Streaming Data From APIs To Github Repositories

We are playing around with different ways of streaming data from common APIs, to demonstrate what is possible with Streamdata.io, but also looking to push the boundaries of what is possible.  A couple of weeks ago we setup a simple project for streaming news using the NewsAPI, and now we wanted to take the concept a little further, by saving the streams to Github. The project already runs 100% on Github, using Github pages, but now we wanted to use the Github repository as storage for each update from the news stream.

To accomplish this, we pull data from the NewsAPI, by proxying it with Streamdata.io, and then we use the Github API to write to the project’s underlying repository. Making for an interesting solution we call “stream to Github”. Here is a copy of the full HTML and JavaScript which will run 100% client-side anywhere, but we recommend running it using Github Pages, to make for a completely standalone solution–no hosting needed.

This script is pulling from the NewsAPI, but could easily be modified to pull from any API. Then taking the resulting updates and write them to the underlying repository. So why would you want to do this? Well, maybe to just save the results for later, but with the data stored on Github, so it can either be made publicly or privately available via a Github repository that could be forked and integrated into other applications. Leveraging Streamdata.io as a way to trickle or stream data from existing APIs into Github repositories that can then be used to build other applications, to create dashboard visualizations, or other types of integrations using the valuable API-driven data.

We will play around with other ways of streaming data, and even aggregating data to Github in this way. Maybe we will take some of the other 311, 511, and 911 data we’ve been pulling, and publish the results to Github repositories using this approach. Maybe we’ll take our Github RSS aggregator solution and transform, then publish the results to a Github repository, making some sort of leaderboard for different segments of the API sector. We don’t have any particular production uses in mind for this approach, we are just looking to think out of the box, and play around with different ways of using Streamdata.io, and find some interesting ways to use the APIs we are coming across in our profiling of the API space.