I’ve been scaling up my monitoring of Github as the API Evangelist. The platform provides me with a wealth of information regarding what companies and individuals are up to in the API space. Each Github user possesses an RSS feed via the platform, making it pretty easy to tune into what any individual user is up to via the platform. I wanted to build me a dashboard for monitoring activity across the platform using Github activity feeds, but as soon as I got to work, I saw there were two major hurdles I needed to get over before I could deliver the real time dashboard I envisioned.
1) Aggregation – I didn’t need a single user, I needed to monitor hundreds, and even thousands of users–I would need to be able to aggregate multiple feeds into a single.
2) No JSON – Github provides RSS feeds for each user on the platform, and to be able to proxy with Streamdata.io, I would need them to be JSON feeds.
To help me achieve my goals I got to work developing a simple RSS aggregator, that would also convert the XML returned into simple JSON. I built my prototype in PHP, but will also be replicating in Node.js, and Python pretty quickly so that I can deploy on a variety of platforms, including as serverless functions running on AWS Lambda.
This script will accept as many Github users as you care to add, then it loop through each one pulling their RSS updates, combine into a single array which it then sorts, and outputs as a simple JSON feed. You can see a demo of this in action where I pull my Github user, as well as apievangelist and streamdataio–providing a simple RSS aggregator and converter. Now I can take this URL and proxy with Streamdata.io, providing me with JSON Patch updates across these three Github accounts using Server-Sent Events (SSE). I’ll still need to stabilize my script, and add a monitor to it to make sure it stays up and running, but it should provide me with what I need to stream Github updates to my monitoring dashboard for the API sector. I am going to organize Github users by industry, and create different leaderboards for each industry I am tracking on as part of my work as the API Evangelist.
Hopefully this article about Github activity feeds provides you with an example of how you can quickly augment existing RSS feeds with JSON APIs, and then aggregate and convert to JSON, which can then easily be made real time using Streamdata.io. We are a big fan of keeping Streamdata.io doing one thing and doing it well, and leave it to people like you to help us create the dead simple JSON APIs which can act as feeds for streaming APIs published via our service. The reports of RSS’s demise have been greatly exaggerated, and there are endless opportunities to publish, aggregate, and translate RSS into more meaningful streams, delivering real time news and other content where it matters.
Follow us on social