Streaming The Github API Rate Limit API

We are always interested in learning how Streamdata.io can be put to use helping API integrators better understand how to monitor their applications. One use case we came across recently focuses on being able to monitor your Github API rate limit API, but as a stream, instead of just via the regular Github web API. It is straightforward to proxy the Github API using Streamdata.io, making it pretty easy to turn the existing Github API rate limit API into a realtime stream that you can tune in via web and mobile applications.

As part of our work on the Streamdata.io API Gallery we’ve been heavily using the Github API to build out the pages, and its machine-readable core. Making it pretty critical for us to be able to tune into API rate limits, and how much consumption we’ve conducted on a day to day basis. Giving us the idea to start streaming the Githhub API rate limit API so that we can empower humans, and our systems to make decisions regarding our API consumption. Here is the first cURL command we implemented to turn the API into a stream using our proxy:

curl -v -h "Accept:{header_value}" -h "User-Agent:{user_agent}" "https://streamdata.motwin.net/http://api.github.com/rate_limit?X-Sd-Token={streamdata_token}&access_token={access_otken}"

This cURL command allowed us to quickly start streaming the rate limit for our API accounts, helping us stay in tune with our API consumption as we are running different jobs, and building out the Streamdata.io API Gallery. Allowing us to horizontally scale out our publishing to the gallery, and spread the API calls out across days, as well as different team Github accounts. As we continue to progress with our work, we are formalizing how we are using the Github API rate limit API, and baking it into our platform so that we can further automate the publishing of APIs to the gallery.

This aspect of our work demonstrates the value of turning the APIs we depend on into streams, and baking them into our operations so that we can automate decision making around our API consumption. Ideally, Github would offer streaming editions of all their web APIs, allowing consumers to choose from between both a request and response model, as well as a more sustained streaming approach–depending on the need. We are continuing to benchmark the Github APIs, understanding which ones make for ideal streams, and will help us improve how we automate our infrastructure. As we continue to expand our approach, we’ll publish more stories here on the blog, and incorporate in our workshops and white papers.