Streaming Twitter Lists Using Streamdata.io Instead of the Twitter Streaming API

I was playing with streaming different APIs, and I found myself playing with the Twitter API. Which is kind of senseless as Twitter has their own streaming API, but it can be a little more investment to get up and running with both technically, and in a businesses sense if you are needing higher volumes, than Streamdata.io. Technically Twitter is just using a long-lived HTTP request, establishing a persistent HTTP request, and making data available in a regular stream of HTTP responses. I’ve integrated with the Twitter Streaming API before, but I wanted to understand the technical details of how its different than using Streamdata.io, and HTTP Server-Sent Events (SSE) and JSON Patch.

To stream a Twitter list using Streamdata.io all I did was use the URL for the Twitter lists statuses API https://api.twitter.com/1.1/lists/statuses.json, with two query parameters.

– slug=api

– owner_screen_name=kinlane

Then you just make sure and include in a Authorization: Bearer [token] with each request. At first I passed the header in with call I made to Streamdata.io.

curl -v "https://streamdata.motwin.net/https://api.twitter.com/1.1/lists/statuses.json?slug=api&owner_screen_name=kinlane&X-Sd-Token=[streamdata api key]" -H "Authorization: Bearer [token]"

However, once I saw that it worked, I added the authorization header to my Streamdata.io settings so I can keep it out of the code, simplifying each request I am making to Streamdata.io.

curl -v "https://streamdata.motwin.net/https://api.twitter.com/1.1/lists/statuses.json?slug=api&owner_screen_name=kinlane&X-Sd-Token=[streamdata api key]"

The result is a pretty efficient stream of Tweets from any accounts on the Twitter list I maintain for the API industry. At first I get the entire response from the API, but then I just get an incremental JSON Patch update with any additional Tweets. Providing a pretty simple stream of Tweets without the overhead of using the Twitter Streaming API. I just need to adjust the polling frequency in my Streamdata.io settings to make sure I am not going to go over my Twitter rate limits, then I have myself a pretty robust streaming and cached source of my Twitter List.

Now, why would I want to do this? I’m not 100%, which is why I’m writing this story. First, I guess it is quicker and easier to implement than the full Twitter Streaming API. Second, it provides a nice cached endpoint that you could reuse across applications, systems, and teams I guess, making efficient use of your Twitter API rate limits. Beyond that, I guess if you have serious volume needs for Twitter data, you probably would want to go with their solution–as you get volume rate limits. I’d say Twitter’s Streaming API isn’t as advanced as using Server-Sent Events (SSE) and JSON Patch–using a persistent connection in HTTP/1.1 just isn’t as efficient as HTTP/2, or Streamdata.io’s approach.

Regardless, my approach provides an interesting look at how you can stream using a pretty well known API resource–Twitter. I’m going to mimic my Tweetdeck client using just Twitter lists, and try to bypass the Twitter algorithm a little bit. I’m kind of over the full stream of my Twitter account(s), and prefer to rely upon the Twitter lists I’ve created. I’m also kind of over Tweetdeck as a client, and would like to see how I can create a bare bones client using Streamdata.io, and take charge over my social streams a little more. Then I am going to try streaming some other Twitter endpoints and see what is possible there, maybe taking streaming via the Twitter platform in a different direction that where the Twitter Streaming API goes.