It can be difficult to keep up with where things are at with the Twitter API over the years, as they’ve changed things up. So we wanted to take a look at where things are at with the Twitter Streaming API to understand how they operate, and document where the opportunities are for Streamdata.io to be applied, or not. The state of the Twitter Streaming API has shifted over the last couple of years because of the Gnip relationship, and is something that seems to be moving to be more about their enterprise offering, but there is still one remaining, publicly available streaming endpoint that we would like to get to know better.
The Twitter Streaming API returns public Twitter status updates by filtering for search phrases, user IDs, and by location. The endpoint supports both GET and POST requests, to avoid longer URLs, when you are looking to stream a large number of user IDs. The Twitter Streaming API return public statuses that match one or more filter predicates. Multiple parameters may be specified which allows most clients to use a single connection to the Streaming API. Both GET and POST requests are supported, but GET requests with too many parameters may cause the request to be rejected for excessive URL length. Use a POST request to avoid long URLs. They allow for filtering up to 400 keywords, 5,000 follow userids and 25 0.1-360 degree location boxes. If you want anything more you have to upgrade to enterprise access.
You use these three parameters to setup filters for the stream:
– follow – A comma separated list of user IDs, indicating the users to return statuses for in the stream.
– track – Keywords to track. Phrases of keywords are specified by a comma-separated list.
– locations – Specifies a set of bounding boxes to track. See locations for more information.
Twitter provides a lot of information about how to keep a connection stable, and how to avoid violating rate limits, to help ensure a stable(r) experience. It is clear that Twitter is still working to figure out the business model, and enterprise strategy between the free layer of Twitter, and their Gnip acquisition, and former partnership. While the Twitter Streaming API provides an avenue for obtaining garden and firehose streams of Twitter users, search, and location data–for delivering more precise streams of individual user, lists, and other searches across the Twitter platform, we still see lots of opportunity for Streamdata.io to be put to use.
Equipped with our understanding of the potential and limitations of the Twitter Streaming API we will keep playing with different ways to proxy the Twitter web API using Streamdata.io, working to deliver more precise, last mile streams that can be delivered via web and mobile dashboards, as well as trickled into training machine learning models. Once we’ve explored all the possibilities, we will publish a more complete guide to streaming via the Twitter platform using Streamdata.io. Demonstrating alternate opportunities for delivering social streams from the Twitter platform.
