We are creating AWS Lambda functions for a variety of Twitter APIs, which allow us to stream Twitter data into Amazon S3 data lakes. We’ve already made one for Twitter searches, allowing anyone to stream a specific search into an AWS data lake. As part of this work we wanted to share the entire list of Twitter API paths we are developing connectors for and will be publishing to the AWS Serverless Application Repository so that anyone can begin streaming data into their data lakes within their own AWS infrastructure.
As part of our profiling of Twitter for the Streamdata.io API Gallery, we’ve identified 15 separate API paths as having a potential for streaming, depending on the user, as well as the use case. Providing us with the Twitter API resources we wanted to make available via serverless streams using AWS Lambda:
– Get Followers
– Get Friends
– Geo Search
– Get Members
– Get List Subscribers
– Get List Subscriptions
– Get Saved Search
– Search Tweets
– Status Timeline
– Mention Timelines
– Show Retweets
– Show Timelines Status
– Show Available Trends
– Show Closest Trends
– Show Place Trends
As we said, we’ve already turned the Search Tweets path into a streaming connector, but we should have the rest of these ready by the end of the week, and published into the AWS Serverless Application Gallery. To operate each one you will need a Streamdata.io account, as well as a Twitter account, however, once you have your Streamdata.io key, and Twitter OAuth token, all you have to do is deploy the function to your account, enter the values, and you can begin streaming. It is up to you how long you run the streams for, and orchestrate them using AWS Cloudwatch events, where you can run indefinitely, or based upon a schedule or other events.
Streamdata.io provides an alternative for the recent deprecation by Twitter of their streaming endpoints. You can proxy the Twitter REST APIs using Streamdata.io and deliver to multiple web or mobile clients, or you can connect and stream into an Amazon S3 data like, and use the data to train machine learning applications and other applications. Our goal with this work is to make all APIs as plug and play as possible, allowing anyone to quickly integrate with a variety of 3rd party APIs using their existing infrastructure. Making AWS Lambda a pretty compelling approach to providing plug and play connectors that are deployable with a single click, and begin streaming into your existing data lakes for use across a variety of existing applications.
Follow us on social