A Serverless Streaming API Prototype Using AWS Lambda

We have begun working on a new line of Streamdata.io connectors for the AWS ecosystem, with the current phase being all about delivering serverless data streams. We’ve delivered our first serverless proof of concept using our Node.js, which delivers a server-sent event (SSE) enabled data stream using Lambda. Taking our existing stock market data demo API, and turning it into serverless streaming that can be fired upon command, on a schedule, or based upon specific events that occur across your AWS-driven infrastructure.

The purpose of this serverless streaming API prototype is to demonstrate how data streaming can be delivered and scaled in the cloud. Serverless approaches provide us with a new way to look at how we deliver and scale the code behind our applications, and provide a powerful new way to deliver real-time streams of data for use in applications, data lakes, and machine learning models.

Helping us focus on the data we are streaming, and leaving the streaming portion of the conversation to Streamdata.io, and the scaling and delivery of the streams to wherever we need it to AWS. Providing a much more modular way to look at how we can fire up data streams.

Like any other environment, delivering data streams using Lambda comes with some constraints–you can only run a Lambda script for up to five minutes at a time, which means the stream will disconnect at that point, but with AWS Lambda there are many ways in which you can retry and fire back up the stream as soon as it disconnects.

You can also define rules, schedules, and other events that help you get more precise in how you fire up a data stream than you can in other environments. Providing a much more modular, flexible, configurable, and scalable way to fire up data streams in the cloud. Making any constraints seem irrelevant when you consider what is possible when it comes to data streaming in a serverless world.

We are working to publish our prototype streaming API to the AWS Serverless Application Repository, but we have a bit more policy work to configure before we are ready. For now, you can download the entire package, complete with any Node.js dependencies as a zip file on Github.

All you need to run the prototype is to create a new Lambda script in your AWS account, import the package, and you will have what you need. You will need to add your Streamdata.io app key to the script, something we’ll configure as environment variables in future editions. We just wanted to get the prototype package out there, so that other people can begin playing with it, as we are developing the next version(s).

This edition doesn’t have any sort of data handling or process for the data stream–it is just a functional prototype. We are working on publishing data using AWS S3, RDS, GitHub, and other locations, as well as potential Webhook editions, so stay tuned. If you have any questions on this prototype, feel free to submit an issue via the GitHub repository for the project, and we’ll see what we can do to help.