We are publishing a bunch of serverless API connectors to the AWS Serverless Application Repository that allow our customers to quickly deploy connectors for common APIs that stream data into Amazon S3 data lakes. We have published one for Stack Overflow, Twitter, Reddit, Hacker News, and Stock Market data. We have a long list of other connectors to publish, but we are needing to automate and streamline the process a little bit using the OpenAPI definitions we’ve published to the Streamdata.io API Gallery, before moving too far forward.
Each API we’ve profiled and StreamRanked™ has an OpenAPI definition behind it. Providing us with the technical details of each API. This OpenAPI definition provides us exactly what we need to generate each of the AWS Lambda functions we will be deploying as a connector in the AWS Serverless Application Repository. Helping us generate:
– Description – Generate the title, description, and other text we will need to publish the README, and AWS Serverless Application Repository profile.
– Github Repository – We will create a new repository, build a README, and commit to Github for managing each individual connector, allowing us to also use GitHub issues for the support and road map.
– Lambda Function – Taking a template Lambda function, we’ll auto generate the Node.js function using the template, but then generating the details of the API integration using the paths, parameters, security, schema definitions from the OpenAPI definition.
– AWS Serverless Application Model (SAM) File – One of the things you need to publish a Lambda function to the AWS Serverless Application Repository is a SAM file, which we’ll generate using the details available in the OpenAPI.
The OpenAPI definition for each API will be used to generate all the artifacts needed for publishing a streaming API integration serverless connector to the AWS Serverless Application Repository. Each connector starts with a base template, but then will be customized base upon the OpenAPI for each of the APIs we are publishing a connector for. Allowing us to somewhat automate the deployment of serverless streaming connectors for a variety of popular APIs.
Our objective is to have a suite of serverless streaming connectors for all of the APIs that have a high StreamRank. Meaning, if an API potentially generates a lot of data, we want there to be a plug and play connector in the AWS Serverless Application Repository for it. Augmenting the value of each API we profile in the gallery, getting us closer to a one click deployment of popular alternative data sources, when it comes to streaming them into your AWS data lake.
