Making Subscribing To Data Streams From Existing APIs As Turnkey As We Can

We have assembled over 10K existing API resources, in over 400 topical areas, from over 300 entities in the Steamdata.io API Gallery so far. We’ve benchmarked a number of them to help us identify the ones that have the highest potential for turning into API streams. No doubt, there is a wealth of data available in the gallery, but now we are facing the challenge of making it as usable as possible when it comes to turning each API into real-time streams that you can subscribe to and feed into data lakes, train machine learning models, and the wealth of other reasons our customers are turning to us to satisfy their data appetite.

It isn’t easy making common public APIs plug and play. We’ve done a fine job to define the surface area of the API using OpenAPI, making the API something we can easily connect to using Streamdata.io or another client. We know where to find the host, base URL, paths, parameters, headers, and other technical details. Some APIs make these technical components more difficult than others, but for the most part, we have this all mapped out and easily made into plug and play connectors that play nicely with Streamdata.io. Where we start having challenges in truly making the APIs into connectors, and something that can easily be connected to is when it comes to the business and security of working with APIs.

What we find to be the most challenging aspect when it comes to working with APIs is that our users have to often times sign up for accounts, obtain API keys, and navigate the billing side of API consumption. Some of the APIs we work with make it easier for us to make the signup and consumption models seamless, but many APIs see the developer or user subscribing to the data as their customer, and they want to capture that account, have them agree to their terms of service, and handle the billing directly. Adding to the complexity these companies often times do not provide any automation when it comes to account, terms of service, and billing provisioning. Requiring seamless integration of all developers and end users, but providing very little support to make it something that is truly easy to do.

In the end, it is tough to make some APIs truly plug and play streams for our customers. Not because of the technical limitations, but because of the business constraints. API providers have insisted on talking to sales teams, manually signing up for services through an interface, and having the direct legal and billing relationship with customers. Adding potentially minutes, hours, or sometimes days to the on-boarding process to subscribing to new data sources. Most of our customers are just going to look elsewhere when it comes to finding streams of data, and won’t jump through the hoops. However, the APIs that we are able to make subscribe-able with one click, abstracting away the business complexity will see an increasing adoption from the API Gallery.

We’ve begun rating APIs based upon the friction involved with getting up and running. Eventually, we will expose the ability to filter these APIs out of the gallery, leaving only the plug and play APIs available for adding to a sort of streaming data shopping cart. Helping data consumers understand not only which APIs have the highest volume data flows, but their providers have chosen to get out of the way and allow for seamless consumption of their valuable resources. As we continue profiling APIs for inclusion in the API Gallery, the overall API landscape continues to come into focus, showing us that API discovery is much more than just a technical thing, and that the business and politics of APIs is really what makes the difference in the end, when you are just trying to put API resources to work in your applications.