Turning OpenAPI Definitions Into A Real Time API Stream

We are working hard to index a variety of OpenAPI definitions as part of the Streamdata.io API Gallery. As part of the work to build a catalog of useful APIs, we gather and craft OpenAPI definitions for each of the APIs we are profiling, as well as the individual API paths that each platform provides. The entire Streamdata.io API Gallery is OpenAPI driven, with all of the static HTML pages being generated from the machine-readable definitions. Allowing us to deliver a human browse-able, as well as a fork-able and integrate-able directory of valuable APIs.

OpenAPI helps us definite the APIs for display in the gallery, but it also helps us integrate with the APIs we are profiling, and turn some of them into real-time streams. Each OpenAPI definition provides us with the technical details around the surface area of an API, which allows us to understand and craft a relevant, and often times topically specific URL that we can turn into a real-time stream. To help demonstrate what we are talking about, let’s take this simple OpenAPI definition from the Stack Exchange API.

swagger: "2.0"
info:
  title: Stack Exchange Get Questions
  description: "Returns all questions for a particular site, possibly with tags"
  version: "2.0"
host: api.stackexchange.com
basePath: /2.2
schemes:
- http
produces:
- application/json
consumes:
- application/json
paths:
  /questions:
    get:
      summary: Get Questions
      description: Gets all the questions on the site. This method allows you make fairly flexible queries across the entire corpus of questions on a site.
      operationId: getQuestions
      parameters:
      - in: query
        name: order
        default: desc
      - in: query
        name: site
        description: The site to return results for.
        default: stackoverflow
      - in: query
        name: sort
        default: activity
      - in: query
        name: tagged
        description: The tag to return.
        default: kafka
      tags:
      - Questions

This OpenAPI provides us with all the technical details for the surface area of the Stack Exchange questions API, which allows us to query for any questions asked across their network of sites. It provides us with the scheme, host, base path, path, as well as parameters needed to craft a useful API call. In this example, we’ve also included default values for each parameter which we can use to populate each of the parameter value and make a specific API call to the Stack Exchange API. From the OpenAPI definition, we are able to programmatically craft this URL to make an API call.

https://api.stackexchange.com/2.2/questions?order=desc&sort=activity&site=stackoverflow&tagged=kafka

Providing us with a simple URL that we can use to get all the questions about Kafka from the Stack Overflow site, and then put to work when we proxy with Streamdata.io with the following code snippet:

curl -v "https://streamdata.motwin.net/https://api.stackexchange.com/2.2/questions?order=desc&sort=activity&site=stackoverflow&tagged=kafka"

This cURL command will turn our Kafka API call into a real-time stream, delivering any questions using Server-Sent Events (SSE), with each incremental update being delivered as a JSON Patch. Turning a simple OpenAPI defined API into a topical stream using Streamadata.io. Moving almost any API we have profiled in the Streamdata.io API Gallery towards being a real-time stream in a single click. Something that we can replicate for each potential topic from Stack Overflow we’d like to turn into a streaming topic that anyone can subscribe to.

The challenge we have now is making these topics accessible in a single click while still needing to authenticate with Streamdata.io as well as potentially each API provider. Stack Exchange is easy because they allow you to make API calls without authentication, but most other APIs will require passing in an API key or token. We are working on a one-click way to authenticate using GitHub, and we’ll abstract away the rest of the authentication with the API and Streamdata.io behind the scenes. We just wanted to take a moment and show how we are using OpenAPI to define each API that we are turning into real-time streams.