What you need to know about AMPLIFY Streams

AMPLIFY™ Streams SaaS is an event-driven proxy that can sit in front of an existing JSON API to provide a reach reactive experience in client applications (i.e., dynamic stock ticker, etc.). It queries the API, caches the response to computing the list of changes against previously requested data and finally pushes the deltas to the clients. It significantly reduces network bandwidth consumption and load on API servers while increasing the user experience with living data.

AMPLIFY Streams information: Getting started?

By simply creating an account on the Streamdata.io by Axway Portal, you are on your way to getting started. Once registered, log in and select the default app. You can now try to stream your first API with provided cURL command in the getting started section.

If you do not receive your “My Account” creation confirmation email, check your spam after registration for account validation.

Transmitting data to clients connecting to AMPLIFY Streams proxy

To push data to the clients, AMPLIFY Streams relies on HTTP/Web standards:

• It relies on Server-sent events(SSE) as a transport protocol. SSE is a server push technology enabling a browser to receive automatic updates from a server via HTTP connection.
  • The Server–Sent EventsEventSource API is standardized as part of HTML5 by the W3C.
• It relies on the JSON-Patchstandard to send the deltas in the form of a list of patches. There are multiple implementations for different languages which can be found under Libraries section of this page.

 Most of the modern browsers support Server-Sent Events natively by implementing the EventSource specification. You can find an exhaustive list of browsers supported here.

Microsoft Internet Explorer (IE) is the only popular browser not supporting SSE natively, but there is JavaScript polyfill that implements the EventSource API allowing us to work around this limitation. A polyfill is included in the AMPLIFY Streams JS SDK.

Wondering if it’s necessary to use a client JS SDK? We strongly recommend to customers to use an SDK that implements an SSE polyfill to make sure that your client application will work on all browsers, including those that do not support SSE natively:

Finding integration examples of AMPLIFY Streams with HTML5 UI frameworks

You can find integration samples with different UI frameworks such as Angular or VueJS here. Furthermore, if you need to know how to connect to AMPLIFY Streams from a nodeJS application, you can find an example here.

Can I connect to AMPLIFY Streams from a server-side App?

Yes, the only thing needed is a library that implements SSE spec and a library that implements JSON-Patch spec.

There are many implementations of SSE (EventSource API) for numerous programming languages.

Visit Wikipedia to find the one that suits your needs and environment. Same thing for the JSON-Patch. You can visit here.

You can try this easy-to-use SDK with some demo code on Github.

Handling errors in a JS application

In a JavaScript Application, both types of errors (SSE connection errors and streaming errors) must be handled in a centralized SSE error-event listener.

The connection must be explicitly closed by the client to release properly the allocated resources and avoid memory leaks or unnecessary server-side resource consumption.

JavaScript code snippet

eventSource.addEventListener('error', function (event) {
  console.log('ERROR!', event);
  disconnect();
  //Handle streaming errors here.
});
function disconnect() {
  if (eventSource != null) {
    eventSource.close();
  }
}

Best practices for handling timeouts

There is no timeout mechanism specific to EventSource implementation. You may want to use the setTimeout/clearTimeout functions which are the standard JavaScript APIs for handling timers.

If you get an error that is not documented, send the error with the replication scenario and ID of the error to the support team.

Does AMPLIFY Streams support target API with a self-signed certificate? The answer is No. The target API must either be not encrypted or encrypted using a signed certificate from a trusted CA. Contact the support team if you need to check Cipher Suites’ compatibility.

Typical error message when trying to stream an API protected by a self-signed (or expired) certificate:

id: "de7fd899-f92e-4036-b16d-f8aa197d714a"
event:error
data: {
    "status": 2001,
    "cause": "An error occurred while streaming https: //self signed.badssl.com. : General SSLEngine problem",
    "message": "General SSLEngine problem",
    "timestamp": 1558345480266,
    "sessionId": "565eb8d5-d872-4308-9a76-ff9dae50089f"
}

Where are AMPLIFY Streams currently hosted?

AMPLIFY Streams is currently hosted on Google Cloud Platform (Europe-west1) distributed across three different availability zones. AMPLIFY Streams can stream a secured API, but depending on how the target API is secured, you can face limitations, for example:

• • If the target API is secured with a static API Key, the API Key can either be provided by the client as an additional request header upon Streams connection or directly injected from the AMPLIFY Streams platform. You inject an additional custom query parameter and headers to the API request by selecting  Configure under My APIs from AMPLIFY Streams Portal.

• • If the target API is secured with a dynamic token (e.g. OAuth), the process of retrieving the token must be handled in the client application code before the establishment of the Streams connection. Once the client application has successfully retrieved the token, it can open a Streams connection bypassing the token to allow Streams to fetch the data from the target API. When the token expires, Streams can no longer request the target API and will throw an error down to the client app. It is then up to the client app to implement a logic that reacts to this error by refreshing the token and re-establishing the connection to Streams.

IMPORTANT: The clients are subscribed to a topic based on the elements of their subscription request, including the URL, query params and HTTP headers. If two requests targeting the same API differ in one of its elements (e.g. additional query param/header, a different value for query param/header), clients will not be subscribed to the same topic leading to different polling requests against the target API (one polling request per topic).

When will AMPLIFY Streams work for private APIs on the Streams roadmap?

Usually, most of the private APIs are secured using OAuth, AWS v4, API Key, Client Cert, etc. APIs like stock lookups are public. Anyone can access them. The ideal use case for Streams is to front non-private APIs as it allows to take advantage of its internal caching and fan-out capabilities. Let’s say you need to serve 1,000 subscribers/clients that are all subscribed to the same resource, Streams will only “poll” the target API once for all subscribed clients. This leads to a significant load reduction for the infrastructure that hosts the target API as it does not have to serve 1,000 clients querying the API directly. Having said that, we should be clear with the notions of public/private resources vs. protected/unprotected APIs.

For example, let’s look at Xignite, who is a Cloud Market Data provider. Some of Xignite’s APIs are protected by an authentication mechanism but provide access to a “public” resource—public in the sense that any authenticated client will receive the same responses. Streams have limited support for protected APIs in the version currently in production (aka SaaS v1) with two different options for two different personas:

1. 1. API Provider: When creating a Streamed version of an API in Streamdata.io Web Portal, the API Provider can add an authentication header that will be injected in all polling requests issued by Streams against the target API. This is transparent for the clients (API Consumers) that subscribe to Streams as they must only provide the Streamdata.io AppToken to subscribe. The limitation here is related to the fact that the header injection can only be configured manually via the Web Portal UI. Streams (v2) will provide an Admin API that will make it possible to do it programmatically, useful when the token needs to be refreshed (e.g. OAuth tokens).

2. 2. API Consumer: The client can perform the authentication handshake to get a token (required to authenticate against the target API) before establishing the connection with Streams.

He can then initiate a Streaming session by providing the token as an additional header that will be forwarded by Streams to the target API.