API is dead — long live the APIs

In this article, I will highlight how the glorious monarchical reign of REST (Representational State Transfer) APIs is declining and how the ecosystem is moving towards democracy. Today the API consumers decide what the format or protocol of the API should be.

History lesson

As I’d previously described in the blog, “There’s gold in them thar hills,” within an enterprise, there are layers upon layers of technologies deployed over time. Many of the older technologies still bring value, making it far from obsolete. Consider that even though many enterprises are undergoing digital transformations, the technology and associated data that is contained within their old and deep layers still provide value.

They cannot simply be removed, because after all it is the layers beneath that create the foundation for new pillars to be placed. Many of these layers will often have legacy interfaces such as CORBA, RMI, SOAP, etc.

So, why do we continue to add layers on top of layers? If we approach this question as a paleontologist we would take a core sample and analyze each layer; let’s try this approach with our technologies and dig through the various layers. Each layer represents a historical view, on the topmost/outer layer you’d find REST which is the most pervasive technology in place today.

Just under REST, you’d hit the Service Oriented Architecture (SOA) layer. Let’s stop there and ask, why did REST replace SOA? While there are many reasons, arguably the biggest reason is that REST greatly simplified the problem of client accessibility – with REST, JSON not XML is exchanged making it simple for use in the browser.

In addition, REST has a simplified approach with its verbs GET, PUT, POST, etc. allowing for simple generic toolings such as cURL and Postman to be used. Creating your own client in virtually any language becomes straightforward.

On the other hand

With SOAP (Simple Object Access Protocol), there was nothing simple about client accessibility. SOAP was defined with the enterprise in mind and while the protocol is platform agnostic, SOAP uses XML which makes it verbose and painful to use in the browser.

In addition, the layers of complexity added by the WS-* standards meant that it was easy to write the server-side but painful to write the client-side. The client implementation required the use of third-party tooling to generate client code in your language of choice which created burdensome dependencies.

As you dig deeper into the technology layers, you will find similar stories. SOAP once was king, and it replaced technologies such as CORBA and RPC. REST claimed the throne from SOAP and today The REST throne is being challenged from calls for democracy.

Who are the challengers to take the throne from REST? There are several aspirants or claimants to the monarchy of the REST throne. The claimants can be divided into three categories: “server to client”, “event-based” or the “data access” family. Just like in Game of Thrones the challengers could be divided into their family lineage (House Targaryen, House Lannister, House Stark, etc.)

The House of “Reverse API”

This house’s sigil is a phone, as it is the server that will call the client and not the reverse.

Webhooks: Webhooks are sometimes referred to as “Reverse APIs” because the initiator of the call is the server rather than the client; the server notifies clients of the change in state. Client applications register a call back URL where they can be reached when there is a state change of a resource on the server.

The client application will be notified with an HTTP payload of the change of state. Webhooks are popular event-based mechanism found on the web (GitHub, Stripe, Twilio). They are not a silver bullet as they don’t provide guaranteed delivery, rarely retry, can get out of sync, and can put a lot of pressure on the client when the volume of events is high (backpressure mechanism required).

WebSockets:  WebSocket is a transport protocol defined by a persistent, bidirectional communication channel between server and client that takes place over a single TCP socket connection. WebSockets are a good choice when the server to client or client to server requires near real-time exchanges. This overcomes the limitations of HTTP’s request/response mechanism (i.e. client polling and associated overhead). WebSockets are suitable for applications that require real-time updates as the server can push data at any time on the connection. WebSocket is supported by all modern browsers.

SSE: Server-Sent Events (SSE) is a server push technology enabling a client to receive automatic updates from a server via an HTTP connection with automatic reconnection mechanism built-in. The Server-Sent Events EventSource API is standardized as part of HTML5. SSEs are sent over traditional HTTP which means they do not require a special protocol or server implementation to get it working.

WebSockets on the other hand, require full-duplex connections and new Web Socket servers to handle the protocol. SSE is suitable in scenarios where data doesn’t need to be sent from the client, but you need updates from some server action (i.e. stock ticker, rideshare updates, friend’s status update, etc.).

The House of “event-driven”

This house’s sigil is Kafka’s icon as it’s the “event-driven” dominant player.

This house is dominated by three characters, namely the two messaging systems based on the use of a commit log Apache Kafka and NATS, and the message queue of RabbitMQ. These technologies have become the de facto backbone for microservice architecture based on asynchronous events. Architectures based on event messaging result in extremely loose coupling between services and are useful for event-driven systems.

With these architectures different design patterns such as CQRS (Command Query Responsibility Segregation) and Event sourcing are popular. Messages sent to the queues are defined in Proto Buf, Avro or Thrift as JSON is too verbose and the events will be consumed by internal microservices.

As event-based systems are asynchronous in nature, the system itself will become eventually consistent. This results in developers having to handle inconsistent data between services. Depending on the messaging backbone in use, the handling of receiving duplicate events or out-of-order or missing events (guaranteed delivery) may have to be taken into account. The tooling for event-based development (design, debugging, testing, monitoring) is more immature compared to traditional REST API.

There are two emerging projects that are worth tracking and contributing to:

CloudEvents project incubating in the Cloud Native Foundation (CNCF) is attempting to define vendor-neutral event format for exchange between different cloud systems.

AsyncAPI whose goal is to make Event-Driven Architecture easier and as familiar to developers as REST APIs – which would encompass documentation, code generation, discovery, and event management.

The house of gRPC

gRPC there are some clues in the name; “g” from Google and RPC as it is designed for Remote Procedure Call (RPC). It is a top-level project in CNCF. gRPC is normally used internally in your organization  (i.e. inside a microservice architecture) where you control the consumer and the provider. gRPC is highly performant as it utilizes HTTP 2.0 and doesn’t suffer compared to traditional HTTP 1.1 REST calls. Service definition is defined using Proto Buf and from this, client and server-side stubs can be generated in a host of languages. gRPC supports bi-directional streaming and a pluggable mechanism for authentication.

The house of “GraphQL”

GraphQL: GraphQL APIs lend themselves to mobile apps and help avoid the chattiness that is seen with REST APIs. With GraphQL, the client determines what data they want, how they want it, and in what format they want it, rather than walking multiple REST requests/responses to find the resource the app requires.

This results in a reduction in the number of networks call clients to need to make and ensures that they are only retrieving the data they need. This provides a better user experience if the client is a mobile app or a single-page application (SPA). The spec for GraphQL and a reference implementation was publicly released in 2015 by Facebook. Today, the stewardship of GraphQL is under the GraphQL Foundation.

The house of “OData”

OData: OData (Open Data Protocol) is an OASIS standard that defines the best practice for building and consuming RESTful APIs. Initially developed by Microsoft in 2007. OData is a set of common conventions on top of HTTP, REST, and JSON.

These conventions, if adopted, mean that API providers end up with a standard and consistent way to deal with aspects such as querying, paging, sorting, and filtering APIs. It is due to these conventions that if you’ve seen one OData service then you’ve seen them all and implementing consuming applications becomes easier. OData APIs are often seen in ecosystems where Microsoft and SAP services are running.

The house of “IoT”

Given the explosion of IoT devices, this house is firmly inside the billionaire club.

MQTT (Message Queue Telemetry Transport):  is an ISO standard publish-subscribe-based messaging protocol. It works on top of the TCP/IP protocol. It is designed for connections with remote locations where a “small code footprint” is required or we limit the network bandwidth. So, it is a perfect protocol for clients such as embedded systems where bandwidth is costly and minimum overhead required.

AMQP (Advanced Message Queuing Protocol) is one of the most popular protocols for asynchronous communication. RabbitMQ and ActiveMQ are some popular implementations or there are hosted solutions like AWS MQ. AMQP is an open standard application layer protocol for message-oriented middleware.

The defining features of AMQP are message orientation, queuing, routing, reliability and security. AMQP has more features when compared to MQTT and as a result, it is more suited to run on client systems requiring higher processing and memory requirements. Thus, it is seen as a rather heavy protocol that does not play well in the sensor/embedded world of IoT.

Conclusion

As this article has shown the monarchical reign of REST APIs/Services is in the past. The required ecosystem of API technologies will be dedicated to the needs of the consuming client applications. There will not be one single winner in the ecosystem, but many winners depending on the experience required by the client. No one technology will sit on the throne – it’s a democracy and clients decide who will represent them!

API is dead — long live the APIs!