Enterprise Comparison of API Gateways and ESBs

It is not uncommon for customers taking their digital transformation (DX) journey to come to a point where they start looking into API Gateways and API Management (APIM), often before they get to a point where they are working heavily with APIs. There is no question about the value of APIM for API Lifecycle Management, but at the earlier crossroads the question is often asked,

Why do I need an API Gateway (APIGW) if I have an ESB (Enterprise Service Bus)? A lot of the functionality looks the same!

If you are in the same boat, you are not wrong.

API Gateways and ESBs – what does my enterprise need?

On the surface, an API Gateway and an ESB share many of the same capabilities and much of the same functionality. In fact, I have often heard API Gateways referred to as a “lightweight ESB” when describing a subset of their functionality. But then, why do you need an API Gateway if you are not ready to embrace an API program? What is the value of the API Gateway to an Enterprise Application Integration (EAI) strategy that is already ESB centric? To understand the value, let’s first talk about digital transformation and some developing requirements we are seeing today.

Digital transformation (DX) and the ESB

ESBs have long been a key workhorse in performing EAI tasks. They have provided an abstraction layer, allowing the orchestration and integration of disparate A2A (application to application) and S2S (system to system) interfaces and flows, focusing strongly on exposing services in a normalized fashion and reuse. This tooling has worked great for many years, but in recent times we are seeing some paradigm shifts that are changing the needs of enterprises around the world.

A move to consumption centric use

Ever been to New York City? If so, you probably have used their elaborate tunnel system. It can be costly (I have the expense reports proving it!), but ultimately as a purpose-built solution to get from point to point, it is the best option in town. But what about if I want to get somewhere a tunnel doesn’t lead? Once I am out on the other end, there is a whole city to explore beyond where the tunnel gets me.

There are landmarks, shows, shopping, and more beyond the end of the tunnel. Restaurants are a staple in NYC, and they are opening and closing all the time. Maybe I want to eat at the same old place or maybe I want to go somewhere new. Maybe I want to go to that same old by-the-slice pizza shop that I normally do, but then want to try something new and finish off my meal with some cannoli from the coffee shop around the block. Maybe I am in a different part of town than normal for a conference, so I want to explore a whole new dining experience, rather than trying to hop back to the boroughs I know.

Enterprise Application Integration (EAI) is a lot like this experience. There is no doubt that traversing New York without using the tunnel system can be an exhausting experience, and by connecting the boroughs with the tunnels it provides connectivity and promotes growth, much like an ESB in an EAI strategy. But what if the tunnel doesn’t take me where I want to go? Should I submit a request and wait until a new tunnel is built? Is that even the right decision for the city given the cost to maintain it? How difficult would it be to move the exit of a tunnel, and what type of disruption would it cause?

Okay, maybe that metaphor is getting a bit extreme, but you get the point. Tunnels are a great way to get point to point and provide value to city infrastructure, but ultimately to consume the services of a city, I need a grid to city streets to get me where I want to go. I don’t want to have someone tell me I can only consume what is right at the exit of the tunnel. I want to explore and try new things as fast as I can and read an article on why I should have my experience and ability to do so not hampered by waiting for a new tunnel.

As enterprises look to accomplish the following simultaneously:

• Facilitate more productivity from their developers
• Enable omnichannel experiences and new ways to monetize their data/services and even more internal reuse of systems and applications

We have seen the rise of the consumption-centric use case.

For many years enterprises have been operating as an Integration Factory, instead of an Integration Enabler. This environment means that when things needed to be hooked up, the responsibility fell on IT to do it, and it has been manageable for a long time.

Today, however, things are looking a little different. We have seen the explosion of Hybrid, with many new:

• Technology domains (e.g. Content Collaboration Platforms)
• User personas (ad-hoc integrators, digital natives) and
• Endpoints (sensors, IoT, mobile devices, external Cloud Services).

These new personas are bringing an exponential amount of new human-driven technology requirements to connect all these endpoints to an ever-growing set of technology domains, while at the same time the technology domains become less siloed and more integrated.

There is a focus on:

• Ease of access
• Self-service and rapid prototyping to support the fastest time to market to enhance revenue
• High productivity and
• Continuous innovation

Having IT be a one-stop-shop for providing every integration point is no longer a viable strategy. The ESB ultimately becomes a bottleneck for every project, killing the independence of business logic and developers, as the ESB team has to be engaged for every rollout.

For example, every time the following event happens, the ESB team must be engaged to build, expose, and maintain the new integration when:

• A new message format comes out
• The organization on-boards a new application
• A new flow or reorchestration is needed to consume these new APIs, domain tools, or services.

Not only is this approach not sustainable from a manpower perspective, but every new integration requires additional infrastructure to support each new orchestration or flow, making it a poor choice from a cost perspective.

In today’s world of generalization and replaceability, with so many more channels and endpoints, this scenario is simply not maintainable. To support the flexibility needed by an enterprise to stay relevant, traditional IT must transform from an Integration Factory, building each abstraction and connection, to Integration Enabler. They must let go of some micromanagement mindset, where IT owns every connection and flow, and instead focus on self-service. They should be enabling catalogs, such as the new AMPLIFY(TM) Central (AMPC) Unified Catalog or our long-standing Axway API Portal, to support discovery and collaboration for internal developers, business analysts, consumers, and third-party app providers. They should be enabling support for integration of third-party applications and services (e.g. Salesforce, SAP, Marketo, etc.) used by business users that are not owned by IT, as well as new, unexpected, and innovative use.

By enabling consumption centric use, there are several benefits to an organization.

• There is an increase in efficiency, productivity, and time-to-market TTM, with an overall lowering of costs associated with enabling new ESB flows.
• By disconnecting the traditional IT-driven integration approach, there is another benefit as well. This culture of enabling gives developers and other users the ability to build a more flexible client interface. By removing dependencies of support on a centralized system, an API-First flow can be embraced.
• This culture also allows developers to change the business logic and applications supporting the consumption contract without a lengthy change management process on the ESB or having to modify the client experience in line with what the ESB supports.

By the way, your customers and consumers don’t have to be an external or third party, these benefits apply to internal apps built for your enterprise users as well! In this day and age with mobile devices and massive app stores, everyone expects a frictionless and feature-rich user experience (UX). To deliver on this expectation, developers need to have the agility and flexibility enabled by self-service; without the bottlenecks a traditional ESB brings.

When we look at EAI, embracing the consumption centric needs of today can be a fearsome topic to consider. A tunnel gives you direction, structure, and control. When you set users loose to travel the city at will and consume services how and when they want, things change. High Control gives way for High Productivity and Flexibility, but that doesn’t mean security has to go with it, or even lower at all.

Governance through Digital Policy and Observability increases, much in line with a microservices strategy as we will cover below. ESB’s still have great value to provide the stoic tunnel to get you where you want to go, but in this new digital age with paradigm shifts reliant on consumption centric use, an API Management Platform with an API Gateway provides the self-service and flexibility you need to support your enterprise requirements. Transform from an Integration Factory building one-offs to an Integration Enabler supporting a diverse and ever-changing requirement set without being the bottleneck for every project.

Decomposing the monolith — low coupling, high cohesion

I mentioned in the last section how Hybrid is seeing the exploding of so many new user personas and endpoints that are driving the need for new ways to integrate. A lot of this activity is focused outside and on components such as new devices, new cloud services, new revenue streams, and new partners. With that said, as much as the human-driven aspect is driving the need to support new technical paradigms outside the enterprise, inside is changing just as quickly.

For years we have started to see the monoliths start to crumble. The ESB largely came about as we saw fully functional-encompassing and siloed applications giving way to the SOA (Service-Oriented Architecture) where they have brought so much value. But decomposition didn’t stop there. From stateful SOA Web Services came stateless REST APIs to allow these services to be used in new and exciting ways, driving not only the consumption centric use described above but also creating new challenges and requirements. For example, now businesses and IT had to fund and partner to achieve:

• Continuous deployment to support agility and innovation with new orchestrations, functionality, and composite integrations of functionality from many services.
• Integration of rapidly advancing and adopted technology in a Hybrid world into current and new services and applications.
• The ability to support fine-grain scaling of functionality, bringing further decomposition into microservices.

And the change from monolithic architectures continues even further now, moving away from any centralized management of orchestration (synchronous) to enabling choreography and event-driven architecture (asynchronous).

ESBs are great at implementing highly reused purpose-built business logic. Even in a world where we see decomposition to smaller and more independent components, coupling together applications and functionality with persistent and stoic orchestration is a key capability for many business processes.

Therefore, SOA Web Services and a need for Electronic Data Interchange (EDI) solutions (such as Axway B2Bi) are relevant. While REST APIs provide the flexibility for reuse, some services benefit from highly controlled and structured processes and messages.

These flows and integrations are not meant to be broken down, decoupled, and used for innovative solutions. These are core and foundational processes that build the foundation of many enterprise offerings. With that said, the paradigm shift to the decomposed monolith is happening and has great value to an organization when embraced.

In a decomposed architecture, we expect to see extensive decoupling. Rather than a single service providing many pieces of functionality (low cohesion), we see it broken down into the sum of its parts and providing only a single targeted set of functionalities (high cohesion), often deployed as microservices.

This approach allows for granular scalability of functionality, more flexibility in its use and more ease in the ability to replace or change components without effecting a larger solution. When greater functionality is encompassed entirely within a single application or integrations are purpose-built to implement specific logic (high coupling) as with an ESB, your solution becomes more brittle.

With strict orchestration and high coupling, any changes will have a large and resonating effect on the whole. You lose out on the ability to scale granularly and replace components with ease.

As we remove the high coupling, we power higher productivity, flexibility, and innovation. This result comes not only from moving away from low cohesion in an application, however, but also how the components are connected, and how the flows operate.

More and more we are seeing a migration from classic Orchestration, defined by asynchronous and highly structured execution of operations, to Choreography (as supported by the new Axway Choreography Engine [ACE]), defined by allowing asynchronous and independent communication between components. More and more we are seeing a shift away from tightly integrated processes to event-driven architectures.

By publishing events to queues, you remove tightly bound dependencies between components, allowing for replacement of components, and enabling the consumption of data in fresh ways without having to define strictly how applications are coupled together.

ESBs are not going away, but they are also purpose-built to enable controlled and tight integrations between components. They are a central bus to enable highly structured integration between services.

This situation is by definition the antithesis of the paradigm shift to a decomposed architecture. With that said, you do ultimately want to put all of these granular pieces together, be it to build composite APIs from many loosely coupled and highly cohesive components, or to orchestrate fine-grain APIs together to provide a higher level abstracted API that just looks for a dynamic component for use cases like onboarding, where most attributes are already set and unchanging.

A Full Lifecycle API Management solution, such as the one that Axway provides as part of the AMPLIFY Platform, allows you to put in place governance and observability around access to and communication between these decomposed components. By doing so, you can expose the components, giving developers, partners and third parties access to the functionality to use it in innovative and unexpected ways while maintaining the benefits of the loosely coupled and highly cohesive architecture paradigm.

Conclusion

ESBs are important to any enterprise’s EAI strategy, but new disruptive digital and Hybrid paradigms have brought about a new need. To be agile enough to remain relevant to the market and your users, you must move away from the manual Integration Factory mindset where IT builds every connection and instead become an Integration Enabler. With so many ways to connect so many new components, self-enablement of your business analysts, developers, and third-party users is a must. API Gateways are central to this strategy, working in conjunction with ESBs to provide the full functionality needed by your enterprise to thrive and deliver the best customer experiences.