Transform event-driven ecosystem management through automated discovery, provisioning, and insights

We regularly engage with companies that are either interested in adopting event-driven technologies or are already working with them. A common topic that arises is the specification for event-driven architectures, which is essential for defining communication between systems or when sharing your architecture setup with other stakeholders.

AsyncAPI is a widely used specification for documenting event-driven architectures, but it also introduces an additional standard that requires maintenance and effective sharing.

As I mentioned in my previous blog post, we discussed how your ecosystem may involve various API standards. So, how can you manage these standards and their associated documentation?

Let’s look at how Amplify Engage can play a key role in addressing this challenge. We’ll explore the Discovery and Traceability Kafka Agent for Confluent Cloud.

See also: Bring Federated API Management to Life with Amplify Platform Agents

Deploy your agents with Axway CLI

Before we dive into the technical details of the Agents, it’s important to know how to install it. You can deploy the agents using the Axway Central CLI. This allows you to manage your Kafka cluster environment within Amplify Engage.

To begin, ensure you have access to the Amplify platform and can install Axway Central CLI. For Confluent Cloud, you will need the environment and cluster identifiers, API keys for Cloud and Schema Registry, and a few other configurations. For Confluent Platform, you’ll need your cluster bootstrap server, SASL credentials, and Schema Registry URL.

Example of command-line configuration process Axway Kafka agents

This blog post will further focus on the agents for Confluent Cloud.

Kafka Confluent Cloud Discovery Agent: how it works

The Discovery Agent plays a critical role in managing Kafka resources within Confluent Cloud. Here’s a brief look at how they work:

Learn more about how Amplify Agents enable true federated API management.

After the agent is deployed, it automatically detects Kafka topics and their associated schemas from the Schema Registry. This information is transformed into AsyncAPI specifications.

Once these specifications are generated, they are published to Amplify Engage as API services. The agent also manages access provisioning for consumers, ensuring that the right permissions are granted for each Kafka topic.

The process is as follows:

• The agent scans the Kafka cluster to identify available topics.
• For each topic, it checks the Schema Registry for any associated schemas.
• It generates AsyncAPI specifications using this data, along with the Kafka broker configuration.
• The agent then registers these specifications as API services in the Service Registry and applies appropriate security policies based on the broker configuration.

This allows you to streamline API documentation, making it easier for developers or consumer to understand and integrate with the available Kafka topics.

Confluent Cloud environment created in Amplify Engage
with running Discovery and Traceability agents through CLI

Overview of discovered services in the Amplify Engage
Service Registry for Confluent Cloud

Automated provisioning and access management

After the AsyncAPI specifications are registered, they can be packaged as products within Amplify Engage and published to a customizable API marketplace. When published, consumers can subscribe to these products, and the Discovery Agent handles the necessary provisioning.

For initial access, the agent creates a dedicated Service Account in Confluent Cloud and generates the Resource API Key. It also configures Access Control Lists (ACLs) to define permissions for the Service Account.

For subsequent access requests, the agent creates new credentials based on the security policies and adjusts the ACLs accordingly, ensuring that consumers retain the right access while maintaining security standards.

This automated provisioning ensures a smooth and secure process for granting access to Kafka topics without the need for manual configuration.

Product in API Marketplace where a consumer can subscribe

Traceability Agent for Kafka

The Traceability Agent complements the Discovery Agent by providing insights into how applications are interacting with Kafka topics. The Traceability Agent monitors consumer metrics by querying consumer offset statistics at regular intervals.

It specifically tracks how many messages each consumer group has read within the Kafka topics. These consumer groups are mapped to the applications registered in the marketplace, providing a clear view of how different applications are using the Kafka topics.

See also: What are API Metrics? Which Ones To Measure & Track For Business Results

This monitoring helps organizations optimize their Kafka infrastructure, identify unused topics, troubleshoot performance issues, and ensure efficient message processing.

Monitoring overview in Amplify Engage

Benefits of marketplace-driven Kafka management

The combination of the Discovery and Traceability Agents creates a solution for managing Kafka environments. The marketplace-driven approach standardizes Kafka topic discovery and documentation, reducing the documentation burden and ensuring that all topics are consistently documented.

See also: API Standards: A Foundation for Secure, Scalable Growth

By automatically generating AsyncAPI specifications, the Discovery Agent ensures that topics are clearly defined, reducing the risk of outdated or incorrect documentation.

The automated provisioning process eliminates the need for manual configuration of service accounts, ACLs, and credentials. This reduces administrative overhead and ensures secure, controlled access to Kafka resources.

With Amplify’s Traceability Agent, you gain valuable insights into how your Kafka topics are being used, which can help you optimize resource allocation and improve message processing efficiency across your ecosystem.