The worldwide growth of event-driven architecture (EDA) is a win-win scenario. Just look at the facts: In a recent IDC InfoBrief, 93% of companies that have deployed EDA across multiple use cases said EDA has met or exceeded their expectations. Furthermore, 82% of IT leaders plan to apply EDA to 2-3 new use cases within the next 24 months.
Use cases are growing as more organisations realise the value of decoupling applications, allowing them to be more responsive to user requests, reduce runtime, and ultimately offer enterprises the freedom to operate in real time.
But more event-streaming use cases mean ever-greater data volumes. It also means more brokers, projects, and products that can be used to stream events. For example, an organisation could be using open-source Kafka for one use case, and brokers such as Confluent to address another and Amazon MSK elsewhere.
While it is good news to see the data processed in real time, many mature EDA adopters are still not realising the best return on their investment. Often, a stream only being consumed once is literally a one-to-one exchange, or an organisation has reached a stage of maturity with multiple streaming use cases, but is left with a tangled web of brokers, clusters, topics, and schemas weaved on top of each other, making it almost impossible to clearly spot where they can improve returns.
In fact, in IDC’s InfoBrief, key challenges identified among organisations in Asia-Pacific after implementing EDA are the integration with existing applications (41.6%), the lack of visibility to track and trace information flowing through systems (31.9%), and the lack of event governance and cataloguing (29.2%).
It’s called growing pains – typical of any evolving IT standard, and EDA of course is no exception.
The pivotal role of an event portal
The real value of EDA lies in being able to easily discover and then reuse all existing real-time data assets, and that can only be done if they are properly audited, managed, and governed. This requires an event management tool, such as an event portal, that will discover the real-time data assets and provide a single source of truth to manage event streams more effectively.
Here are three common pain points for organisations who find themselves on the path to event-driven maturity, and how the introduction of an event portal can help overcome them.
- Poor visibility into your event streaming estate – you need a portal with a view
Application decoupling is great for runtime, but this lack of visibility on both the publish and subscribe side can cause issues when making changes to existing applications. Application producers don’t know who the consumers are, and consumers don’t know who the publishers are.
For example, how do you know who the downstream consumers are to notify them of impending changes or even application decommissioning? From a user/consumer perspective, the person using the application daily may have a requirement to add an extra attribute to the data. But they have no idea who to contact, as they don’t have visibility of who the producer of the data is.
An event portal offers a single window into an event steaming ecosystem. It provides a native discovery agent to scan, for instance, a Kafka cluster and its schema registry to produce a visual representation of every topic and its schema(s) across multiple versions, and importantly, to identify the consumers of each topic.
Case in point – getting a health check on event streaming
Freeus, LLC, is a global manufacturer of mobile medical alerts and personal safety systems. After decomposing its monolithic applications into event-driven microservices that communicate with each other via Apache Kafka, the company found itself struggling to understand and manage its infrastructure and information flows.
Specifically, they needed to see the microservices affected by a given change before deploying a new feature or function to ensure it wouldn’t bring that system down – even for as few as five seconds.
Using an event portal, Freeus can now automatically scan its system and visualise a complete map of endpoints and event streams instead of manually diagraming event streams between microservices. Today, the Freeus development team is able to deploy new services much more quickly, running at a rate of a new release every two weeks in 2023 versus just four in all of 2022.
- Limited sharing and reuse – event streaming must live up to its hype
Let’s extrapolate the poor visibility issue from one application, across the entire enterprise.
For instance, if you’ve been using Kafka for several years, especially with many departmental or specific application type use cases, your Kafka cluster now is a treasure trove of real-time data. But data is most valuable at the moment it’s produced – as per Forrester: “Data is, without a doubt, valuable. But when stored in vaults and locked down, it is not.” Real-time data is the most valuable data that exists. But siloed event streaming data means other departments, decision-makers, customers and partners, don’t know about it – therefore it’s not getting shared or reused to its full potential.
Typically, developers don’t have anywhere to go to find this data treasure trove. Some have resorted to building Confluence or Wiki pages that try to document that this data exists, using SharePoint or Word documents. Noble intentions, but without real-time data mining and updates, this information quickly gets stale and out of date.
Again, this is where an event portal makes a difference, providing a perpetually up-to-date catalogue of data detailing all topics, event streams, schemas, and pub/sub interfaces for each application, along with owners and points of contact, as well as changes for each of the managed EDA entities. This helps expedite development by letting developers easily share, discover and re-use any existing Kafka or event streaming asset, both inside and outside the organisation.
Open up whole new opportunities to do more with your data – case in point
For a real-world example of this in action, look at the Federal Aviation Administration and its SWIM (System Wide Information Management) infrastructure, which distributes real-time information to FAA systems across the United States.
Through secure gateways, external partners of the FAA can tap into the flow of events. Airlines and other industry partners that need the SWIM data get it in real time, without perpetually requesting updates. Whether it’s the availability of gates at an airport, the position data of planes in the sky, or weather for a region, external partners have the latest information without needing to ask.
- Inability to effectively secure and govern event streams
The decentralised and dynamic nature of event-driven systems introduces unique security challenges. One of the common trade-offs with event streaming is that it does include access control rules, but developers can err on the side of being too permissive to ensure agility. If streams are not visible, properly catalogued, and data updated regularly, this presents problems with data security, governance, and compliance.
This issue will only worsen with time, as applications and use of data evolves. Every producer or consumer added to the system creates a new potential vulnerability.
Become proactive, not reactive – let the software do the talking
Using an event portal designed to give organisations visibility and control over their event streaming can turn security from reactive to proactive. Users can:
- Organise systems into application domains.
- Create and import payload schema definitions in a variety of formats including AsyncAPI.
- Define event interactions between application and microservices.
- Create events and associated topic addresses using topic structure proven practices.
Ultimately this allows them to govern and control who can access which resources, the ability to create and track every version of each EDA object as they evolve from cradle to grave, and actively promote new versions throughout development, staging, and production environments. With the right visibility, administrators can ensure security, governance, and compliance with internal policies and government regulations.
Unlock the power of EDA
Real-time streams and powerful architecture success has brought with it the need to address the rapidly growing complexity of event-streaming estates. Single, multi-broker event portal technology is the solution to helping organisations discover, govern, and manage the lifecycle of their real-time event streams across the enterprise. This technology will become increasingly important as more organisations embrace EDA as a foundational platform.
