Implementations of event-driven architecture (EDA) platforms are accelerating, with 82% of IT leaders indicating their company plans to apply EDA to 2-3 new use cases within the next 24 months, according to a new IDC Infobrief. The adoption of EDA is closely linked with digital maturity, with 47% of respondents describing their EDA journey as either maturing (“centralised”) or “advanced.”
And the rise of EDA is industry agnostic. Research shows more and more businesses across all industries – such as retail, financial services, aviation, manufacturing, transportation, and logistics – are seeing the need to adopt this pattern of application connectivity, to effectively enable modern use cases such as curbside pickup, preventative maintenance of machinery, and digital twinning.
At the heart of EDA transformation is the event broker, middleware that routes events and other data between applications, systems, and devices. As EDA has gained appreciation and use, the market for event brokers has also grown, offering a wide variety of options today. The capabilities of event brokers can vary significantly, making it crucial for architects and application teams to carefully consider their evaluation criteria.
Choosing the right broker: A key to integration success
An event broker, as message-oriented middleware, facilitates the reliable transmission of events between different components of a system, serving as a mediator between publishers and subscribers. It is foundational to EDA, with all event-driven applications relying on some form of event broker to exchange information.
For IT leaders, the decision is no longer about whether to embrace EDA. The critical first step is choosing the right event broker to underpin their EDA strategy. This decision includes determining which broker is best for specific use cases. Often, an enterprise might find that it needs more than one type of broker, as some are better suited to certain use cases.
Analysts are jumping into the debate, notably, David Mooter of Forrester who recently outlined the choice between a “log stream broker” vs a “smart broker.” A log stream broker can support high data throughput and tolerate some degree of complexity, as well as message replay when combined with real-time analytics and event sourcing. A smart broker, on the other hand, can support complex message routing, granular controls on message filtering, global order guarantees, and transactional commits.
Use cases signpost the choice – analytical vs operational
The fundamentals of event brokers, Mooter explains, allow organisations to build their applications as a collection of composable services that are decoupled in nature. This provides the benefits of agility, scalability, and resilience but, depending on the use cases, he argues, choosing between a log stream broker or a smart broker can make all the difference.
Log stream brokers are best epitomised by Apache Kafka. Over the last few years Apache Kafka has taken the data streaming world by storm because it’s very good at its intended purpose of aggregating massive amounts of “log” data, and streaming it to analytics engines and big data repositories.
Not all event brokers are equal to the job in hand
Unfortunately, the popularity and prevalence of Kafka has led many developers to use Kafka for use cases for which it is not ideally suited – namely operational, “run-the-business” scenarios which often involve a mix of applications, systems, and devices that need to tap into specific event streams to run effectively.
Log-based brokers use rigid, flat topic structures to describe the data they transmit, which puts the onus on the applications to do the work to filter through all the data that’s being streamed. This can be like drinking from a firehose, as applications need to consume, filter, and discard events they don’t need, leading to increased cost and complexity, not to mention security concerns.
As businesses get more connected, the broker must get smarter
On the flip side, ‘smart’ brokers do a lot of the thinking, filtering, and routing – especially as enterprises work to become more integrated, connected, and real time across diverse information technology (IT) and operational technology (OT).
These brokers feature rich and flexible topic hierarchies which enable applications to easily publish and subscribe to the very specific subsets of data they are interested in. From an event streaming perspective, smart brokers support a wide range of message exchange patterns beyond publish/subscribe, including request/reply, streaming and replay, as well as different qualities of service, including best effort and guaranteed delivery.
This makes smart brokers ideal for operational use cases, acting as the ‘digital nervous system’ of a distributed enterprise. Smart brokers go beyond simple analytics – imagine a global bank streaming over 150 billion events a day between low-latency trading platforms and market data centres in multiple locations across the globe. Prices could be published in New York and consumed in London in no time, allowing orders to be placed quickly based on the latest information.
Choosing the right smart broker to run your business
The choice becomes clear: If an organisation is looking to address operational “run-the-business” applications and use cases across a distributed enterprise, a smart broker is essential, offering three critical attributes:
- Fine-grained routing and filtering
When dealing with complex business operations, events need to be intelligently routed and filtered to be made available to the right people at the right time, not just served up en masse. A smart event broker allows consumers to subscribe to only receive the subset of events they need, in the original sequence. Take the operational example of a retailer using microservices to stream order-related events, where different functions are handled in various locations. Those application users who only want to process new orders do not require information such as shipped orders or returns; they just require information based on new orders.
It’s also the job of a smart broker to publish information in a way that allows fine-grained filtering of events based on topic taxonomy. Consider an example of an aviation authority that is required to manage incoming and outgoing flights from a particular airport. This sounds straightforward, but there will be a multitude of events related to each flight. A smart broker can handle the volume of events related to all flights, but allow subscribers to drill down by airline, arrival/departure, on time/delayed, inbound and outbound gate, meaning they can receive detailed information based on the factors relevant to their job function or area of responsibility.
- Event mesh-ability
Organisations requiring real-time event streaming for operational use cases need to look at how their broker can support an “event mesh.” An event mesh is an architecture layer consisting of a network of event brokers interconnected to allow events from one application to be dynamically routed and received by any other application, regardless of whether these applications are deployed on no-cloud, private cloud, or public cloud environments. An event mesh makes all data that touches the mesh available on demand in a secure, reliable manner, exactly where and when needed.
It enables the integration of legacy applications, data stores, modern microservices, SaaS, IoT, and mobile devices dynamically, all in real time.
If IT and analytics represent legacy integration, and OT involves the sensing of devices, then an event mesh is the glue that ties old technology with the new.
Consider a manufacturing organisation using IoT edge devices for asset tracking, product quality monitoring, and predictive maintenance. With an event mesh, sensor events in manufacturing processes can be leveraged with real-time streaming analytics to improve quality and detect machine maintenance issues sooner. Operations are always on, and so are customer and employee expectations. An event mesh supporting real-time data delivery ultimately means no waiting. Spikes in demand are deftly handled so systems don’t crash. - Rich application integration
Catering to the complex business ecosystems they support, a smart broker easily connects a variety of applications and devices without worrying about protocol translation from one to another. A distributed organisation, for example, has many custom applications in many different languages that they need to share information.
Operational use cases will usually involve this complex mix of legacy and modern applications, as well as taking on further data streams from IoT devices, all communicating using different languages and protocols. These streams may also need to be accessed by partners or third parties using standards-based protocols such as AMQP, MQTT, or webhooks; hence the need for a smart broker to support integration across this application landscape.
Integration of SaaS, legacy, and packaged applications and middleware are also required across an organisation. This requires the use of specialised event connectors to event-enable these applications so the event broker you choose needs to have many of these off-the-shelf connectors as well.
Heineken: the case for rich application integration
Heineken, a multinational brewing company, operates in 190 countries and exemplifies sophisticated EDA deployments. The company employs an event-driven integration approach to facilitate event streaming across thousands of business-critical applications, encompassing payments, logistics, inventory management, and more.
Such integration, requiring consistently fast, reliable, and robust processes across all key business functions, underscores the utility of an effective event broker.
Making business operations smarter – with the right broker for the job
In today’s event-driven environment, business operations are evolving to be more data-driven, dynamic, and real time, necessitating support for event streaming that extends beyond basic analytics-based log brokering. The role of a smart broker is crucial in providing the necessary architecture to ensure stakeholders remain interconnected, informed, and capable of making timely data-driven decisions.
We’re talking developers, users and consumers, employees, partners and end-customers. It’s a community of stakeholders that is only going to keep growing.
