At Confluent, Stephen Deasy points to a structural constraint emerging as enterprises adopt agentic AI: Many systems still rely on batch pipelines that can’t support time-sensitive decisions. As organisations move towards workflows that require continuous input and iteration, the gap between when data is generated and when it is available becomes a limiting factor.
The company’s new Chief Technology Officer spoke with Frontier Enterprise about his tech background, how AI is reshaping real-time architecture, and why connecting data more directly to business decisions is becoming critical.
What made you join Confluent?
I joined in mid-August last year, but I’m already familiar with Confluent and Kafka. I’ve been running data infrastructure and data platforms for years, and I was an early adopter of the open-source Kafka.
One of the reasons I joined Confluent was that I had seen the transformation that products like Kafka, Flink, and others can have in an organisation. It powered a lot of the technology, architecture, and end-user impact I had been building in platform teams. When I got the chance to be part of building the product itself, it was a great opportunity for me.
Was the move from product teams to infrastructure a big change?
My experience has been all up and down the stack. I started in infrastructure companies like EMC and VMware, and I’ve also run application stacks. Where I’ve seen the most impact with customers is when they integrate that full set of capabilities and make it available to application development teams to deliver business impact.
That has been what’s driving the Confluent story. We don’t think of ourselves as infrastructure. We think of ourselves as real-time data and a central nervous system, which is what customers are looking for, and we’re able to provide that.
How does agentic AI change the shift from batch to real-time streaming?
I think what agentic AI changes is how we think about building and operating software. At a high level, businesses are moving from a business intelligence model, where humans take in data, analyse it, and make decisions periodically in batch. That will continue for certain use cases. What’s changing is the move towards AI and agentic workflows.
The transformation of these workflows is driving demand for data, because real-time, fresh data enables decisions to be made continuously within those workflows. This can include use cases like personalisation of customer experiences. Some organisations are creating guest experiences that are personalised by building a complete view of the customer and delivering that through real-time interactions.
To support this shift, organisations have traditionally built what can be described as an operational estate, which includes ERP and SaaS applications that run the business. Alongside this are analytical systems, such as data warehouses and BI platforms, where the business is analysed. Historically, these have been connected using batch pipelines and ETL, which can be slow, costly, and fragile.
To support real-time context and quickly reprocess historical data, you need to shift more of that logic left. We call that a “shift left,” where governance and data usage are moved as close to the source of the event as possible.
How do you work with the product team?
We work very closely together. Between myself and (Confluent CPO) Shaun (Clowes), it starts with both of us being highly aligned on the outcomes we’re trying to drive. One of the things I value is that we’re anchored around customer outcomes.
With that alignment, we’re able to work through prioritisation, impact, and how we decide what our engineering and product teams should focus on. The most effective teams, including what I see with Shaun and across Confluent, are highly aligned on that impact.
Is relying on historical data still a problem for enterprises?
It is still very much a problem. An example is a fraud detection pipeline, where decisions are made based on very recent data. In a customer service scenario, whether handled by a human or an agentic workflow, you might be referring to a product purchased an hour ago. You then need to decide whether to ship that product and restock inventory.
If those decisions rely on data that only becomes available through an overnight batch process and is then post-processed, it slows down those use cases. That is why the shift to real-time data is critical.
The second thing we see is a shift in how software is built. Traditionally, it has been based on more deterministic models and testing. Now, it is moving towards probabilistic models, where decisions are guided by data and context. This requires a continuous stream of real-time data for making decisions, and for continuously testing and re-evaluating logic.
New use cases, models, or requirements require ongoing evaluation and tweaking before being promoted to production, often based on statistical validation rather than traditional testing. This drives the need to unify data across both streaming and batch environments and make it available for these capabilities.
How will the AI revolution play out?
It’s the speed that makes this phase of AI different. What I see is that the most innovative companies, and those focused on driving impact, are looking at how to use these capabilities. Those are the conversations I’ve been having this week, where companies are trying to understand how to connect that speed of change to what is happening in their business and how that is reflected in their data, because they see that connection as critical. The ambitions of getting that out and having impact depend on connecting your data to the business case and being able to iterate on it in real time. That will continue to speed up.
The models are continuing to evolve, along with the use cases, and the comfort level among customers, developers, and product managers is growing. More broadly, when we look at what governments are doing in training and enablement, this is likely to become a society-wide shift as organisations learn and adopt these technologies.
I’m optimistic about the range of applications, from science and medicine to civil services. Some organisations are already using this to provide citizen services more efficiently. We will continue to see this not just in high-tech companies, but more widely. AI is increasingly being treated as an enabler across organisations.
