In the first of a two-part feature, Confluent Chief Product Officer Shaun Clowes explained why most AI projects fail to scale. In this continuation, he challenges the long-standing belief that batch processing is cheaper than streaming, traces Kafka’s role in redefining how enterprises move data, and outlines what separates effective product managers from the rest.
What enabled the shift from batch to real-time streaming?
Kafka was invented about 13 years ago at LinkedIn by the founders of Confluent. They created it because they were running a data platform at LinkedIn and had many data producers and consumers, but no clear way to manage them at scale. The challenge was how to govern it. These weren’t database tables or messages, but streams. They’re event streaming. That was the context in which Kafka was invented.
Kafka has been around for 13 years, and Confluent for 10. In that time, Kafka became widely adopted. The Kafka API is now a common way developers think about streaming data. Most developers know what Kafka is and what it’s for. When they need to move data from one system to another, Kafka often comes to mind first.
Over time, Kafka itself has also evolved. It began as a monolith: an open-source project installed on a set of machines. Those machines had to be large enough to store and move all the data, and organisations had to self-manage it.
Meanwhile, the scale of enterprise data has grown dramatically. In 2024, about 64 zettabytes of data were created worldwide. That figure is predicted to nearly triple to 182 zettabytes in 2025. That’s a lot of data, but not all of it is equally important. Some data requires very low latency, while other data does not. This diversity means no single approach fits all use cases.
Kafka solved the problem of moving data, but in its early form it became a one-size-fits-all solution. That meant it was too costly for some types of data, even though it was a perfect fit for others.
To address this, Confluent has introduced different types of Kafka, all of which use the Kafka protocol but with different characteristics. For example, there are freight clusters and enterprise clusters. Freight clusters use a different deployment technology to move data at very high volumes but with higher latency, making them less expensive for that type of workload. Enterprise clusters move data at high volumes with very low latency. Dedicated clusters add further options.
What they all have in common is that they are Kafka-based and interoperable, but they allow organisations to align how data is moved with its value and intended use.
That expands the solution space for streaming in a significant way. If I can cost-effectively move all of my data using streaming, then I have one universal approach for moving it. On top of that, I can layer Flink for processing, governance for management, and then Tableflow, with each building on the previous one.
In that sense, it’s cost effective and efficient to move data using Kafka, process it with Flink, manage it with governance, and have it repeated across both the analytics estate and the operational estate. Taken together, these are what enables this kind of streaming to be real.
For a long time, people thought about streaming in real time as an expensive solution, applicable only to certain types of data. But there are two things to remember: all data is created in real time — all of it. There’s no such thing as data that isn’t created in real time. Data is always generated in reaction to something happening in the world. The only reason it was ever moved in batch was that earlier technology made batch transfers easier.
The other perception was that once the technology existed, streaming was still too costly for certain types of data. That’s not true anymore. Today, there is one way to move data that can handle all types, and do so cost-effectively. This has been a 10-year journey of building a streaming approach that is not only technically stronger than batch, but also less expensive, more effective, and better at delivering outcomes.
Is streaming superior in speed, cost, and volume?
That’s exactly right. People often don’t believe it, because there’s a deeply held belief that batch is inexpensive. I don’t know why, because in reality, batch is very costly computationally. You put files into cold storage, then you have to retrieve them, which means moving them back up through the tiers of the computing stack, processing them, writing them back into storage, sending them across the network, and repeating the cycle.
There’s nothing cheap about batch. It became the standard only because it was the method we knew for moving data at the time. But batch is neither cheap nor easy; it’s complicated, expensive, and fragile. The only reason it persisted was that there was no better alternative.
On another note, what do you look for when hiring product managers?
The main thing I look for in product managers is whether they focus on what customers are trying to do. When I talk to a prospective product manager, I’ll often ask them about a project they’ve worked on: what they decided to do, how they decided it was the right answer, and how confident they were in that decision.
I keep digging, and the strong candidates can describe the journey they took. It’s almost like a hero’s journey: they explain how they chased the rabbit.
Ultimately, the job of a product manager is to take limited resources and solve a meaningful set of problems that generate revenue for the company in a way that is unique to its position in the market. It sounds simple, you could write it on the back of a napkin, but in practice it’s complicated.
Customers rarely state exactly what they need, or even know for sure what their problems are. They aren’t always clear on how they choose tools. They’re just reacting to inputs and doing what they do. So regardless of whether they are technical, the best product managers are those who relentlessly work to uncover the real problem, identify viable solutions, and succeed by driving adoption.
There are many ways to fail in product management. One is to miss the right problem altogether. Another is to identify a good problem, but not one the company is uniquely positioned to solve. Or it might be a good problem that the company can uniquely solve, but customers don’t care enough about it.
I’ve seen product managers come from engineering, sales, project management, and other backgrounds. I don’t think pedigree matters. What matters is that they can explain how they approached a problem, the steps they took to understand the customer context, and how they identified problems worth solving.
They should also be able to describe failures, reflect on what went wrong, and show humility. For example, saying: “I tried this because it seemed obvious at the time, but I learned it was wrong, and here’s why.” That ability to tell a continuous learning story is what makes someone grow into a strong product leader.
