VAST Data was built before AI entered the enterprise mainstream. Research breakthroughs existed alongside isolated applications, but there was no shared infrastructure model for how organisations would store, move, and analyse data at AI scale. That uncertainty shaped the company’s earliest architectural decisions, long before it emerged from stealth in 2019.
In this interview, Jeff Denworth, Co-Founder of VAST Data, explains how those early assumptions influenced the company’s approach to distributed systems, why consolidation became central to its strategy, and how that thinking now plays out in competition with established storage, data, and streaming platforms.
What was your stealth period like from 2016 to 2019?
We started the company in January 2016, which was about a month after OpenAI was founded. We had an idea to build a new type of distributed systems architecture that we believed could break many of the long-standing trade-offs in how organisations manage and store data, including database systems. At the same time, we sensed that something was about to happen with AI. Companies such as Baidu, Facebook, Meta, and Google were beginning to do interesting work, but there was no clear commercial business application for any of it yet.
We began with a small team of architects. I was the only person focused on the go-to-market side, while they spent roughly a year thinking through what the product would need to look like over the next decade. My role was to speak directly with customers. Before we came out of stealth mode in 2019, we spoke with around 300 to 400 organisations. Early on, we would tell people that we had something we couldn’t disclose unless they were willing to sign an NDA. Once that was in place, we explained how we thought it was possible to change many of the dynamics that had defined enterprise data centres for the past 20 to 30 years, which required a new type of systems architecture.
In the earliest days, we just had a blank website which said “coming soon,” but we were shipping our software out to some of the most important data processing organisations in the world, a lot of which have now become customers like NASA and Pixar, as well as large banks, trading firms, life sciences institutions, like the National Institutes of Health.
By the time we came out of stealth mode, I was serving as CMO, and I actually argued that we should not come out of stealth at all, which runs counter to what a marketing person is usually expected to do. We liked the idea that we knew something others did not, and that we could engage customers without relying on a big marketing engine.
What was the point of departure for you when you were developing your product?
It came from two places. One was the realisation that something new could be built; the other was the recognition that something new had to be built. In the earliest days, we said two things to ourselves: First, over time, we wanted to build a new distributed computing system for AI, and we wanted to do that from the ground up, covering everything required for AI computing. We didn’t have a large team, and we didn’t have unlimited investment, so we decided to start at the data layer, because our background was in storage and it was an area we knew well.
Fifteen years before Google worked on transformers, one of the things it was known for was re-architecting the data centre to reach scale ahead of everyone else. In 2003, Google published a white paper describing a distributed systems architecture it called “shared nothing.” The concept was simple: take a commodity server, attach storage devices to it, and build clusters of these nodes to reach the required capacity or performance. That model was used for YouTube storage and later gave rise to technologies such as object storage systems, file systems, Hadoop, and many of the companies that followed Hadoop, effectively shaping modern distributed computing.
At the time, networks were relatively slow, so it made sense to bring data to the compute node rather than move it across the network. As a result, systems were broken apart and distributed across large numbers of commodity servers. This introduced several challenges. One was that flash was not considered viable for high-capacity use, so it was not used for real-time access, largely because of cost.
Another challenge was consistency. In that style of system, if you wanted to update something like a database table or another dataset, the nodes had to coordinate with each other to manage an atomic write. Without that coordination, the system would become inconsistent. As these systems scaled, write performance degraded to the point that real-time analysis became effectively impossible.
By the time we started the company, one of the key realisations we had was that networks were about 1,600 times faster than they were back when you could only buy one-gigabit networking. The first thing we said was, “Can we break apart these systems and have a data centre with containers that run software, alongside separate systems that manage storage devices but run no application software?” That’s called disaggregation.
The second requirement was a new data structure implemented in software, allowing all of those containers to see a single global volume at data centre scale. When containers can see which parts of the data are locked or released, consistency can be enforced at the data layer. In that model, processors don’t need to coordinate with each other; they simply read from the system. By removing communication across large clusters for consistency management, we built an extremely parallelised distributed architecture in which nodes do not communicate directly with one another.
What’s your strategy in competing with companies like NetApp, Snowflake, and Kafka?
Consolidation is nice if it means not having to build a data centre stack out of 17 different products. That brings some implied benefits, such as unified management and unified data governance. But there’s something more important at play. If you think about a set of capabilities that are each fit for purpose, what happens if they are brought together with the specific intention of building a more integrated AI infrastructure?
The question then becomes why nobody has tried to build something this unified before. The companies you referenced represent technologies we are displacing, but you only see a comparable combination of these components in large cloud environments, typically within tier-one cloud providers. Even there, those environments rely on stitching together many separate services to approximate what we’re doing.
The only way this strategy works is by not introducing compromise as more capability is added to the platform. If we build an event streaming system, to your point about Kafka, it has to scale efficiently. When that is combined with data warehousing, so that you don’t have an event bus in one place and a data lake somewhere else, it becomes possible to correlate events happening in real time with the rest of the data in the estate.
What does the partnership with Ola involve?
Ola has been working with us for about two years. At the time, the company was starting to build out its own cloud, initially to support a big data environment across its different businesses, driven by the tons of telemetry collected from its various endpoints. In parallel, it was also working on generative AI tools and developing its own AI models.
That work led Ola to pursue the launch of an indigenous AI cloud in India, called Krutrim, which is built on VAST infrastructure. We’re working with them on data pipelines and storage, supporting not only Ola’s internal use of Krutrim but also other customers. The environment is designed as a multi-tenant cloud infrastructure with secure, confidential computing capabilities.
