Data and organisations need to get moving as demands for businesses evolve. However, managing data can be a daunting task. At the recent Kafka Summit APAC 2021 organised by Kafka-specialist software firm Confluent, speakers from Grab, National Australia Bank, and BRI Agro talked about how their organisations are putting data to use and how they are building the next generation of cloud-based software and infrastructure.
While some of the speakers talked about leveraging data in motion, others stressed on using cloud-native managed services. The banking professionals at the sessions emphasised building a digital platform in their organisation that utilises event-driven architecture.
Jun Rao, the co-founder of Confluent, kick-started the session by revealing that the technology used by the Kafka platform – an open-source software developed by the Apache Software Foundation – was originally started at the LinkedIn social network. According to Rao, in a period of four years, LinkedIn’s business had grown by a factor of 10.
He explains what brought about the social network’s growth: “It’s all about LinkedIn’s ability to leverage its data which made it scale its business. Today a lot of companies talk about being data-driven and digital-first, but what’s unique about LinkedIn is it had two significant realisations that allowed it to go to the deeper level of leveraging its data.”
“The first realisation LinkedIn had is that leveraging data in motion is a much more effective way than leveraging data at rest. Traditionally, most places leverage data at rest. Once something interesting happens, it’s stored in the system and after that typically nothing happens. You have to wait a long time to fetch that digitalised information.”
“LinkedIn realised that it’s not the best way to leverage data. A better way to leverage the data is you first define what you want to do with the data instead of storing it first and worrying about what you want to do with the data later. When interesting things happen, it triggers predefined actions.”
“Over time, LinkedIn has moved a lot of services from data at rest to data in motion like email, recommendation engine, and search index. These are all configured such that it would be beneficial in events like job changes.”
Kafka and the Cloud
In another session, Wui Ngiap Foo, Head of Technology at Grab, said his organisation is fortunate enough to have a cloud-native framework. “One conscious decision we took is we tried to be an AI-native company from the start. We process billions of transactions every year. You can imagine the scale of the computing power to meet the needs of millions of people who are using our platform on a regular basis.”
“Using cloud services is a core part of our strategy as it allows flexibility and scalability, supporting use cases like traffic congestion and safety, and also minimising our extra effort of managing dedicated infra. As a company, what we try to do is avoid this, not-invented-here syndrome that hits a lot of tech companies. Instead what we have is this hybrid build-plus-buy strategy. We build what is core to us, and we partner with the best-in-class players for what they do well and things that we don’t think we should be building,” said Foo.
Kaspar Situmorang, CEO of Bank Rakyat Indonesia Agroniaga (BRI Agro), believes that digitalisation wasn’t easy because his organisation previously served agribusiness sectors, and now they have tried to pivot to become a fully digital bank serving new segments such as the gig economy or the informal sectors in Indonesia. According to Situmorang, BRI Agro saw a growth of 27% during the pandemic.
On using cloud services, the BRI Agro CEO said: “When we were building the new digital bank, it took us less than eight months to implement Confluent Kafka. In Indonesia, when Google Cloud Platform (GCP) was entering data centres back in 2020, we were the first bank to utilise their services in our organisation. We’ve been utilising them ever since. Lots of our services are running on Confluent Kafka within the GCP because the volume of transactions is so large and customers are expecting a reliable service, so we needed a reliable backbone supported by an event-driven infrastructure,“ he said
National Australia Bank (NAB), one of the largest financial institutions in Australia, is striving to make ample use of its self-service technology. Leng Be, Head of Messaging, Eventing and Catching GCS and API (MEGA) at NAB said, “The technology is quite new to the enterprise, so we wanted to make sure that we set the right standards, frameworks, and code base, and focus on resiliency. We decided that for the enterprise, this technology will be centrally managed. We deployed two instances of Kafka: one for a data warehouse, less time-critical usage; and then we’ve got the real-time instance for more time-critical events.”
Be works for NAB’s IT and technology division, focusing on messaging and Cloud IPS (Intrusion Prevention System). He said the organisation never had any major issues using Kafka services.
Plans for the future
According to Be, NAB is going to place a strong focus on multi-cloud. They want to make self-services easier with real-time analytics so that it’s beneficial for customers. The next stage for NAB, says Be, will focus on Kafka streams and real-time analytics leveraging a centralised architecture.
Situmorang from BRI Agro hopes to help with the recovery of the Indonesian economy by supporting small and medium-sized businesses. “We are trying to build a massive supplier financing platform so we can become a super aggregator of supplier financing in Indonesia. We always think about platforms and platform economies while serving the gig economy, so we have to think like a platform owner as well,” he said.
By doing this, according to Situmorang, BRI Agro is trying to build two sides of the same coin: the merchant side and the supplier side. Their goal is to build a reliable, easy-to-use platform for supplier financing, and build it as quickly as possible for both sides, especially in this pandemic.
Grab, on the other hand, wants to continue to push its efforts to deliver more features with the help of artificial intelligence (AI) and the cloud to help address opportunities in the market.
“For example, in our financial service verticals, fintechs have generally been more regulated and cautious with AI adoption, so we are starting to invest deeper into AI – everything from credit scoring to financial product personalisation, to recent compliance products delivered through Grab – and there’s a ton of potential there,” said Foo.
“With the pandemic still around, food and mart deliveries have been the core to the consumer behaviour, so people rely on us for deliveries to ensure that they get their daily essentials, cooked food, groceries, even household products. We continue to build upon AI to bring step changes to how we operate in the market — everything from batching orders for better efficiency, balancing demand and supply, and personalising things like search and recommendations,” he added.
