In just a few Budget cycles, AI has shifted from experimentation to execution in Singapore. With the National AI Impact Programme and fresh support under Budget 2026, the government plans to support 10,000 enterprises in adopting AI and help 100,000 workers become “AI-bilingual.” AI “missions” in finance, healthcare, and other sectors will accelerate that shift further.
At the centre of this transformation sit large language models (LLMs), the engines behind chatbots, co-pilots, and an expanding range of “smart” applications. They draft emails, summarise documents, guide customer conversations, and increasingly assist in security operations. For many leadership teams, they are still seen as smart assistants with a thin layer of vendor-supplied safety controls.
That view is no longer sufficient. Once an LLM can see sensitive data, answer customer questions, or influence operational decisions, it becomes part of your core risk surface. Because of the way these models work internally, they fail, and can be attacked, in ways that do not resemble traditional cyber incidents.
For a long time, LLMs were treated as clever productivity tools that somehow sat outside the normal rules of security. In reality, the moment you let an LLM touch sensitive data, talk to customers, or influence decisions, it becomes part of your attack surface. In Singapore, where AI is being pushed into the mainstream economy at speed, that blind spot can scale into a systemic risk very quickly.
All of this is happening now. As Singapore moves from AI pilots to full-scale deployment across finance, healthcare, government, and enterprises, any blind spots in how LLMs behave will be copied into every new chatbot, co-pilot, and AI agent. At the same time, regulators are tightening expectations on AI risk management and placing responsibility squarely on boards and senior management, making “black-box” practices around mission-critical LLMs increasingly hard to defend.
For boards and CEOs adopting LLMs, it is imperative that organisations understand enough about how they behave to govern them responsibly.
From pilots to plumbing
In many Singapore companies, LLMs have already moved from pilots to plumbing.
Customer-facing teams in banks and insurers rely on conversational assistants to handle routine enquiries and triage more complex issues. Relationship managers and sales teams use co-pilots embedded in office suites and CRM tools to craft proposals, summarise meetings, and surface relevant information. Internal service desks and HR teams are also rolling out chatbots that answer policy questions and guide employees through processes.
Security operations centres are changing too. Instead of poring over raw logs and alerts, analysts lean on AI assistants to group similar events, translate technical data into plain language, and suggest next steps. In some cases, agent-style systems are being trialled to perform low-risk tasks automatically, such as opening tickets or enriching alerts with additional context.
All of this makes sound business sense. AI helps address capacity constraints in a tight labour market, improves customer experience, and unlocks new insights from data. But it also changes the nature of operational risk. A misjudged LLM output is no longer just a bad answer; it can move money, share sensitive information, or steer a customer or citizen down the wrong path. LLM-generated decisions increasingly flow into downstream systems, updating records, triggering workflows, or influencing pricing and approvals, so a single incorrect decision can be amplified quickly. Many deployments still rely heavily on default guardrails and broad internal guidelines, with limited independent analysis of how a determined attacker might try to manipulate the system.
Regulators have noticed. MAS’s AI Risk Management Guidelines for financial institutions put governance responsibility at the top of the organisation and require boards to approve AI risk frameworks, set risk appetite, and oversee inventories of AI systems in use. AI risk is being integrated into existing three-lines-of-defence models, not treated as a standalone technical concern. That is difficult to do if LLMs are still seen as opaque “magic boxes” owned by the IT department.
Inside the black box
The technical details of LLMs can seem distant from the boardroom, but a few basic ideas explain why these systems can be misled.
Instead of reading whole sentences the way humans do, LLMs break text into small units and convert them into mathematical representations that capture relationships in a high-dimensional space. They operate within a fixed memory window and constantly decide which parts of a conversation to pay the most attention to. Attackers exploit these characteristics in ways that matter for business leaders.
They can get around simple guardrails. By slightly changing spelling, phrasing, or structure, an attacker can frame a question that looks harmless to surface-level filters but is interpreted very differently by the model. Safety checks based only on spotting banned words or obvious patterns may be easier to evade than many assume. Attackers can also push safety rules out of “memory”.
Because LLMs only remember a finite amount of text at once, a patient attacker can flood a conversation with additional content — long stories, fake logs, and repeated questions — until earlier safety instructions or business rules effectively fall out of scope, and only then ask the risky question. Finally, attackers can quietly steer the model’s focus. By carefully structuring prompts, they can cause the model to focus more on their instructions — “ignore previous guidance”, “answer only this way” — than on embedded policies.
When we talk to CISOs, we don’t expect them to memorise the maths behind transformers. But they do need to appreciate that small changes in how text is handled can be the difference between a safe response and a policy-breaking one. Attackers are already experimenting with these levers. Defenders have to move beyond “we turned on the guardrails” to asking how those guardrails actually interact with the way the model works under the hood.
From a business perspective, the key point is that many of these attacks travel over completely legitimate channels. They look like ordinary API calls or chat interactions, contain no malware, and may not trigger any of the traditional warning signs that security teams rely on. An organisation can be misled by its own AI without a single virus or exploit in sight. For sectors that depend heavily on trust (e.g., financial services, healthcare, and public services) that is a reputational risk as much as a technical one.
What leaders should be asking
Directors and executives do not need to run penetration tests themselves, but they do need to change the way they think and talk about LLMs.
The first step is visibility. Most organisations already have models sitting in frontline chatbots, internal co-pilots, and analytics tools, yet few can produce a clear list of where these systems are in production and what they are allowed to do. Treating LLMs like any other critical system means asking for an inventory of use cases that touch customers, sensitive data, or core operations, and making sure each one has a named business owner, not just an IT team, accountable for its performance and risk.
Once visibility is established, LLMs should move from the “innovation” box into mainstream threat models alongside payments platforms, trading systems, and core applications. Boards should be asking whether safeguards reflect how LLMs really behave over long, multi-step interactions, and whether prompts and outputs, including those from internal co-pilots, are logged, reviewed, and integrated into the same detection and response processes that already apply to networks, endpoints, and cloud services.
The lesson from endpoint and cloud security is that intelligence cannot simply be bolted on afterwards. The same is true for LLMs. Organisations need security capabilities that understand how attackers abuse identity, endpoints, cloud workloads, and now AI systems as one connected campaign. This requires treating LLMs as first-class assets in detection and response strategy and governance instead of experimental side projects. Singapore’s AI programs are designed to give the economy a competitive edge, and that edge will be strongest where organisations can move fast with AI, confident they are not inadvertently widening the door for attackers.
