The Emperor Has No Clothes

The Emperor Has No Clothes. Or: The Robot Is Naked.

By Michelle Johnson, Founder of Ideal Intelligence August 26, 2025

The AI Governance Gap in Financial Services

JD Vance recently remarked that “the AI future will not be won by hand-wringing about safety.” He was speaking in the context of the international race to produce the fastest generative AI models: a race run by OpenAI, Anthropic, Google, and others. Yet, intriguingly, trustworthy, explainable, or risk-resilient AI models are not part of the finish line many are racing toward. Safety, it seems, is a second-order concern. That may be acceptable when deploying chatbots for benign tasks like FAQs or menu planning. But when those same models are deployed inside compliance- and regulation-heavy industries such as financial services, the stakes change instantly. Banks and insurers must be able to explain how their systems work. Increasingly, they cannot. Even the builders acknowledge the risk. Sam Altman has warned that biometric authentication is already broken, and banks face a deepfake-driven fraud crisis. Yet safety is too often cast as “slowing innovation.” In financial services, ignoring AI governance isn’t naïve. It’s career-ending. Reputational fallout can be swift and significant. Fraud, mis-selling, AML breaches: these are not hypothetical. They are already here.

The Critical vs. Non-Critical Myth

Traditional governance divides systems into “critical” (like fraud detection, AML, credit scoring) and “non-critical” (chatbots, marketing automation, HR tools). That distinction is fast becoming obsolete. Take the Air Canada chatbot incident in 2024. A bereaved customer interacting with the airline’s chatbot was misled about refund eligibility. The chatbot promised a policy that did not exist. When Air Canada refused to honour it, the British Columbia Civil Resolution Tribunal ruled the airline was responsible and ordered it to pay damages. Now ask: if this had been a financial institution, would the regulator have let it slide? Closer to home, even automation in critical compliance functions has failed: Metro Bank was fined £16.7 million by the FCA for a flaw in its automated transaction‑monitoring system that missed over 60 million potentially suspicious transactions. There’s also potential for Generative AI in marketing, which is often seen as “non-critical,” to cross into regulated territory in an instant. Moral: Any AI system can go critical the moment it touches customers, compliance, or fraud vectors, especially when that system is lacking context, guardrails, or oversight.

Invisible Risks: Drift, SaaS, and Poisoned Context

Even for systems meant to be “critical,” control is slipping: Credit risk drift: A retrain at JPMorgan reportedly led to a 7% mis-scoring spike before being caught. AML drift: National Australia Bank’s AML systems produced false negatives that led to millions in remediation costs. Then there are the hidden threats: Silent SaaS updates: OpenAI shifted all users onto a new ChatGPT model without notice in 2025, altering behaviour overnight before partially reversing course. Context poisoning: Microsoft 365 Copilot was recently patched after a zero-click prompt injection vulnerability (dubbed “EchoLeak”) allowed attackers to exfiltrate sensitive data. Put drift, SaaS opacity, and poisoned context together, and you have a compliance nightmare.

Black Box vs Glass Box

Modern generative AI often functions as a black box: results appear without clear logic trails or auditability. Regulators will not accept “the AI told us so.” Some institutions are experimenting with explainability: Capital One has invested in in-house tools for credit scoring transparency. HSBC has launched AML explainability initiatives. Methods like SHAP and LIME help, but they are post-hoc fixes and not governance. In highly regulated industries, the only defensible path is glass box AI: traceable, explainable, and reproducible. If you can’t explain what your AI did, you can’t defend it.

Context as Governance (and Performance)

Generative AI rarely fails from lack of computing power. More often it fails from lack of context. Jennifer Bemert highlights “context engineering” as the missing discipline: Static: rules and policies Dynamic: transaction feeds Latent: customer histories Temporal: shifting fraud patterns Without orchestrated context such as filtering, versioning, securing, AI systems fail. One corrupted document or poisoned feed can tip the balance. Governance collapses without explainable context flows.

Turning Context into Control

Yet context on its own isn’t enough. To make governance real, financial institutions also need policies that define acceptable AI behaviour and observation layers that ensure those policies are actually followed in practice. Policies set the rules. For example, “no AI system can recommend an investment product without a human sign-off,” or “all fraud models must be version-locked and drift-tested before deployment.” Observation turns those rules into reality. It tracks how AI systems behave day to day, surfaces anomalies, and creates the audit trail regulators expect. Without observation, context engineering risks becoming another technical patch rather than a governance discipline. Together, context, policy, and observation form the checks and balances that turn abstract AI principles into defensible business practice.

Your Organisation and AI

Ask yourself (and be honest): Do you know every AI model in use (including SaaS or employee-adopted tools)? Are those models version-locked and monitored for drift? Could you explain every AI decision to a regulator tomorrow? Who owns AI governance? And do they have authority across your stack? Few financial services firms can answer “yes” to all four. That alone should spark urgent action. Vance may be right that the AI race isn’t won with hand-wringing. But in financial services, it will be lost through hand-waving. Perhaps a wiser note belongs to DeepMind’s Demis Hassabis: “I would advocate not moving fast and breaking things.” In financial services, that isn’t caution. It’s survival.

First posted on LinkedIn on 26 August 2025.