The AI Model Is Not the Compliance System
If you are evaluating an AI platform, you need to distinguish between a genuine control capability from an impressive demonstration. An AI model can perform well for certain use-cases, but the system around it remains difficult to govern, explain or defend.
AI is making software faster and cheaper to build. And Financial institutions will continue to develop more compliance technology internally. While AI-first vendors will also promise to automate work that once depended on experienced professionals and the firm’s experience with be frictionless.
Both may produce useful tools. Neither arrives as a complete compliance system.
Judgment Does Not Sit in One Place
Compliance decisions rarely follow a clean path from question to answer. They depend on the data, applicable policy, employee’s role, prior decisions, documented exceptions and facts that may sit outside the system. A model is one component within that environment.
A reliable system must determine when the model should act, when a person should intervene and how disagreements are resolved. It must preserve evidence, support escalation and allow the firm to reconstruct what happened months or years later.
A strong model inserted into a weak operating environment will simply produce weak decisions faster.
Automation Can Hide Vulnerability
Recent failures show why this matters. A Deloitte report prepared for the Australian government using generative AI contained incorrect references and a fabricated judicial quotation. Deloitte corrected the report and agreed to refund part of its fee, although it did not attribute the errors directly to AI.
The recommendations remained unchanged but the damage was done. It affected the credibility of the report, its authors and Deloitte. AI can support heavy work, but credible output still depends on testing, traceability and human review throughout the system. As we’ve discussed in this space before, functions that rely on judgement need friction – humans throughout the loop – to function properly.
Traditional software usually follows rules that can be tested against expected outcomes. AI introduces more variation. Responses may change as models, data and prompts change.
Outputs that appear reasonable may still be incomplete, inconsistent or wrong.
Firms need clear answers to practical questions. Who validates the model? How are errors identified? Which decisions require human approval? What happens when employees rely on recommendations beyond their intended use? Can the firm explain why an outcome was accepted, rejected or escalated?
Building Is Tip of the Iceberg
AI also makes internal construction look more attractive. If software can be produced in weeks rather than months, extending an existing platform may appear faster and less expensive than buying specialist technology.
The initial build is only part of the commitment. The firm still owns integration, testing, security, documentation, monitoring, maintenance and support. It must retain people who understand how years of policies, exceptions and workflows have been translated into code. When that knowledge rests with a few developers, key-person risk becomes control risk.
The same scrutiny applies to vendors. Firms should understand how providers validate their models, manage change, retain institutional knowledge and maintain resilience. AI can reduce the cost of creating software without reducing the obligation to govern it. Count the Opportunity Cost
AI talent is scarce. Using it to rebuild compliance infrastructure already available in the market carries a cost beyond the project budget: the higher-value work those teams cannot pursue.
Internal construction should provide enough strategic or control value to justify its full cost and continuing risk.
Evaluate the System Around the Model
Let me be clear: I am entirely pro-innovation. I just saying that AI can improve triage, investigation, and anomaly detection. The question is whether AI is being added as a feature or designed as part of a defensible control system. It’s different.
Compliance leaders need to examine the data, workflows, integrations, audit trails, escalation paths and human oversight surrounding the model. What surrounds the model? Can the firm reconstruct and defend the outcome? Who remains accountable and whose credibility will be damaged?
The model is the most visible part of an AI-enabled application and it is most impressive part of the demo. But a polished demonstration does not prove the system can be governed, explained or defended. The model is not the control.