Putting a person between an AI system and its decisions has become the default answer to AI risk. It only works if you can prove it does.
Ask almost any organisation how it manages the risks of AI and the answer comes quickly: there’s a human in the loop. It’s reassuring and intuitive, and regulators are increasingly questioning it. In its review of AI in financial services in October, the Financial Conduct Authority found that many firms have responded to AI risks by introducing “human-in-the-loop” controls. However, the FCA warns that simply adding human review does not automatically reduce risk. It also highlighted the danger of human oversight becoming a box-ticking exercise rather than a meaningful challenge process.
That warning comes at the right time. AI is moving from tools that suggest to agents that act: booking, paying, emailing, changing records and calling other systems. As it does, the comfortable idea of a person approving every decision is running into reality. The question for boards and risk teams is no longer “Is there a human in the loop?” It’s “Can you prove the loop works?” Answering that is the job of AI assurance.
Human oversight comes in more than one form
“Human in the loop” is often used as shorthand for any human oversight, but it’s only one model. Strictly, it means a person approves every AI decision, which suits cases where accuracy matters more than speed. A human on the loop steps in periodically or when needed, for example when the system flags a low-confidence output. Human in command sits further up the chain: people set the AI’s autonomy and guardrails and manage its overall impact, a model designed with agentic AI in mind. AI in the loop uses AI to help people oversee AI, for instance by raising alerts, while humans stay in control. Underpinning all of them are kill switches to halt the system, and routes for the people affected to contest its outputs.
None of these models is right everywhere. In October the Financial Stability Board published a final report setting out 12 sound practices to facilitate responsible artificial intelligence (AI) adoption by financial institutions. They include implementing effective human oversight considering the materiality, risk, autonomy, complexity of each use. The principle applies well beyond finance: mix and match forms of oversight according to what’s at stake.
For agentic AI this matters a great deal. An agent carrying out a multi-step task across several systems can’t have a person approve every step without ceasing to be an agent at all. Oversight therefore shifts from approving individual outputs to setting the boundaries an agent works within, and checking that it stays inside them.
What makes oversight meaningful
Regulators on both sides of the Channel are converging on the same ingredients. The FCA, the Information Commissioner’s Office and Article 14 of the EU AI Act all point to six: authority, timing, information, capability, escalation and records. Miss one and oversight exists only on paper.
Authority means the reviewer can change or stop the outcome, not just comment on it. The ICO’s draft guidance on automated decision-making emphasises that human involvement must be active and not a “token gesture”, and makes clear that a human decision‑maker must have real authority.
The other five ingredients are just as practical:
- Timing: the review happens before the decision takes effect.
- Information: the reviewer can see the data, the reasoning and the system’s confidence.
- Capability: the reviewer’s competence is checked, not assumed.
- Escalation: the routes for raising concerns actually get used.
- Records: there’s a record of who decided what, and why.
Each of these gets harder with agents. Timing is the obvious one – once an agent has sent the email or moved the money, review is an inquest rather than a control. Information is another. An agent’s reasoning may span dozens of steps, tools and data sources, and a reviewer shown only the final action has been shown almost nothing. Records matter more too. When several agents act on delegated authority, organisations need to know which agent did what, and on whose behalf.
How oversight wears down
Even well-designed oversight erodes, and the ways it happens are well understood. The best known is automation bias: people over-trust a system that is usually right, and research shows experts aren’t immune. Thin sampling misses rare but serious errors. Reviewers feel pressure not to slow things down or raise faults. Poorly designed screens make the AI’s answer the default, or bury people in alerts. Unclear escalation lets low-confidence outputs slip through.
A newer failure is the way we talk about AI. In a Boston Consulting Group study of more than 1,200 managers, when AI was framed as an employee, managers identified 18% fewer errors, individual accountability for errors dropped by 9 percentage points, and accountability attributed to the AI rose by 8 percentage points. This isn’t a fringe practice. BCG found nearly one-third of managers across the US, Canada, and European Union framed AI as a teammate or employee, and more than 20% listed those AI agents on their company’s work charts.
For agentic AI, the lesson is pointed. Agents look and behave more like colleagues than any software before them. The more human they seem, the more carefully their work needs checking, not less.
One slow failure, one fast
Our framing is around two failures that unfold over different timescales.
The slow one is erosion. Skills fade and rubber-stamping grows. New staff who never did the task themselves may never build the judgement the control assumes. Aviation learned this the hard way. In 2009, ice blocked the airspeed sensors on Air France flight 447 over the Atlantic. The autopilot disconnected and handed a still-flyable aircraft back to a crew with little practice of flying by hand at high altitude. The aircraft stalled and fell into the ocean – the automation didn’t fail; the human fallback did.
The FSB makes the same point about AI. It warns that sustained reliance on AI may diminish staff expertise, judgement and ability to challenge, weakening oversight and undermining fallback arrangements. It expects firms to:
- preserve opportunities for staff to decide on and challenge AI outputs;
- retain the skills to work when the AI is down, degraded or wrong;
- train everyone to spot automation bias;
- reward the quality of oversight decisions;
- test their fallbacks.
The fast failure is speed. Real review isn’t possible at machine speed and volume, yet speed and volume is what drives AI deployment. Slow everything to human pace and the business case vanishes. This is the defining oversight challenge of agentic AI, because an agent can take thousands of actions in the time a person reviews one. The answer isn’t more reviewers -it’s better-designed loops.
From oversight on paper to oversight you can prove
Closing the gap takes two kinds of safeguard.
Keeping reviewers sharp:
- Manual tests. Regularly handling cases without AI keeps skills alive.
- Blind review. People decide before they see the AI’s answer, so they aren’t influenced by it.
- Seeded errors. Planting known errors and measuring how many are caught.
- Override tracking. An override rate near zero is a warning sign, not a success.
- Competence checks. Training on automation bias should be followed by checks that it has worked.
- Learning first. People should learn a task before they review AI doing it.
- Rehearsed fallbacks. Running the process without the AI should be practised – not just documented.
Building checks into the system. This is an area where the FCA found that monitoring was sometimes narrow, reactive or static, and oversight functions sometimes lacked AI-specific capability. Good practice it identified includes clearly designed human oversight and escalation triggers.
- Tiering. Decisions are tiered by impact.
- Escalation. Escalation is triggered by risk and uncertainty, and the confidence scores that drive it are tested rather than trusted.
- Hard limits. Agents have hard limits on what they can do, whatever their instructions say.
- Monitoring. Outcomes, drift and override patterns are monitored.
- Manageable alerts. Alert volumes are set at what people can actually handle.
- A stop button that actually stops. That includes actions already under way across connected systems.
- Continuous assurance. Assurance continues after launch rather than happening once.
- Records. Every review leaves a record of who reviewed what, and why.
Why this is an AI assurance problem
Every one of these safeguards produces evidence – catch rates, override patterns, escalation volumes, fallback drills and audit trails. That evidence separates oversight that works from oversight that merely exists, and it’s what regulators are starting to ask for. The FCA has reiterated that firms using AI-supported tools remain responsible for consumer outcomes, and reminds firms that while they can outsource AI services, they remain responsible for them.
This is where AI assurance earns its place. EthicAI’s BeehAIve® AI assurance platform helps organisations assess the AI they actually deploy, including the human controls around it, and keep assessing it as systems, staff and risks change. For agentic AI in particular, assurance can’t stop at the model. It has to test whether the people, limits and stop buttons around an agent would hold when it matters.
A human in the loop can be a powerful safeguard. But the question isn’t whether there’s a human in the loop. It’s whether you can prove the loop works.



