The Co-pilot Fallacy by James Duez

AI systems are evolving from passive tools into autonomous agents, yet organisations are still leaning on the ‘co-pilot’ model for safety. The premise is simple: for every AI decision-maker, there’s a human overseer ready to catch mistakes. In theory, this human safety net should ensure any error by the AI is spotted and fixed before harm is done. In practice, however, this assumption is proving dangerously false. A cognitive phenomenon known as automation bias means that the very people meant to keep AI in check often fail to intervene when the AI is wrong. This undermines the fundamental promise of the co-pilot approach, that human experts will reliably verify AI outputs or step in when the AI goes awry.

THE ILLUSION OF A HUMAN SAFETY NET

Relying on human oversight as the ultimate failsafe fundamentally misunderstands human psychology and the nature of modern AI. Automation bias is the tendency for people to trust automated systems too readily, even in the face of contradictory evidence. In other words, humans are wired to defer to a confident machine output over their own judgment. This isn’t mere laziness; it’s a deeply ingrained bias. When an AI presents a decision or recommendation – especially with an air of authority – we feel it must know better Over time our active vigilance erodes, and we become complacent, assuming the system is probably right.

Automation bias is the tendency for people to trust automated systems too readily, even. in the face of contradictory evidence.

Real-world incidents tragically illustrate this fallacy. In 2018, a self-driving Uber test vehicle struck and killed a pedestrian in Arizona. A human safety driver was behind the wheel, whose sole job was to monitor the AI and intervene if it made a mistake. But video evidence showed the driver had grown complacent and distracted, failing to react in time. This was not a one-off lapse in attention – it was a foreseeable outcome of how our brains respond when an automated autopilot seems to handle things well for long stretches, until it doesn’t. We begin to trust it blindly. The human co-pilot in this case was physically present, but mentally absent when needed most.

Psychology research and decades of automation experience (from aircraft cockpits to industrial control rooms) confirm that humans make poor monitors of highly automated systems. We are not vigilant guardians by nature; our cognitive architecture simply isn’t built to sit idle, watching for rare failures in complex, fastmoving processes. Instead, we become lulled into inaction. This co-pilot fallacy – believing that a human can pay full attention and jump in at exactly the right moment – has created a false sense of security in AI governance.

Why HUMAN OVERSIGHT FAILS

It turns out that the limitations of human oversight go far beyond the risk of momentary inattention or complacency. Multiple factors conspire to make humans unreliable AI guardians:

Automation Bias – Trusting the Machine by Default: As described, people tend to accept automated outputs uncritically. We overly trust that the AI knows what it’s doing . This bias makes objective oversight nearly impossible. The longer an AI system runs without obvious error, the more our scepticism fades. Eventually the human overseer may be rubber-stamping decisions, intervening late or not at all. Example carnage includes regulatory breaches, reputational damage and even the erroneous overwriting of systems of record.

‘Black Box’ Opacity – We Can’t Verify What We Don’t Understand: Many advanced AI models (especially deep neural networks and large language models) are utterly opaque in their reasoning. They arrive at outcomes via complex internal processes that even experts cannot readily interpret. How can a human effectively vet an AI’s decision if the logic behind it is inscrutable? This comprehension gap means oversight often devolves into just trusting the AI’s output. For instance, multiple embarrassments in the legal field have seen lawyers submitting AI-generated court filings full of fictitious citations – because the text looked plausible, and the human reviewers assumed it must be correct. The AI (in this case, an LLM) fabricated case law references out of thin air, and the humans, unable to tell real from fake, failed to intervene. Black-box AI can produce answers that sound authoritative yet are subtly wrong, and without transparent logic, the human co-pilot is effectively flying blind.

Speed and Volume – Too Much Information, Too Little Time: Unlike human experts, who typically make considered decisions carefully and methodically, AI systems generate outputs at a scale and speed impossible for humans to manage effectively. Take tax compliance, for example: a generative AI drafting tax advice could produce dozens of pages of analysis or commentary for each client. A human reviewer, tasked with verifying the accuracy of these AI-generated reports, quickly faces a mountain of paperwork to assess. By the time they scrutinise a fraction of these outputs, subtle but critical errors – such as incorrect interpretations of complex tax regulations or misapplied exemptions – may have already slipped through and become embedded into official records. For a sector that is needing to lean into leveraging its hard-earned trust as the cost of time-based work trends towards zero, that’s a problem.

This overwhelming volume creates two critical issues. Firstly, it simply exceeds human cognitive and physical capacities, making meaningful oversight impractical and unreliable. Secondly, if inaccurate AI-generated insights are allowed into official systems of record, they can quickly pollute vital data repositories. Over time, inaccuracies compound, corrupting trusted information sources and significantly increasing risk and regulatory exposure.

As one technologist wryly noted: ‘ LLMs are predictive, not deterministic. In high-stakes applications precision is paramount. Relying solely on predictive models inevitably results in errors with significant and compounding consequences. ’ Simply put, expecting humans to reliably sift through a firehose of AI-produced content is unrealistic. Unless organisations embed more robust deterministic checks directly into their AI systems, the burden of oversight quickly becomes unmanageable, resulting in systemic errors and polluted records that undermine trust and compliance.

It’s clear that the standard human-in-the-loop strategy is not scalable under these pressures. The very safeguard meant to ensure AI reliability – human oversight – too often becomes a weak link due to cognitive bias, lack of AI explainability, and sheer volume. This undermines the core premise of the AI co-pilot. If the human cannot or does not effectively intervene when needed, the safety net might as well not be there. Automation bias has effectively pulled the rug out from under the co-pilot model, suggesting we need to rethink how we achieve reliability and trust in AI systems.

FROM OVERSIGHT TO GUARDRAILS –RETHINKING RELIABILITY

If having a human ride shotgun isn’t a dependable safety solution, what’s the alternative? The answer is to bake the safety net directly into the AI’s design. In practice, this means shifting from an oversight model (‘let the AI do its thing and hope a human catches errors’) to an architecture with deterministic guardrails: engineered constraints within which the AI must operate. Rather than trusting a human to filter out bad outputs after the fact, the system itself prevents undesirable actions by design.

Early attempts at this kind of assurance used handcrafted rules (‘if output contains forbidden phrase X, reject it’ or ‘if parameter Y exceeds threshold, require approval’). But traditional linear rule systems quickly became brittle and unmanageable in complex domains and just don’t provide the breadth of coverage.

Today’s approach is far more sophisticated. Modern deterministic guardrails utilise graph-based knowledge structures to represent the nuanced web of rules, facts, and relationships that should govern decisions. In essence, the AI is imbued with a structured model of the domain (for example, a regulatory rulebook, or a company’s policy hierarchy), encoded as an explicit knowledge graph. This graph acts as a decision boundary : any potential action or recommendation from the AI is checked against it, and if it doesn’t satisfy the encoded constraints, it won’t be allowed.

Unlike a black box neural network, a graph-based system follows explicit logical pathways thanks to the use of a symbolic inference engine – capable of reasoning over the knowledge in the graph as precisely as Excel reasons over numbers. Every inference or decision it makes can be traced back step-by-step through the graph’s rules and data. This means its outputs are consistent and repeatable given the same inputs. Where a probabilistic AI might one day say ‘ Approved ’ and the next day ‘ Denied ’ for an identical case, a deterministic inference engine will always produce the same result for the same inputs. The power of this deterministic approach is that it can handle real-world complexity without sacrificing reliability. The knowledge graph can capture intricate, interdependent conditions – far beyond simple yes/no rules – but because it’s grounded in formal logic, it won’t wing it or hallucinate an answer outside those conditions.

By removing the probabilistic element from the loop, a pure deterministic system can guarantee several crucial properties:

• No Hallucinations: The system will never fabricate non-existent facts or arbitrary answers in critical decision areas. Every output is derived from the rules and data in a knowledge graph, eliminating the bizarre ‘hallucinations’ that even the best LLMs sometimes produce.
• Repeatability: Given the same inputs and context, the system’s decisions are perfectly repeatable. There’s no roll of the dice; identical cases yield identical outcomes, an essential feature for fairness and auditability.
• Traceability and Explainability: Each decision can be traced back to specific rules and facts that were applied. This is invaluable for explainability – regulators, auditors, or end-users can be shown the exact logical steps or rule references that led to an outcome. Instead of ‘ the neural network somehow decided this loan is high-risk ’, we can point to a chain of reasoning: e.g. ‘ Declined because the applicant’s income was below £X as required by rule Y, and their credit history showed Z .’
• Causal Consistency: The deterministic engine follows an integrated map of explicit logical pathways that reflect cause-and-effect relationships. This means it’s performing true reasoning (applying domain expertise or policy logic), as opposed to a statistical correlation. The results therefore hold up under scenario changes and stress tests better than a fragile statistical guess  would.
• Freedom from Training Bias: Because decisions come from encoded domain knowledge, they aren’t skewed by the quirks and biases hidden in vast training data. A knowledge graph doesn’t pick up irrelevant cultural or historical biases the way an ML model might; it only knows what it has been explicitly given as ground truth.

In short, a pure deterministic architecture provides total confidence in critical decision-making. To give a concrete example: consider an anti-money laundering (AML) agent that flags suspicious transactions. In a deterministic setup, an LLM might help by reading unstructured reports or extracting entities from a PDF, but it will not decide whether a transaction is suspicious. Instead, the decision comes from a knowledge graph containing regulations, policies and typologies relating to financial crime. The AI agent might converse with a compliance officer in natural language (making it user-friendly), but under the hood, it is following a sophisticated yet strict rule-based logic to reach its conclusions. The outcome is an AI that is simultaneously conversational and trustworthy : it can explain its reasoning, and it will not overlook a red flag nor raise a false alarm outside of defined rules.

Overcoming The Knowledge Bottleneck

One historical challenge to implementing deterministic, knowledge-driven AI at scale has been the effort required to build and maintain knowledge graphs. In the past, defining a comprehensive rule-based system for a complex domain (like tax law or claims handling) meant months of knowledge engineering: interviewing experts, coding rules by hand, and meticulously updating them as things changed. This was a serious bottleneck, often cited as a reason why purely symbolic systems fell out of favour during the rise of machine learning. Today, however, that equation has changed dramatically.

Ironically, the latest generation of AI (including LLMs) can be turned toward the task of building the knowledge graphs themselves. Specialised LLMs finetuned on reasoning types and domain knowledge can ingest regulatory texts, policy documents, and expert explanations, and then automatically generate structured knowledge graph candidates for automated testing and human review. Instead of manually encoding thousands of rules, we can have an AI helper read the law and output a draft knowledge graph of interlinked concepts, relationships, weights and rules. Recent breakthroughs have shown this can compress what used to be months of work into a few hours. Furthermore, these systems can assist in keeping the knowledge updated – monitoring for changes in regulations or policy and suggesting updates to the knowledge graph followed by automated regression tests. The result is that deterministically governed AI is far more economically and practically feasible now than it was even a few years ago.

In other words, we’re approaching a point where every organisation can realistically encode its layer of expert knowledge and make it accessible to humans and machines, without prohibitive cost or delay. This development has knocked down the last major barrier to adopting deterministic AI solutions widely.

Deterministic Ai In Action

The paradigm of deterministic, knowledge-driven AI offers a compelling alternative to the current co-pilot model. This approach involves transforming complex regulatory frameworks and expert knowledge into executable knowledge graphs that serve as either AI guardrails or a primary reasoning engine. Rather than relying on either brittle hand-written rules or hoping humans catch LLM-generated errors, this enables a third way: a system where every automated decision is rigorously reasoned over a model of institutional knowledge. Graph-based inference engines ensure AI behaviour remains precise, deterministic, and accountable by default.

A key innovation in this space is the use of programmatically generated graphs. Modern knowledge engineering tools can automatically extract and structure knowledge from documents, policies, and human experts – essentially building a draft of the knowledge graph automatically. This tackles the knowledge bottleneck head-on.

Consider a banking example: lending policy manuals and relevant lending rules can be fed into such a system, converting that information into a web of conditions, calculations, and logical rules that are transparent. The outcome is a verifiable knowledge graph reflecting the bank’s lending criteria and regulatory obligations which can be easily tested. The knowledge graph can guarantee that every acceptance or rejection aligns with policy with an audit trail to prove it.

This approach demonstrates how deterministic reasoning can be both robust and adaptable. Knowledge graphs are flexible enough to capture complex, evolving interrelationships (such as a change in tax law or a new compliance requirement) while remaining rigorous in execution. This means organisations don’t have to choose between the efficiency of AI and the peace of mind of human oversight – they can embed the oversight into the AI. Notably, because the decisions are explainable and rooted in expert-approved logic, regulators and stakeholders gain confidence that the AI isn’t operating unpredictably. Every AI-generated recommendation or action is backed by a chain of reasoning that a human domain expert would recognise and agree with.

By deploying solutions based on these principles, companies in high-stakes sectors (finance, healthcare, legal, etc.) are finding they can embrace AI automation without losing control. The AI can act as a true copilot – not a potentially rogue agent checked by an inattentive human, but a responsible partner constrained by the same rules a diligent human would apply. In effect, the machine now has an in-built second pair of eyes: the codified knowledge of regulation, policy or experts.

Toward Ai We Can Trust

The lesson in all of this is that trustworthy AI can’t rely on after-the-fact human fixes; it must be trustworthy by design. The co-pilot model assumed that a human in the loop always be vigilant enough to catch mistakes. We now see that was a hopeful assumption, undermined by automation bias and practical limits on human attention. To achieve AI that is reliable in the real world, especially under demanding conditions, we need to move from hoping someone will catch it to ensuring it can’t go off course in the first place.

This isn’t about abandoning LLMs, or reverting to rules-based systems. It’s about combining the strengths of both, leveraging the culmination of over 50 years of AI research. As AI thought leaders have noted, the future of AI governance will not be about choosing between innovation and safety, but about a hybrid neurosymbolic approach that delivers both. We can let probabilistic AI (machine learning, neural nets, LLMs, etc.) do what it does best – learn patterns, process language, generate suggestions – while a deterministic layer provides the unbreachable reasoning or guardrails to ensure those suggestions are correct and acceptable. This synthesis enables AI to tackle complex tasks without sacrificing compliance, reliability or trust.

A well-designed AI with deterministic oversight won’t run a red light, figuratively or literally, because it’s incapable of breaking the rules encoded within it. This flips the script: the human’s role shifts from being a desperate last-chance goalie to setting the rules of the game upfront. AI co-pilots governed by deterministic logic allow human experts to impart their knowledge and values directly into the system, rather than nervously watching the system from the sidelines.

Ultimately, overcoming automation bias and unreliable oversight is about engineering AI for assurance. When AI outputs can be trusted to begin with – and produce clear reasoning – the burden on human supervisors diminishes. The humans can then focus on higher-level judgment and edge cases, rather than mindnumbing review of AI actions.

AI the world can trust is built on transparency, determinism, and robust design. By recognising the illusion of the co-pilot safety net and replacing it with built-in safety through knowledge graphs and symbolic inference, we empower AI to be both innovative and dependable. The path forward for developers, leaders and CAIOs is clear: integrate knowledge and governance into the fabric of AI systems from day one. In doing so, we create autonomous systems that earn our trust – not by asking us to watch them tirelessly, but by operating within the boundaries of what we define as correct. This approach turns automation bias on its head: rather than humans having blind faith in the machine, we’ve given the machine a strict fidelity to human knowledge. And that, ultimately, is how we ensure that our AI ‘co-pilots’ never fly solo into danger, but instead deliver results we can reliably stand behind.