Living safely with AI: the danger of automation bias

You send a message to your primary care doctor and receive an empathetic response. The message looks authentic but contains inaccuracies and harmful omissions. Your doctor has never made such errors. What happened?

It turns out the messages were drafted by generative AI and approved by your doctor. This actually occurred in a study published in the npj Digital Medicine journal in 2025.

The failure did not stem from the doctor’s competence but from overtrust in AI. They approved what was proposed by the system without proper verification. This phenomenon, known as “automation bias”, is becoming increasingly prevalent.

In 2025, a US lawyer was sanctioned by a state court after submitting a legal brief containing fake AI-generated case citations. The lawyer accepted responsibility, explaining that a law clerk had used ChatGPT to draft the petition without independent verification.

In education, a similar issue occurred. An Australian university accused thousands of students of academic misconduct in 2024 after relying on an AI detection tool. The university then reportedly discontinued using the tool after finding it to be ineffective following appeals from students.

All of the above are examples of automation bias:the tendency for people to favour a computer’s suggestions over their own judgment, even when contradictory evidence is present.

This creates a dangerous dynamic: as AI performance improves, human attentiveness often declines. When AI systems appear reliable, our critical thinking skills can atrophy from lack of use. Instead of acting as vigilant safeguards, human operators risk becoming “rubber stamps”, blindly approving machine outputs.

This relates to a profound ethical dilemma known as the “moral crumple zone”, introduced by American anthropologist Madeleine Clare Elish in 2019.

In automotive engineering, a crumple zone is designed to absorb the force of a collision to protect passengers. In AI-mediated decision-making systems, the human operator often becomes that crumple zone.

When an AI system fails in high-stakes domains such as healthcare or military operations, the human is frequently blamed for “failing to intervene.” Although humans are positioned as safeguards, prolonged reliance on AI might significantly harm their ability to detect errors.

A real-world example of the “moral crumple zone” occurred in 2018, when a pedestrian was killed by a self-driving car in Tempe, Arizona. She was struck by an Uber test vehicle while crossing the road at night. Following the collision, the safety backup driver was charged with negligent homicide and later pled guilty to endangerment, even though investigations revealed critical failures in both the autonomous system and the driver’s attention. This is a clear and painful example of the “moral crumple zone” problem. In the AI era, automation bias may amplify this problem, making it more widespread and severe.

To address automation bias at its root, we propose a three-layer framework for human-AI collaboration, covering technical, human, and legal aspects.

Technical layer: Treating AI as a second opinion

To prevent cognitive disengagement, system design should incorporate “cognitive forcing functions”that require active human participation. For example, in educational or professional contexts, users could be required to produce and submit their own preliminary solutions before accessing AI-generated responses. This approach encourages users to treat AI output as a second opinion rather than an authoritative answer

Human layer: Calibrating trust through training

Organisations must provide comprehensive training programs that empower users to recognise automation bias and engage critically with AI tools. Importantly, training should not only showcase AI capabilities but also highlight its limitations, such as hallucinations, overconfidence, and flawed reasoning, thereby helping user trust to align with actual AI performance.

Legal layer: Mandating oversight and accountability

Finally, robust legal and regulatory frameworks are essential to govern AI development, ensuring that organisations comply with safety standards and are held accountable for harm.

Recent landmark policies have begun to address automation bias directly, such as the EU AI Act, which explicitly addresses automation bias in high-risk AI systems through Article 14. It mandates human oversight and requires system designs that enable users to fully understand AI limitations, especially in high-risk AI systems.

Meanwhile, the US Executive Order 14110 emphasises preventing over-reliance on AI in high-consequence domains, such as healthcare, criminal justice, and employment, ensuring that human judgement and accountability remain central.

Vietnam's first-ever Law on Artificial Intelligence, effective since 1 March 2026, also highlights the need to ensure the maintenance of human control and intervention capabilities over all decisions and behaviours of AI systems. This principle can be interpreted as indirectly addressing automation bias.

Taken together, these responses point to a common principle: AI should support human judgement, not replace it.

AI is like water to some extent: it can bring immense benefits to our lives, but it can also cause significant damage. That’s why we should remain vigilant about its risks before it can drown us in negative effects.

Automation bias is one such danger, quietly shaping how we trust and rely on machines. To live safely with AI, we must keep a sharp mind and verify what it says before accepting them.

Story: Dr Nhat-Quang Tran, Lecturer in IT, School of Science, Engineering & Technology, RMIT University Vietnam

---

Masthead image: ipopba – stock.adobe.com | Thumbnail image: Andrii – stock.adobe.com