Module 7 · 70 min

Safety, Human Factors & Responsible AI

Designing for Harm, Not Just Accuracy. The safety-case chain from failure mode to hazard, harm, control, monitoring signal, stop rule and owner: human factors and meaningful oversight, differential harm, explainability claims, security read as patient safety, and an applied safety review. Failure modes themselves are recalled from earlier modules rather than re-taught.

Lesson progress

0 / 8 steps

Not started

Learning objectives

Construct a simple socio-technical safety argument for a healthcare AI use case, running from failure mode through hazard and harm to control, monitoring signal and residual risk.
Distinguish a model failure from a workflow, human-factors or organisational failure, and recognise that a technically correct output can still contribute to harm.
Translate a failure mode you already know — predictive, generative, retrieval or agentic — into the hazard and harm it could create in a specific workflow.
Recognise automation bias, alert fatigue and out-of-the-loop effects, and judge whether a proposed human oversight step is a real control or a formality.
Identify where unfairness can enter — representation, proxies, labels, access, workflow allocation and differential error burden — and specify how differential harm would be evaluated.
Judge explainability and uncertainty claims without mistaking a plausible explanation for evidence of correctness, causality or safety.
Read security weaknesses as routes to patient harm rather than as IT concerns, and say what the safety case has to state about them.
Organise controls as prevention, detection, mitigation and recovery, and map each control to a specific failure mode rather than a general reassurance.
Specify a monitoring set with named owners, covering safety events, override behaviour, subgroup signals, data and model change, workflow impact and action errors.
Write pre-agreed stop and rollback rules that are operationally executable by the people on shift.

From failure to harm: the safety-case chain

You already know how these systems fail. Earlier Healthcare AI course modules 2–5 covered it: leakage and shift in the data, miscalibration and missed action in prediction, fabrication and omission in generation, stale sources and wrong-target actions in retrieval and agents. This safety-case section does not re-teach that catalogue.

What this safety-case tool adds is the translation. A failure mode only becomes a safety concern when you can say what situation it creates in a specific workflow, who could be harmed, what would catch it, and what you would watch to know whether the catch is working. That is the chain, and it is the work of a safety case.

Failure mode

A specific way the system can behave other than intended.

Example: The discharge summary omits a medication change made on the ward round; the model's risk score is systematically too high after a lab supplier changes units.
Caution: Failure modes are behaviours, not vague worries. If you cannot describe what the system did, you cannot design a control for it.

Hazard

A situation created by that failure that could lead to harm.

Example: A patient leaves with a medication list that contradicts the ward-round decision; a low-risk patient is not reviewed because the score looked reassuring.
Caution: A hazard is not yet harm. Most hazards are intercepted by people, habits or luck — which is exactly why counting harms alone underestimates risk.

Harm

The actual injury or loss to a patient, a group of patients or the service.

Example: A missed dose adjustment leading to readmission; delayed treatment; loss of trust; wasted clinical time that would have gone to other patients.
Caution: Harm includes non-clinical harm. Workload, moral injury and eroded trust are real consequences and are frequently left out of risk registers.

Control

Something deliberately put in place to prevent, detect, reduce or recover from a specific failure.

Example: Structured medication reconciliation before the summary can be signed; a rule that blocks scoring when a required input is missing; a rollback plan.
Caution: A control that is not tied to a named failure mode is decoration. 'A clinician reviews it' is a staffing arrangement until you say what they are checking and how they could tell.

Residual risk

What remains after the controls, honestly stated.

Example: Rare omissions of free-text nuance that reconciliation would not catch; performance in a subgroup too small to estimate reliably.
Caution: Residual risk is meant to be accepted explicitly by someone with the authority to accept it — not quietly rounded down to zero.

The chain, worked through

One row, worked through, using the generative omission introduced in Healthcare AI Module 4. The point is the movement from left to right: each column constrains the next, and a gap in any column is where the argument breaks.

Failure mode: The drafted discharge summary omits a medication change made on the ward round.
Hazard: The patient and their GP receive a medication list that contradicts the decision actually taken.
Harm: A dose adjustment is not made in the community; avoidable deterioration or readmission, plus the clinical time spent unpicking it.
Control: The draft is shown against a structured view of medications and ward-round decisions, so review is a comparison rather than open-ended recall.
Monitoring signal: Sampled review of signed summaries against the record, reported with the approval rate and time-to-approval.
Stop or escalation rule: A clinically significant medication omission found in sampling triggers same-day review by the named owner and suspension of the medication section if a second is found.
Owner: Named: the ward's clinical lead, with the pharmacy lead for the medication section.

Work one chain through

Take one real or plausible failure and follow it all the way to a named owner. Your text is saved in this browser only, and is the same wherever you open this chain.

1. Failure mode

What did the system actually do, other than intended?

2. Hazard

What situation does that failure create?

3. Harm

Who could be harmed, and how?

4. Control

What breaks the link between hazard and harm, and in which layer does it act?

5. Signal

What would you watch that shows this is happening?

6. Stop rule

What threshold or event stops or restricts use, and who can invoke it?

7. Owner

Who is named and accountable, by role?

0 of 7 links written

Failure families you already know

A one-line reminder of each family and where it was taught, followed by the question this module asks about it. If a row is unfamiliar, revisit that module rather than treating this table as the teaching.

Predictive models

Modules 2, 3 and 6

False positives and negatives at the operating threshold, miscalibration, dataset and population shift, proxy or label failure.

Which clinical decision changes because of this output, and what situation arises when it is wrong in each direction?

Generative systems

Module 4

Fabrication, omission, instruction-following failure, uniformly confident tone, behaviour drift across versions.

Where does this text become an artefact someone relies on, and how would anyone notice what it left out?

Retrieval-grounded systems

Module 5

Retrieval misses, stale or withdrawn sources, citation mismatch, permission and provenance failure, unowned corpus.

Who owns each source, and what happens clinically when an answer is correctly cited to a document that was withdrawn?

Tool-using and agentic systems

Module 5

Wrong action, wrong target, duplicate or non-idempotent action, excessive privilege, stale state carried between sessions.

What can this system change without a human, and how is an incorrect action reversed once it has left the building?

Security as patient safety

Security in an AI-enabled clinical system is not a separate IT concern running in parallel to patient safety. It is a route to the same harms: a wrong action taken on a patient, a wrong instruction given, information disclosed that should not have been, or a service unavailable during care. Module 5 covered the mechanics; here it is four rows of the same chain.

Content is treated as instruction (prompt injection)

The assistant does something nobody asked for — discloses information, changes behaviour, or acts — because text it read told it to.

Where does the authority to act sit, and can anything the model reads change it?

Information reaches a context it should not

A confidentiality breach, which for many patients is the harm itself, plus loss of trust in the service.

What is sent, what is retained, what is logged, and who can see each of those?

The system can do more than its task requires

One reasoning error or manipulated input reaches beyond the intended patient, encounter or specialty — including actions that cannot simply be withdrawn.

What can this system do, as opposed to what is it supposed to do, and how is each action undone?

The wrong target, or no reliable record

An action lands on the wrong patient or episode, and without a usable audit trail the incident cannot be characterised, so it recurs.

Is every action bound to an authenticated user and a verified patient context, and could we reconstruct one from the log?

Previously, in Module 5

Module 5 covered the containment mechanics in detail — least privilege, allow-listed tools, separating instructions from untrusted content, approval gates, idempotency, sandboxing, kill switch and audit logging. This module does not repeat them; it asks what they mean for patients and what the safety case has to state.

Two habits carry most of the value here. Ask what the system can do rather than what it is supposed to do — capability, not intent, is what an incident will exploit. And treat every consequential action as requiring an answer to three questions: who authorised it, against which patient context, and how is it undone.

Correct and still harmful

The subtlest category is not in that table at all: an output that is entirely correct and still contributes to harm because of the context it lands in.

A correct risk score displayed after the discharge decision has been made changes nothing. A correct, well-written patient instruction leaflet at a reading level the patient cannot use is not a safe communication. A correct alert that arrives forty times a shift trains the recipient to dismiss the forty-first. A correct summary that becomes the record without anyone reading the source shifts what 'documented' means.

This is why safety review has to follow the output into the workflow, and why 'the model was right' is not a complete answer to an incident.

Sources & evidence · 13 sources

This module cites public or consensus guidance, scholarly literature.

Content reviewed: September 2026. Publication dates of the individual sources are shown in each citation.

World Health Organization. Ethics and governance of artificial intelligence for health. Geneva: WHO; 2021. ISBN 9789240029200.
WHO guidance setting out ethical principles and governance considerations for health AI, including autonomy, safety, transparency, accountability, equity and responsiveness. A normative guidance document, not a regulation or a technical safety standard.
Open source
World Health Organization. Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models. Geneva: WHO; 2024. ISBN 9789240084759.
Extends the 2021 guidance to large multi-modal and generative models, covering uses, risks and governance across the value chain. Guidance rather than evidence about the performance or safety of any specific system.
Open source
National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. Gaithersburg, MD: NIST; 2023. doi:10.6028/NIST.AI.100-1
A voluntary framework organising AI risk work into Govern, Map, Measure and Manage. Useful as a structure for a safety case and for assigning ownership; it is not sector-specific, not mandatory, and does not certify any system.
Open source
Autio C, Schwartz R, Dunietz J, et al. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1. Gaithersburg, MD: NIST; 2024. doi:10.6028/NIST.AI.600-1
A companion profile enumerating risks characteristic of generative AI — including confabulation, information security and data privacy — with suggested actions. A cross-sector risk catalogue, not clinical guidance.
Open source
Goddard K, Roudsari A, Wyatt JC. Automation bias: a systematic review of frequency, effect mediators, and mitigators. J Am Med Inform Assoc. 2012;19(1):121-127. doi:10.1136/amiajnl-2011-000089
The standard systematic review of automation bias in clinical decision support, covering how often incorrect advice is accepted and which factors moderate it. The included studies vary in design and setting, so treat it as establishing the phenomenon rather than a rate that transfers to your deployment.
Open source
Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366(6464):447-453. doi:10.1126/science.aax2342
An empirical dissection of one widely used population-health algorithm whose cost-based target understated need for Black patients at equal illness. Evidence about that algorithm and that proxy choice; it does not establish a general prevalence of bias across healthcare AI.
Open source
Lekadir K, Frangi AF, Porras AR, et al. FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare. BMJ. 2025;388:e081554. doi:10.1136/bmj-2024-081554
An international consensus guideline built on six principles — fairness, universality, traceability, usability, robustness and explainability — with 30 lifecycle best practices. Supports the lifecycle framing of safety work; following it is not proof that any specific system is safe, and it does not prescribe the six-column chain used in this module.
Open source
Ray CE, Wilson GM, Hughes AM, et al. Alert fatigue measurement in clinical decision support: a systematic review. J Am Med Inform Assoc. 2026;33(8):1523-1531. doi:10.1093/jamia/ocag064
A review of 22 systematic reviews finding that alert-fatigue measurement is poorly standardised; alert quantity, override and acceptance rates dominate. The authors recommend a significant sustained decrease in appropriate alert response from an established baseline as a more defensible operational measure — which is why override rate alone is treated here as a signal, not a diagnosis.
Open source
Ghassemi M, Oakden-Rayner L, Beam AL. The false hope of current approaches to explainable artificial intelligence in health care. Lancet Digit Health. 2021;3(11):e745-e750. doi:10.1016/S2589-7500(21)00208-9
Argues that current post-hoc explanation methods do not reliably convey why an individual prediction was made and should not substitute for rigorous validation. Supports the boundary drawn in step 5; it is an argued position paper rather than an empirical result about a specific tool.
Open source
Kleinberg J, Mullainathan S, Raghavan M. Inherent Trade-Offs in the Fair Determination of Risk Scores. ITCS 2017. doi:10.4230/LIPIcs.ITCS.2017.43
The formal result behind the fairness trade-off taught in step 4: calibration within groups and balance in error rates cannot generally be satisfied simultaneously when base rates differ, except in special cases such as perfect prediction or equal base rates. A mathematical result, not a statement about any healthcare system.
Open source
Chouldechova A. Fair prediction with disparate impact: a study of bias in recidivism prediction instruments. Big Data. 2017;5(2):153-163. doi:10.1089/big.2016.0047
Complementary evidence for the same incompatibility, showing how a calibrated instrument can still produce unequal error rates across groups when prevalence differs. Developed in criminal justice, not healthcare; the formal argument transfers, the setting does not.
Open source
Greshake K, Abdelnabi S, Mishra S, et al. Not what you've signed up for: compromising real-world LLM-integrated applications with indirect prompt injection. AISec '23. doi:10.1145/3605764.3623985
Reused from Module 5. Demonstrates that content an application retrieves can act as instruction, which is the mechanism behind the security hazards in step 6. It shows feasibility in general LLM applications; it does not establish incidence in clinical deployments.
Open source
Debenedetti E, Zhang JZ, Balunović M, et al. AgentDojo: a dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. NeurIPS 2024 Datasets and Benchmarks. arXiv:2406.13352
Reused from Module 5. A benchmark showing that current defences reduce but do not eliminate successful injection against tool-using agents, which is why containment and reversibility, rather than filtering alone, carry the safety argument here.
Open source

Applied practice: AI Safety Incident Lab

Work a near miss in an AI-drafted discharge summary through the safety-case chain: failure mode, hazard, harm, control, monitoring signal, stop rule and owner.

Open lab

Capstone Board Pack

Add what you just learned to your own strategy document while it is fresh.

Open capstone