Module 7 · 70 min
Safety, Human Factors & Responsible AI
Designing for Harm, Not Just Accuracy. The safety-case chain from failure mode to hazard, harm, control, monitoring signal, stop rule and owner: human factors and meaningful oversight, differential harm, explainability claims, security read as patient safety, and an applied safety review. Failure modes themselves are recalled from earlier modules rather than re-taught.
- Construct a simple socio-technical safety argument for a healthcare AI use case, running from failure mode through hazard and harm to control, monitoring signal and residual risk.
- Distinguish a model failure from a workflow, human-factors or organisational failure, and recognise that a technically correct output can still contribute to harm.
- Translate a failure mode you already know — predictive, generative, retrieval or agentic — into the hazard and harm it could create in a specific workflow.
- Recognise automation bias, alert fatigue and out-of-the-loop effects, and judge whether a proposed human oversight step is a real control or a formality.
- Identify where unfairness can enter — representation, proxies, labels, access, workflow allocation and differential error burden — and specify how differential harm would be evaluated.
- Judge explainability and uncertainty claims without mistaking a plausible explanation for evidence of correctness, causality or safety.
- Read security weaknesses as routes to patient harm rather than as IT concerns, and say what the safety case has to state about them.
- Organise controls as prevention, detection, mitigation and recovery, and map each control to a specific failure mode rather than a general reassurance.
- Specify a monitoring set with named owners, covering safety events, override behaviour, subgroup signals, data and model change, workflow impact and action errors.
- Write pre-agreed stop and rollback rules that are operationally executable by the people on shift.
Controls, monitoring & stop rules
A safety case is a chain of specific statements: this failure mode, this hazard, this harm, this control, this monitoring signal, this stop rule, this owner. Generic reassurance breaks the chain, and the most common break is an oversight step that has not been designed.
Organising controls by where they act in the sequence keeps a design review honest, because it exposes the layers you have left empty.
Prevention
Make the failure less likely to occur at all.
Worked examples — preventionOptional. Concrete examples of controls acting in this layer.
- Restrict scope: narrow population, narrow task, narrow set of permitted actions.
- Input validation and refusal to score on incomplete or implausible data.
- Least privilege and an explicit allow-list for tools and write actions.
- Change control: no silent model, prompt or corpus updates in a clinical system.
Detection
Notice the failure quickly when it happens anyway.
Worked examples — detectionOptional. Concrete examples of controls acting in this layer.
- Automated output checks: structure, required elements, source support, implausible values.
- Visible provenance so a reviewer can verify a claim in seconds rather than minutes.
- Independent verification for high-risk actions — a second check that does not share the first one's blind spot.
- An easy in-workflow way to report a near miss, with someone who reads the reports.
Mitigation
Limit the harm once the failure has occurred.
Worked examples — mitigationOptional. Concrete examples of controls acting in this layer.
- Approval gates before any consequential action leaves the system.
- Rate limits and volume caps that bound the blast radius of a systematic error.
- Graded autonomy: propose-only for the riskiest actions, execute-with-approval for the rest.
Recovery
Return to a safe state and learn from it.
Worked examples — recoveryOptional. Concrete examples of controls acting in this layer.
- Rollback to a previous model, prompt or corpus version, tested before it is needed.
- A kill switch that a named person on shift can actually operate, out of hours.
- A defined reversal procedure per action type, including how affected patients are contacted.
Every control should name the failure mode it addresses. Write the pair out: 'Omitted medication change → structured medication reconciliation against the ward-round entry before sign-off, with the two lists shown side by side.' That is a control. 'A clinician reviews the summary' is not — it does not say what is being checked, how the reviewer would detect it, or what makes detection feasible in the time available.
Apply the six oversight conditions from the human-oversight step of Healthcare AI Module 7 to any claimed human review step: does the reviewer have time, information, authority, competence, an escalation path and feedback? If not, the control is nominal, and the safety case should say so rather than claim the mitigation.
Beware of controls that address a different failure than the one you are worried about. Provenance links help with fabrication and stale sources; they do very little for omission. A confidence score helps with borderline cases; it does nothing about a wrong-patient action. Matching the control to the mechanism is most of the skill.
Exercise: match the failure to the control
Controls are only meaningful against a named failure. Match each failure to the control that plausibly addresses that mechanism — not the one that sounds most reassuring. Attempt all six to continue; correctness is not required.
1. The medication change that never made the summary
A drafted discharge summary leaves out a dose change agreed on the ward round. The reviewer read the draft carefully and saw nothing wrong with it.
2. The score built on half a patient
A risk score is produced for a patient whose observations have not been filed yet. It looks like every other score on the list.
3. The appointment booked for the wrong patient
An assistant with write access across all specialties books a follow-up against the wrong encounter after a retry.
4. The protocol withdrawn eight months ago
An assistant answers correctly and with an accurate citation, from a document the trust retired last spring.
5. The update that changed behaviour overnight
After a supplier model update, summaries become terser and start dropping a section clinicians relied on. Nobody can revert.
6. The score everyone reads as an explanation
A missing-vitals case is scored anyway and the number is shown with a generated sentence justifying it.
Monitoring: what to watch, and who owns it
Safety events and near misses
Incidents linked to the system, near misses reported in workflow, complaints, and clinically significant errors found in sampled outputs.
How to read these signals — safety events and near missesOptional. Interpretation notes for this monitoring domain.
Near misses are the earlier signal and are under-reported by default. Sampling outputs prospectively finds what passive reporting misses.
Override and acceptance behaviour
Acceptance rate of drafts and recommendations, edit distance before sign-off, dismissal latency, override reasons.
How to read these signals — override and acceptance behaviourOptional. Interpretation notes for this monitoring domain.
Both extremes are warning signals rather than conclusions: near-total acceptance is consistent with automation bias, and near-total dismissal suggests the human step has stopped functioning. Interpret either against an established baseline and against how relevant the outputs are, before concluding what is happening.
Subgroup signals
Performance, alert volume and error burden by the groups named in advance, with event counts and uncertainty.
How to read these signals — subgroup signalsOptional. Interpretation notes for this monitoring domain.
Pre-specify the groups and the review cadence. Numbers accumulate slowly, so the first honest report is often 'still too imprecise to say'.
Data and model change
Input distributions and missingness, upstream coding or device changes, model/prompt/corpus versions, calibration drift.
How to read these signals — data and model changeOptional. Interpretation notes for this monitoring domain.
Silent degradation can arise from upstream data, coding, device or workflow changes even when the model code has not changed. Version every component that can change behaviour, and watch the inputs as well as the outputs.
Workflow and human impact
Alert volume per shift, time spent per output, interruption load, staff experience, and whether promised time savings materialised.
How to read these signals — workflow and human impactOptional. Interpretation notes for this monitoring domain.
A tool that saves clinician time but relocates the work to another team has moved a cost, not removed one.
Action-layer correctness
Executed versus proposed actions, wrong-target rate, duplicate actions, failed or reversed actions, permission errors.
How to read these signals — action-layer correctnessOptional. Interpretation notes for this monitoring domain.
Only relevant where the system acts; where it does, this is the domain with the shortest path to irreversible harm.
Stop rules that work at 22:00 on a Saturday
Stop rules are the part of a safety case most often written last and most often written unusably. A rule that requires a committee to convene is not a stop rule; it is an agenda item.
A usable stop rule states the trigger, the immediate action, the person who can take it without seeking permission, and how it is communicated to people mid-shift. It is agreed before go-live, when nobody is defending a decision.
Triggers worth pre-agreeing include: any incident with a plausible causal link to the system; a wrong-patient action; sustained alert volume beyond the responding service's capacity; a marked change in acceptance or override behaviour; a subgroup signal crossing the threshold set in advance; loss of a required input feed; and any unversioned change to the model, prompt or corpus.
Say what stopping means in operational terms, because it is rarely all-or-nothing: revert to the previous version, disable the agentic actions but keep the draft, restrict to a single ward, or turn it off entirely and fall back to the documented manual process. Rehearse that fallback — a manual process nobody has run for eight months is not a fallback.
Every control, monitoring signal and stop rule needs a named owner with the authority and the capacity to act. 'The digital team' is not an owner. Where the deployment spans clinical, informatics and supplier boundaries, the gaps between those groups are where monitoring quietly stops happening — so assign ownership across the boundary explicitly and review it when people change roles.
Work one chain through
Take one real or plausible failure and follow it all the way to a named owner. Your text is saved in this browser only, and is the same wherever you open this chain.
What did the system actually do, other than intended?
What situation does that failure create?
Who could be harmed, and how?
What breaks the link between hazard and harm, and in which layer does it act?
What would you watch that shows this is happening?
What threshold or event stops or restricts use, and who can invoke it?
Who is named and accountable, by role?
Optional: which layer is each control acting in?Which layer does each control mostly act in? Several sit near a boundary, and the useful part of the exercise is noticing which layers your own design has left empty. Not assessed.
1. Narrow the permitted actions
The assistant may book follow-ups in two specialties and can do nothing else.
2. Sampled review of signed summaries
Twenty signed summaries a week are checked against the record for clinically significant omissions.
3. Cap bookings per hour
No more than a set number of appointment requests can be created in any hour.
4. Rehearsed fallback to manual discharge
The ward runs a documented manual process twice a year so it still works when the system is switched off.
5. Alert on a shift in acceptance rate
A weekly report flags when the approval rate or approval latency moves away from its baseline.
6. Propose-only for the riskiest actions
The system may draft a referral but never send one; a clinician issues it from their own account.
Sources & evidence · 13 sources
This module cites public or consensus guidance, scholarly literature.
Content reviewed: September 2026. Publication dates of the individual sources are shown in each citation.
World Health Organization. Ethics and governance of artificial intelligence for health. Geneva: WHO; 2021. ISBN 9789240029200.
WHO guidance setting out ethical principles and governance considerations for health AI, including autonomy, safety, transparency, accountability, equity and responsiveness. A normative guidance document, not a regulation or a technical safety standard.
Open sourceWorld Health Organization. Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models. Geneva: WHO; 2024. ISBN 9789240084759.
Extends the 2021 guidance to large multi-modal and generative models, covering uses, risks and governance across the value chain. Guidance rather than evidence about the performance or safety of any specific system.
Open sourceNational Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. Gaithersburg, MD: NIST; 2023. doi:10.6028/NIST.AI.100-1
A voluntary framework organising AI risk work into Govern, Map, Measure and Manage. Useful as a structure for a safety case and for assigning ownership; it is not sector-specific, not mandatory, and does not certify any system.
Open sourceAutio C, Schwartz R, Dunietz J, et al. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1. Gaithersburg, MD: NIST; 2024. doi:10.6028/NIST.AI.600-1
A companion profile enumerating risks characteristic of generative AI — including confabulation, information security and data privacy — with suggested actions. A cross-sector risk catalogue, not clinical guidance.
Open sourceGoddard K, Roudsari A, Wyatt JC. Automation bias: a systematic review of frequency, effect mediators, and mitigators. J Am Med Inform Assoc. 2012;19(1):121-127. doi:10.1136/amiajnl-2011-000089
The standard systematic review of automation bias in clinical decision support, covering how often incorrect advice is accepted and which factors moderate it. The included studies vary in design and setting, so treat it as establishing the phenomenon rather than a rate that transfers to your deployment.
Open sourceObermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366(6464):447-453. doi:10.1126/science.aax2342
An empirical dissection of one widely used population-health algorithm whose cost-based target understated need for Black patients at equal illness. Evidence about that algorithm and that proxy choice; it does not establish a general prevalence of bias across healthcare AI.
Open sourceLekadir K, Frangi AF, Porras AR, et al. FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare. BMJ. 2025;388:e081554. doi:10.1136/bmj-2024-081554
An international consensus guideline built on six principles — fairness, universality, traceability, usability, robustness and explainability — with 30 lifecycle best practices. Supports the lifecycle framing of safety work; following it is not proof that any specific system is safe, and it does not prescribe the six-column chain used in this module.
Open sourceRay CE, Wilson GM, Hughes AM, et al. Alert fatigue measurement in clinical decision support: a systematic review. J Am Med Inform Assoc. 2026;33(8):1523-1531. doi:10.1093/jamia/ocag064
A review of 22 systematic reviews finding that alert-fatigue measurement is poorly standardised; alert quantity, override and acceptance rates dominate. The authors recommend a significant sustained decrease in appropriate alert response from an established baseline as a more defensible operational measure — which is why override rate alone is treated here as a signal, not a diagnosis.
Open sourceGhassemi M, Oakden-Rayner L, Beam AL. The false hope of current approaches to explainable artificial intelligence in health care. Lancet Digit Health. 2021;3(11):e745-e750. doi:10.1016/S2589-7500(21)00208-9
Argues that current post-hoc explanation methods do not reliably convey why an individual prediction was made and should not substitute for rigorous validation. Supports the boundary drawn in step 5; it is an argued position paper rather than an empirical result about a specific tool.
Open sourceKleinberg J, Mullainathan S, Raghavan M. Inherent Trade-Offs in the Fair Determination of Risk Scores. ITCS 2017. doi:10.4230/LIPIcs.ITCS.2017.43
The formal result behind the fairness trade-off taught in step 4: calibration within groups and balance in error rates cannot generally be satisfied simultaneously when base rates differ, except in special cases such as perfect prediction or equal base rates. A mathematical result, not a statement about any healthcare system.
Open sourceChouldechova A. Fair prediction with disparate impact: a study of bias in recidivism prediction instruments. Big Data. 2017;5(2):153-163. doi:10.1089/big.2016.0047
Complementary evidence for the same incompatibility, showing how a calibrated instrument can still produce unequal error rates across groups when prevalence differs. Developed in criminal justice, not healthcare; the formal argument transfers, the setting does not.
Open sourceGreshake K, Abdelnabi S, Mishra S, et al. Not what you've signed up for: compromising real-world LLM-integrated applications with indirect prompt injection. AISec '23. doi:10.1145/3605764.3623985
Reused from Module 5. Demonstrates that content an application retrieves can act as instruction, which is the mechanism behind the security hazards in step 6. It shows feasibility in general LLM applications; it does not establish incidence in clinical deployments.
Open sourceDebenedetti E, Zhang JZ, Balunović M, et al. AgentDojo: a dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. NeurIPS 2024 Datasets and Benchmarks. arXiv:2406.13352
Reused from Module 5. A benchmark showing that current defences reduce but do not eliminate successful injection against tool-using agents, which is why containment and reversibility, rather than filtering alone, carry the safety argument here.
Open source
Applied practice: AI Safety Incident Lab
Work a near miss in an AI-drafted discharge summary through the safety-case chain: failure mode, hazard, harm, control, monitoring signal, stop rule and owner.
Capstone Board Pack
Add what you just learned to your own strategy document while it is fresh.