Case · Module 7
AI Safety Incident
A near miss reaches your desk. The tempting response is to tell clinicians to read more carefully. The safety question is different: what in the system allowed an unverified statement to travel, and what would stop the next one?
Your group has used an AI drafting assistant for discharge summaries on four wards for five months. Clinicians rate it highly: median documentation time per discharge has fallen substantially and junior doctors say evenings are calmer. Nobody wants it withdrawn.
Last Tuesday a summary for a patient discharged after an infective exacerbation stated that anticoagulation had been stopped and would be reviewed by the GP in two weeks. The medication had in fact been continued at a reduced dose. The pharmacist caught the discrepancy at the medicines-reconciliation step and the summary was corrected before it left the hospital.
No harm occurred. The incident was logged as a near miss. You chair the review.
Fictional organisation and fictional incident, constructed for teaching. No real product, patient or event is described.
The evidence in front of you
Where the statement came from
The draft was generated from the ward round notes, in which a plan to stop anticoagulation was recorded on day two and reversed verbally on day four without a corresponding note.
The assistant reproduced the most recent written plan. The failure mode is a confident summary of stale source data, not a fabricated fact — which matters, because a guardrail aimed at fabrication would not have caught it.
How the summary is signed
Sign-off happens on a single screen showing the finished narrative. The source notes are in a different system, requiring a separate login.
Verification is nominally required and practically expensive. The clinician has no way to see, on the sign-off screen, which statements came from which note.
When it happens
78% of summaries are signed between 16:00 and 18:00, in batches of five to twelve, at a median of forty seconds each.
Forty seconds is enough to read a narrative and not enough to reconcile it against a separate record. The workflow makes the required check impossible, so the check is not really required.
How it was caught
By pharmacy medicines reconciliation — a control that exists for other reasons and covers medication only.
Nothing in the pathway routinely checks non-medication statements. The organisation's confidence in the tool currently rests on a control that was never designed for it.
What is being monitored
Adoption, time saved and user satisfaction. No measure of edit rate, no sampling of signed summaries, no discrepancy reporting route.
Every metric in place measures value; none measures whether the value is being taken safely. There is no signal that would have shown this trend before the near miss.
Is it isolated?
Two similar corrections were made informally on the ward in the previous quarter and were not reported, because there was no obvious place to report them.
The absence of incidents was an absence of reporting, not an absence of events. Any judgement of frequency here is uncertain.
Your decisions
You are choosing what actually reduces the hazard, at a cost the organisation will sustain.
You can change any answer until you confirm.