What can go wrong

Mode 01

Hallucination

A generative tool invents a medication, a date or a citation that reads perfectly. Example: a discharge summary lists a drug the patient never received.

Mode 02

Bias and unequal performance

The tool works less well for a group under-represented in training. Example: a skin-lesion model trained mostly on lighter skin tones performs worse on darker skin.

Mode 03

Data quality

The model inherits whatever the records got wrong. Example: a coding artefact — pain scores only entered when severe — teaches the model a pattern that reflects documentation habits, not patients.

Mode 04

Dataset shift and drift

The world moves; the frozen model does not. Example: a new triage pathway changes who arrives on the ward, and last year's risk score no longer calibrates.

Mode 05

Automation bias and deskilling

People stop checking. Example: a reader accepts a 'normal' flag and stops looking, or trainees lose practice at unaided interpretation.

Mode 06

Workflow mismatch and alert fatigue

Right output, wrong moment, too many of them. Example: a risk alert fires at 03:00 to a nurse with no mandate to escalate, twenty times a night.

Mode 07

Privacy and security

Patient data ends up where it should not, or the tool becomes a route into the record. At this level, the basics: know where data goes, who processes it and under what agreement.