Lab · Module 6
The proposal that should be approved
Most evidence review teaches you to find the hole. This lab does the opposite: the pack in front of you is genuinely good. The skill being practised is approving well — saying yes to a bounded next step, with conditions and a stop rule, without letting the yes drift into a claim the evidence does not support.
Situation
You chair the AI assurance group of a regional hospital group. A deterioration score for adult medical wards has come back for a second time, after the group asked for external evidence eighteen months ago.
The vendor has done the work. The pack is unusually complete for this market, and the clinical director, the outreach lead and the ward matron all support a next step. The finance lead wants to know what the group is actually committing to.
Your decision today is not whether the product works everywhere. It is whether this evidence justifies the next bounded stage in your own hospitals — and on what terms.
Fictional vendor, fictional sites and fictional figures, constructed for teaching. No real product, study or dataset is described.
The evidence pack
External validation at three independent sites
Three health systems that supplied no development data. Discrimination and calibration reported site by site, not pooled: AUC 0.86, 0.84 and 0.88; calibration slopes 0.94, 1.07 and 0.91.
Nothing was refitted at the sites. Reporting per site rather than pooled is the meaningful part: it shows the spread across settings instead of hiding a weak site inside an average. The variation that is present is modest and is stated openly.
Performance at the intended operating point
At the proposed threshold: sensitivity 71%, specificity 92%, PPV 14% at the three sites' prevalence, and an expected 6 to 9 alerts per 30-bed ward per day.
The pack states the operating point rather than a headline AUC alone, so the workload can be sized before anyone commits. Your outreach service can be asked directly whether it can answer 6 to 9 alerts per ward per day, which is a decidable question.
Subgroups, with honest sample sizes
Age, sex, ethnicity and admission route reported with event counts and intervals. Two subgroups are declared underpowered by the vendor rather than reported as reassuring.
Declaring an underpowered subgroup instead of quoting a clean-looking percentage from a handful of events is a marker of a pack written to be used rather than to be sold.
Comparator against current practice
Compared against the early-warning score the wards already use, at equal alert volume: modest incremental sensitivity, chiefly for patients the current score picks up late.
The comparison is against what you do today, not against nothing, and it is made at equal alert volume so the gain is not bought with extra alerts. The size of the gain is described as modest, not transformational.
Prospective silent-mode period at one site
Eight weeks running live but invisible to clinicians: who would have been alerted, when, what the workload would have been, and how often the alert preceded the existing escalation.
This is prospective evidence about operations, not about outcomes. It tells you the system can run on live data and what it would have asked of staff — the two things that most often break in the first month of a pilot.
What the pack states it does not show
No effect on mortality, escalation rates or length of stay. No site has yet run it live with clinicians acting on the alerts.
The vendor writes this down rather than leaving you to discover it. The remaining uncertainty is therefore named precisely enough to design the next stage around it — which is what makes a bounded approval possible.
Your decisions
Decision 1. Proceed, change, or stop?
You are deciding on one bounded next stage, not on permanent adoption.
Decision 2. Condition 1 — scope of the approval
Pick the condition that keeps the pilot bounded and reversible.
Decision 3. Condition 2 — what is measured during the pilot
One condition. Which measurement set would actually inform the renewal decision?
Decision 4. Condition 3 — human oversight and accountability during the pilot
Decision 5. The stop rule — what would end the pilot immediately?
One explicit, pre-agreed trigger. It has to be observable during the pilot.
You can change any answer until you confirm.