Module 6 · 85 min
Evaluating Healthcare AI
From Vendor Claims to Evidence-Based Decisions. How to judge whether a healthcare AI claim is credible: which evidence to demand at each layer, what the headline metrics do and do not tell you, whether a result transfers to your setting, the red flags in an impressive-looking deck, the evidence ladder from validation to real-world impact, and the questions to ask a vendor or data-science team. Technical methodology is available as optional deep dives.
- Separate the three claims an AI proposal usually blends together: model performance, workflow performance, and patient or operational impact.
- Interpret sensitivity, specificity, PPV, NPV, AUC and calibration in plain language, using absolute counts and alert workload rather than formulas.
- State precisely what each headline metric cannot tell you, and convert that gap into a question for the vendor or data-science team.
- Judge whether evidence generated elsewhere is likely to hold for your patients, your case mix and your workflow, and treat transportability as an empirical question rather than a permanent badge.
- Recognise the recurring red flags in impressive-looking evidence packs, including subgroup claims resting on a handful of events.
- Place a piece of evidence on the ladder from retrospective validation to post-deployment monitoring, and say which claim that rung can and cannot support.
- Run a structured evidence review of a realistic package and produce a defensible go / limited pilot / more evidence first / no-go recommendation, naming the evidence that would change it.
What the metrics do not tell you
A metric is a narrow instrument. Almost every serious misreading of healthcare AI evidence comes from stretching one metric across a gap it was never built to cross.
Five gaps recur often enough to be worth memorising as a set: discrimination without calibration; sensitivity without workload; retrospective accuracy without outcomes; the average without the subgroup; and someone else's setting without yours.
AUC 0.89
- What it tells you
- The model ranks higher-risk patients above lower-risk ones in the evaluated cohort.
- What it cannot tell you
- Whether a displayed 20% means 20%, what happens at the operating threshold, or whether anyone benefits.
- Leadership question
- What are the calibration results, and what do sensitivity, PPV and alert volume look like at the threshold you propose?
Sensitivity 92%
- What it tells you
- Most patients who go on to have the outcome are flagged at that threshold.
- What it cannot tell you
- How many patients are flagged in total, or whether the responding service can absorb them.
- Leadership question
- How many alerts per day does that produce in a service like ours, and what is the PPV?
Validated on 40,000 records
- What it tells you
- The estimates are reasonably precise as statistical summaries of that dataset.
- What it cannot tell you
- That the data reflected live conditions, or that outcomes improved. Size buys precision, not relevance.
- Leadership question
- Was this data prepared retrospectively, and has the system ever run prospectively on live data?
Overall accuracy is consistent
- What it tells you
- Performance averaged across the whole cohort was stable.
- What it cannot tell you
- That it holds in the smaller groups that make up the average — where events may number in single figures.
- Leadership question
- Show me performance by subgroup with the number of events behind each estimate, not just the percentages.
Published in a peer-reviewed journal
- What it tells you
- The study was described in enough detail to be appraised by others.
- What it cannot tell you
- That the result transports to your patients, your case mix, your data capture or your workflow.
- Leadership question
- How does that study population compare with ours, and what local evidence exists?
None of this makes the metrics untrustworthy. It makes them narrow. The leadership skill is not distrust — it is knowing exactly which question the number in front of you has answered, and asking for the one it has not.
Technical deep dive — how calibration is actually reportedOptional. Useful when you are reading a methods section or briefing a data-science team. Not required for the assessment.
Calibration is usually summarised in two numbers plus a plot. The calibration intercept (sometimes 'calibration-in-the-large') describes systematic over- or under-prediction: 0 is ideal, a negative value means the model predicts more risk than occurs. The calibration slope describes how spread out the predictions are: 1 is ideal, and a slope below 1 indicates predictions that are too extreme at both ends — typically a sign of overfitting.
The plot matters more than either number. A flexible calibration curve of predicted against observed risk shows where the model is wrong, which is often not uniform: many models are adequate in the middle and badly off in the tail where decisions are actually made.
Van Calster and colleagues describe a hierarchy of calibration levels, from mean calibration (the overall event rate is right) up to strong calibration (the probability is right for every covariate pattern). Moderate calibration — observed risk matches predicted risk within groups of similar predictions — is the usual practical target.
Recalibration to a local population is often possible and often sufficient. It is also a change to the deployed system, and should be evaluated and governed as one rather than treated as configuration.
Visual Guide 06 — AI Evaluation in Healthcare
Keep the terminology map next to you while you continue the technical track.
Sources & evidence · 8 sources
This module cites scholarly literature.
Content reviewed: September 2026. Publication dates of the individual sources are shown in each citation.
Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. doi:10.1136/bmj-2023-078378. Published 16 April 2024. (A correction was published on 18 April 2024: BMJ. 2024;385:q902, doi:10.1136/bmj.q902; the substantive guidance is unchanged.)
The reporting standard for prediction-model studies, including AI methods. It specifies what must be reported so a study can be appraised. It is not a quality or safety standard, and adherence says nothing about whether the model performs well.
Open sourceMoons KGM, Damen JAA, Kaul T, et al. PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. BMJ. 2025;388:e082505. doi:10.1136/bmj-2024-082505
A structured tool for appraising risk of bias and applicability in prediction-model studies. It supports a judgement about a study; it does not certify a model or license deployment.
Open sourceVan Calster B, Collins GS, Vickers AJ, et al. Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance. Lancet Digital Health. 2025;7(12):100916. doi:10.1016/j.landig.2025.100916. PMID 41391983.
A contemporary overview of which performance measures answer which question, and how they are misused. Methodological guidance from an expert group, not a regulation or a mandatory standard.
Open sourceVan Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. Calibration: the Achilles heel of predictive analytics. BMC Medicine. 2019;17:230. doi:10.1186/s12916-019-1466-7. PMID 31842878.
The standard reference for why calibration matters and how discrimination can look strong while probabilities are wrong. Background for the calibration deep dive; a methodological review rather than evidence about any particular system.
Open sourceVickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making. 2006;26(6):565-574. doi:10.1177/0272989X06295361. PMID 17099194.
The original description of net benefit and decision curves, referenced by the optional threshold deep dive. It compares strategies on evaluation data; it does not demonstrate that deploying a model improves outcomes.
Open sourceRiley RD, Debray TPA, Collins GS, et al. Minimum sample size for external validation of a clinical prediction model with a binary outcome. Statistics in Medicine. 2021;40(19):4230-4251. doi:10.1002/sim.9025. PMID 34031906.
Sizing for external validation driven by the precision required, given the expected event proportion. Referenced by the optional validation deep dive; useful for retiring fixed event-count folklore.
Open sourceVasey B, Nagendran M, Campbell B, et al. Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nature Medicine. 2022;28:924-933. doi:10.1038/s41591-022-01772-9. (A publisher correction has been issued; cite the current record.)
A reporting frame for early live evaluation, covering human factors, safety and implementation — the third rung of the evidence ladder. It structures how such a study is described; it is not evidence that a system is safe or effective.
Open sourceSounderajah V, Guni A, Liu X, et al. The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence. Nature Medicine. 2025;31(10):3283-3289. doi:10.1038/s41591-025-03953-8. Published online 15 September 2025. (An Author Correction, doi:10.1038/s41591-026-04570-9, corrected an author name; the methodological content is unchanged.)
The AI-specific extension of STARD, including reporting of setting, population, reference standard and operating point — the details that determine whether an accuracy estimate travels. A reporting guideline, not an evidence standard.
Open source
Applied practice: Model Evaluation Lab
Review a hypothetical vendor evidence pack as a decision-maker: read the headline metrics, find what is missing, choose the next evidence step and make a pilot recommendation.
Capstone Board Pack
Add what you just learned to your own strategy document while it is fresh.