Ask most AI device teams what their clinical evidence is, and they will quote you a number. Ninety-four per cent sensitivity. An AUC of 0.96. It is a good number, hard-won on a held-out test set, and they treat it as the answer. Reviewers on both sides of the Atlantic treat it as the start of the conversation — the first question, not the last.
A single accuracy figure is not evidence that your model works. It is evidence that your model scored well on one dataset once. The FDA and EU notified bodies have converged, almost word for word, on what actually turns a performance number into clinical evidence — and the gap between the two is the single most common reason an otherwise strong AI submission stalls.
The headline number is the least interesting thing you'll submit
The reason a lone accuracy figure fails is that it answers a question nobody is really asking. Reviewers do not want to know how well the model performed in aggregate on your test set. They want to know whether it will perform on this patient, in this setting, for this claim — and whether it will still perform in eighteen months when the data has drifted. An aggregate number is silent on all four.
That is not a US quirk or an EU quirk. It is the shared logic underneath the FDA's draft lifecycle guidance and its Good Machine Learning Practice principles, and underneath what notified bodies are already asking for under the MDR today. The vocabulary differs; the underlying artefacts are the same four.
What teams submit vs what reviewers actually want
| What teams submit | What reviewers actually want |
|---|---|
| A single headline accuracy / AUC on a held-out set | Performance tied to the specific claim — evidence mapped to the exact intended use and population, not a figure floating free of the indication. |
| "Performance was consistent across the cohort" | Subgroup and demographic performance broken down across the axes that matter for the indication — age, sex, ethnicity, device, site — with documented mitigation wherever it degrades. |
| Internal test-set results from the development team | Genuinely independent validation — data and, ideally, evaluators separated from development, so the result isn't the model grading its own homework. |
| A validation snapshot frozen at submission | Real-world drift monitoring — a concrete plan to detect degradation and distribution shift once the device is in use, not a one-time measurement. |
Read the right-hand column and a pattern emerges: every item turns a static claim into a defended one. The number is the same; what surrounds it is the evidence.
The four artefacts, and why both regulators ask for each
1. Subgroup and demographic performance
This is the single most common gap we see. A model that is 94% accurate overall can be 82% accurate for the subgroup that matters most clinically, and an aggregate number hides exactly that. The FDA's Good Machine Learning Practice principles and the draft lifecycle guidance both push for performance broken down by demographic and clinical axes, with mitigation where it varies. EU notified bodies ask the same question under the MDR's existing clinical-evaluation expectations — long before the AI Act's Annex I obligations for embedded medical-device AI arrive on 2 August 2028. You do not get to wait for 2028; the question is live now.
2. Genuinely independent validation
Reviewers have learned to spot a test set that is really a second training set. Independence means the validation data — and, where possible, the people evaluating it — are separated from the team that built the model, so the result reflects generalisation rather than memorisation. A performance figure from data the model has effectively seen is not evidence; it is a mirror.
3. Real-world drift monitoring
An AI device is not static, and neither is the world it runs in. Populations shift, scanners get replaced, coding practice changes — and performance quietly decays. Both regulators want a plan to detect that: how you will monitor real-world performance, what triggers a review, and how you will act. This is also where your predetermined change control plan earns its keep — the monitoring that catches drift is the same machinery that governs the updates you make in response.
4. Performance tied to the specific claim
The cardinal rule underneath all of it: evidence must support the exact claim, not a general capability. A number that demonstrates the model is good at "detecting abnormalities" does not support a claim to "triage suspected intracranial haemorrhage in adult emergency patients." Reviewers map every performance figure back to the stated intended use, and a figure that doesn't map is a figure that doesn't count.
A note on FDA status, because it changes how hard you build. The FDA's lifecycle / total-product-lifecycle (TPLC) guidance — the source of most of these content elements — is still a DRAFT, issued January 2025 and marked "Not for implementation." The PCCP guidance is FINAL (December 2024, updated August 2025). So the change-control envelope is binding today; the evidence content elements are directional — build toward them in spirit, but know which document the FDA can actually hold you to. We drew that line in detail in The FDA's AI rulebook in 2026.
Build the evidence once, defend it twice
The reassuring part, for teams going to the EU and US in parallel, is that these four artefacts are not two evidence packages. They are one. Subgroup performance, independent validation, drift monitoring and claim-tied evidence answer the FDA and a notified body from the same source of truth — the vocabulary diverges, the artefacts converge. The MDR asks for it now; the AI Act's Annex I formalises it for embedded medical-device AI from 2 August 2028; the FDA's draft signals the same set today. Build the evidence to satisfy the strictest reader and you have built it for both. We set out how the AI Act and MDR fit together for AIaMD in The EU AI Act meets MDR.
The one-line version: a single accuracy number is not clinical evidence. The FDA and EU notified bodies both want the same four artefacts — subgroup performance with mitigation, genuinely independent validation, real-world drift monitoring, and performance tied to the specific claim — so build them once and defend them in both markets.
What to do this quarter
01 · Break performance down by subgroup
Report performance across the demographic and clinical axes that matter for your indication, and document mitigation wherever it degrades. This is the single most common gap in AI submissions.
02 · Make validation genuinely independent
Separate your validation data — and, where you can, your evaluators — from the development team. A test set the model has effectively seen is not evidence.
03 · Write the drift-monitoring plan now
Define how you'll detect real-world degradation, what triggers a review, and how the response ties to your PCCP. Don't submit a frozen snapshot.
04 · Map every number to the claim
Tie each performance figure to the exact intended use and population. A figure that doesn't map to the claim doesn't count — restructure your reporting around the indication.
Sources & further reading
- Federal Register — AI-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations (draft, 7 Jan 2025).
- FDA / Health Canada / MHRA — Good Machine Learning Practice: 10 guiding principles (2021).
- Meridian 103.8 — The EU AI Act meets MDR: the integrated route for AI as a medical device.
- Meridian 103.8 — The FDA's AI rulebook in 2026: what's binding, and what's still just a draft.
This article is general information current as at 27 October 2026, not regulatory or legal advice. FDA guidance status and EU AI Act timelines change; confirm the current requirements for your specific device, claim and submission type before acting.