Evaluate an ambient AI scribe on omissions first, hallucinations second, and time saved third. Published data puts hallucinations around 1.5% of note sentences and omissions around 3.5% in one large study, and a 2025 randomized trial found clinicians still saw "occasional" clinically significant errors. The signing clinician carries the liability, so your evaluation must prove they can catch them.
Ambient scribes are now the most widely deployed generative AI in US care delivery. Epic shipped its own in February 2026, Microsoft folded DAX into Dragon Copilot in 2025, and most health systems are on their second or third vendor bake-off. The hard question has moved from "does it save time?" to "how do we know it is safe enough, and who is on the hook when it is not?" This guide covers what the published studies actually measured, the metrics that matter, how to run a local evaluation, and where liability sits. Facts were checked on 12 November 2026.
What do published studies say about ambient AI scribe error rates?
The best evidence shows low but real error rates, with omissions more common than fabrications. In a 2025 npj Digital Medicine study of 12,999 clinician-annotated note sentences, 1.47% were hallucinated and 3.45% of transcript sentences were omitted; 44% of the hallucinations were rated major.
Three sources are worth reading in full before any vendor meeting:
- Asgari et al., npj Digital Medicine, May 2025. A framework for clinical safety and hallucination rates in LLM-generated notes. 191 hallucinated sentences out of 12,999; 1,712 omitted sentences out of 49,590 transcript sentences; 84 of the 191 hallucinations (44%) were "major", meaning they could change diagnosis or management if left uncorrected. The authors work for a scribe vendor and used GPT-4-era models, so treat the absolute numbers as a reference point, not a benchmark for today's products. The method is the valuable part.
- Lukac et al., NEJM AI, November 2025. A three-arm randomized trial of 238 outpatient physicians at UCLA Health comparing Microsoft DAX Copilot, Nabla and usual care. Nabla reduced time-in-note by 9.5% versus control (statistically significant); DAX by 1.7% (not significant). Physicians rated clinically significant inaccuracies at 2.7–2.8 on a 5-point scale, which means "occasionally". There was one mild adverse event: counseling that was omitted from the plan.
- Kaiser Permanente, 2025. The Permanente Medical Group reported that 7,260 physicians used ambient scribes in about 2.58 million encounters between October 2023 and December 2024, saving an estimated 15,791 hours. Its accuracy review was small, based on a sample of transcripts.
The pattern across all three is consistent. Time savings are modest and concentrated in heavy users. Errors are uncommon but not rare, and the most common kind is something missing rather than something invented. That matters, because omissions are much harder for a busy reviewer to spot than a wrong number.
Which metrics should you use to evaluate an AI scribe?
Measure four things: major omission rate, major hallucination rate, edit burden, and time-in-note including editing time. Report each per note and per sentence, split by specialty and visit type, and never accept a single "accuracy" percentage.
| Metric | How to measure | Why it matters | Common trap |
|---|---|---|---|
| Omission rate (major / minor) | Annotators mark clinically relevant transcript facts missing from the note | Most common error type in published work | Vendors report only hallucinations |
| Hallucination rate (major / minor) | Note sentences with no support in the transcript | Directly creates false record entries | Counting style changes as errors, or ignoring wrong negations |
| Attribution errors | Statements assigned to the wrong speaker; pronoun and negation flips | Reported in the UCLA trial; easy to miss | Not in most vendor scorecards |
| Edit burden | Characters or sections changed between draft and signed note | Proxy for real review effort | Near-zero edits may mean clinicians stopped reading |
| Time-in-note | EHR audit log time, including editing | The business case | The UCLA trial notes its metric excluded editing time |
| Subgroup performance | All of the above by language, accent, specialty, visit length | Errors are not evenly distributed | Pilots run only with enthusiastic English-speaking clinicians |
Define "major" before you start. The npj paper's definition is a good default: an error is major if, uncorrected, it could change diagnosis or management. Write it down, train annotators on twenty examples, and measure inter-annotator agreement. Without that, two reviewers will disagree on a third of the borderline cases and your numbers will not mean much.
How do health systems run an AI scribe evaluation?
The systems that do this well run three stages: an offline test on recorded, consented encounters with gold-standard annotation; a pragmatic pilot with matched controls; and continuous post-go-live sampling. Each stage answers a different question, and skipping the first is the most common mistake.
Stage 1: offline test set
Collect 50–100 consented encounters per major specialty, with the transcript and the clinician's own note as references. Run every candidate vendor on the same audio. Two trained annotators score each draft for omissions, hallucinations and attribution errors using a fixed taxonomy. This is the only stage where you can compare vendors fairly, because the input is identical. It is also the stage where you test hard cases on purpose: interpreters, family members talking over each other, poor audio, long visits, and specialty vocabulary.
Stage 2: pragmatic pilot
Randomize or match clinicians into vendor and control groups, as the UCLA trial did, and measure time-in-note from EHR audit logs, burnout surveys, and a sample of note reviews. Track usage: in that trial, about 15% of clinicians assigned a scribe never used it, and usage was around a third of eligible visits. A pilot that only reports results from enthusiastic users overstates the benefit.
Stage 3: production monitoring
After go-live, sample a fixed number of notes each month per clinician group, re-score them against audio, and watch the trend. Re-run the offline set whenever the vendor changes its model, which may happen without much notice. Ask in the contract how you will be told. We describe how to build these test harnesses in LLM testing for regulated industries.
Who is liable when an AI scribe makes a mistake?
In practice, the clinician who signs the note. CMS says that when you use a scribe, including AI technology, you sign to authenticate the entry, and state medical boards hold physicians accountable for how they use AI. Vendor contracts usually limit the vendor's liability, so the health system carries the operational risk.
Three sources set the frame:
- CMS signature rules. The July 2025 Medicare signature requirements fact sheet states that when a scribe, "including artificial intelligence technology", documents for you, you sign the entry to authenticate it; the scribe does not sign.
- State medical boards. The FSMB's 2024 policy on AI in clinical practice says the physician is ultimately responsible for the use of AI. It is guidance, not law, but boards use it.
- State law. Some states now add their own rules. Texas SB 1188, for example, requires practitioners who use AI in diagnosis to review AI-generated records. We track these in healthcare AI laws by state.
The practical consequence: your evaluation is also a liability control. If you can show that you measured error rates, trained clinicians on the error types that occur, designed the review screen to surface them, and monitor after launch, you are in a much better position than an organization that turned the tool on and trusted it. Ask vendors for indemnification terms, error reporting obligations and model change notice in writing.
What should the review workflow look like to catch errors?
Design the review screen around the errors you measured. Show the source transcript next to each section, highlight sentences with weak support, force attention on assessment and plan, and flag negations and medication changes for a second look.
- Source linking. Click any sentence to hear or read the transcript segment it came from. Sentences without a source are highlighted.
- Omission prompts. A checklist of items discussed in the transcript but absent from the note ("medication change mentioned at 06:12 not in plan"). Omissions are invisible unless something points at them.
- High-risk tokens. Negations, laterality, doses and allergies get visual emphasis, because a single word flip is a major error.
- Store the draft and the signed version. The difference is your best ongoing signal of model quality.
This is a human-in-the-loop design problem more than an AI problem. A review step that clinicians click through in three seconds is not a control.
Should you build your own ambient scribe or buy one?
For most health systems and practices, buy. The leading products integrate with major EHRs and have scale you will not match. Build only when you need a specialty or language the products handle poorly, an unsupported EHR, a scribe embedded in your own product, or audio and model inside your own cloud account.
If you do build, the hard parts are not the model. They are diarization (who said what), medical vocabulary in noisy rooms, note templates per specialty, EHR write-back as an unsigned draft, and the evaluation harness above. AWS lists Transcribe including HealthScribe and Bedrock as HIPAA-eligible services, which lets audio, transcription and the language model run under one BAA. Even when you buy, you need the evaluation capability; it is the only way to compare vendors and to notice when a model update makes things worse. Our HIPAA-compliant AI agent architecture covers the surrounding controls.
What should you ask an AI scribe vendor before signing?
Ask for their omission and hallucination rates with definitions and test set size, results by specialty and language, how they notify you of model changes, what they retain and for how long, whether your data trains their models, and what liability they accept.
- Signed BAA, and the list of subcontractors that touch audio.
- Audio and transcript retention period, and deletion on request.
- Whether customer data is used for model training, with an opt-out in the contract.
- Error metrics with definitions; willingness to run on your offline test set.
- Advance notice of model changes and a way to pin or test before rollout.
- Safety event reporting: how you report an error and how they respond.
The Joint Commission and Coalition for Health AI guidance released in September 2025 asks organizations to monitor AI quality continuously and report AI safety events. These vendor questions are how you make that possible.
How BeevR builds this
We usually come in at one of two points. Health systems and digital health vendors ask us to build the evaluation harness: a consented test set, an annotation tool with a fixed error taxonomy, automated scoring where it is reliable, and dashboards by specialty and vendor. Product companies ask us to build a scribe or documentation feature into their own platform, with source-linked drafts, omission prompts, EHR write-back as unsigned drafts, and the same harness running in CI. The model can run inside your own AWS account under your AWS BAA, and BeevR signs a BAA as your business associate.
We are a senior, founder-led team in Hanoi, rated 5.0 on Clutch, and we built a multi-tenant EHR now live across four US clinics at about 300 patients a day. We work in fixed phases, from $10K per phase, with a working demo every Friday, and you own the code from day one. See our healthcare AI agent development and EHR integration work.
If you are planning a scribe bake-off or building documentation AI into a product, book a 30-minute call. We will review your evaluation plan and tell you where it will give misleading numbers.