Whitepaper

Pioneering the Science of AI Evaluation

Written by

Michael Oberst, Davis Liang, Zachary C. Lipton

Published

September 19, 2024

Last updated

August 7, 2025

Introduction

"Pajama time" is now commonplace across all clinical specialties

2 hours/day

Rotenstein LS, Holmgren AJ, Horn DM, et al. System-Level Factors and Time Spent on Electronic Health Records by Primary Care Physicians. JAMA Network Open. 11:e2344713. doi:10.1001/jamanetworkopen.2023.44713.

time spent after clinical hours to catch up on clinical documentation

63%

Shanafelt TD, West CP, Dyrbye LN, et al. Changes in Burnout and Satisfaction With Work-Life Integration in Physicians During the First 2 Years of the COVID-19 Pandemic. Mayo Clinic Proceedings. 12:2248-58. doi:10.1016/j.mayocp.2022.09.002.

of physicians report burnout

Ambient documentation systems (or “AI scribes”) reduce documentation burden by drafting clinical notes based on raw audio of the clinical encounter. Clinicians edit and verify these notes before signing them in the Electronic Health Record (EHR). However, evaluation of these systems, and the quality of the documentation they draft, is complicated by the free-form nature of generated text and the various uses of clinical documentation.

This whitepaper describes our process for evaluating AI-generated documentation at Abridge, quantitative performance measures for many of our key systems, and our process for conducting holistic clinician-in-the-loop studies prior to green-lighting updated AI components into production. These evaluations have informed the development and deployment of our AI systems; systems that already process more than a million clinical encounters every week, across 150+ health systems. By sharing our approach, we hope to foster greater transparency and facilitate dialogue among stakeholders in this fast-moving space.

Executive summary

Commitment and expertise

As a clinician-led company with roots in academia, we are committed to building trustworthy AI systems, and have the expertise to do so.

Quality and monitoring process

For assessing the quality of complex free-text documents, human judgment remains the gold standard, and our evaluation process reflects the need to keep humans in the loop.

Performance

Our systems already outperform off-the-shelf clinical and open models. In this report, we provide some deep dives on particular aspects of performance we have found critical to having an impact with our health system partners.

Looking ahead

The state of evaluation in generative AI remains a rapidly developing area with novel ideas arising every week.

Quality and monitoring

At the highest level, we can think of our clinical documentation engine as being comprised of two primary components: a world-class medically tailored speech recognition system and a note-generation system that transforms raw transcripts into drafted clinical notes.

Automated Speech Recognition (ASR)

The first component is our Automated Speech Recognition (ASR) system, which takes raw clinical audio and produces a transcript of the encounter.

Note Generation

The second component is our note-generation system which uses the transcript to produce a draft of the clinical documentation.

Each component uses a variety of underlying models. These are evaluated both individually and in an end-to-end fashion. We take a deliberate approach to evaluating and releasing upgrades to our core components:

1. Model development guided by automated metrics and clinician spot-checks

We rely on many automated metrics to guide early model development, using a large internal benchmark dataset containing clinical audio, gold standard transcripts, human-written reference notes, and rich metadata on patient characteristics.

For our automatic speech recognition system, we use canonical metrics, including word error rate and medically tailored metrics (e.g., recall of medical terms), alongside more targeted analyses (e.g., capture of newly minted medication names). For our note-generation system, we compute automated metrics of quality that compare AI-generated and human-written summaries, including variants of precision and recall (e.g., of medical concepts), which serve as proxies for factuality and completeness. Throughout, we perform stratified analyses to assess performance across diverse patient subpopulations.

2. Validation via blinded head-to-head trials Adjudicated by licensed clinicians

Before deploying models, we perform blinded, head-to-head evaluations with licensed clinicians as evaluators. To this end, we developed a software platform that presents notes (one from the current system, the other from candidate systems) side-by-side, with clinical reviewers blinded to the system that authored each.

3. Staged release process

Even once our models survive a rigorous blinded test of clinician judgment, we take a careful staged-release approach to productionizing our models, verifying performance in vivo on selected cohorts prior to approving a broader release.

4. Ongoing post-deployment monitoring

Quantitative feedback at scale to track performance

First, we capture edits made to the AI-generated note before they are finalized shared in the medical record. Clinician-generated edits are inherently scalable as a feedback mechanism, as editing is already a natural workflow. These edits provide us with a rich signal for modeling purposes and enable the computation of high-level metrics to gauge how much editing is required.

Conclusion

Evaluation of AI-driven ambient documentation is an evolving challenge, requiring more than simply determining the right set of metrics, or doing a single evaluation at a given point in time. Equally important is the process by which human feedback is incorporated into model development, auditing, and the construction of novel evaluation criteria and test cases.

Performance

In this section, we provide selected deep dives on particular aspects of performance we have found critical to having an impact with our health system partners. While these examples do not cover every dimension of performance that we monitor, each example illustrates how targeted evaluation helps drive clinically meaningful improvements both over current off-the-shelf models, and in our system over time.

Measuring the clinical fidelity of automatic speech recognition

One of the core components of our product is our Automatic Speech Recognition (ASR) system, which converts patient-clinician conversations into a transcript, which in turn constitutes the key input to our note-generation system. Accurate transcription is critical to ensuring the quality of the final output, as well as an accurate record of the conversation that the clinician can reference later.

Curating diverse data for evaluation

In order to appropriately evaluate our transcription model, we need diverse data on clinical conversations, including audio and gold standard transcripts. To that end, we use several benchmark datasets, including:

• Over 10,000 hours of clinical conversations with associated audio, and human annotations.
• Curated "challenge datasets" designed by our team to pose difficulties and probe weaknesses of clinical ASR models, including clinical conversations laden with new medication names.
• Standard research benchmark datasets, such as the Librispeech benchmark dataset.

Metrics for assessing quality of clinical transcription

To evaluate the quality of a transcript produced by an ASR system, we typically compare it to a gold standard reference transcript, written by a human listener.

Two important metrics are:

1. Word Error Rate (WER) - Calculated as the minimum number of word-level edits (substitutions, deletions, and insertions) to transform the reference transcript into the generated transcript, divided by the length of the reference transcript.
2. Medical Term Recall Rate (MTR) - Tracks the fraction of medical terms in the reference that are captured in the generated transcript.

Assessing and improving performance across multilingual settings

In this section, we consider real-world performance of our end-to-end system as judged by our users, and illustrate how we assess performance across a diverse set of users who speak multiple languages.

Multilingual support at Abridge

Abridge is designed to support situations where non-English is spoken during visits. Our transcription software (i.e., ASR) is designed to faithfully render in the transcript whatever language was spoken, and does not attempt to translate into English.

At Abridge, we officially support clinical conversations in 28 languages.

Conclusion

The feedback flywheel: Continuous evaluation of our product is not just designed to catch or prevent issues, but to drive improvements of our product at scale. Rapid product improvements in response to feedback create a virtuous cycle—when users see that their feedback is taken seriously, they are more likely to provide feedback in the future.