Evaluation

Rubric design for the AI scribe: Comparing Likert and Boolean approaches

20 Jul, 2026

Allu Rachel Kurien, Shane Khaki

Rubric design for AI medical scribe evaluation

Ambient AI medical scribes, tools that listen to the clinician–patient encounter and draft the clinical note automatically, have become one of the most rapidly adopted technologies in modern healthcare. The Peterson Health Technology Institute's 2025 AI Taskforce report described ambient documentation as being on track to become one of the fastest technology adoptions in healthcare history.​

The scale of deployment supports that assessment: The Permanente Medical Group reported that more than 7,000 of its physicians used ambient scribes across over 2.5 million patient encounters in the 14 months following rollout, with estimated documentation time savings exceeding 15,700 hours.

Adoption at this pace places unusual weight on evaluation. Randomized trials of commercial scribe platforms have observed that generated notes occasionally omit information, misattribute details, or introduce inaccuracies. Because these drafts may become part of the legal medical record after clinician review and sign-off, the question of how to measure the quality and safety of ambient documentation has shifted from an academic concern to an operational requirement for vendors and health systems alike.

In this blog, we examine the rating scales most commonly used to evaluate AI-generated clinical notes, consider their limitations from a model-improvement perspective, and describe an alternative rubric design built on binary, failure-mode-specific criteria. We illustrate the approach with a worked example and provide the full rubric as a downloadable template.

How AI-generated clinical notes are evaluated today

Human evaluation of ambient documentation typically relies on ordinal rating scales completed by physician reviewers. Among validated instruments, the Physician Documentation Quality Instrument (PDQI-9) is one of the best known: reviewers rate a note across nine attributes (up to date, accurate, thorough, useful, organized, comprehensible, succinct, synthesized, and internally consistent), each on a five-point Likert scale. Alternatives and variants exist - QNOTE offers an element-level assessment of outpatient notes, and many evaluations construct bespoke Likert questionnaires suited to their setting. Systematic reviews of human evaluation practice across healthcare language model applications have repeatedly identified Likert scales as the most common rating approach in the field.

The limitation appears when the resulting scores reach a model development team. Consider a note that receives 36 of a possible 45 on the PDQI-9 scale. The aggregate conveys that reviewers judged the note to be moderately good, yet it compresses many independent judgments, including factual fidelity, completeness, organization, and succinctness, into a single figure. A hallucinated examination finding and an overly verbose assessment may lower the score by similar amounts while differing enormously in clinical consequence. For the engineers responsible for improving the system, the relevant question shifts from how good the note is overall to which specific errors it contains, where those errors occur, and how frequently each error type recurs across encounters.

A failure-mode view of documentation errors

An alternative starting point is a defect model: an explicit taxonomy of the error types that ambient documentation systems are known to produce. Published evaluations and post-deployment reports suggest that most clinically meaningful errors fall into a small number of recurring categories.

#Failure modeDefinitionExampleClinical consequence
1FabricationContent appears in the note without support from the encounter."Patient reports quitting smoking" when tobacco use was never discussed.False entries in a legal record; decisions built on invented facts.
2Fabricated pertinent negativeA normal finding is recorded for an examination or question that never occurred."Denies palpitations" when the topic was never raised.Conceals gaps in the workup; creates false reassurance.
3OmissionClinically significant content from the encounter is absent from the note.Orthostatic dizziness the patient mentioned is missing from the history.Lost findings; delayed diagnosis.
4Attribution errorPatient-reported items are recorded as clinician findings, or history is merged with examination."Murmur noted" sourced from patient recollection rather than auscultation.Overstates the evidence behind an assessment.
5Medication infidelityIncorrect drug, dose, route, frequency, or active vs discontinued status.Lisinopril 20 mg documented as 10 mg.Direct patient safety exposure.
6Negation & temporal errorA denied symptom is recorded as present, or past history is written as current."Chest pain" documented where the patient denied it.Corrupts the problem list.
7Unsupported inferenceAn assessment or plan extends beyond what the encounter established.A diagnosis appears without supporting findings.Anchors future care on weak evidence.

These categories require different weights. Fabrication and fabricated pertinent negatives occupy the top of the severity ranking because they introduce falsehoods into the medical record; stylistic issues such as verbosity and formatting sit at the bottom, worth monitoring but rarely grounds for blocking a release. A taxonomy of this kind also gives an evaluation program a shared vocabulary: reviewers, engineers, and clinical safety teams can discuss a "fabricated pertinent negative rate" with far greater precision than a movement in mean quality score.

Assigning severity weights

A weighted rubric is only as defensible as its weights, so the basis for them should be explicit rather than intuited. Four factors should drive each weight:

  • Potential patient harm: the clinical consequence if the error reaches care undetected. A halved dose or an inverted risk assessment outranks a formatting lapse.
  • Legal-record risk: the note is a legal document, so fabricated content and false attestations carry medico-legal exposure beyond their clinical effect.
  • Downstream decision impact: whether the error propagates into orders, coding, referrals, or the next visit, rather than staying contained.
  • Likelihood of clinician detection: a subtle, plausible error that survives review deserves more weight than an obvious one a clinician will almost certainly catch and correct. This factor runs against intuition and is the one teams most often overlook.

Detectability is why a fabricated pertinent negative belongs alongside outright fabrication rather than below it. A fabricated positive finding is an assertion, and assertions draw the reviewing eye. A fabricated negative reads as clinical routine, carries no internal contradiction, and offers a reviewer nothing to snag on. It survives sign-off, and a later clinician reasonably believes the question was asked and the answer was normal. 

Weights should be calibrated with clinicians and revisited as real detection data accrues. An error type that clinicians reliably catch in practice can be down-weighted over time, while one that repeatedly slips through warrants escalation.

Establishing the source of truth

A rubric for ambient documentation has to begin by defining what the note is measured against, and the transcript alone is an incomplete answer. Legitimate note content routinely originates outside the spoken encounter: the prior EHR record, an active medication list, the problem list, clinician-entered structured fields, standard follow-up instructions, and accepted clinical norms. A rubric that treats the transcript as the sole source of truth will misclassify this legitimate content as fabrication, and will forfeit the trust of the clinicians whose notes it grades.

The resolution is to score each piece of note content by its relationship to a defined source-of-truth set, rather than to the transcript alone. That set should include the transcript together with whatever context the scribe could legitimately draw on, and the raters should hold access to the same sources. The distinctions that matter:

Status of note contentExampleRubric treatment
Supported by the transcriptA symptom the patient described aloud.Pass
Supported by another valid sourceA chronic diagnosis from the problem list; a home medication from the med list.Pass (rater can see the source)
Reasonable standard content beyond the encounterRoutine safety-netting such as "return if symptoms worsen".Program decision - pass or flag
Absent from every available sourceAn invented symptom or result.Fail - fabrication
Contradicted by the transcriptA dose or finding recorded differently from the encounter.Fail - high severity
Unsafe unsupported inferenceA diagnosis or treatment beyond what any source supports.Fail - high severity

This is why the fabrication criterion should reference the source-of-truth set rather than the transcript. Phrased as "every clinical fact in the note is supported by a defined source of truth," the criterion accepts a chronic diagnosis pulled from the problem list while still failing an invented symptom. The middle case, reasonable but undiscussed content, is the one a program should rule on explicitly, since institutions differ on how much standard boilerplate a note may carry.

Fidelity, rather than restatement

A related distinction saves a rubric from a common and costly error. Several categories of clinical data live in structured EHR fields: allergies, the active medication list, the problem list. A generated note is under no obligation to restate them, and a follow-up note routinely omits an unchanged allergy status without any defect having occurred.

A rubric that scores allergy documentation by asking whether the note repeats the allergy list will therefore fail perfectly good notes. The criterion has to test fidelity instead. What matters is whether the note contradicts the recorded allergy status, and whether an allergy or adverse reaction newly elicited during the encounter reaches the record at all. The second case is the dangerous one: a patient describes a reaction to amoxicillin, the scribe drops it, the structured list is never updated, and the next prescriber sees no alert. An omitted restatement of existing data removes no safeguard. A dropped new allergy removes the safeguard entirely.

The same logic applies wherever a criterion could be misread as demanding repetition rather than accuracy.

From failure modes to boolean criteria for AI medical scribe evaluation

There are indications that evaluation practice is already moving in this direction. A 2025 study evaluated 97 patient encounters, each with paired LLM-generated and physician-authored notes, and retained Likert ratings for overall note quality while introducing a separate binary criterion for hallucination, scored as present or absent, an acknowledgment that fabricated content resists placement on an ordinal scale.

Recent work from Google Research generalizes this observation. In a study of health language model evaluation, the authors transformed rubric criteria based on Likert scales into a larger set of granular binary questions, an approach they term Precise Boolean rubrics, and compared the two designs directly. Boolean rubrics produced significantly higher inter-rater reliability than their Likert counterparts, as measured by intraclass correlation coefficients, across expert raters, non-expert raters, and automated evaluation. An adaptive variant, which filters the question set to those relevant for a given response, reduced evaluation time by more than half relative to Likert scoring while preserving the reliability gains. That study evaluated open-ended health-model responses rather than ambient scribe notes, so it supports the rubric-design principle rather than validating this scribe-specific rubric.

Applying this design philosophy to ambient documentation means converting each failure mode in the taxonomy into binary, individually verifiable criteria. An excerpt from the full rubric:

IDFailure modeCriterion (Yes / No)
AL1Allergy fidelity Allergy content in the note matches the defined source of truth, and any newly elicited allergy or adverse reaction is documented.
F1Fabrication Every clinical fact in the note is supported by a defined source of truth.
F3Fabricated pertinent negative Every recorded pertinent negative corresponds to a question asked or an examination performed.
M2Medication fidelity Every medication dose, route, and frequency matches the encounter.
N1Negation Every reported and denied symptom carries the polarity the patient gave it.
O1Omission Every clinically relevant patient-reported symptom is captured.
SN1Safety-netting Safety-netting advice, red-flag symptoms, and return precautions discussed in the encounter are documented accurately.
I1Diagnosis support Each diagnosis is supported by the documented note content or the defined source-of-truth set.

Each criterion carries scoring guidance that resolves its boundaries, because a criteria list without those rules invites two raters to score the same defect under different headings. Three examples of the kind of rule this requires. A symptom the patient reported but the note records as denied is a negation error rather than an omission, so O1 is treated as met and N1 fails. "Supported" in I1 means at least one documented finding is consistent with the diagnosis, deliberately a low bar, because a diagnosis contradicted by the note's own examination is an internal-consistency failure scored elsewhere. And a criterion returns N/A rather than a failure when the encounter puts it out of scope, which keeps it out of both the numerator and the denominator.

Three properties of this design merit emphasis:

  • Reliability: two physicians are far more likely to converge on whether a documented medication appears in the transcript than on whether a note's thoroughness merits a 3 or a 4, and the intraclass correlation results above bear this out empirically.
  • Actionability: each failed criterion identifies a specific defect of a specific type at a specific location, so the output of an evaluation run is a ranked list of correctable behaviors rather than a summary judgment.
  • Trendability: binary criteria yield per-failure-mode rates that can be tracked across model versions, allowing a team to report, for instance, that fabrication fell from 9% to 3% of notes between releases, a claim with far more engineering content than an improvement in mean quality score.

A limitation deserves acknowledgment: certain judgments remain genuinely scalar. The completeness of a complex assessment, or the reasoning quality behind a differential diagnosis, may be poorly served by decomposition into binary checks. A pragmatic design retains a small Likert layer for the few criteria that resist decomposition, while making binary criteria the default rather than the exception. This inverts the structure of most current instruments, where ordinal ratings dominate and binary checks appear only as supplements.

Gating criteria

A weighted score carries one structural risk: it lets a severe failure be offset by a long tail of criteria the note satisfies. Some errors should never be offset. A rubric handles this with gating criteria, where a single failure fails the note regardless of the total.

The candidates are the errors a program refuses to ship at any score: an inverted or fabricated risk assessment, a wrong laterality or misidentified patient, a contradicted or dropped allergy. Medication name and dose are common additions in higher-risk settings. Gating is what allows a rubric to hold a broad, forgiving criteria set and still refuse a note that would be indefensible.

A worked example

The example below is synthetic and constructed for illustration. It applies both scoring approaches to a single primary care encounter, so the difference in what each produces becomes visible on the same note. The Boolean pass is scored against the full downloadable rubric, and appears there as Example 1.

Panel 1 · The encounter

A 58-year-old woman with type 2 diabetes (metformin 1000 mg twice daily) and hypertension (amlodipine 5 mg daily) attends a routine follow-up. Over the visit:

  • Bilateral foot burning and tingling for two months, worse at night.
  • Occasional lightheadedness on standing, no falls.
  • Misses her evening metformin roughly twice a week.
  • Home fasting glucose 140–180 mg/dL.
  • Denies chest pain and visual changes.
  • Clinic: BP 148/86, weight stable.
  • Exam: reduced monofilament sensation both great toes, pedal pulses intact, no ulcers, heart regular, lungs clear.

Plan agreed: HbA1c, BMP, urine ACR; start gabapentin 100 mg nightly and titrate; reinforce evening metformin; continue amlodipine, recheck BP in 4 weeks; refer to podiatry and ophthalmology.

Panel 2 · AI-generated note

Subjective: 58-y woman with T2DM and HTN for follow-up. Bilateral foot burning/tingling ×2 months, worse at night. Reports increased thirst and urinary frequency. Home fasting glucose 140–180. Reports full adherence. Denies chest pain, visual changes, and dizziness.

Medications: Metformin 500 mg BID. Amlodipine 5 mg daily.

Objective: BP 148/86. Foot exam: monofilament intact bilaterally, no deficit; pulses intact; no ulcers. Heart regular; lungs clear.

Assessment and plan:

  • Type 2 diabetes, suboptimally controlled. Order HbA1c and basic metabolic panel. Continue metformin.
  • Bilateral foot pain, likely diabetic peripheral neuropathy. Start gabapentin 100 mg nightly and titrate. Refer to podiatry.
  • Hypertension. Continue amlodipine; recheck in four weeks.
  • Recommend annual diabetic eye examination; refer to ophthalmology.

Reads fluently, yet the exam is recorded normal while the assessment diagnoses neuropathy and starts gabapentin.

Panel 3 · Scored rubric
57/100
Fidelity score · 8 of 18 criteria failed
M2 Metformin dose halved
F2 Exam finding inverted
F1 Symptoms fabricated
N1 Dizziness polarity flipped
A2 Adherence overstated
O2 Urine ACR order dropped
O3b Counselling omitted
I3 Sections disagree

Gated: M2 (medication dose) is a gating criterion: The note fails outright regardless of score.

The note reads fluently and is well organized. It also carries one internal contradiction worth noting in advance: the examination is recorded as normal, while the assessment diagnoses neuropathy and starts gabapentin.

Scoring pass 1: Likert (PDQI-9)

To illustrate the difference, suppose two reviewers score the note on the nine PDQI-9 attributes, each from 1 to 5. The ratings below are constructed for this example rather than drawn from an empirical review:

PDQI-9 attributeReviewer AReviewer B
Up to date44
Accurate41
Thorough43
Useful53
Organized55
Comprehensible55
Succinct44
Synthesized43
Internally consistent42
Total (of 45)3930

Accuracy is the attribute where the danger concentrates - rated 4 by one reviewer and 1 by the other.

PDQI-9 has no universal release threshold, so without a predefined gating rule a program might read both totals as broadly acceptable, with the mean of roughly 34 of 45 suggesting modest disagreement about something broadly good. The single attribute where the danger concentrates, Accuracy, is rated 4 by one reviewer and 1 by the other, and that divergence is averaged into the aggregate rather than surfaced. Neither total tells the engineering team that the metformin dose is halved, that the foot examination is inverted, or that a symptom the patient never reported has been added.

Scoring pass 2: Weighted Boolean rubric

Now apply the full rubric to the same note. Each criterion is answered Yes, No, or N/A, and each carries a severity weight. Eighteen criteria apply to this encounter; four fall out of scope. Under the worked answer key, eight fail:

IDCriterionSeverityLocated defect
M2Every medication dose, route, and frequency matches the encounterCriticalMetformin recorded at 500 mg; actual 1000 mg twice daily.
F2Every recorded examination finding matches what was performedCriticalMonofilament recorded intact; actual reduced sensation both great toes.
F1Every clinical fact is supported by a defined source of truthCriticalAdded thirst and urinary frequency, absent from every source.
N1Every reported and denied symptom carries the polarity the patient gave itCritical"Denies dizziness"; lightheadedness on standing was reported.
A2Adherence statements match what the patient reportedMajor"Full adherence"; patient reported missing evening metformin twice weekly.
O2Every order, test, or referral discussed appears in the planMajorUrine albumin-to-creatinine ratio omitted.
O3bLifestyle and adherence counselling discussed appears in the noteMinorEvening-metformin adherence counselling omitted.
I3The note's sections agree with one anotherMinorExamination recorded normal while neuropathy is diagnosed and treated.

Because each criterion is a specific factual check rather than a holistic judgment, two reviewers applying this key record the same result on every criterion. Ten further criteria pass, and the note earns a fidelity score of 57 of 100.

That number deserves a comment, because it is higher than the defects suggest. A broad rubric credits what a note gets right, and this note gets a good deal right: the presenting complaint is captured, the gabapentin start is accurate, the referrals are present, the vital signs are correct. A halved diabetes drug dose and an inverted neuropathy examination still leave it above half marks. This is precisely the argument for gating rather than for a harsher scale. A program that designates M2 as a gating criterion fails this note outright, at 57 or at any other number, and the fidelity score becomes a description of the note rather than a verdict on it.

The two passes side by side

Likert · PDQI-9
VS
Weighted Boolean
Two totals, 39 and 30 of 45
Headline output
One fidelity score, 57 of 100
Diverges on the attribute that matters most (Accuracy, 4 vs 1)
Rater agreement
Any disagreement is localised to specific criteria
None
Defects localized
Eight - each typed, placed, and weighted
A judgment call on a disputed number
Release decision
Gating criteria decide it explicitly
Unclear
Engineering action
Fix dose extraction; correct exam capture; suppress fabricated symptoms; preserve symptom polarity; reconcile adherence.

Measuring inter-rater reliability

Whatever rubric a team adopts, the evaluation itself should be audited for agreement between raters, and the appropriate statistic depends on the criterion type.

Criterion typeRecommended statisticWorking guidance
Binary, two ratersCohen's κκ above 0.6 substantial, above 0.8 approaching ceiling; hold safety-critical and gating criteria to the higher bar.
Binary, three or more ratersFleiss' κSame thresholds.
Ordinal or LikertICCState the ICC model used — for example, two-way agreement.
Any typeChance-corrected figureReport a chance-corrected figure; treat raw percent agreement with caution — rare failure modes push it toward 100% even when raters catch different errors.

One caveat applies for readers of the scribe evaluation literature. Some published studies report high inter-rater agreement on Likert instruments using RWG, a within-group statistic generally considered more lenient than chance-corrected measures. Head-to-head comparisons using intraclass correlation have favored Boolean designs, and reliability is in any case only part of the argument: an itemized account of a note's specific errors remains more useful to a model team than a quality score, even on the occasions when raters agree about that score.

Applying this approach

The method summarized: define the source-of-truth set, construct a defect taxonomy for the application, convert each failure mode into binary criteria, weight them by severity, designate the criteria that gate, audit rater agreement with chance-corrected statistics, and track per-failure-mode rates across model releases.

The complete rubric behind this article is available as a template. It carries 22 criteria across three sections, clinical rationale and edge-case scoring guidance for each, a source-of-truth mode toggle, and four worked examples scored end to end.

FREE DOWNLOAD

Get the full Boolean rubric template

22 criteria across three sections, clinical rationale and edge-case scoring guidance for each, a source-of-truth mode toggle, and four worked examples scored end to end.

Grayde designs application-specific evaluation rubrics of this kind and administers them with credentialed clinicians on synthetic or de-identified data. Contact us about evaluation design, evaluation by experts and medical annotation.

Designing an evaluation program?

FAQ

What is the difference between a Likert rubric and a Boolean rubric for AI medical scribe evaluation?

A Likert rubric rates a note on a scale, usually 1 to 5, across attributes like accuracy and thoroughness. A Boolean rubric asks yes or no questions, each tied to one failure mode.

Likert scoring returns one aggregate number. Boolean scoring returns a list of specific defects, each with a type and a location.

Google Research found Boolean rubrics produced significantly higher inter-rater reliability than Likert equivalents.

Is PDQI-9 enough to evaluate an ambient AI scribe?

PDQI-9 describes overall note quality well. It is not enough on its own to improve a model. A note scoring 36 of 45 tells a team that reviewers found it moderately good. It does not say which errors are present or where.

A hallucinated exam finding and a verbose assessment can cost similar points. Their clinical consequences are nowhere close.

What is a fabricated pertinent negative, and why is it weighted so highly?

It is a normal finding recorded for an examination or question that never happened. An example is "denies palpitations" when the topic was never raised.

It ranks alongside outright fabrication because it is so hard to catch. A fabricated positive finding is an assertion, and assertions draw the reviewing eye. A fabricated negative reads as routine. It survives sign-off, and the next clinician believes the question was asked.

What should the source of truth be when scoring an AI-generated clinical note?

A defined set of sources, not the transcript alone. Legitimate content often comes from the prior EHR record, the medication list, the problem list, and clinician-entered fields.

A rubric scored against the transcript only will mark that content as fabrication and fail good notes. Raters need access to the same sources the scribe could use.

How do you score allergy documentation without failing correct notes?

Score fidelity, not restatement. A note that leaves out an unchanged allergy status contains no defect. Test two things instead. Does the note contradict the recorded allergy status? Does a newly elicited allergy reach the record?

The second case is the dangerous one. A dropped new allergy removes a safeguard.

Which inter-rater reliability statistic should be used for a Boolean scribe rubric?

Use a chance-corrected statistic matched to the criterion type. Cohen's kappa fits binary criteria with two raters. Fleiss' kappa fits three or more. Kappa above 0.6 is substantial and above 0.8 approaches the ceiling.

Ordinal criteria call for ICC, with the model stated. Treat raw percent agreement with caution, since rare failure modes push it toward 100 percent.