Annals of Internal MedicineRapid Evaluation of Artificial Intelligence Technology Used for Ambient Dictation in Primary Care: Comparing the Quality of Documentation of Artificial Intelligence–Generated and Human-Produced Clinical Notes

⚠️ Small Study / Early Comparative Evidence

A VHA evaluation of 11 ambient AI scribe tools found AI-generated notes scored lower than human-produced notes across all 5 standardized primary care cases and all 10 PDQI-9 quality domains. The largest gap appeared in the acute low back pain case.

Clinical Considerations

• AI notes scored 23.5 points below human notes on the modified PDQI-9 in the acute low back pain case (20.3 vs. 43.8 out of 50).
• The largest domain deficits were thoroughness, organization, and usefulness, the elements most tied to clinical decision-making.
• All 11 vendor tools underperformed, suggesting the gap is not isolated to a single product or training approach.
• Findings come from simulated cases, so real-world performance under time pressure remains unmeasured in either direction.

Practice Applications

• Recognize that current ambient AI scribes may not match human documentation quality in primary care encounters.
• Review AI-generated notes carefully before signing, with particular attention to thoroughness and clinical reasoning.
• Avoid treating vendor demonstrations as substitutes for independent quality evaluation.
• Monitor for vendor-neutral performance data before expanding ambient AI deployment across a practice.

More in AI