{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,22]],"date-time":"2026-10-22T08:40:35Z","timestamp":1792658435317,"version":"4.5.0"},"reference-count":39,"publisher":"Oxford University Press (OUP)","issue":"11","license":[{"start":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T00:00:00Z","timestamp":1790121600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"name":"Rochester Regional Health"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,11,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>To demonstrate a replicable in-house framework for evaluating ambient digital scribe (ADS) systems and, applying it, to assess 3 commercial systems and examine alignment between clinician Likert ratings and a manual transcript-level error audit.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>Two standardized clinical encounters\u2014a brief problem-focused visit and a comprehensive annual visit\u2014were simulated and audio-recorded. Three ADS vendors generated notes from identical recordings. A multidisciplinary panel of 56 physicians rated the notes on 7-point Likert scales across 4 domains: relevance, accuracy, quality, and provider confidence. In parallel, physician reviewers cross-referenced each note against the source recording, enumerating hallucinations, omissions, and inaccuracies using a pre-specified rubric.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>In this exploratory sample, Vendor A received the highest mean Likert ratings across all 4 domains (Friedman \u03c7\u00b2(2)\u2009=\u200915.529, P\u2009&amp;lt;\u2009.001 short visit; \u03c7\u00b2(2)\u2009=\u200910.753, P\u2009=\u2009.005 long visit). After Bonferroni correction, only Vendor A vs Vendor B on the short visit remained significant (P\u2009=\u2009.0098). The audit identified fewest errors in Vendor B\u2019s notes (n\u2009=\u20093), followed by Vendor A (n\u2009=\u20098) and Vendor C (n\u2009=\u20099). Physicians flagged hallucinations more often than the audit confirmed, a pattern we tentatively label \u201cpseudo-hallucinations.\u201d<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>In this sample, clinician preference did not track audit-identified error counts, raising the hypothesis that note-style features influence ratings alongside documentation correctness; larger samples are required.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Pairing clinician Likert assessments with transcript-level error audits may be a useful adjunct in evaluating ADS and other generative AI clinical tools; confirmation in larger studies is needed before this framework drives procurement decisions.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocag157","type":"journal-article","created":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T11:45:17Z","timestamp":1787312717000},"page":"2177-2185","source":"Crossref","is-referenced-by-count":0,"title":["Comparative evaluation of ambient digital scribe systems in clinical documentation"],"prefix":"10.1093","volume":"33","author":[{"given":"Everett","family":"Weiss","sequence":"first","affiliation":[{"name":"Department of Clinical Informatics, Rochester Regional Health , Rochester, NY 14617,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Balazs","family":"Zsenits","sequence":"additional","affiliation":[{"name":"Department of Clinical Informatics, Rochester Regional Health , Rochester, NY 14617,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Farhad","family":"Nasar","sequence":"additional","affiliation":[{"name":"Department of Clinical Informatics, Rochester Regional Health , Rochester, NY 14617,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matt","family":"Phillips","sequence":"additional","affiliation":[{"name":"Department of Clinical Informatics, Rochester Regional Health , Rochester, NY 14617,","place":["United States"]},{"name":"Department of Management Information Systems, Saunders College of Business, Rochester Institute of Technology , Rochester, NY 14623,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Venugopal","family":"Mudgundi","sequence":"additional","affiliation":[{"name":"Department of Clinical Informatics, Rochester Regional Health , Rochester, NY 14617,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tamer","family":"Salhab","sequence":"additional","affiliation":[{"name":"Department of Clinical Informatics, Rochester Regional Health , Rochester, NY 14617,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrew","family":"Muth","sequence":"additional","affiliation":[{"name":"Department of Clinical Informatics, Rochester Regional Health , Rochester, NY 14617,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kaiwen","family":"Zhu","sequence":"additional","affiliation":[{"name":"Department of Clinical Informatics, Rochester Regional Health , Rochester, NY 14617,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joanna","family":"Kolosko","sequence":"additional","affiliation":[{"name":"Department of Clinical Informatics, Rochester Regional Health , Rochester, NY 14617,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sean","family":"Hansen","sequence":"additional","affiliation":[{"name":"Department of Management Information Systems, Saunders College of Business, Rochester Institute of Technology , Rochester, NY 14623,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Quang Neo","family":"Bui","sequence":"additional","affiliation":[{"name":"Department of Management Information Systems, Saunders College of Business, Rochester Institute of Technology , Rochester, NY 14623,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ahmad","family":"Yaseen","sequence":"additional","affiliation":[{"name":"Department of Clinical Informatics, Rochester Regional Health , Rochester, NY 14617,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mostafa","family":"Balboul","sequence":"additional","affiliation":[{"name":"Department of Clinical Informatics, Rochester Regional Health , Rochester, NY 14617,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jee","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Clinical Informatics, Rochester Regional Health , Rochester, NY 14617,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Orlando Andres","family":"Nunez Isaac","sequence":"additional","affiliation":[{"name":"Department of Clinical Informatics, Rochester Regional Health , Rochester, NY 14617,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,9,23]]},"reference":[{"key":"2026102204065411100_ocag157-B1","doi-asserted-by":"publisher","first-page":"21501319231166921","DOI":"10.1177\/21501319231166921","article-title":"Burnout related to electronic health record use in primary care","volume":"14","author":"Budd","year":"2023","journal-title":"J Prim Care Community Health"},{"key":"2026102204065411100_ocag157-B2","doi-asserted-by":"publisher","first-page":"371","DOI":"10.3122\/jabfm.2015.03.140224","article-title":"The use of medical scribes in health care settings: a systematic review and future directions","volume":"28","author":"Shultz","year":"2015","journal-title":"J Am Board Fam Med"},{"key":"2026102204065411100_ocag157-B3","doi-asserted-by":"publisher","first-page":"e41223","DOI":"10.2196\/41223","article-title":"Novel note templates to enhance signal and reduce noise in medical documentation: prospective improvement study","volume":"7","author":"Feldman","year":"2023","journal-title":"JMIR Form Res"},{"key":"2026102204065411100_ocag157-B4","doi-asserted-by":"publisher","first-page":"e169","DOI":"10.1093\/jamia\/ocv152","article-title":"Risks and benefits of speech recognition for clinical documentation: a systematic review","volume":"23","author":"Hodgson","year":"2016","journal-title":"J Am Med Inform Assoc"},{"key":"2026102204065411100_ocag157-B5","doi-asserted-by":"publisher","first-page":"e57673","DOI":"10.2196\/57673","article-title":"The utility and implications of ambient scribes in primary care","volume":"3","author":"Seth","year":"2024","journal-title":"Jmir Ai"},{"key":"2026102204065411100_ocag157-B6","doi-asserted-by":"publisher","first-page":"84","DOI":"10.1016\/j.dss.2004.04.014","article-title":"Design and evaluation of an intelligent decision support system for nuclear emergencies","volume":"41","author":"Papamichail","year":"2005","journal-title":"Decis Support Syst"},{"key":"2026102204065411100_ocag157-B7","doi-asserted-by":"publisher","first-page":"100292","DOI":"10.1016\/j.mcpdig.2025.100292","article-title":"Evaluating the quality and safety of ambient digital scribe platforms using simulated ambulatory encounters","volume":"3","author":"Anderson","year":"2025","journal-title":"Mayo Clinic Proc Digital Health"},{"key":"2026102204065411100_ocag157-B8","doi-asserted-by":"publisher","first-page":"20552076251395588","DOI":"10.1177\/20552076251395588","article-title":"Evaluating an artificial intelligence scribe for clinical documentation","volume":"11","author":"Alpert","year":"2025","journal-title":"Digit Health"},{"key":"2026102204065411100_ocag157-B9","doi-asserted-by":"publisher","first-page":"1691499","DOI":"10.3389\/frai.2025.1691499","article-title":"Assessing the quality of AI-generated clinical notes: validated evaluation of a large language model ambient scribe","volume":"8","author":"Palm","year":"2025","journal-title":"Front Artif Intell"},{"key":"2026102204065411100_ocag157-B10","doi-asserted-by":"publisher","DOI":"10.48550\/arxiv.2410.15528","article-title":"Improving clinical documentation with AI: a comparative study of Sporo AI Scribe and GPT-4o mini","author":"Lee","year":"2024"},{"key":"2026102204065411100_ocag157-B11","doi-asserted-by":"publisher","DOI":"10.48550\/arxiv.2411.06713","article-title":"Ambient AI scribing support: comparing the performance of specialized AI agentic architecture to leading foundational models","author":"Lee","year":"2024"},{"key":"2026102204065411100_ocag157-B12","doi-asserted-by":"publisher","author":"Jain","year":"2025","DOI":"10.1101\/2025.06.24.25330085"},{"key":"2026102204065411100_ocag157-B13","doi-asserted-by":"publisher","first-page":"273","DOI":"10.1093\/jamia\/ocaf180","article-title":"Evaluating ambient artificial intelligence documentation: effects on work efficiency, documentation burden, and patient-centered care","volume":"33","author":"Guo","year":"2025","journal-title":"J Am Med Inform Assoc"},{"key":"2026102204065411100_ocag157-B14","doi-asserted-by":"publisher","first-page":"1447","DOI":"10.3390\/healthcare13121447","article-title":"The impact of AI scribes on streamlining clinical documentation: a systematic review","volume":"13","author":"Sasseville","year":"2025","journal-title":"Healthcare (Basel)"},{"key":"2026102204065411100_ocag157-B15","doi-asserted-by":"publisher","first-page":"e76743","DOI":"10.2196\/76743","article-title":"Real-world evidence synthesis of digital scribes using ambient listening and generative artificial intelligence for clinician documentation workflows: a rapid review","volume":"4","author":"Kanaparthy","year":"2025","journal-title":"Jmir Ai"},{"key":"2026102204065411100_ocag157-B16","doi-asserted-by":"publisher","first-page":"2993091X251384313","DOI":"10.1177\/2993091X251384313","article-title":"Ambient artificial intelligence in health care documentation: a review of tools, integration, and clinical implications","author":"Hundal","year":"2024","journal-title":"SAGE Open Med"},{"key":"2026102204065411100_ocag157-B17","doi-asserted-by":"publisher","first-page":"753","DOI":"10.7326\/M16-0961","article-title":"Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties","volume":"165","author":"Sinsky","year":"2016","journal-title":"Ann Intern Med"},{"key":"2026102204065411100_ocag157-B18","doi-asserted-by":"publisher","first-page":"491","DOI":"10.1016\/j.mayocp.2021.11.021","article-title":"Changes in burnout and satisfaction with work-life integration in physicians and the general US working population between 2011 and 2020","volume":"97","author":"Shanafelt","year":"2022","journal-title":"Mayo Clin Proc"},{"key":"2026102204065411100_ocag157-B19","doi-asserted-by":"publisher","author":"Lukac","year":"2025","DOI":"10.1101\/2025.07.10.25331333"},{"key":"2026102204065411100_ocag157-B20","doi-asserted-by":"publisher","first-page":"e2460637","DOI":"10.1001\/jamanetworkopen.2024.60637","article-title":"Clinician experiences with ambient scribe technology to assist with documentation burden and efficiency","volume":"8","author":"Duggan","year":"2025","journal-title":"JAMA Netw Open"},{"key":"2026102204065411100_ocag157-B21","doi-asserted-by":"publisher","first-page":"e2528056","DOI":"10.1001\/jamanetworkopen.2025.28056","article-title":"Ambient documentation technology in clinician experience of documentation burden and burnout","volume":"8","author":"You","year":"2025","journal-title":"JAMA Netw Open"},{"key":"2026102204065411100_ocag157-B22","doi-asserted-by":"publisher","DOI":"10.21203\/rs.3.rs-7033639\/v1","article-title":"Performance, acceptability, and impact of ambient listening scribe technology in an outpatient context: a mixed methods trial evaluation","volume":"26","author":"Memon","year":"2025","journal-title":"Res Sq"},{"key":"2026102204065411100_ocag157-B23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3571730","article-title":"Survey of hallucination in natural language generation","volume":"55","author":"Ji","year":"2023","journal-title":"ACM Comput Surv"},{"key":"2026102204065411100_ocag157-B24","doi-asserted-by":"publisher","first-page":"e35179","DOI":"10.7759\/cureus.35179","article-title":"Artificial hallucinations in ChatGPT: implications in scientific writing","volume":"15","author":"Alkaissi","year":"2023","journal-title":"Cureus"},{"key":"2026102204065411100_ocag157-B25","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1038\/s41746-023-00939-z","article-title":"Large language models propagate race-based medicine","volume":"6","author":"Omiye","year":"2023","journal-title":"NPJ Digit Med"},{"key":"2026102204065411100_ocag157-B26","doi-asserted-by":"publisher","author":"Knoll","year":"2022","DOI":"10.18653\/v1\/2022.naacl-main.29"},{"key":"2026102204065411100_ocag157-B27","doi-asserted-by":"publisher","first-page":"2625","DOI":"10.1007\/s11606-024-08924-2","article-title":"Impact of an artificial intelligence-based solution on clinicians\u2019 clinical documentation experience: initial findings using ambient listening technology","volume":"39","author":"Galloway","year":"2024","journal-title":"J Gen Intern Med"},{"key":"2026102204065411100_ocag157-B28","doi-asserted-by":"publisher","first-page":"ocae295","DOI":"10.1093\/jamia\/ocae295","article-title":"Ambient artificial intelligence scribes: physician burnout and perspectives on usability and documentation burden","volume":"32","author":"Shah","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2026102204065411100_ocag157-B29","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1055\/a-2657-8087","article-title":"The influence of artificial intelligence scribes on clinician experience and efficiency among pediatric subspecialists: a rapid, randomized quality improvement trial","volume":"16","author":"Jahan","year":"2025","journal-title":"Appl Clin Inform"},{"key":"2026102204065411100_ocag157-B30","doi-asserted-by":"publisher","first-page":"319","DOI":"10.2307\/249008","article-title":"Perceived usefulness, perceived ease of use, and user acceptance of information technology","volume":"13","author":"Davis","year":"1989","journal-title":"MIS Q"},{"key":"2026102204065411100_ocag157-B31","doi-asserted-by":"publisher","first-page":"186","DOI":"10.1287\/mnsc.46.2.186.11926","article-title":"A theoretical extension of the technology acceptance model: four longitudinal field studies","volume":"46","author":"Venkatesh","year":"2000","journal-title":"Manage Sci"},{"key":"2026102204065411100_ocag157-B32","doi-asserted-by":"publisher","first-page":"53","DOI":"10.5116\/ijme.4dfb.8dfd","article-title":"Making sense of Cronbach\u2019s alpha","volume":"2","author":"Tavakol","year":"2011","journal-title":"Int J Med Educ"},{"key":"2026102204065411100_ocag157-B33","doi-asserted-by":"publisher","first-page":"13","DOI":"10.20982\/tqmp.04.1.p013","article-title":"The Mann-Whitney U: a test for assessing whether two independent samples come from the same distribution","volume":"4","author":"Nachar","year":"2008","journal-title":"TQMP"},{"key":"2026102204065411100_ocag157-B34","doi-asserted-by":"publisher","first-page":"591","DOI":"10.1093\/biomet\/52.3-4.591","article-title":"An analysis of variance test for normality (complete samples)","volume":"52","author":"Shapiro","year":"1965","journal-title":"Biometrika"},{"key":"2026102204065411100_ocag157-B35","doi-asserted-by":"publisher","first-page":"e47184","DOI":"10.2196\/47184","article-title":"Investigating the impact of user trust on the adoption and use of ChatGPT: survey analysis","volume":"25","author":"Choudhury","year":"2023","journal-title":"J Med Internet Res"},{"key":"2026102204065411100_ocag157-B36","doi-asserted-by":"publisher","first-page":"1124","DOI":"10.1126\/science.185.4157.1124","article-title":"Judgment under uncertainty: heuristics and biases","volume":"185","author":"Tversky","year":"1974","journal-title":"Science"},{"key":"2026102204065411100_ocag157-B37","doi-asserted-by":"publisher","first-page":"e367","DOI":"10.2196\/jmir.8775","article-title":"Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies","volume":"19","author":"Greenhalgh","year":"2017","journal-title":"J Med Internet Res"},{"key":"2026102204065411100_ocag157-B38","doi-asserted-by":"publisher","first-page":"158","DOI":"10.1038\/s41746-023-00896-7","article-title":"Evaluating large language models on medical evidence summarization","volume":"6","author":"Tang","year":"2023","journal-title":"NPJ Digit Med"},{"key":"2026102204065411100_ocag157-B39","doi-asserted-by":"publisher","author":"Liu","DOI":"10.1162\/tacl_a_00638"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocag157\/71351982\/ocag157.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/11\/2177\/71351982\/ocag157.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/11\/2177\/71351982\/ocag157.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,22]],"date-time":"2026-10-22T08:06:58Z","timestamp":1792656418000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/33\/11\/2177\/8832258"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,23]]},"references-count":39,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2026,9,23]]},"published-print":{"date-parts":[[2026,11,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocag157","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,11]]},"published":{"date-parts":[[2026,9,23]]}}}