{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T09:03:07Z","timestamp":1777626187235,"version":"3.51.4"},"reference-count":23,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,6,13]],"date-time":"2025-06-13T00:00:00Z","timestamp":1749772800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0"},{"start":{"date-parts":[[2025,6,13]],"date-time":"2025-06-13T00:00:00Z","timestamp":1749772800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0"}],"funder":[{"DOI":"10.13039\/100000133","name":"Agency for Healthcare Research and Quality","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000133","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Duke Endowment grant for Evaluation & Governance"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"DOI":"10.1038\/s41746-025-01622-1","type":"journal-article","created":{"date-parts":[[2025,6,13]],"date-time":"2025-06-13T16:04:44Z","timestamp":1749830684000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["An evaluation framework for ambient digital scribing tools in clinical applications"],"prefix":"10.1038","volume":"8","author":[{"given":"Haoyuan","family":"Wang","sequence":"first","affiliation":[]},{"given":"Rui","family":"Yang","sequence":"additional","affiliation":[]},{"given":"Mahmoud","family":"Alwakeel","sequence":"additional","affiliation":[]},{"given":"Ankit","family":"Kayastha","sequence":"additional","affiliation":[]},{"given":"Anand","family":"Chowdhury","sequence":"additional","affiliation":[]},{"given":"Joshua M.","family":"Biro","sequence":"additional","affiliation":[]},{"given":"Anthony D.","family":"Sorrentino","sequence":"additional","affiliation":[]},{"given":"Jessica L.","family":"Handley","sequence":"additional","affiliation":[]},{"given":"Sarah","family":"Hantzmon","sequence":"additional","affiliation":[]},{"given":"Sophia","family":"Bessias","sequence":"additional","affiliation":[]},{"given":"Nicoleta J.","family":"Economou-Zavlanos","sequence":"additional","affiliation":[]},{"given":"Armando","family":"Bedoya","sequence":"additional","affiliation":[]},{"given":"Monica","family":"Agrawal","sequence":"additional","affiliation":[]},{"given":"Raj M.","family":"Ratwani","sequence":"additional","affiliation":[]},{"given":"Eric G.","family":"Poon","sequence":"additional","affiliation":[]},{"given":"Michael J.","family":"Pencina","sequence":"additional","affiliation":[]},{"given":"Kathryn I.","family":"Pollak","sequence":"additional","affiliation":[]},{"given":"Chuan","family":"Hong","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2025,6,13]]},"reference":[{"key":"1622_CR1","doi-asserted-by":"publisher","first-page":"753","DOI":"10.7326\/M16-0961","volume":"165","author":"C Sinsky","year":"2016","unstructured":"Sinsky, C. et al. Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties. Ann. Intern. Med. 165, 753\u2013760 (2016).","journal-title":"Ann. Intern. Med."},{"key":"1622_CR2","doi-asserted-by":"publisher","first-page":"564","DOI":"10.1001\/jamainternmed.2022.0372","volume":"182","author":"A Gaffney","year":"2022","unstructured":"Gaffney, A. et al. Medical documentation burden among US office-based physicians in 2019: a national study: a national study. JAMA Intern. Med. 182, 564\u2013566 (2022).","journal-title":"JAMA Intern. Med."},{"key":"1622_CR3","doi-asserted-by":"publisher","first-page":"202","DOI":"10.1093\/fampra\/cmn018","volume":"25","author":"C Pearce","year":"2008","unstructured":"Pearce, C., Trumble, S., Arnold, M., Dwan, K. & Phillips, C. Computers in the new consultation: within the first minute. Fam. Pract. 25, 202\u20138 (2008).","journal-title":"Fam. Pract."},{"key":"1622_CR4","doi-asserted-by":"publisher","first-page":"548","DOI":"10.1007\/s11606-015-3582-1","volume":"31","author":"MA Alkureishi","year":"2016","unstructured":"Alkureishi, M. A. et al. Impact of electronic medical record use on the patient\u2013doctor relationship and communication: a systematic review. J. Gen. Intern. Med. 31, 548\u201360 (2016).","journal-title":"J. Gen. Intern. Med."},{"key":"1622_CR5","first-page":"75","volume":"12","author":"N Booth","year":"2004","unstructured":"Booth, N., Robinson, P. & Kohannejad, J. Identification of high-quality consultation practice in primary care: the effects of computer use on doctor\u2013patient rapport. Inform. Prim. Care 12, 75\u201383 (2004).","journal-title":"Inform. Prim. Care"},{"key":"1622_CR6","first-page":"pp.CAT-23","volume":"5","author":"AA Tierney","year":"2024","unstructured":"Tierney, A. A. et al. Ambient artificial intelligence scribes to alleviate the burden of clinical documentation. JM Catal. Innov. Care Deliv. 5, pp.CAT-23 (2024).","journal-title":"JM Catal. Innov. Care Deliv."},{"key":"1622_CR7","doi-asserted-by":"publisher","first-page":"e2432460","DOI":"10.1001\/jamanetworkopen.2024.32460","volume":"7","author":"T-L Liu","year":"2024","unstructured":"Liu, T.-L. et al. AI-powered clinical documentation and clinicians\u2019 electronic health record experience: a nonrandomized clinical trial: a nonrandomized clinical trial. JAMA Netw. Open 7, e2432460 (2024).","journal-title":"JAMA Netw. Open"},{"key":"1622_CR8","unstructured":"Ma, S. P. et al. PRIME PubMed https:\/\/www.unboundmedicine.com\/medline\/citation\/39688515\/Ambient_artificial_intelligence_scribes:_utilization_and_impact_on_documentation_time (2024)."},{"key":"1622_CR9","doi-asserted-by":"publisher","first-page":"375","DOI":"10.1093\/jamia\/ocae295","volume":"32","author":"SJ Shah","year":"2024","unstructured":"Shah, S. J. et al. Ambient artificial intelligence scribes: physician burnout and perspectives on usability and documentation burden. J. Am. Med.Inform. Assoc. 32, 375\u2013380 (2024).","journal-title":"J. Am. Med.Inform. Assoc."},{"key":"1622_CR10","doi-asserted-by":"publisher","first-page":"491","DOI":"10.1093\/jamia\/ocz192","volume":"27","author":"S Reddy","year":"2020","unstructured":"Reddy, S., Allan, S., Coghlan, S. & Cooper, P. A governance model for the application of AI in health care. J. Am. Med. Inform. Assoc. 27, 491\u2013497 (2020).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"1622_CR11","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1109\/MIC.2017.4180835","volume":"21","author":"U Gasser","year":"2017","unstructured":"Gasser, U. & Almeida, V. A. F. A layered model for AI governance. IEEE Internet Comput. 21, 58\u201362 (2017).","journal-title":"IEEE Internet Comput."},{"key":"1622_CR12","doi-asserted-by":"publisher","first-page":"660","DOI":"10.1055\/a-2337-4739","volume":"15","author":"LM Owens","year":"2024","unstructured":"Owens, L. M., Wilda, J. J., Grifka, R., Westendorp, J. & Fletcher, J. J. Effect of ambient voice technology, natural language processing, and artificial intelligence on the patient\u2013physician relationship. Appl. Clin. Inf. 15, 660\u2013667 (2024).","journal-title":"Appl. Clin. Inf."},{"key":"1622_CR13","doi-asserted-by":"crossref","unstructured":"Selvaraj, S. P. & Konam, S. Medication regimen extraction from medical conversations: building a culture of transparency and accountability. In Explainable AI in Healthcare and Medicine Vol. 914 (eds Shaban-Nejad, A., Michalowski, M. & Buckeridge, D. L.) 195\u2013209 (Springer International Publishing, Cham, 2021).","DOI":"10.1007\/978-3-030-53352-6_18"},{"key":"1622_CR14","unstructured":"Pioneering the Science of AI Evaluation https:\/\/www.abridge.com\/ai\/science-ai-evaluation."},{"key":"1622_CR15","doi-asserted-by":"publisher","first-page":"e64993","DOI":"10.2196\/64993","volume":"27","author":"J Biro","year":"2025","unstructured":"Biro, J. et al. Accuracy and safety of AI-enabled scribe technology: instrument validation study. J. Med. Internet Res. 27, e64993 (2025).","journal-title":"J. Med. Internet Res."},{"key":"1622_CR16","doi-asserted-by":"publisher","first-page":"258","DOI":"10.1038\/s41746-024-01258-7","volume":"7","author":"TYC Tam","year":"2024","unstructured":"Tam, T. Y. C. et al. A framework for human evaluation of large language models in healthcare derived from literature review. NPJ Digit. Med. 7, 258 (2024).","journal-title":"NPJ Digit. Med."},{"key":"1622_CR17","doi-asserted-by":"publisher","first-page":"e49240","DOI":"10.2196\/49240","volume":"25","author":"E Sezgin","year":"2023","unstructured":"Sezgin, E., Chekeni, F., Lee, J. & Keim, S. Clinical accuracy of large language models and Google search responses to postpartum depression questions: cross-sectional study. J. Med. Internet Res. 25, e49240 (2023).","journal-title":"J. Med. Internet Res."},{"key":"1622_CR18","doi-asserted-by":"crossref","unstructured":"Abacha, A. B., Yim, W. W., Michalopoulos, G. & Lin, T. An investigation of evaluation methods in automatic medical note generation. In Findings of the Association for Computational Linguistics: ACL 2023 (pp. 2575\u20132588) (ACL, 2023).","DOI":"10.18653\/v1\/2023.findings-acl.161"},{"key":"1622_CR19","unstructured":"Panickssery, A., Bowman, S. R., & Feng, S. LLM Evaluators Recognize and Favor Their Own Generations. In A. Globerson, L. et al. (Eds.), Advances in Neural Information Processing Systems, Vol. 37. (Curran Associates, Inc, 2024)"},{"key":"1622_CR20","unstructured":"Radford, A. et al. Robust speech recognition via large-scale weak supervision. In Krause, A. et al. (Eds.), Proc. 40th International Conference on Machine Learning, Vol. 202, 28492\u201328518 (JMLR.org, 2023)."},{"key":"1622_CR21","unstructured":"Research. https:\/\/openai.com\/research\/."},{"key":"1622_CR22","doi-asserted-by":"publisher","first-page":"652","DOI":"10.1056\/NEJM196803212781204","volume":"278","author":"LL Weed","year":"1968","unstructured":"Weed, L. L. Medical records that guide and teach. N. Engl. J. Med. 278, 652\u2013657 (1968).","journal-title":"N. Engl. J. Med."},{"key":"1622_CR23","unstructured":"GitHub. unitaryai\/detoxify: Trained models & code to predict toxic comments on all 3 Jigsaw Toxic Comment Challenges. Built using Pytorch lightning and transformers. For access to our API, please email us at contact@unitary.ai. GitHub https:\/\/github.com\/unitaryai\/detoxify."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-01622-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-01622-1","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-01622-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,13]],"date-time":"2025-06-13T16:04:47Z","timestamp":1749830687000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-01622-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,13]]},"references-count":23,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["1622"],"URL":"https:\/\/doi.org\/10.1038\/s41746-025-01622-1","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,13]]},"assertion":[{"value":"29 December 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 April 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 June 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"358"}}