{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"medRxiv"}],"indexed":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T08:51:27Z","timestamp":1791190287872,"version":"4.1.0"},"posted":{"date-parts":[[2026,9,7]]},"group-title":"Health Systems and Quality Improvement","reference-count":16,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2026,9,7]],"date-time":"2026-09-07T00:00:00Z","timestamp":1788739200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.medrxiv.org\/about\/FAQ#license"}],"funder":[{"name":"National Institutes of Health","award":["K08DK131286"],"award-info":[{"award-number":["K08DK131286"]}]},{"name":"National Institutes of Health","award":["R01DK133539"],"award-info":[{"award-number":["R01DK133539"]}]},{"award":["K08DK131286"],"award-info":[{"award-number":["K08DK131286"]}],"id":[{"id":"https:\/\/ror.org\/045p44t13","id-type":"ROR","asserted-by":"publisher"}]},{"award":["R01DK133539"],"award-info":[{"award-number":["R01DK133539"]}],"id":[{"id":"https:\/\/ror.org\/045p44t13","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2026,9,7]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To evaluate Epic IP Insights, an electronic health record integrated generative AI summarizer used across a seven-hospital health system, using a multidimensional assurance framework.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We developed and validated an agentic hallucination detector that decomposed summaries into atomic content units (ACUs) and verified each against source documentation. We then applied it to remaining summaries to estimate the hallucination rate. We also measured source note utilization, textual and semantic similarity of regenerated summaries, and clinician perceptions through structured evaluations.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The EPIC IP Insights generated 706 summaries across 445 encounters and used a mean of 5.8% of available notes. The detector was developed on 30 summaries (2,012 ACUs) and validated on 5 held out summaries (295 ACUs). The detector agreed with physician adjudication on 96.95% ACUs in the validation set. Across 671 remaining summaries containing 40,452 ACUs, the hallucination rate was 11.79% (95% CI, 11.47% to 12.10%). Among 385 regenerated pairs of summaries, 32.7% were textually identical and 37.9% were semantically identical. Clinicians found the tool easy to use (97.8%), while responses were more mixed regarding reliance on the output with little verification (45.9%), expected efficiency gains (45.3%), and frequent use (46.1%).<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Clinicians found the tool easy to use but believed that outputs required verification. Systematic evaluation identified the frequency of hallucinations and limited use of available notes which provide directions for future improvement.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>Multidimensional assurance frameworks are needed to evaluate the safety, reliability, and consistency of generative AI tools.<\/jats:p>\n                <\/jats:sec>","DOI":"10.64898\/2026.08.31.26361660","type":"posted-content","created":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T03:25:15Z","timestamp":1788837915000},"source":"Crossref","is-referenced-by-count":0,"title":["Multidimensional Assurance of an Electronic Health Record Embedded Generative Artificial Intelligence Summarization Tool"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-5196-7831","authenticated-orcid":false,"given":"Pavan","family":"Bodanki","sequence":"first","affiliation":[{"name":"The Windreich Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA"},{"name":"The Mount Sinai AI Assurance Lab, Mount Sinai Health System, New York, NY, USA"},{"name":"Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-9687-3421","authenticated-orcid":false,"given":"Christopher","family":"Capone","sequence":"additional","affiliation":[{"name":"The Windreich Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA"},{"name":"The Mount Sinai AI Assurance Lab, Mount Sinai Health System, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-4511-6495","authenticated-orcid":false,"given":"Nimay S","family":"Hazare","sequence":"additional","affiliation":[{"name":"The Windreich Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA"},{"name":"The Mount Sinai AI Assurance Lab, Mount Sinai Health System, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shamay","family":"Agaron","sequence":"additional","affiliation":[{"name":"The Windreich Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mahima","family":"Vijayaraghavan","sequence":"additional","affiliation":[{"name":"Division of Hospital Medicine, Department of Internal Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sohan","family":"Japa","sequence":"additional","affiliation":[{"name":"Division of Hospital Medicine, Department of Internal Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Irina","family":"Zaretsky","sequence":"additional","affiliation":[{"name":"Division of Hospital Medicine, Department of Internal Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeffrey","family":"Epstein","sequence":"additional","affiliation":[{"name":"Division of Hospital Medicine, Department of Internal Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ashwin","family":"Sawant","sequence":"additional","affiliation":[{"name":"The Windreich Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA"},{"name":"Division of Hospital Medicine, Department of Internal Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA"},{"name":"The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ahmed","family":"Shaikh","sequence":"additional","affiliation":[{"name":"Institute for Critical Care Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Evan","family":"Leibner","sequence":"additional","affiliation":[{"name":"Institute for Critical Care Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA"},{"name":"Department of Emergency Medicine, Institute of Critical Care Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aniket","family":"Sharma","sequence":"additional","affiliation":[{"name":"Institute for Critical Care Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sandeep","family":"Gangadharan","sequence":"additional","affiliation":[{"name":"Department of Pediatrics, Kravis Children\u2019s Hospital, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Timsal","family":"Ghani","sequence":"additional","affiliation":[{"name":"The Mount Sinai AI Assurance Lab, Mount Sinai Health System, New York, NY, USA"},{"name":"Division of Hospital Medicine, Department of Internal Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hua-Hsin","family":"Tai","sequence":"additional","affiliation":[{"name":"The Mount Sinai AI Assurance Lab, Mount Sinai Health System, New York, NY, USA"},{"name":"Division of Hospital Medicine, Department of Internal Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Ngai","sequence":"additional","affiliation":[{"name":"The Windreich Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA"},{"name":"The Mount Sinai AI Assurance Lab, Mount Sinai Health System, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sean","family":"El-Haj","sequence":"additional","affiliation":[{"name":"Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Prathamesh","family":"Parchure","sequence":"additional","affiliation":[{"name":"The Mount Sinai AI Assurance Lab, Mount Sinai Health System, New York, NY, USA"},{"name":"Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Annelia","family":"Itwaru","sequence":"additional","affiliation":[{"name":"The Mount Sinai AI Assurance Lab, Mount Sinai Health System, New York, NY, USA"},{"name":"Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ben","family":"Kaplan","sequence":"additional","affiliation":[{"name":"The Windreich Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA"},{"name":"The Mount Sinai AI Assurance Lab, Mount Sinai Health System, New York, NY, USA"},{"name":"Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aditi","family":"Vakil","sequence":"additional","affiliation":[{"name":"The Windreich Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ronald","family":"Tamler","sequence":"additional","affiliation":[{"name":"Division of Endocrinology, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Avniel","family":"Klein","sequence":"additional","affiliation":[{"name":"Department of Rehabilitation and Human Performance, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert","family":"Freeman","sequence":"additional","affiliation":[{"name":"The Windreich Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA"},{"name":"Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patricia","family":"Kovatch","sequence":"additional","affiliation":[{"name":"The Windreich Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bruce","family":"Darrow","sequence":"additional","affiliation":[{"name":"Mount Sinai Fuster Heart Hospital, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jolion","family":"McGreevy","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Institute of Critical Care Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lisa","family":"Stump","sequence":"additional","affiliation":[{"name":"The Windreich Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Girish N","family":"Nadkarni","sequence":"additional","affiliation":[{"name":"The Windreich Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA"},{"name":"The Charles Bronfman Department of Personalized Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA"},{"name":"The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, USA"},{"name":"Division of Nephrology, Department of Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Prem","family":"Timsina","sequence":"additional","affiliation":[{"name":"The Windreich Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA"},{"name":"The Mount Sinai AI Assurance Lab, Mount Sinai Health System, New York, NY, USA"},{"name":"Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ankit","family":"Sakhuja","sequence":"additional","affiliation":[{"name":"The Windreich Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA"},{"name":"The Mount Sinai AI Assurance Lab, Mount Sinai Health System, New York, NY, USA"},{"name":"The Charles Bronfman Department of Personalized Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA"},{"name":"The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, USA"},{"name":"Institute for Critical Care Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"54368","reference":[{"key":"2026100501000971000_2026.08.31.26361660v1.1","doi-asserted-by":"publisher","DOI":"10.1101\/2025.06.02.25328807"},{"key":"2026100501000971000_2026.08.31.26361660v1.2","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-02855-5"},{"key":"2026100501000971000_2026.08.31.26361660v1.3","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01670-7"},{"key":"2026100501000971000_2026.08.31.26361660v1.4","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocaf068"},{"key":"2026100501000971000_2026.08.31.26361660v1.5","doi-asserted-by":"publisher","DOI":"10.1038\/s44401-025-00064-x"},{"key":"2026100501000971000_2026.08.31.26361660v1.6","unstructured":"OpenAI. OpenAI. Introducing GPT-4. Secondary OpenAI. Introducing GPT-4 2023. https:\/\/openai.com\/index\/gpt-4-research\/?hl=en-IN."},{"key":"2026100501000971000_2026.08.31.26361660v1.7","unstructured":"ACUEval: Fine-grained Hallucination Evaluation and Correction for Abstractive Summarization; 2024 August; Bangkok, Thailand. Association for Computational Linguistics."},{"key":"2026100501000971000_2026.08.31.26361660v1.8","unstructured":"OpenAI. OpenAI. Introducing GPT-5-2. Secondary OpenAI. Introducing GPT-5-2 2026. https:\/\/openai.com\/index\/introducing-gpt-5-2\/."},{"key":"2026100501000971000_2026.08.31.26361660v1.9","unstructured":"Akl E , Altman D , Aluko P , et al. Cochrane Handbook for Systematic Reviews of Interventions, 2019."},{"key":"2026100501000971000_2026.08.31.26361660v1.10","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1952.10483441"},{"key":"2026100501000971000_2026.08.31.26361660v1.11","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177730491"},{"key":"2026100501000971000_2026.08.31.26361660v1.12","unstructured":"Zhang T , Kishore V , Wu F , Weinberger KQ , Artzi Y. BERTScore: Evaluating Text Generation with BERT. 2019. https:\/\/ui.adsabs.harvard.edu\/abs\/2019arXiv190409675Z (accessed April 01, 2019)."},{"key":"2026100501000971000_2026.08.31.26361660v1.13","doi-asserted-by":"publisher","DOI":"10.2307\/1422689"},{"key":"2026100501000971000_2026.08.31.26361660v1.14","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"key":"2026100501000971000_2026.08.31.26361660v1.15","unstructured":"Gwet K. Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters, 2012."},{"key":"2026100501000971000_2026.08.31.26361660v1.16","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000899"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.64898\/2026.08.31.26361660","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T08:00:18Z","timestamp":1791187218000},"score":1,"resource":{"primary":{"URL":"http:\/\/medrxiv.org\/lookup\/doi\/10.64898\/2026.08.31.26361660"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,7]]},"references-count":16,"URL":"https:\/\/doi.org\/10.64898\/2026.08.31.26361660","relation":{},"subject":[],"published":{"date-parts":[[2026,9,7]]},"subtype":"preprint"}}