{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T17:41:01Z","timestamp":1781545261987,"version":"3.54.5"},"reference-count":55,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2025,5,5]],"date-time":"2025-05-05T00:00:00Z","timestamp":1746403200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"name":"National Institute of Health"},{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","award":["5T15LM007359"],"award-info":[{"award-number":["5T15LM007359"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","award":["R00 LM014308-02"],"award-info":[{"award-number":["R00 LM014308-02"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","award":["R01LM012973"],"award-info":[{"award-number":["R01LM012973"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,6,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>As large language models (LLMs) are integrated into electronic health record (EHR) workflows, validated instruments are essential to evaluate their performance before implementation and as models and documentation practices evolve. Existing instruments for provider documentation quality are often unsuitable for the complexities of LLM-generated text and lack validation on real-world data. The Provider Documentation Summarization Quality Instrument (PDSQI-9) was developed to evaluate LLM-generated clinical summaries. This study aimed to validate the PDSQI-9 across key aspects of construct validity.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>Multi-document summaries were generated from real-world EHR data across multiple specialties using several LLMs (GPT-4o, Mixtral 8x7b, and Llama 3-8b). Validation included Pearson correlation analyses for substantive validity, factor analysis and Cronbach\u2019s \u03b1 for structural validity, inter-rater reliability (ICC and Krippendorff\u2019s \u03b1) for generalizability, a semi-Delphi process for content validity, and comparisons of high- versus low-quality summaries for discriminant validity. Raters underwent standardized training to ensure consistent application of the instrument.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Seven physician raters evaluated 779 summaries and answered 8329 questions, achieving over 80% power for inter-rater reliability. The PDSQI-9 demonstrated strong internal consistency (Cronbach\u2019s \u03b1 = 0.879; 95% CI, 0.867-0.891) and high inter-rater reliability (ICC\u2009=\u20090.867; 95% CI, 0.867-0.868), supporting structural validity and generalizability. Factor analysis identified a 4-factor model explaining 58% of the variance, representing organization, clarity, accuracy, and utility. Substantive validity was supported by correlations between note length and scores for Succinct (\u03c1 = \u22120.200, P\u2009=\u2009.029) and Organized (\u03c1 = \u22120.190, P\u2009=\u2009.037). The semi-Delphi process ensured clinically relevant attributes, and discriminant validity distinguished high- from low-quality summaries (P&amp;lt;.001).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>The PDSQI-9 showed high inter-rater reliability, internal consistency, and a meaningful factor structure that reliably captured key dimensions of documentation quality. It distinguished between high- and low-quality summaries, supporting its practical utility for health systems needing an evaluation instrument for LLMs.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>The PDSQI-9 demonstrates robust construct validity, supporting its use in clinical practice to evaluate LLM-generated summaries and facilitate safer, more effective integration of LLMs into healthcare workflows.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocaf068","type":"journal-article","created":{"date-parts":[[2025,4,17]],"date-time":"2025-04-17T09:46:12Z","timestamp":1744883172000},"page":"1050-1060","source":"Crossref","is-referenced-by-count":23,"title":["Development and validation of the provider documentation summarization quality instrument for large language models"],"prefix":"10.1093","volume":"32","author":[{"given":"Emma","family":"Croxford","sequence":"first","affiliation":[{"name":"Department of Biostatistics and Medical Informatics, University of Wisconsin , Madison, WI 53792,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9341-7360","authenticated-orcid":false,"given":"Yanjun","family":"Gao","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, University of Colorado\u2014Anschutz Medical , Aurora, CO 80045,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicholas","family":"Pellegrino","sequence":"additional","affiliation":[{"name":"Epic Systems , Verona, WI 53593,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Karen","family":"Wong","sequence":"additional","affiliation":[{"name":"Epic Systems , Verona, WI 53593,","place":["United States"]},{"name":"UW Health , Madison, WI 53726,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Graham","family":"Wills","sequence":"additional","affiliation":[{"name":"UW Health , Madison, WI 53726,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elliot","family":"First","sequence":"additional","affiliation":[{"name":"Epic Systems , Verona, WI 53593,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Miranda","family":"Schnier","sequence":"additional","affiliation":[{"name":"Epic Systems , Verona, WI 53593,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kyle","family":"Burton","sequence":"additional","affiliation":[{"name":"UW Health , Madison, WI 53726,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cris","family":"Ebby","sequence":"additional","affiliation":[{"name":"Department of Pediatrics, University of Wisconsin , Madison, WI 53792,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jillian","family":"Gorski","sequence":"additional","affiliation":[{"name":"UW Health , Madison, WI 53726,","place":["United States"]},{"name":"BerbeeWalsh Department of Emergency Medicine, University of Wisconsin , Madison, WI 53792,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew","family":"Kalscheur","sequence":"additional","affiliation":[{"name":"UW Health , Madison, WI 53726,","place":["United States"]},{"name":"Department of Medicine, University of Wisconsin , Madison, WI 53792,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samy","family":"Khalil","sequence":"additional","affiliation":[{"name":"UW Health , Madison, WI 53726,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-0436-6631","authenticated-orcid":false,"given":"Marie","family":"Pisani","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Wisconsin , Madison, WI 53792,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tyler","family":"Rubeor","sequence":"additional","affiliation":[{"name":"UW Health , Madison, WI 53726,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter","family":"Stetson","sequence":"additional","affiliation":[{"name":"Memorial Sloan Kettering Cancer Center , New York, NY 10065,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7577-7530","authenticated-orcid":false,"given":"Frank","family":"Liao","sequence":"additional","affiliation":[{"name":"UW Health , Madison, WI 53726,","place":["United States"]},{"name":"BerbeeWalsh Department of Emergency Medicine, University of Wisconsin , Madison, WI 53792,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cherodeep","family":"Goswami","sequence":"additional","affiliation":[{"name":"UW Health , Madison, WI 53726,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4584-3808","authenticated-orcid":false,"given":"Brian","family":"Patterson","sequence":"additional","affiliation":[{"name":"UW Health , Madison, WI 53726,","place":["United States"]},{"name":"BerbeeWalsh Department of Emergency Medicine, University of Wisconsin , Madison, WI 53792,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6368-4652","authenticated-orcid":false,"given":"Majid","family":"Afshar","sequence":"additional","affiliation":[{"name":"Department of Biostatistics and Medical Informatics, University of Wisconsin , Madison, WI 53792,","place":["United States"]},{"name":"UW Health , Madison, WI 53726,","place":["United States"]},{"name":"Department of Medicine, University of Wisconsin , Madison, WI 53792,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,5,5]]},"reference":[{"key":"2025052712455580700_ocaf068-B1","doi-asserted-by":"crossref","first-page":"ooae039","DOI":"10.1093\/jamiaopen\/ooae039","article-title":"Call me Dr Ishmael: trends in electronic health record notes available at emergency department visits and admissions","volume":"7","author":"Patterson","year":"2024","journal-title":"JAMIA Open"},{"key":"2025052712455580700_ocaf068-B2","volume-title":"To Err is Human: Building a Safer Health System","author":"Institute of Medicine (US) Committee on Quality of Health Care in America","year":"2000"},{"key":"2025052712455580700_ocaf068-B3","doi-asserted-by":"crossref","first-page":"718","DOI":"10.1136\/amiajnl-2012-000946","article-title":"Computerized provider documentation: findings and implications of a multisite study of clinicians and administrators","volume":"20","author":"Embi","year":"2013","journal-title":"J Am Med Inform Assoc"},{"key":"2025052712455580700_ocaf068-B4","author":"Team","year":"2024"},{"key":"2025052712455580700_ocaf068-B5","author":"Xiong","year":"2024"},{"key":"2025052712455580700_ocaf068-B6","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1162\/tacl_a_00638","article-title":"Lost in the middle: how language models use long contexts","volume":"12","author":"Liu","year":"2024","journal-title":"Trans Assoc Comput Linguist"},{"key":"2025052712455580700_ocaf068-B7","author":"Li","year":"2024"},{"key":"2025052712455580700_ocaf068-B8","author":"Croxford","year":"2024"},{"key":"2025052712455580700_ocaf068-B9","author":"Croxford","year":"2024"},{"key":"2025052712455580700_ocaf068-B10","author":"Moramarco","year":"2022"},{"key":"2025052712455580700_ocaf068-B11","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models: a systematic review","volume":"333","author":"Bedi","year":"2025","journal-title":"JAMA"},{"key":"2025052712455580700_ocaf068-B12","doi-asserted-by":"crossref","first-page":"258","DOI":"10.1038\/s41746-024-01258-7","article-title":"A framework for human evaluation of large language models in healthcare derived from literature review","volume":"7","author":"Tam","year":"2024","journal-title":"NPJ Digit Med"},{"key":"2025052712455580700_ocaf068-B13","doi-asserted-by":"crossref","first-page":"e54419","DOI":"10.2196\/54419","article-title":"Using ChatGPT-4 to create structured medical notes from audio recordings of physician-patient encounters: comparative study","volume":"26","author":"Kernberg","year":"2024","journal-title":"J Med Internet Res."},{"key":"2025052712455580700_ocaf068-B14","doi-asserted-by":"crossref","first-page":"660","DOI":"10.1055\/a-2337-4739","article-title":"Effect of ambient voice technology, natural language processing, and artificial intelligence on the patient-physician relationship","volume":"15","author":"Owens","year":"2024","journal-title":"Appl Clin Inform"},{"key":"2025052712455580700_ocaf068-B15","doi-asserted-by":"crossref","first-page":"CAT.23.0404","DOI":"10.1056\/CAT.23.0404","article-title":"Ambient artificial intelligence scribes to alleviate the burden of clinical documentation","volume":"5","author":"Tierney","year":"2024","journal-title":"NEJM Catalyst"},{"key":"2025052712455580700_ocaf068-B16","doi-asserted-by":"crossref","first-page":"164","DOI":"10.4338\/ACI-2011-11-RA-0070","article-title":"Assessing electronic note quality using the Physician Documentation Quality Instrument (PDQI-9)","volume":"3","author":"Stetson","year":"2012","journal-title":"Appl Clin Inform"},{"key":"2025052712455580700_ocaf068-B17","author":"Zhao","year":"2023"},{"key":"2025052712455580700_ocaf068-B18","volume-title":"The Delphi Method: Techniques and Applications","author":"Turoff"},{"key":"2025052712455580700_ocaf068-B19","author":"He","year":"2024"},{"key":"2025052712455580700_ocaf068-B20","volume-title":"Words Matter: Strategies to Reduce Bias in Electronic Health Records","author":"Casau","year":"2022"},{"key":"2025052712455580700_ocaf068-B21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3485766","article-title":"A survey of evaluation metrics used for NLG systems","volume":"55","author":"Sai","year":"2023","journal-title":"ACM Comput Surv"},{"key":"2025052712455580700_ocaf068-B22","author":"Cai","year":"2022"},{"key":"2025052712455580700_ocaf068-B23","author":"Adams","year":"2023"},{"key":"2025052712455580700_ocaf068-B24","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"2025052712455580700_ocaf068-B25","author":"Umapathi","year":"2023"},{"key":"2025052712455580700_ocaf068-B26","author":"Wallace","year":"2020"},{"key":"2025052712455580700_ocaf068-B27","author":"Otmakhova","year":"2022"},{"key":"2025052712455580700_ocaf068-B28","first-page":"806","author":"Cohan"},{"key":"2025052712455580700_ocaf068-B29","author":"Yadav","year":"2021"},{"key":"2025052712455580700_ocaf068-B30","author":"Guo","year":"2022"},{"key":"2025052712455580700_ocaf068-B31","author":"Abacha","year":"2023"},{"key":"2025052712455580700_ocaf068-B32","doi-asserted-by":"crossref","first-page":"377","DOI":"10.1016\/j.jbi.2008.08.010","article-title":"Research Electronic Data Capture (REDCap) \u2013 a metadata-driven methodology and workflow process for providing translational research informatics support","volume":"42","author":"Harris","year":"2009","journal-title":"J Biomed Inform"},{"key":"2025052712455580700_ocaf068-B33","doi-asserted-by":"crossref","first-page":"103208","DOI":"10.1016\/j.jbi.2019.103208","article-title":"The REDCap consortium: building an international community of software platform partners","volume":"95","author":"Harris","year":"2019","journal-title":"J Biomed Inform"},{"key":"2025052712455580700_ocaf068-B34","author":"OpenAI","year":"2024"},{"key":"2025052712455580700_ocaf068-B35","author":"Jiang","year":"2024"},{"key":"2025052712455580700_ocaf068-B36","author":"Grattafiori","year":"2024"},{"key":"2025052712455580700_ocaf068-B37","author":"Hugging Face","year":"2024"},{"key":"2025052712455580700_ocaf068-B38","author":"Rotondi","year":"2018"},{"key":"2025052712455580700_ocaf068-B39","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1111\/j.1745-3992.1995.tb00881.x","volume":"14","author":"Messick","year":"1995","journal-title":"Educ Meas Issues Pract"},{"key":"2025052712455580700_ocaf068-B40","volume-title":"Content Analysis: An Introduction to Its Methodology","author":"Krippendorff","year":"2018"},{"key":"2025052712455580700_ocaf068-B41","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1007\/978-1-4612-4380-9_6","volume-title":"Statistical Methods for Research Workers","author":"Fisher","year":"1992"},{"key":"2025052712455580700_ocaf068-B42","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A guideline of selecting and reporting intraclass correlation coefficients for reliability research","volume":"15","author":"Koo","year":"2016","journal-title":"J Chiropr Med"},{"key":"2025052712455580700_ocaf068-B43","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1007\/BF02310555","article-title":"Coefficient alpha and the internal structure of tests","volume":"16","author":"Cronbach","year":"1951","journal-title":"Psychometrika"},{"key":"2025052712455580700_ocaf068-B44","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1177\/014662168701100107","article-title":"Statistical inference for coefficient alpha","volume":"11","author":"Feldt","year":"1987","journal-title":"Appl Psychol Meas"},{"key":"2025052712455580700_ocaf068-B45","doi-asserted-by":"crossref","first-page":"420","DOI":"10.1037\/0033-2909.86.2.420","article-title":"Intraclass correlations: uses in assessing rater reliability","volume":"86","author":"Shrout","year":"1979","journal-title":"Psychol Bull"},{"key":"2025052712455580700_ocaf068-B46","author":"Wolf","year":"2020"},{"key":"2025052712455580700_ocaf068-B47","volume-title":"Natural Language Processing with Python","author":"Bird","year":"2009"},{"key":"2025052712455580700_ocaf068-B48","doi-asserted-by":"crossref","first-page":"802894","DOI":"10.3389\/fgene.2021.802894","article-title":"smplot: an R package for easy and elegant data visualization","volume":"12","author":"Min","year":"2021","journal-title":"Front Genet"},{"key":"2025052712455580700_ocaf068-B49","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-24277-4","volume-title":"ggplot2: Elegant Graphics for Data Analysis","author":"Wickham","year":"2016"},{"key":"2025052712455580700_ocaf068-B50","author":"Revelle","year":"2024"},{"key":"2025052712455580700_ocaf068-B51","author":"Hughes","year":"2021"},{"key":"2025052712455580700_ocaf068-B52","author":"Signorell","year":"2017"},{"key":"2025052712455580700_ocaf068-B53","doi-asserted-by":"crossref","first-page":"e55957","DOI":"10.2196\/55957","article-title":"Toward clinical generative AI: conceptual framework","volume":"3","author":"Bragazzi","year":"2024","journal-title":"JMIR AI."},{"key":"2025052712455580700_ocaf068-B54","doi-asserted-by":"crossref","first-page":"e2117052","DOI":"10.1001\/jamanetworkopen.2021.17052","article-title":"Physician use of stigmatizing language in patient medical records","volume":"4","author":"Park","year":"2021","journal-title":"JAMA Network Open"},{"key":"2025052712455580700_ocaf068-B55","doi-asserted-by":"crossref","first-page":"320","DOI":"10.1038\/s41746-024-01315-1","article-title":"A strategy for cost-effective large language model use at health system-scale","volume":"7","author":"Klang","year":"2024","journal-title":"NPJ Digit Med"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/6\/1050\/63054828\/ocaf068.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/6\/1050\/63054828\/ocaf068.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,5,27]],"date-time":"2025-05-27T16:46:08Z","timestamp":1748364368000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/6\/1050\/8125016"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,5]]},"references-count":55,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2025,5,5]]},"published-print":{"date-parts":[[2025,6,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaf068","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,6]]},"published":{"date-parts":[[2025,5,5]]}}}