{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T18:59:23Z","timestamp":1790276363295,"version":"4.1.0"},"reference-count":49,"publisher":"Oxford University Press (OUP)","license":[{"start":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T00:00:00Z","timestamp":1790208000000},"content-version":"am","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100009645","name":"National Academy of Medicine","doi-asserted-by":"publisher","award":["SCON-10001137"],"award-info":[{"award-number":["SCON-10001137"]}],"id":[{"id":"10.13039\/100009645","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000936","name":"Gordon and Betty Moore Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000936","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000909","name":"John A. Hartford Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000909","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Penn AITech"},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000049","name":"NIA","doi-asserted-by":"publisher","award":["P30AG073105"],"award-info":[{"award-number":["P30AG073105"]}],"id":[{"id":"10.13039\/100000049","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>Older adults face disproportionate risk of diagnostic errors in primary care, yet effective diagnostic decision support systems (DDSSs) remain lacking. Existing artificial intelligence (AI)-based DDSSs rely on gold-standard labels infeasible to obtain, have narrow diagnostic scope, and fail to reflect real-world uncertainty. We developed and validated a deep learning DDSS using imitation learning and collective clinician intelligence to generate diagnostic and order recommendations.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>We trained a multi-label neural network on structured and unstructured electronic health record data from primary care encounters of adults \u226565 years old to imitate clinician decisions. Models were evaluated using discrimination, calibration, threshold-based, and composite metrics on a temporally held-out test set. Clinical validity was assessed using a randomized, blinded equivalence study comparing model-generated recommendations with observed clinician decisions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The study included 707 598 primary care encounters, generating recommendations across 669 diagnoses and 1000 orders. The general model demonstrated excellent performance (micro c-statistic 0.995 [95% CI 0.995-0.996], macro c-statistic 0.896 [0.888-0.897], micro F1-score 0.904 [0.903-0.905], macro F1-score 0.384 [0.381-0.385], integrated calibration index (ICI) 0.0025 [0.0024-0.0026]). Order prediction was lower (micro c-statistic 0.812 [0.811-0.814], macro c-statistic 0.786 [0.777-0.786], micro F1-score 0.305 [0.303-0.307], macro F1-score 0.0286 [0.0277-0.0294], ICI 0.0145 [0.0141-0.0150]). In clinician-validation, diagnostic disagreements were statistically equivalent between model recommendations and observed practice while order disagreements were not.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Imitation learning and collective clinician intelligence provide a feasible framework for generating diagnostic and ordering recommendations that reflect real-world practice without expert-adjudicated training labels. Randomized, blinded clinician evaluation pragmatically establishes preliminary clinical validity beyond traditional retrospective metrics.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocag132","type":"journal-article","created":{"date-parts":[[2026,7,14]],"date-time":"2026-07-14T11:39:45Z","timestamp":1784029185000},"source":"Crossref","is-referenced-by-count":0,"title":["Artificial intelligence-based diagnostic decision support for primary care of older adults using collective intelligence and imitation learning"],"prefix":"10.1093","author":[{"given":"Christopher D","family":"Streiffer","sequence":"first","affiliation":[{"name":"Palliative and Advanced Illness Research (PAIR) Center, Perelman School of Medicine, University of Pennsylvania , Philadelphia, PA 19104,","place":["United States"]},{"name":"Department of Medicine, Perelman School of Medicine, University of Pennsylvania , Philadelphia, PA 19104,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew J","family":"Press","sequence":"additional","affiliation":[{"name":"Department of Medicine, Perelman School of Medicine, University of Pennsylvania , Philadelphia, PA 19104,","place":["United States"]},{"name":"Leonard Davis Institute of Health Economics, University of Pennsylvania , Philadelphia, PA 19104,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicholas S","family":"Bishop","sequence":"additional","affiliation":[{"name":"Palliative and Advanced Illness Research (PAIR) Center, Perelman School of Medicine, University of Pennsylvania , Philadelphia, PA 19104,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benjamin E","family":"Schmid","sequence":"additional","affiliation":[{"name":"Palliative and Advanced Illness Research (PAIR) Center, Perelman School of Medicine, University of Pennsylvania , Philadelphia, PA 19104,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alyssa","family":"Sliwa","sequence":"additional","affiliation":[{"name":"Palliative and Advanced Illness Research (PAIR) Center, Perelman School of Medicine, University of Pennsylvania , Philadelphia, PA 19104,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anuradha L","family":"Vyas","sequence":"additional","affiliation":[{"name":"Palliative and Advanced Illness Research (PAIR) Center, Perelman School of Medicine, University of Pennsylvania , Philadelphia, PA 19104,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lyle H","family":"Ungar","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Science, University of Pennsylvania , Philadelphia, PA 19104,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"MaryAnne","family":"Peifer","sequence":"additional","affiliation":[{"name":"Leonard Davis Institute of Health Economics, University of Pennsylvania , Philadelphia, PA 19104,","place":["United States"]},{"name":"Department of Family Medicine and Community Health, Perelman School of Medicine, University of Pennsylvania , Philadelphia, PA 19104,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gary E","family":"Weissman","sequence":"additional","affiliation":[{"name":"Palliative and Advanced Illness Research (PAIR) Center, Perelman School of Medicine, University of Pennsylvania , Philadelphia, PA 19104,","place":["United States"]},{"name":"Department of Medicine, Perelman School of Medicine, University of Pennsylvania , Philadelphia, PA 19104,","place":["United States"]},{"name":"Leonard Davis Institute of Health Economics, University of Pennsylvania , Philadelphia, PA 19104,","place":["United States"]},{"name":"Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania , Philadelphia, PA 19104,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,9,24]]},"reference":[{"key":"2026092414045577200_ocag132-B1","doi-asserted-by":"crossref","first-page":"137","DOI":"10.2147\/IJGM.S96741","article-title":"Diagnostic errors in older patients: a systematic review of incidence and potential causes in seven prevalent diseases","volume":"9","author":"Skinner","year":"2016","journal-title":"Int J Gen Med"},{"key":"2026092414045577200_ocag132-B2","doi-asserted-by":"crossref","first-page":"727","DOI":"10.1136\/bmjqs-2013-002627","article-title":"The frequency of diagnostic errors in outpatient care: estimations from three large observational studies involving US adult populations","volume":"23","author":"Singh","year":"2014","journal-title":"BMJ Qual Saf"},{"key":"2026092414045577200_ocag132-B3","volume-title":"Improving Diagnosis in Health Care","author":"Ball","year":"2015"},{"key":"2026092414045577200_ocag132-B4","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1136\/bmjqs-2016-005401","article-title":"The global burden of diagnostic errors in primary care","volume":"26","author":"Singh","year":"2017","journal-title":"BMJ Qual Saf"},{"key":"2026092414045577200_ocag132-B5","volume-title":"25-Year Summary of US Malpractice Claims for Diagnostic Errors 1986-2010","author":"Tehrani","year":"2013"},{"key":"2026092414045577200_ocag132-B6","doi-asserted-by":"crossref","first-page":"1821","DOI":"10.1377\/hlthaff.2018.0698","article-title":"Learning from patients\u2019 experiences related to diagnostic errors is essential for progress in patient safety","volume":"37","author":"Giardina","year":"2018","journal-title":"Health Aff"},{"key":"2026092414045577200_ocag132-B7","first-page":"335","article-title":"Foundations of medical decision-making for older adults with cardiovascular disease","volume":"12","author":"Lipman","year":"2015","journal-title":"J Geriatr Cardiol"},{"key":"2026092414045577200_ocag132-B8","article-title":"Updates in the role of health IT in patient safety","author":"Hall","year":"2020","journal-title":"PSNet Internet"},{"key":"2026092414045577200_ocag132-B9","doi-asserted-by":"crossref","first-page":"1905","DOI":"10.1001\/jama.2021.19493","article-title":"Diagnostic excellence","volume":"326","author":"Yang","year":"2021","journal-title":"JAMA"},{"key":"2026092414045577200_ocag132-B10","doi-asserted-by":"crossref","first-page":"4754","DOI":"10.1038\/s41467-020-18310-1","article-title":"Improving the accuracy of medical diagnosis with causal machine learning","volume":"11","author":"Richens","year":"2020","journal-title":"Nat Commun"},{"key":"2026092414045577200_ocag132-B11","doi-asserted-by":"crossref","first-page":"e271","DOI":"10.1016\/S2589-7500(19)30123-2","article-title":"A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging","volume":"1","author":"Liu","year":"2019","journal-title":"Lancet Digit Health"},{"key":"2026092414045577200_ocag132-B12","doi-asserted-by":"crossref","first-page":"2943","DOI":"10.1007\/s11606-020-06428-3","article-title":"Advancing diagnostic safety research","volume":"36","author":"Zwaan","year":"2021","journal-title":"J Gen Intern Med"},{"key":"2026092414045577200_ocag132-B13","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1126\/science.aax2342","article-title":"Dissecting racial bias in an algorithm used to manage the health of populations","volume":"366","author":"Obermeyer","year":"2019","journal-title":"Science"},{"key":"2026092414045577200_ocag132-B14","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1186\/s12875-017-0650-0","article-title":"Managing diagnostic uncertainty in primary care","volume":"18","author":"Alam","year":"2017","journal-title":"BMC Fam Pract"},{"key":"2026092414045577200_ocag132-B15","doi-asserted-by":"crossref","first-page":"8435","DOI":"10.3390\/ijerph18168435","article-title":"Clinical decision support systems for diagnosis in primary care: a scoping review","volume":"18","author":"Harada","year":"2021","journal-title":"Int J Environ Res Public Health"},{"key":"2026092414045577200_ocag132-B16","doi-asserted-by":"crossref","first-page":"256","DOI":"10.1136\/postgradmedj-2016-134328","article-title":"Application of visually based, computerised diagnostic decision support system in dermatological medical education","volume":"93","author":"Chou","year":"2017","journal-title":"Postgrad Med J"},{"key":"2026092414045577200_ocag132-B17","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1038\/s41746-018-0040-6","article-title":"Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy","volume":"1","author":"Abr\u00e0moff","year":"2018","journal-title":"NPJ Digit Med"},{"key":"2026092414045577200_ocag132-B18","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1515\/dx-2024-0151","article-title":"Diagnostic scope: the AI can\u2019t see what the mind doesn\u2019t know","volume":"12","author":"Weissman","year":"2025","journal-title":"Diagnosis"},{"key":"2026092414045577200_ocag132-B19","doi-asserted-by":"crossref","first-page":"506","DOI":"10.1093\/fampra\/cmt033","article-title":"Most common diseases diagnosed in primary care in Stockholm, Sweden","volume":"30","author":"W\u00e4ndell","year":"2013","journal-title":"Fam Pract"},{"key":"2026092414045577200_ocag132-B20","doi-asserted-by":"crossref","first-page":"1261","DOI":"10.1136\/bmj.39203.658970.55","article-title":"Diagnostic scope of and exposure to primary care physicians","volume":"334","author":"Bindman","year":"2007","journal-title":"BMJ"},{"key":"2026092414045577200_ocag132-B21","doi-asserted-by":"crossref","first-page":"418","DOI":"10.1001\/jamainternmed.2013.2777","article-title":"Types and origins of diagnostic errors in primary care settings","volume":"173","author":"Singh","year":"2013","journal-title":"JAMA Intern Med"},{"key":"2026092414045577200_ocag132-B22","doi-asserted-by":"crossref","DOI":"10.1093\/ageing\/afad219","article-title":"New horizons in artificial intelligence in the healthcare of older people","volume":"52","author":"Shiwani","year":"2023","journal-title":"Age Ageing"},{"key":"2026092414045577200_ocag132-B23","doi-asserted-by":"crossref","first-page":"1278","DOI":"10.1093\/geronb\/gbx106","article-title":"Activity engagement among older adult spousal caregivers","volume":"74","author":"Queen","year":"2019","journal-title":"J Gerontol Ser B"},{"key":"2026092414045577200_ocag132-B24","doi-asserted-by":"crossref","DOI":"10.1001\/jamainternmed.2015.7664","article-title":"A national profile of family and unpaid caregivers","author":"Wolff","year":"2016","journal-title":"JAMA Intern Med"},{"key":"2026092414045577200_ocag132-B25","doi-asserted-by":"crossref","first-page":"2916","DOI":"10.1111\/jgs.17337","article-title":"Caregiver involvement in managing medications among older adults","volume":"69","author":"O\u2019Conor","year":"2021","journal-title":"J Am Geriatr Soc"},{"key":"2026092414045577200_ocag132-B26","author":"Soysal","year":"2022"},{"key":"2026092414045577200_ocag132-B27","author":"Weissman","year":"2023"},{"key":"2026092414045577200_ocag132-B28","doi-asserted-by":"crossref","first-page":"312","DOI":"10.15288\/jsad.2022.83.312","article-title":"The influence of descriptive and injunctive norms on US adolescent marijuana use","volume":"83","author":"Mbaba","year":"2022","journal-title":"J Stud Alcohol Drugs"},{"key":"2026092414045577200_ocag132-B29","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1038\/s41467-022-35052-4","article-title":"Providing normative information increases intentions to accept a COVID-19 vaccine","volume":"14","author":"Moehring","year":"2023","journal-title":"Nat Commun"},{"key":"2026092414045577200_ocag132-B30","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1515\/dx-2022-0090","article-title":"Collective intelligence improves probabilistic diagnostic assessments","volume":"10","author":"Stehouwer","year":"2023","journal-title":"Diagnosis (Berl)"},{"key":"2026092414045577200_ocag132-B31","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2019.0096","article-title":"Comparative accuracy of diagnosis by collective intelligence","volume":"2","author":"Barnett","year":"2019","journal-title":"JAMA Netw Open"},{"key":"2026092414045577200_ocag132-B32","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0134269","article-title":"Collective intelligence meets medical decision-making","volume":"10","author":"Wolf","year":"2015","journal-title":"PLoS One"},{"key":"2026092414045577200_ocag132-B33","doi-asserted-by":"crossref","first-page":"2467","DOI":"10.1001\/jama.2021.22396","article-title":"Next-generation artificial intelligence for diagnosis","volume":"326","author":"Adler-Milstein","year":"2021","journal-title":"JAMA"},{"key":"2026092414045577200_ocag132-B34","author":"McIlroy-Young","year":"2020"},{"key":"2026092414045577200_ocag132-B35","author":"McIlroy-Young R, Wang R, Sen S, Kleinberg J, Anderson A. Learning models of individual behavior in chess. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022 Aug 14:1253-1263."},{"key":"2026092414045577200_ocag132-B36","year":"2022"},{"key":"2026092414045577200_ocag132-B37","author":"Turing","year":"2007"},{"key":"2026092414045577200_ocag132-B38","author":"Rios","year":"2019"},{"key":"2026092414045577200_ocag132-B39","author":"Alsentzer E, Murphy J, Boag W, Weng WH, Jindi D, Naumann T, McDermott M. Publicly available clinical BERT embeddings. In Proceedings of the 2nd clinical natural language processing workshop; 2019 Jun;72-78."},{"key":"2026092414045577200_ocag132-B40","doi-asserted-by":"crossref","first-page":"128","DOI":"10.1097\/EDE.0b013e3181c30fb2","article-title":"Assessing the performance of prediction models","volume":"21","author":"Steyerberg","year":"2010","journal-title":"Epidemiology"},{"key":"2026092414045577200_ocag132-B41","author":"Page","year":"1999"},{"key":"2026092414045577200_ocag132-B42","doi-asserted-by":"crossref","first-page":"604","DOI":"10.1145\/324133.324140","article-title":"Authoritative sources in a hyperlinked environment","volume":"46","author":"Kleinberg","year":"1999","journal-title":"J ACM"},{"key":"2026092414045577200_ocag132-B43","article-title":"Performance on patient experience measures","author":"Chen","year":"2023","journal-title":"JAMA Intern Med"},{"key":"2026092414045577200_ocag132-B44","author":"Caldwell","year":"2022"},{"key":"2026092414045577200_ocag132-B45","first-page":"259","article-title":"Equivalence testing for psychological research","volume":"1","author":"Lakens","year":"2018","journal-title":"Adv Methods Pract Psychol Sci"},{"key":"2026092414045577200_ocag132-B46","doi-asserted-by":"crossref","first-page":"e30104","DOI":"10.2196\/30104","article-title":"A data-driven algorithm to recommend initial clinical workup for outpatient specialty referral: algorithm development and validation using electronic health record data and expert surveys","volume":"10","author":"Ip","year":"2022","journal-title":"JMIR Med Inform"},{"key":"2026092414045577200_ocag132-B47","year":"2006"},{"key":"2026092414045577200_ocag132-B48","first-page":"201","article-title":"A focus theory of normative conduct","volume":"24","author":"Cialdini","year":"1990","journal-title":"J Pers Soc Psychol"},{"key":"2026092414045577200_ocag132-B49","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1038\/s41746-020-0258-y","article-title":"Protected health information filter (Philter)","volume":"3","author":"Norgeot","year":"2020","journal-title":"NPJ Digit Med"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocag132\/71383432\/ocag132.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocag132\/71383432\/ocag132.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T18:05:09Z","timestamp":1790273109000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article\/doi\/10.1093\/jamia\/ocag132\/8833982"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,24]]},"references-count":49,"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocag132","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,24]]},"article-number":"ocag132"}}