{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,24]],"date-time":"2026-10-24T01:43:17Z","timestamp":1792806197098,"version":"4.5.0"},"reference-count":23,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,3,30]],"date-time":"2021-03-30T00:00:00Z","timestamp":1617062400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,3,30]],"date-time":"2021-03-30T00:00:00Z","timestamp":1617062400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Machine learning can help clinicians to make individualized patient predictions only if researchers demonstrate models that contribute novel insights, rather than learning the most likely next step in a set of actions a clinician will take. We trained deep learning models using only clinician-initiated, administrative data for 42.9 million admissions using three subsets of data: demographic data only, demographic data and information available at admission, and the previous data plus charges recorded during the first day of admission. Models trained on charges during the first day of admission achieve performance close to published full EMR-based benchmarks for inpatient outcomes: inhospital mortality (0.89 AUC), prolonged length of stay (0.82 AUC), and 30-day readmission rate (0.71 AUC). Similar performance between models trained with only clinician-initiated data and those trained with full EMR data purporting to include information about patient state and physiology should raise concern in the deployment of these models. Furthermore, these models exhibited significant declines in performance when evaluated over only myocardial infarction (MI) patients relative to models trained over MI patients alone, highlighting the importance of physician diagnosis in the prognostic performance of these models. These results provide a benchmark for predictive accuracy trained only on prior clinical actions and indicate that models with similar performance may derive their signal by looking over clinician\u2019s shoulders\u2014using clinical behavior as the expression of preexisting intuition and suspicion to generate a prediction. For models to guide clinicians in individual decisions, performance exceeding these benchmarks is necessary.<\/jats:p>","DOI":"10.1038\/s41746-021-00426-3","type":"journal-article","created":{"date-parts":[[2021,3,30]],"date-time":"2021-03-30T10:03:41Z","timestamp":1617098621000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":148,"title":["Machine learning for patient risk stratification: standing on, or looking over, the shoulders of clinicians?"],"prefix":"10.1038","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6700-1468","authenticated-orcid":false,"given":"Brett K.","family":"Beaulieu-Jones","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"William","family":"Yuan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3928-5931","authenticated-orcid":false,"given":"Gabriel A.","family":"Brat","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6657-2787","authenticated-orcid":false,"given":"Andrew L.","family":"Beam","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Griffin","family":"Weber","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marshall","family":"Ruffin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2192-5160","authenticated-orcid":false,"given":"Isaac S.","family":"Kohane","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,3,30]]},"reference":[{"key":"426_CR1","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1038\/s41591-018-0300-7","volume":"25","author":"EJ Topol","year":"2019","unstructured":"Topol, E. J. High-performance medicine: the convergence of human and artificial intelligence. Nat. Med. 25, 44\u201356 (2019).","journal-title":"Nat. Med."},{"key":"426_CR2","doi-asserted-by":"publisher","first-page":"2368","DOI":"10.1001\/jama.2016.17217","volume":"316","author":"AL Beam","year":"2016","unstructured":"Beam, A. L. & Kohane, I. S. Translating artificial intelligence into clinical care. JAMA 316, 2368\u20132369 (2016).","journal-title":"JAMA"},{"key":"426_CR3","doi-asserted-by":"publisher","first-page":"20170387","DOI":"10.1098\/rsif.2017.0387","volume":"15","author":"T Ching","year":"2018","unstructured":"Ching, T. et al. Opportunities and obstacles for deep learning in biology and medicine. J. R. Soc. Interface 15, 20170387 (2018).","journal-title":"J. R. Soc. Interface"},{"key":"426_CR4","first-page":"33","volume":"33","author":"JC Weiss","year":"2012","unstructured":"Weiss, J. C., Natarajan, S., Peissig, P. L., McCarty, C. A. & Page, D. Machine learning for personalized medicine: predicting primary myocardial infarction from electronic health records. AI Mag. 33, 33 (2012).","journal-title":"AI Mag."},{"key":"426_CR5","doi-asserted-by":"publisher","first-page":"k1479","DOI":"10.1136\/bmj.k1479","volume":"361","author":"D Agniel","year":"2018","unstructured":"Agniel, D., Kohane, I. S. & Weber, G. M. Biases in electronic health record data due to processes within the healthcare system: retrospective observational study. BMJ 361, k1479 (2018).","journal-title":"BMJ"},{"key":"426_CR6","first-page":"e362","volume":"127","author":"PT O'Gara","year":"2013","unstructured":"O'Gara, P. T., Kushner, F. G. & Ascheim, D. D. ACCF\/AHA guideline for the management of ST-elevation myocardial infarction: a report of the American College of Cardiology Foundation\/American Heart Association Task Force on Practice Guidelines. Circulation 127, e362\u2013e425 (2013).","journal-title":"Circulation"},{"key":"426_CR7","doi-asserted-by":"publisher","first-page":"1347","DOI":"10.1056\/NEJMra1814259","volume":"380","author":"A Rajkomar","year":"2019","unstructured":"Rajkomar, A., Dean, J. & Kohane, I. Machine learning in medicine. N. Engl. J. Med. 380, 1347\u20131358 (2019).","journal-title":"N. Engl. J. Med."},{"key":"426_CR8","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-018-0029-1","volume":"1","author":"A Rajkomar","year":"2018","unstructured":"Rajkomar, A. et al. Scalable and accurate deep learning with electronic health records. npj Digit. Med. 1, 18 (2018).","journal-title":"npj Digit. Med."},{"key":"426_CR9","doi-asserted-by":"publisher","first-page":"1187","DOI":"10.1377\/hlthaff.2014.0038","volume":"33","author":"PJ Wallace","year":"2014","unstructured":"Wallace, P. J., Shah, N. D., Dennen, T., Bleicher, P. A. & Crown, W. H. Optum Labs: building a novel node in the learning health care system. Health Aff. 33, 1187\u20131194 (2014).","journal-title":"Health Aff."},{"key":"426_CR10","first-page":"1","volume":"2010","author":"T Botsis","year":"2010","unstructured":"Botsis, T., Hartvigsen, G., Chen, F. & Weng, C. Secondary use of EHR: data quality issues and informatics opportunities. Summit Transl. Bioinform. 2010, 1\u20135 (2010).","journal-title":"Summit Transl. Bioinform."},{"key":"426_CR11","doi-asserted-by":"publisher","first-page":"144","DOI":"10.1136\/amiajnl-2011-000681","volume":"20","author":"NG Weiskopf","year":"2013","unstructured":"Weiskopf, N. G. & Weng, C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J. Am. Med. Inform. Assoc. 20, 144\u2013151 (2013).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"426_CR12","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1093\/jamia\/ocw054","volume":"24","author":"S van der Bij","year":"2017","unstructured":"van der Bij, S., Khan, N., Ten Veen, P., de Bakker, D. H. & Verheij, R. A. Improving the quality of EHR recording in primary care: a data quality feedback tool. J. Am. Med. Inform. Assoc. 24, 81\u201387 (2017).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"426_CR13","doi-asserted-by":"publisher","DOI":"10.2196\/medinform.8960","volume":"6","author":"BK Beaulieu-Jones","year":"2018","unstructured":"Beaulieu-Jones, B. K. et al. Characterizing and managing missing structured data in electronic health records: data analysis. JMIR Med Inform. 6, e11 (2018).","journal-title":"JMIR Med Inform."},{"key":"426_CR14","doi-asserted-by":"publisher","first-page":"1589","DOI":"10.1109\/JBHI.2017.2767063","volume":"22","author":"B Shickel","year":"2018","unstructured":"Shickel, B., Tighe, P. J., Bihorac, A. & Rashidi, P. Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE J. Biomed. Health Inform. 22, 1589\u20131604 (2018).","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"426_CR15","unstructured":"Premier Applied Sciences. Premier Healthcare Database. https:\/\/products.premierinc.com\/downloads\/PremierHealthcareDatabaseWhitepaper.pdf (2019)."},{"key":"426_CR16","first-page":"8","volume":"24","author":"BK Beaulieu-Jones","year":"2019","unstructured":"Beaulieu-Jones, B. K., Kohane, I. S. & Beam, A. L. Learning contextual hierarchical structure of medical concepts with poincaire embeddings to clarify phenotypes. Pac. Symp. Biocomput. 24, 8\u201317 (2019).","journal-title":"Pac. Symp. Biocomput."},{"key":"426_CR17","unstructured":"Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations (Bengio, Y. & LeCun, Y. eds) San Diego, CA, USA, http:\/\/arxiv.org\/abs\/1412.6980 (2015)."},{"key":"426_CR18","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1145\/2934664","volume":"59","author":"M Zaharia","year":"2016","unstructured":"Zaharia, M. et al. Apache Spark: a unified engine for big data processing. Commun. ACM 59, 56\u201365 (2016).","journal-title":"Commun. ACM"},{"key":"426_CR19","first-page":"1235","volume":"17","author":"X Meng","year":"2016","unstructured":"Meng, X. et al. MLlib: machine learning in apache spark. J. Mach. Learn. Res. 17, 1235\u20131241 (2016).","journal-title":"J. Mach. Learn. Res."},{"key":"426_CR20","unstructured":"Abadi, M. et al. Tensorflow: A System for Large-scale Machine Learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI \u201816) (Keeton, K. and Roscoe, T. eds) USENIX Association, Savannah, GA, USA, 265\u2013283, https:\/\/www.usenix.org\/conference\/osdi16\/technical-sessions\/presentation\/abadi (2016)."},{"key":"426_CR21","unstructured":"Gruener, R. Introducing Petastorm: Uber ATG\u2019s Data Access Library for Deep Learning. Uber Engineering Blog https:\/\/eng.uber.com\/petastorm\/ (2018)."},{"key":"426_CR22","doi-asserted-by":"publisher","unstructured":"Beaulieu-Jones, B. & Yuan, W. Source Code For\u2014machine Learning For Patient Risk Stratification: Standing On, Or Looking Over, The Shoulders Of Clinicians? https:\/\/doi.org\/10.6084\/m9.figshare.12628595.v1 (2020).","DOI":"10.6084\/m9.figshare.12628595.v1"},{"key":"426_CR23","unstructured":"Rajkomar, A. et al. Scalable and accurate deep learning for electronic health records. CoRR abs\/1801.07860, http:\/\/arxiv.org\/abs\/1801.07860 (2018)."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00426-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00426-3","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00426-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,3]],"date-time":"2022-12-03T18:48:29Z","timestamp":1670093309000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00426-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,30]]},"references-count":23,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["426"],"URL":"https:\/\/doi.org\/10.1038\/s41746-021-00426-3","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,30]]},"assertion":[{"value":"22 May 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 February 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 March 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"B.B.J. and M.R. are co-founders of Salutary Inc., and W.Y. has received consulting fees from Salutary Inc. A.L.B. received consulting fees and stock options from Generate Biomedicines. The remaining authors declare that there are no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"62"}}