{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,6]],"date-time":"2025-12-06T17:13:00Z","timestamp":1765041180859},"reference-count":58,"publisher":"Georg Thieme Verlag KG","issue":"02","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Appl Clin Inform"],"published-print":{"date-parts":[[2021,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>\n          Background\u2003Machine learning (ML) has captured the attention of many clinicians who may not have formal training in this area but are otherwise increasingly exposed to ML literature that may be relevant to their clinical specialties. ML papers that follow an outcomes-based research format can be assessed using clinical research appraisal frameworks such as PICO (Population, Intervention, Comparison, Outcome). However, the PICO frameworks strain when applied to ML papers that create new ML models, which are akin to diagnostic tests. There is a need for a new framework to help assess such papers.<\/jats:p><jats:p>\n          Objective\u2003We propose a new framework to help clinicians systematically read and evaluate medical ML papers whose aim is to create a new ML model: ML-PICO (Machine Learning, Population, Identification, Crosscheck, Outcomes). We describe how the ML-PICO framework can be applied toward appraising literature describing ML models for health care.<\/jats:p><jats:p>\n          Conclusion\u2003The relevance of ML to practitioners of clinical medicine is steadily increasing with a growing body of literature. Therefore, it is increasingly important for clinicians to be familiar with how to assess and best utilize these tools. In this paper we have described a practical framework on how to read ML papers that create a new ML model (or diagnostic test): ML-PICO. We hope that this can be used by clinicians to better evaluate the quality and utility of ML papers.<\/jats:p>","DOI":"10.1055\/s-0041-1729752","type":"journal-article","created":{"date-parts":[[2021,5,19]],"date-time":"2021-05-19T23:00:32Z","timestamp":1621465232000},"page":"407-416","source":"Crossref","is-referenced-by-count":11,"title":["Rethinking PICO in the Machine Learning Era: ML-PICO"],"prefix":"10.1055","volume":"12","author":[{"given":"Xinran","family":"Liu","sequence":"additional","affiliation":[{"name":"Division of Hospital Medicine, University of California, San Francisco, San Francisco, California, United States"},{"name":"University of California, San Francisco, San Francisco, California, United States"}]},{"given":"James","family":"Anstey","sequence":"additional","affiliation":[{"name":"Division of Hospital Medicine, University of California, San Francisco, San Francisco, California, United States"}]},{"given":"Ron","family":"Li","sequence":"additional","affiliation":[{"name":"Division of Hospital Medicine, Stanford University, Stanford, California, United States"}]},{"given":"Chethan","family":"Sarabu","sequence":"additional","affiliation":[{"name":"doc.ai, Palo Alto, California, United States"},{"name":"Department of Pediatrics, Stanford University, Stanford, California, United States"}]},{"given":"Reiri","family":"Sono","sequence":"additional","affiliation":[{"name":"University of California, San Francisco, San Francisco, California, United States"}]},{"given":"Atul J.","family":"Butte","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute, University of California, San Francisco, San Francisco, California, United States"}]}],"member":"194","published-online":{"date-parts":[[2021,5,19]]},"reference":[{"issue":"01","key":"ref1","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1038\/s41591-018-0300-7","article-title":"High-performance medicine: the convergence of human and artificial intelligence","volume":"25","author":"E J Topol","year":"2019","journal-title":"Nat Med"},{"issue":"23","key":"ref2","doi-asserted-by":"crossref","first-page":"2283","DOI":"10.1001\/jama.2019.17831","article-title":"Machine learning, predictive analytics, and clinical practice: can the past inform the present?","volume":"322","author":"E D Peterson","year":"2019","journal-title":"JAMA"},{"issue":"04","key":"ref3","doi-asserted-by":"crossref","first-page":"e013924","DOI":"10.1161\/JAHA.119.013924","article-title":"State-of-the-art machine learning techniques aiming to improve patient outcomes pertaining to the cardiovascular system","volume":"9","author":"R K Sevakula","year":"2020","journal-title":"J Am Heart Assoc"},{"key":"ref4","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.media.2017.07.005","article-title":"A survey on deep learning in medical image analysis","volume":"42","author":"G Litjens","year":"2017","journal-title":"Med Image Anal"},{"issue":"01","key":"ref5","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1055\/s-0039-3400270","article-title":"Artificial intelligence in radiology residency training","volume":"24","author":"M C Forney","year":"2020","journal-title":"Semin Musculoskelet Radiol"},{"issue":"04","key":"ref6","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1007\/s10140-020-01795-0","article-title":"Developing a curriculum in artificial intelligence for emergency radiology","volume":"27","author":"E M Weisberg","year":"2020","journal-title":"Emerg Radiol"},{"issue":"14","key":"ref7","doi-asserted-by":"crossref","first-page":"1347","DOI":"10.1056\/NEJMra1814259","article-title":"Machine learning in medicine","volume":"380","author":"A Rajkomar","year":"2019","journal-title":"N Engl J Med"},{"issue":"18","key":"ref8","doi-asserted-by":"crossref","first-page":"1806","DOI":"10.1001\/jama.2019.16489","article-title":"How to read articles that use machine learning: users' guides to the medical literature","volume":"322","author":"Y Liu","year":"2019","journal-title":"JAMA"},{"issue":"01","key":"ref9","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1001\/jama.2018.5602","article-title":"Big data and predictive analytics: recalibrating expectations","volume":"320","author":"N D Shah","year":"2018","journal-title":"JAMA"},{"issue":"01","key":"ref10","doi-asserted-by":"crossref","first-page":"47","DOI":"10.4103\/0253-7184.69003","article-title":"Formulating a researchable question: a critical step for facilitating good clinical research","volume":"31","author":"S Aslam","year":"2010","journal-title":"Indian J Sex Transm Dis AIDS"},{"issue":"01","key":"ref11","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1016\/j.cmi.2018.06.011","article-title":"How to: evaluate a diagnostic test","volume":"25","author":"M MG Leeflang","year":"2019","journal-title":"Clin Microbiol Infect"},{"issue":"06","key":"ref12","first-page":"587","article-title":"A framework for clinical evaluation of diagnostic technologies","volume":"134","author":"G H Guyatt","year":"1986","journal-title":"CMAJ"},{"issue":"02","key":"ref13","doi-asserted-by":"crossref","first-page":"199","DOI":"10.11613\/BM.2014.022","article-title":"Observational and interventional study design types; an overview","volume":"24","author":"M S Thiese","year":"2014","journal-title":"Biochem Med (Zagreb)"},{"key":"ref14","doi-asserted-by":"crossref","first-page":"m689","DOI":"10.1136\/bmj.m689","article-title":"Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies","volume":"368","author":"M Nagendran","year":"2020","journal-title":"BMJ"},{"issue":"05","key":"ref15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1371\/journal.pmed.1001221","article-title":"Reporting and methods in clinical prediction research: a systematic review","volume":"9","author":"W Bouwmeester","year":"2012","journal-title":"PLoS Med"},{"key":"ref16","doi-asserted-by":"crossref","first-page":"i3140","DOI":"10.1136\/bmj.i3140","article-title":"External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges","volume":"353","author":"R D Riley","year":"2016","journal-title":"BMJ"},{"key":"ref17","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1016\/j.jclinepi.2016.03.002","article-title":"Modern modeling techniques had limited external validity in predicting mortality from traumatic brain injury","volume":"78","author":"T van der Ploeg","year":"2016","journal-title":"J Clin Epidemiol"},{"issue":"12","key":"ref18","doi-asserted-by":"crossref","first-page":"1651","DOI":"10.1093\/jamia\/ocz130","article-title":"Predictive analytics in health care: how can we know it works?","volume":"26","author":"B Van Calster","year":"2019","journal-title":"J Am Med Inform Assoc"},{"issue":"09","key":"ref19","doi-asserted-by":"crossref","first-page":"4754","DOI":"10.1007\/s00330-019-06244-2","article-title":"Machine learning classifiers can predict Gleason pattern 4 prostate cancer with greater accuracy than experienced radiologists","volume":"29","author":"M Antonelli","year":"2019","journal-title":"Eur Radiol"},{"key":"ref20","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1038\/s41746-020-00318-y","article-title":"Developing a delivery science for artificial intelligence in healthcare","volume":"3","author":"R C Li","year":"2020","journal-title":"NPJ Digit Med"},{"issue":"16","key":"ref21","doi-asserted-by":"crossref","first-page":"1529","DOI":"10.1056\/NEJMoa1804988","article-title":"Effects of aspirin for primary prevention in persons with diabetes mellitus","volume":"379","author":"L Bowman","year":"2018","journal-title":"N Engl J Med"},{"key":"ref22","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1016\/j.compbiomed.2016.05.003","article-title":"A computational approach to early sepsis detection","volume":"74","author":"J S Calvert","year":"2016","journal-title":"Comput Biol Med"},{"issue":"03","key":"ref23","doi-asserted-by":"crossref","first-page":"e28","DOI":"10.2196\/medinform.5909","article-title":"Prediction of sepsis in the intensive care unit with minimal electronic health record data: a machine learning approach","volume":"4","author":"T Desautels","year":"2016","journal-title":"JMIR Med Inform"},{"issue":"06","key":"ref24","doi-asserted-by":"crossref","first-page":"739","DOI":"10.1016\/j.jelectrocard.2017.08.013","article-title":"Early sepsis detection in critical care patients using multiscale blood pressure and heart rate dynamics","volume":"50","author":"S P Shashikumar","year":"2017","journal-title":"J Electrocardiol"},{"key":"ref25","volume-title":"MIT Critical Data","year":"2016"},{"issue":"02","key":"ref26","first-page":"345","article-title":"From development to deployment: dataset shift, causality, and shift-stable models in health AI","volume":"21","author":"A Subbaswamy","year":"2020","journal-title":"Biostatistics"},{"issue":"01","key":"ref27","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1038\/s41591-018-0307-0","article-title":"The practical implementation of artificial intelligence technologies in medicine","volume":"25","author":"J He","year":"2019","journal-title":"Nat Med"},{"key":"ref28","first-page":"191","article-title":"A review of challenges and opportunities in machine learning for health","volume":"2020","author":"M Ghassemi","year":"2020","journal-title":"AMIA Jt Summits Transl Sci Proc"},{"key":"ref29","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1186\/s12874-015-0022-1","article-title":"The rise of multiple imputation: a review of the reporting and implementation of the method in medical research","volume":"15","author":"P Hayati Rezvan","year":"2015","journal-title":"BMC Med Res Methodol"},{"issue":"03","key":"ref30","doi-asserted-by":"crossref","first-page":"491","DOI":"10.1093\/jamia\/ocz192","article-title":"A governance model for the application of AI in health care","volume":"27","author":"S Reddy","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"02","key":"ref31","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1172\/JCI129197","article-title":"Opportunities and challenges in using real-world data for health care","volume":"130","author":"V A Rudrapatna","year":"2020","journal-title":"J Clin Invest"},{"issue":"10","key":"ref32","doi-asserted-by":"crossref","first-page":"1419","DOI":"10.1093\/jamia\/ocy068","article-title":"Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review","volume":"25","author":"C Xiao","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"11","key":"ref33","doi-asserted-by":"crossref","first-page":"1544","DOI":"10.1001\/jamainternmed.2018.3763","article-title":"Potential biases in machine learning algorithms using electronic health record data","volume":"178","author":"M A Gianfrancesco","year":"2018","journal-title":"JAMA Intern Med"},{"issue":"24","key":"ref34","doi-asserted-by":"crossref","first-page":"2285","DOI":"10.1056\/NEJMp1907407","article-title":"Machine learning and the cancer-diagnosis problem\u2014no gold standard","volume":"381","author":"A S Adamson","year":"2019","journal-title":"N Engl J Med"},{"issue":"13","key":"ref35","doi-asserted-by":"crossref","first-page":"1317","DOI":"10.1001\/jama.2017.18391","article-title":"Big data and machine learning in health care","volume":"319","author":"A L Beam","year":"2018","journal-title":"JAMA"},{"issue":"01","key":"ref36","doi-asserted-by":"crossref","first-page":"e017833","DOI":"10.1136\/bmjopen-2017-017833","article-title":"Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU","volume":"8","author":"Q Mao","year":"2018","journal-title":"BMJ Open"},{"issue":"03","key":"ref37","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1111\/acem.12876","article-title":"Prediction of in-hospital mortality in emergency department patients with sepsis: a local big data-driven, machine learning approach","volume":"23","author":"R A Taylor","year":"2016","journal-title":"Acad Emerg Med"},{"issue":"01","key":"ref38","doi-asserted-by":"crossref","first-page":"e000856","DOI":"10.1136\/bmjoq-2019-000856","article-title":"Accuracy of medical billing data against the electronic health record in the measurement of colorectal cancer screening rates","volume":"9","author":"V A Rudrapatna","year":"2020","journal-title":"BMJ Open Qual"},{"issue":"06","key":"ref39","doi-asserted-by":"crossref","first-page":"e39","DOI":"10.1097\/MLR.0b013e318268ac86","article-title":"Identifying patients with severe sepsis using administrative claims: patient-level validation of the angus implementation of the international consensus conference definition of severe sepsis","volume":"52","author":"T J Iwashyna","year":"2014","journal-title":"Med Care"},{"issue":"13","key":"ref40","doi-asserted-by":"crossref","first-page":"1241","DOI":"10.1001\/jama.2017.13836","article-title":"Incidence and trends of sepsis in US Hospitals using clinical vs claims data, 2009-2014","volume":"318","author":"C Rhee","year":"2017","journal-title":"JAMA"},{"issue":"08","key":"ref44","doi-asserted-by":"crossref","first-page":"913","DOI":"10.1111\/ecog.02881","article-title":"Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure","volume":"40","author":"D R Roberts","year":"2017","journal-title":"Ecography"},{"key":"ref45","first-page":"883","article-title":"Cross-validation for correlated data","volume":"97","author":"A Rabinowicz","year":"2020","journal-title":"J Am Stat Assoc"},{"issue":"04","key":"ref46","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2382577.2382579","article-title":"Leakage in data mining: formulation, detection, and avoidance","volume":"6","author":"S Kaufman","year":"2012","journal-title":"ACM Trans Knowl Discov Data"},{"issue":"02","key":"ref47","doi-asserted-by":"crossref","first-page":"678","DOI":"10.1213\/ANE.0000000000002224","article-title":"Defining the primary outcomes and justifying secondary outcomes of a study: usually, the fewer, the better","volume":"125","author":"T R Vetter","year":"2017","journal-title":"Anesth Analg"},{"issue":"12","key":"ref48","doi-asserted-by":"crossref","first-page":"1655","DOI":"10.1093\/jamia\/ocz088","article-title":"The number needed to benefit: estimating the value of predictive analytics in healthcare","volume":"26","author":"V X Liu","year":"2019","journal-title":"J Am Med Inform Assoc"},{"issue":"15","key":"ref49","doi-asserted-by":"crossref","first-page":"E2811","DOI":"10.3390\/molecules24152811","article-title":"Multi-level comparison of machine learning classifiers and their performance metrics","volume":"24","author":"A R\u00e1cz","year":"2019","journal-title":"Molecules"},{"key":"ref50","first-page":"45","article-title":"A new typology design of performance metrics to measure errors in machine learning regression algorithms","volume":"14","author":"A Botchkarev","year":"2019","journal-title":"Interdiscip J Inf Knowl Manag"},{"key":"ref51","volume-title":"Statistics for Machine Learning","author":"P Dangeti","year":"2017"},{"issue":"14","key":"ref52","doi-asserted-by":"crossref","first-page":"1377","DOI":"10.1001\/jama.2017.12126","article-title":"Discrimination and calibration of clinical prediction models: users' guides to the medical literature","volume":"318","author":"A C Alba","year":"2017","journal-title":"JAMA"},{"key":"ref53","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2019.10306","article-title":"Making machine learning models clinically useful","author":"N H Shah","year":"2019","journal-title":"JAMA"},{"issue":"01","key":"ref54","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1186\/s13054-015-0999-1","article-title":"Why the C-statistic is not informative to evaluate early warning scores and what metrics to use","volume":"19","author":"S Romero-Brufau","year":"2015","journal-title":"Crit Care"},{"issue":"08","key":"ref55","doi-asserted-by":"crossref","first-page":"855","DOI":"10.1016\/j.jclinepi.2015.02.010","article-title":"The precision\u2013recall curve overcame the optimism of the receiver operating characteristic curve in rare diseases","volume":"68","author":"B Ozenne","year":"2015","journal-title":"J Clin Epidemiol"},{"issue":"02","key":"ref56","doi-asserted-by":"crossref","first-page":"87","DOI":"10.3322\/caac.21262","article-title":"Global cancer statistics, 2012","volume":"65","author":"L A Torre","year":"2015","journal-title":"CA Cancer J Clin"},{"issue":"06","key":"ref57","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1038\/nrcardio.2016.25","article-title":"Epidemiology and aetiology of heart failure","volume":"13","author":"B Ziaeian","year":"2016","journal-title":"Nat Rev Cardiol"},{"issue":"05","key":"ref58","doi-asserted-by":"crossref","first-page":"1047","DOI":"10.2337\/diacare.27.5.1047","article-title":"Global prevalence of diabetes: estimates for the year 2000 and projections for 2030","volume":"27","author":"S Wild","year":"2004","journal-title":"Diabetes Care"},{"issue":"01","key":"ref59","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1002\/cpt.1826","article-title":"Predicting inpatient medication orders from electronic health record data","volume":"108","author":"K Rough","year":"2020","journal-title":"Clin Pharmacol Ther"},{"key":"ref61","volume-title":"Artificial Intelligence Special Publication","author":"M Matheny","year":"2019"},{"issue":"01","key":"ref62","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1186\/s41512-017-0017-y","article-title":"A novel method for interrogating receiver operating characteristic curves for assessing prognostic tests","volume":"1","author":"G Thomas","year":"2017","journal-title":"Diagn Progn Res"}],"container-title":["Applied Clinical Informatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.thieme-connect.de\/products\/ejournals\/pdf\/10.1055\/s-0041-1729752.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,5,19]],"date-time":"2021-05-19T23:01:04Z","timestamp":1621465264000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.thieme-connect.de\/DOI\/DOI?10.1055\/s-0041-1729752"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3]]},"references-count":58,"journal-issue":{"issue":"02","published-online":{"date-parts":[[2021,3,17]]},"published-print":{"date-parts":[[2021,3]]}},"URL":"https:\/\/doi.org\/10.1055\/s-0041-1729752","relation":{},"ISSN":["1869-0327"],"issn-type":[{"value":"1869-0327","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3]]}}}