{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T07:43:24Z","timestamp":1771573404556,"version":"3.50.1"},"reference-count":45,"publisher":"Oxford University Press (OUP)","issue":"7","license":[{"start":{"date-parts":[[2020,6,17]],"date-time":"2020-06-17T00:00:00Z","timestamp":1592352000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"Stanford Graduate Fellowship"},{"name":"Stanford Major Grant"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,7,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Machine learning (ML) diagnostic tools have significant potential to improve health care. However, methodological pitfalls may affect diagnostic test accuracy studies used to appraise such tools. We aimed to evaluate the prevalence and reporting of design characteristics within the literature. Further, we sought to empirically assess whether design features may be associated with different estimates of diagnostic accuracy.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We systematically retrieved 2 \u00d7 2 tables (n\u2009=\u2009281) describing the performance of ML diagnostic tools, derived from 114 publications in 38 meta-analyses, from PubMed. Data extracted included test performance, sample sizes, and design features. A mixed-effects metaregression was run to quantify the association between design features and diagnostic accuracy.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Participant ethnicity and blinding in test interpretation was unreported in 90% and 60% of studies, respectively. Reporting was occasionally lacking for rudimentary characteristics such as study design (28% unreported). Internal validation without appropriate safeguards was used in 44% of studies. Several design features were associated with larger estimates of accuracy, including having unreported (relative diagnostic odds ratio [RDOR], 2.11; 95% confidence interval [CI], 1.43-3.1) or case-control study designs (RDOR, 1.27; 95% CI, 0.97-1.66), and recruiting participants for the index test (RDOR, 1.67; 95% CI, 1.08-2.59).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Significant underreporting of experimental details was present. Study design features may affect estimates of diagnostic performance in the ML diagnostic test accuracy literature.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>The present study identifies pitfalls that threaten the validity, generalizability, and clinical value of ML diagnostic tools and provides recommendations for improvement.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocaa075","type":"journal-article","created":{"date-parts":[[2020,4,25]],"date-time":"2020-04-25T03:39:43Z","timestamp":1587785983000},"page":"1092-1101","source":"Crossref","is-referenced-by-count":25,"title":["Empirical assessment of bias in machine learning diagnostic test accuracy studies"],"prefix":"10.1093","volume":"27","author":[{"given":"Ryan J","family":"Crowley","sequence":"first","affiliation":[{"name":"Meta-Research Innovation Center at Stanford, Stanford University, Stanford, California, USA"},{"name":"Department of Bioengineering, Stanford School of Engineering, Stanford University, Stanford, California, USA"}]},{"given":"Yuan Jin","family":"Tan","sequence":"additional","affiliation":[{"name":"Meta-Research Innovation Center at Stanford, Stanford University, Stanford, California, USA"},{"name":"Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, California, USA"}]},{"given":"John P A","family":"Ioannidis","sequence":"additional","affiliation":[{"name":"Meta-Research Innovation Center at Stanford, Stanford University, Stanford, California, USA"},{"name":"Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, California, USA"},{"name":"Stanford Prevention Research Center, Department of Medicine, Stanford Medicine, Stanford University, Stanford, California, USA"},{"name":"Department of Biomedical Data Science, Stanford Medicine, Stanford University, Stanford, California, USA"},{"name":"Department of Statistics, School of Humanities and Science, Stanford University, Stanford, California, USA"}]}],"member":"286","published-online":{"date-parts":[[2020,6,17]]},"reference":[{"issue":"7639","key":"2020110613112540400_ocaa075-B1","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1038\/nature21056","article-title":"Dermatologist-level classification of skin cancer with deep neural networks","volume":"542","author":"Esteva","year":"2017","journal-title":"Nature"},{"issue":"1","key":"2020110613112540400_ocaa075-B2","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1038\/s41591-018-0268-3","article-title":"Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network","volume":"25","author":"Hannun","year":"2019","journal-title":"Nat Med"},{"issue":"1","key":"2020110613112540400_ocaa075-B3","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1038\/s41591-018-0300-7","article-title":"High-performance medicine: the convergence of human and artificial intelligence","volume":"25","author":"Topol","year":"2019","journal-title":"Nat Med"},{"issue":"6","key":"2020110613112540400_ocaa075-B4","doi-asserted-by":"crossref","first-page":"1224","DOI":"10.1038\/ajg.2010.173","article-title":"Machine learning in medicine: a primer for physicians","volume":"105","author":"Waljee","year":"2010","journal-title":"Am J Gastroenterol"},{"issue":"20","key":"2020110613112540400_ocaa075-B5","doi-asserted-by":"crossref","first-page":"1920","DOI":"10.1161\/CIRCULATIONAHA.115.001593","article-title":"Machine learning in medicine","volume":"132","author":"Deo","year":"2015","journal-title":"Circulation"},{"issue":"10","key":"2020110613112540400_ocaa075-B6","doi-asserted-by":"crossref","first-page":"719","DOI":"10.1038\/s41551-018-0305-z","article-title":"Artificial intelligence in healthcare","volume":"2","author":"Yu","year":"2018","journal-title":"Nat Biomed Eng"},{"issue":"1","key":"2020110613112540400_ocaa075-B7","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/S0933-3657(01)00077-X","article-title":"Machine learning for medical diagnosis: history, state of the art and perspective","volume":"23","author":"Kononenko","year":"2001","journal-title":"Artif Intell Med"},{"issue":"8","key":"2020110613112540400_ocaa075-B8","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1001\/jama.1995.03530080061042","article-title":"Use of methodological standards in diagnostic test research: getting better but still not good","volume":"274","author":"Reid","year":"1995","journal-title":"JAMA"},{"issue":"8","key":"2020110613112540400_ocaa075-B9","first-page":"1650","article-title":"Compliance with methodological standards when evaluating ophthalmic diagnostic tests","volume":"40","author":"Harper","year":"1999","journal-title":"Invest Ophthalmol Vis Sci"},{"key":"2020110613112540400_ocaa075-B10","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1186\/1472-6874-11-7","article-title":"Methodological quality of test accuracy studies included in systematic reviews in obstetrics and gynaecology: sources of bias","volume":"11","author":"Morris","year":"2011","journal-title":"BMC Womens Health"},{"issue":"4","key":"2020110613112540400_ocaa075-B11","doi-asserted-by":"crossref","first-page":"469","DOI":"10.1503\/cmaj.050090","article-title":"Evidence of bias and variation in diagnostic accuracy studies","volume":"174","author":"Rutjes","year":"2006","journal-title":"CMAJ"},{"issue":"10","key":"2020110613112540400_ocaa075-B12","doi-asserted-by":"crossref","first-page":"1093","DOI":"10.1016\/j.jclinepi.2013.05.014","article-title":"A systematic review classifies sources of bias and variation in diagnostic test accuracy studies","volume":"66","author":"Whiting","year":"2013","journal-title":"J Clin Epidemiol"},{"issue":"11","key":"2020110613112540400_ocaa075-B13","doi-asserted-by":"crossref","first-page":"1061","DOI":"10.1001\/jama.282.11.1061","article-title":"Empirical evidence of design-related bias in studies of diagnostic tests","volume":"282","author":"Lijmer","year":"1999","journal-title":"JAMA"},{"issue":"3","key":"2020110613112540400_ocaa075-B14","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1046\/j.1525-1497.2000.03189.x","article-title":"Reporting and concordance of methodologic criteria between abstracts and articles in diagnostic test studies","volume":"15","author":"Estrada","year":"2000","journal-title":"J Gen Intern Med"},{"issue":"11","key":"2020110613112540400_ocaa075-B15","doi-asserted-by":"crossref","first-page":"e012799","DOI":"10.1136\/bmjopen-2016-012799","article-title":"STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration","volume":"6","author":"Cohen","year":"2016","journal-title":"BMJ Open"},{"issue":"3","key":"2020110613112540400_ocaa075-B16","doi-asserted-by":"crossref","first-page":"326","DOI":"10.1145\/212094.212114","article-title":"Overfitting and undercomputing in machine learning","volume":"27","author":"Dietterich","year":"1995","journal-title":"ACM Comput Surv"},{"issue":"3","key":"2020110613112540400_ocaa075-B17","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1016\/j.jacr.2017.12.026","article-title":"Artificial intelligence and machine learning in radiology: opportunities, challenges, pitfalls, and criteria for success","volume":"15","author":"Thrall","year":"2018","journal-title":"J Am Coll Radiol"},{"key":"2020110613112540400_ocaa075-B18","doi-asserted-by":"crossref","first-page":"177","DOI":"10.3389\/fpsyt.2016.00177","article-title":"Promises, pitfalls, and basic guidelines for applying machine learning classifiers to psychiatric imaging data, with autism as an example","volume":"7","author":"Kassraian-Fard","year":"2016","journal-title":"Front Psychiatry"},{"issue":"9","key":"2020110613112540400_ocaa075-B19","doi-asserted-by":"crossref","first-page":"1314","DOI":"10.1002\/mds.26693","article-title":"Machine learning for large-scale wearable sensor data in Parkinson\u2019s disease: concepts, promises, pitfalls, and futures","volume":"31","author":"Kubota","year":"2016","journal-title":"Mov Disord"},{"issue":"5","key":"2020110613112540400_ocaa075-B20","doi-asserted-by":"crossref","first-page":"1121","DOI":"10.1007\/s10803-014-2268-6","article-title":"Applying machine learning to facilitate autism diagnostics: pitfalls and promises","volume":"45","author":"Bone","year":"2015","journal-title":"J Autism Dev Disord"},{"key":"2020110613112540400_ocaa075-B21","first-page":"2079","article-title":"On over-fitting in model selection and subsequent selection bias in performance evaluation","volume":"11","author":"Cawley","year":"2010","journal-title":"J Mach Learn Res"},{"issue":"11","key":"2020110613112540400_ocaa075-B22","doi-asserted-by":"crossref","first-page":"1544","DOI":"10.1001\/jamainternmed.2018.3763","article-title":"Potential biases in machine learning algorithms using electronic health record data","volume":"178","author":"Gianfrancesco","year":"2018","journal-title":"JAMA Intern Med"},{"key":"2020110613112540400_ocaa075-B23","author":"McDermott"},{"issue":"6","key":"2020110613112540400_ocaa075-B24","doi-asserted-by":"crossref","first-page":"e271","DOI":"10.1016\/S2589-7500(19)30123-2","article-title":"A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis","volume":"1","author":"Liu","year":"2019","journal-title":"Lancet Digit Health"},{"key":"2020110613112540400_ocaa075-B25","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.jclinepi.2019.02.004","article-title":"A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models","volume":"110","author":"Christodoulou","year":"2019","journal-title":"J Clin Epidemiol"},{"issue":"3","key":"2020110613112540400_ocaa075-B26","doi-asserted-by":"crossref","first-page":"405","DOI":"10.3348\/kjr.2019.0025","article-title":"Design characteristics of studies reporting the performance of artificial intelligence algorithms for diagnostic analysis of medical images: results from recently published papers","volume":"20","author":"Kim","year":"2019","journal-title":"Korean J Radiol"},{"issue":"14","key":"2020110613112540400_ocaa075-B27","doi-asserted-by":"crossref","first-page":"1293","DOI":"10.1002\/sim.4780121403","article-title":"Combining independent studies of a diagnostic test into a summary roc curve: Data-analytic approaches and some additional considerations","volume":"12","author":"Moses","year":"1993","journal-title":"Stat Med"},{"issue":"12","key":"2020110613112540400_ocaa075-B28","doi-asserted-by":"crossref","first-page":"i101","DOI":"10.1093\/bioinformatics\/btw282","article-title":"Comparative analyses of population-scale phenomic data in electronic medical records reveal race-specific disease networks","volume":"32","author":"Glicksberg","year":"2016","journal-title":"Bioinformatics"},{"issue":"25","key":"2020110613112540400_ocaa075-B29","doi-asserted-by":"crossref","first-page":"3137","DOI":"10.1161\/01.CIR.102.25.3137","article-title":"Trends and disparities in coronary heart disease, stroke, and other cardiovascular diseases in the United States: findings of the national conference on cardiovascular disease prevention","volume":"102","author":"Cooper","year":"2000","journal-title":"Circulation"},{"key":"2020110613112540400_ocaa075-B30","author":"Merler"},{"issue":"8","key":"2020110613112540400_ocaa075-B31","doi-asserted-by":"crossref","first-page":"660","DOI":"10.1111\/j.1471-0528.1995.tb11407.x","article-title":"Maternal serum screening for Down\u2019s syndrome in the first trimester of pregnancy","volume":"102","author":"Biagiotti","year":"1995","journal-title":"Br J Obstet Gynaecol"},{"issue":"4","key":"2020110613112540400_ocaa075-B32","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1016\/S0009-9120(97)00004-0","article-title":"Screening for Down syndrome during first trimester: a prospective study using free beta-human chorionic gonadotropin and pregnancy-associated plasma protein A","volume":"30","author":"Forest","year":"1997","journal-title":"Clin Biochem"},{"issue":"3","key":"2020110613112540400_ocaa075-B33","doi-asserted-by":"crossref","first-page":"680","DOI":"10.1002\/jmri.22095","article-title":"Machine learning study of several classifiers trained with texture analysis features to differentiate benign from malignant soft-tissue tumors in T1-MRI images","volume":"31","author":"Juntu","year":"2010","journal-title":"J Magn Reson Imaging"},{"issue":"9","key":"2020110613112540400_ocaa075-B34","doi-asserted-by":"crossref","first-page":"1316","DOI":"10.1016\/j.mri.2008.02.013","article-title":"Are signal intensity and homogeneity useful parameters for distinguishing between benign and malignant soft tissue masses on MR images? Objective evaluation by means of texture analysis","volume":"26","author":"Mayerhoefer","year":"2008","journal-title":"Magn Reson Imaging"},{"issue":"2","key":"2020110613112540400_ocaa075-B35","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1016\/S0304-3835(01)00508-0","article-title":"Feature extraction and classification of breast cancer on dynamic magnetic resonance imaging using artificial neural network","volume":"171","author":"Abdolmaleki","year":"2001","journal-title":"Cancer Lett"},{"issue":"5","key":"2020110613112540400_ocaa075-B36","first-page":"283","article-title":"Neural network analysis of breast cancer from MRI findings","volume":"15","author":"Abdolmaleki","year":"1997","journal-title":"Radiat Med"},{"issue":"S1","key":"2020110613112540400_ocaa075-B37","doi-asserted-by":"crossref","first-page":"58","DOI":"10.1007\/BF03190297","article-title":"A use of a neural network to evaluate contrast enhancement curves in breast magnetic resonance images","volume":"14","author":"Vergnaghi","year":"2001","journal-title":"J Digit Imaging"},{"issue":"8","key":"2020110613112540400_ocaa075-B38","doi-asserted-by":"crossref","first-page":"3940","DOI":"10.1118\/1.3446799","article-title":"Multilevel analysis of spatiotemporal association features for differentiation of tumor enhancement patterns in breast DCE-MRI","volume":"37","author":"Lee","year":"2010","journal-title":"Med Phys"},{"key":"2020110613112540400_ocaa075-B39","first-page":"CD011975","article-title":"First trimester serum tests for Down\u2019s syndrome screening","volume":"11","author":"Alldred","year":"2015","journal-title":"Cochrane Database Syst Rev"},{"issue":"3","key":"2020110613112540400_ocaa075-B40","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1002\/uog.6318","article-title":"Fetal nasal bone in screening for trisomies 21, 18 and 13 and Turner syndrome at 11-13 weeks of gestation","volume":"33","author":"Kagan","year":"2009","journal-title":"Ultrasound Obstet Gynecol"},{"issue":"4","key":"2020110613112540400_ocaa075-B41","doi-asserted-by":"crossref","first-page":"449","DOI":"10.1007\/s40846-016-0163-7","article-title":"Pattern recognition approaches for breast cancer DCE-MRI classification: a systematic review","volume":"36","author":"Fusco","year":"2016","journal-title":"J Med Biol Eng"},{"key":"2020110613112540400_ocaa075-B42","first-page":"65","author":"Lee","year":"2009"},{"issue":"6","key":"2020110613112540400_ocaa075-B43","doi-asserted-by":"crossref","first-page":"1016","DOI":"10.1002\/jmri.1880070613","article-title":"Multifeature analysis of Gd-enhanced MR images of breast lesions","volume":"7","author":"Sinha","year":"1997","journal-title":"J Magn Reson Imaging"},{"issue":"4","key":"2020110613112540400_ocaa075-B44","doi-asserted-by":"crossref","first-page":"570","DOI":"10.1373\/49.4.570","article-title":"The D-dimer test for deep venous thrombosis: gold standards and bias in negative predictive value","volume":"49","author":"Philbrick","year":"2003","journal-title":"Clin Chem"},{"issue":"4","key":"2020110613112540400_ocaa075-B45","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1056\/NEJMoa021659","article-title":"Effect of verification bias on screening for prostate cancer by measurement of prostate-specific antigen","volume":"349","author":"Punglia","year":"2003","journal-title":"N Engl J Med"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/7\/1092\/34153804\/ocaa075.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/7\/1092\/34153804\/ocaa075.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T19:36:45Z","timestamp":1604691405000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/27\/7\/1092\/5858298"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,17]]},"references-count":45,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2020,6,17]]},"published-print":{"date-parts":[[2020,7,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaa075","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,7]]},"published":{"date-parts":[[2020,6,17]]}}}