{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,23]],"date-time":"2026-04-23T10:18:32Z","timestamp":1776939512141,"version":"3.51.4"},"reference-count":33,"publisher":"Oxford University Press (OUP)","issue":"12","license":[{"start":{"date-parts":[[2019,8,2]],"date-time":"2019-08-02T00:00:00Z","timestamp":1564704000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"funder":[{"DOI":"10.13039\/100005930","name":"Research Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100005930","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>There is increasing awareness that the methodology and findings of research should be transparent. This includes studies using artificial intelligence to develop predictive algorithms that make individualized diagnostic or prognostic risk predictions. We argue that it is paramount to make the algorithm behind any prediction publicly available. This allows independent external validation, assessment of performance heterogeneity across settings and over time, and algorithm refinement or updating. Online calculators and apps may aid uptake if accompanied with sufficient information. For algorithms based on \u201cblack box\u201d machine learning methods, software for algorithm implementation is a must. Hiding algorithms for commercial exploitation is unethical, because there is no possibility to assess whether algorithms work as advertised or to monitor when and how algorithms are updated. Journals and funders should demand maximal transparency for publications on predictive algorithms, and clinical guidelines should only recommend publicly available algorithms.<\/jats:p>","DOI":"10.1093\/jamia\/ocz130","type":"journal-article","created":{"date-parts":[[2019,7,4]],"date-time":"2019-07-04T15:36:58Z","timestamp":1562254618000},"page":"1651-1654","source":"Crossref","is-referenced-by-count":183,"title":["Predictive analytics in health care: how can we know it works?"],"prefix":"10.1093","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1613-7450","authenticated-orcid":false,"given":"Ben","family":"Van Calster","sequence":"first","affiliation":[{"name":"Department of Development and Regeneration, KU Leuven, Leuven, Belgium"},{"name":"Department of Biomedical Data Sciences, Leiden University Medical Center (LUMC), Leiden, The Netherlands"}]},{"given":"Laure","family":"Wynants","sequence":"additional","affiliation":[{"name":"Department of Development and Regeneration, KU Leuven, Leuven, Belgium"}]},{"given":"Dirk","family":"Timmerman","sequence":"additional","affiliation":[{"name":"Department of Development and Regeneration, KU Leuven, Leuven, Belgium"},{"name":"Department of Obstetrics and Gynaecology, University Hospitals Leuven, Leuven, Belgium"}]},{"given":"Ewout W","family":"Steyerberg","sequence":"additional","affiliation":[{"name":"Department of Biomedical Data Sciences, Leiden University Medical Center (LUMC), Leiden, The Netherlands"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2772-2316","authenticated-orcid":false,"given":"Gary S","family":"Collins","sequence":"additional","affiliation":[{"name":"Centre for Statistics in Medicine, Nuffield, Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, UK"},{"name":"Oxford University Hospitals NHS Foundation Trust, Oxford, UK"}]}],"member":"286","published-online":{"date-parts":[[2019,8,2]]},"reference":[{"key":"2020110612460826500_ocz130-B1","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-77244-8","volume-title":"Clinical Prediction Models","author":"Steyerberg","year":"2009"},{"issue":"1","key":"2020110612460826500_ocz130-B2","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1001\/jama.2018.5602","article-title":"Big data and predictive analytics: recalibrating expectations","volume":"320","author":"Shah","year":"2018","journal-title":"JAMA"},{"issue":"13","key":"2020110612460826500_ocz130-B3","doi-asserted-by":"crossref","first-page":"1317","DOI":"10.1001\/jama.2017.18391","article-title":"Big data and machine learning in health care","volume":"319","author":"Beam","year":"2018","journal-title":"JAMA"},{"issue":"1","key":"2020110612460826500_ocz130-B4","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1136\/amiajnl-2012-001145","article-title":"Next-generation phenotyping of electronic health records","volume":"20","author":"Hripcsak","year":"2013","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2020110612460826500_ocz130-B5","doi-asserted-by":"crossref","first-page":"e1002333.","DOI":"10.1371\/journal.pbio.1002333","article-title":"Reproducible research practices and transparency across the biomedical literature","volume":"14","author":"Iqbal","year":"2016","journal-title":"PLoS Biol"},{"issue":"6242","key":"2020110612460826500_ocz130-B6","doi-asserted-by":"crossref","first-page":"1422","DOI":"10.1126\/science.aab2374","article-title":"Promoting an open research culture","volume":"348","author":"Nosek","year":"2015","journal-title":"Science"},{"key":"2020110612460826500_ocz130-B7","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1016\/j.jclinepi.2015.12.005","article-title":"A calibration hierarchy for risk models was defined: from utopia to empirical data","volume":"74","author":"Van Calster","year":"2016","journal-title":"J Clin Epidemiol"},{"issue":"9","key":"2020110612460826500_ocz130-B8","doi-asserted-by":"crossref","first-page":"1304","DOI":"10.1038\/s41591-018-0178-4","article-title":"New machine-learning technologies for computer-aided diagnosis","volume":"24","author":"Lynch","year":"2018","journal-title":"Nat Med"},{"key":"2020110612460826500_ocz130-B9","doi-asserted-by":"crossref","first-page":"i3140.","DOI":"10.1136\/bmj.i3140","article-title":"External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges","volume":"353","author":"Riley","year":"2016","journal-title":"BMJ"},{"key":"2020110612460826500_ocz130-B10","author":"Ghassemi","year":"2018"},{"issue":"6","key":"2020110612460826500_ocz130-B11","doi-asserted-by":"crossref","first-page":"1052","DOI":"10.1093\/jamia\/ocx030","article-title":"Calibration drift in regression and machine learning models for acute kidney injury","volume":"24","author":"Davis","year":"2017","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"2020110612460826500_ocz130-B12","doi-asserted-by":"crossref","first-page":"680","DOI":"10.1038\/bjc.2014.333","article-title":"Strategies to diagnose ovarian cancer: new evidence from phase 3 of the multicenter international IOTA study","volume":"111","author":"Testa","year":"2014","journal-title":"Br J Cancer"},{"issue":"25","key":"2020110612460826500_ocz130-B13","doi-asserted-by":"crossref","first-page":"2935","DOI":"10.1016\/j.jacc.2013.11.005","article-title":"2013 ACC\/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology\/American Heart Association Task Force on practice guidelines","volume":"63","author":"Goff","year":"2014","journal-title":"J Am Coll Cardiol"},{"key":"2020110612460826500_ocz130-B14","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1016\/j.ijmedinf.2017.03.006","article-title":"Decaying relevance of clinical data towards future decisions in data-driven inpatient clinical order sets","volume":"102","author":"Chen","year":"2017","journal-title":"Int J Med Inform"},{"key":"2020110612460826500_ocz130-B15","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1016\/j.jbi.2018.07.019","article-title":"When to re-order laboratory tests? Learning laboratory test shelf-life","volume":"85","author":"Levy-Fix","year":"2018","journal-title":"J Biomed Inform"},{"issue":"1","key":"2020110612460826500_ocz130-B16","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1038\/s41591-018-0307-0","article-title":"The practical implementation of artificial intelligence technologies in medicine","volume":"25","author":"He","year":"2019","journal-title":"Nat Med"},{"key":"2020110612460826500_ocz130-B17","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1016\/j.jclinepi.2016.03.002","article-title":"Modern modeling techniques had limited external validity in predicting mortality from traumatic brain injury","volume":"78","author":"van der Ploeg","year":"2016","journal-title":"J Clin Epidemiol"},{"issue":"11","key":"2020110612460826500_ocz130-B18","doi-asserted-by":"crossref","first-page":"e1002708","DOI":"10.1371\/journal.pmed.1002708","article-title":"Advancing the beneficial use of machine learning in health care and medicine: toward a community understanding","volume":"15","author":"Nevin","year":"2018","journal-title":"PLoS Med"},{"issue":"9","key":"2020110612460826500_ocz130-B19","doi-asserted-by":"crossref","first-page":"976.","DOI":"10.1001\/jamacardio.2016.3826","article-title":"Risk prediction with electronic health records. The importance of model validation and clinical context","volume":"1","author":"Goldstein","year":"2016","journal-title":"JAMA Cardiol"},{"issue":"2","key":"2020110612460826500_ocz130-B20","doi-asserted-by":"crossref","first-page":"e1001381","DOI":"10.1371\/journal.pmed.1001381","article-title":"Prognosis research strategy (PROGRESS) 3: prognostic model research","volume":"10","author":"Steyerberg","year":"2013","journal-title":"PLoS Med"},{"issue":"1","key":"2020110612460826500_ocz130-B21","doi-asserted-by":"crossref","first-page":"W1","DOI":"10.7326\/M14-0698","article-title":"Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration","volume":"162","author":"Moons","year":"2015","journal-title":"Ann Intern Med"},{"issue":"9913","key":"2020110612460826500_ocz130-B22","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1016\/S0140-6736(13)62228-X","article-title":"Reducing waste from incomplete and unusable reports of biomedical research","volume":"383","author":"Glasziou","year":"2014","journal-title":"Lancet"},{"issue":"1","key":"2020110612460826500_ocz130-B23","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1016\/j.ajog.2015.06.013","article-title":"Prognostic models in obstetrics: available, but far from applicable","volume":"214","author":"Kleinrouweler","year":"2016","journal-title":"Am J Obstet Gynecol"},{"issue":"3","key":"2020110612460826500_ocz130-B24","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1007\/s11914-012-0108-1","article-title":"Fracture risk assessment: state of the art, methodologically unsound, or poorly reported","volume":"10","author":"Collins","year":"2012","journal-title":"Curr Osteoporos Rep"},{"issue":"10","key":"2020110612460826500_ocz130-B25","doi-asserted-by":"crossref","first-page":"1273.","DOI":"10.1093\/jamia\/ocy136","article-title":"Data science and artificial intelligence to improve clinical practice and research","volume":"25","author":"Ohno-Machado","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"11","key":"2020110612460826500_ocz130-B26","doi-asserted-by":"crossref","first-page":"e1002689.","DOI":"10.1371\/journal.pmed.1002689","article-title":"Machine learning in medicine: addressing ethical challenges","volume":"15","author":"Vayena","year":"2018","journal-title":"PLoS Med"},{"issue":"8","key":"2020110612460826500_ocz130-B27","doi-asserted-by":"crossref","first-page":"969","DOI":"10.1093\/jamia\/ocy032","article-title":"Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data","volume":"25","author":"Reps","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"10192","key":"2020110612460826500_ocz130-B28","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1016\/S0140-6736(19)30762-7","article-title":"WHO and ITU establish benchmarking process for artificial intelligence in health","volume":"394","author":"Wiegand","year":"2019","journal-title":"Lancet"},{"issue":"11","key":"2020110612460826500_ocz130-B29","doi-asserted-by":"crossref","first-page":"981","DOI":"10.1056\/NEJMp1714229","article-title":"Implementing machine learning in health care\u2013addressing ethical challenges","volume":"378","author":"Char","year":"2018","journal-title":"N Engl J Med"},{"issue":"3","key":"2020110612460826500_ocz130-B30","doi-asserted-by":"crossref","first-page":"910","DOI":"10.1148\/radiol.2018181310","article-title":"Regulatory approval versus clinical validation of artificial intelligence diagnostic tools","volume":"288","author":"Park","year":"2018","journal-title":"Radiology"},{"key":"2020110612460826500_ocz130-B31","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.jclinepi.2019.02.004","article-title":"A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models","volume":"110","author":"Christodoulou","year":"2019","journal-title":"J Clin Epidemiol"},{"issue":"27","key":"2020110612460826500_ocz130-B32","doi-asserted-by":"crossref","first-page":"7329","DOI":"10.1073\/pnas.1510502113","article-title":"Characterizing treatment pathways at scale using the OHDSI network","volume":"113","author":"Hripcsak","year":"2016","journal-title":"Proc Natl Acad Sci USA"},{"issue":"6","key":"2020110612460826500_ocz130-B33","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1159\/000342138","article-title":"Legal liability and the uncertain nature of risk prediction: the case of breast cancer risk prediction models","volume":"15","author":"Black","year":"2012","journal-title":"Public Health Genomics"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/12\/1651\/34152002\/ocz130.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/12\/1651\/34152002\/ocz130.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T18:34:22Z","timestamp":1604687662000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/26\/12\/1651\/5542900"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,8,2]]},"references-count":33,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2019,8,2]]},"published-print":{"date-parts":[[2019,12,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocz130","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,12]]},"published":{"date-parts":[[2019,8,2]]}}}