{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T05:48:36Z","timestamp":1781156916561,"version":"3.54.1"},"reference-count":40,"publisher":"Oxford University Press (OUP)","issue":"8","license":[{"start":{"date-parts":[[2018,4,14]],"date-time":"2018-04-14T00:00:00Z","timestamp":1523664000000},"content-version":"vor","delay-in-days":1,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000925","name":"National Health and Medical Research Council","doi-asserted-by":"publisher","award":["APP1134919"],"award-info":[{"award-number":["APP1134919"]}],"id":[{"id":"10.13039\/501100000925","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2018,8,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>Many research fields, including psychology and basic medical sciences, struggle with poor reproducibility of reported studies. Biomedical and health informatics is unlikely to be immune to these challenges. This paper explores replication in informatics and the unique challenges the discipline faces.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Narrative review of recent literature on research replication challenges.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>While there is growing interest in re-analysis of existing data, experimental replication studies appear uncommon in informatics. Context effects are a particular challenge as they make ensuring replication fidelity difficult, and the same intervention will never quite reproduce the same result in different settings. Replication studies take many forms, trading-off testing validity of past findings against testing generalizability. Exact and partial replication designs emphasize testing validity while quasi and conceptual studies test generalizability of an underlying model or hypothesis with different methods or in a different setting.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>The cost of poor replication is a weakening in the quality of published research and the evidence-based foundation of health informatics. The benefits of replication include increased rigor in research, and the development of evaluation methods that distinguish the impact of context and the nonreproducibility of research. Taking replication seriously is essential if biomedical and health informatics is to be an evidence-based discipline.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocy028","type":"journal-article","created":{"date-parts":[[2018,3,16]],"date-time":"2018-03-16T16:09:22Z","timestamp":1521216562000},"page":"963-968","source":"Crossref","is-referenced-by-count":89,"title":["Does health informatics have a replication crisis?"],"prefix":"10.1093","volume":"25","author":[{"given":"Enrico","family":"Coiera","sequence":"first","affiliation":[{"name":"Australian Institute of Health Innovation, Macquarie University, NSW 2109, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elske","family":"Ammenwerth","sequence":"additional","affiliation":[{"name":"University for Health Sciences, Medical Informatics and Technology, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrew","family":"Georgiou","sequence":"additional","affiliation":[{"name":"Australian Institute of Health Innovation, Macquarie University, NSW 2109, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Farah","family":"Magrabi","sequence":"additional","affiliation":[{"name":"Australian Institute of Health Innovation, Macquarie University, NSW 2109, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2018,4,13]]},"reference":[{"key":"2020110612230319800_ocy028-B1","volume-title":"Conjectures and Refutations","author":"Popper","year":"1963"},{"issue":"8","key":"2020110612230319800_ocy028-B2","doi-asserted-by":"crossref","first-page":"e124","DOI":"10.1371\/journal.pmed.0020124","article-title":"Why most published research findings are false","volume":"2","author":"Ioannidis","year":"2005","journal-title":"PLoS Med"},{"issue":"6","key":"2020110612230319800_ocy028-B3","doi-asserted-by":"crossref","first-page":"e1002165","DOI":"10.1371\/journal.pbio.1002165","article-title":"The economics of reproducibility in preclinical research","volume":"13","author":"Freedman","year":"2015","journal-title":"PLOS Biol"},{"issue":"9","key":"2020110612230319800_ocy028-B4","doi-asserted-by":"crossref","first-page":"977","DOI":"10.1001\/jama.2009.1242","article-title":"Comparison of registered and published primary outcomes in randomized controlled trials","volume":"302","author":"Mathieu","year":"2009","journal-title":"JAMA"},{"issue":"6","key":"2020110612230319800_ocy028-B5","doi-asserted-by":"crossref","first-page":"1146","DOI":"10.1037\/xge0000104","article-title":"Better P-curves: making P-curve analysis more robust to errors, fraud, and ambitious P-hacking, a Reply to Ulrich and Miller","volume":"144","author":"Simonsohn","year":"2015","journal-title":"J Exp Psychol Gen"},{"issue":"10","key":"2020110612230319800_ocy028-B6","doi-asserted-by":"crossref","first-page":"1405","DOI":"10.1001\/jama.1990.03440100121018","article-title":"Underreporting research is scientific misconduct","volume":"263","author":"Chalmers","year":"1990","journal-title":"JAMA"},{"issue":"10","key":"2020110612230319800_ocy028-B7","doi-asserted-by":"crossref","first-page":"e1002273","DOI":"10.1371\/journal.pbio.1002273","article-title":"Risk of bias in reports of in vivo research: a focus for improvement","volume":"13","author":"Macleod","year":"2015","journal-title":"PLoS Biol"},{"issue":"4","key":"2020110612230319800_ocy028-B8","doi-asserted-by":"crossref","first-page":"e00164","DOI":"10.1002\/prp2.164","article-title":"Replication \u2013 why we need to publish our findings","volume":"3","author":"Curtis","year":"2015","journal-title":"Pharmacol Res Perspect"},{"key":"2020110612230319800_ocy028-B9","doi-asserted-by":"crossref","first-page":"1037","DOI":"10.1126\/science.aad9163","article-title":"Response to Comment on \u201cEstimating the reproducibility of psychological science\u201d","volume":"351","author":"Anderson","year":"2016","journal-title":"Science"},{"issue":"6251","key":"2020110612230319800_ocy028-B10","doi-asserted-by":"crossref","DOI":"10.1126\/science.aac4716","article-title":"Estimating the reproducibility of psychological science","volume":"349","author":"Open Science Collaboration","year":"2015","journal-title":"Science"},{"issue":"3","key":"2020110612230319800_ocy028-B11","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1093\/biostatistics\/kxp014","article-title":"Reproducible research and Biostatistics","volume":"10","author":"Peng","year":"2009","journal-title":"Biostatistics"},{"key":"2020110612230319800_ocy028-B12","article-title":"Replicability is not reproducibility: nor is it good science","author":"Drummond"},{"issue":"2","key":"2020110612230319800_ocy028-B13","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1136\/jamia.2010.010306","article-title":"The case for randomized controlled trials to assess the impact of clinical information systems","volume":"18","author":"Liu","year":"2011","journal-title":"J Am Med Inform Assoc"},{"issue":"6","key":"2020110612230319800_ocy028-B14","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1177\/1745691612463401","article-title":"Is the replicability crisis overblown? Three arguments examined","volume":"7","author":"Pashler","year":"2012","journal-title":"Perspect Psychol Sci"},{"issue":"1","key":"2020110612230319800_ocy028-B15","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1186\/s12915-015-0196-3","article-title":"Replicating research in ecology and evolution: feasibility, incentives, and the cost-benefit conundrum","volume":"13","author":"Nakagawa","year":"2015","journal-title":"BMC Biol"},{"key":"2020110612230319800_ocy028-B16","volume-title":"Evidence-Based Health Informatics","author":"Ammenwerth","year":"2016"},{"issue":"2","key":"2020110612230319800_ocy028-B17","doi-asserted-by":"crossref","first-page":"246","DOI":"10.1093\/jamia\/ocw154","article-title":"Problems with health information technology and their effects on care delivery and patient outcomes: a systematic review","volume":"24","author":"Kim","year":"2017","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2020110612230319800_ocy028-B18","doi-asserted-by":"crossref","first-page":"e1000387","DOI":"10.1371\/journal.pmed.1000387","article-title":"The impact of eHealth on the quality and safety of health care: a systematic overview","volume":"8","author":"Black","year":"2011","journal-title":"PLoS Med"},{"key":"2020110612230319800_ocy028-B19","doi-asserted-by":"crossref","DOI":"10.23970\/AHRQEPCERTA132","volume-title":"Costs and Benefits of Health Information Technology. Evidence Report\/Technology Assessment","author":"Shekelle","year":"2006"},{"issue":"4","key":"2020110612230319800_ocy028-B20","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1093\/aje\/kwt010","article-title":"Evaluating the impact of database heterogeneity on observational study results","volume":"178","author":"Madigan","year":"2013","journal-title":"Am J Epidemiol"},{"issue":"6","key":"2020110612230319800_ocy028-B21","doi-asserted-by":"crossref","first-page":"1506","DOI":"10.1542\/peds.2005-1287","article-title":"Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system","volume":"116","author":"Han","year":"2005","journal-title":"Pediatrics"},{"issue":"2","key":"2020110612230319800_ocy028-B22","doi-asserted-by":"crossref","first-page":"114","DOI":"10.5811\/westjem.2012.9.6601","article-title":"Implementation of computerized physician order entry for critical patients in an Academic Emergency Department is not associated with a change in mortality rate","volume":"14","author":"Brunette","year":"2013","journal-title":"Western J Emerg Med"},{"key":"2020110612230319800_ocy028-B23","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1186\/1472-6947-11-71","article-title":"Impact of computerized physician order entry (CPOE) system on the outcome of critically ill adult patients: a before-after study","volume":"11","author":"Al-Dorzi","year":"2011","journal-title":"BMC Med Inform Decis Making"},{"issue":"1","key":"2020110612230319800_ocy028-B24","doi-asserted-by":"crossref","first-page":"290","DOI":"10.1542\/peds.2006-0367","article-title":"Computerized provider order entry implementation: no association with increased mortality rates in an intensive care unit","volume":"118","author":"Del Beccaro","year":"2006","journal-title":"Pediatrics"},{"issue":"1","key":"2020110612230319800_ocy028-B25","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1542\/peds.2009-3271","article-title":"Decrease in hospital-wide mortality rate after implementation of a commercially sold computerized physician order entry system","volume":"126","author":"Longhurst","year":"2010","journal-title":"Pediatrics"},{"issue":"6","key":"2020110612230319800_ocy028-B26","first-page":"586","article-title":"Impact of CPOE on mortality rates \u2013 contradictory findings, important messages","volume":"45","author":"Ammenwerth","year":"2006","journal-title":"Methods Archive"},{"key":"2020110612230319800_ocy028-B27","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1186\/1748-5908-5-82","article-title":"A critical synthesis of literature on the promoting action on research implementation in health services (PARIHS) framework","volume":"5","author":"Helfrich","year":"2010","journal-title":"Implement Sci"},{"key":"2020110612230319800_ocy028-B28","doi-asserted-by":"crossref","first-page":"160035","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Scientific Data"},{"key":"2020110612230319800_ocy028-B29","first-page":"574","article-title":"Observational Health Data Sciences and Informatics (OHDSI): Opportunities for Observational Researchers","volume":"216","author":"Hripcsak","year":"2015","journal-title":"Stud Health Technol Inform"},{"issue":"3","key":"2020110612230319800_ocy028-B30","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1086\/506236","article-title":"Replicating empirical research in behavioral ecology: how and why it should be done but rarely ever is","volume":"81","author":"Kelly","year":"2006","journal-title":"Quart Rev Biol"},{"issue":"1","key":"2020110612230319800_ocy028-B31","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1146\/annurev.ecolsys.31.1.441","article-title":"Quasi-replication and the contract of error: lessons from sex ratios, heritabilities and fluctuating asymmetry","volume":"31","author":"Palmer","year":"2000","journal-title":"Ann Rev Ecol Syst"},{"issue":"3","key":"2020110612230319800_ocy028-B32","doi-asserted-by":"crossref","first-page":"331","DOI":"10.4338\/ACI-2013-04-RA-0024","article-title":"STARE-HI \u2013 Statement on Reporting of Evaluation Studies in Health Informatics: Explanation and Elaboration","volume":"4","author":"Brender","year":"2013","journal-title":"Appl Clin Inform"},{"key":"2020110612230319800_ocy028-B33","doi-asserted-by":"crossref","first-page":"1250","DOI":"10.1056\/NEJMe048225","article-title":"Clinical trial registration: a statement from the International Committee of Medical Journal Editors","volume":"351","author":"De Angelis","year":"2004","journal-title":"New Engl J Med"},{"key":"2020110612230319800_ocy028-B34","doi-asserted-by":"crossref","first-page":"2277","DOI":"10.1056\/NEJMe1705439","article-title":"Data Sharing Statements for Clinical Trials\u2014A Requirement of the International Committee of Medical Journal Editors","volume":"376","author":"Taichman","year":"2017","journal-title":"N Engl J Med"},{"issue":"1","key":"2020110612230319800_ocy028-B35","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s41073-016-0006-7","article-title":"Conflict of interest disclosure in biomedical research: a review of current practices, biases, and the role of public registries in improving transparency","volume":"1","author":"Dunn","year":"2016","journal-title":"Res Integrity Peer Rev"},{"issue":"1","key":"2020110612230319800_ocy028-B36","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1093\/jamia\/ocx120","article-title":"Reproducible Bioconductor workflows using browser-based interactive notebooks and containers","volume":"25","author":"Almugbel","year":"2018","journal-title":"J Am Med Inform Assoc"},{"key":"2020110612230319800_ocy028-B37","doi-asserted-by":"crossref","DOI":"10.1093\/intqhc\/mzx076","article-title":"Linking quality indicators to clinical trials: an automated approach","author":"Coiera","year":"2017","journal-title":"Int J Qual Healthcare"},{"issue":"1","key":"2020110612230319800_ocy028-B38","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1186\/s12874-017-0461-y","article-title":"Getting messier with TIDieR: embracing context and complexity in intervention reporting","volume":"18","author":"Cotterill","year":"2018","journal-title":"BMC Med Res Methodol"},{"issue":"169","key":"2020110612230319800_ocy028-B39","article-title":"Emerging trends in peer review\u2014a survey","volume":"9","author":"Walker","year":"2015","journal-title":"Front Neurosci"},{"issue":"9","key":"2020110612230319800_ocy028-B40","doi-asserted-by":"crossref","first-page":"e1001864","DOI":"10.1371\/journal.pmed.1001864","article-title":"Four proposals to help improve the medical research literature","volume":"12","author":"Moher","year":"2015","journal-title":"PLOS Med"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/25\/8\/963\/34150203\/ocy028.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/25\/8\/963\/34150203\/ocy028.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,8,16]],"date-time":"2022-08-16T02:47:23Z","timestamp":1660618043000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/25\/8\/963\/4970161"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,4,13]]},"references-count":40,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2018,4,13]]},"published-print":{"date-parts":[[2018,8,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocy028","relation":{"has-review":[{"id-type":"doi","id":"10.3410\/f.733061229.793544960","asserted-by":"object"}]},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2018,8]]},"published":{"date-parts":[[2018,4,13]]}}}