{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T05:55:27Z","timestamp":1776318927375,"version":"3.50.1"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,5,11]],"date-time":"2020-05-11T00:00:00Z","timestamp":1589155200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,5,11]],"date-time":"2020-05-11T00:00:00Z","timestamp":1589155200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000092","name":"U.S. Department of Health & Human Services | NIH | U.S. National Library of Medicine","doi-asserted-by":"publisher","award":["R01LM009886-10"],"award-info":[{"award-number":["R01LM009886-10"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"U.S. Department of Health & Human Services | NIH | U.S. National Library of Medicine","doi-asserted-by":"publisher","award":["T15LM007079"],"award-info":[{"award-number":["T15LM007079"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"name":"U.S. Department of Health & Human Services | NIH | U.S. National Library of Medicine"},{"DOI":"10.13039\/100008897","name":"Janssen Pharmaceuticals","doi-asserted-by":"publisher","award":["CU15-2317"],"award-info":[{"award-number":["CU15-2317"]}],"id":[{"id":"10.13039\/100008897","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Randomized controlled trials (RCTs) are regarded as the most reputable source of evidence. In some studies, factors beyond the intervention itself may contribute to the measured effect, an occurrence known as heterogeneity of treatment effect (HTE). If the RCT population differs from the real-world population on factors that induce HTE, the trials effect will not replicate. The RCTs eligibility criteria should identify the sub-population in which its evidence will replicate. However, the extent to which the eligibility criteria identify the appropriate population is unknown, which raises concerns for generalizability. We compared reported data from RCTs with real-world data from the electronic health records of a large, academic medical center that was curated according to RCT eligibility criteria. Our results show fundamental differences between the RCT population and our observational cohorts, which suggests that eligibility criteria may be insufficient for identifying the applicable real-world population in which RCT evidence will replicate.<\/jats:p>","DOI":"10.1038\/s41746-020-0277-8","type":"journal-article","created":{"date-parts":[[2020,5,11]],"date-time":"2020-05-11T10:03:49Z","timestamp":1589191429000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":114,"title":["Translating evidence into practice: eligibility criteria fail to eliminate clinically significant differences between real-world and study populations"],"prefix":"10.1038","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1221-1123","authenticated-orcid":false,"given":"Amelia J.","family":"Averitt","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9624-0214","authenticated-orcid":false,"given":"Chunhua","family":"Weng","sequence":"additional","affiliation":[]},{"given":"Patrick","family":"Ryan","sequence":"additional","affiliation":[]},{"given":"Adler","family":"Perotte","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,5,11]]},"reference":[{"key":"277_CR1","unstructured":"Wong, V. C. & Steiner P. M. Replication designs for causal inference. EdPolicyWorks Working Paper Series. 2018 [cited 2019 Mar 26]. Available from: http:\/\/curry.virginia.edu\/uploads\/epw\/62_Replication_Designs.pdfhttp:\/\/curry.virginia.edu\/edpolicyworks\/wp"},{"key":"277_CR2","doi-asserted-by":"crossref","unstructured":"Djulbegovic, B. & Guyatt, G. H. Progress in evidence-based medicine: a quarter century on. Lancet 390, 415\u2013423 (2017).","DOI":"10.1016\/S0140-6736(16)31592-6"},{"key":"277_CR3","unstructured":"Djulbegovic, B. & Guyatt G. in Users Guide to Medical Literature. 3rd edn. (McGraw-Hill Education, 1976)."},{"key":"277_CR4","doi-asserted-by":"crossref","unstructured":"Djulbegovic, B., Guyatt, G. H. & Ashcroft, R. E. Epistemologic inquiries in evidence-based medicine. Cancer Cont. 16, 158\u2013168.","DOI":"10.1177\/107327480901600208"},{"key":"277_CR5","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1136\/bmj.312.7023.71","volume":"312","author":"DL Sackett","year":"1996","unstructured":"Sackett, D. L., Rosenberg, W. M., Gray, J. A., Haynes, R. B. & Richardson, W. S. Evidence based medicine: what it is and what it isn\u2019t. BMJ 312, 71\u201372 (1996).","journal-title":"BMJ"},{"key":"277_CR6","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1001747","volume":"11","author":"JPA Ioannidis","year":"2014","unstructured":"Ioannidis, J. P. A. How to make more published research true. PLoS Med. 11, e1001747 (2014).","journal-title":"PLoS Med."},{"key":"277_CR7","doi-asserted-by":"publisher","first-page":"1298","DOI":"10.1126\/science.1160622","volume":"321","author":"DG Contopoulos-Ioannidis","year":"2008","unstructured":"Contopoulos-Ioannidis, D. G., Alexiou, G. A., Gouvias, T. C. & Ioannidis, J. P. A. Life cycle of translational research for medical interventions. Science 321, 1298\u20131299 (2008).","journal-title":"Science"},{"key":"277_CR8","doi-asserted-by":"publisher","first-page":"dyw118","DOI":"10.1093\/ije\/dyw118","volume":"45","author":"DM Kent","year":"2016","unstructured":"Kent, D. M. et al. Risk and treatment effect heterogeneity: re-analysis of individual participant data from 32 large clinical trials. Int. J. Epidemiol. 45, dyw118 (2016).","journal-title":"Int. J. Epidemiol."},{"key":"277_CR9","doi-asserted-by":"publisher","first-page":"435","DOI":"10.1198\/073500108000000033","volume":"26","author":"P Fredriksson","year":"2008","unstructured":"Fredriksson, P. & Johansson, P. Dynamic treatment assignment. J. Bus. Econ. Stat. 26, 435\u2013445 (2008).","journal-title":"J. Bus. Econ. Stat."},{"key":"277_CR10","doi-asserted-by":"publisher","first-page":"314","DOI":"10.1177\/0081175012452652","volume":"42","author":"Y Xie","year":"2012","unstructured":"Xie, Y., Brand, J. E. & Jann, B. Estimating heterogeneous treatment effects with observational data. Socio. Methodol. 42, 314\u2013347 (2012).","journal-title":"Socio. Methodol."},{"key":"277_CR11","volume-title":"Experimental and Quasi-experimental Designs for Research","author":"DT Campbell","year":"1963","unstructured":"Campbell, D. T. & Stanley, J. C. Experimental and Quasi-experimental Designs for Research. (Houghton Mifflin Company, Boston, 1963)."},{"key":"277_CR12","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1097\/PRS.0b013e318219c171","volume":"128","author":"PB Burns","year":"2011","unstructured":"Burns, P. B., Rohrich, R. J. & Chung, K. C. The levels of evidence and their role in evidence-based medicine. Plast. Reconstr. Surg. 128, 305\u2013310 (2011).","journal-title":"Plast. Reconstr. Surg."},{"key":"277_CR13","unstructured":"Campbell, D. T. & Stanley, J. C. Handbook of Research on Teaching. 1\u201384 (Houghton Mifflin Company, Boston, 1963)."},{"key":"277_CR14","doi-asserted-by":"crossref","unstructured":"Hyman, R. Quasi-Experimentation: Design and Analysis Issues for Field Settings. Vol. 46, 96\u201397 (Houghton Mifflin, 1982).","DOI":"10.1207\/s15327752jpa4601_16"},{"key":"277_CR15","doi-asserted-by":"crossref","unstructured":"Anderson-Cook, C. M. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Vol. 100 (Wiley, 2005).","DOI":"10.1198\/jasa.2005.s22"},{"key":"277_CR16","unstructured":"Velasco, E. in Encyclopedia of Research Design (ed. Salkind, N) (SAGE, Thousand Oaks Publications, 2010)."},{"key":"277_CR17","doi-asserted-by":"publisher","first-page":"893","DOI":"10.1016\/j.brat.2009.06.019","volume":"47","author":"JA Wales","year":"2009","unstructured":"Wales, J. A., Palmer, R. L. & Fairburn, C. G. Can treatment trial samples be representative? Behav. Res. Ther. 47, 893\u2013896 (2009).","journal-title":"Behav. Res. Ther."},{"key":"277_CR18","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1017\/S0266462300009570","volume":"12","author":"D Moher","year":"1996","unstructured":"Moher, D., Jadad, A. R. & Tugwell, P. Assessing the quality of randomized controlled trials. Current issues and future directions. Int. J. Technol. Assess. Health Care. 12, 195\u2013208 (1996).","journal-title":"Int. J. Technol. Assess. Health Care."},{"key":"277_CR19","doi-asserted-by":"publisher","first-page":"112","DOI":"10.1177\/135581969900400210","volume":"4","author":"A Britton","year":"1999","unstructured":"Britton, A. et al. Threats to applicability of randomised trials: exclusions and selective participation. J. Heal. Serv. Res. Policy 4, 112\u2013121 (1999).","journal-title":"J. Heal. Serv. Res. Policy"},{"key":"277_CR20","doi-asserted-by":"publisher","unstructured":"Karanis, Y. B., Canta, F. A. B., Mitrofan, L., Mistry, H. & Anger, C. \u2018Research\u2019 vs \u2018real world\u2019 patients: the representativeness of clinical trial participants. Ann. Oncol. (2016) https:\/\/doi.org\/10.1093\/annonc\/mdw392.51\/2800468\/Research-vs-real-world-patients-the","DOI":"10.1093\/annonc\/mdw392.51\/2800468\/Research-vs-real-world-patients-the"},{"key":"277_CR21","doi-asserted-by":"publisher","first-page":"475","DOI":"10.1007\/s11121-014-0513-z","volume":"16","author":"EA Stuart","year":"2015","unstructured":"Stuart, E. A., Bradshaw, C. P. & Leaf, P. J. Assessing the generalizability of randomized trial results to target populations. Prev. Sci. 16, 475\u2013485 (2015).","journal-title":"Prev. Sci."},{"key":"277_CR22","doi-asserted-by":"publisher","first-page":"2293","DOI":"10.1056\/NEJMsb1609216","volume":"375","author":"RE Sherman","year":"2016","unstructured":"Sherman, R. E. et al. Real-world evidence \u2014 what is it and what can it tell us? N. Engl. J. Med. 375, 2293\u20132297 (2016).","journal-title":"N. Engl. J. Med."},{"key":"277_CR23","doi-asserted-by":"publisher","first-page":"495","DOI":"10.1186\/s13063-015-1023-4","volume":"16","author":"T Kennedy-Martin","year":"2015","unstructured":"Kennedy-Martin, T., Curtis, S., Faries, D., Robinson, S. & Johnston, J. A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results. Trials 16, 495 (2015).","journal-title":"Trials"},{"key":"277_CR24","first-page":"84","volume":"41","author":"LP Badano","year":"2003","unstructured":"Badano, L. P. et al. Patients with chronic heart failure encountered in daily clinical practice are different from the \u201ctypical\u201d patient enrolled in therapeutic trials. Ital. Hear. J. 41, 84\u201391 (2003).","journal-title":"Ital. Hear. J."},{"key":"277_CR25","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1016\/j.ijcard.2006.12.027","volume":"124","author":"X Bosch","year":"2008","unstructured":"Bosch, X. et al. Causes of ineligibility in randomized controlled trials and long-term mortality in patients with non-ST-segment elevation acute coronary syndromes. Int. J. Cardiol. 124, 86\u201391 (2008).","journal-title":"Int. J. Cardiol."},{"key":"277_CR26","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1016\/S0735-1097(02)02664-5","volume":"41","author":"JP Collet","year":"2003","unstructured":"Collet, J. P. et al. Enoxaparin in unstable angina patients who would have been excluded from randomized pivotal trials. J. Am. Coll. Cardiol. 41, 8\u201314 (2003).","journal-title":"J. Am. Coll. Cardiol."},{"key":"277_CR27","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1007\/s11739-008-0180-9","volume":"4","author":"G Costantino","year":"2009","unstructured":"Costantino, G. et al. Eligibility criteria in heart failure randomized controlled trials: a gap between evidence and clinical practice. Intern. Emerg. Med. 4, 117\u2013122 (2009).","journal-title":"Intern. Emerg. Med."},{"key":"277_CR28","doi-asserted-by":"publisher","first-page":"136","DOI":"10.1001\/archinternmed.2007.56","volume":"168","author":"SS Dhruva","year":"2008","unstructured":"Dhruva, S. S. & Redberg, R. F. Variations between clinical trial participants and medicare beneficiaries in evidence used for medicare national coverage decisions. Arch. Intern. Med. 168, 136 (2008).","journal-title":"Arch. Intern. Med."},{"key":"277_CR29","doi-asserted-by":"publisher","first-page":"735","DOI":"10.1161\/CIRCHEARTFAILURE.112.968974","volume":"5","author":"JA Ezekowitz","year":"2012","unstructured":"Ezekowitz, J. A. et al. Acute heart failure. Circ. Hear. Fail. 5, 735\u2013741 (2012).","journal-title":"Circ. Hear. Fail."},{"key":"277_CR30","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2012-000833","volume":"2","author":"BA Golomb","year":"2012","unstructured":"Golomb, B. A. et al. The older the better: are elderly study participants more non-representative? A cross-sectional analysis of clinical trial and observational study samples. BMJ Open. 2, e000833 (2012).","journal-title":"BMJ Open."},{"key":"277_CR31","doi-asserted-by":"publisher","first-page":"1389","DOI":"10.1016\/j.amjcard.2010.06.070","volume":"106","author":"AB Hutchinson-Jaffe","year":"2010","unstructured":"Hutchinson-Jaffe, A. B. et al. Comparison of baseline characteristics, management and outcome of patients with non\u2013ST-segment elevation acute coronary syndrome in versus not in clinical trials. Am. J. Cardiol. 106, 1389\u20131396 (2010).","journal-title":"Am. J. Cardiol."},{"key":"277_CR32","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1161\/CIRCOUTCOMES.110.868307","volume":"3","author":"C Melloni","year":"2010","unstructured":"Melloni, C. et al. Representation of women in randomized clinical trials of cardiovascular disease prevention. Circ. Cardiovasc. Qual. Outcomes 3, 135\u2013142 (2010).","journal-title":"Circ. Cardiovasc. Qual. Outcomes"},{"key":"277_CR33","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1016\/j.ahj.2007.03.047","volume":"154","author":"BA Steinberg","year":"2007","unstructured":"Steinberg, B. A. et al. Global outcomes of ST-elevation myocardial infarction: comparisons of the enoxaparin and thrombolysis reperfusion for acute myocardial infarction treatment-thrombolysis in myocardial infarction study 25 (ExTRACT-TIMI 25) registry and trial. Am. Heart J. 154, 54\u201361 (2007).","journal-title":"Am. Heart J."},{"key":"277_CR34","doi-asserted-by":"publisher","first-page":"330","DOI":"10.1016\/j.jclinepi.2006.05.015","volume":"60","author":"AA Uijen","year":"2007","unstructured":"Uijen, A. A., Bakx, J. C., Mokkink, H. G. A. & van Weel, C. Hypertension patients participating in trials differ in many aspects from patients treated in general practices. J. Clin. Epidemiol. 60, 330\u2013335 (2007).","journal-title":"J. Clin. Epidemiol."},{"key":"277_CR35","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1001\/jama.297.11.1233","volume":"297","author":"HGC Van Spall","year":"2007","unstructured":"Van Spall, H. G. C., Toren, A., Kiss, A. & Fowler, R. A. Eligibility criteria of randomized controlled trials published in high-impact general medical journals. JAMA 297, 1233 (2007).","journal-title":"JAMA"},{"key":"277_CR36","doi-asserted-by":"publisher","first-page":"c332","DOI":"10.1136\/bmj.c332","volume":"340","author":"KF Schulz","year":"2010","unstructured":"Schulz, K. F., Altman, D. G. & Moher, D. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMJ 340, c332 (2010).","journal-title":"BMJ"},{"key":"277_CR37","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1186\/1475-9276-11-14","volume":"11","author":"J Furler","year":"2012","unstructured":"Furler, J., Magin, P., Pirotta, M. & van Driel, M. Participant demographics reported in \u201cTable 1\u201d of randomised controlled trials: a case of \u201cinverse evidence\u201d? Int. J. Equity Health. 11, 14 (2012).","journal-title":"Int. J. Equity Health."},{"key":"277_CR38","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2019.12869","volume":"2","author":"VL Bartlett","year":"2019","unstructured":"Bartlett, V. L., Dhruva, S. S., Shah, N. D., Ryan, P. & Ross, J. S. Feasibility of using real-world data to replicate clinical trial evidence. JAMA Netw. Open. 2, e1912869 (2019).","journal-title":"JAMA Netw. Open."},{"key":"277_CR39","doi-asserted-by":"publisher","first-page":"861","DOI":"10.1056\/NEJMoa011161","volume":"345","author":"BM Brenner","year":"2001","unstructured":"Brenner, B. M. et al. Effects of losartan on renal and cardiovascular outcomes in patients with type 2 diabetes and nephropathy. N. Engl. J. Med. 345, 861\u2013869 (2001).","journal-title":"N. Engl. J. Med."},{"key":"277_CR40","doi-asserted-by":"crossref","unstructured":"Jamerson, K. et al. Benazepril plus Amlodipine or hydrochlorothiazide for hypertension in high-risk patients. N. Engl. J. Med. 359, 2417\u20132428 (2008) http:\/\/www.nejm.org\/doi\/abs\/10.1056\/NEJMoa0806182.","DOI":"10.1056\/NEJMoa0806182"},{"key":"277_CR41","doi-asserted-by":"publisher","first-page":"1495","DOI":"10.1056\/NEJMoa040583","volume":"350","author":"CP Cannon","year":"2004","unstructured":"Cannon, C. P. et al. Intensive versus moderate lipid lowering with statins after acute coronary syndromes. N. Engl. J. Med. 350, 1495\u20131504 (2004).","journal-title":"N. Engl. J. Med."},{"key":"277_CR42","doi-asserted-by":"publisher","first-page":"469","DOI":"10.1007\/s40266-015-0271-z","volume":"32","author":"P Hartley","year":"2015","unstructured":"Hartley, P. Efficacy and tolerability of sitagliptin compared with glimepiride in elderly patients with type 2 diabetes mellitus and inadequate glycemic control: a randomized, double-blind, non-inferiority trial. Drugs Aging 32, 469\u2013476 (2015).","journal-title":"Drugs Aging"}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0277-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0277-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0277-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T02:12:40Z","timestamp":1670379160000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0277-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5,11]]},"references-count":42,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,12]]}},"alternative-id":["277"],"URL":"https:\/\/doi.org\/10.1038\/s41746-020-0277-8","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,5,11]]},"assertion":[{"value":"10 December 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 April 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 May 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"P.R. is an employee of Janssen Research and Development and a shareholder of Johnson & Johnson. All other authors have no relevant conflicts of interest.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"67"}}