{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,25]],"date-time":"2026-08-25T22:26:48Z","timestamp":1787696808281,"version":"build-2784847793"},"reference-count":26,"publisher":"Public Library of Science (PLoS)","issue":"7","license":[{"start":{"date-parts":[[2025,7,18]],"date-time":"2025-07-18T00:00:00Z","timestamp":1752796800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"The British Heart Foundation Data Science Centre","award":["SP\/19\/3\/34678"],"award-info":[{"award-number":["SP\/19\/3\/34678"]}]},{"DOI":"10.13039\/100014013","name":"UK Research and Innovation","doi-asserted-by":"crossref","award":["MC_PC_20058"],"award-info":[{"award-number":["MC_PC_20058"]}],"id":[{"id":"10.13039\/100014013","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100014653","name":"NIHR Manchester Biomedical Research Centre","doi-asserted-by":"crossref","award":["NIHR203308"],"award-info":[{"award-number":["NIHR203308"]}],"id":[{"id":"10.13039\/100014653","id-type":"DOI","asserted-by":"crossref"}]},{"name":"the NIHR Applied Research Collaboration Greater Manchester","award":["NIHR200174"],"award-info":[{"award-number":["NIHR200174"]}]},{"name":"the BHF Data Science Centre","award":["HDRUK2023.0239"],"award-info":[{"award-number":["HDRUK2023.0239"]}]},{"name":"NIHR Research Professor","award":["NIHR303137"],"award-info":[{"award-number":["NIHR303137"]}]},{"DOI":"10.13039\/501100000274","name":"British Heart Foundation","doi-asserted-by":"publisher","award":["RG\/18\/13\/33946"],"award-info":[{"award-number":["RG\/18\/13\/33946"]}],"id":[{"id":"10.13039\/501100000274","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100018956","name":"NIHR Cambridge Biomedical Research Centre","doi-asserted-by":"publisher","award":["BRC-1215-20014"],"award-info":[{"award-number":["BRC-1215-20014"]}],"id":[{"id":"10.13039\/501100018956","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100018956","name":"NIHR Cambridge Biomedical Research Centre","doi-asserted-by":"publisher","award":["NIHR203312"],"award-info":[{"award-number":["NIHR203312"]}],"id":[{"id":"10.13039\/501100018956","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100024893","name":"Cambridge BHF Centre of Research Excellence","doi-asserted-by":"crossref","award":["RE\/18\/1\/34212"],"award-info":[{"award-number":["RE\/18\/1\/34212"]}],"id":[{"id":"10.13039\/501100024893","id-type":"DOI","asserted-by":"crossref"}]},{"name":"BHF Chair Award","award":["CH\/12\/2\/29428"],"award-info":[{"award-number":["CH\/12\/2\/29428"]}]},{"DOI":"10.13039\/501100023699","name":"Health Data Research UK","doi-asserted-by":"publisher","award":["N\/A"],"award-info":[{"award-number":["N\/A"]}],"id":[{"id":"10.13039\/501100023699","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"<jats:sec id=\"sec041\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>The ability to reproduce the work of others is an essential part of the scientific disciplines. Replicating observational studies using electronic health record (EHR) data can be challenging due to complexities in data access, variations in EHR systems across institutions, and the potential for unaccounted confounding variables. Our aim is to identify the barriers to methods reproducibility for replication studies using EHR data.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec042\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We replicated a study that examined the risk of hospitalisation following a positive COVID-19 test in individuals with diabetes. Using EHR data from the NHS England\u2019s Secure Data Environment (SDE) covering the whole of England, UK (population 57m), we sought to replicate findings from the original study, which used data from Greater Manchester (a large urban region in the UK, population 2.9m). Both analyses were conducted in Trusted Research Environments (TREs) or SDEs, containing linked primary and secondary care data, however methods reproducibility was not straightforward. Differences between the environments that contributed to the difficulties were documented, categorized into themes, and converted into a list of recommendations for TRE\/SDEs.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec043\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Small differences between the environments and the data sources led to several challenges in methods reproducibility. Our recommendations of TRE\/SDEs should facilitate future replication studies. The recommendations include: a need for improved machine-readable metadata for EHR data; standardization of governance processes to facilitate federated analysis; mandating of code sharing; and for environments to have a support structure for data engineers and analysts. We also propose a new theme for research, \u201cdata reproducibility\u201d, as the ability to prepare, extract and clean data from a different database for a replication study.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec044\">\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Even with perfect code sharing, data reproducibility remains a challenge. Our recommendations have the potential to reduce the barriers to replication studies and therefore enhance the potential of observational studies using EHR data.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1371\/journal.pone.0326335","type":"journal-article","created":{"date-parts":[[2025,7,18]],"date-time":"2025-07-18T17:32:46Z","timestamp":1752859966000},"page":"e0326335","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":3,"title":["The challenges of replication: A worked example of methods reproducibility using electronic health record data"],"prefix":"10.1371","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0920-1103","authenticated-orcid":true,"given":"Richard","family":"Williams","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"Bolton","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Jenkins","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mehrdad A.","family":"Mizani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew","family":"Sperrin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7725-7520","authenticated-orcid":true,"given":"Cathie","family":"Sudlow","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Angela","family":"Wood","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adrian","family":"Heald","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Niels","family":"Peek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"name":"on behalf of the CVD-COVID-UK\/COVID-IMPACT Consortium","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2025,7,18]]},"reference":[{"issue":"7604","key":"pone.0326335.ref001","doi-asserted-by":"crossref","first-page":"452","DOI":"10.1038\/533452a","article-title":"1,500 scientists lift the lid on reproducibility","volume":"533","author":"M Baker","year":"2016","journal-title":"Nature"},{"issue":"8","key":"pone.0326335.ref002","doi-asserted-by":"crossref","first-page":"963","DOI":"10.1093\/jamia\/ocy028","article-title":"Does health informatics have a replication crisis?","volume":"25","author":"E Coiera","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"2128","key":"pone.0326335.ref003","article-title":"Improving reproducibility by using high-throughput observational studies with empirical calibration","volume":"376","author":"MJ Schuemie","year":"2018","journal-title":"Philos Trans A Math Phys Eng Sci"},{"issue":"4","key":"pone.0326335.ref004","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1093\/aje\/kwt010","article-title":"Evaluating the impact of database heterogeneity on observational study results","volume":"178","author":"D Madigan","year":"2013","journal-title":"Am J Epidemiol"},{"key":"pone.0326335.ref005","first-page":"96","article-title":"What does research reproducibility mean?","volume-title":"Getting to Good Research Integrity Biomedical Science","author":"SN Goodman","year":"2018"},{"issue":"5","key":"pone.0326335.ref006","doi-asserted-by":"crossref","first-page":"1007","DOI":"10.1007\/s13300-022-01230-2","article-title":"The risk factors potentially influencing hospital admission in people with diabetes, following SARS-CoV-2 infection: a population-level analysis","volume":"13","author":"AH Heald","year":"2022","journal-title":"Diabetes Ther"},{"issue":"4","key":"pone.0326335.ref007","doi-asserted-by":"crossref","DOI":"10.1136\/bmjopen-2024-093080","article-title":"Replicating a COVID-19 study in a national England database to assess the generalisability of research with regional electronic health record data","volume":"15","author":"R Williams","year":"2025","journal-title":"BMJ Open"},{"key":"pone.0326335.ref008","author":"R Core Team","year":"2022"},{"key":"pone.0326335.ref009","volume-title":"dplyr: A Grammar of Data Manipulation","author":"H Wickham","year":"2022"},{"issue":"2","key":"pone.0326335.ref010","doi-asserted-by":"crossref","first-page":"142","DOI":"10.32614\/RJ-2017-037","article-title":"Furniture for quantitative scientists","volume":"9","author":"TS Barrett","year":"2017","journal-title":"R J"},{"key":"pone.0326335.ref011","author":"NHS Digital"},{"key":"pone.0326335.ref012","volume-title":"Better, broader, safer: using health data for research and analysis","author":"B Goldacre","year":"2022"},{"key":"pone.0326335.ref013","author":"N. England"},{"key":"pone.0326335.ref014","volume-title":"SATRE: Standardised Architecture for Trusted Research Environments","author":"DC Cole","year":"2023"},{"key":"pone.0326335.ref015","author":"DARE UK"},{"key":"pone.0326335.ref016","author":"DARE UK"},{"key":"pone.0326335.ref017","author":"SAIL"},{"key":"pone.0326335.ref018","first-page":"66","article-title":"Can researchers assess the suitability of datasets to answer their research questions, with access to metadata only?","volume":"290","author":"G Tilston","year":"2022","journal-title":"Stud Health Technol Inform"},{"issue":"6","key":"pone.0326335.ref019","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0099825","article-title":"ClinicalCodes: an online clinical codes repository to improve the validity and reproducibility of research using electronic medical records","volume":"9","author":"DA Springate","year":"2014","journal-title":"PLoS One"},{"key":"pone.0326335.ref020","author":"HDRUK","year":"2021"},{"key":"pone.0326335.ref021","author":"P Sefton","year":"2023"},{"issue":"2","key":"pone.0326335.ref022","doi-asserted-by":"crossref","first-page":"97","DOI":"10.3233\/DS-210053","article-title":"Packaging research artefacts with RO-Crate","volume":"5","author":"S Soiland-Reyes","year":"2022","journal-title":"Data Sci"},{"issue":"9","key":"pone.0326335.ref023","doi-asserted-by":"crossref","first-page":"1105","DOI":"10.1007\/s00228-016-2071-8","article-title":"A methodological comparison of two European primary care databases and replication in a US claims database: inhaled long-acting beta-2-agonists and the risk of acute myocardial infarction","volume":"72","author":"A Afonso","year":"2016","journal-title":"Eur J Clin Pharmacol"},{"key":"pone.0326335.ref024","author":"OpenSAFELY","year":"2021"},{"key":"pone.0326335.ref025","first-page":"1","author":"OHDSI","year":"2016"},{"key":"pone.0326335.ref026","author":"CPRD"}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0326335","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,18]],"date-time":"2025-07-18T17:32:50Z","timestamp":1752859970000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0326335"}},"subtitle":[],"editor":[{"given":"Youssef","family":"El Khatib","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2025,7,18]]},"references-count":26,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2025,7,18]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0326335","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,18]]}}}