{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T09:04:18Z","timestamp":1781773458273,"version":"3.54.5"},"reference-count":40,"publisher":"Oxford University Press (OUP)","issue":"10","license":[{"start":{"date-parts":[[2021,7,27]],"date-time":"2021-07-27T00:00:00Z","timestamp":1627344000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100005205","name":"Janssen Research & Development, LLC","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100005205","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Innovative Medicines Initiative 2 Joint Undertaking","award":["806968"],"award-info":[{"award-number":["806968"]}]},{"name":"European Union\u2019s Horizon 2020 research and innovation program and EFPIA"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,9,18]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>Advances in standardization of observational healthcare data have enabled methodological breakthroughs, rapid global collaboration, and generation of real-world evidence to improve patient outcomes. Standardizations in data structure, such as use of common data models, need to be coupled with standardized approaches for data quality assessment. To ensure confidence in real-world evidence generated from the analysis of real-world data, one must first have confidence in the data itself.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>We describe the implementation of check types across a data quality framework of conformance, completeness, plausibility, with both verification and validation. We illustrate how data quality checks, paired with decision thresholds, can be configured to customize data quality reporting across a range of observational health data sources. We discuss how data quality reporting can become part of the overall real-world evidence generation and dissemination process to promote transparency and build confidence in the resulting output.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The Data Quality Dashboard is an open-source R package that reports potential quality issues in an OMOP CDM instance through the systematic execution and summarization of over 3300 configurable data quality checks.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>Transparently communicating how well common data model-standardized databases adhere to a set of quality measures adds a crucial piece that is currently missing from observational research.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Assessing and improving the quality of our data will inherently improve the quality of the evidence we generate.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocab132","type":"journal-article","created":{"date-parts":[[2021,6,16]],"date-time":"2021-06-16T07:12:01Z","timestamp":1623827521000},"page":"2251-2257","source":"Crossref","is-referenced-by-count":99,"title":["Increasing trust in real-world evidence through evaluation of observational data quality"],"prefix":"10.1093","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2570-2124","authenticated-orcid":false,"given":"Clair","family":"Blacketer","sequence":"first","affiliation":[{"name":"Observational Health Data Analytics, Janssen Research and Development, LLC, Titusville, New Jersey, USA"},{"name":"Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Frank J","family":"Defalco","sequence":"additional","affiliation":[{"name":"Observational Health Data Analytics, Janssen Research and Development, LLC, Titusville, New Jersey, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patrick B","family":"Ryan","sequence":"additional","affiliation":[{"name":"Observational Health Data Analytics, Janssen Research and Development, LLC, Titusville, New Jersey, USA"},{"name":"Department of Biomedical Informatics, Columbia University, New York, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter R","family":"Rijnbeek","sequence":"additional","affiliation":[{"name":"Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2021,7,27]]},"reference":[{"key":"2021091817521952200_ocab132-B1","year":"2018"},{"key":"2021091817521952200_ocab132-B2","year":"2019"},{"key":"2021091817521952200_ocab132-B3","year":"2021"},{"issue":"8","key":"2021091817521952200_ocab132-B4","doi-asserted-by":"crossref","first-page":"969","DOI":"10.1093\/jamia\/ocy032","article-title":"Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data","volume":"25","author":"Reps","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"Suppl 1","key":"2021091817521952200_ocab132-B5","doi-asserted-by":"crossref","first-page":"S59","DOI":"10.1007\/s40264-013-0099-6","article-title":"Empirical performance of a new user cohort method: lessons for developing a risk identification and analysis system","volume":"36","author":"Ryan","year":"2013","journal-title":"Drug Saf"},{"issue":"27","key":"2021091817521952200_ocab132-B6","doi-asserted-by":"crossref","first-page":"7329","DOI":"10.1073\/pnas.1510502113","article-title":"Characterizing treatment pathways at scale using the OHDSI network","volume":"113","author":"Hripcsak","year":"2016","journal-title":"Proc Natl Acad Sci USA"},{"issue":"10211","key":"2021091817521952200_ocab132-B7","doi-asserted-by":"crossref","first-page":"1816","DOI":"10.1016\/S0140-6736(19)32317-7","article-title":"Comprehensive comparative effectiveness and safety of first-line antihypertensive drug classes: a systematic, multinational, large-scale analysis","volume":"394","author":"Suchard","year":"2019","journal-title":"Lancet"},{"key":"2021091817521952200_ocab132-B8","first-page":"1","article-title":"Secondary use of EHR: data quality issues and informatics opportunities","volume":"2010","author":"Botsis","year":"2010","journal-title":"Summit Transl Bioinform"},{"issue":"1","key":"2021091817521952200_ocab132-B9","doi-asserted-by":"crossref","first-page":"202","DOI":"10.1186\/s12911-019-0939-0","article-title":"Feasibility analysis of conducting observational studies with the electronic health record","volume":"19","author":"von Lucadou","year":"2019","journal-title":"BMC Med Inform Decis Mak"},{"issue":"1","key":"2021091817521952200_ocab132-B10","doi-asserted-by":"crossref","first-page":"38","DOI":"10.3414\/ME9132","article-title":"Perspectives for medical informatics. reusing the electronic medical record for clinical research","volume":"48","author":"Prokosch","year":"2009","journal-title":"Methods Inf Med"},{"issue":"1","key":"2021091817521952200_ocab132-B11","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1186\/s13643-019-0971-2","article-title":"Evaluation of interventions to improve electronic health record documentation within the inpatient setting: a protocol for a systematic review","volume":"8","author":"Otero Varela","year":"2019","journal-title":"Syst Rev"},{"issue":"4","key":"2021091817521952200_ocab132-B12","doi-asserted-by":"crossref","first-page":"209","DOI":"10.2515\/therapie\/2013035","article-title":"How can the quality of medical data in pharmacovigilance, pharmacoepidemiology and clinical studies be guaranteed?","volume":"68","author":"Alla","year":"2013","journal-title":"Therapie"},{"issue":"Suppl 11","key":"2021091817521952200_ocab132-B13","doi-asserted-by":"crossref","first-page":"S79","DOI":"10.7326\/M19-0873","article-title":"Research and reporting considerations for observational studies using electronic health record data","volume":"172","author":"Callahan","year":"2020","journal-title":"Ann Intern Med"},{"issue":"6","key":"2021091817521952200_ocab132-B14","doi-asserted-by":"crossref","first-page":"498","DOI":"10.1056\/NEJMp1014427","article-title":"Developing the Sentinel System\u2013a national resource for evidence development","volume":"364","author":"Behrman","year":"2011","journal-title":"N Engl J Med"},{"issue":"3","key":"2021091817521952200_ocab132-B15","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1002\/cpt.320","article-title":"The FDA\u2019s sentinel initiative\u2013a comprehensive approach to medical product surveillance","volume":"99","author":"Ball","year":"2016","journal-title":"Clin Pharmacol Ther"},{"issue":"4","key":"2021091817521952200_ocab132-B16","doi-asserted-by":"crossref","first-page":"576","DOI":"10.1136\/amiajnl-2014-002864","article-title":"PCORnet: turning a dream into reality","volume":"21","author":"Collins","year":"2014","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2021091817521952200_ocab132-B17","first-page":"3","article-title":"Evaluating foundational data quality in the national patient-centered clinical research network (PCORnet\u00ae)","volume":"6","author":"Qualls","year":"2018","journal-title":"EGEMS (Wash DC)"},{"issue":"4","key":"2021091817521952200_ocab132-B18","doi-asserted-by":"crossref","first-page":"602","DOI":"10.1136\/amiajnl-2014-002743","article-title":"PEDSnet: a national pediatric learning health system","volume":"21","author":"Forrest","year":"2014","journal-title":"J Am Med Inform Assoc"},{"key":"2021091817521952200_ocab132-B19","author":"Utidjian","year":"2015"},{"issue":"1","key":"2021091817521952200_ocab132-B20","first-page":"1239","article-title":"Multisite evaluation of a data quality tool for patient-level clinical data sets","volume":"4","author":"Huser","year":"2016","journal-title":"EGEMS (Wash DC)"},{"issue":"1","key":"2021091817521952200_ocab132-B21","first-page":"8","article-title":"A comparison of data quality assessment checks in six data sharing networks","volume":"5","author":"Callahan","year":"2017","journal-title":"EGEMS (Wash DC)"},{"key":"2021091817521952200_ocab132-B22","first-page":"17","article-title":"Renin-angiotensin system blockers and susceptibility to COVID-19: an international, open science, cohort analysis","author":"Morales","year":"2020","journal-title":"Lancet Digit Health"},{"issue":"8","key":"2021091817521952200_ocab132-B23","doi-asserted-by":"crossref","first-page":"1268","DOI":"10.1093\/jamia\/ocaa124","article-title":"Large-scale evidence generation and evaluation across a network of databases (LEGEND): assessing validity using hypertension as a case study","volume":"27","author":"Schuemie","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2021091817521952200_ocab132-B24","first-page":"27","article-title":"Baseline phenotype and 30-day outcomes of people tested for COVID-19: an international network cohort including &gt;3.32 million people tested with real-time PCR and &gt;219,000 tested positive for SARS-CoV-2 in South Korea, Spain, and the United States","author":"Golozar","year":"2020","journal-title":"MedRxiv Prepr Serv Health Sci"},{"key":"2021091817521952200_ocab132-B25","author":"Schuemie","year":"2020"},{"key":"2021091817521952200_ocab132-B26","author":"SNOMED","year":"2021"},{"key":"2021091817521952200_ocab132-B27","year":"2021"},{"issue":"1","key":"2021091817521952200_ocab132-B28","first-page":"1244","article-title":"A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data","volume":"4","author":"Kahn","year":"2016","journal-title":"EGEMS (Wash DC)"},{"key":"2021091817521952200_ocab132-B29","author":"Schuemie","year":"2020"},{"key":"2021091817521952200_ocab132-B30","author":"Chang","year":"2020"},{"key":"2021091817521952200_ocab132-B31","author":"Blacketer","year":"2021"},{"key":"2021091817521952200_ocab132-B32","volume-title":"R: A Language and Environment for Statistical Computing","year":"2013"},{"key":"2021091817521952200_ocab132-B33","article-title":"Scientists are drowning in COVID-19 articles. Can new tools keep them afloat?","author":"Brainard","year":"2020","journal-title":"Sci AAAS"},{"key":"2021091817521952200_ocab132-B34","year":"2021"},{"issue":"3","key":"2021091817521952200_ocab132-B35","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1093\/jamia\/ocaa196","article-title":"The National COVID Cohort Collaborative (N3C): rationale, design, infrastructure, and deployment","volume":"28","author":"Haendel","year":"2021","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2021091817521952200_ocab132-B36","first-page":"36","article-title":"Design and refinement of a data quality assessment workflow for a large pediatric research network","volume":"7","author":"Khare","year":"2019","journal-title":"EGEMS (Wash DC)"},{"key":"2021091817521952200_ocab132-B37","year":"2019"},{"issue":"2","key":"2021091817521952200_ocab132-B38","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1001\/jamapediatrics.2020.5052","article-title":"Assessment of 135\u202f794 pediatric patients tested for severe acute respiratory syndrome coronavirus 2 across the United States","volume":"175","author":"Bailey","year":"2021","journal-title":"JAMA Pediatr"},{"issue":"5","key":"2021091817521952200_ocab132-B39","doi-asserted-by":"crossref","first-page":"e00637","DOI":"10.1002\/prp2.637","article-title":"Diabetes medication regimens and patient clinical characteristics in the national patient-centered clinical research network, PCORnet","volume":"8","author":"Bachmann","year":"2020","journal-title":"Pharmacol Res Perspect"},{"key":"2021091817521952200_ocab132-B40","doi-asserted-by":"crossref","DOI":"10.1093\/jncics\/pkab009","article-title":"Risk of non-melanoma skin cancer in association with use of hydrochlorothiazide-containing products in the United States","author":"Eworuke","year":"2021","journal-title":"JNCI Cancer Spectr"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/10\/2251\/40408800\/ocab132.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/10\/2251\/40408800\/ocab132.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,18]],"date-time":"2021-09-18T13:52:47Z","timestamp":1631973167000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/28\/10\/2251\/6328963"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,27]]},"references-count":40,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2021,7,27]]},"published-print":{"date-parts":[[2021,9,18]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocab132","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.03.25.21254341","asserted-by":"object"}]},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,10,1]]},"published":{"date-parts":[[2021,7,27]]}}}