{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,4]],"date-time":"2026-04-04T04:14:39Z","timestamp":1775276079574,"version":"3.50.1"},"reference-count":42,"publisher":"Oxford University Press (OUP)","issue":"12","license":[{"start":{"date-parts":[[2020,11,10]],"date-time":"2020-11-10T00:00:00Z","timestamp":1604966400000},"content-version":"vor","delay-in-days":1,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"University of Florida\u2019s Creating the Healthiest Generation\u2014Moonshot"},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"crossref","award":["UL1TR001427"],"award-info":[{"award-number":["UL1TR001427"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"crossref","award":["R01CA246418"],"award-info":[{"award-number":["R01CA246418"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"crossref","award":["R21AG068717"],"award-info":[{"award-number":["R21AG068717"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100006093","name":"Patient-Centered Outcomes Research Institute","doi-asserted-by":"crossref","award":["PCORI ME-2018C3-14754"],"award-info":[{"award-number":["PCORI ME-2018C3-14754"]}],"id":[{"id":"10.13039\/100006093","id-type":"DOI","asserted-by":"crossref"}]},{"name":"OneFlorida Clinical Research Consortium","award":["CDRN-1501-26692"],"award-info":[{"award-number":["CDRN-1501-26692"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,12,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To synthesize data quality (DQ) dimensions and assessment methods of real-world data, especially electronic health records, through a systematic scoping review and to assess the practice of DQ assessment in the national Patient-centered Clinical Research Network (PCORnet).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We started with 3 widely cited DQ literature\u20142 reviews from Chan et al (2010) and Weiskopf et al (2013a) and 1 DQ framework from Kahn et al (2016)\u2014and expanded our review systematically to cover relevant articles published up to February 2020. We extracted DQ dimensions and assessment methods from these studies, mapped their relationships, and organized a synthesized summarization of existing DQ dimensions and assessment methods. We reviewed the data checks employed by the PCORnet and mapped them to the synthesized DQ dimensions and methods.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>We analyzed a total of 3 reviews, 20 DQ frameworks, and 226 DQ studies and extracted 14 DQ dimensions and 10 assessment methods. We found that completeness, concordance, and correctness\/accuracy were commonly assessed. Element presence, validity check, and conformance were commonly used DQ assessment methods and were the main focuses of the PCORnet data checks.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Definitions of DQ dimensions and methods were not consistent in the literature, and the DQ assessment practice was not evenly distributed (eg, usability and ease-of-use were rarely discussed). Challenges in DQ assessments, given the complex and heterogeneous nature of real-world data, exist.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>The practice of DQ assessment is still limited in scope. Future work is warranted to generate understandable, executable, and reusable DQ measures.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocaa245","type":"journal-article","created":{"date-parts":[[2020,9,16]],"date-time":"2020-09-16T19:24:05Z","timestamp":1600284245000},"page":"1999-2010","source":"Crossref","is-referenced-by-count":83,"title":["Assessing the practice of data quality evaluation in a national clinical data research network through a systematic scoping review in the era of real-world data"],"prefix":"10.1093","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2238-5429","authenticated-orcid":false,"given":"Jiang","family":"Bian","sequence":"first","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA"},{"name":"Cancer Informatics Shared Resource, University of Florida Health Cancer Center, Gainesville, Florida, USA"}]},{"given":"Tianchen","family":"Lyu","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA"}]},{"given":"Alexander","family":"Loiacono","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA"}]},{"given":"Tonatiuh Mendoza","family":"Viramontes","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA"}]},{"given":"Gloria","family":"Lipori","sequence":"additional","affiliation":[{"name":"Clinical and Translational Institute, University of Florida, Gainesville, Florida, USA"}]},{"given":"Yi","family":"Guo","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA"}]},{"given":"Yonghui","family":"Wu","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9021-5595","authenticated-orcid":false,"given":"Mattia","family":"Prosperi","sequence":"additional","affiliation":[{"name":"Department of Epidemiology, College of Public Health and Health Professions & College of Medicine, University of Florida, Gainesville, Florida, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6249-9180","authenticated-orcid":false,"suffix":"Jr","given":"Thomas J","family":"George","sequence":"additional","affiliation":[{"name":"Hematology & Oncology, Department of Medicine, College of Medicine, University of Florida, Gainesville, Florida, USA"}]},{"given":"Christopher A","family":"Harle","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA"}]},{"given":"Elizabeth A","family":"Shenkman","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9881-1017","authenticated-orcid":false,"given":"William","family":"Hogan","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA"}]}],"member":"286","published-online":{"date-parts":[[2020,11,9]]},"reference":[{"issue":"4","key":"2020121009244595000_ocaa245-B1","doi-asserted-by":"crossref","first-page":"576","DOI":"10.1136\/amiajnl-2014-002864","article-title":"PCORnet: turning a dream into reality","volume":"21","author":"Collins","year":"2014","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"2020121009244595000_ocaa245-B2","doi-asserted-by":"crossref","first-page":"204","DOI":"10.1200\/JOP.2014.003194","article-title":"Building data infrastructure to evaluate and improve quality: PCORnet","volume":"11","author":"Corley","year":"2015","journal-title":"J Oncol Pract"},{"key":"2020121009244595000_ocaa245-B3"},{"issue":"3","key":"2020121009244595000_ocaa245-B4","doi-asserted-by":"crossref","first-page":"451","DOI":"10.1097\/ACM.0000000000002029","article-title":"OneFlorida Clinical Research Consortium: linking a clinical and translational science institute with a community-based distributive medical education model","volume":"93","author":"Shenkman","year":"2018","journal-title":"Acad Med"},{"key":"2020121009244595000_ocaa245-B5"},{"key":"2020121009244595000_ocaa245-B6"},{"issue":"23","key":"2020121009244595000_ocaa245-B7","doi-asserted-by":"crossref","first-page":"2293","DOI":"10.1056\/NEJMsb1609216","article-title":"Real-world evidence\u2014what is it and what can it tell us?","volume":"375","author":"Sherman","year":"2016","journal-title":"N Engl J Med"},{"key":"2020121009244595000_ocaa245-B8","volume-title":"The Learning Healthcare System: Workshop Summary","author":"Olsen","year":"2007"},{"key":"2020121009244595000_ocaa245-B9","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1016\/j.jbi.2016.09.018","article-title":"The learning healthcare system: where are we now? A systematic review","volume":"64","author":"Budrionis","year":"2016","journal-title":"J Biomed Inform"},{"issue":"3","key":"2020121009244595000_ocaa245-B10","doi-asserted-by":"crossref","first-page":"203","DOI":"10.7326\/0003-4819-151-3-200908040-00125","article-title":"Comparative effectiveness research: a report from the Institute of Medicine","volume":"151","author":"Sox","year":"2009","journal-title":"Ann Intern Med"},{"issue":"5","key":"2020121009244595000_ocaa245-B11","doi-asserted-by":"crossref","first-page":"454","DOI":"10.1056\/NEJMra1510059","article-title":"Pragmatic trials","volume":"375","author":"Ford","year":"2016","journal-title":"N Engl J Med"},{"key":"2020121009244595000_ocaa245-B12","first-page":"1","article-title":"Secondary use of EHR: data quality issues and informatics opportunities","volume":"2010","author":"Botsis","year":"2010","journal-title":"Summit Transl Bioinform"},{"issue":"1","key":"2020121009244595000_ocaa245-B13","doi-asserted-by":"crossref","first-page":"12","DOI":"10.13063\/2327-9214.1125","article-title":"The challenges of data quality evaluation in a joint data warehouse","volume":"3","author":"Bae","year":"2015","journal-title":"eGEMs"},{"issue":"4","key":"2020121009244595000_ocaa245-B14","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1080\/07421222.1996.11518099","article-title":"Beyond accuracy: what data quality means to data consumers","volume":"12","author":"Wang","year":"1996","journal-title":"J Manag Infn Systems"},{"key":"2020121009244595000_ocaa245-B15","author":"McGilvray"},{"issue":"5","key":"2020121009244595000_ocaa245-B16","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1177\/1077558709359007","article-title":"Review: electronic health records and the reliability and validity of quality measures: a review of the literature","volume":"67","author":"Chan","year":"2010","journal-title":"Med Care Res Rev"},{"key":"2020121009244595000_ocaa245-B17","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1007\/978-1-84882-448-5_10","volume-title":"Clinical Research Informatics","author":"Nahm","year":"2012"},{"key":"2020121009244595000_ocaa245-B18","doi-asserted-by":"crossref","first-page":"S21","DOI":"10.1097\/MLR.0b013e318257dd67","article-title":"A pragmatic framework for single-site and multisite data quality assessment in electronic health record-based clinical research","volume":"50","author":"Kahn","year":"2012","journal-title":"Medl Care"},{"issue":"1","key":"2020121009244595000_ocaa245-B19","doi-asserted-by":"crossref","first-page":"144","DOI":"10.1136\/amiajnl-2011-000681","article-title":"Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research","volume":"20","author":"Weiskopf","year":"2013","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2020121009244595000_ocaa245-B20","doi-asserted-by":"crossref","first-page":"830","DOI":"10.1016\/j.jbi.2013.06.010","article-title":"Defining and measuring completeness of electronic health records for secondary use","volume":"46","author":"Weiskopf","year":"2013","journal-title":"J Biomed Inform"},{"issue":"1","key":"2020121009244595000_ocaa245-B21","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1016\/j.ijmedinf.2012.10.001","article-title":"Towards an ontology for data quality in integrated chronic disease management: a realist review of the literature","volume":"82","author":"Liaw","year":"2013","journal-title":"Intl J Med Inform"},{"key":"2020121009244595000_ocaa245-B22","author":"Zozus"},{"key":"2020121009244595000_ocaa245-B23","first-page":"1937","author":"Johnson"},{"key":"2020121009244595000_ocaa245-B24","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1016\/j.compbiomed.2015.09.024","article-title":"Construction of quality-assured infant feeding process of care data repositories: definition and design (Part 1)","volume":"67","author":"Garc\u00ed A-de-Le\u00f3n-Chocano","year":"2015","journal-title":"Comput Biol Med"},{"issue":"1","key":"2020121009244595000_ocaa245-B25","doi-asserted-by":"crossref","first-page":"18","DOI":"10.13063\/2327-9214.1244","article-title":"A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data","volume":"4","author":"Kahn","year":"2016","journal-title":"eGEMs"},{"key":"2020121009244595000_ocaa245-B26","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1016\/j.ijmedinf.2016.03.006","article-title":"Data quality assessment framework to assess electronic medical record data for use in research","volume":"90","author":"Reimer","year":"2016","journal-title":"Int J Med Inform"},{"issue":"6","key":"2020121009244595000_ocaa245-B27","doi-asserted-by":"crossref","first-page":"1072","DOI":"10.1093\/jamia\/ocx033","article-title":"A longitudinal analysis of data quality in a large pediatric data research network","volume":"24","author":"Khare","year":"2017","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"2020121009244595000_ocaa245-B28","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1093\/jamia\/ocx078","article-title":"Assessing the quality of administrative data for research: a framework from the Manitoba Centre for Health Policy","volume":"25","author":"Smith","year":"2018","journal-title":"J Am Med Inform Assoc"},{"key":"2020121009244595000_ocaa245-B29","first-page":"14","article-title":"A data quality assessment guideline for electronic health record data reuse","author":"Weiskopf","year":"2017"},{"key":"2020121009244595000_ocaa245-B30","first-page":"1080","author":"Lee"},{"issue":"5","key":"2020121009244595000_ocaa245-B31","doi-asserted-by":"crossref","first-page":"753","DOI":"10.1177\/0193945916689084","article-title":"Data quality in electronic health records research: quality domains and assessment methods","volume":"40","author":"Feder","year":"2018","journal-title":"West J Nurs Res"},{"issue":"1","key":"2020121009244595000_ocaa245-B32","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1186\/s12911-019-0740-0","article-title":"A basic model for assessing primary health care electronic medical record data quality","volume":"19","author":"Terry","year":"2019","journal-title":"BMC Med Inform Decis Mak"},{"key":"2020121009244595000_ocaa245-B33","first-page":"333","article-title":"Evaluative outcomes in direct extraction and use of EHR data in clinical trials","author":"Nordo","year":"2019","journal-title":"Stud Health Technol Inform"},{"issue":"1","key":"2020121009244595000_ocaa245-B34","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1186\/s12889-019-6709-1","article-title":"Defining & assessing the quality, usability, and utilization of immunization data","volume":"19","author":"Bloland","year":"2019","journal-title":"BMC Public Health"},{"issue":"1","key":"2020121009244595000_ocaa245-B35","doi-asserted-by":"crossref","first-page":"38","DOI":"10.5334\/egems.298","article-title":"Improving a secondary use health data warehouse: proposing a multi-level data quality framework","volume":"7","author":"Henley-Smith","year":"2019","journal-title":"eGEMs"},{"issue":"1","key":"2020121009244595000_ocaa245-B36","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1111\/hir.12249","article-title":"Electronic healthcare records and data quality","volume":"36","author":"Charnock","year":"2019","journal-title":"Health Info Libr J"},{"issue":"1","key":"2020121009244595000_ocaa245-B37","doi-asserted-by":"crossref","first-page":"8","DOI":"10.5334\/egems.223","article-title":"A comparison of data quality assessment checks in six data sharing networks","volume":"5","author":"Callahan","year":"2017","journal-title":"eGEMs"},{"issue":"1","key":"2020121009244595000_ocaa245-B38","doi-asserted-by":"crossref","first-page":"3","DOI":"10.5334\/egems.199","article-title":"Evaluating foundational data quality in the National Patient-Centered Clinical Research Network (PCORnet)","volume":"6","author":"Qualls","year":"2018","journal-title":"eGEMs"},{"key":"2020121009244595000_ocaa245-B39"},{"issue":"5","key":"2020121009244595000_ocaa245-B40","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1145\/253769.253804","article-title":"Data quality in context","volume":"40","author":"Strong","year":"1997","journal-title":"Commun ACM"},{"issue":"16","key":"2020121009244595000_ocaa245-B41","first-page":"1","article-title":"A grounding-based ontology of data quality measures","author":"Mocnik","year":"2018","journal-title":"JOSIS"},{"issue":"1","key":"2020121009244595000_ocaa245-B42","doi-asserted-by":"crossref","first-page":"7","DOI":"10.13063\/2327-9214.1052","article-title":"Transparent reporting of data quality in distributed data networks","volume":"3","author":"Kahn","year":"2015","journal-title":"eGEMs"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/12\/1999\/34838730\/ocaa245.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/12\/1999\/34838730\/ocaa245.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,12,10]],"date-time":"2020-12-10T14:42:57Z","timestamp":1607611377000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/27\/12\/1999\/5964088"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,9]]},"references-count":42,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2020,11,9]]},"published-print":{"date-parts":[[2020,12,9]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaa245","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,12]]},"published":{"date-parts":[[2020,11,9]]}}}