{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,4]],"date-time":"2026-04-04T19:19:38Z","timestamp":1775330378207,"version":"3.50.1"},"reference-count":40,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2017,10,24]],"date-time":"2017-10-24T00:00:00Z","timestamp":1508803200000},"content-version":"vor","delay-in-days":1,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000030","name":"CDC","doi-asserted-by":"publisher","award":["200-2015-87699"],"award-info":[{"award-number":["200-2015-87699"]}],"id":[{"id":"10.13039\/100000030","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000030","name":"CDC","doi-asserted-by":"publisher","award":["NIH R01-HG009174"],"award-info":[{"award-number":["NIH R01-HG009174"]}],"id":[{"id":"10.13039\/100000030","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"NLM","doi-asserted-by":"publisher","award":["T15LM007092"],"award-info":[{"award-number":["T15LM007092"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2018,1,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>To provide an open source, interoperable, and scalable data quality assessment tool for evaluation and visualization of completeness and conformance in electronic health record (EHR) data repositories.<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and Methods<\/jats:title><jats:p>This article describes the tool\u2019s design and architecture and gives an overview of its outputs using a sample dataset of 200\u2009000 randomly selected patient records with an encounter since January 1, 2010, extracted from the Research Patient Data Registry (RPDR) at Partners HealthCare. All the code and instructions to run the tool and interpret its results are provided in the Supplementary Appendix.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>DQe-c produces a web-based report that summarizes data completeness and conformance in a given EHR data repository through descriptive graphics and tables. Results from running the tool on the sample RPDR data are organized into 4 sections: load and test details, completeness test, data model conformance test, and test of missingness in key clinical indicators.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>Open science, interoperability across major clinical informatics platforms, and scalability to large databases are key design considerations for DQe-c. Iterative implementation of the tool across different institutions directed us to improve the scalability and interoperability of the tool and find ways to facilitate local setup.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>EHR data quality assessment has been hampered by implementation of ad hoc processes. The architecture and implementation of DQe-c offer valuable insights for developing reproducible and scalable data science tools to assess, manage, and process data in clinical data repositories.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocx109","type":"journal-article","created":{"date-parts":[[2017,9,15]],"date-time":"2017-09-15T11:08:27Z","timestamp":1505473707000},"page":"17-24","source":"Crossref","is-referenced-by-count":23,"title":["Exploring completeness in clinical data research networks with DQe-c"],"prefix":"10.1093","volume":"25","author":[{"given":"Hossein","family":"Estiri","sequence":"first","affiliation":[{"name":"Harvard Medical School"},{"name":"Massachusetts General Hospital"},{"name":"Partners HealthCare, Boston, MA, USA"}]},{"given":"Kari A","family":"Stephens","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Medical Education"},{"name":"Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA"}]},{"given":"Jeffrey G","family":"Klann","sequence":"additional","affiliation":[{"name":"Harvard Medical School"},{"name":"Massachusetts General Hospital"},{"name":"Partners HealthCare, Boston, MA, USA"}]},{"given":"Shawn N","family":"Murphy","sequence":"additional","affiliation":[{"name":"Harvard Medical School"},{"name":"Massachusetts General Hospital"},{"name":"Partners HealthCare, Boston, MA, USA"}]}],"member":"286","published-online":{"date-parts":[[2017,10,23]]},"reference":[{"issue":"9","key":"2020110612454451300_ocx109-B1","first-page":"1","article-title":"Electronic health record systems and intent to apply for meaningful use incentives among office-based physician practices: United States, 2001-2011","author":"Hsiao","year":"2011","journal-title":"NCHS Data Brief."},{"issue":"13","key":"2020110612454451300_ocx109-B2","doi-asserted-by":"crossref","first-page":"1351","DOI":"10.1001\/jama.2013.393","article-title":"The inevitable application of big data to health care","volume":"309","author":"Murdoch","year":"2013","journal-title":"J Am Med Inform Assoc."},{"key":"2020110612454451300_ocx109-B3","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1016\/j.ijmedinf.2012.10.001","article-title":"Towards an ontology for data quality in integrated chronic disease management: a realist review of the literature","volume":"82","author":"Liaw","year":"2013","journal-title":"Int J Med Inform."},{"key":"2020110612454451300_ocx109-B4","first-page":"97","article-title":"\u201cBig data\u201d and the electronic health record","volume":"9","author":"Ross","year":"2014","journal-title":"Yearb Med Inform."},{"key":"2020110612454451300_ocx109-B5","first-page":"277","article-title":"Adding value to the electronic health record through secondary use of data for quality assurance, research, and surveillance","volume":"13","author":"Hersh","year":"2007","journal-title":"Am J Manag Care."},{"key":"2020110612454451300_ocx109-B6","doi-asserted-by":"crossref","first-page":"144","DOI":"10.1136\/amiajnl-2011-000681","article-title":"Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research","volume":"20","author":"Weiskopf","year":"2013","journal-title":"J Am Med Inform Assoc."},{"issue":"2","key":"2020110612454451300_ocx109-B7","doi-asserted-by":"crossref","first-page":"w181","DOI":"10.1377\/hlthaff.26.2.w181","article-title":"Bridging the inferential gap: the electronic health record and clinical evidence","volume":"26","author":"Stewart","year":"2007","journal-title":"Health Aff"},{"key":"2020110612454451300_ocx109-B8","doi-asserted-by":"crossref","first-page":"578","DOI":"10.1136\/amiajnl-2014-002747","article-title":"Launching PCORnet, a national patient-centered clinical research network","volume":"21","author":"Fleurence","year":"2014","journal-title":"J Am Med Inform Assoc."},{"key":"2020110612454451300_ocx109-B9","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1186\/1755-8794-4-13","article-title":"The eMERGE Network: a consortium of biorepositories linked to electronic medical records data for conducting genomic studies","volume":"4","author":"McCarty","year":"2011","journal-title":"BMC Med Genomics."},{"key":"2020110612454451300_ocx109-B10","first-page":"57","article-title":"LC Data QUEST: a technical architecture for community federated clinical data sharing","volume":"2012","author":"Stephens","year":"2012","journal-title":"AMIA Summits Transl Sci Proc."},{"issue":"6","key":"2020110612454451300_ocx109-B11","first-page":"1009","article-title":"Data science and informatics: when it comes to biomedical data, is there a real distinction? J Am Med Inform Assoc","volume":"20","author":"Ohno-Machado","journal-title":"2013"},{"issue":"6","key":"2020110612454451300_ocx109-B12","doi-asserted-by":"crossref","first-page":"1126","DOI":"10.1093\/jamia\/ocv077","article-title":"Big biomedical data as the key resource for discovery science","volume":"22","author":"Toga","year":"2015","journal-title":"J Am Med Inform Assoc."},{"key":"2020110612454451300_ocx109-B13","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1016\/j.jbi.2017.03.017","article-title":"Envisioning the future of \u201cbig data\u201d biomedicine","volume":"69","author":"Bui","year":"2017","journal-title":"J Biomed Inform."},{"issue":"6","key":"2020110612454451300_ocx109-B14","doi-asserted-by":"crossref","first-page":"957","DOI":"10.1136\/amiajnl-2014-002974","article-title":"The National Institutes of Health\u2019s Big Data to Knowledge (BD2K) initiative: capitalizing on biomedical big data","volume":"21","author":"Margolis","year":"2014","journal-title":"J Am Med Inform Assoc."},{"issue":"6","key":"2020110612454451300_ocx109-B15","doi-asserted-by":"crossref","first-page":"1114","DOI":"10.1093\/jamia\/ocv136","article-title":"The NIH Big Data to Knowledge (BD2K) initiative","volume":"22","author":"Bourne","year":"2015","journal-title":"J Am Med Inform Assoc."},{"key":"2020110612454451300_ocx109-B16","doi-asserted-by":"crossref","first-page":"S22","DOI":"10.1097\/MLR.0b013e31829b1e2c","article-title":"Data quality assessment for comparative effectiveness research in distributed data networks","volume":"51","author":"Brown","year":"2013","journal-title":"Med Care."},{"key":"2020110612454451300_ocx109-B17","doi-asserted-by":"crossref","first-page":"S60","DOI":"10.1097\/MLR.0b013e318259bff4","article-title":"Data model considerations for clinical effectiveness researchers","volume":"50","author":"Kahn","year":"2012","journal-title":"Med Care."},{"key":"2020110612454451300_ocx109-B18","doi-asserted-by":"crossref","first-page":"830","DOI":"10.1016\/j.jbi.2013.06.010","article-title":"Defining and measuring completeness of electronic health records for secondary use","volume":"46","author":"Weiskopf","year":"2013","journal-title":"J Biomed Inform."},{"key":"2020110612454451300_ocx109-B19","doi-asserted-by":"crossref","first-page":"456","DOI":"10.1002\/9781119940012.ch23","article-title":"Quality of electronic medical records","volume-title":"Statistical Methods in Healthcare","author":"Gregori","year":"2012"},{"key":"2020110612454451300_ocx109-B20","doi-asserted-by":"crossref","first-page":"S21","DOI":"10.1097\/MLR.0b013e318257dd67","article-title":"A pragmatic framework for single-site and multisite data quality assessment in electronic health record-based clinical research","volume":"50","author":"Kahn","year":"2012","journal-title":"Med Care."},{"key":"2020110612454451300_ocx109-B21","doi-asserted-by":"crossref","first-page":"5170","DOI":"10.3390\/ijerph110505170","article-title":"A review of data quality assessment methods for public health information systems","volume":"11","author":"Chen","year":"2014","journal-title":"Int J Environ Res Public Health."},{"key":"2020110612454451300_ocx109-B22","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1177\/1062860609336627","article-title":"The challenge of measuring quality of care from the electronic health record","volume":"24","author":"Roth","year":"2009","journal-title":"Am J Med Qual."},{"key":"2020110612454451300_ocx109-B23","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1093\/fampra\/cmn047","article-title":"Accuracy and completeness of electronic patient records in primary care","volume":"25","author":"Majeed","year":"2008","journal-title":"Fam Pract."},{"key":"2020110612454451300_ocx109-B24","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1016\/j.canep.2014.02.013","article-title":"Control of data quality for population-based cancer survival analysis","volume":"38","author":"Li","year":"2014","journal-title":"Cancer Epidemiol."},{"issue":"1","key":"2020110612454451300_ocx109-B25","first-page":"1244","article-title":"A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data","volume":"4","author":"Kahn","year":"2016","journal-title":"EGEMS (Wash DC)."},{"key":"2020110612454451300_ocx109-B26","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1177\/1077558709359007","article-title":"Review: electronic health records and the reliability and validity of quality measures: a review of the literature","volume":"67","author":"Chan","year":"2010","journal-title":"Med Care Res Rev."},{"key":"2020110612454451300_ocx109-B27","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1177\/1460458208096555","article-title":"The strategic management of data quality in healthcare","volume":"14","author":"Kerr","year":"2008","journal-title":"Health Informatics J."},{"key":"2020110612454451300_ocx109-B28","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1007\/s40264-013-0098-7","article-title":"Managing data quality for a drug safety surveillance system","volume":"36","author":"Hartzema","year":"2013","journal-title":"Drug Saf."},{"key":"2020110612454451300_ocx109-B29","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1197\/jamia.M1087","article-title":"Defining and improving data quality in medical registries: a literature review, case study, and generic framework","volume":"9","author":"Arts","year":"2002","journal-title":"J Am Med Inform Assoc."},{"key":"2020110612454451300_ocx109-B30","doi-asserted-by":"crossref","first-page":"342","DOI":"10.1136\/jamia.1997.0040342","article-title":"Accuracy of data in computer-based patient records","volume":"4","author":"Hogan","year":"1997","journal-title":"J Am Med Inform Assoc."},{"issue":"2","key":"2020110612454451300_ocx109-B31","first-page":"1206","article-title":"Extracting electronic health record data in a practice-based research network: processes to support translational research across diverse practice organizations","volume":"4","author":"Cole","year":"2016","journal-title":"EGEMS (Wash DC)."},{"key":"2020110612454451300_ocx109-B32","first-page":"548","article-title":"Architecture of the open-source clinical research chart from Informatics for Integrating Biology and the Bedside","author":"Murphy","year":"2007","journal-title":"AMIA Annu Symp Proc."},{"issue":"2","key":"2020110612454451300_ocx109-B33","doi-asserted-by":"crossref","first-page":"124","DOI":"10.1136\/jamia.2009.000893","article-title":"Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2)","volume":"17","author":"Murphy","year":"2010","journal-title":"J Am Med Inform Assoc."},{"issue":"4","key":"2020110612454451300_ocx109-B34","doi-asserted-by":"crossref","first-page":"615","DOI":"10.1136\/amiajnl-2014-002727","article-title":"Scalable collaborative infrastructure for a learning healthcare system (SCILHS): architecture","volume":"21","author":"Mandl","year":"2014","journal-title":"J Am Med Inform Assoc."},{"key":"2020110612454451300_ocx109-B35","first-page":"1044","article-title":"Calculating the benefits of a research patient data repository","author":"Nalichowski","year":"2006","journal-title":"AMIA Annu Symp Proc"},{"issue":"11","key":"2020110612454451300_ocx109-B36","first-page":"779","article-title":"Open code for open science? Nat Geosci","volume":"7","author":"Easterbrook","journal-title":"2014"},{"key":"2020110612454451300_ocx109-B37","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1016\/j.ijmedinf.2016.05.008","article-title":"Implementing partnership-driven clinical federated electronic health record data sharing networks","volume":"93","author":"Stephens","year":"2016","journal-title":"Int J Med Inform."},{"issue":"2","key":"2020110612454451300_ocx109-B38","doi-asserted-by":"crossref","first-page":"1063","DOI":"10.13063\/2327-9214.1063","article-title":"The DARTNet Institute: seeking a sustainable support mechanism for electronic data enabled research networks","volume":"2","author":"Pace","year":"2014","journal-title":"EGEMS."},{"key":"2020110612454451300_ocx109-B39","first-page":"574","article-title":"Observational health data sciences and informatics (OHDSI): opportunities for observational researchers","volume":"216","author":"Hripcsak","year":"2015","journal-title":"Stud Health Technol Inform."},{"issue":"5","key":"2020110612454451300_ocx109-B40","doi-asserted-by":"crossref","first-page":"909","DOI":"10.1093\/jamia\/ocv188","article-title":"Data interchange using i2b2","volume":"23","author":"Klann","year":"2016","journal-title":"J Am Med Inform Assoc."}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/25\/1\/17\/34149518\/ocx109.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/25\/1\/17\/34149518\/ocx109.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,25]],"date-time":"2023-08-25T22:51:01Z","timestamp":1693003861000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/25\/1\/17\/4562678"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,10,23]]},"references-count":40,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2017,10,23]]},"published-print":{"date-parts":[[2018,1,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocx109","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2018,1]]},"published":{"date-parts":[[2017,10,23]]}}}