{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T20:25:19Z","timestamp":1790367919078,"version":"4.1.0"},"reference-count":32,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2021,12,16]],"date-time":"2021-12-16T00:00:00Z","timestamp":1639612800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,4,13]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Given that electronic clinical quality measures (eCQMs) are playing a central role in quality improvement applications nationwide, a stronger evidence base demonstrating their reliability is critically needed. To assess the reliability of electronic health record-extracted data elements and measure results for the Elective Delivery and Exclusive Breast Milk Feeding measures (vs manual abstraction) among a national sample of US acute care hospitals, as well as common sources of discrepancies and change over time.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>eCQM and chart-abstracted data for the same patients were matched and compared at the data element and measure level for hospitals submitting both sources of data to The Joint Commission between 2017 and 2019. Sensitivity, specificity, and kappa statistics were used to assess reliability.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Although eCQM denominator reliability had moderate to substantial agreement for both measures and both improved over time (Elective Delivery: kappa\u2009=\u20090.59 [95% confidence interval (CI), 0.58\u20130.61] in 2017 and 0.84 [95% CI, 083\u20130.85] in 2019; Exclusive Breast Milk Feeding: kappa\u2009=\u20090.58 [95% CI, 0.54\u20130.62] in 2017 and 0.70 [95% CI, 0.67\u20130.73] in 2019), the numerator status reliability was poor for Elective Delivery (kappa\u2009=\u20090.08 [95% CI, 0.03\u20130.12] in 2017 and 0.10 [95% CI, 0.05\u20130.15] in 2019) but near perfect for Exclusive Breast Milk Feeding (kappa\u2009=\u20090.85 [0.83, 0.87] in 2017 and 0.84 [0.83, 0.85] in 2019). The failure of the eCQM to accurately capture estimated gestational age, conditions possibly justifying elective delivery, active labor, and medical induction were the main reasons for the discrepancies.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>Although eCQM denominator reliability for the Elective Delivery and Exclusive Breast Milk Feeding measures had moderate agreement when compared to medical record review, the numerator status reliability was poor for Elective Delivery, but near perfect for Exclusive Breast Milk Feeding. Improvements in eCQM data capture of some key data elements would greatly improve the reliability.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocab276","type":"journal-article","created":{"date-parts":[[2021,12,1]],"date-time":"2021-12-01T20:21:30Z","timestamp":1638390090000},"page":"789-797","source":"Crossref","is-referenced-by-count":2,"title":["Comparison of electronic versus manual abstraction for 2 standardized perinatal care measures"],"prefix":"10.1093","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2486-3268","authenticated-orcid":false,"given":"Stephen","family":"Schmaltz","sequence":"first","affiliation":[{"name":"Department of Research, The Joint Commission, Division of Healthcare Quality Evaluation, Oakbrook Terrace, Illinois, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jocelyn","family":"Vaughn","sequence":"additional","affiliation":[{"name":"Division of Epidemiology and Biostatistics, The University of Illinois Chicago School of Public Health, Chicago, Illinois, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tricia","family":"Elliott","sequence":"additional","affiliation":[{"name":"Department of Measurement Science and Application, National Quality Forum, Washington, District of Columbia, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2021,12,16]]},"reference":[{"key":"2022041311584754100_ocab276-B1"},{"issue":"4","key":"2022041311584754100_ocab276-B2","doi-asserted-by":"crossref","first-page":"226","DOI":"10.1136\/bmjqs-2015-004486","article-title":"Measuring and improving patient safety through health information technology: the Health IT Safety Framework","volume":"25","author":"Singh","year":"2016","journal-title":"BMJ Qual Saf"},{"issue":"3","key":"2022041311584754100_ocab276-B3","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1093\/intqhc\/mzs011","article-title":"New paradigms for measuring clinical performance using electronic health records","volume":"24","author":"Weiner","year":"2012","journal-title":"Int J Qual Health Care"},{"issue":"2","key":"2022041311584754100_ocab276-B4","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1016\/j.pop.2012.03.006","article-title":"Health information technology: transforming chronic disease management and care transitions","volume":"39","author":"Rao","year":"2012","journal-title":"Prim Care"},{"issue":"1","key":"2022041311584754100_ocab276-B5","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1136\/amiajnl-2013-001789","article-title":"E-measures: insight into the challenges and opportunities of automating publicly reported quality measures","volume":"21","author":"Garrido","year":"2014","journal-title":"J Am Med Inform Assoc"},{"key":"2022041311584754100_ocab276-B6","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1186\/s12887-016-0592-z","article-title":"Quality of EHR data extractions for studies of preterm birth in a tertiary care center: guidelines for obtaining reliable data","volume":"16","author":"Knake","year":"2016","journal-title":"BMC Pediatr"},{"issue":"2","key":"2022041311584754100_ocab276-B7","doi-asserted-by":"crossref","first-page":"77","DOI":"10.7326\/0003-4819-158-2-201301150-00001","article-title":"Accuracy of electronically reported \u201cmeaningful use\u201d clinical quality measures: a cross-sectional study","volume":"158","author":"Kern","year":"2013","journal-title":"Ann Intern Med"},{"key":"2022041311584754100_ocab276-B8"},{"issue":"6","key":"2022041311584754100_ocab276-B9","doi-asserted-by":"crossref","first-page":"697","DOI":"10.1093\/intqhc\/mzr056","article-title":"The perceived impact of public reporting hospital performance data: interviews with hospital staff","volume":"23","author":"Hafner","year":"2011","journal-title":"Int J Qual Health Care"},{"issue":"2","key":"2022041311584754100_ocab276-B10","doi-asserted-by":"crossref","first-page":"409","DOI":"10.1136\/amiajnl-2014-002865","article-title":"Completeness, accuracy, and computability of National Quality Forum-specified eMeasures","volume":"22","author":"Amster","year":"2015","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2022041311584754100_ocab276-B11","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1055\/s-0038-1627475","article-title":"The reliability of electronic health record data used for obstetrical research","volume":"9","author":"Altman","year":"2018","journal-title":"Appl Clin Inform"},{"issue":"9","key":"2022041311584754100_ocab276-B12","doi-asserted-by":"crossref","DOI":"10.1161\/CIRCOUTCOMES.116.003157","article-title":"Development and validation of electronic quality measures to assess care for patients with transient ischemic attack and minor ischemic stroke","volume":"10","author":"Bravata","year":"2017","journal-title":"Circ Cardiovasc Qual Outcomes"},{"issue":"4","key":"2022041311584754100_ocab276-B13","doi-asserted-by":"crossref","first-page":"1226","DOI":"10.1111\/1475-6773.12159","article-title":"Using computer-extracted data from electronic health records to measure the quality of adolescent well-care","volume":"49","author":"Gardner","year":"2014","journal-title":"Health Serv Res"},{"issue":"6","key":"2022041311584754100_ocab276-B14","first-page":"313","article-title":"Automating care quality measurement with health information technology","volume":"18","author":"Hazelhurst","year":"2012","journal-title":"Am J Manag Care"},{"issue":"7","key":"2022041311584754100_ocab276-B15","doi-asserted-by":"crossref","first-page":"626","DOI":"10.1097\/MLR.0000000000000916","article-title":"In data we trust? Comparison of electronic versus manual abstraction of antimicrobial prescribing quality metrics for hospitalized veterans with pneumonia","volume":"56","author":"Jones","year":"2018","journal-title":"Med Care"},{"issue":"3","key":"2022041311584754100_ocab276-B16","first-page":"1102","article-title":"Evaluating the reliability of EHR-generated clinical outcomes reports: a case study","volume":"2","author":"Kanger","year":"2014","journal-title":"EGEMS (Wash DC)"},{"issue":"4","key":"2022041311584754100_ocab276-B17","doi-asserted-by":"crossref","first-page":"604","DOI":"10.1136\/amiajnl-2011-000557","article-title":"Validity of electronic health record-derived quality measurement for performance monitoring","volume":"19","author":"Parsons","year":"2012","journal-title":"J Am Med Inform Assoc"},{"issue":"8","key":"2022041311584754100_ocab276-B18","doi-asserted-by":"crossref","DOI":"10.1161\/JAHA.117.007762","article-title":"Errors in Electronic health record-based data query of statin prescriptions in patients with coronary artery disease in a large, academic, multispecialty clinic practice","volume":"7","author":"Shin","year":"2018","journal-title":"J Am Heart Assoc"},{"issue":"10","key":"2022041311584754100_ocab276-B19","doi-asserted-by":"crossref","first-page":"901","DOI":"10.1097\/MLR.0000000000000418","article-title":"Calculations of financial incentives for providers in a pay-for-performance program: manual review versus data from structured fields in electronic health records","volume":"53","author":"Urech","year":"2015","journal-title":"Med Care"},{"issue":"5","key":"2022041311584754100_ocab276-B20","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1177\/1062860609336627","article-title":"The challenge of measuring quality of care from the electronic health record","volume":"24","author":"Roth","year":"2009","journal-title":"Am J Med Qual"},{"issue":"10","key":"2022041311584754100_ocab276-B21","doi-asserted-by":"crossref","first-page":"2128","DOI":"10.1007\/s10995-015-1725-y","article-title":"The feasibility of tracking elective deliveries prior to 39 gestational weeks: lessons from three California projects","volume":"19","author":"Korst","year":"2015","journal-title":"Matern Child Health J"},{"issue":"24","key":"2022041311584754100_ocab276-B22","doi-asserted-by":"crossref","first-page":"2240","DOI":"10.1056\/NEJMp1203102","article-title":"Escaping the EHR trap\u2014the future of health IT","volume":"366","author":"Mandl","year":"2012","journal-title":"N Engl J Med"},{"issue":"5","key":"2022041311584754100_ocab276-B23","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1177\/1077558709359007","article-title":"Review: electronic health records and the reliability and validity of quality measures: a review of the literature","volume":"67","author":"Chan","year":"2010","journal-title":"Med Care Res Rev"},{"issue":"4","key":"2022041311584754100_ocab276-B24","doi-asserted-by":"crossref","first-page":"e7","DOI":"10.1097\/MLR.0b013e31821d2b1d","article-title":"Quality of data in perinatal population health databases: a systematic review","volume":"50","author":"Lain","year":"2012","journal-title":"Med Care"},{"key":"2022041311584754100_ocab276-B25"},{"key":"2022041311584754100_ocab276-B26"},{"key":"2022041311584754100_ocab276-B27"},{"issue":"1","key":"2022041311584754100_ocab276-B28","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"issue":"4","key":"2022041311584754100_ocab276-B29","doi-asserted-by":"crossref","first-page":"514","DOI":"10.1093\/jamia\/ocz225","article-title":"Measuring implementation feasibility of clinical decision support alerts for clinical practice recommendations","volume":"27","author":"Richesson","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2022041311584754100_ocab276-B30"},{"key":"2022041311584754100_ocab276-B31","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jbi.2016.12.004","article-title":"Single reviewer electronic phenotyping validation in operational settings: comparison of strategies and recommendations","volume":"66","author":"Kukhareva","year":"2017","journal-title":"J Biomed Inform"},{"issue":"4","key":"2022041311584754100_ocab276-B32","doi-asserted-by":"crossref","first-page":"301","DOI":"10.7326\/M14-2128","article-title":"Clinical documentation in the 21st century: executive summary of a policy position paper from the American College of Physicians","volume":"162","author":"Kuhn","year":"2015","journal-title":"Ann Intern Med"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/5\/789\/43372448\/ocab276.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/5\/789\/43372448\/ocab276.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,13]],"date-time":"2022-04-13T12:19:29Z","timestamp":1649852369000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/29\/5\/789\/6464069"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,16]]},"references-count":32,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2021,12,16]]},"published-print":{"date-parts":[[2022,4,13]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocab276","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,5,1]]},"published":{"date-parts":[[2021,12,16]]}}}