{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,19]],"date-time":"2026-06-19T08:16:54Z","timestamp":1781857014489,"version":"3.54.5"},"reference-count":41,"publisher":"Oxford University Press (OUP)","issue":"10","license":[{"start":{"date-parts":[[2019,4,16]],"date-time":"2019-04-16T00:00:00Z","timestamp":1555372800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,10,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To assess measurement practice in clinical decision support evaluation studies.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We identified empirical studies evaluating clinical decision support systems published from 1998 to 2017. We reviewed titles, abstracts, and full paper contents for evidence of attention to measurement validity, reliability, or reuse. We used Friedman and Wyatt\u2019s typology to categorize the studies.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>There were 391 studies that met the inclusion criteria. Study types in this cohort were primarily field user effect studies (n\u2009=\u2009210) or problem impact studies (n\u2009=\u2009150). Of those, 280 studies (72%) had no evidence of attention to measurement methodology, and 111 (28%) had some evidence with 33 (8%) offering validity evidence; 45 (12%) offering reliability evidence; and 61 (16%) reporting measurement artefact reuse.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Only 5 studies offered validity assessment within the study. Valid measures were predominantly observed in problem impact studies with the majority of measures being clinical or patient reported outcomes with validity measured elsewhere.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>Measurement methodology is frequently ignored in empirical studies of clinical decision support systems and particularly so in field user effect studies. Authors may in fact be attending to measurement considerations and not reporting this or employing methods of unknown validity and reliability in their studies. In the latter case, reported study results may be biased and effect sizes misleading. We argue that replication studies to strengthen the evidence base require greater attention to measurement practice in health informatics research.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocz035","type":"journal-article","created":{"date-parts":[[2019,3,8]],"date-time":"2019-03-08T12:36:53Z","timestamp":1552048613000},"page":"1120-1128","source":"Crossref","is-referenced-by-count":18,"title":["A review of measurement practice in studies of clinical decision support systems 1998\u20132017"],"prefix":"10.1093","volume":"26","author":[{"given":"Philip J","family":"Scott","sequence":"first","affiliation":[{"name":"Centre for Healthcare Modelling and Informatics, University of Portsmouth, Portsmouth, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Angela W","family":"Brown","sequence":"additional","affiliation":[{"name":"Centre for Healthcare Modelling and Informatics, University of Portsmouth, Portsmouth, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Taiwo","family":"Adedeji","sequence":"additional","affiliation":[{"name":"Centre for Healthcare Modelling and Informatics, University of Portsmouth, Portsmouth, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeremy C","family":"Wyatt","sequence":"additional","affiliation":[{"name":"Wessex Institute of Health Research, University of Southampton, Southampton, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7619-3668","authenticated-orcid":false,"given":"Andrew","family":"Georgiou","sequence":"additional","affiliation":[{"name":"Australian Institute of Health Innovation, Macquarie University, Sydney, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eric L","family":"Eisenstein","sequence":"additional","affiliation":[{"name":"Duke Clinical Research Institute, Duke University Medical Center, Durham, North Carolina, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Charles P","family":"Friedman","sequence":"additional","affiliation":[{"name":"Schools of Medicine, Information and Public Health, University of Michigan, Ann Arbor, Michigan, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2019,4,17]]},"reference":[{"key":"2020110613063559700_ocz035-B1","author":"Friedman"},{"issue":"23","key":"2020110613063559700_ocz035-B2","doi-asserted-by":"crossref","first-page":"2276","DOI":"10.2146\/ajhp070364","article-title":"Validity and reliability of measurement instruments used in research","volume":"65","author":"Kimberlin","year":"2008","journal-title":"Am J Health Syst Pharm"},{"key":"2020110613063559700_ocz035-B3","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1007\/0-387-36278-9_11","volume-title":"Biomedical Informatics","author":"Friedman","year":"2006"},{"issue":"1","key":"2020110613063559700_ocz035-B4","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1080\/14639230601097903","article-title":"Electronic health records: is the evidence base any use?","volume":"32","author":"Clamp","year":"2007","journal-title":"Med Inform Internet Med"},{"key":"2020110613063559700_ocz035-B5","author":"Scott","year":"2010"},{"issue":"8","key":"2020110613063559700_ocz035-B6","doi-asserted-by":"crossref","first-page":"963","DOI":"10.1093\/jamia\/ocy028","article-title":"Does health informatics have a replication crisis?","volume":"25","author":"Coiera","year":"2018","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613063559700_ocz035-B7","author":"EBSCO","year":"2018"},{"key":"2020110613063559700_ocz035-B8","volume-title":"Research Methods Laboratory Manual for Psychology","author":"Langston","year":"2010"},{"issue":"1","key":"2020110613063559700_ocz035-B9","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1080\/0141192870130107","article-title":"Some notes on the terms \u2018validity\u2019 and \u2018reliability\u2019","volume":"13","author":"Hammersley","year":"1987","journal-title":"Br Educ Res J"},{"key":"2020110613063559700_ocz035-B10","doi-asserted-by":"crossref","DOI":"10.4135\/9781412985642","volume-title":"Reliability and Validity Assessment","author":"Carmines","year":"1979"},{"key":"2020110613063559700_ocz035-B11","volume-title":"Measurement in Health Behavior: Methods for Research and Evaluation","author":"DiIorio","year":"2006"},{"issue":"2\u20133","key":"2020110613063559700_ocz035-B12","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1016\/S1386-5056(02)00109-0","article-title":"Is medical informatics a mature science? A review of measurement practice in outcome studies of clinical systems","volume":"69","author":"Friedman","year":"2003","journal-title":"Int J Med Inform"},{"key":"2020110613063559700_ocz035-B13","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1016\/j.jbi.2017.07.014","article-title":"Development and classification of a robust inventory of near real-time outcome measurements for assessing information technology interventions in health care","volume":"73","author":"Colicchio","year":"2017","journal-title":"J Biomed Inform"},{"key":"2020110613063559700_ocz035-B14","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1016\/j.jbi.2016.07.018","article-title":"Health information technology adoption: Understanding research protocols and outcome measurements for IT interventions in health care","volume":"63","author":"Colicchio","year":"2016","journal-title":"J Biomed Inform"},{"key":"2020110613063559700_ocz035-B15","author":"US National Library of Medicine","year":"2012"},{"issue":"1","key":"2020110613063559700_ocz035-B16","doi-asserted-by":"crossref","first-page":"e1000387.","DOI":"10.1371\/journal.pmed.1000387","article-title":"The impact of eHealth on the quality and safety of health care: a systematic overview","volume":"8","author":"Black","year":"2011","journal-title":"PLoS Med"},{"issue":"2","key":"2020110613063559700_ocz035-B17","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1136\/jamia.1998.0050164","article-title":"Development and initial validation of an instrument to measure physicians' use of, knowledge about, and attitudes toward computers","volume":"5","author":"Cork","year":"1998","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"2020110613063559700_ocz035-B18","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1136\/jamia.2010.010306","article-title":"The case for randomized controlled trials to assess the impact of clinical information systems","volume":"18","author":"Liu","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613063559700_ocz035-B19","first-page":"3","article-title":"Field trials of medical decision-aids: potential problems and solutions","author":"Wyatt","year":"1991","journal-title":"Proc Ann Symp Comp Appl Med Care"},{"issue":"1","key":"2020110613063559700_ocz035-B20","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"issue":"19","key":"2020110613063559700_ocz035-B21","doi-asserted-by":"crossref","first-page":"1851","DOI":"10.1001\/jama.282.19.1851","article-title":"Enhancement of clinicians\u2019 diagnostic reasoning by computer-based consultation: a multisite study of 2 systems","volume":"282","author":"Friedman","year":"1999","journal-title":"JAMA"},{"issue":"6","key":"2020110613063559700_ocz035-B22","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1197\/jamia.M1338","article-title":"Measuring the impact of diagnostic decision support on the quality of clinical decision making: development of a reliable and valid composite score","volume":"10","author":"Ramnarayan","year":"2003","journal-title":"J Am Med Inform Assoc"},{"issue":"Pt 1","key":"2020110613063559700_ocz035-B23","first-page":"648","article-title":"Evaluation of KNAVE-II: a tool for intelligent query and exploration of patient data","volume":"107","author":"Martins","year":"2004","journal-title":"Stud Health Technol Inform"},{"issue":"1","key":"2020110613063559700_ocz035-B24","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1016\/j.artmed.2008.03.006","article-title":"Evaluation of an architecture for intelligent query and exploration of time-oriented clinical data","volume":"43","author":"Martins","year":"2008","journal-title":"Artif Intell Med"},{"issue":"5","key":"2020110613063559700_ocz035-B25","doi-asserted-by":"crossref","first-page":"433","DOI":"10.1016\/j.ijmedinf.2004.04.001","article-title":"Clinicians\u2019 perceptions of clinical decision support integrated into computerized provider order entry","volume":"73","author":"Rosenbloom","year":"2004","journal-title":"Int J Med Inform"},{"key":"2020110613063559700_ocz035-B26","first-page":"36","article-title":"Negative CPOE attitudes correlate with diminished power in the workplace","author":"Bartos","year":"2008","journal-title":"AMIA Ann Symp Proc"},{"issue":"1","key":"2020110613063559700_ocz035-B27","first-page":"7","article-title":"Is healthcare information technology based on evidence?","volume":"8","author":"Koppel","year":"2013","journal-title":"Yearb Med Inform"},{"key":"2020110613063559700_ocz035-B28","first-page":"889","article-title":"Inter-observer reliability assessments in time motion studies: the foundation for meaningful clinical workflow analysis","volume":"2013","author":"Lopetegui","year":"2013","journal-title":"AMIA Annu Symp Proc"},{"key":"2020110613063559700_ocz035-B29","first-page":"34","article-title":"Evidence based health informatics: 10 years of efforts to promote the principle","author":"Rigby","year":"2013","journal-title":"Yearb Med Inform"},{"issue":"12","key":"2020110613063559700_ocz035-B30","doi-asserted-by":"crossref","first-page":"815","DOI":"10.1016\/j.ijmedinf.2011.08.004","article-title":"Guideline for good evaluation practice in health informatics (GEP-HI)","volume":"80","author":"Nyk\u00e4nen","year":"2011","journal-title":"Int J Med Inform"},{"issue":"3","key":"2020110613063559700_ocz035-B31","doi-asserted-by":"crossref","first-page":"331","DOI":"10.4338\/ACI-2013-04-RA-0024","article-title":"Statement on Reporting of Evaluation Studies in Health Informatics: explanation and elaboration","volume":"4","author":"Brender","year":"2013","journal-title":"Appl Clin Inform"},{"issue":"1","key":"2020110613063559700_ocz035-B32","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ijmedinf.2008.09.002","article-title":"STARE-HI\u2014statement on reporting of evaluation studies in health informatics","volume":"78","author":"Talmon","year":"2009","journal-title":"Int J Med Inform"},{"issue":"1","key":"2020110613063559700_ocz035-B33","doi-asserted-by":"crossref","first-page":"18.","DOI":"10.1186\/1741-7015-8-18","article-title":"CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials","volume":"8","author":"Schulz","year":"2010","journal-title":"BMC Med"},{"key":"2020110613063559700_ocz035-B34","volume-title":"Handbook of Evaluation Methods for Health Informatics","author":"McNair","year":"2006"},{"key":"2020110613063559700_ocz035-B35","volume-title":"Evidence-Based Health Informatics: Promoting Safety and Efficiency through Scientific Methods and Ethical Policy","author":"Ammenwerth","year":"2016"},{"key":"2020110613063559700_ocz035-B36","author":"Agency for Healthcare Research and Quality (AHRQ)"},{"key":"2020110613063559700_ocz035-B37","author":"Agency for Healthcare Research and Quality, A","year":"2014"},{"key":"2020110613063559700_ocz035-B38","author":"Wakefield","year":"2014"},{"issue":"1","key":"2020110613063559700_ocz035-B39","doi-asserted-by":"crossref","first-page":"135.","DOI":"10.1186\/1472-6947-13-135","article-title":"Systematic review of clinical decision support interventions with potential for inpatient cost reduction","volume":"13","author":"Fillmore","year":"2013","journal-title":"BMC Med Inform Dec Making"},{"issue":"3","key":"2020110613063559700_ocz035-B40","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1177\/1558689809334209","article-title":"A pragmatist argument for mixed methodology in medical informatics","volume":"3","author":"Scott","year":"2009","journal-title":"J Mixed Methods Res"},{"issue":"3","key":"2020110613063559700_ocz035-B41","doi-asserted-by":"crossref","first-page":"147","DOI":"10.17705\/1jais.00354","article-title":"Evaluation as a multi-ontological endeavour: a case from the English National Program for IT in healthcare","volume":"15","author":"Klecun","year":"2014","journal-title":"JAIS"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/10\/1120\/34151653\/ocz035.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/10\/1120\/34151653\/ocz035.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T19:09:16Z","timestamp":1604689756000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/26\/10\/1120\/5470864"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,4,17]]},"references-count":41,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2019,4,17]]},"published-print":{"date-parts":[[2019,10,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocz035","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,10]]},"published":{"date-parts":[[2019,4,17]]}}}