{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T10:17:30Z","timestamp":1778667450635,"version":"3.51.4"},"reference-count":54,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2016,1,17]],"date-time":"2016-01-17T00:00:00Z","timestamp":1452988800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2016,5,1]]},"abstract":"<jats:p>Objective For health information technology tools to fully inform evidence-based decisions, recommendations must be reliably assessed for quality and strength of evidence. We aimed to create an annotation framework for grading recommendations regarding appropriate use of diagnostic imaging examinations.<\/jats:p>\n               <jats:p>Methods The annotation framework was created by an expert panel (clinicians in three medical specialties, medical librarians, and biomedical scientists) who developed a process for achieving consensus in assessing recommendations, and evaluated by measuring agreement in grading the strength of evidence for 120 empirically selected recommendations using the Oxford Levels of Evidence.<\/jats:p>\n               <jats:p>Results Eighty-two percent of recommendations were assigned to Level 5 (expert opinion). Inter-annotator agreement was 0.70 on initial grading (\u03ba\u2009=\u20090.35, 95% CI, 0.23-0.48). After systematic discussion utilizing the annotation framework, agreement increased significantly to 0.97 (\u03ba\u2009=\u20090.88, 95% CI, 0.77-0.99).<\/jats:p>\n               <jats:p>Conclusions A novel annotation framework was effective for grading the strength of evidence supporting appropriate use criteria for diagnostic imaging exams.<\/jats:p>","DOI":"10.1093\/jamia\/ocv194","type":"journal-article","created":{"date-parts":[[2016,1,18]],"date-time":"2016-01-18T01:07:26Z","timestamp":1453079246000},"page":"649-653","source":"Crossref","is-referenced-by-count":16,"title":["Assessing Strength of Evidence of Appropriate Use Criteria for Diagnostic Imaging Examinations"],"prefix":"10.1093","volume":"23","author":[{"given":"Ronilda","family":"Lacson","sequence":"first","affiliation":[{"name":"Center for Evidence Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA 02115, USA"},{"name":"Harvard Medical School, Boston, MA 02115, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ali S","family":"Raja","sequence":"additional","affiliation":[{"name":"Harvard Medical School, Boston, MA 02115, USA"},{"name":"Department of Emergency Medicine, Massachusetts General Hospital, Boston, MA 02114, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David","family":"Osterbur","sequence":"additional","affiliation":[{"name":"Harvard Medical School, Boston, MA 02115, USA"},{"name":"Countway Library of Medicine, Boston, MA 02115, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ivan","family":"Ip","sequence":"additional","affiliation":[{"name":"Center for Evidence Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA 02115, USA"},{"name":"Harvard Medical School, Boston, MA 02115, USA"},{"name":"Department of Medicine, Brigham and Women's Hospital, Boston, MA 02115, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Louise","family":"Schneider","sequence":"additional","affiliation":[{"name":"Harvard Medical School, Boston, MA 02115, USA"},{"name":"Department of Medicine, Brigham and Women's Hospital, Boston, MA 02115, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Paul","family":"Bain","sequence":"additional","affiliation":[{"name":"Harvard Medical School, Boston, MA 02115, USA"},{"name":"Countway Library of Medicine, Boston, MA 02115, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Carol","family":"Mita","sequence":"additional","affiliation":[{"name":"Harvard Medical School, Boston, MA 02115, USA"},{"name":"Countway Library of Medicine, Boston, MA 02115, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Julia","family":"Whelan","sequence":"additional","affiliation":[{"name":"Harvard Medical School, Boston, MA 02115, USA"},{"name":"Countway Library of Medicine, Boston, MA 02115, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Patricia","family":"Silveira","sequence":"additional","affiliation":[{"name":"Center for Evidence Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA 02115, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David","family":"Dement","sequence":"additional","affiliation":[{"name":"Center for Evidence Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA 02115, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ramin","family":"Khorasani","sequence":"additional","affiliation":[{"name":"Center for Evidence Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA 02115, USA"},{"name":"Harvard Medical School, Boston, MA 02115, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2016,1,17]]},"reference":[{"issue":"Pt 2","key":"2021010611501532500_ocv194-B1","first-page":"814","article-title":"Measuring and improving quality using information systems","volume":"52","author":"Bates","year":"1998","journal-title":"Stud Health Technol Inform."},{"issue":"3","key":"2021010611501532500_ocv194-B2","doi-asserted-by":"crossref","first-page":"320","DOI":"10.1016\/j.ajem.2014.11.005","article-title":"Impact of clinical decision support on head computed tomography use in patients with mild traumatic brain injury in the ED","volume":"33","author":"Ip","year":"2015","journal-title":"Am J Emerg Med."},{"issue":"8","key":"2021010611501532500_ocv194-B3","doi-asserted-by":"crossref","first-page":"687","DOI":"10.1016\/j.amjmed.2012.11.034","article-title":"Impact of provider-led, technology-enabled radiology management program on imaging","volume":"126","author":"Ip","year":"2013","journal-title":"Am J Med."},{"issue":"2","key":"2021010611501532500_ocv194-B4","doi-asserted-by":"crossref","first-page":"468","DOI":"10.1148\/radiol.11110951","article-title":"Effect of computerized clinical decision support on the use and yield of CT pulmonary angiography in the emergency department","volume":"262","author":"Raja","year":"2012","journal-title":"Radiology."},{"issue":"6","key":"2021010611501532500_ocv194-B5","doi-asserted-by":"crossref","first-page":"512","DOI":"10.1016\/j.amjmed.2014.01.024","article-title":"Impact of IT-enabled intervention on mri use for back pain","volume":"127","author":"Ip","year":"2014","journal-title":"Am J Med."},{"issue":"4","key":"2021010611501532500_ocv194-B6","doi-asserted-by":"crossref","first-page":"232","DOI":"10.1056\/NEJM199801223380406","article-title":"A computer-assisted management program for antibiotics and other antiinfective agents","volume":"338","author":"Evans","year":"1998","journal-title":"N Engl J Med."},{"issue":"1","key":"2021010611501532500_ocv194-B7","doi-asserted-by":"crossref","first-page":"26","DOI":"10.5415\/apallergy.2012.2.1.26","article-title":"Educational and decision-support tools for asthma-management guideline implementation","volume":"2","author":"Kim","year":"2012","journal-title":"Asia Pac Allergy."},{"issue":"10","key":"2021010611501532500_ocv194-B8","doi-asserted-by":"crossref","first-page":"969","DOI":"10.1056\/NEJMoa041533","article-title":"Electronic alerts to prevent venous thromboembolism among hospitalized patients","volume":"352","author":"Kucher","year":"2005","journal-title":"N Engl J Med."},{"issue":"6","key":"2021010611501532500_ocv194-B9","doi-asserted-by":"crossref","first-page":"1276","DOI":"10.1097\/CCM.0000000000000948","article-title":"Impact of the electronic medical record on mortality, length of stay, and cost in the hospital and ICU: a systematic review and metaanalysis","volume":"43","author":"Thompson","year":"2015","journal-title":"Crit Care Med."},{"issue":"17","key":"2021010611501532500_ocv194-B10","doi-asserted-by":"crossref","first-page":"1578","DOI":"10.1001\/archinternmed.2009.263","article-title":"Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are electronic medical records achieving their potential?","volume":"169","author":"Singh","year":"2009","journal-title":"Arch Intern Med."},{"issue":"171","key":"2021010611501532500_ocv194-B11","first-page":"53967","article-title":"Final rule","volume":"77","author":"Medicare and Medicaid programs; electronic health record incentive program\u2013stage 2","year":"2012","journal-title":"Fed Regist."},{"issue":"7494","key":"2021010611501532500_ocv194-B12","doi-asserted-by":"crossref","first-page":"765","DOI":"10.1136\/bmj.38398.500764.8F","article-title":"Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success","volume":"330","author":"Kawamoto","year":"2005","journal-title":"BMJ."},{"issue":"2","key":"2021010611501532500_ocv194-B13","doi-asserted-by":"crossref","first-page":"135","DOI":"10.7326\/0003-4819-120-2-199401150-00007","article-title":"Effects of computer-based clinical decision support systems on clinician performance and patient outcome. A critical appraisal of research","volume":"120","author":"Johnston","year":"1994","journal-title":"Ann Intern Med."},{"key":"2021010611501532500_ocv194-B14","first-page":"626","article-title":"A critical appraisal of the literature on the effects of computer-based clinical decision support systems on clinician performance and patient outcomes","author":"Langton","year":"1992","journal-title":"Proc Annu Symp Comput Appl Med Care."},{"issue":"15","key":"2021010611501532500_ocv194-B15","doi-asserted-by":"crossref","first-page":"1339","DOI":"10.1001\/jama.280.15.1339","article-title":"Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review","volume":"280","author":"Hunt","year":"1998","journal-title":"JAMA."},{"key":"2021010611501532500_ocv194-B16"},{"issue":"1","key":"2021010611501532500_ocv194-B17","doi-asserted-by":"crossref","first-page":"e21","DOI":"10.2196\/resprot.4024","article-title":"Implementation and evaluation of a Wiki involving multiple stakeholders including patients in the promotion of best practices in trauma care: the WikiTrauma Interrupted Time Series Protocol","volume":"4","author":"Archambault","year":"2015","journal-title":"JMIR Res Protoc."},{"issue":"7","key":"2021010611501532500_ocv194-B18","doi-asserted-by":"crossref","first-page":"727","DOI":"10.1016\/j.jclinepi.2014.12.015","article-title":"Quality of evidence is a key determinant for making a strong guidelines recommendation","volume":"68","author":"Djulbegovic","year":"2015","journal-title":"J Clin Epidemiol."},{"key":"2021010611501532500_ocv194-B19","doi-asserted-by":"crossref","first-page":"S217","DOI":"10.2105\/AJPH.2014.302077","article-title":"Advancing the use of evidence-based decision-making in local health departments with systems science methodologies","volume":"105","author":"Li","year":"2015","journal-title":"Am J Public Health."},{"issue":"2","key":"2021010611501532500_ocv194-B20","first-page":"159","article-title":"Preventable medication errors: identifying and eliminating serious drug interactions","volume":"41","author":"Peterson","year":"2001","journal-title":"J Am Pharm Assoc."},{"issue":"2","key":"2021010611501532500_ocv194-B21","doi-asserted-by":"crossref","first-page":"395","DOI":"10.1148\/radiol.2372041887","article-title":"Guidelines for management of small pulmonary nodules detected on CT scans: a statement from the Fleischner Society","volume":"237","author":"MacMahon","year":"2005","journal-title":"Radiology."},{"issue":"1","key":"2021010611501532500_ocv194-B22","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1148\/radiol.2491070783","article-title":"Management of the incidental renal mass","volume":"249","author":"Silverman","year":"2008","journal-title":"Radiology."},{"issue":"12","key":"2021010611501532500_ocv194-B23","doi-asserted-by":"crossref","first-page":"1511","DOI":"10.1001\/jama.294.12.1511","article-title":"Comparison of the Canadian CT Head Rule and the New Orleans Criteria in patients with minor head injury","volume":"294","author":"Stiell","year":"2005","journal-title":"JAMA."},{"issue":"2","key":"2021010611501532500_ocv194-B24","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1067\/mem.2001.116796","article-title":"The Canadian CT Head Rule Study for patients with minor head injury: rationale, objectives, and methodology for phase I (derivation)","volume":"38","author":"Stiell","year":"2001","journal-title":"Ann Emerg Med."},{"issue":"996","key":"2021010611501532500_ocv194-B25","first-page":"111","article-title":"Ottawa ankle rules for radiography of acute injuries","volume":"108","author":"Stiell","year":"1995","journal-title":"N Z Med J."},{"issue":"9","key":"2021010611501532500_ocv194-B26","doi-asserted-by":"crossref","first-page":"1127","DOI":"10.1001\/jama.269.9.1127","article-title":"Decision rules for the use of radiography in acute ankle injuries. Refinement and prospective validation","volume":"269","author":"Stiell","year":"1993","journal-title":"JAMA."},{"key":"2021010611501532500_ocv194-B27"},{"issue":"4","key":"2021010611501532500_ocv194-B28","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1097\/RUQ.0000000000000099","article-title":"ACR Appropriateness Criteria(R) acute pancreatitis","volume":"30","author":"Baker","year":"2014","journal-title":"Ultrasound Q."},{"issue":"6","key":"2021010611501532500_ocv194-B29","doi-asserted-by":"crossref","first-page":"364","DOI":"10.1097\/RTI.0000000000000113","article-title":"ACR Appropriateness Criteria(R) rib fractures","volume":"29","author":"Henry","year":"2014","journal-title":"J Thorac Imaging."},{"issue":"3","key":"2021010611501532500_ocv194-B30","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1016\/j.jacr.2014.11.015","article-title":"ACR Appropriateness Criteria Acute Trauma to the Ankle","volume":"12","author":"Mosher","year":"2015","journal-title":"J Am Coll Radiol."},{"issue":"12 Pt A","key":"2021010611501532500_ocv194-B31","doi-asserted-by":"crossref","first-page":"1160","DOI":"10.1016\/j.jacr.2014.08.020","article-title":"ACR Appropriateness Criteria stage I breast cancer: initial workup and surveillance for local recurrence and distant metastases in asymptomatic women","volume":"11","author":"Moy","year":"2014","journal-title":"J Am Coll Radiol."},{"issue":"6","key":"2021010611501532500_ocv194-B32","doi-asserted-by":"crossref","first-page":"380","DOI":"10.1016\/j.profnurs.2005.10.004","article-title":"Evidence-based resources and the role of librarians in developing evidence-based practice curricula","volume":"21","author":"Klem","year":"2005","journal-title":"J Prof Nurs."},{"issue":"1","key":"2021010611501532500_ocv194-B33","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1097\/01.NNE.0000299497.23119.21","article-title":"Customized library tutorial for online BSN students","volume":"33","author":"Morrison","year":"2008","journal-title":"Nurse Educator."},{"key":"2021010611501532500_ocv194-B34","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1111\/j.1470-3327.2005.00612.x","article-title":"Healthcare librarians and learner support: a review of competences and methods","volume":"22","author":"Robinson","year":"2005","journal-title":"Health Info Libr J."},{"key":"2021010611501532500_ocv194-B35"},{"key":"2021010611501532500_ocv194-B36","first-page":"47","article-title":"Reuse of knowledge represented in the Arden syntax","author":"Shwe","year":"1992","journal-title":"Proc Annu Symp Comput Appl Med Care."},{"issue":"1","key":"2021010611501532500_ocv194-B37","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1186\/1472-6963-5-25","article-title":"Systems for grading the quality of evidence and the strength of recommendations II: pilot study of a new system","volume":"5","author":"Atkins","year":"2005","journal-title":"BMC Health Serv Res."},{"issue":"7","key":"2021010611501532500_ocv194-B38","doi-asserted-by":"crossref","first-page":"760","DOI":"10.1016\/j.jclinepi.2014.01.006","article-title":"Applying Grading of Recommendations Assessment, Development and Evaluation (GRADE) to diagnostic tests was challenging but doable","volume":"67","author":"Gopalakrishna","year":"2014","journal-title":"J Clin Epidemiol."},{"issue":"5","key":"2021010611501532500_ocv194-B39","doi-asserted-by":"crossref","first-page":"465","DOI":"10.1001\/jama.2014.17082","article-title":"A piece of my mind. Should C be a passing grade for the USPSTF?","volume":"313","author":"Saver","year":"2015","journal-title":"JAMA."},{"issue":"2","key":"2021010611501532500_ocv194-B40","doi-asserted-by":"crossref","first-page":"123","DOI":"10.7326\/0003-4819-147-2-200707170-00171","article-title":"How to read the new recommendation statement: methods update from the U.S. Preventive Services Task Force","volume":"147","author":"Barton","year":"2007","journal-title":"Ann Intern Med."},{"issue":"10","key":"2021010611501532500_ocv194-B41","doi-asserted-by":"crossref","first-page":"716","DOI":"10.7326\/0003-4819-151-10-200911170-00008","article-title":"U.S. Preventive Services Task Force recommendation statement","volume":"151","author":"The U.S. Preventive Services Task Force. Screening for breast cancer","year":"2009","journal-title":"Ann Intern Med."},{"issue":"11","key":"2021010611501532500_ocv194-B42","doi-asserted-by":"crossref","first-page":"1312","DOI":"10.1016\/j.jclinepi.2014.11.023","article-title":"Grading the strength of a body of evidence when assessing health care interventions: an EPC update","volume":"68","author":"Berkman","year":"2015","journal-title":"J Clin Epidemiol."},{"key":"2021010611501532500_ocv194-B43","author":"Berkman"},{"key":"2021010611501532500_ocv194-B44"},{"key":"2021010611501532500_ocv194-B45","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1016\/S0749-3797(01)00261-6","article-title":"Current methods of the US Preventive Services Task Force: a review of the process","volume":"20","author":"Harris","year":"2001","journal-title":"Am J Prev Med."},{"issue":"10","key":"2021010611501532500_ocv194-B46","first-page":"RA181","article-title":"A new scheme for grading the quality of scientific reports that evaluate imaging modalities for cerebrovascular diseases","volume":"13","author":"Qureshi","year":"2007","journal-title":"Med Sci Monit."},{"issue":"9","key":"2021010611501532500_ocv194-B47","doi-asserted-by":"crossref","first-page":"602","DOI":"10.1016\/j.jacr.2011.05.008","article-title":"ACR Appropriateness Criteria(R) on acute shoulder pain","volume":"8","author":"Wise","year":"2011","journal-title":"J Am Coll Radiol."},{"issue":"1","key":"2021010611501532500_ocv194-B48","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics."},{"issue":"4","key":"2021010611501532500_ocv194-B49","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1037\/h0026256","article-title":"Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit","volume":"70","author":"Cohen","year":"1968","journal-title":"Psychol Bull."},{"issue":"5","key":"2021010611501532500_ocv194-B50","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1016\/0010-4825(94)90002-7","article-title":"Writing Arden Syntax Medical Logic Modules","volume":"24","author":"Hripcsak","year":"1994","journal-title":"Comput Biol Med."},{"issue":"2","key":"2021010611501532500_ocv194-B51","first-page":"76, 78","article-title":"Arden Syntax for Medical Logic Modules","volume":"8","author":"Hripcsak","year":"1991","journal-title":"MD Comput."},{"issue":"2","key":"2021010611501532500_ocv194-B52","first-page":"126","article-title":"Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials","volume":"19","author":"Smith","year":"2006","journal-title":"Int J Prosthodont."},{"issue":"1","key":"2021010611501532500_ocv194-B53","doi-asserted-by":"crossref","first-page":"55","DOI":"10.7812\/TPP\/11-134","article-title":"Transparency matters: Kaiser Permanente's National Guideline Program methodological processes","volume":"16","author":"Davino-Ramaya","year":"2012","journal-title":"Perm J."},{"key":"2021010611501532500_ocv194-B54","doi-asserted-by":"crossref","first-page":"853","DOI":"10.1186\/1756-0500-7-853","article-title":"Tools for assessing the content of guidelines are needed to enable their effective use\u2013a systematic comparison","volume":"7","author":"Eikermann","year":"2014","journal-title":"BMC Res Notes."}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/23\/3\/649\/34938666\/ocv194.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/23\/3\/649\/34938666\/ocv194.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,1,10]],"date-time":"2021-01-10T21:27:41Z","timestamp":1610314061000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/23\/3\/649\/2909016"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,1,17]]},"references-count":54,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2016,1,17]]},"published-print":{"date-parts":[[2016,5,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocv194","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2016,5]]},"published":{"date-parts":[[2016,1,17]]}}}