{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,20]],"date-time":"2026-11-20T15:19:12Z","timestamp":1795187952558,"version":"4.10.3"},"reference-count":36,"publisher":"Ovid Technologies (Wolters Kluwer Health)","issue":"3","license":[{"start":{"date-parts":[[2022,9,9]],"date-time":"2022-09-09T00:00:00Z","timestamp":1662681600000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["lww.com","ovid.com"],"crossmark-restriction":true},"short-container-title":["Clin Orthop Relat Res"],"published-print":{"date-parts":[[2023,3]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Missed fractures are the most common diagnostic errors in musculoskeletal imaging and can result in treatment delays and preventable morbidity. Deep learning, a subfield of artificial intelligence, can be used to accurately detect fractures by training algorithms to emulate the judgments of expert clinicians. Deep learning systems that detect fractures are often limited to specific anatomic regions and require regulatory approval to be used in practice. Once these hurdles are overcome, deep learning systems have the potential to improve clinician diagnostic accuracy and patient care.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Questions\/purposes<\/jats:title>\n                    <jats:p>This study aimed to evaluate whether a Food and Drug Administration\u2013cleared deep learning system that identifies fractures in adult musculoskeletal radiographs would improve diagnostic accuracy for fracture detection across different types of clinicians. Specifically, this study asked: (1) What are the trends in musculoskeletal radiograph interpretation by different clinician types in the publicly available Medicare claims data? (2) Does the deep learning system improve clinician accuracy in diagnosing fractures on radiographs and, if so, is there a greater benefit for clinicians with limited training in musculoskeletal imaging?<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We used the publicly available Medicare Part B Physician\/Supplier Procedure Summary data provided by the Centers for Medicare &amp; Medicaid Services to determine the trends in musculoskeletal radiograph interpretation by clinician type. In addition, we conducted a multiple-reader, multiple-case study to assess whether clinician accuracy in diagnosing fractures on radiographs was superior when aided by the deep learning system compared with when unaided. Twenty-four clinicians (radiologists, orthopaedic surgeons, physician assistants, primary care physicians, and emergency medicine physicians) with a median (range) of 16 years (2 to 37) of experience postresidency each assessed 175 unique musculoskeletal radiographic cases under aided and unaided conditions (4200 total case-physician pairs per condition). These cases were comprised of radiographs from 12 different anatomic regions (ankle, clavicle, elbow, femur, forearm, hip, humerus, knee, pelvis, shoulder, tibia and fibula, and wrist) and were randomly selected from 12 hospitals and healthcare centers. The gold standard for fracture diagnosis was the majority opinion of three US board-certified orthopaedic surgeons or radiologists who independently interpreted the case. The clinicians\u2019 diagnostic accuracy was determined by the area under the curve (AUC) of the receiver operating characteristic (ROC) curve, sensitivity, and specificity. Secondary analyses evaluated the fracture miss rate (1-sensitivity) by clinicians with and without extensive training in musculoskeletal imaging.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Medicare claims data revealed that physician assistants showed the greatest increase in interpretation of musculoskeletal radiographs within the analyzed time period (2012 to 2018), although clinicians with extensive training in imaging (radiologists and orthopaedic surgeons) still interpreted the majority of the musculoskeletal radiographs. Clinicians aided by the deep learning system had higher accuracy diagnosing fractures in radiographs compared with when unaided (unaided AUC: 0.90 [95% CI 0.89 to 0.92]; aided AUC: 0.94 [95% CI 0.93 to 0.95]; difference in least square mean per the Dorfman, Berbaum, Metz model AUC: 0.04 [95% CI 0.01 to 0.07]; p &lt; 0.01). Clinician sensitivity increased when aided compared with when unaided (aided: 90% [95% CI 88% to 92%]; unaided: 82% [95% CI 79% to 84%]), and specificity increased when aided compared with when unaided (aided: 92% [95% CI 91% to 93%]; unaided: 89% [95% CI 88% to 90%]). Clinicians with limited training in musculoskeletal imaging missed a higher percentage of fractures when unaided compared with radiologists (miss rate for clinicians with limited imaging training: 20% [95% CI 17% to 24%]; miss rate for radiologists: 14% [95% CI 9% to 19%]). However, when assisted by the deep learning system, clinicians with limited training in musculoskeletal imaging reduced their fracture miss rate, resulting in a similar miss rate to radiologists (miss rate for clinicians with limited imaging training: 9% [95% CI 7% to 12%]; miss rate for radiologists: 10% [95% CI 6% to 15%]).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Clinicians were more accurate at diagnosing fractures when aided by the deep learning system, particularly those clinicians with limited training in musculoskeletal image interpretation. Reducing the number of missed fractures may allow for improved patient care and increased patient mobility.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Level of Evidence<\/jats:title>\n                    <jats:p>Level III, diagnostic study.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1097\/corr.0000000000002385","type":"journal-article","created":{"date-parts":[[2022,9,9]],"date-time":"2022-09-09T16:57:37Z","timestamp":1662742657000},"page":"580-588","update-policy":"https:\/\/doi.org\/10.1097\/lww.0000000000001000","source":"Crossref","is-referenced-by-count":58,"title":["Deep Learning Assistance Closes the Accuracy Gap in Fracture Detection Across Clinician Types"],"prefix":"10.1097","volume":"481","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6602-9186","authenticated-orcid":false,"given":"Pamela G.","family":"Anderson","sequence":"first","affiliation":[{"name":"Imagen Technologies, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Graham L.","family":"Baum","sequence":"additional","affiliation":[{"name":"Imagen Technologies, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nora","family":"Keathley","sequence":"additional","affiliation":[{"name":"Imagen Technologies, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Serge","family":"Sicular","sequence":"additional","affiliation":[{"name":"Imagen Technologies, New York, NY, USA"},{"name":"The Mount Sinai Hospital, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Srivas","family":"Venkatesh","sequence":"additional","affiliation":[{"name":"Imagen Technologies, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anuj","family":"Sharma","sequence":"additional","affiliation":[{"name":"Imagen Technologies, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aaron","family":"Daluiski","sequence":"additional","affiliation":[{"name":"Hospital for Special Surgery, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hollis","family":"Potter","sequence":"additional","affiliation":[{"name":"Hospital for Special Surgery, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert","family":"Hotchkiss","sequence":"additional","affiliation":[{"name":"Hospital for Special Surgery, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert V.","family":"Lindsey","sequence":"additional","affiliation":[{"name":"Imagen Technologies, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rebecca M.","family":"Jones","sequence":"additional","affiliation":[{"name":"Imagen Technologies, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"276","published-online":{"date-parts":[[2022,9,9]]},"reference":[{"key":"R1-20250508","doi-asserted-by":"crossref","first-page":"e0045","DOI":"10.2106\/JBJS.OA.18.00045","article-title":"Incidence and economic burden of intertrochanteric fracture: a Medicare claims database analysis","volume":"4","author":"Adeyemi","year":"2019","journal-title":"JB JS Open Access"},{"key":"R2-20250508","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1002\/jbmr.2072","article-title":"Trends in fracture incidence: a population-based study over 20 years","volume":"29","author":"Amin","year":"2014","journal-title":"J Bone Miner Res"},{"key":"R3-20250508","doi-asserted-by":"crossref","first-page":"317","DOI":"10.2214\/ajr.176.2.1760317","article-title":"Defending the \u201cmissed\u201d radiographic diagnosis","volume":"176","author":"Berlin","year":"2001","journal-title":"AJR Am J Roentgenol"},{"key":"R4-20250508","doi-asserted-by":"crossref","first-page":"465","DOI":"10.1359\/jbmr.061113","article-title":"Incidence and economic burden of osteoporosis-related fractures in the United States, 2005\u20132025","volume":"22","author":"Burge","year":"2007","journal-title":"J Bone Miner Res"},{"key":"R6-20250508","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1007\/s00256-019-03284-z","article-title":"Current applications and future directions of deep learning in musculoskeletal radiology","volume":"49","author":"Chea","year":"2020","journal-title":"Skeletal Radiol"},{"key":"R7-20250508","doi-asserted-by":"crossref","first-page":"e0245992","DOI":"10.1371\/journal.pone.0245992","article-title":"Application of deep learning algorithm to detect and visualize vertebral fractures on plain frontal radiographs","volume":"16","author":"Chen","year":"2021","journal-title":"PLoS One"},{"key":"R8-20250508","first-page":"1","article-title":"A scalable physician-level deep learning algorithm detects universal trauma on pelvic radiographs","volume":"12","author":"Cheng","year":"2021","journal-title":"Nat Commun"},{"key":"R9-20250508","doi-asserted-by":"crossref","first-page":"468","DOI":"10.1080\/17453674.2018.1453714","article-title":"Automated detection and classification of the proximal humerus fracture by using deep learning algorithm","volume":"89","author":"Chung","year":"2018","journal-title":"Acta Orthop"},{"key":"R10-20250508","doi-asserted-by":"crossref","first-page":"806","DOI":"10.1302\/0301-620X.93B6.25596","article-title":"The outcome of fractures in very elderly patients","volume":"96","author":"Clement","year":"2011","journal-title":"J Bone Joint Surg Br"},{"key":"R11-20250508","doi-asserted-by":"crossref","first-page":"723","DOI":"10.1097\/00004424-199209000-00015","article-title":"Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method","volume":"27","author":"Dorfman","year":"1992","journal-title":"Invest Radiol"},{"key":"R12-20250508","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1148\/radiol.2021203886","article-title":"Assessment of an AI aid in detection of adult appendicular skeletal fractures by emergency physicians and radiologists: a multicenter cross-sectional diagnostic study","volume":"300","author":"Duron","year":"2021","journal-title":"Radiology"},{"key":"R13-20250508","doi-asserted-by":"crossref","first-page":"128","DOI":"10.1080\/13814788.2019.1625886","article-title":"Diagnostic errors reported in primary healthcare and emergency departments: a retrospective and descriptive cohort study of 4830 reported cases of preventable harm in Sweden","volume":"25","author":"Fernholm","year":"2019","journal-title":"Eur J Gen Pract"},{"key":"R16-20250508","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1007\/s11420-005-0137-8","article-title":"Osteoporosis and skeletal fractures","volume":"2","author":"Gardner","year":"2006","journal-title":"HSS Journal"},{"key":"R17-20250508","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1177\/1758998316659676","article-title":"Can we reduce the incidence of complex regional pain syndrome type I in distal radius fractures? The Liverpool experience","volume":"21","author":"Gillespie","year":"2016","journal-title":"Hand Therapy"},{"key":"R18-20250508","doi-asserted-by":"crossref","first-page":"627","DOI":"10.1148\/radiol.210937","article-title":"Improving radiographic fracture recognition performance and efficiency using artificial intelligence","volume":"302","author":"Guermazi","year":"2022","journal-title":"Radiology"},{"key":"R19-20250508","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1136\/emj.18.4.263","article-title":"Diagnostic errors in an accident and emergency department","volume":"18","author":"Guly","year":"2001","journal-title":"Emerg Med J"},{"key":"R20-20250508","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-227X-6-4","article-title":"Errors in fracture diagnoses in the emergency department \u2013 characteristics of patients and diurnal variation","volume":"6","author":"Hallas","year":"2006","journal-title":"BMC Emerg Med"},{"key":"R21-20250508","doi-asserted-by":"crossref","first-page":"647","DOI":"10.1016\/j.acra.2007.12.015","article-title":"Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis","volume":"15","author":"Hillis","year":"2008","journal-title":"Acad Radiol."},{"key":"R22-20250508","doi-asserted-by":"crossref","first-page":"1899","DOI":"10.2106\/JBJS.G.00883","article-title":"Outcome after surgery for the treatment of hip fracture in the extremely elderly","volume":"90","author":"Holt","year":"2008","journal-title":"J Bone Joint Surg Am"},{"key":"R23-20250508","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12873-019-0289-3","article-title":"Diagnostic error in the emergency department: learning from national patient safety incident report analysis","volume":"19","author":"Hussain","year":"2019","journal-title":"BMC Emerg Med"},{"key":"R24-20250508","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-020-00352-w","article-title":"Assessment of a deep-learning system for fracture detection in musculoskeletal radiographs","volume":"3","author":"Jones","year":"2020","journal-title":"NPJ Digit Med"},{"key":"R25-20250508","doi-asserted-by":"crossref","first-page":"e190023","DOI":"10.1148\/ryai.2020190023","article-title":"Automatic hip fracture identification and functional subclassification with deep learning","volume":"2","author":"Krogue","year":"2020","journal-title":"Radiol Artif Intell"},{"key":"R26-20250508","doi-asserted-by":"crossref","first-page":"1479","DOI":"10.2106\/JBJS.G.00675","article-title":"Mortality following the diagnosis of a vertebral compression fracture in the Medicare population","volume":"90","author":"Lau","year":"2008","journal-title":"J Bone Joint Surg Am"},{"key":"R27-20250508","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1016\/j.jacr.2010.06.027","article-title":"Medicare payments for noninvasive diagnostic imaging are now higher to nonradiologist physicians than to radiologists","volume":"8","author":"Levin","year":"2011","journal-title":"J Am Coll Radiol"},{"key":"R28-20250508","doi-asserted-by":"crossref","first-page":"1299","DOI":"10.1007\/s00198-020-05334-y","article-title":"Osteoporotic fracture trends in a population of US managed care enrollees from 2007 to 2017","volume":"31","author":"Lewiecki","year":"2020","journal-title":"Osteoporos Int"},{"key":"R29-20250508","doi-asserted-by":"crossref","first-page":"11591","DOI":"10.1073\/pnas.1806905115","article-title":"Deep neural network improves fracture detection by clinicians","volume":"115","author":"Lindsey","year":"2018","journal-title":"Proc Natl Acad Sci USA"},{"key":"R30-20250508","doi-asserted-by":"crossref","first-page":"475739","DOI":"10.1155\/2012\/475739","article-title":"Survivorship and severe complications are worse for octogenarians and elderly patients with pelvis fractures as compared to adults: data from the national trauma data bank","volume":"2012","author":"Matityahu","year":"2012","journal-title":"J Osteoporos"},{"key":"R31-20250508","doi-asserted-by":"crossref","first-page":"1408","DOI":"10.1016\/j.jacr.2017.10.007","article-title":"National trends in the utilization of skeletal radiography from 2003 to 2015","volume":"15","author":"Mizrahi","year":"2018","journal-title":"J Am Coll Radiol"},{"key":"R32-20250508","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1136\/emj.14.5.295","article-title":"Interpretation of trauma radiographs by junior doctors in accident and emergency departments: a cause for concern?","volume":"14","author":"McLauchlan","year":"1997","journal-title":"J Accid Emerg Med"},{"key":"R33-20250508","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13049-017-0361-5","article-title":"Diagnostic error in the emergency department: follow up of patients with minor trauma in the outpatient clinic","volume":"25","author":"Moonen","year":"2017","journal-title":"Scand J Trauma Resusc Emerg Med"},{"key":"R35-20250508","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1080\/17453674.2017.1344459","article-title":"Artificial intelligence for analyzing orthopedic trauma radiographs","volume":"88","author":"Olczak","year":"2017","journal-title":"Acta Orthop"},{"key":"R36-20250508","doi-asserted-by":"crossref","first-page":"1210","DOI":"10.2214\/AJR.16.16078","article-title":"Radiologists\u2019 training, experience, and attitudes about elder abuse detection","volume":"207","author":"Rosen","year":"2016","journal-title":"AJR Am J Roentgenol"},{"key":"R37-20250508","doi-asserted-by":"crossref","first-page":"1042","DOI":"10.2214\/AJR.14.13020","article-title":"Regional variation in Medicare payments for medical imaging: radiologists versus nonradiologists","volume":"204","author":"Rosman","year":"2015","journal-title":"AJR Am J Roentgenol"},{"key":"R38-20250508","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1093\/biostatistics\/kxi011","article-title":"A marginal model approach for analysis of multi-reader multi-test receiver operating characteristic (ROC) data","volume":"6","author":"Song","year":"2005","journal-title":"Biostatistics"},{"key":"R39-20250508","doi-asserted-by":"crossref","first-page":"1195","DOI":"10.1007\/s00198-020-05769-3","article-title":"Long-term direct and indirect economic burden associated with osteoporotic fracture in US postmenopausal women","volume":"32","author":"Tran","year":"2021","journal-title":"Osteoporos Int"},{"key":"R40-20250508","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1007\/s00256-018-3016-3","article-title":"Detecting intertrochanteric hip fractures with orthopedist-level accuracy using a deep convolutional neural network","volume":"48","author":"Urakawa","year":"2019","journal-title":"Skeletal Radiol"}],"container-title":["Clinical Orthopaedics &amp; Related Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.lww.com\/10.1097\/CORR.0000000000002385","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,5,8]],"date-time":"2025-05-08T11:05:04Z","timestamp":1746702304000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.lww.com\/10.1097\/CORR.0000000000002385"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,9]]},"references-count":36,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023]]}},"URL":"https:\/\/doi.org\/10.1097\/corr.0000000000002385","relation":{},"ISSN":["0009-921X","1528-1132"],"issn-type":[{"value":"0009-921X","type":"print"},{"value":"1528-1132","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,9]]},"assertion":[{"value":"2022-01-04","name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-08-05","name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}}]}}