{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,6]],"date-time":"2026-11-06T06:27:53Z","timestamp":1793946473413,"version":"4.8.0"},"reference-count":30,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2025,4,4]],"date-time":"2025-04-04T00:00:00Z","timestamp":1743724800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Institute Of Arthritis and Musculoskeletal and Skin Diseases (NIAMS) of the National Institutes of Health (NIH)","award":["U19AR076737"],"award-info":[{"award-number":["U19AR076737"]}],"id":[{"id":"https:\/\/ror.org\/006zn3t30","id-type":"ROR","asserted-by":"crossref"}]},{"name":"National Institute Of Arthritis and Musculoskeletal and Skin Diseases (NIAMS) of the National Institutes of Health (NIH)","award":["UH2AR076724"],"award-info":[{"award-number":["UH2AR076724"]}],"id":[{"id":"https:\/\/ror.org\/006zn3t30","id-type":"ROR","asserted-by":"crossref"}]},{"name":"National Institute Of Arthritis and Musculoskeletal and Skin Diseases (NIAMS) of the National Institutes of Health (NIH)","award":["UH2AR076719"],"award-info":[{"award-number":["UH2AR076719"]}],"id":[{"id":"https:\/\/ror.org\/006zn3t30","id-type":"ROR","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Diagnostics"],"abstract":"<jats:p>Background\/Objectives: This study aimed to create a pipeline for standardized data extraction from lumbar-spine MRI radiology reports using a large language model (LLM) and assess the agreement of the extracted data with research-grade semi-quantitative scoring. Methods: We included a subset of data from a multi-site NIH-funded cohort study of chronic low back pain (cLBP) participants. After initial prompt development, a secure application programming interface (API) deployment of OpenAIs GPT-4 was used to extract different classes of pathology from the clinical radiology report. Unsupervised UMAP and agglomerative clustering of the pathology terms\u2019 embeddings provided insight into model comprehension for optimized prompt design. Model extraction was benchmarked against human extraction (gold standard) with F1 scores and false-positive and false-negative rates (FPR\/FNR). Then, an expert MSK radiologist provided comprehensive research-grade scores of the images, and agreement with report-extracted data was calculated using Cohen\u2019s kappa. Results: Data from 230 patients with cLBP were included (mean age 53.2 years, 54% women). The overall model performance for extracting data from clinical reports was excellent, with a mean F1 score of 0.96 across pathologies. The mean FPR was marginally higher than the FNR (5.1% vs. 3.0%). Agreement with comprehensive scoring was moderate (kappa 0.424), and the underreporting of lateral recess stenosis (FNR 63.6%) and overreporting of disc pathology (FPR 42.7%) were noted. Conclusions: LLMs can accurately extract highly detailed information on lumbar spine imaging pathologies from radiology reports. Moderate agreement between the LLM and comprehensive scores underscores the need for less subjective, machine-based data extraction from imaging.<\/jats:p>","DOI":"10.3390\/diagnostics15070930","type":"journal-article","created":{"date-parts":[[2025,4,10]],"date-time":"2025-04-10T11:26:41Z","timestamp":1744284401000},"page":"930","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Information Extraction from Lumbar Spine MRI Radiology Reports Using GPT4: Accuracy and Benchmarking Against Research-Grade Comprehensive Scoring"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9763-2420","authenticated-orcid":false,"given":"Katharina","family":"Ziegeler","sequence":"first","affiliation":[{"name":"Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA 94143, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-7378-1068","authenticated-orcid":false,"given":"Virginie","family":"Kreutzinger","sequence":"additional","affiliation":[{"name":"Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA 94143, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8766-7542","authenticated-orcid":false,"given":"Michelle W.","family":"Tong","sequence":"additional","affiliation":[{"name":"Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA 94143, USA"},{"name":"Department of Bioengineering, University of California Berkeley, Berkeley, CA 94720, USA"},{"name":"Department of Bioengineering, University of California San Francisco, San Francisco, CA 94143, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cynthia T.","family":"Chin","sequence":"additional","affiliation":[{"name":"Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA 94143, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6542-9711","authenticated-orcid":false,"given":"Emma","family":"Bahroos","sequence":"additional","affiliation":[{"name":"Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA 94143, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Po-Hung","family":"Wu","sequence":"additional","affiliation":[{"name":"Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA 94143, USA"},{"name":"The UCSF REACH Center, The Core Center for Patient-Centric Mechanistic Phenotyping in Chronic Low Back Pain, San Francisco, CA 94143, USA"},{"name":"Department of Orthopaedic Surgery, University of California San Francisco, San Francisco, CA 94143, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2191-180X","authenticated-orcid":false,"given":"Noah","family":"Bonnheim","sequence":"additional","affiliation":[{"name":"The UCSF REACH Center, The Core Center for Patient-Centric Mechanistic Phenotyping in Chronic Low Back Pain, San Francisco, CA 94143, USA"},{"name":"Department of Orthopaedic Surgery, University of California San Francisco, San Francisco, CA 94143, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6460-4506","authenticated-orcid":false,"given":"Aaron J.","family":"Fields","sequence":"additional","affiliation":[{"name":"The UCSF REACH Center, The Core Center for Patient-Centric Mechanistic Phenotyping in Chronic Low Back Pain, San Francisco, CA 94143, USA"},{"name":"Department of Orthopaedic Surgery, University of California San Francisco, San Francisco, CA 94143, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeffrey C.","family":"Lotz","sequence":"additional","affiliation":[{"name":"Department of Orthopaedic Surgery, University of California San Francisco, San Francisco, CA 94143, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas M.","family":"Link","sequence":"additional","affiliation":[{"name":"Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA 94143, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0201-871X","authenticated-orcid":false,"given":"Sharmila","family":"Majumdar","sequence":"additional","affiliation":[{"name":"Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA 94143, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,4,4]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/j.berh.2009.11.002","article-title":"Measuring the Global Burden of Low Back Pain","volume":"24","author":"Hoy","year":"2010","journal-title":"Best Pract. Res. Clin. Rheumatol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"487","DOI":"10.14245\/ns.2448372.186","article-title":"Comparative Review of the Socioeconomic Burden of Lower Back Pain in the United States and Globally","volume":"21","author":"Chang","year":"2024","journal-title":"Neurospine"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"482","DOI":"10.1016\/S0140-6736(11)60610-7","article-title":"Non-Specific Low Back Pain","volume":"379","author":"Mannion","year":"2012","journal-title":"Lancet"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"222","DOI":"10.1038\/s41746-024-01219-0","article-title":"A Scoping Review of Large Language Model Based Approaches for Information Extraction from Radiology Reports","volume":"7","author":"Reichenpfader","year":"2024","journal-title":"NPJ Digit. Med."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1418","DOI":"10.1038\/s41467-024-45563-x","article-title":"Structured Information Extraction from Scientific Text with Large Language Models","volume":"15","author":"Dagdelen","year":"2024","journal-title":"Nat. Commun."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"707","DOI":"10.1053\/j.gastro.2023.12.019","article-title":"A Comparison of a Large Language Model vs Manual Chart Review for the Extraction of Data Elements From the Electronic Health Record","volume":"166","author":"Ge","year":"2024","journal-title":"Gastroenterology"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"103985","DOI":"10.1016\/j.ijmedinf.2019.103985","article-title":"Extracting Comprehensive Clinical Information for Breast Cancer Using Deep Learning Methods","volume":"132","author":"Zhang","year":"2019","journal-title":"Int. J. Med. Inf."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"e231362","DOI":"10.1148\/radiol.231362","article-title":"Potential of ChatGPT and GPT-4 for Data Mining of Free-Text CT Reports on Lung Cancer","volume":"308","author":"Fink","year":"2023","journal-title":"Radiology"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Hue, T.F., Lotz, J.C., Zheng, P., Black, D.M., Bailey, J., Ewing, S.K., Fields, A.J., Mehling, W., Scheffler, A., and Strigo, I. (2024). Design of the COMEBACK and BACKHOME Studies, Longitudinal Cohorts for Comprehensive Deep Phenotyping of Adults with Chronic Low-Back Pain (cLBP): A Part of the BACPAC Research Program. MedRxiv Prepr. Serv. Health Sci., 2024.04.09.24305574.","DOI":"10.1101\/2024.04.09.24305574"},{"key":"ref_10","first-page":"S3","article-title":"The Back Pain Consortium (BACPAC) Research Program: Structure, Research Priorities, and Methods","volume":"24","author":"Mauck","year":"2023","journal-title":"Pain Med. Malden Mass"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"S95","DOI":"10.1093\/pm\/pnad008","article-title":"The Back Pain Consortium (BACPAC) Research Program Data Harmonization: Rationale for Data Elements and Standards","volume":"24","author":"Batorsky","year":"2023","journal-title":"Pain Med."},{"key":"ref_12","first-page":"S81","article-title":"Magnetic Resonance Imaging of the Lumbar Spine: Recommendations for Acquisition and Image Evaluation from the BACPAC Spine Imaging Working Group","volume":"24","author":"Sollmann","year":"2023","journal-title":"Pain Med. Malden Mass"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1186\/s13054-023-04425-6","article-title":"Artificial Hallucination: GPT on LSD?","volume":"27","author":"Beutel","year":"2023","journal-title":"Crit. Care"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"861","DOI":"10.21105\/joss.00861","article-title":"UMAP: Uniform Manifold Approximation and Projection","volume":"3","author":"McInnes","year":"2018","journal-title":"J. Open Source Softw."},{"key":"ref_15","unstructured":"McInnes, L., Healy, J., and Melville, J. (2020). UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"857","DOI":"10.1016\/j.jbi.2013.06.013","article-title":"An Ontology-Based Similarity Measure for Biomedical Data\u2014Application to Radiology Reports","volume":"46","author":"Mabotuwana","year":"2013","journal-title":"J. Biomed. Inform."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1037\/h0026256","article-title":"Weighted Kappa: Nominal Scale Agreement with Provision for Scaled Disagreement or Partial Credit","volume":"70","author":"Cohen","year":"1968","journal-title":"Psychol. Bull."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The Measurement of Observer Agreement for Categorical Data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"ooad027","DOI":"10.1093\/jamiaopen\/ooad027","article-title":"Weakly Supervised Spatial Relation Extraction from Radiology Reports","volume":"6","author":"Datta","year":"2023","journal-title":"JAMIA Open"},{"key":"ref_20","first-page":"1079","article-title":"A Fusion NLP Model for the Inference of Standardized Thyroid Nodule Malignancy Scores from Radiology Report Text","volume":"2021","author":"Santos","year":"2021","journal-title":"AMIA Annu. Symp. Proc. AMIA Symp."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1422","DOI":"10.1016\/j.acra.2018.03.008","article-title":"Comparison of Natural Language Processing Rules-Based and Machine-Learning Systems to Identify Lumbar Spine Imaging Findings Related to Low Back Pain","volume":"25","author":"Tan","year":"2018","journal-title":"Acad. Radiol."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Banitalebi, H., Espeland, A., Anvar, M., Hermansen, E., Hellum, C., Brox, J.I., Myklebust, T.\u00c5., Indrekvam, K., Brisby, H., and Weber, C. (2022). Reliability of Preoperative MRI Findings in Patients with Lumbar Spinal Stenosis. BMC Musculoskelet. Disord., 23.","DOI":"10.1186\/s12891-021-04949-4"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"e004044","DOI":"10.1136\/rmdopen-2023-004044","article-title":"Diagnostic Accuracy in Axial Spondyloarthritis: A Systematic Evaluation of the Role of Clinical Information in the Interpretation of Sacroiliac Joint Imaging","volume":"10","author":"Pohlner","year":"2024","journal-title":"RMD Open"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"240","DOI":"10.1148\/radiol.12121502","article-title":"Improving Consistency in Radiology Reporting through the Use of Department-Wide Standardized Structured Reporting","volume":"267","author":"Larson","year":"2013","journal-title":"Radiology"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"541","DOI":"10.2214\/ajr.175.2.1750541","article-title":"Satisfaction of Search in Osteoradiology","volume":"175","author":"Ashman","year":"2000","journal-title":"Am. J. Roentgenol."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"20190043","DOI":"10.1259\/bjr.20190043","article-title":"Fatigue in Radiology: A Fertile Area for Future Research","volume":"92","author":"Stinton","year":"2019","journal-title":"Br. J. Radiol."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1148\/radiol.16142770","article-title":"Natural Language Processing in Radiology: A Systematic Review","volume":"279","author":"Pons","year":"2016","journal-title":"Radiology"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"e230725","DOI":"10.1148\/radiol.230725","article-title":"Leveraging GPT-4 for Post Hoc Transformation of Free-Text Radiology Reports into Structured Reporting: A Multilingual Feasibility Study","volume":"307","author":"Adams","year":"2023","journal-title":"Radiology"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"e240320","DOI":"10.1148\/radiol.240320","article-title":"Optimal Large Language Model Characteristics to Balance Accuracy and Energy Use for Sustainable Medical Applications","volume":"312","author":"Doo","year":"2024","journal-title":"Radiology"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"e231147","DOI":"10.1148\/radiol.231147","article-title":"Feasibility of Using the Privacy-Preserving Large Language Model Vicuna for Labeling Radiology Reports","volume":"309","author":"Mukherjee","year":"2023","journal-title":"Radiology"}],"container-title":["Diagnostics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-4418\/15\/7\/930\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:10:14Z","timestamp":1760029814000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-4418\/15\/7\/930"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,4]]},"references-count":30,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2025,4]]}},"alternative-id":["diagnostics15070930"],"URL":"https:\/\/doi.org\/10.3390\/diagnostics15070930","relation":{},"ISSN":["2075-4418"],"issn-type":[{"value":"2075-4418","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4,4]]}}}