{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T11:20:19Z","timestamp":1792063219971,"version":"4.4.0"},"reference-count":29,"publisher":"Springer Science and Business Media LLC","issue":"12","license":[{"start":{"date-parts":[[2025,9,24]],"date-time":"2025-09-24T00:00:00Z","timestamp":1758672000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,9,24]],"date-time":"2025-09-24T00:00:00Z","timestamp":1758672000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Universit\u00e4tsklinikum Freiburg"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Radiol med"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Purpose<\/jats:title>\n                    <jats:p>To assess the performance of state-of-the-art large language models in classifying vertebral metastasis stability using the Spinal Instability Neoplastic Score (SINS) compared to human experts, and to evaluate the impact of task-specific refinement including in-context learning on their performance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Material and methods<\/jats:title>\n                    <jats:p>This retrospective study analyzed 100 synthetic CT and MRI reports encompassing a broad range of SINS scores. Four human experts (two radiologists and two neurosurgeons) and four large language models (Mistral, Claude, GPT-4 turbo, and GPT-4o) evaluated the reports. Large language models were tested in both generic form and with task-specific refinement. Performance was assessed based on correct SINS category assignment and attributed SINS points.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Human experts demonstrated high median performance in SINS classification (98.5% correct) and points calculation (92% correct), with a median point offset of 0 [0\u20130]. Generic large language models performed poorly with 26\u201363% correct category and 4\u201315% correct SINS points allocation. In-context learning significantly improved chatbot performance to near-human levels (96\u201398\/100 correct for classification, 86\u201395\/100 for scoring, no significant difference to human experts). Refined large language models performed 71\u201385% better in SINS points allocation.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>In-context learning enables state-of-the-art large language models to perform at near-human expert levels in SINS classification, offering potential for automating vertebral metastasis stability assessment. The poor performance of generic large language models highlights the importance of task-specific refinement in medical applications of artificial intelligence.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1007\/s11547-025-02096-7","type":"journal-article","created":{"date-parts":[[2025,9,24]],"date-time":"2025-09-24T09:45:55Z","timestamp":1758707155000},"page":"2073-2080","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["In-context learning enables large language models to achieve human-level performance in spinal instability neoplastic score classification from synthetic CT and MRI reports"],"prefix":"10.1007","volume":"130","author":[{"given":"Maximilian F.","family":"Russe","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marco","family":"Reisert","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anna","family":"Fink","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marc","family":"Hohenhaus","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Julia M.","family":"Nakagawa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Caroline","family":"Wilpert","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carl P.","family":"Simon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elmar","family":"Kotter","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Horst","family":"Urbach","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5881-6043","authenticated-orcid":false,"given":"Alexander","family":"Rau","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,9,24]]},"reference":[{"key":"2096_CR1","doi-asserted-by":"publisher","first-page":"87","DOI":"10.2147\/CLEP.S28339","volume":"4","author":"S Li","year":"2012","unstructured":"Li S, Peng Y, Weinhandl ED et al (2012) Estimated number of prevalent cases of metastatic bone disease in the US adult population. Clin Epidemiol 4:87\u201393. https:\/\/doi.org\/10.2147\/CLEP.S28339","journal-title":"Clin Epidemiol"},{"key":"2096_CR2","doi-asserted-by":"publisher","first-page":"335","DOI":"10.2147\/CLEP.S85496","volume":"7","author":"RK Hernandez","year":"2015","unstructured":"Hernandez RK, Adhia A, Wade SW et al (2015) Prevalence of bone metastases and bone-targeting agent use among solid tumor patients in the United States. Clin Epidemiol 7:335\u2013345. https:\/\/doi.org\/10.2147\/CLEP.S85496","journal-title":"Clin Epidemiol"},{"key":"2096_CR3","doi-asserted-by":"publisher","first-page":"107969","DOI":"10.1155\/2011\/107969","volume":"2011","author":"G Maccauro","year":"2011","unstructured":"Maccauro G, Spinelli MS, Mauro S et al (2011) Physiopathology of spine metastasis. Int J Surg Oncol 2011:107969. https:\/\/doi.org\/10.1155\/2011\/107969","journal-title":"Int J Surg Oncol"},{"key":"2096_CR4","doi-asserted-by":"publisher","first-page":"e720","DOI":"10.1016\/S1470-2045(17)30612-5","volume":"18","author":"DE Spratt","year":"2017","unstructured":"Spratt DE, Beeler WH, de Moraes FY et al (2017) An integrated multidisciplinary algorithm for the management of spinal metastases: an International Spine Oncology Consortium report. Lancet Oncol 18:e720\u2013e730. https:\/\/doi.org\/10.1016\/S1470-2045(17)30612-5","journal-title":"Lancet Oncol"},{"key":"2096_CR5","doi-asserted-by":"publisher","first-page":"3072","DOI":"10.1200\/JCO.2010.34.3897","volume":"29","author":"DR Fourney","year":"2011","unstructured":"Fourney DR, Frangou EM, Ryken TC et al (2011) Spinal instability neoplastic score: an analysis of reliability and validity from the spine oncology study group. J Clin Oncol 29:3072\u20133077. https:\/\/doi.org\/10.1200\/JCO.2010.34.3897","journal-title":"J Clin Oncol"},{"key":"2096_CR6","doi-asserted-by":"publisher","first-page":"213","DOI":"10.6061\/clinics\/2013(02)OA15","volume":"68","author":"WGJ Teixeira","year":"2013","unstructured":"Teixeira WGJ, de Mesquita Coutinho PR, Marchese LD et al (2013) Interobserver agreement for the spine instability neoplastic score varies according to the experience of the evaluator. Clinics 68:213\u2013217. https:\/\/doi.org\/10.6061\/clinics\/2013(02)OA15","journal-title":"Clinics"},{"key":"2096_CR7","doi-asserted-by":"publisher","first-page":"e230970","DOI":"10.1148\/radiol.230970","volume":"308","author":"A Rau","year":"2023","unstructured":"Rau A, Rau S, Zoeller D et al (2023) A context-based chatbot surpasses trained radiologists and generic ChatGPT in following the ACR appropriateness guidelines. Radiology 308:e230970. https:\/\/doi.org\/10.1148\/radiol.230970","journal-title":"Radiology"},{"key":"2096_CR8","doi-asserted-by":"publisher","first-page":"e232561","DOI":"10.1148\/radiol.232561","volume":"309","author":"KS Amin","year":"2023","unstructured":"Amin KS, Davis MA, Doshi R et al (2023) Accuracy of ChatGPT, Google Bard, and Microsoft Bing for simplifying radiology reports. Radiology 309:e232561. https:\/\/doi.org\/10.1148\/radiol.232561","journal-title":"Radiology"},{"key":"2096_CR9","doi-asserted-by":"publisher","first-page":"e232133","DOI":"10.1148\/radiol.232133","volume":"311","author":"A Cozzi","year":"2024","unstructured":"Cozzi A, Pinker K, Hidber A et al (2024) BI-RADS category assignments by GPT-3.5, GPT-4, and Google Bard: a multilanguage study. Radiology 311:e232133. https:\/\/doi.org\/10.1148\/radiol.232133","journal-title":"Radiology"},{"key":"2096_CR10","doi-asserted-by":"publisher","first-page":"e230922","DOI":"10.1148\/radiol.230922","volume":"307","author":"AA Rahsepar","year":"2023","unstructured":"Rahsepar AA, Tavakoli N, Kim GHJ et al (2023) How AI responds to common lung cancer questions: chatgpt versus Google Bard. Radiology 307:e230922. https:\/\/doi.org\/10.1148\/radiol.230922","journal-title":"Radiology"},{"key":"2096_CR11","doi-asserted-by":"publisher","first-page":"e231362","DOI":"10.1148\/radiol.231362","volume":"308","author":"MA Fink","year":"2023","unstructured":"Fink MA, Bischoff A, Fink CA et al (2023) Potential of ChatGPT and GPT-4 for data mining of free-text CT reports on lung cancer. Radiology 308:e231362. https:\/\/doi.org\/10.1148\/radiol.231362","journal-title":"Radiology"},{"key":"2096_CR12","doi-asserted-by":"publisher","first-page":"e241191","DOI":"10.1148\/radiol.241191","volume":"312","author":"LC Adams","year":"2024","unstructured":"Adams LC, Truhn D, Busch F et al (2024) Llama 3 challenges proprietary state-of-the-art large language models in radiology board\u2013style examination questions. Radiology 312:e241191. https:\/\/doi.org\/10.1148\/radiol.241191","journal-title":"Radiology"},{"key":"2096_CR13","doi-asserted-by":"publisher","DOI":"10.1055\/a-2264-5631","author":"MF Russe","year":"2024","unstructured":"Russe MF, Reisert M, Bamberg F, Rau A (2024) Improving the use of LLMs in radiology through prompt engineering: from precision prompts to zero-shot learning. Rofo. https:\/\/doi.org\/10.1055\/a-2264-5631","journal-title":"Rofo"},{"key":"2096_CR14","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1038\/s41746-023-00873-0","volume":"6","author":"B Mesk\u00f3","year":"2023","unstructured":"Mesk\u00f3 B, Topol EJ (2023) The imperative for regulatory oversight of large language models (or generative AI) in healthcare. NPJ Digit Med 6:120. https:\/\/doi.org\/10.1038\/s41746-023-00873-0","journal-title":"NPJ Digit Med"},{"key":"2096_CR15","doi-asserted-by":"publisher","first-page":"e231040","DOI":"10.1148\/radiol.231040","volume":"308","author":"D Ueda","year":"2023","unstructured":"Ueda D, Mitsuyama Y, Takita H et al (2023) Diagnostic performance of ChatGPT from patient history and imaging findings on the diagnosis please quizzes. Radiology 308:e231040. https:\/\/doi.org\/10.1148\/radiol.231040","journal-title":"Radiology"},{"key":"2096_CR16","doi-asserted-by":"publisher","first-page":"e232715","DOI":"10.1148\/radiol.232715","volume":"311","author":"S Krishna","year":"2024","unstructured":"Krishna S, Bhambra N, Bleakney R et al (2024) Evaluation of reliability, repeatability, robustness, and confidence of GPT-3.5 and GPT-4 on a radiology board\u2013style examination. Radiology 311:e232715. https:\/\/doi.org\/10.1148\/radiol.232715","journal-title":"Radiology"},{"key":"2096_CR17","first-page":"1877","volume":"33","author":"TB Brown","year":"2020","unstructured":"Brown TB, Mann B, Ryder N et al (2020) Language models are few-shot learners. Adv Neural Inf Process Syst 33:1877\u20131901","journal-title":"Adv Neural Inf Process Syst"},{"key":"2096_CR18","doi-asserted-by":"publisher","DOI":"10.1093\/dmfr\/twad015","author":"MF Russe","year":"2024","unstructured":"Russe MF, Rau A, Ermer MA et al (2024) A content-aware chatbot based on GPT 4 provides trustworthy recommendations for cone beam computed tomography guidelines in dental imaging. Dentomaxillofacial Radiol. https:\/\/doi.org\/10.1093\/dmfr\/twad015","journal-title":"Dentomaxillofacial Radiol"},{"key":"2096_CR19","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1016\/j.pcad.2024.10.010","volume":"87","author":"TE Workman","year":"2024","unstructured":"Workman TE, Ahmed A, Sheriff HM et al (2024) ChatGPT-4 extraction of heart failure symptoms and signs from electronic health records. Prog Cardiovasc Dis 87:44\u201349. https:\/\/doi.org\/10.1016\/j.pcad.2024.10.010","journal-title":"Prog Cardiovasc Dis"},{"key":"2096_CR20","doi-asserted-by":"publisher","DOI":"10.1093\/bjr\/tqae236","author":"K-L Lee","year":"2024","unstructured":"Lee K-L, Kessler DA, Caglic I et al (2024) Assessing the performance of ChatGPT and Bard\/Gemini against radiologists for PI-RADS classification based on prostate multiparametric MRI text reports. Br J Radiol. https:\/\/doi.org\/10.1093\/bjr\/tqae236","journal-title":"Br J Radiol"},{"key":"2096_CR21","doi-asserted-by":"publisher","first-page":"1390774","DOI":"10.3389\/fradi.2024.1390774","volume":"4","author":"P Fervers","year":"2024","unstructured":"Fervers P, Hahnfeldt R, Kottlors J et al (2024) ChatGPT yields low accuracy in determining LI-RADS scores based on free-text and structured radiology reports in German language. Front Radiol 4:1390774. https:\/\/doi.org\/10.3389\/fradi.2024.1390774","journal-title":"Front Radiol"},{"issue":"11","key":"2096_CR22","first-page":"23299107","volume":"12","author":"Y Nakamura","year":"2023","unstructured":"Nakamura Y, Kikuchi T, Yamagishi Y et al (2023) ChatGPT for automating lung cancer staging: feasibility study on open radiology report dataset. medRxiv 12(11):23299107","journal-title":"medRxiv"},{"key":"2096_CR23","doi-asserted-by":"publisher","first-page":"14215","DOI":"10.1038\/s41598-023-41512-8","volume":"13","author":"MF Russe","year":"2023","unstructured":"Russe MF, Fink A, Ngo H et al (2023) Performance of ChatGPT, human radiologists, and context-aware ChatGPT in identifying AO codes from radiology reports. Sci Rep 13:14215. https:\/\/doi.org\/10.1038\/s41598-023-41512-8","journal-title":"Sci Rep"},{"key":"2096_CR24","doi-asserted-by":"publisher","first-page":"506","DOI":"10.1007\/s00330-024-10902-5","volume":"35","author":"D Horiuchi","year":"2025","unstructured":"Horiuchi D, Tatekawa H, Oura T et al (2025) ChatGPT\u2019s diagnostic performance based on textual vs. visual information compared to radiologists\u2019 diagnostic performance in musculoskeletal radiology. Eur Radiol 35:506\u2013516. https:\/\/doi.org\/10.1007\/s00330-024-10902-5","journal-title":"Eur Radiol"},{"key":"2096_CR25","doi-asserted-by":"publisher","first-page":"e232741","DOI":"10.1148\/radiol.232741","volume":"311","author":"NC Lehnen","year":"2024","unstructured":"Lehnen NC, Dorn F, Wiest IC et al (2024) Data extraction from free-text reports on mechanical thrombectomy in acute ischemic stroke using ChatGPT: a retrospective analysis. Radiology 311:e232741. https:\/\/doi.org\/10.1148\/radiol.232741","journal-title":"Radiology"},{"key":"2096_CR26","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1186\/1748-717X-9-69","volume":"9","author":"CG Fisher","year":"2014","unstructured":"Fisher CG, Schouten R, Versteeg AL et al (2014) Reliability of the spinal instability neoplastic score (SINS) among radiation oncologists: an assessment of instability secondary to spinal metastases. Radiat Oncol 9:69. https:\/\/doi.org\/10.1186\/1748-717X-9-69","journal-title":"Radiat Oncol"},{"key":"2096_CR27","doi-asserted-by":"publisher","first-page":"363","DOI":"10.3348\/kjr.2023.0671","volume":"25","author":"DH Kim","year":"2024","unstructured":"Kim DH, Seo J, Lee JH et al (2024) Automated detection and segmentation of bone metastases on spine MRI using U-net: a multicenter study. Korean J Radiol 25:363\u2013373. https:\/\/doi.org\/10.3348\/kjr.2023.0671","journal-title":"Korean J Radiol"},{"key":"2096_CR28","doi-asserted-by":"publisher","first-page":"e233270","DOI":"10.1148\/radiol.233270","volume":"311","author":"Y Zhou","year":"2024","unstructured":"Zhou Y, Ong H, Kennedy P et al (2024) Evaluating GPT-4V (GPT-4 with vision) on detection of radiologic findings on chest radiographs. Radiology 311:e233270. https:\/\/doi.org\/10.1148\/radiol.233270","journal-title":"Radiology"},{"key":"2096_CR29","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-020-00341-z","volume":"3","author":"S-C Huang","year":"2020","unstructured":"Huang S-C, Pareek A, Seyyedi S et al (2020) Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Digit Med 3:1\u20139. https:\/\/doi.org\/10.1038\/s41746-020-00341-z","journal-title":"NPJ Digit Med"}],"container-title":["La radiologia medica"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11547-025-02096-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11547-025-02096-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11547-025-02096-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,1]],"date-time":"2025-12-01T11:20:26Z","timestamp":1764588026000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11547-025-02096-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,24]]},"references-count":29,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["2096"],"URL":"https:\/\/doi.org\/10.1007\/s11547-025-02096-7","relation":{},"ISSN":["1826-6983"],"issn-type":[{"value":"1826-6983","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,24]]},"assertion":[{"value":"3 October 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 September 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 September 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no relevant financial or non-financial interests to disclose.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Research involving human participants and\/or animals"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}}]}}