{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:43:28Z","timestamp":1785545008376,"version":"3.56.0"},"reference-count":46,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Vertebral compression fractures (VCFs) impose a substantial clinical and health care burden, and their management relies on timely access to evidence-based guidelines. Large language models (LLMs) may help clinicians rapidly obtain guideline-related information, but their performance on VCF guidelines remains unclear.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to evaluate the performance of LLMs, including DeepSeek-R1 and ChatGPT-5, in generating responses consistent with VCF clinical guidelines.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Using the 2024 North American Spine Society VCF clinical guidelines as the reference standard, 34 open-ended and 87 closed-ended questions were submitted to DeepSeek-R1 and ChatGPT-5. Four senior spine surgeons independently rated responses to both closed-ended and open-ended questions using a 5-point Likert scale for accuracy, consistency, self-awareness, and fabrication\/falsification. For open-ended questions, comprehensiveness, clarity, and trust and confidence were additionally assessed. Subgroup analyses were performed by question type, recommendation grade, and VCF subtype, with direct comparisons between models.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n A total of 726 responses were generated for 121 questions. For closed-ended questions, ChatGPT-5 and DeepSeek-R1 showed comparable performance in accuracy (\n P<\/jats:italic>\n =.11), self-awareness (\n P<\/jats:italic>\n =.10), and fabrication\/falsification (\n P<\/jats:italic>\n =.10). DeepSeek-R1 demonstrated better consistency than ChatGPT-5 for both closed-ended and open-ended questions (\n P<\/jats:italic>\n <.001 and\n P<\/jats:italic>\n =.001, respectively). For open-ended questions, the models differed significantly in comprehensiveness (\n P<\/jats:italic>\n =.03) and trust and confidence (\n P<\/jats:italic>\n =.02), but not in accuracy (\n P<\/jats:italic>\n =.42), self-awareness (\n P<\/jats:italic>\n =.22), fabrication\/falsification (\n P<\/jats:italic>\n =.64), or clarity (\n P<\/jats:italic>\n =.48). Closed-ended questions generally outperformed open-ended questions. Responses to grade A-C recommendations outperformed grade I recommendations in accuracy, consistency, and fabrication\/falsification (all\n P<\/jats:italic>\n \u2264.001) but scored lower in self-awareness (\n P<\/jats:italic>\n <.001). No significant differences were observed across VCF subtypes.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Under a standardized clinician-oriented prompting condition, ChatGPT-5 and DeepSeek-R1 showed generally high but variable scores across evaluation dimensions, with important deficiencies remaining, particularly in interventional and surgical treatment recommendations and in questions linked to recommendation grade I. Because these findings were obtained in a controlled prompting setting, caution is warranted when extrapolating them to other query styles, clinical scenarios, or LLMs.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/87816","type":"journal-article","created":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T18:41:40Z","timestamp":1783708900000},"page":"e87816-e87816","source":"Crossref","is-referenced-by-count":0,"title":["Performance of DeepSeek-R1 and ChatGPT-5 in the Generation of North American Spine Society Clinical Guidelines for Adult Vertebral Compression Fractures: Comparative Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-0745-4427","authenticated-orcid":false,"given":"Ruiyuan","family":"Chen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-6995-5718","authenticated-orcid":false,"given":"Yue","family":"Pan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-4010-0243","authenticated-orcid":false,"given":"Minghui","family":"Liang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3271-1953","authenticated-orcid":false,"given":"Aobo","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1245-378X","authenticated-orcid":false,"given":"Ziqian","family":"Ma","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-3022-9281","authenticated-orcid":false,"given":"Yu","family":"Xi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0095-9476","authenticated-orcid":false,"given":"Ning","family":"Fan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5668-9527","authenticated-orcid":false,"given":"Shuo","family":"Yuan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5017-8507","authenticated-orcid":false,"given":"Peng","family":"Du","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5016-858X","authenticated-orcid":false,"given":"Tianyi","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1403-4159","authenticated-orcid":false,"given":"Lei","family":"Zang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,7,10]]},"reference":[{"issue":"3","key":"R1","doi-asserted-by":"publisher","first-page":"465","DOI":"10.1359\/jbmr.061113","article-title":"Incidence and economic burden of osteoporosis-related fractures in the United States, 2005-2025","volume":"22","author":"Burge","journal-title":"J Bone Miner Res"},{"issue":"5","key":"R2","doi-asserted-by":"publisher","DOI":"10.1007\/s11916-020-00849-9","article-title":"Current concepts in the management of vertebral compression fractures","volume":"24","author":"Hoyt","journal-title":"Curr Pain Headache Rep"},{"issue":"2","key":"R3","doi-asserted-by":"publisher","first-page":"317","DOI":"10.3349\/ymj.2018.59.2.317","article-title":"Prevalence of frailty in patients with osteoporotic vertebral compression fracture and its association with numbers of fractures","volume":"59","author":"Kim","journal-title":"Yonsei Med J"},{"issue":"12","key":"R4","doi-asserted-by":"publisher","first-page":"1932","DOI":"10.1016\/j.spinee.2017.07.174","article-title":"An overview of clinical guidelines for the management of vertebral compression fracture: a systematic review","volume":"17","author":"Parreira","journal-title":"Spine J"},{"issue":"3","key":"R5","first-page":"209","volume":"18","author":"Wang","journal-title":"Pain Physician"},{"issue":"4","key":"R6","doi-asserted-by":"publisher","first-page":"817","DOI":"10.1016\/j.mcna.2014.03.007","article-title":"Osteoporosis and its complications","volume":"98","author":"Varacallo","journal-title":"Med Clin North Am"},{"issue":"7","key":"R7","doi-asserted-by":"publisher","first-page":"1384","DOI":"10.1359\/jbmr.2000.15.7.1384","article-title":"Health-related quality of life in postmenopausal women with low BMD with or without prevalent vertebral fractures","volume":"15","author":"Oleksik","journal-title":"J Bone Miner Res"},{"issue":"8","key":"R8","doi-asserted-by":"publisher","first-page":"861","DOI":"10.1007\/s00198-004-1774-3","article-title":"Impact of incident vertebral fractures on health related quality of life (HRQOL) in postmenopausal women with prevalent vertebral fractures","volume":"16","author":"Oleksik","journal-title":"Osteoporos Int"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.2147\/JMDH.S31659","article-title":"Vertebral compression fractures: a review of current management and multimodal therapy","volume":"6","author":"Wong","journal-title":"J Multidiscip Healthc"},{"issue":"7","key":"R10","doi-asserted-by":"publisher","first-page":"815","DOI":"10.1016\/j.amjmed.2022.02.035","article-title":"Diagnosis and management of vertebral compression fracture","volume":"135","author":"Alsoof","journal-title":"Am J Med"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.2196\/75567","article-title":"Evaluating the performance of state-of-the-art artificial intelligence chatbots based on the WHO global guidelines for the prevention of surgical site infection: cross-sectional study","volume":"27","author":"Wang","journal-title":"J Med Internet Res"},{"issue":"2","key":"R12","doi-asserted-by":"publisher","first-page":"e345","DOI":"10.1093\/infdis\/jiae403","article-title":"Breaking boundaries in spinal surgery: GPT-4\u2019s quest to revolutionize surgical site infection management","volume":"231","author":"Zhao","journal-title":"J Infect Dis"},{"issue":"8","key":"R13","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.30320","article-title":"Comparison of ophthalmologist and large language model chatbot responses to online patient eye care questions","volume":"6","author":"Bernstein","journal-title":"JAMA Netw Open"},{"issue":"8","key":"R14","doi-asserted-by":"publisher","first-page":"843","DOI":"10.1089\/end.2023.0696","article-title":"Evaluation of the current status of artificial intelligence for endourology patient education: a blind comparison of ChatGPT and Google Bard against traditional information resources","volume":"38","author":"Connors","journal-title":"J Endourol"},{"issue":"1","key":"R15","doi-asserted-by":"publisher","first-page":"128","DOI":"10.14245\/ns.2347310.655","article-title":"Performance of a large language model in the generation of clinical guidelines for antibiotic prophylaxis in spine surgery","volume":"21","author":"Zaidat","journal-title":"Neurospine"},{"issue":"11","key":"R16","doi-asserted-by":"publisher","first-page":"1684","DOI":"10.1016\/j.spinee.2023.07.015","article-title":"Thromboembolic prophylaxis in spine surgery: an analysis of ChatGPT recommendations","volume":"23","author":"Duey","journal-title":"Spine J"},{"issue":"9","key":"R17","doi-asserted-by":"publisher","first-page":"640","DOI":"10.1097\/BRS.0000000000004915","article-title":"Performance of ChatGPT on NASS clinical guidelines for the diagnosis and treatment of low back pain: a comparison study","volume":"49","author":"Shrestha","journal-title":"Spine (Phila Pa 1976)"},{"issue":"11","key":"R18","doi-asserted-by":"publisher","first-page":"4182","DOI":"10.1007\/s00586-024-08198-6","article-title":"ChatGPT versus NASS clinical guidelines for degenerative spondylolisthesis: a comparative analysis","volume":"33","author":"Ahmed","journal-title":"Eur Spine J"},{"issue":"3","key":"R19","doi-asserted-by":"publisher","first-page":"385","DOI":"10.3171\/2024.4.SPINE231148","article-title":"An analysis of ChatGPT recommendations for the diagnosis and treatment of cervical radiculopathy","volume":"41","author":"Hoang","journal-title":"J Neurosurg Spine"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01258-7","article-title":"A framework for human evaluation of large language models in healthcare derived from literature review","volume":"7","author":"Tam","journal-title":"NPJ Digit Med"},{"issue":"7624","key":"R21","doi-asserted-by":"publisher","first-page":"806","DOI":"10.1136\/bmj.39335.541782.AD","article-title":"Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies","volume":"335","author":"von Elm","journal-title":"BMJ"},{"key":"R22","unstructured":"Evidence-based clinical guidelines multidisciplinary spine care: diagnosis and treatment of adults with osteoporotic vertebral compression fractures. North American Spine Society; 2024. URL: https:\/\/www.spine.org\/Portals\/0\/assets\/downloads\/ResearchClinicalCare\/Guidelines\/Osteoporotic-Vertebral-Compression-Fractures.pdf [Accessed 15-08-2025]"},{"key":"R23","unstructured":"Evidence-based clinical guidelines for multidisciplinary spine care: diagnosis and treatment of adults with neoplastic vertebral fractures. North American Spine Society; 2024. URL: https:\/\/www.spine.org\/Portals\/0\/assets\/downloads\/ResearchClinicalCare\/Guidelines\/Neoplastic-Vertebral-Fractures.pdf [Accessed 15-08-2025]"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.2196\/50638","article-title":"Prompt engineering as an important emerging skill for medical professionals: tutorial","volume":"25","author":"Mesk\u00f3","journal-title":"J Med Internet Res"},{"issue":"2","key":"R25","doi-asserted-by":"publisher","DOI":"10.1002\/imt2.178","article-title":"ChatGPT and generative AI are revolutionizing the scientific community: a Janus-faced conundrum","volume":"3","author":"Pu","journal-title":"Imeta"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.3389\/fmed.2024.1504532","article-title":"A guide to prompt design: foundations and applications for healthcare simulationists","volume":"11","author":"Maaz","journal-title":"Front Med (Lausanne)"},{"issue":"1","key":"R27","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-025-02213-z","article-title":"Performance of DeepSeek-R1 and ChatGPT-4o on the Chinese national medical licensing examination: a comparative study","volume":"49","author":"Wu","journal-title":"J Med Syst"},{"issue":"6","key":"R28","doi-asserted-by":"publisher","first-page":"4056","DOI":"10.1097\/JS9.0000000000002386","article-title":"DeepSeek-R1 and GPT-4 are comparable in a complex diagnostic challenge: a historical control study","volume":"111","author":"Chan","journal-title":"Int J Surg"},{"issue":"1","key":"R29","doi-asserted-by":"publisher","DOI":"10.1186\/s13018-025-05831-y","article-title":"Assessment of ChatGPT\u2019s adherence to evidence-based clinical practice guidelines for plantar fasciitis management","volume":"20","author":"Zhang","journal-title":"J Orthop Surg Res"},{"issue":"2","key":"R30","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A guideline of selecting and reporting intraclass correlation coefficients for reliability research","volume":"15","author":"Koo","journal-title":"J Chiropr Med"},{"issue":"1","key":"R31","first-page":"44","volume":"94","author":"McCarthy","journal-title":"Am Fam Physician"},{"key":"R32","doi-asserted-by":"publisher","DOI":"10.1227\/NEU.0000000000000947","article-title":"Management of the elderly with vertebral compression fractures","volume":"77 Suppl 4","author":"Goldstein","journal-title":"Neurosurgery"},{"issue":"1","key":"R33","doi-asserted-by":"publisher","first-page":"149","DOI":"10.14245\/ns.2347052.526","article-title":"Use of ChatGPT for determining clinical and surgical treatment of lumbar disc herniation with radiculopathy: a North American Spine Society guideline comparison","volume":"21","author":"Mejia","journal-title":"Neurospine"},{"issue":"3","key":"R34","doi-asserted-by":"publisher","first-page":"222","DOI":"10.2519\/jospt.2024.12151","article-title":"Performance of ChatGPT compared to clinical practice guidelines in making informed decisions for lumbosacral radicular pain: a cross-sectional study","volume":"54","author":"Gianola","journal-title":"J Orthop Sports Phys Ther"},{"key":"R35","doi-asserted-by":"publisher","DOI":"10.2196\/47479","article-title":"Reliability of medical information provided by ChatGPT: assessment against clinical guidelines and patient information quality instrument","volume":"25","author":"Walker","journal-title":"J Med Internet Res"},{"issue":"2","key":"R36","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1016\/j.arthro.2024.07.040","article-title":"Currently available large language models do not provide musculoskeletal treatment recommendations that are concordant with evidence-based clinical practice guidelines","volume":"41","author":"Nwachukwu","journal-title":"Arthroscopy"},{"issue":"8","key":"R37","doi-asserted-by":"publisher","first-page":"1215","DOI":"10.1093\/ecco-jcc\/jjae040","article-title":"Accuracy of information given by ChatGPT for patients with inflammatory bowel disease in relation to ECCO guidelines","volume":"18","author":"Sciberras","journal-title":"J Crohns Colitis"},{"key":"R38","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2025.106013","article-title":"Clinical feasibility of AI Doctors: evaluating the replacement potential of large language models in outpatient settings for central nervous system tumors","volume":"203","author":"Pan","journal-title":"Int J Med Inform"},{"issue":"1","key":"R39","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1136\/ard-2024-226202","article-title":"Assessing the performance of AI chatbots in answering patients\u2019 common questions about low back pain","volume":"84","author":"Scaff","journal-title":"Ann Rheum Dis"},{"issue":"6630","key":"R40","doi-asserted-by":"publisher","DOI":"10.1126\/science.adg7879","article-title":"ChatGPT is fun, but not an author","volume":"379","author":"Thorp","journal-title":"Science"},{"issue":"10","key":"R41","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.36483","article-title":"Accuracy and reliability of chatbot responses to physician questions","volume":"6","author":"Goodman","journal-title":"JAMA Netw Open"},{"issue":"1","key":"R42","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-02071-6","article-title":"Uncertainty-aware large language models for explainable disease diagnosis","volume":"8","author":"Zhou","journal-title":"NPJ Digit Med"},{"issue":"3","key":"R43","doi-asserted-by":"publisher","first-page":"479","DOI":"10.1002\/art.42737","article-title":"Doctor versus artificial intelligence: patient and physician evaluation of large language model responses to rheumatology patient questions in a cross-sectional study","volume":"76","author":"Ye","journal-title":"Arthritis Rheumatol"},{"key":"R44","doi-asserted-by":"publisher","DOI":"10.3389\/fdgth.2025.1659134","article-title":"Large language models in real-world clinical workflows: a systematic review of applications and implementation","volume":"7","author":"Artsi","journal-title":"Front Digit Health"},{"key":"R45","unstructured":"Ethics and governance of artificial intelligence for health: guidance on large multi-modal models. World Health Organization; 2024. URL: https:\/\/iris.who.int\/server\/api\/core\/bitstreams\/e9e62c65-6045-481e-bd04-20e206bc5039\/content [Accessed 20-06-2026]"},{"issue":"12","key":"R46","doi-asserted-by":"publisher","first-page":"1123","DOI":"10.3348\/kjr.2025.1522","article-title":"Minimum reporting items for CLEAR evaluation of accuracy reports of large language models in healthcare (MI-CLEAR-LLM): 2025 updates","volume":"26","author":"Park","journal-title":"Korean J Radiol"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2026\/1\/e87816","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:25:07Z","timestamp":1785536707000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e87816"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,10]]},"references-count":46,"URL":"https:\/\/doi.org\/10.2196\/87816","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,10]]},"article-number":"v28i3e87816"}}