{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T09:55:26Z","timestamp":1790934926709,"version":"4.1.0"},"reference-count":28,"publisher":"Springer Science and Business Media LLC","issue":"9","license":[{"start":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T00:00:00Z","timestamp":1788998400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T00:00:00Z","timestamp":1788998400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Ruhr-Universit\u00e4t Bochum","id":[{"id":"https:\/\/ror.org\/04tsk2644","id-type":"ROR","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Rheumatol Int"],"abstract":"<jats:sec>\n                    <jats:title>Abstract<\/jats:title>\n                    <jats:p>Specialized pregnancy counselling improves maternal and infant outcomes in women with inflammatory rheumatic and musculoskeletal diseases (iRMDs). As such services are not universally available, large language models (LLMs) may increasingly serve as an accessible source of information for patients seeking guidance on family planning, pregnancy, and lactation. This highlights the need to evaluate this software in terms of their adequacy, accuracy and overall reliability. The aim of the analysis is to evaluate the possibility of using LLMs as reliable sources of information regarding female patients suffering from iRMD and going through the period of pregnancy.\u00a054 questions were formulated, which concern the impact and general relationship between pregnancy and rheumatic diseases. The questions were given to four LLMs, ChatGPT, DeepSeek, Gemini and Grok, and the corresponding answers were obtained. The responses were rated across three clinical domains on a 5-point Likert scale (ranging from 1\u2009=\u2009strongly disagree\/very poor to 5\u2009=\u2009strongly agree\/excellent) by ten European rheumatology experts and with the help of appropriate statistical indicators calculated, the reliability of the LLMs was assessed.\u00a02160 items were evaluated. All models demonstrated high performance in \u201cClinical Safety\u201d (Means\u2009&gt;\u20094.10). For \u201cClinical Accuracy\u201d, DeepSeek and ChatGPT scored highest (Median 4.40 for both). Significant deviations were observed in \u201cCompleteness and Relevance\u201d, where DeepSeek dominated (Mean 4.24, 95% CI 4.17\u20134.30) and Gemini showed the lowest performance (Mean 3.46, 95% CI 3.27\u20133.64). Inter-rater agreement was poor to slight (Fleiss\u2019 kappa range: -0.043 to 0.059) and moderate for ICC (range: 0.259 to 0.692).\u00a0LLMs currently exhibit uniformly strong safety behaviors when providing medical information about family planning, pregnancy and lactation in iRMD. However, significant differences exist regarding completeness and relevance, where DeepSeek and ChatGPT are currently the most reliable and complete models for this domain. Furthermore, the variability in responses and low inter-rater agreement highlight that LLMs require further parameterization, are useful tools for routine care, but cannot replace expert medical judgment.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1007\/s00296-026-06271-5","type":"journal-article","created":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T08:51:46Z","timestamp":1789030306000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Large language models show high clinical safety but differences in completeness for reproductive counselling in women with inflammatory rheumatic and musculoskeletal diseases: a comparative expert evaluation"],"prefix":"10.1007","volume":"46","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2602-7677","authenticated-orcid":false,"given":"Diana","family":"Vossen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9475-9362","authenticated-orcid":false,"given":"Xenofon","family":"Baraliakos","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maria","family":"Polyzou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,9,10]]},"reference":[{"issue":"3","key":"6271_CR1","doi-asserted-by":"publisher","first-page":"0299545","DOI":"10.1371\/journal.pone.0299545","volume":"19","author":"L Alzubaidi","year":"2024","unstructured":"Alzubaidi L, Salhi A, Afadhel M, Bai J, Hollman F, Italia K, Pareyon R, Albahri AS, Ouyang C, Santamar\u00eda J, Cutbush K, Gupta A, Abbosh A, Gu Y (2024) Trustworthy deep learning framework for the detection of abnormalities in X-ray shoulder images. PLoS ONE 19(3):0299545. https:\/\/doi.org\/10.1371\/journal.pone.0299545","journal-title":"PLoS ONE"},{"key":"6271_CR2","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2023.1838","author":"J Ayers","year":"2023","unstructured":"Ayers J, Poliak A, Dredze M, Leas E, Zhu Z, Kelley J, Faix D, Goodman A, Longhurst C, Hogarth M, Smith D (2023) Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Intern Med. https:\/\/doi.org\/10.1001\/jamainternmed.2023.1838","journal-title":"JAMA Intern Med"},{"key":"6271_CR3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1177\/1759720X251331529","volume":"17","author":"D Benavent","year":"2025","unstructured":"Benavent D, Venerito V, Michelena X (2025) RAGing ahead in rheumatology: new language model architectures to tame artificial intelligence. Ther Adv Musculoskelet Dis 17:1\u201310. https:\/\/doi.org\/10.1177\/1759720X251331529","journal-title":"Ther Adv Musculoskelet Dis"},{"issue":"1","key":"6271_CR4","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1038\/s43856-024-00717-2","volume":"5","author":"F Busch","year":"2025","unstructured":"Busch F, Hoffmann L, Rueger C et al (2025) Current applications and challenges in large language models for patient care: a systematic review. Commun Med 5(1):26. https:\/\/doi.org\/10.1038\/s43856-024-00717-2","journal-title":"Commun Med"},{"key":"6271_CR5","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01013-y","author":"AP Creagh","year":"2024","unstructured":"Creagh AP, Hamy V, Yuan H, Mertes G, Tomlinson R, Chen WH, Williams R, Llop C, Yee C, Duh MS, Doherty A, Garcia-Gancedo L, Clifton DA (2024) Digital health technologies and machine learning augment patient reported outcomes to remotely characterise rheumatoid arthritis. NPJ Digit Med. https:\/\/doi.org\/10.1038\/s41746-024-01013-y","journal-title":"NPJ Digit Med"},{"issue":"12","key":"6271_CR6","doi-asserted-by":"publisher","first-page":"3593","DOI":"10.1007\/s10067-020-05173-6","volume":"39","author":"Y El Miedany","year":"2020","unstructured":"El Miedany Y, Palmer D (2020) Rheumatology-led pregnancy clinic: enhancing the care of women with rheumatic diseases during pregnancy. Clin Rheumatol 39(12):3593\u20133601. https:\/\/doi.org\/10.1007\/s10067-020-05173-6","journal-title":"Clin Rheumatol"},{"key":"6271_CR7","doi-asserted-by":"publisher","DOI":"10.3389\/fimmu.2024.1453587","author":"W Fu","year":"2024","unstructured":"Fu W, Wang T, Lu Y, Shi T, Yang Q (2024) The role of lactylation in plasma cells and its impact on rheumatoid arthritis pathogenesis: insights from single-cell RNA sequencing and machine learning. Front Immunol. https:\/\/doi.org\/10.3389\/fimmu.2024.1453587","journal-title":"Front Immunol"},{"key":"6271_CR8","volume-title":"Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement among Raters","author":"K Gwet","year":"2014","unstructured":"Gwet K (2014) Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement among Raters, 4th Edition. Advanced Analytics, LLC","edition":"4th Edition"},{"issue":"4","key":"6271_CR9","doi-asserted-by":"publisher","first-page":"325","DOI":"10.3348\/kjr.2023.0965","volume":"25","author":"K Han","year":"2024","unstructured":"Han K, Ryu L (2024) Statistical methods for the analysis of inter-reader agreement among three or more readers. Korean J Radiol 25(4):325\u2013327. https:\/\/doi.org\/10.3348\/kjr.2023.0965","journal-title":"Korean J Radiol"},{"issue":"1","key":"6271_CR10","doi-asserted-by":"publisher","first-page":"350","DOI":"10.1186\/s12911-025-03202-5","volume":"25","author":"T Ke\u00e7eci","year":"2025","unstructured":"Ke\u00e7eci T, Karag\u00f6z B (2025) Can large language models follow guidelines? a comparative study of ChatGPT-4o and DeepSeek AI in clavicle fracture management based on AAOS recommendations. BMC Med Inf Decis Mak 25(1):350. https:\/\/doi.org\/10.1186\/s12911-025-03202-5","journal-title":"BMC Med Inf Decis Mak"},{"issue":"10","key":"6271_CR11","doi-asserted-by":"publisher","first-page":"2701","DOI":"10.1093\/rheumatology\/keae122","volume":"63","author":"AM Kerola","year":"2024","unstructured":"Kerola AM, Palom\u00e4ki A, Laivuori H et al (2024) Patterns of reproductive health in inflammatory rheumatic diseases and other immune-mediated diseases: a nationwide registry study. Rheumatology (Oxford) 63(10):2701\u20132710. https:\/\/doi.org\/10.1093\/rheumatology\/keae122","journal-title":"Rheumatology (Oxford)"},{"issue":"2","key":"6271_CR12","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","volume":"15","author":"TK Koo","year":"2016","unstructured":"Koo TK, Li MY (2016) A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 15(2):155\u2013163","journal-title":"J Chiropr Med"},{"issue":"1","key":"6271_CR13","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1007\/s00296-025-06068-y","volume":"46","author":"P Kremer","year":"2026","unstructured":"Kremer P, Langballe E, Haase I, Bamberger J, Kuhn S, Krusche M, Knitza J (2026) Diagnostic performance of Prof. Valmed, ChatGPT-5 thinking, and openevidence in rheumatology: a comparative evaluation. Rheumatol Int 46(1):31. https:\/\/doi.org\/10.1007\/s00296-025-06068-y","journal-title":"Rheumatol Int"},{"issue":"12","key":"6271_CR14","doi-asserted-by":"publisher","first-page":"269","DOI":"10.1007\/s00296-025-06023-x","volume":"45","author":"H Labinsky","year":"2025","unstructured":"Labinsky H, Klemm P, Graalmann L, Thiele T, Hornig J, Fink D, Morf H, Mucke J, Kiltz U, Pecher AC, Pfeil A, Elling-Audersch C, Bendzuck G, Krusche M, Hueber AJ, Knitza J (2025) Patient experiences, attitudes, and profiles regarding artificial intelligence in rheumatology: a German national cross-sectional survey study. Rheumatol Int 45(12):269. https:\/\/doi.org\/10.1007\/s00296-025-06023-x","journal-title":"Rheumatol Int"},{"key":"6271_CR15","doi-asserted-by":"publisher","DOI":"10.1002\/acr.80014","author":"D Marcy","year":"2026","unstructured":"Marcy D, Sturm K, Skorupa T et al (2026) Reproductive health counseling in rheumatology: gaps, barriers, and patient impact. Arthritis Care Res. https:\/\/doi.org\/10.1002\/acr.80014","journal-title":"Arthritis Care Res"},{"issue":"10","key":"6271_CR16","doi-asserted-by":"publisher","first-page":"677","DOI":"10.1016\/j.annepidem.2017.09.001","volume":"27","author":"A Mitani","year":"2017","unstructured":"Mitani A, Freer P, Nelson K (2017) Summary measures of agreement and association between many raters\u2019 ordinal classifications. Ann Epidemiol 27(10):677\u2013685e4. https:\/\/doi.org\/10.1016\/j.annepidem.2017.09.001","journal-title":"Ann Epidemiol"},{"issue":"3","key":"6271_CR17","doi-asserted-by":"publisher","first-page":"rkaf079","DOI":"10.1093\/rap\/rkaf079","volume":"9","author":"M Polyzou","year":"2025","unstructured":"Polyzou M, Baraliakos X (2025) Estimation of key indicators for bibliometric analysis in the applications of artificial intelligence in rheumatology. Rheumatol Adv Pract 9(3):rkaf079. https:\/\/doi.org\/10.1093\/rap\/rkaf079","journal-title":"Rheumatol Adv Pract"},{"key":"6271_CR18","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1186\/s41927-026-00618-y","volume":"10","author":"M Polyzou","year":"2026","unstructured":"Polyzou M, Baraliakos X (2026) Artificial Intelligence (AI) in rheumatology: a comparative evaluation of the ChatGPT and DeepSeek application. BMC Rheumatol 10:13. https:\/\/doi.org\/10.1186\/s41927-026-00618-y","journal-title":"BMC Rheumatol"},{"issue":"7","key":"6271_CR19","doi-asserted-by":"publisher","first-page":"857","DOI":"10.3899\/jrheum.230139","volume":"50","author":"MH Radhakrishna","year":"2023","unstructured":"Radhakrishna MH, Kayidhi S, Ravindran V (2023) Optimizing care for pregnancy in rheumatic diseases: barriers and potential solutions. J Rheumatol 50(7):857\u2013859. https:\/\/doi.org\/10.3899\/jrheum.230139","journal-title":"J Rheumatol"},{"key":"6271_CR20","doi-asserted-by":"publisher","first-page":"1380148","DOI":"10.3389\/fmed.2024.1380148","volume":"11","author":"A R\u00edos-Hoyo","year":"2024","unstructured":"R\u00edos-Hoyo A, Shan NL, Li A, Pearson AT, Pusztai L, Howard FM (2024) Evaluation of large language models as a diagnostic aid for complex medical cases. Front Med 11:1380148. https:\/\/doi.org\/10.3389\/fmed.2024.1380148","journal-title":"Front Med"},{"issue":"6","key":"6271_CR21","doi-asserted-by":"publisher","first-page":"910","DOI":"10.1016\/j.ard.2025.02.023","volume":"84","author":"L R\u00fcegg","year":"2025","unstructured":"R\u00fcegg L, Pluma A, Hamroun S et al (2025) EULAR recommendations for use of antirheumatic drugs in reproduction, pregnancy, and lactation: 2024 update. Ann Rheum Dis 84(6):910\u2013926. https:\/\/doi.org\/10.1016\/j.ard.2025.02.023","journal-title":"Ann Rheum Dis"},{"key":"6271_CR22","doi-asserted-by":"publisher","first-page":"1759720X2513500","DOI":"10.1177\/1759720X251350087","volume":"17","author":"K Schreiber","year":"2025","unstructured":"Schreiber K, Graversgaard C, Parodis I et al (2025) Impact of conflicting information on the use of antirheumatic drugs in pregnancy and breastfeeding: perspectives of healthcare providers from the global PRAISE survey. Ther Adv Musculoskelet Dis 17:1759720X251350087. https:\/\/doi.org\/10.1177\/1759720X251350087","journal-title":"Ther Adv Musculoskelet Dis"},{"issue":"6","key":"6271_CR23","doi-asserted-by":"publisher","first-page":"3685","DOI":"10.1007\/s10067-026-08073-3","volume":"45","author":"Z Shao","year":"2026","unstructured":"Shao Z, Zhang Y, Cheng B, Zhang F, Zhu X, Wang Z, Wang Y, Xu L, Mu Z (2026) Accuracy and readability of LLM-generated responses for gout management: a benchmark study based on ACR guidelines. Clin Rheumatol 45(6):3685\u20133693. https:\/\/doi.org\/10.1007\/s10067-026-08073-3","journal-title":"Clin Rheumatol"},{"key":"6271_CR24","volume-title":"Handbook of Parametric and Nonparametric Statistical Procedures","author":"D Sheskin","year":"2011","unstructured":"Sheskin D (2011) Handbook of Parametric and Nonparametric Statistical Procedures, 5th Edition. Taylor and Francis Group, LLC","edition":"5th Edition"},{"issue":"2","key":"6271_CR25","doi-asserted-by":"publisher","first-page":"rkaf035","DOI":"10.1093\/rap\/rkaf035","volume":"9","author":"V Venerito","year":"2025","unstructured":"Venerito V, Gupta L, Mileto S, Iannone F, Bilgin E (2025) Ethical challenges and regulatory pathways for artificial intelligence in rheumatology. Rheumatol Adv Pract 9(2):rkaf035. https:\/\/doi.org\/10.1093\/rap\/rkaf035","journal-title":"Rheumatol Adv Pract"},{"key":"6271_CR26","doi-asserted-by":"publisher","first-page":"262","DOI":"10.3390\/life15020262","volume":"15","author":"S Yang","year":"2025","unstructured":"Yang S, Kim Y, Chang MC, Jeon J, Hong K, Yi YG (2025) Assessing the validity, safety, and utility of ChatGPT\u2019s responses for patients with frozen shoulder. Life 15:262. https:\/\/doi.org\/10.3390\/life15020262","journal-title":"Life"},{"key":"6271_CR27","doi-asserted-by":"publisher","DOI":"10.1007\/s11678-024-00836-1","author":"Y Youssef","year":"2024","unstructured":"Youssef Y, Youssef S, Melcher P, Henkelmann R, Osterhoff G, Theopold J (2024) How accurately can ChatGPT 3.5 answer frequently asked questions by patients on glenohumeral osteoarthritis? Obere Extremit\u00e4t. https:\/\/doi.org\/10.1007\/s11678-024-00836-1","journal-title":"Obere Extremit\u00e4t"},{"key":"6271_CR28","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-016-0200-9","author":"A Zapf","year":"2016","unstructured":"Zapf A, Castell S, Morawietz L, Karch A (2016) Measuring inter-rater reliability for nominal data \u2013 which coefficients and confidence intervals are appropriate? BMC Med Res Methodol. https:\/\/doi.org\/10.1186\/s12874-016-0200-9","journal-title":"BMC Med Res Methodol"}],"container-title":["Rheumatology International"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00296-026-06271-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00296-026-06271-5","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00296-026-06271-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T09:09:52Z","timestamp":1790932192000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00296-026-06271-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,10]]},"references-count":28,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2026,9]]}},"alternative-id":["6271"],"URL":"https:\/\/doi.org\/10.1007\/s00296-026-06271-5","relation":{},"ISSN":["1437-160X"],"issn-type":[{"value":"1437-160X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,10]]},"assertion":[{"value":"8 July 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 August 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 September 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors have declared no conflicts of interest.","order":1,"name":"Ethics","label":"Conflict of interest","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This study evaluated the outputs of artificial intelligence language models and did not involve human patients, medical interventions, or the collection of personal health data. The participating expert rheumatologists acted solely as independent raters of the software-generated text in their professional capacity. As this study does not fall under the definition of human subjects research according to the Declaration of Helsinki, formal approval from an Ethics Committee was waived. All participating experts were informed about the study\u2019s purpose, and their voluntary completion of the evaluation constitutes implied consent for their aggregated ratings to be used and published.","order":2,"name":"Ethics","label":"Ethical approval and consent to participate","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":3,"name":"Ethics","label":"Consent for publication","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"ChatGPT (OpenAI) was used to improve the English language of the manuscript. All content was reviewed and approved by the authors. No part of this report is copied, published elsewhere, or generated by AI.\u00a0No part of this report is copied, published elsewhere, or generated by AI.","order":4,"name":"Ethics","label":"Acknowledge ethical AI use","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"258"}}