{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,3]],"date-time":"2026-09-03T03:49:59Z","timestamp":1788407399989,"version":"build-2803163510"},"reference-count":22,"publisher":"SAGE Publications","issue":"4","license":[{"start":{"date-parts":[[2025,4,1]],"date-time":"2025-04-01T00:00:00Z","timestamp":1743465600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"},{"start":{"date-parts":[[2025,4,1]],"date-time":"2025-04-01T00:00:00Z","timestamp":1743465600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Orthopaedic Journal of Sports Medicine"],"published-print":{"date-parts":[[2025,4]]},"abstract":"<jats:sec>\n                    <jats:title>Background:<\/jats:title>\n                    <jats:p>Large language model (LLM)\u2013based chatbots have shown potential in providing health information and patient education. However, the reliability of these chatbots in offering medical advice for specific conditions like Achilles tendinopathy remains uncertain. Mixed outcomes in the field of orthopaedics highlight the need for further examination of these chatbots\u2019 reliability.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Hypothesis:<\/jats:title>\n                    <jats:p>Three leading LLM-based chatbots can provide accurate and complete responses to inquiries related to Achilles tendinopathy.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Study Design:<\/jats:title>\n                    <jats:p>Cross-sectional study.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods:<\/jats:title>\n                    <jats:p>Eighteen questions derived from the Dutch clinical guideline on Achilles tendinopathy were posed to 3 leading LLM-based chatbots: ChatGPT 4.0, Claude 2, and Gemini. The responses were incorporated into an online survey assessed by orthopaedic surgeons specializing in Achilles tendinopathy. Responses were evaluated using a 4-point scoring system, where 1 indicates unsatisfactory and 4 indicates excellent. The total scores for the 18 responses were aggregated for each rater and compared across the chatbots. The intraclass correlation coefficient was calculated to assess consistency among the raters\u2019 evaluations.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results:<\/jats:title>\n                    <jats:p>\n                      Thirteen specialists from 9 diverse countries and regions participated. Analysis showed no significant difference in the mean total scores among the chatbots: ChatGPT (59.7 \u00b1 5.5), Claude 2 (53.4 \u00b1 9.7), and Gemini (53.6 \u00b1 8.4). The proportions of unsatisfactory responses (score 1) were low and comparable across chatbots: 0.9% for ChatGPT 4.0, 3.4% for Claude 2, and 3.4% for Gemini. In terms of excellent responses (score 4), ChatGPT 4.0 outperformed the others, with 43.6% of the responses rated as excellent, significantly higher than Claude 2 at 27.4% and Gemini at 25.2% (\n                      <jats:italic toggle=\"yes\">P<\/jats:italic>\n                      &lt; .001 for both comparisons). Intraclass correlation coefficients indicated poor reliability for ChatGPT 4.0 (0.420) and moderate reliability for Claude 2 (0.522) and Gemini (0.575).\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion:<\/jats:title>\n                    <jats:p>While LLM-based chatbots such as ChatGPT 4.0 can deliver high-quality responses to queries regarding Achilles tendinopathy, the inconsistency among specialist evaluations and the absence of standardized assessment criteria significantly challenge our ability to draw definitive conclusions. These issues underscore the need for a cautious and standardized approach when considering the integration of LLM-based chatbots into clinical settings.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/23259671251332596","type":"journal-article","created":{"date-parts":[[2025,4,30]],"date-time":"2025-04-30T20:35:48Z","timestamp":1746045348000},"update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":5,"title":["Are Large Language Model\u2013Based Chatbots Effective in Providing Reliable Medical Advice for Achilles Tendinopathy? An International Multispecialist Evaluation"],"prefix":"10.1177","volume":"13","author":[{"given":"Zuru","family":"Liang","sequence":"first","affiliation":[{"name":"Department of Orthopaedics and Traumatology, The Chinese University of Hong Kong, Hong Kong, SAR, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ming","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Orthopaedics and Traumatology, The Chinese University of Hong Kong, Hong Kong, SAR, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nasef Mohamed Nasef","family":"Abdelatif","sequence":"additional","affiliation":[{"name":"DrNasef OrthoClinic for Foot and Ankle Orthopedic Disorders, Cairo, Egypt"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marut","family":"Arunakul","sequence":"additional","affiliation":[{"name":"Department of Orthopedic Surgery, Faculty of Medicine, Thammasat University, Pathumthani, Thailand"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carlo Angelo V","family":"Borbon","sequence":"additional","affiliation":[{"name":"Makati Medical Center, Makati, Philippines"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Keen Wai","family":"Chong","sequence":"additional","affiliation":[{"name":"Duke-NUS Graduate Medical School, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Man Wai","family":"Chow","sequence":"additional","affiliation":[{"name":"Department of Orthopaedics and Traumatology, The Chinese University of Hong Kong, Hong Kong, SAR, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yinghui","family":"Hua","sequence":"additional","affiliation":[{"name":"Department of Sports Medicine, Huashan Hospital, Fudan University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Oji","sequence":"additional","affiliation":[{"name":"Foot and Ankle Surgery, Department of Orthopaedic Surgery, Stanford University School of Medicine, Redwood City, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ximena","family":"Ahumada","sequence":"additional","affiliation":[{"name":"Hospital Cl\u00ednico San Borja Arriar\u00e1n, Santiago, Chile"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kwai Ming","family":"Siu","sequence":"additional","affiliation":[{"name":"Department of Orthopaedics and Traumatology, Princess Margaret Hospital, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ken Jin","family":"Tan","sequence":"additional","affiliation":[{"name":"OrthoSports Clinic for Orthopedic Surgery and Sports Medicine, Mt. Elizabeth Novena Specialist Centre, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yasuhito","family":"Tanaka","sequence":"additional","affiliation":[{"name":"Department of Orthopaedic Surgery, Nara Medical University, Kashihara, Nara, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Akira","family":"Taniguchi","sequence":"additional","affiliation":[{"name":"Department of Orthopaedic Surgery, Nara Medical University, Kashihara, Nara, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patrick Shu-Hang","family":"Yung","sequence":"additional","affiliation":[{"name":"Department of Orthopaedics and Traumatology, The Chinese University of Hong Kong, Hong Kong, SAR, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samuel Ka-Kin","family":"Ling","sequence":"additional","affiliation":[{"name":"Department of Orthopaedics and Traumatology, The Chinese University of Hong Kong, Hong Kong, SAR, China"},{"name":"Investigation performed at The Chinese University of Hong Kong, Hong Kong"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2025,4,30]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.30320"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230582"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1097\/j.pain.0000000000002720"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2023.23906"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1136\/bjsports-2020-103867"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.36997"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1177\/0033354919874074"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.3201"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-023-02656-2"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.arthro.2024.02.039"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.arthro.2024.01.017"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00167-023-07529-2"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2021.6986"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.arthro.2023.12.009"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ebiom.2023.104770"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.2106\/JBJS.23.00209"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1177\/03635465211018217"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.46721"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1136\/bjsports-2019-101872"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12891-022-06090-2"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1186\/s43019-024-00218-5"}],"container-title":["Orthopaedic Journal of Sports Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/23259671251332596","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/23259671251332596","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/23259671251332596","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T00:27:16Z","timestamp":1777595236000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/23259671251332596"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4]]},"references-count":22,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,4]]}},"alternative-id":["10.1177\/23259671251332596"],"URL":"https:\/\/doi.org\/10.1177\/23259671251332596","relation":{},"ISSN":["2325-9671","2325-9671"],"issn-type":[{"value":"2325-9671","type":"print"},{"value":"2325-9671","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4]]},"article-number":"23259671251332596"}}