{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T17:13:25Z","timestamp":1783012405377,"version":"3.54.6"},"reference-count":32,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2024,1,4]],"date-time":"2024-01-04T00:00:00Z","timestamp":1704326400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100014276","name":"Pershing Square Sohn Cancer Research Alliance","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100014276","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Pershing Square MIND Prize Awards"},{"name":"Irma T. Hirschl Career Scientist Award"},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["DP5OD021351"],"award-info":[{"award-number":["DP5OD021351"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01AR075585"],"award-info":[{"award-number":["R01AR075585"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["P30AR072581"],"award-info":[{"award-number":["P30AR072581"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Burroughs Welcome Foundation"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,3,22]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Artificial intelligence (AI) chatbots utilizing large language models (LLMs) have recently garnered significant interest due to their ability to generate humanlike responses to user inquiries in an interactive dialog format. While these models are being increasingly utilized to obtain medical information by patients, scientific and medical providers, and trainees to address biomedical questions, their performance may vary from field to field. The opportunities and risks these chatbots pose to the widespread understanding of skeletal health and science are unknown. Here we assess the performance of 3 high-profile LLM chatbots, Chat Generative Pre-Trained Transformer (ChatGPT) 4.0, BingAI, and Bard, to address 30 questions in 3 categories: basic and translational skeletal biology, clinical practitioner management of skeletal disorders, and patient queries to assess the accuracy and quality of the responses. Thirty questions in each of these categories were posed, and responses were independently graded for their degree of accuracy by four reviewers. While each of the chatbots was often able to provide relevant information about skeletal disorders, the quality and relevance of these responses varied widely, and ChatGPT 4.0 had the highest overall median score in each of the categories. Each of these chatbots displayed distinct limitations that included inconsistent, incomplete, or irrelevant responses, inappropriate utilization of lay sources in a professional context, a failure to take patient demographics or clinical context into account when providing recommendations, and an inability to consistently identify areas of uncertainty in the relevant literature. Careful consideration of both the opportunities and risks of current AI chatbots is needed to formulate guidelines for best practices for their use as source of information about skeletal health and biology.<\/jats:p>","DOI":"10.1093\/jbmr\/zjad007","type":"journal-article","created":{"date-parts":[[2024,1,4]],"date-time":"2024-01-04T19:02:15Z","timestamp":1704394935000},"page":"106-115","source":"Crossref","is-referenced-by-count":24,"title":["The performance of artificial intelligence chatbot large language models to address skeletal biology and bone health queries"],"prefix":"10.1093","volume":"39","author":[{"given":"Michelle","family":"Cung","sequence":"first","affiliation":[{"name":"Department of Pathology and Laboratory Medicine, Weill Cornell Medical College , New York, NY 10065 , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1054-2917","authenticated-orcid":false,"given":"Branden","family":"Sosa","sequence":"additional","affiliation":[{"name":"Department of Pathology and Laboratory Medicine, Weill Cornell Medical College , New York, NY 10065 , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"He S","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Pathology and Laboratory Medicine, Weill Cornell Medical College , New York, NY 10065 , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3928-2506","authenticated-orcid":false,"given":"Michelle M","family":"McDonald","sequence":"additional","affiliation":[{"name":"Skeletal Diseases Program, The Garvan Institute of Medical Research , Darlinghurst, 2010 , Australia"},{"name":"St Vincent\u2019s Clinical Campus School of Clinical Medicine, University of New South Wales , Kensington, 2052 , Australia"},{"name":"School of Medicine Science , Faculty of Medicine and Health, , Sydney, 2006 , Australia"},{"name":"The University of Sydney , Faculty of Medicine and Health, , Sydney, 2006 , Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brya G","family":"Matthews","sequence":"additional","affiliation":[{"name":"Department of Molecular Medicine and Pathology, University of Auckland , Auckland, 1142 , New Zealand"},{"name":"Center for Regenerative Medicine and Skeletal Development, School of Dental Medicine, UConn Health , Farmington, CT 06030 , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Annegreet G","family":"Vlug","sequence":"additional","affiliation":[{"name":"Center for Bone Quality , Department of Internal Medicine, , Leiden, 2300 , The Netherlands"},{"name":"Leiden University Medical Center , Department of Internal Medicine, , Leiden, 2300 , The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7284-3467","authenticated-orcid":false,"given":"Erik A","family":"Imel","sequence":"additional","affiliation":[{"name":"Indiana Center for Musculoskeletal Health , Department of Medicine, , Indianapolis, IN 46202 , United States"},{"name":"Indiana University School of Medicine , Department of Medicine, , Indianapolis, IN 46202 , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6015-8147","authenticated-orcid":false,"given":"Marc N","family":"Wein","sequence":"additional","affiliation":[{"name":"Endocrine Unit, Massachusetts General Hospital , Boston, MA 02114 , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emily Margaret","family":"Stein","sequence":"additional","affiliation":[{"name":"Division of Endocrinology, Hospital for Special Surgery , New York, NY 10021 , United States"},{"name":"Metabolic Bone Service, Hospital for Special Surgery , New York, NY 10021 , United States"},{"name":"Research Division, Hospital for Special Surgery , New York, NY 10021 , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9794-8532","authenticated-orcid":false,"given":"Matthew B","family":"Greenblatt","sequence":"additional","affiliation":[{"name":"Department of Pathology and Laboratory Medicine, Weill Cornell Medical College , New York, NY 10065 , United States"},{"name":"Research Division, Hospital for Special Surgery , New York, NY 10021 , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,1,4]]},"reference":[{"key":"2024032223035604900_ref1","volume-title":"GPT-4 Technical Report","author":"OpenAI"},{"key":"2024032223035604900_ref2","article-title":"LaMDA: language models for dialog applications","author":"Thoppilan"},{"issue":"6","key":"2024032223035604900_ref3","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","article-title":"Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum","volume":"183","author":"Ayers","year":"2023","journal-title":"JAMA Intern Med"},{"issue":"8","key":"2024032223035604900_ref4","doi-asserted-by":"crossref","first-page":"939","DOI":"10.1093\/clinchem\/hvad058","article-title":"Assessing the accuracy and clinical utility of ChatGPT in laboratory medicine","volume":"69","author":"Munoz-Zuluaga","year":"2023","journal-title":"Clin Chem"},{"issue":"11","key":"2024032223035604900_ref5","doi-asserted-by":"crossref","first-page":"1238","DOI":"10.1093\/clinchem\/hvad106","article-title":"AI chatbots in clinical laboratory medicine: foundations and trends","volume":"69","author":"Yang","year":"2023","journal-title":"Clin Chem"},{"key":"2024032223035604900_ref6","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1016\/j.urology.2023.05.010","article-title":"ChatGPT performance on the American urological association (AUA) self-assessment study program and the potential influence of artificial intelligence (AI) in urologic training","volume":"177","author":"Deebel","year":"2023","journal-title":"Urology"},{"key":"2024032223035604900_ref7","doi-asserted-by":"crossref","DOI":"10.2196\/45312","article-title":"How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment","volume":"9","author":"Gilson","year":"2023","journal-title":"JMIR Med Educ"},{"issue":"6","key":"2024032223035604900_ref8","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1001\/jamaophthalmol.2023.1144","article-title":"Performance of an artificial intelligence chatbot in ophthalmic knowledge assessment","volume":"141","author":"Mihalache","year":"2023","journal-title":"JAMA Ophthalmol"},{"key":"2024032223035604900_ref9","doi-asserted-by":"crossref","DOI":"10.2196\/40533","article-title":"Usability and credibility of a COVID-19 vaccine Chatbot for young adults and health Workers in the United States: formative mixed methods study","volume":"10","author":"Weeks","year":"2023","journal-title":"JMIR Hum factors"},{"issue":"7","key":"2024032223035604900_ref10","doi-asserted-by":"crossref","DOI":"10.2196\/38418","article-title":"Chatbot-delivered COVID-19 vaccine communication message preferences of young adults and public health Workers in Urban American Communities: qualitative study","volume":"24","author":"Weeks","year":"2022","journal-title":"J Med Internet Res"},{"key":"2024032223035604900_ref11","doi-asserted-by":"crossref","DOI":"10.2196\/40922","article-title":"User-Chatbot conversations during the COVID-19 pandemic: study based on topic modeling and sentiment analysis","volume":"25","author":"Chin","year":"2023","journal-title":"J Med Internet Res"},{"issue":"6","key":"2024032223035604900_ref12","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1001\/jamainternmed.2023.1832","article-title":"Medicine in the era of artificial intelligence: hey chatbot, write me an H&P","volume":"183","author":"Brender","year":"2023","journal-title":"JAMA Intern Med"},{"issue":"7934","key":"2024032223035604900_ref13","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1038\/d41586-022-03479-w","article-title":"Could AI help you to write your next paper?","volume":"611","author":"Hutson","year":"2022","journal-title":"Nature"},{"issue":"4","key":"2024032223035604900_ref14","first-page":"1148","article-title":"The role of ChatGPT in scientific communication: writing better scientific review articles","volume":"13","author":"Huang","year":"2023","journal-title":"Am J Cancer Res"},{"issue":"6","key":"2024032223035604900_ref15","doi-asserted-by":"crossref","first-page":"3170","DOI":"10.1245\/s10434-023-13436-0","article-title":"A ghostwriter for the masses: ChatGPT and the future of writing","volume":"30","author":"Zimmerman","year":"2023","journal-title":"Ann Surg Oncol"},{"key":"2024032223035604900_ref16","doi-asserted-by":"crossref","first-page":"238212052311784","DOI":"10.1177\/23821205231178449","article-title":"Practical applications of ChatGPT in undergraduate medical education","volume":"10","author":"Tsang","year":"2023","journal-title":"J Med Educ Curric Dev"},{"issue":"3","key":"2024032223035604900_ref17","doi-asserted-by":"crossref","first-page":"532","DOI":"10.1038\/d41591-023-00014-w","article-title":"Caution with AI-generated content in biomedicine","volume":"29","author":"Zhavoronkov","year":"2023","journal-title":"Nat Med"},{"issue":"7947","key":"2024032223035604900_ref18","doi-asserted-by":"crossref","first-page":"214","DOI":"10.1038\/d41586-023-00340-6","article-title":"What ChatGPT and generative AI mean for science","volume":"614","author":"Stokel-Walker","year":"2023","journal-title":"Nature"},{"issue":"7730","key":"2024032223035604900_ref19","doi-asserted-by":"crossref","first-page":"254","DOI":"10.1038\/s41586-018-0662-5","article-title":"Resting zone of the growth plate houses a unique class of skeletal stem cells","volume":"563","author":"Mizuhashi","year":"2018","journal-title":"Nature"},{"issue":"5","key":"2024032223035604900_ref20","doi-asserted-by":"crossref","first-page":"1330","DOI":"10.1016\/j.cell.2021.02.002","article-title":"Osteoclasts recycle via osteomorphs during RANKL-stimulated bone resorption","volume":"184","author":"McDonald","year":"2021","journal-title":"Cell"},{"issue":"4","key":"2024032223035604900_ref21","first-page":"e38166","article-title":"A case study demonstrating applications of ChatGPT in the clinical management of treatment-resistant schizophrenia","volume":"15","author":"Galido","year":"2023","journal-title":"Cureus"},{"issue":"1","key":"2024032223035604900_ref22","doi-asserted-by":"crossref","DOI":"10.1038\/s41467-019-14029-w","article-title":"A Wnt-mediated transformation of the bone marrow stromal cell identity orchestrates skeletal regeneration","volume":"11","author":"Matsushita","year":"2020","journal-title":"Nat Commun"},{"issue":"2","key":"2024032223035604900_ref23","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1016\/j.stem.2014.06.008","article-title":"Leptin-receptor-expressing mesenchymal stromal cells represent the main source of bone formed by adult bone marrow","volume":"15","author":"Zhou","year":"2014","journal-title":"Cell Stem Cell"},{"key":"2024032223035604900_ref24","doi-asserted-by":"crossref","first-page":"9","DOI":"10.7554\/eLife.54695","article-title":"Single cell transcriptomics identifies a unique adipose lineage cell population that regulates bone marrow environment","volume":"9","author":"Zhong","year":"2020","journal-title":"elife"},{"issue":"8","key":"2024032223035604900_ref25","doi-asserted-by":"crossref","first-page":"891","DOI":"10.1038\/ncb3570","article-title":"Bone marrow adipocytes promote the regeneration of stem cells and haematopoiesis by secreting SCF","volume":"19","author":"Zhou","year":"2017","journal-title":"Nat Cell Biol"},{"issue":"9","key":"2024032223035604900_ref26","doi-asserted-by":"crossref","first-page":"2471","DOI":"10.1016\/j.cell.2021.03.011","article-title":"The bone microenvironment invigorates metastatic seeds for further dissemination","volume":"184","author":"Zhang","year":"2021","journal-title":"Cell"},{"issue":"7547","key":"2024032223035604900_ref27","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1038\/nature14347","article-title":"The evolutionary history of lethal metastatic prostate cancer","volume":"520","author":"Gundem","year":"2015","journal-title":"Nature"},{"issue":"9","key":"2024032223035604900_ref28","doi-asserted-by":"crossref","first-page":"1072","DOI":"10.1038\/nm.4385","article-title":"Targeting cellular senescence prevents age-related bone loss in mice","volume":"23","author":"Farr","year":"2017","journal-title":"Nat Med"},{"issue":"8","key":"2024032223035604900_ref29","doi-asserted-by":"crossref","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"issue":"8","key":"2024032223035604900_ref30","doi-asserted-by":"crossref","first-page":"1485","DOI":"10.1002\/jbmr.2888","article-title":"A crisis in the treatment of osteoporosis","volume":"31","author":"Khosla","year":"2016","journal-title":"J Bone Miner Res"},{"issue":"2","key":"2024032223035604900_ref31","first-page":"e35179","article-title":"Artificial hallucinations in ChatGPT: implications in scientific writing","volume":"15","author":"Alkaissi","year":"2023","journal-title":"Cureus"},{"issue":"7972","key":"2024032223035604900_ref32","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"}],"container-title":["Journal of Bone and Mineral Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jbmr\/advance-article-pdf\/doi\/10.1093\/jbmr\/zjad007\/56912975\/zjad007.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jbmr\/article-pdf\/39\/2\/106\/57063335\/zjad007.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jbmr\/article-pdf\/39\/2\/106\/57063335\/zjad007.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,22]],"date-time":"2024-03-22T23:07:09Z","timestamp":1711148829000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jbmr\/article\/39\/2\/106\/7505083"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,4]]},"references-count":32,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,1,4]]},"published-print":{"date-parts":[[2024,3,22]]}},"URL":"https:\/\/doi.org\/10.1093\/jbmr\/zjad007","relation":{},"ISSN":["0884-0431","1523-4681"],"issn-type":[{"value":"0884-0431","type":"print"},{"value":"1523-4681","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,2]]},"published":{"date-parts":[[2024,1,4]]}}}