{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T00:20:38Z","timestamp":1791418838583,"version":"4.3.3"},"reference-count":22,"publisher":"Ovid Technologies (Wolters Kluwer Health)","issue":"12","content-domain":{"domain":["lww.com","ovid.com"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,6,19]]},"abstract":"<jats:sec>\n                    <jats:title>Background:<\/jats:title>\n                    <jats:p>In today\u2019s digital age, patients increasingly rely on online search engines for medical information. The integration of large language models such as GPT-4 into search engines such as Bing raises concerns over the potential transmission of misinformation when patients search for information online regarding spine surgery.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods:<\/jats:title>\n                    <jats:p>\n                      SearchResponse.io, a database that archives People Also Ask (PAA) data from Google, was utilized to determine the most popular patient questions regarding 4 specific spine surgery topics: anterior cervical discectomy and fusion, lumbar fusion, laminectomy, and spinal deformity. Bing\u2019s responses to these questions, along with the cited sources, were recorded for analysis. Two fellowship-trained spine surgeons assessed the accuracy of the answers on a 6-point scale and the completeness of the answers on a 3-point scale. Inaccurate answers were re-queried 2 weeks later. Cited sources were categorized and evaluated against\n                      <jats:italic toggle=\"yes\">Journal of the American Medical Association (JAMA)<\/jats:italic>\n                      benchmark criteria. Interrater reliability was measured with use of the kappa statistic. A linear regression analysis was utilized to explore the relationship between answer accuracy and the type of source, number of sources, and mean\n                      <jats:italic toggle=\"yes\">JAMA<\/jats:italic>\n                      benchmark score.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results:<\/jats:title>\n                    <jats:p>\n                      Bing\u2019s responses to 71 PAA questions were analyzed. The average completeness score was 2.03 (standard deviation [SD], 0.36), and the average accuracy score was 4.49 (SD, 1.10). Among the question topics, spinal deformity had the lowest mean completeness score. Re-querying the questions that initially had answers with low accuracy scores resulted in responses with improved accuracy. Among the cited sources, commercial sources were the most prevalent. The\n                      <jats:italic toggle=\"yes\">JAMA<\/jats:italic>\n                      benchmark score across all sources averaged 2.63. Government sources had the highest mean benchmark score (3.30), whereas social media had the lowest (1.75).\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions:<\/jats:title>\n                    <jats:p>\n                      Bing\u2019s answers were generally accurate and adequately complete, with incorrect responses rectified upon re-querying. The plurality of information was sourced from commercial websites. The type of source, number of sources, and mean\n                      <jats:italic toggle=\"yes\">JAMA<\/jats:italic>\n                      benchmark score were not significantly correlated with answer accuracy. These findings underscore the importance of ongoing evaluation and improvement of large language models to ensure reliable and informative results for patients seeking information regarding spine surgery online amid the integration of these models in the search experience.\n                    <\/jats:p>\n                  <\/jats:sec>","DOI":"10.2106\/jbjs.23.00914","type":"journal-article","created":{"date-parts":[[2024,2,9]],"date-time":"2024-02-09T14:10:39Z","timestamp":1707487839000},"page":"1136-1142","update-policy":"https:\/\/doi.org\/10.1097\/lww.0000000000001000","source":"Crossref","is-referenced-by-count":27,"title":["Assessing the Accuracy and Reliability of AI-Generated Responses to Patient Questions Regarding Spine Surgery"],"prefix":"10.2106","volume":"106","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0611-5963","authenticated-orcid":false,"given":"Viknesh S.","family":"Kasthuri","sequence":"first","affiliation":[{"name":"Warren Alpert Medical School of Brown University, Providence, Rhode Island"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4382-1274","authenticated-orcid":false,"given":"Jacob","family":"Glueck","sequence":"additional","affiliation":[{"name":"Warren Alpert Medical School of Brown University, Providence, Rhode Island"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-0547-3155","authenticated-orcid":false,"given":"Han","family":"Pham","sequence":"additional","affiliation":[{"name":"Warren Alpert Medical School of Brown University, Providence, Rhode Island"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9256-9952","authenticated-orcid":false,"given":"Mohammad","family":"Daher","sequence":"additional","affiliation":[{"name":"Department of Orthopedic Surgery, Warren Alpert Medical School of Brown University, Providence, Rhode Island"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5065-6062","authenticated-orcid":false,"given":"Mariah","family":"Balmaceno-Criss","sequence":"additional","affiliation":[{"name":"Warren Alpert Medical School of Brown University, Providence, Rhode Island"},{"name":"Department of Orthopedic Surgery, Warren Alpert Medical School of Brown University, Providence, Rhode Island"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9633-5336","authenticated-orcid":false,"given":"Christopher L.","family":"McDonald","sequence":"additional","affiliation":[{"name":"Department of Orthopedic Surgery, Warren Alpert Medical School of Brown University, Providence, Rhode Island"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7835-2263","authenticated-orcid":false,"given":"Bassel G.","family":"Diebo","sequence":"additional","affiliation":[{"name":"Warren Alpert Medical School of Brown University, Providence, Rhode Island"},{"name":"Department of Orthopedic Surgery, Warren Alpert Medical School of Brown University, Providence, Rhode Island"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9597-4139","authenticated-orcid":false,"given":"Alan H.","family":"Daniels","sequence":"additional","affiliation":[{"name":"Warren Alpert Medical School of Brown University, Providence, Rhode Island"},{"name":"Department of Orthopedic Surgery, Warren Alpert Medical School of Brown University, Providence, Rhode Island"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"276","published-online":{"date-parts":[[2024,2,9]]},"reference":[{"issue":"1","key":"bib1-20250430","doi-asserted-by":"crossref","first-page":"e9","DOI":"10.2196\/jmir.5729","article-title":"Internet Health Information Seeking and the Patient-Physician Relationship: A Systematic Review","volume":"19","author":"Tan","year":"2017","journal-title":"J Med Internet Res."},{"key":"bib3-20250430","first-page":"5","article-title":"The Representation of Trust in Artificial Intelligence Healthcare Research","volume":"305","author":"Kutza","year":"2023","journal-title":"Stud Health Technol Inform."},{"issue":"12","key":"bib4-20250430","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3571730","article-title":"Survey of Hallucination in Natural Language Generation","volume":"55","author":"Ji","year":"2023","journal-title":"ACM Comput Surv."},{"issue":"6","key":"bib5-20250430","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","article-title":"Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum","volume":"183","author":"Ayers","year":"2023","journal-title":"JAMA Intern Med."},{"key":"bib6-20250430","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1016\/j.ajem.2023.06.010","article-title":"Instructional support on first aid in choking by an artificial intelligence-powered chatbot","volume":"70","author":"Birkun","year":"2023","journal-title":"Am J Emerg Med."},{"issue":"1","key":"bib7-20250430","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1001\/jama.2023.8288","article-title":"Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge","volume":"330","author":"Kanjee","year":"2023","journal-title":"JAMA."},{"issue":"8","key":"bib8-20250430","doi-asserted-by":"crossref","first-page":"NP670","DOI":"10.1093\/asj\/sjad134","article-title":"GPT-4: The Future of Cosmetic Procedure Consultation?","volume":"43","author":"Sun","year":"2023","journal-title":"Aesthet Surg J."},{"key":"bib10-20250430","article-title":"Assessing the Accuracy and Reliability of AI-Generated Medical Responses: An Evaluation of the Chat-GPT Model","author":"Johnson","year":"2023","journal-title":"Res Sq [Preprint]."},{"issue":"15","key":"bib11-20250430","doi-asserted-by":"crossref","first-page":"1244","DOI":"10.1001\/jama.1997.03540390074039","article-title":"Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor\u2014Let the reader and viewer beware","volume":"277","author":"Silberg","year":"1997","journal-title":"JAMA."},{"key":"bib12-20250430","first-page":"100214","article-title":"Modern internet search analytics and spine: what are patients asking and reading online?","volume":"14","author":"Kasthuri","year":"2023","journal-title":"N Am Spine Soc J."},{"issue":"4","key":"bib13-20250430","doi-asserted-by":"crossref","first-page":"1224","DOI":"10.1016\/j.arth.2020.10.024","article-title":"Modern Internet Search Analytics and Total Joint Arthroplasty: What Are Patients Asking and Reading Online?","volume":"36","author":"Shen","year":"2021","journal-title":"J Arthroplasty."},{"issue":"4","key":"bib14-20250430","doi-asserted-by":"crossref","first-page":"553","DOI":"10.1016\/j.igie.2023.10.001","article-title":"Evaluating the performance of ChatGPT in responding to questions about endoscopic procedures for patients","volume":"2","author":"Ali","year":"2023","journal-title":"IGIE."},{"issue":"1","key":"bib15-20250430","doi-asserted-by":"crossref","first-page":"17885","DOI":"10.1038\/s41598-023-45223-y","article-title":"Does ChatGPT Provide Appropriate and Equitable Medical Advice?: A Vignette-Based, Clinical Evaluation Across Care Contexts","volume":"13","author":"Nastasi","year":"2023","journal-title":"Sci Rep."},{"issue":"6","key":"bib16-20250430","doi-asserted-by":"crossref","first-page":"1562","DOI":"10.1111\/opo.13207","article-title":"Assessing the utility of ChatGPT as an artificial intelligence-based large language model for information to answer questions on myopia","volume":"43","author":"Biswas","year":"2023","journal-title":"Ophthalmic Physiol Opt."},{"key":"bib17-20250430","doi-asserted-by":"crossref","first-page":"1176617","DOI":"10.3389\/fonc.2023.1176617","article-title":"AI-based online chat and the future of oncology care: a promising technology or a solution in search of a problem?","volume":"13","author":"Kassab","year":"2023","journal-title":"Front Oncol."},{"issue":"2","key":"bib19-20250430","doi-asserted-by":"crossref","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","year":"2023","journal-title":"PLOS Digit Health."},{"issue":"1","key":"bib20-20250430","doi-asserted-by":"crossref","first-page":"e45312","DOI":"10.2196\/45312","article-title":"How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment","volume":"9","author":"Gilson","year":"2023","journal-title":"JMIR Med Educ."},{"issue":"9","key":"bib21-20250430","doi-asserted-by":"crossref","first-page":"1028","DOI":"10.1001\/jamainternmed.2023.2909","article-title":"Chatbot vs Medical Student Performance on Free-Response Clinical Reasoning Examinations","volume":"183","author":"Strong","year":"2023","journal-title":"JAMA Intern Med."},{"key":"bib22-20250430","article-title":"Performance of ChatGPT, GPT-4, and Google Bard on a Neurosurgery Oral Boards Preparation Question Bank","author":"Ali","year":"2023","journal-title":"Neurosurgery."},{"issue":"7","key":"bib24-20250430","doi-asserted-by":"crossref","first-page":"1195","DOI":"10.1016\/j.arth.2023.04.007","article-title":"Using a Google Web Search Analysis to Assess the Utility of ChatGPT in Total Joint Arthroplasty","volume":"38","author":"Dubin","year":"2023","journal-title":"J Arthroplasty."},{"issue":"11","key":"bib25-20250430","first-page":"342","article-title":"Lumbar spinal stenosis: evaluation of information on the internet","volume":"95","author":"Feller","year":"2012","journal-title":"Med Health R I."},{"issue":"12","key":"bib26-20250430","doi-asserted-by":"crossref","first-page":"1041","DOI":"10.1097\/BRS.0000000000001446","article-title":"Is the Readability of Spine-Related Patient Education Material Improving?: An Assessment of Subspecialty Websites","volume":"41","author":"Eltorai","year":"2016","journal-title":"Spine (Phila Pa 1976)."}],"container-title":["Journal of Bone and Joint Surgery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.ovid.com\/10.2106\/JBJS.23.00914","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T13:44:24Z","timestamp":1787751864000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.ovid.com\/10.2106\/JBJS.23.00914"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,9]]},"references-count":22,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2024]]}},"URL":"https:\/\/doi.org\/10.2106\/jbjs.23.00914","relation":{},"ISSN":["0021-9355","1535-1386"],"issn-type":[{"value":"0021-9355","type":"print"},{"value":"1535-1386","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,9]]}}}