{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,12]],"date-time":"2026-07-12T03:10:59Z","timestamp":1783825859486,"version":"3.55.0"},"reference-count":54,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T00:00:00Z","timestamp":1714348800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objectives<\/jats:title><jats:p>To evaluate demographic biases in diagnostic accuracy and health advice between generative artificial intelligence (AI) (ChatGPT GPT-4) and traditional symptom checkers like WebMD.<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and Methods<\/jats:title><jats:p>Combination symptom and demographic vignettes were developed for 27 most common symptom complaints. Standardized prompts, written from a patient perspective, with varying demographic permutations of age, sex, and race\/ethnicity were entered into ChatGPT (GPT-4) between July and August 2023. In total, 3 runs of 540 ChatGPT prompts were compared to the corresponding WebMD Symptom Checker output using a mixed-methods approach. In addition to diagnostic correctness, the associated text generated by ChatGPT was analyzed for readability (using Flesch-Kincaid Grade Level) and qualitative aspects like disclaimers and demographic tailoring.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>ChatGPT matched WebMD in 91% of diagnoses, with a 24% top diagnosis match rate. Diagnostic accuracy was not significantly different across demographic groups, including age, race\/ethnicity, and sex. ChatGPT's urgent care recommendations and demographic tailoring were presented significantly more to 75-year-olds versus 25-year-olds (P\u2009&amp;lt;\u2009.01) but were not statistically different among race\/ethnicity and sex groups. The GPT text was suitable for college students, with no significant demographic variability.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>The use of non-health-tailored generative AI, like ChatGPT, for simple symptom-checking functions provides comparable diagnostic accuracy to commercially available symptom checkers and does not demonstrate significant demographic bias in this setting. The text accompanying differential diagnoses, however, suggests demographic tailoring that could potentially introduce bias.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>These results highlight the need for continued rigorous evaluation of AI-driven medical platforms, focusing on demographic biases to ensure equitable care.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocae086","type":"journal-article","created":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T05:29:21Z","timestamp":1714368561000},"page":"2002-2009","source":"Crossref","is-referenced-by-count":17,"title":["Mixed methods assessment of the influence of demographics on medical advice of ChatGPT"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8586-450X","authenticated-orcid":false,"given":"Katerina","family":"Andreadis","sequence":"first","affiliation":[{"name":"Department of Population Health, NYU Grossman School of Medicine , New York, NY 10016, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Devon R","family":"Newman","sequence":"additional","affiliation":[{"name":"Department of Population Health, NYU Grossman School of Medicine , New York, NY 10016, United States"},{"name":"Brown University , Providence, RI 02912, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chelsea","family":"Twan","sequence":"additional","affiliation":[{"name":"Department of Population Health, NYU Grossman School of Medicine , New York, NY 10016, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amelia","family":"Shunk","sequence":"additional","affiliation":[{"name":"Department of Population Health, NYU Grossman School of Medicine , New York, NY 10016, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Devin M","family":"Mann","sequence":"additional","affiliation":[{"name":"Department of Population Health, NYU Grossman School of Medicine , New York, NY 10016, United States"},{"name":"Medical Center Information Technology, NYU Langone Health , New York, NY 10016, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elizabeth R","family":"Stevens","sequence":"additional","affiliation":[{"name":"Department of Population Health, NYU Grossman School of Medicine , New York, NY 10016, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,4,29]]},"reference":[{"key":"2024082207520289700_ocae086-B1","doi-asserted-by":"crossref","first-page":"h3727","DOI":"10.1136\/bmj.h3727","article-title":"Fifty million people use computerised self triage","volume":"351","author":"Wyatt","year":"2015","journal-title":"BMJ"},{"issue":"23","key":"2024082207520289700_ocae086-B2","doi-asserted-by":"crossref","first-page":"2367","DOI":"10.1001\/jama.2020.4263","article-title":"Addressing medical misinformation in the patient-clinician relationship","volume":"324","author":"Arora","year":"2020","journal-title":"JAMA"},{"issue":"6","key":"2024082207520289700_ocae086-B3","doi-asserted-by":"crossref","first-page":"e0234663","DOI":"10.1371\/journal.pone.0234663","article-title":"Studying health-related internet and mobile device use using web logs and smartphone records","volume":"15","author":"Bach","year":"2020","journal-title":"PLoS One"},{"key":"2024082207520289700_ocae086-B4","author":"Pew Research","year":"2023"},{"issue":"1","key":"2024082207520289700_ocae086-B5","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1038\/s41746-022-00667-w","article-title":"The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review","volume":"5","author":"Wallace","year":"2022","journal-title":"NPJ Digit Med"},{"issue":"3","key":"2024082207520289700_ocae086-B6","doi-asserted-by":"crossref","first-page":"e19713","DOI":"10.2196\/19713","article-title":"Patients\u2019 utilization and perception of an artificial intelligence\u2013based symptom assessment and advice technology in a British primary care waiting room: exploratory pilot study","volume":"7","author":"Miller","year":"2020","journal-title":"JMIR Hum Factors"},{"issue":"9","key":"2024082207520289700_ocae086-B7","doi-asserted-by":"crossref","first-page":"1379","DOI":"10.3390\/jpm13091379","article-title":"Redesigning primary care: the emergence of artificial-intelligence-driven symptom diagnostic tools","volume":"13","author":"Wiedermann","year":"2023","journal-title":"J Pers Med"},{"key":"2024082207520289700_ocae086-B8","doi-asserted-by":"crossref","first-page":"e43803","DOI":"10.2196\/43803","article-title":"Triage and diagnostic accuracy of online symptom checkers: systematic review","volume":"25","author":"Riboli-Sasco","year":"2023","journal-title":"J Med Internet Res"},{"issue":"10","key":"2024082207520289700_ocae086-B9","doi-asserted-by":"crossref","first-page":"e0000132","DOI":"10.1371\/journal.pdig.0000132","article-title":"Artificial intelligence applications used in the clinical response to COVID-19: A scoping review","volume":"1","author":"Mann","year":"2022","journal-title":"PLOS Digit Health"},{"key":"2024082207520289700_ocae086-B10","article-title":"Faithful AI in medicine: a systematic review with large language models and beyond","author":"Xie","journal-title":"medRxiv"},{"issue":"3","key":"2024082207520289700_ocae086-B11","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1097\/01.NEP.0000000000001129","article-title":"ChatGPT: disruptive educational technology","volume":"44","author":"Frith","year":"2023","journal-title":"Nurs Educ Perspect"},{"issue":"6","key":"2024082207520289700_ocae086-B12","doi-asserted-by":"crossref","first-page":"887","DOI":"10.3390\/healthcare11060887","article-title":"ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns","volume":"11","author":"Sallam","year":"2023","journal-title":"Healthcare"},{"key":"2024082207520289700_ocae086-B13","author":"Pew Research Center","year":"2023"},{"issue":"10","key":"2024082207520289700_ocae086-B14","doi-asserted-by":"crossref","first-page":"1459","DOI":"10.1001\/jamaoncol.2023.2954","article-title":"Use of artificial intelligence chatbots for cancer treatment information","volume":"9","author":"Chen","year":"2023","journal-title":"JAMA Oncol"},{"key":"2024082207520289700_ocae086-B15","doi-asserted-by":"crossref","first-page":"e51580","DOI":"10.2196\/51580","article-title":"Evaluation of generative artificial intelligence large language models ChatGPT, Google Bard, and Microsoft Bing Chat in supporting evidence-based dentistry: a comparative mixed-methods study","volume":"25","author":"Giannakopoulos","year":"2023","journal-title":"J Med Internet Res. Published"},{"issue":"1","key":"2024082207520289700_ocae086-B16","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1007\/s00234-023-03252-4","article-title":"Accuracy of ChatGPT generated diagnosis from patient\u2019s medical history and imaging findings in neuroradiology cases","volume":"66","author":"Horiuchi","year":"2023","journal-title":"Neuroradiology."},{"issue":"4","key":"2024082207520289700_ocae086-B17","doi-asserted-by":"crossref","first-page":"24730114231209919","DOI":"10.1177\/24730114231209919","article-title":"Evaluating the quality and usability of artificial intelligence\u2013generated responses to common patient questions in foot and ankle surgery","volume":"8","author":"Anastasio","year":"2023","journal-title":"Foot Ankle Orthop"},{"key":"2024082207520289700_ocae086-B18","article-title":"Assessing ChatGPT\u2019s ability to answer questions pertaining to erectile dysfunction: can our patients trust it? [published online ahead of print November 20, 2023]","author":"Razdan","journal-title":"Int J Impot Res."},{"key":"2024082207520289700_ocae086-B19","doi-asserted-by":"crossref","first-page":"e49995","DOI":"10.2196\/49995","article-title":"Comparison of diagnostic and triage accuracy of Ada Health and WebMD Symptom Checkers, ChatGPT, and Physicians for Patients in an emergency department: clinical data analysis study","volume":"11","author":"Fraser","year":"2023","journal-title":"JMIR Mhealth Uhealth"},{"key":"2024082207520289700_ocae086-B20","article-title":"Artificial intelligence chatbot performance in triage of ophthalmic conditions [published online ahead of print August 9, 2023]","author":"Lyons","journal-title":"Can J Ophthalmol"},{"issue":"2","key":"2024082207520289700_ocae086-B21","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1192\/bjp.2023.136","article-title":"Artificial intelligence and increasing misinformation","volume":"224","author":"Monteith","year":"2023","journal-title":"Br J Psychiatry"},{"issue":"1","key":"2024082207520289700_ocae086-B22","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1038\/s41746-023-00858-z","article-title":"Bias in AI-based models for medical applications: challenges and mitigation strategies","volume":"6","author":"Mittermaier","year":"2023","journal-title":"NPJ Digit Med"},{"issue":"10","key":"2024082207520289700_ocae086-B23","doi-asserted-by":"crossref","first-page":"e0000367","DOI":"10.1371\/journal.pdig.0000367","article-title":"Centering health equity in large language model deployment","volume":"2","author":"Singh","year":"2023","journal-title":"PLOS Digit Health"},{"issue":"1-2","key":"2024082207520289700_ocae086-B24","first-page":"3","article-title":"More human than human: measuring ChatGPT political bias","volume":"198","author":"Motoki","year":"2023"},{"key":"2024082207520289700_ocae086-B25","author":"Hanna"},{"issue":"9","key":"2024082207520289700_ocae086-B26","first-page":"e45911","article-title":"Bias and Inaccuracy in AI Chatbot Ophthalmologist Recommendations","volume":"15","author":"Oca","year":"2023","journal-title":"Published"},{"issue":"1","key":"2024082207520289700_ocae086-B27","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1038\/s41746-023-00939-z","article-title":"Large language models propagate race-based medicine","volume":"6","author":"Omiye","year":"2023","journal-title":"NPJ Digit Med"},{"key":"2024082207520289700_ocae086-B28","author":"WebMD symptom checker","year":"2023"},{"key":"2024082207520289700_ocae086-B29","first-page":"24","author":"Busker","year":"2023"},{"key":"2024082207520289700_ocae086-B30","author":"Salinas","year":"2023"},{"issue":"1","key":"2024082207520289700_ocae086-B31","doi-asserted-by":"crossref","first-page":"17885","DOI":"10.1038\/s41598-023-45223-y","article-title":"A vignette-based evaluation of ChatGPT\u2019s ability to provide appropriate and equitable medical advice across care contexts","volume":"13","author":"Nastasi","year":"2023","journal-title":"Sci Rep"},{"key":"2024082207520289700_ocae086-B32","author":"MayoClinic Symptom Checker","year":"2023"},{"key":"2024082207520289700_ocae086-B33","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1016\/B978-012369378-5\/50025-9","volume-title":"Handbook of Biosurveillance","author":"Wagner","year":"2006"},{"issue":"4","key":"2024082207520289700_ocae086-B34","doi-asserted-by":"crossref","first-page":"3378","DOI":"10.3390\/ijerph20043378","article-title":"diagnostic accuracy of differential-diagnosis lists generated by generative pretrained transformer 3 chatbot for clinical vignettes with common chief complaints: a pilot study","volume":"20","author":"Hirosawa","year":"2023","journal-title":"Int J Environ Res Public Health"},{"key":"2024082207520289700_ocae086-B35","author":"OpenAI"},{"issue":"1-3","key":"2024082207520289700_ocae086-B36","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1016\/j.tcs.2006.05.024","article-title":"An efficient approach for the rank aggregation problem","volume":"359","author":"Dinu","year":"2006","journal-title":"Theor Comput Sci"},{"key":"2024082207520289700_ocae086-B37"},{"issue":"2","key":"2024082207520289700_ocae086-B38","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1191\/1478088706qp063oa","article-title":"Using thematic analysis in psychology","volume":"3","author":"Braun","year":"2006","journal-title":"Qual Res Psychol"},{"key":"2024082207520289700_ocae086-B39","author":"Benoit","year":"2023"},{"key":"2024082207520289700_ocae086-B40","author":"Zack","year":"2023"},{"key":"2024082207520289700_ocae086-B41","author":"Wennberg","year":"2007"},{"issue":"1","key":"2024082207520289700_ocae086-B42","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1080\/08998280.2017.11929552","article-title":"Improving health outcomes through patient education and partnerships with patients","volume":"30","author":"Paterick","year":"2017","journal-title":"Proc (Baylor Univ Med Center)"},{"key":"2024082207520289700_ocae086-B43","doi-asserted-by":"crossref","first-page":"e48966","DOI":"10.2196\/48966","article-title":"ChatGPT vs Google for queries related to dementia and other cognitive decline: comparison of results","volume":"25","author":"Hristidis","year":"2023","journal-title":"J Med Internet Res"},{"key":"2024082207520289700_ocae086-B44","doi-asserted-by":"crossref","first-page":"e1138","DOI":"10.1016\/j.wneu.2023.11.062","article-title":"Information quality and readability: ChatGPT\u2019s responses to the most common questions about spinal cord injury","volume":"181","author":"Temel","year":"2023","journal-title":"World Neurosurg"},{"key":"2024082207520289700_ocae086-B45","doi-asserted-by":"crossref","first-page":"2374373521998847","DOI":"10.1177\/2374373521998847","article-title":"Readability of patient education materials from high-impact medical journals: a 20-year analysis","volume":"8","author":"Rooney","year":"2021","journal-title":"J Patient Exp"},{"key":"2024082207520289700_ocae086-B46","volume-title":"Health Literacy: A Manual for Clinicians","author":"Weiss","year":"2003"},{"key":"2024082207520289700_ocae086-B47","author":"Cai","year":"2023"},{"key":"2024082207520289700_ocae086-B48","doi-asserted-by":"crossref","first-page":"107908","DOI":"10.1016\/j.chb.2023.107908","article-title":"Social media information literacy: conceptualization and associations with information overload, news avoidance and conspiracy mentality","volume":"148","author":"Heiss","year":"2023","journal-title":"Comput Human Behav"},{"issue":"9","key":"2024082207520289700_ocae086-B49","doi-asserted-by":"crossref","first-page":"1324","DOI":"10.1080\/0144929X.2022.2072768","article-title":"Measuring user competence in using artificial intelligence: validity and reliability of artificial intelligence literacy scale","volume":"42","author":"Wang","year":"2023","journal-title":"Behav Inform Technol"},{"issue":"7","key":"2024082207520289700_ocae086-B50","doi-asserted-by":"crossref","first-page":"1472","DOI":"10.1016\/j.acra.2022.10.002","article-title":"Artificial intelligence literacy: developing a multi-institutional infrastructure for AI education","volume":"30","author":"Perchik","year":"2023","journal-title":"Acad Radiol"},{"issue":"5","key":"2024082207520289700_ocae086-B51","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1177\/1524839914538969","article-title":"The CDC Clear Communication Index is a new evidence-based tool to prepare and review health information","volume":"15","author":"Baur","year":"2014","journal-title":"Health Promot Pract"},{"issue":"3","key":"2024082207520289700_ocae086-B52","doi-asserted-by":"crossref","first-page":"395","DOI":"10.1016\/j.pec.2014.05.027","article-title":"Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information","volume":"96","author":"Shoemaker","year":"2014","journal-title":"Patient Educ Couns"},{"key":"2024082207520289700_ocae086-B53","author":"OpenAI API Reference","year":"2023"},{"key":"2024082207520289700_ocae086-B54","author":"Chen","year":"2023"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/2002\/58868232\/ocae086.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/2002\/58868232\/ocae086.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,15]],"date-time":"2024-11-15T16:59:00Z","timestamp":1731689940000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/9\/2002\/7659471"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,29]]},"references-count":54,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,4,29]]},"published-print":{"date-parts":[[2024,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae086","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,9]]},"published":{"date-parts":[[2024,4,29]]}}}