{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T21:04:36Z","timestamp":1790197476481,"version":"4.1.0"},"reference-count":38,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR AI"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Large language models (LLMs) are increasingly deployed in mental health applications, yet growing evidence suggests they encode algorithmic biases that influence clinical outputs. Because these models now mediate patient-facing decisions, such biases carry the potential for direct harm. Whether they systematically affect psychiatric diagnosis across demographic groups remains underexplored.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to examine whether LLMs exhibit implicit demographic biases when generating psychiatric diagnoses.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We developed 1152 synthetic clinical vignettes using a matched-pair design that manipulated gender, race and ethnicity, age, socioeconomic status, English proficiency, and urbanicity while holding clinical content constant. Vignettes were divided into control (unambiguous anorexia nervosa [AN]) and ambiguous conditions designed to permit differential diagnosis. Ten LLM configurations across 5 model families were tested.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Control vignettes produced near-unanimous AN diagnoses (mean 100%, SD 0.1%), while ambiguous vignettes elicited greater variability (mean 23.6%, SD 10.1%). Intermodel agreement was moderate for ambiguous vignettes (Fleiss \u03ba=0.410, 95% CI 0.397\u20100.422). Mixed-effects logistic regression with LLM as a random intercept revealed significant demographic biases: Black patients were over 6 times more likely to receive a major depressive disorder (MDD) diagnosis than White patients with identical presentations (odds ratio [OR] 6.09, 95% CI 5.13\u20107.24), Latine patients were over 9 times more likely (OR 9.57, 95% CI 8.00\u201011.45), and Asian patients were nearly 3 times more likely to receive an AN diagnosis (OR 2.88, 95% CI 2.44\u20103.42). Female patients were less likely than males to be diagnosed with AN (OR 0.43, 95% CI 0.37\u20100.49).<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n These findings demonstrate that LLMs exhibit systematic demographic biases in psychiatric diagnosis even when clinical content is held constant, revealing measurable patterns that can inform improvements to training data, model architecture, and clinical deployment frameworks.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/93498","type":"journal-article","created":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T13:25:06Z","timestamp":1786022706000},"page":"e93498-e93498","source":"Crossref","is-referenced-by-count":0,"title":["Implicit Bias in Large Language Model Diagnosis of Eating Disorders: Experimental Vignette Study"],"prefix":"10.2196","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-4165-2222","authenticated-orcid":false,"given":"Deija","family":"McCalla","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-6151-3461","authenticated-orcid":false,"given":"Bochen","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-1202-1092","authenticated-orcid":false,"given":"Saul","family":"Jaeger","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8574-5380","authenticated-orcid":false,"given":"Justin","family":"Jacques","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1350-0234","authenticated-orcid":false,"given":"Leo","family":"Gonzalez Jr","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8927-0431","authenticated-orcid":false,"given":"Charles","family":"Silber","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7708-1409","authenticated-orcid":false,"given":"Cass","family":"Dykeman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,23]]},"reference":[{"issue":"8","key":"R1","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","journal-title":"Nat Med"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","DOI":"10.1038\/s41398-024-02911-1","article-title":"Optimising the use of electronic medical records for large scale research in psychiatry","volume":"14","author":"Newby","journal-title":"Transl Psychiatry"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.2196\/59479","article-title":"The opportunities and risks of large language models in mental health","volume":"11","author":"Lawrence","journal-title":"JMIR Ment Health"},{"issue":"2","key":"R4","doi-asserted-by":"publisher","DOI":"10.2196\/mental.7785","article-title":"Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial","volume":"4","author":"Fitzpatrick","journal-title":"JMIR Ment Health"},{"issue":"11","key":"R5","doi-asserted-by":"publisher","DOI":"10.2196\/12106","article-title":"An empathy-driven, conversational artificial intelligence agent (Wysa) for digital mental well-being: real-world data evaluation mixed-methods study","volume":"6","author":"Inkster","journal-title":"JMIR Mhealth Uhealth"},{"issue":"1","key":"R6","doi-asserted-by":"publisher","DOI":"10.61888\/2692-4129.1130","article-title":"Evaluating generative AI for depression diagnosis: implications for counselor education and supervision","volume":"6","author":"Rensi","journal-title":"J Technol Couns Educ Superv"},{"issue":"1","key":"R7","doi-asserted-by":"publisher","first-page":"e12","DOI":"10.1016\/S2589-7500(23)00225-X","article-title":"Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study","volume":"6","author":"Zack","journal-title":"Lancet Digit Health"},{"issue":"1","key":"R8","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01746-4","article-title":"Racial bias in AI-mediated psychiatric diagnosis and treatment: a qualitative comparison of four large language models","volume":"8","author":"Bouguettaya","journal-title":"NPJ Digit Med"},{"issue":"3","key":"R9","doi-asserted-by":"publisher","first-page":"330","DOI":"10.1145\/230538.230561","article-title":"Bias in computer systems","volume":"14","author":"Friedman","journal-title":"ACM Trans Inf Syst"},{"issue":"6","key":"R10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3457607","article-title":"A survey on bias and fairness in machine learning","volume":"54","author":"Mehrabi","journal-title":"ACM Comput Surv"},{"issue":"6464","key":"R11","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1126\/science.aax2342","article-title":"Dissecting racial bias in an algorithm used to manage the health of populations","volume":"366","author":"Obermeyer","journal-title":"Science"},{"key":"R12","unstructured":"Bolukbasi T Chang KW Zou JY Saligrama V Kalai AT . Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Presented at: 30th International Conference on Neural Information Processing Systems (NIPS 2016); Dec 5-10, 2016:4349-4357; URL: https:\/\/proceedings.neurips.cc\/paper\/2016\/hash\/a486cd07e4ac3d270571622f4f316ec5-Abstract.html [Accessed 31-08-2026]"},{"issue":"6334","key":"R13","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1126\/science.aal4230","article-title":"Semantics derived automatically from language corpora contain human-like biases","volume":"356","author":"Caliskan","journal-title":"Science"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Dixon L Li J Sorensen J Thain N Vasserman L . Measuring and mitigating unintended bias in text classification. Presented at: AIES \u201918: Proceedings of the 2018 AAAI\/ACM Conference on AI, Ethics, and Society; Feb 2-3, 2018:67-73; New Orleans, LA, USA. URL: https:\/\/dl.acm.org\/doi\/proceedings\/10.1145\/3278721 [Accessed 21-08-2026] [doi: 10.1145\/3278721.3278729]","DOI":"10.1145\/3278721.3278729"},{"key":"R15","unstructured":"Pleiss G Raghavan M Wu F Kleinberg J Weinberger KQ . On fairness and calibration. Presented at: Advances in Neural Information Processing Systems 30 (NIPS 2017); Dec 4-9, 2017:5680-5689; Long Beach, CA, USA. URL: https:\/\/proceedings.neurips.cc\/paper\/2017\/hash\/b8b9c74ac526fffbeb2d39ab038d1cd7-Abstract.html? [Accessed 31-08-2026]"},{"issue":"10","key":"R16","doi-asserted-by":"publisher","first-page":"1359","DOI":"10.1377\/hlthaff.2023.00553","article-title":"Mitigating racial and ethnic bias and advancing health equity in clinical algorithms: a scoping review","volume":"42","author":"Cary","journal-title":"Health Aff (Millwood)"},{"key":"R17","doi-asserted-by":"crossref","unstructured":"Bender EM Gebru T McMillan-Major A Shmitchell S . On the dangers of stochastic parrots: can language models be too big. Presented at: ACM Conference on Fairness, Accountability, and Transparency (FAccT '21); Mar 3-10, 2021:610-623; Virtual Event, Toronto, Canada. [doi: 10.1145\/3442188.3445922]","DOI":"10.1145\/3442188.3445922"},{"issue":"4","key":"R18","doi-asserted-by":"publisher","first-page":"395","DOI":"10.1176\/appi.ajp.2014.14020138","article-title":"The NIMH Research Domain Criteria (RDoC) Project: precision medicine for psychiatry","volume":"171","author":"Insel","journal-title":"Am J Psychiatry"},{"issue":"2","key":"R19","doi-asserted-by":"publisher","first-page":"130","DOI":"10.1176\/appi.ps.201800223","article-title":"A naturalistic study of racial disparities in diagnoses at an outpatient behavioral health clinic","volume":"70","author":"Gara","journal-title":"Psychiatr Serv"},{"issue":"6","key":"R20","doi-asserted-by":"publisher","first-page":"518","DOI":"10.1002\/eat.22846","article-title":"Disparities in eating disorder diagnosis and treatment according to weight status, race\/ethnicity, socioeconomic background, and sex among college students","volume":"51","author":"Sonneville","journal-title":"Int J Eat Disord"},{"issue":"1","key":"R21","doi-asserted-by":"publisher","DOI":"10.1186\/s12939-024-02320-2","article-title":"Is the diagnosis and treatment of depression gender-biased? Evidence from a population-based aging cohort in Sweden","volume":"23","author":"Bacigalupe","journal-title":"Int J Equity Health"},{"issue":"1","key":"R22","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1097\/HRP.0000000000000128","article-title":"Differences in the expression of symptoms in men versus women with depression: a systematic review and meta-analysis","volume":"25","author":"Cavanagh","journal-title":"Harv Rev Psychiatry"},{"issue":"5","key":"R23","doi-asserted-by":"publisher","first-page":"1366","DOI":"10.1111\/hsc.12968","article-title":"Challenges and barriers in mental healthcare systems and their impact on the family: a systematic integrative review","volume":"28","author":"Carbonell","journal-title":"Health Soc Care Community"},{"issue":"2","key":"R24","doi-asserted-by":"publisher","first-page":"239","DOI":"10.2105\/ajph.93.2.239","article-title":"Bias in mental health assessment and intervention: theory and evidence","volume":"93","author":"Snowden","journal-title":"Am J Public Health"},{"issue":"2","key":"R25","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1007\/s43681-026-01012-7","article-title":"Uncovering AI\u2019s hidden risks: an empirical analysis of health-related AI incidents and their ethical implications","volume":"6","author":"Denecke","journal-title":"AI Ethics"},{"issue":"4","key":"R26","doi-asserted-by":"publisher","first-page":"351","DOI":"10.1177\/1094428114547952","article-title":"Best practice recommendations for designing and implementing experimental vignette methodology studies","volume":"17","author":"Aguinis","journal-title":"Organ Res Methods"},{"key":"R27","unstructured":"Kusner MJ Silva R Russell C Loftus JR . Counterfactual fairness. Presented at: Advances in Neural Information Processing Systems 30 (NIPS 2017); Dec 4-9, 2017:4066-4077; Long Beach, CA, USA. URL: https:\/\/proceedings.neurips.cc\/paper\/2017\/hash\/a486cd07e4ac3d270571622f4f316ec5-Abstract.html [Accessed 26-08-2026]"},{"key":"R28","unstructured":"Russell C Kusner MJ Loftus J Silva R . When worlds collide: integrating different counterfactual assumptions in fairness. Presented at: Advances in Neural Information Processing Systems (NIPS 2017); Dec 4-9, 2017:6414-6423; Long Beach, CA, USA. URL: https:\/\/proceedings.neurips.cc\/paper\/2017\/hash\/1271a7029c9df08643b631b02cf9e116-Abstract.html [Accessed 26-08-2026]"},{"key":"R29","doi-asserted-by":"crossref","unstructured":"Kincaid JP Fishburne RP JrRogers RL Chissom BS . Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted personnel. Naval Technical Training Command, Research Branch; 1975. URL: https:\/\/archive.org\/details\/DTIC_ADA006655 [Accessed 26-08-2026] [doi: 10.21236\/ADA006655]","DOI":"10.21236\/ADA006655"},{"key":"R30","doi-asserted-by":"publisher","DOI":"10.1016\/j.eatbeh.2020.101458","article-title":"Eating disorder symptoms in Asian American college students","volume":"40","author":"Uri","journal-title":"Eat Behav"},{"issue":"3","key":"R31","doi-asserted-by":"publisher","first-page":"595","DOI":"10.1002\/eat.24594","article-title":"Eating disorder risk and diagnosis among East Asian youth in the United States: findings from the Healthy Minds Study, 2020-2023","volume":"59","author":"Gao","journal-title":"Int J Eat Disord"},{"issue":"4","key":"R32","doi-asserted-by":"publisher","first-page":"455","DOI":"10.1002\/eat.23666","article-title":"Reporting racial and ethnic diversity in eating disorder research over the past 20\u2009years","volume":"55","author":"Egbert","journal-title":"Int J Eat Disord"},{"issue":"3","key":"R33","doi-asserted-by":"publisher","first-page":"1097","DOI":"10.1162\/coli_a_00524","article-title":"Bias and fairness in large language models: a survey","volume":"50","author":"Gallegos","journal-title":"Comput Linguist"},{"key":"R34","doi-asserted-by":"crossref","unstructured":"Chen E Zhan RJ Lin YB Chen HH . More women, same stereotypes: unpacking the gender bias paradox in large language models. Presented at: 34th ACM International Conference on Information and Knowledge Management (CIKM \u201925); Nov 10-14, 2025; Seoul, Republic of Korea. URL: https:\/\/dl.acm.org\/doi\/proceedings\/10.1145\/3746252 [doi: 10.1145\/3746252.3760969]","DOI":"10.1145\/3746252.3760969"},{"key":"R35","doi-asserted-by":"publisher","DOI":"10.2196\/90759","article-title":"Toward retrieval-grounded evaluation for conversational large language model-based risk assessment","volume":"5","author":"Hu","journal-title":"JMIR AI"},{"key":"R36","doi-asserted-by":"publisher","DOI":"10.2196\/80348","article-title":"Large language model-based chatbots and agentic AI for mental health counseling: systematic review of methodologies, evaluation frameworks, and ethical safeguards","volume":"5","author":"Cho","journal-title":"JMIR AI"},{"key":"R37","doi-asserted-by":"crossref","unstructured":"Xie C Zhu D Wang Z Zhang H Wei Z . Explainable AI for mental health detection from social media: a comparative study of traditional machine learning and a large language model. SSRN. Preprint posted online on Mar 19, 2026. [doi: 10.2139\/ssrn.6429778]","DOI":"10.2139\/ssrn.6429778"},{"key":"R38","doi-asserted-by":"crossref","unstructured":"McCalla D Li B Jaeger S et al. Implicit bias in large language model diagnosis of eating disorders: experimental vignette study. JMIR Preprints. Preprint posted online on Feb 13, 2026. [doi: 10.2196\/preprints.93498]","DOI":"10.2196\/preprints.93498"}],"container-title":["JMIR AI"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T20:24:48Z","timestamp":1790195088000},"score":1,"resource":{"primary":{"URL":"https:\/\/ai.jmir.org\/2026\/1\/e93498"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,23]]},"references-count":38,"URL":"https:\/\/doi.org\/10.2196\/93498","relation":{},"ISSN":["2817-1705"],"issn-type":[{"value":"2817-1705","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,23]]},"article-number":"v5i8e93498"}}