{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,11]],"date-time":"2026-11-11T22:00:25Z","timestamp":1794434425157,"version":"4.9.1"},"reference-count":28,"publisher":"SAGE Publications","license":[{"start":{"date-parts":[[2023,1,1]],"date-time":"2023-01-01T00:00:00Z","timestamp":1672531200000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000025","name":"National Institute of Mental Health","doi-asserted-by":"publisher","award":["R01 MH123482-01"],"award-info":[{"award-number":["R01 MH123482-01"]}],"id":[{"id":"10.13039\/100000025","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["DIGITAL HEALTH"],"published-print":{"date-parts":[[2023,1]]},"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>With a rapidly expanding gap between the need for and availability of mental health care, artificial intelligence (AI) presents a promising, scalable solution to mental health assessment and treatment. Given the novelty and inscrutable nature of such systems, exploratory measures aimed at understanding domain knowledge and potential biases of such systems are necessary for ongoing translational development and future deployment in high-stakes healthcare settings.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We investigated the domain knowledge and demographic bias of a generative, AI model using contrived clinical vignettes with systematically varied demographic features. We used balanced accuracy (BAC) to quantify the model\u2019s performance. We used generalized linear mixed-effects models to quantify the relationship between demographic factors and model interpretation.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Findings<\/jats:title>\n                    <jats:p>We found variable model performance across diagnoses; attention deficit hyperactivity disorder, posttraumatic stress disorder, alcohol use disorder, narcissistic personality disorder, binge eating disorder, and generalized anxiety disorder showed high BAC (0.70\u2009\u2264\u2009BAC\u2009\u2264\u20090.82); bipolar disorder, bulimia nervosa, barbiturate use disorder, conduct disorder, somatic symptom disorder, benzodiazepine use disorder, LSD use disorder, histrionic personality disorder, and functional neurological symptom disorder showed low BAC (BAC\u2009\u2264\u20090.59).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Interpretation<\/jats:title>\n                    <jats:p>Our findings demonstrate initial promise in the domain knowledge of a large AI model, with performance variability perhaps due to the more salient hallmark symptoms, narrower differential diagnosis, and higher prevalence of some disorders. We found limited evidence of model demographic bias, although we do observe some gender and racial differences in model outcomes mirroring real-world differential prevalence estimates.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/20552076231170499","type":"journal-article","created":{"date-parts":[[2023,4,18]],"date-time":"2023-04-18T01:54:41Z","timestamp":1681782881000},"update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":32,"title":["Testing domain knowledge and risk of bias of a large-scale general artificial intelligence model in mental health"],"prefix":"10.1177","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0866-0508","authenticated-orcid":false,"given":"Michael V.","family":"Heinz","sequence":"first","affiliation":[{"name":"Center for Technology and Behavioral Health, Geisel School of Medicine, Dartmouth College, Lebanon, NH, USA"},{"name":"Department of Psychiatry, Geisel School of Medicine, Dartmouth College, Hanover, NH, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sukanya","family":"Bhattacharya","sequence":"additional","affiliation":[{"name":"Center for Technology and Behavioral Health, Geisel School of Medicine, Dartmouth College, Lebanon, NH, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brianna","family":"Trudeau","sequence":"additional","affiliation":[{"name":"Center for Technology and Behavioral Health, Geisel School of Medicine, Dartmouth College, Lebanon, NH, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rachel","family":"Quist","sequence":"additional","affiliation":[{"name":"Center for Technology and Behavioral Health, Geisel School of Medicine, Dartmouth College, Lebanon, NH, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2970-2746","authenticated-orcid":false,"given":"Seo Ho","family":"Song","sequence":"additional","affiliation":[{"name":"Center for Technology and Behavioral Health, Geisel School of Medicine, Dartmouth College, Lebanon, NH, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Camilla M.","family":"Lee","sequence":"additional","affiliation":[{"name":"Center for Technology and Behavioral Health, Geisel School of Medicine, Dartmouth College, Lebanon, NH, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicholas C.","family":"Jacobson","sequence":"additional","affiliation":[{"name":"Center for Technology and Behavioral Health, Geisel School of Medicine, Dartmouth College, Lebanon, NH, USA"},{"name":"Department of Psychiatry, Geisel School of Medicine, Dartmouth College, Hanover, NH, USA"},{"name":"Quantitative Biomedical Sciences Program, Dartmouth College, Hanover, NH, USA"},{"name":"Department of Biomedical Data Science, Geisel School of Medicine, Dartmouth College, Lebanon, NH, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2023,4,17]]},"reference":[{"key":"bibr1-20552076231170499","doi-asserted-by":"publisher","DOI":"10.1016\/S2215-0366(21)00395-3"},{"key":"bibr2-20552076231170499","doi-asserted-by":"publisher","DOI":"10.15252\/embr.201642951"},{"key":"bibr3-20552076231170499","doi-asserted-by":"publisher","DOI":"10.1001\/archpsyc.62.6.629"},{"key":"bibr4-20552076231170499","first-page":"PCC.10m01013","volume":"13","author":"Vermani M","year":"2011","journal-title":"Prim Care Companion CNS Disord"},{"key":"bibr5-20552076231170499","doi-asserted-by":"publisher","DOI":"10.1016\/j.copsyc.2020.04.005"},{"key":"bibr6-20552076231170499","doi-asserted-by":"publisher","DOI":"10.1007\/s11920-019-1094-0"},{"key":"bibr7-20552076231170499","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-021-81368-4"},{"key":"bibr8-20552076231170499","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445922"},{"key":"bibr9-20552076231170499","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-021-00530-0"},{"key":"bibr10-20552076231170499","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-021-00464-x"},{"key":"bibr11-20552076231170499","doi-asserted-by":"publisher","DOI":"10.1176\/appi.ajp.2012.12070999"},{"key":"bibr12-20552076231170499","doi-asserted-by":"publisher","DOI":"10.1186\/s12910-017-0179-8"},{"key":"bibr13-20552076231170499","doi-asserted-by":"publisher","DOI":"10.1186\/s13229-015-0019-y"},{"key":"bibr14-20552076231170499","doi-asserted-by":"publisher","DOI":"10.1023\/A:1026087410516"},{"key":"bibr15-20552076231170499","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2019.18058"},{"key":"bibr16-20552076231170499","doi-asserted-by":"publisher","DOI":"10.2196\/32875"},{"key":"bibr17-20552076231170499","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2018.3763"},{"key":"bibr18-20552076231170499","volume-title":"Subject examinations: Content, outlines, and sample items","author":"National Board of Medical Examiners","year":"2021"},{"key":"bibr19-20552076231170499","doi-asserted-by":"publisher","DOI":"10.1176\/appi.books.9780890425596"},{"key":"bibr20-20552076231170499","unstructured":"Brown TB, Mann B, Ryder N, et al. Language models are few-shot learners. ArXiv200514165 Cs [Internet]. 2020 Jul 22 [cited 2022 Feb 1]. Available from: http:\/\/arxiv.org\/abs\/2005.14165"},{"key":"bibr21-20552076231170499","unstructured":"Open AI. OpenAI API [Internet]. OpenAI. 2020 [cited 2022 Mar 25]. Available from: https:\/\/openai.com\/blog\/openai-api\/"},{"key":"bibr22-20552076231170499","unstructured":"Kuhn M. The caret package [Internet]. [cited 2022 Mar 28]. Available from: https:\/\/topepo.github.io\/caret\/"},{"key":"bibr23-20552076231170499","unstructured":"Bates D, Maechler M, Bolker B, et al. lme4: Linear mixed-effects models using \u201cEigen\u201d and S4 [Internet]. 2022 [cited 2022 Mar 31]. Available from: https:\/\/CRAN.R-project.org\/package=lme4"},{"key":"bibr24-20552076231170499","doi-asserted-by":"publisher","DOI":"10.1007\/s11747-019-00710-5"},{"key":"bibr25-20552076231170499","doi-asserted-by":"publisher","DOI":"10.5498\/wjp.v4.i4.133"},{"key":"bibr26-20552076231170499","unstructured":"U.S. DHHS. 2020 National survey of drug use and health (NSDUH) releases | CBHSQ Data [Internet]. Substance Abuse and Mental Health Services Administration. 2020 [cited 2022 Apr 3]. Available from: https:\/\/www.samhsa.gov\/data\/release\/2020-national-survey-drug-use-and-health-nsduh-releases"},{"key":"bibr27-20552076231170499","doi-asserted-by":"publisher","DOI":"10.1186\/1471-244X-13-298"},{"key":"bibr28-20552076231170499","unstructured":"Solaiman I, Dennison C. Process for adapting language models to society (PALMS) with values-targeted datasets. 43."}],"container-title":["DIGITAL HEALTH"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/20552076231170499","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/20552076231170499","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/20552076231170499","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T17:53:07Z","timestamp":1777398787000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/20552076231170499"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1]]},"references-count":28,"alternative-id":["10.1177\/20552076231170499"],"URL":"https:\/\/doi.org\/10.1177\/20552076231170499","relation":{},"ISSN":["2055-2076","2055-2076"],"issn-type":[{"value":"2055-2076","type":"print"},{"value":"2055-2076","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1]]},"article-number":"20552076231170499"}}