{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T06:20:59Z","timestamp":1789021259004,"version":"build-2803163510"},"reference-count":33,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Large language models (LLMs) have emerged as promising tools for addressing global disparities in mental health care. However, cloud-based proprietary models raise concerns about data privacy and limited adaptability to local health care systems. In contrast, open-source LLMs offer several advantages, including enhanced data security, the ability to operate offline in resource-limited settings, and greater adaptability to non-English clinical environments. Nevertheless, their performance in psychiatric applications involving non-English language inputs remains largely unexplored.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to systematically evaluate the clinical reasoning capabilities and diagnostic accuracy of a locally deployable open-source LLM in both Korean and English psychiatric contexts.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n The openbuddy-mistral-7b-v13.1 model, fine-tuned from Mistral 7B to enable conversational capabilities in Korean, was selected. A total of 200 deidentified psychiatric interview notes, documented during initial assessments of emergency department patients, were randomly selected from the electronic medical records of a tertiary hospital in South Korea. The dataset included 50 cases each of schizophrenia, bipolar disorder, depressive disorder, and anxiety disorder. The model translated the Korean notes into English and was prompted to extract 5 clinically meaningful diagnostic clues and generate the 2 most likely diagnoses using both the original Korean and translated English inputs. The hallucination rate and clinical relevance of the generated clues were manually evaluated. Top-1 and top-2 diagnostic accuracy were assessed by comparing the model\u2019s prediction with the ground truth labels. Additionally, the model\u2019s performance on a structured diagnostic task was evaluated using the psychiatry section of the Korean Medical Licensing Examination and its English-translated version.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n The model generated 997 clues from Korean interview notes and 1003 clues from English-translated notes. Hallucinations were more frequent with Korean input (n=301, 30.2%) than with English (n=134, 13.4%). Diagnostic relevance was also higher in English (n=429, 42.8%) compared to Korean (n=341, 34.2%). The model showed significantly higher top-1 diagnostic accuracy with English input (74.5% vs 59%;\n P<\/jats:italic>\n <.001), while top-2 accuracy was comparable (89.5% vs 90%;\n P<\/jats:italic>\n =.56). Across 115 questions from the medical licensing examination, the model performed better in English (n=53, 46.1%) than in Korean (n=37, 32.2%), with superior results in 7 of 11 diagnostic categories.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n This study provides an in-depth evaluation of an open-source LLM in multilingual psychiatric settings. The model\u2019s performance varied notably by language, with English input consistently outperforming Korean. These findings highlight the importance of assessing LLMs in diverse linguistic and clinical contexts. To ensure equitable mental health artificial intelligence, further development of high-quality psychiatric datasets in underrepresented languages and culturally adapted training strategies will be essential.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/69857","type":"journal-article","created":{"date-parts":[[2025,6,4]],"date-time":"2025-06-04T09:50:08Z","timestamp":1749030608000},"page":"e69857-e69857","source":"Crossref","is-referenced-by-count":13,"title":["Performance of Open-Source Large Language Models in Psychiatry: Usability Study Through Comparative Analysis of Non-English Records and English Translations"],"prefix":"10.2196","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-8243-0287","authenticated-orcid":false,"given":"Min-Gyu","family":"Kim","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2293-4555","authenticated-orcid":false,"given":"Gyubeom","family":"Hwang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9622-0232","authenticated-orcid":false,"given":"Junhyuk","family":"Chang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-2778-5383","authenticated-orcid":false,"given":"Seheon","family":"Chang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1333-358X","authenticated-orcid":false,"given":"Hyun Woong","family":"Roh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4989-3287","authenticated-orcid":false,"given":"Rae Woong","family":"Park","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,8,18]]},"reference":[{"issue":"2","key":"R1","doi-asserted-by":"publisher","first-page":"166","DOI":"10.1007\/s40471-017-0107-y","article-title":"Social epidemiology and global mental health: expanding the evidence from high-income to low- and middle-income countries","volume":"4","author":"Maselko","journal-title":"Curr Epidemiol Rep"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.1177\/1178632917694350","article-title":"Mental health service provision in low- and middle-income countries","volume":"10","author":"Rathod","journal-title":"Health Serv Insights"},{"issue":"1","key":"R3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.amepre.2022.09.003","article-title":"Geographic disparities in the availability of mental health services in U.S. public schools","volume":"64","author":"Graves","journal-title":"Am J Prev Med"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","DOI":"10.1001\/jamahealthforum.2022.4936","article-title":"Mental health service utilization rates among commercially insured adults in the US during the first year of the COVID-19 pandemic","volume":"4","author":"McBain","journal-title":"JAMA Health Forum"},{"issue":"11","key":"R5","doi-asserted-by":"publisher","first-page":"1560","DOI":"10.7326\/M22-1507","article-title":"The impact of the COVID-19 pandemic and associated control measures on the mental health of the general population\u202f: a systematic review and dose-response meta-analysis","volume":"175","author":"Salanti","journal-title":"Ann Intern Med"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.2196\/43730","article-title":"The usability, feasibility, acceptability, and efficacy of digital mental health services in the COVID-19 pandemic: scoping review, systematic review, and meta-analysis","volume":"9","author":"Zhong","journal-title":"JMIR Public Health Surveill"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.2196\/59479","article-title":"The opportunities and risks of large language models in mental health","volume":"11","author":"Lawrence","journal-title":"JMIR Ment Health"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.2196\/59939","article-title":"Equity in digital mental health interventions in the United States: where to next?","volume":"26","author":"Robinson","journal-title":"J Med Internet Res"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.1016\/j.lanwpc.2023.100905","article-title":"ChatGPT: promise and challenges for deployment in low- and middle-income countries","volume":"41","author":"Wang","journal-title":"Lancet Reg Health West Pac"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","first-page":"1177","DOI":"10.1186\/s12913-022-08519-9","article-title":"Enhancing the quality of cognitive behavioral therapy in community mental health through artificial intelligence generated fidelity feedback (Project AFFECT): a study protocol","volume":"22","author":"Creed","journal-title":"BMC Health Serv Res"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyt.2024.1422807","article-title":"Applications of large language models in psychiatry: a systematic review","volume":"15","author":"Omar","journal-title":"Front Psychiatry"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.2196\/48009","article-title":"Ethical considerations of using ChatGPT in health care","volume":"25","author":"Wang","journal-title":"J Med Internet Res"},{"issue":"1","key":"R13","doi-asserted-by":"publisher","first-page":"239","DOI":"10.1038\/s41746-024-01239-w","article-title":"Closing the gap between open source and commercial large language models for medical evidence summarization","volume":"7","author":"Zhang","journal-title":"NPJ Digital Med"},{"issue":"12","key":"R14","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000397","article-title":"How does ChatGPT-4 preform on non-English national medical licensing examination? An evaluation in Chinese language","volume":"2","author":"Fang","journal-title":"PLOS Digital Health"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1016\/j.bjao.2024.100296","article-title":"A comparative study of English and Japanese ChatGPT responses to anaesthesia-related medical questions","volume":"10","author":"Ando","journal-title":"BJA Open"},{"key":"R16","unstructured":"OpenBuddy\/openbuddy-mistral-7b-v13.1. Hugging Face; URL: https:\/\/huggingface.co\/OpenBuddy\/openbuddy-mistral-7b-v13.1 [Accessed 13-11-2023]"},{"key":"R17","unstructured":"Jiang AQ Sablayrolles A Mensch A et al. Mistral 7B. arXiv. Preprint posted online on Oct 10, 2023. [doi: 10.48550\/arXiv.2310.06825]"},{"key":"R18","unstructured":"guidance-ai\/guidance: A guidance language for controlling large language models. Github; URL: https:\/\/github.com\/guidance-ai\/guidance [Accessed 13-11-2023]"},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.2196\/51926","article-title":"Uncovering language disparity of ChatGPT on retinal vascular disease classification: cross-sectional study","volume":"26","author":"Liu","journal-title":"J Med Internet Res"},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.2196\/60807","article-title":"Performance of ChatGPT across different versions in medical licensing examinations worldwide: systematic review and meta-analysis","volume":"26","author":"Liu","journal-title":"J Med Internet Res"},{"issue":"6","key":"R21","doi-asserted-by":"publisher","first-page":"1460","DOI":"10.3390\/clinpract13060130","article-title":"Evaluating the efficacy of ChatGPT in navigating the Spanish Medical Residency Entrance Examination (MIR): promising horizons for AI in clinical medicine","volume":"13","author":"Guillen-Grima","journal-title":"Clin Pract"},{"issue":"12","key":"R22","doi-asserted-by":"publisher","first-page":"3590","DOI":"10.1038\/s41591-024-03258-2","article-title":"A toolbox for surfacing health equity harms and biases in large language models","volume":"30","author":"Pfohl","journal-title":"Nat Med"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-025-03626-6","article-title":"Socio-demographic biases in medical decision-making by large language models: a large-scale multi-model analysis","volume":"31","author":"Omar","journal-title":"Nat Med"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","first-page":"e12","DOI":"10.1016\/S2589-7500(23)00225-X","article-title":"Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study","volume":"6","author":"Zack","journal-title":"Lancet Digital Health"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.1186\/s13033-017-0170-2","article-title":"Translating mental health diagnostic and symptom terminology to train health workers and engage patients in cross-cultural, non-English speaking populations","volume":"11","author":"Acharya","journal-title":"Int J Ment Health Syst"},{"issue":"6","key":"R26","doi-asserted-by":"publisher","first-page":"864","DOI":"10.4306\/pi.2017.14.6.864","article-title":"Comparative analysis of emotional symptoms in elderly Koreans with hwa-byung and depression","volume":"14","author":"Im","journal-title":"Psychiatry Investig"},{"issue":"8","key":"R27","doi-asserted-by":"publisher","first-page":"798","DOI":"10.1176\/appi.ps.201400351","article-title":"Language barriers and access to psychiatric care: a systematic review","volume":"66","author":"Ohtani","journal-title":"Psychiatr Serv"},{"issue":"9","key":"R28","doi-asserted-by":"publisher","DOI":"10.1093\/pnasnexus\/pgae346","article-title":"Cultural bias and cultural alignment of large language models","volume":"3","author":"Tao","journal-title":"PNAS Nexus"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.2196\/58158","article-title":"Evaluating and enhancing large language models\u2019 performance in domain-specific medicine: development and usability study with DocOA","volume":"26","author":"Chen","journal-title":"J Med Internet Res"},{"key":"R30","doi-asserted-by":"publisher","DOI":"10.1177\/20552076211061922","article-title":"Data equity to advance health and health equity in low- and middle-income countries: a scoping review","volume":"7","author":"O\u2019Neil","journal-title":"Digital Health"},{"issue":"1","key":"R31","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.231147","article-title":"Feasibility of using the privacy-preserving large language model vicuna for labeling radiology reports","volume":"309","author":"Mukherjee","journal-title":"Radiology"},{"issue":"1","key":"R32","doi-asserted-by":"publisher","first-page":"339","DOI":"10.1038\/s41746-024-01344-w","article-title":"The path forward for large language models in medicine is open","volume":"7","author":"Riedemann","journal-title":"NPJ Digital Med"},{"issue":"5","key":"R33","doi-asserted-by":"publisher","first-page":"501","DOI":"10.1097\/YCO.0000000000000626","article-title":"Digital technology for management of severe mental disorders in low-income and middle-income countries","volume":"33","author":"Merchant","journal-title":"Curr Opin Psychiatry"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2025\/1\/e69857","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:58:04Z","timestamp":1782842284000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2025\/1\/e69857"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,18]]},"references-count":33,"URL":"https:\/\/doi.org\/10.2196\/69857","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,18]]},"article-number":"v27i2e69857"}}