{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,3]],"date-time":"2026-09-03T13:14:48Z","timestamp":1788441288925,"version":"build-2803163510"},"reference-count":36,"publisher":"IMR Press","issue":"4","license":[{"start":{"date-parts":[[2026,8,13]],"date-time":"2026-08-13T00:00:00Z","timestamp":1786579200000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"An-Nan Hospital, China Medical University Hospital, Taiwan","award":["ANHRF114-23"],"award-info":[{"award-number":["ANHRF114-23"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Alpha Psychiatry"],"abstract":"<jats:p>Background:Artificial intelligence (AI) integration offers significant potential to improve mental healthcare, however, the reliability of large language models (LLMs) in performing nuanced clinical tasks remains an important and largely unanswered question. This study aimed to evaluate ChatGPT\u2019s performance in scoring the Hamilton Depression Rating Scale (HAMD-21) compared with expert raters using standardized patients (SPs).Methods:Three senior mental health experts created and portrayed scenarios for ten SPs representing diverse depressive symptom profiles. Recorded interviews were transcribed and used as input for ChatGPT-4o. HAMD-21 scores generated by ChatGPT were compared with those assigned by expert raters and with predefined script-based reference scores. Inter-rater reliability was assessed using intraclass correlation coefficient (ICC), and differences between raters were evaluated using Steiger\u2019s tests.Results:ChatGPT and the expert raters achieved good-to-excellent reliability for total HAMD-21 scores (experts: ICC = 0.9921; ChatGPT: ICC = 0.9739). However, expert raters achieved perfect ICCs on 11 individual items, whereas ChatGPT achieved perfect agreement on only 2 items. Steiger\u2019s test demonstrated that experts significantly outperformed ChatGPT on 10 individual items as well as on total scores (Z = 1.931, p = 0.0268). Qualitative review revealed that ChatGPT tended to overestimate scores on items related to insomnia and somatic symptoms (items 4\u20136 and 13) and frequently miscalculated total scores.Conclusions:ChatGPT demonstrated excellent agreement on total HAMD-21 scores in structured, text-based depression assessments, supporting the potential role of LLMs as adjunctive tools for standardized depression severity evaluation. However, item-level discrepancies and systematic scoring errors indicate that human oversight remains essential for clinically nuanced interpretation.<\/jats:p>","DOI":"10.31083\/ap51483","type":"journal-article","created":{"date-parts":[[2026,9,3]],"date-time":"2026-09-03T12:23:51Z","timestamp":1788438231000},"source":"Crossref","is-referenced-by-count":0,"title":["A Cautious Integration With AI in the Clinic: A Standardized-Patient Pilot Study of ChatGPT\u2019s Reliability in Hamilton Depression Rating Scale Scoring"],"prefix":"10.31083","volume":"27","author":[{"given":"Chun-Hung","family":"Chang","sequence":"first","affiliation":[{"name":"Department of Psychiatry, An-Nan Hospital, China Medical University, 709 Tainan, Taiwan"},{"name":"Department of Nursing, Shu-Zen Junior College of Medicine and Management, 82144 Kaohsiung, Taiwan"},{"name":"Department of Psychiatry, Chi Mei Medical Center, 710 Tainan, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Szu-Wei","family":"Cheng","sequence":"additional","affiliation":[{"name":"Mind-Body Interface Research Center (MBI Lab & Care), China Medical University Hospital, 40447 Taichung, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei-Jen","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Psychiatry, An-Nan Hospital, China Medical University, 709 Tainan, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chung-Wen","family":"Chang","sequence":"additional","affiliation":[{"name":"Digital Service for Smart Healthcare, Chia Nan University of Pharmacy & Science, 717301 Tainan, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ting-Hui","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Psychiatry, Chi Mei Medical Center, 710 Tainan, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jia-Hau","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Psychiatry, An-Nan Hospital, China Medical University, 709 Tainan, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sheng-Che","family":"Lin","sequence":"additional","affiliation":[{"name":"Department of Psychiatry, An-Nan Hospital, China Medical University, 709 Tainan, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4501-2502","authenticated-orcid":true,"given":"Kuan-Pin","family":"Su","sequence":"additional","affiliation":[{"name":"Department of Psychiatry, An-Nan Hospital, China Medical University, 709 Tainan, Taiwan"},{"name":"Mind-Body Interface Research Center (MBI Lab & Care), China Medical University Hospital, 40447 Taichung, Taiwan"},{"name":"Office of Research and Development, Asia University, 41354 Taichung, Taiwan"},{"name":"College of Medicine, China Medical University, 40402 Taichung, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"16242","published-online":{"date-parts":[[2026,8,13]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","unstructured":"Cruz-Gonzalez P, He AWJ, Lam EP, Ng IMC, Li MW, Hou R, et al. Artificial intelligence in mental health care: a systematic review of diagnosis, monitoring, and intervention applications. Psychological Medicine. 2025; 55: e18.","DOI":"10.1017\/S0033291724003295"},{"key":"ref2","doi-asserted-by":"publisher","unstructured":"Dehbozorgi R, Zangeneh S, Khooshab E, Nia DH, Hanif HR, Samian P, et al. The application of artificial intelligence in the field of mental health: a systematic review. BMC Psychiatry. 2025; 25: 132.","DOI":"10.1186\/s12888-025-06483-2"},{"key":"ref3","doi-asserted-by":"publisher","unstructured":"Li J, Dada A, Puladi B, Kleesiek J, Egger J. ChatGPT in healthcare: A taxonomy and systematic review. Computer Methods and Programs in Biomedicine. 2024; 245: 108013.","DOI":"10.1016\/j.cmpb.2024.108013"},{"key":"ref4","doi-asserted-by":"publisher","unstructured":"Wang L, Wan Z, Ni C, Song Q, Li Y, Clayton EW, et al. Applications and Concerns of ChatGPT and Other Conversational Large Language Models in Health Care: Systematic Review. Journal of Medical Internet Research. 2024; 26: e22769.","DOI":"10.2196\/22769"},{"key":"ref5","doi-asserted-by":"publisher","unstructured":"Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, et al. How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Medical Education. 2023; 9: e45312.","DOI":"10.2196\/45312"},{"key":"ref6","doi-asserted-by":"publisher","unstructured":"Wang Y, Chen Y, Sheng J. Assessing ChatGPT as a Medical Consultation Assistant for Chronic Hepatitis B: Cross-Language Study of English and Chinese. JMIR Medical Informatics. 2024; 12: e56426.","DOI":"10.2196\/56426"},{"key":"ref7","doi-asserted-by":"publisher","unstructured":"Arruzza ES, Evangelista CM, Chau M. The performance of ChatGPT-4.0o in medical imaging evaluation: a cross-sectional study. Journal of Educational Evaluation for Health Professions. 2024; 21: 29.","DOI":"10.3352\/jeehp.2024.21.29"},{"key":"ref8","doi-asserted-by":"publisher","unstructured":"Rajjoub R, Arroyave JS, Zaidat B, Ahmed W, Mejia MR, Tang J, et al. ChatGPT and its Role in the Decision-Making for the Diagnosis and Treatment of Lumbar Spinal Stenosis: A Comparative Analysis and Narrative Review. Global Spine Journal. 2024; 14: 998\u20131017.","DOI":"10.1177\/21925682231195783"},{"key":"ref9","doi-asserted-by":"publisher","unstructured":"Raballo A, Ravenda F, Mira A. Diagnosing schizophrenia spectrum disorders: Large language models (LLMs) vs. leading international psychiatrists (LIPs). Psychiatry and Clinical Neurosciences. 2025; 79: 599\u2013600.","DOI":"10.1111\/pcn.13864"},{"key":"ref10","doi-asserted-by":"publisher","unstructured":"Omar M, Levkovich I. Exploring the efficacy and potential of large language models for depression: A systematic review. Journal of Affective Disorders. 2025; 371: 234\u2013244.","DOI":"10.1016\/j.jad.2024.11.052"},{"key":"ref11","doi-asserted-by":"publisher","unstructured":"Zhong W, Luo J, Zhang H. The therapeutic effectiveness of artificial intelligence-based chatbots in alleviation of depressive and anxiety symptoms in short-course treatments: A systematic review and meta-analysis. Journal of Affective Disorders. 2024; 356: 459\u2013469.","DOI":"10.1016\/j.jad.2024.04.057"},{"key":"ref12","doi-asserted-by":"publisher","unstructured":"Sha Y, Pan H, Xu W, Meng W, Luo G, Du X, et al. MDD-LLM: Towards accuracy large language models for major depressive disorder diagnosis. Journal of Affective Disorders. 2025; 388: 119774.","DOI":"10.1016\/j.jad.2025.119774"},{"key":"ref13","doi-asserted-by":"publisher","unstructured":"Cheng SW, Chang CW, Chang WJ, Wang HW, Liang CS, Kishimoto T, et al. The now and future of ChatGPT and GPT in psychiatry. Psychiatry and Clinical Neurosciences. 2023; 77: 592\u2013596.","DOI":"10.1111\/pcn.13588"},{"key":"ref14","doi-asserted-by":"publisher","unstructured":"Lho SK, Park SC, Lee H, Oh DY, Kim H, Jang S, et al. Large Language Models and Text Embeddings for Detecting Depression and Suicide in Patient Narratives. JAMA Network Open. 2025; 8: e2511922.","DOI":"10.1001\/jamanetworkopen.2025.11922"},{"key":"ref15","doi-asserted-by":"publisher","unstructured":"Weber S, Deperrois N, Heun R, Fr\u00fchsch\u00fctz L, Monn A, Homan S, et al. Using a fine-tuned large language model for symptom-based depression evaluation. NPJ Digital Medicine. 2025; 8: 598.","DOI":"10.1038\/s41746-025-01982-8"},{"key":"ref16","doi-asserted-by":"publisher","unstructured":"Li N, Feng L, Hu J, Jiang L, Wang J, Han J, et al. Using deeply time-series semantics to assess depressive symptoms based on clinical interview speech. Frontiers in Psychiatry. 2023; 14: 1104190.","DOI":"10.3389\/fpsyt.2023.1104190"},{"key":"ref17","doi-asserted-by":"publisher","unstructured":"Hong RH, Murphy JK, Michalak EE, Chakrabarty T, Wang Z, Parikh SV, et al. Implementing Measurement-Based Care for Depression: Practical Solutions for Psychiatrists and Primary Care Physicians. Neuropsychiatric Disease and Treatment. 2021; 17: 79\u201390.","DOI":"10.2147\/NDT.S283731"},{"key":"ref18","doi-asserted-by":"publisher","unstructured":"Hamilton M. A rating scale for depression. Journal of Neurology, Neurosurgery & Psychiatry. 1960; 23: 56\u201362.","DOI":"10.1136\/jnnp.23.1.56"},{"key":"ref19","doi-asserted-by":"publisher","unstructured":"Belmaker RH, Agam G. Major depressive disorder. The New England Journal of Medicine. 2008; 358: 55\u201368.","DOI":"10.1056\/NEJMra073096"},{"key":"ref20","doi-asserted-by":"publisher","unstructured":"Undurraga J, Baldessarini RJ. Randomized, placebo-controlled trials of antidepressants for acute major depression: thirty-year meta-analytic review. Neuropsychopharmacology. 2012; 37: 851\u2013864.","DOI":"10.1038\/npp.2011.306"},{"key":"ref21","doi-asserted-by":"publisher","unstructured":"Mayes TL, Deane AE, Aramburu H, Yagnik K, Trivedi MH. Improving Identification and Treatment Outcomes of Treatment-Resistant Depression Through Measurement-Based Care. The Psychiatric Clinics of North America. 2023; 46: 227\u2013245.","DOI":"10.1016\/j.psc.2023.02.002"},{"key":"ref22","doi-asserted-by":"publisher","unstructured":"Zhu M, Hong RH, Yang T, Yang X, Wang X, Liu J, et al. The Efficacy of Measurement-Based Care for Depressive Disorders: Systematic Review and Meta-Analysis of Randomized Controlled Trials. The Journal of Clinical Psychiatry. 2021; 82: 21r14034.","DOI":"10.4088\/JCP.21r14034"},{"key":"ref23","doi-asserted-by":"publisher","unstructured":"Hamilton M. Development of a rating scale for primary depressive illness. The British Journal of Social and Clinical Psychology. 1967; 6: 278\u2013296.","DOI":"10.1111\/j.2044-8260.1967.tb00530.x"},{"key":"ref24","doi-asserted-by":"publisher","unstructured":"Rabinowitz J, Williams JBW, Anderson A, Fu DJ, Hefting N, Kadriu B, et al. Consistency checks to improve measurement with the Hamilton Rating Scale for Depression (HAM-D). Journal of Affective Disorders. 2022; 302: 273\u2013279.","DOI":"10.1016\/j.jad.2022.01.105"},{"key":"ref25","doi-asserted-by":"publisher","unstructured":"American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 5th edn. American Psychiatric Publishing: Arlington (VA). 2013.","DOI":"10.1176\/appi.books.9780890425596"},{"key":"ref26","doi-asserted-by":"publisher","unstructured":"Barrows HS. An overview of the uses of standardized patients for teaching and evaluating clinical skills. AAMC. Academic Medicine. 1993; 68: 443\u2013451.","DOI":"10.1097\/00001888-199306000-00002"},{"key":"ref27","doi-asserted-by":"publisher","unstructured":"Shirazi M, Sadeghi M, Emami A, Kashani AS, Parikh S, Alaeddini F, et al. Training and validation of standardized patients for unannounced assessment of physicians' management of depression. Academic Psychiatry. 2011; 35: 382\u2013387.","DOI":"10.1176\/appi.ap.35.6.382"},{"key":"ref28","doi-asserted-by":"publisher","unstructured":"Steiger JH. Tests for comparing elements of a correlation matrix. Psychological Bulletin. 1980; 87: 245\u2013251.","DOI":"10.1037\/0033-2909.87.2.245"},{"key":"ref29","doi-asserted-by":"publisher","unstructured":"Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal of Chiropractic Medicine. 2016; 15: 155\u2013163.","DOI":"10.1016\/j.jcm.2016.02.012"},{"key":"ref30","doi-asserted-by":"publisher","unstructured":"Beheshti M, Toubal IE, Alaboud K, Almalaysha M, Ogundele OB, Turabieh H, et al. Evaluating the Reliability of ChatGPT for Health-Related Questions: A Systematic Review. Informatics. 2025; 12: 9.","DOI":"10.3390\/informatics12010009"},{"key":"ref31","doi-asserted-by":"publisher","unstructured":"Hirosawa T, Harada Y, Mizuta K, Sakamoto T, Tokumasu K, Shimizu T. Evaluating ChatGPT-4's Accuracy in Identifying Final Diagnoses Within Differential Diagnoses Compared With Those of Physicians: Experimental Study for Diagnostic Cases. JMIR Formative Research. 2024; 8: e59267.","DOI":"10.2196\/59267"},{"key":"ref32","doi-asserted-by":"publisher","unstructured":"Wei Q, Yao Z, Cui Y, Wei B, Jin Z, Xu X. Evaluation of ChatGPT-generated medical responses: A systematic review and meta-analysis. Journal of Biomedical Informatics. 2024; 151: 104620.","DOI":"10.1016\/j.jbi.2024.104620"},{"key":"ref33","doi-asserted-by":"publisher","unstructured":"Ellis JL, Sijtsma K. Proof of Reliability Convergence to 1 at Rate of Spearman-Brown Formula for Random Test Forms and Irrespective of Item Pool Dimensionality. Psychometrika. 2024; 89: 774\u2013795.","DOI":"10.1007\/s11336-024-09956-7"},{"key":"ref34","doi-asserted-by":"publisher","unstructured":"Ko\u00e7ak D. Examination of ChatGPT's Performance as a Data Analysis Tool. Educational and Psychological Measurement. 2025; 85: 641\u2013671.","DOI":"10.1177\/00131644241302721"},{"key":"ref35","doi-asserted-by":"publisher","unstructured":"Shiferaw MW, Zheng T, Winter A, Mike LA, Chan LN. Assessing the accuracy and quality of artificial intelligence (AI) chatbot-generated responses in making patient-specific drug-therapy and healthcare-related decisions. BMC Medical Informatics and Decision Making. 2024; 24: 404.","DOI":"10.1186\/s12911-024-02824-5"},{"key":"ref36","unstructured":"Greist JH. Computer interviews for depression management. The Journal of Clinical Psychiatry. 1998; 59: 20\u201324."}],"container-title":["Alpha Psychiatry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.imrpress.com\/journal\/AP\/27\/4\/10.31083\/AP51483","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,3]],"date-time":"2026-09-03T12:24:10Z","timestamp":1788438250000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.imrpress.com\/journal\/AP\/27\/4\/10.31083\/AP51483"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,13]]},"references-count":36,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2026]]}},"alternative-id":["S2757-8038(26)03499-0"],"URL":"https:\/\/doi.org\/10.31083\/ap51483","relation":{},"ISSN":["2757-8038"],"issn-type":[{"value":"2757-8038","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,13]]},"article-number":"51483"}}