{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T19:09:04Z","timestamp":1788980944053,"version":"build-2803163510"},"reference-count":32,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Psychiatric distress is common among patients with cancer; yet, systematic interview-based screening remains difficult to scale in routine clinical care. Large language models (LLMs) have shown promise as scalable tools for mental health assessment, but most existing evidence is derived from clinician-authored records, translated text, or proxy data. The performance characteristics, error patterns, and explanatory behaviors of contemporary LLMs when applied to authentic, non-English psychiatric interviews remain insufficiently characterized.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This exploratory study evaluated how contemporary LLMs reproduced psycho-oncologists\u2019 item-level symptom ratings from real-world Korean psycho-oncology interviews, focusing on concordance, directional bias, and clinician-adjudicated error characteristics.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Between April 2024 and May 2025, 101 adults receiving oncologic care in South Korea underwent semistructured interviews. Board-certified psycho-oncologists provided real-time, time-stamped ratings of 39 items. Generative pretrained transformer 4o (GPT-4o), Claude 3.5 Sonnet, and Gemini 2.5 Flash generated item scores and brief rationales using identical Korean zero-shot rubrics. Concordance between clinician ratings was evaluated using ordinal and binary screening metrics. A paired Wilcoxon test compared patient-level total symptom burden. Binary mismatches were clinically adjudicated as ambiguous or definite overestimation or underestimation with an 8-etiology taxonomy. Model-generated rationales were further meta-evaluated using GPT-5.4, Claude Sonnet 4.6, and Gemini Pro 3.1 across 4 dimensions: citation (use of quoted supporting statements), structure (logical organization of the rationale), mapping (consistency between the rationale and the assigned item rating), and expansion (degree of interpretive elaboration beyond the explicit transcript content). The associations between meta-evaluation results and absolute error were examined using cross-classified mixed-effects models.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Out of 101 participants, 88 (87.1%) were predominantly female and had breast cancer as the primary cancer type (n=70, 69.3%). Across 3931 item-level ratings, all models showed good agreement with clinicians (intraclass correlation coefficient: 0.816\u20100.872), with GPT-4o showing the highest agreement. Claude 3.5 and Gemini 2.5 yielded significantly higher patient-level symptom burden (adjusted\n P<\/jats:italic>\n <.001 and adjusted\n P<\/jats:italic>\n =.002, respectively), whereas GPT-4o did not (adjusted\n P<\/jats:italic>\n =.15). Clinician adjudication attributed 97 of 248 (39.1%) GPT-4o mismatches, 88 of 301 (29.2%) Claude 3.5 mismatches, and 118 of 355 (33.2%) Gemini 2.5 mismatches to intrinsic ambiguity in patient speech. Among definite errors, misapplication of severity thresholds was the predominant mechanism across models. Exploratory mixed-effects models showed that higher expansion and lower mapping scores were associated with larger absolute errors. However, the association with expansion may reflect case difficulty or ambiguity rather than causation.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n In this exploratory clinician-benchmarked evaluation of authentic interviews from a Korean psycho-oncology sample comprising predominantly women and patients with breast cancer, LLMs showed high aggregate concordance with psycho-oncologists\u2019 item-level ratings while differing in their error profiles. These findings support further evaluation of LLM-based item-level symptom-rating approaches in psycho-oncology. Validation in larger, more diverse, and independent cohorts is needed.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/91405","type":"journal-article","created":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T18:41:21Z","timestamp":1788979281000},"page":"e91405-e91405","source":"Crossref","is-referenced-by-count":0,"title":["Large Language Models for Distress Rating in Korean Psycho-Oncology Interviews: Exploratory Clinician-Benchmarked Evaluation Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6553-9563","authenticated-orcid":false,"given":"Jaehyun","family":"Kim","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9332-8659","authenticated-orcid":false,"given":"Kyung-Lak","family":"Son","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2265-3291","authenticated-orcid":false,"given":"Chan-Woo","family":"Yeom","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6650-3685","authenticated-orcid":false,"given":"Won-Hyoung","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8559-8117","authenticated-orcid":false,"given":"Sun Hyung","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2425-4406","authenticated-orcid":false,"given":"Joon Sung","family":"Shin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-2578-7567","authenticated-orcid":false,"given":"Hyunsun","family":"Yang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-7177-3885","authenticated-orcid":false,"given":"Daehun","family":"Yoo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2366-3275","authenticated-orcid":false,"given":"Bong-Jin","family":"Hahm","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,9]]},"reference":[{"issue":"2","key":"R1","doi-asserted-by":"publisher","first-page":"160","DOI":"10.1016\/S1470-2045(11)70002-X","article-title":"Prevalence of depression, anxiety, and adjustment disorder in oncological, haematological, and palliative-care settings: a meta-analysis of 94 interview-based studies","volume":"12","author":"Mitchell","journal-title":"Lancet Oncol"},{"issue":"12","key":"R2","doi-asserted-by":"publisher","first-page":"e628","DOI":"10.1016\/S1470-2045(25)00530-3","article-title":"The human crisis in cancer: a Lancet Oncology Commission","volume":"26","author":"Rodin","journal-title":"Lancet Oncol"},{"issue":"14","key":"R3","doi-asserted-by":"publisher","first-page":"2101","DOI":"10.1001\/archinte.160.14.2101","article-title":"Depression is a risk factor for noncompliance with medical treatment: meta-analysis of the effects of anxiety and depression on patient adherence","volume":"160","author":"DiMatteo","journal-title":"Arch Intern Med"},{"issue":"3","key":"R4","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1002\/ejsp.2420150303","article-title":"Methods of coping with social desirability bias: a review","volume":"15","author":"Nederhof","journal-title":"Euro J Social Psych"},{"key":"R5","unstructured":"First MB Williams JB Karg RS Spitzer RL . The structured clinical interview for DSM-5\u00ae. American Psychiatric Association Publishing. 2015. URL: https:\/\/www.appi.org\/products\/structured-clinical-interview-for-dsm-5-scid-5 [Accessed 12-08-2026]"},{"issue":"1","key":"R6","doi-asserted-by":"publisher","DOI":"10.1016\/S2589-7500(23)00254-6","article-title":"Large language models: a new chapter in digital health","volume":"6","author":"The Lancet Digital Health","journal-title":"Lancet Digit Health"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2025.109808","article-title":"Natural language processing techniques applied to the electronic health record in clinical research and practice\u2014an introduction to methodologies","volume":"188","author":"Clay","journal-title":"Comput Biol Med"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2023.106649","article-title":"Natural language processing in electronic health records in relation to healthcare decision-making: a systematic review","volume":"155","author":"Hossain","journal-title":"Comput Biol Med"},{"issue":"4","key":"R9","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models: a systematic review","volume":"333","author":"Bedi","journal-title":"JAMA"},{"issue":"11","key":"R10","doi-asserted-by":"publisher","first-page":"e829","DOI":"10.1016\/S2589-7500(22)00153-4","article-title":"From promise to practice: towards the realisation of AI-informed mental health care","volume":"4","author":"Koutsouleris","journal-title":"Lancet Digit Health"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","first-page":"108","DOI":"10.1093\/schbul\/sbl054","article-title":"DSM and the death of phenomenology in America: an example of unintended consequences","volume":"33","author":"Andreasen","journal-title":"Schizophr Bull"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","first-page":"547","DOI":"10.1146\/annurev.psych.54.101601.145041","article-title":"Psychological aspects of natural language use: our words, our selves","volume":"54","author":"Pennebaker","journal-title":"Annu Rev Psychol"},{"issue":"9","key":"R13","doi-asserted-by":"publisher","first-page":"856","DOI":"10.1016\/j.bpsc.2021.02.001","article-title":"Artificial intelligence for mental health care: clinical applications, barriers, facilitators, and artificial wisdom","volume":"6","author":"Lee","journal-title":"Biol Psychiatry Cogn Neurosci Neuroimaging"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.98","article-title":"Documenting large webtext corpora: a case study on the Colossal Clean Crawled Corpus","author":"Dodge","journal-title":"Proc 2021 Conf Empir Methods Nat Lang Process"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.2196\/69857","article-title":"Performance of open-source large language models in psychiatry: usability study through comparative analysis of non-English records and English translations","volume":"27","author":"Kim","journal-title":"J Med Internet Res"},{"issue":"1","key":"R16","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-024-00717-2","article-title":"Current applications and challenges in large language models for patient care: a systematic review","volume":"5","author":"Busch","journal-title":"Commun Med (Lond)"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.2196\/57400","article-title":"Large language models for mental health applications: systematic review","volume":"11","author":"Guo","journal-title":"JMIR Ment Health"},{"issue":"11","key":"R18","doi-asserted-by":"publisher","first-page":"6488","DOI":"10.3390\/curroncol31110481","article-title":"Two-step screening for depression and anxiety in patients with cancer: a retrospective validation study using real-world data","volume":"31","author":"Gascon","journal-title":"Curr Oncol"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01611-4","article-title":"A scoping review of large language models for generative tasks in mental health care","volume":"8","author":"Hua","journal-title":"NPJ Digit Med"},{"issue":"2","key":"R20","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1198\/sbr.2011.10019","article-title":"Measures of agreement and concordance with clinical research applications","volume":"3","author":"Landis","journal-title":"Stat Biopharm Res"},{"issue":"1","key":"R21","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1038\/s41591-024-03425-5","article-title":"The TRIPOD-LLM reporting guideline for studies using large language models","volume":"31","author":"Gallifant","journal-title":"Nat Med"},{"issue":"1","key":"R22","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01969-5","article-title":"Optimizing large language models for detecting symptoms of depression\/anxiety in chronic diseases patient communications","volume":"8","author":"Kim","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R23","doi-asserted-by":"publisher","DOI":"10.1007\/s40501-025-00363-y","article-title":"Large language models in mental health care: a scoping review","volume":"12","author":"Hua","journal-title":"Curr Treat Options Psych"},{"issue":"9","key":"R24","doi-asserted-by":"publisher","DOI":"10.1093\/jnci\/dju244","article-title":"Development of the National Cancer Institute\u2019s patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE)","volume":"106","author":"Basch","journal-title":"J Natl Cancer Inst"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.2196\/75030","article-title":"Aiding large language models using clinical scoresheets for neurobehavioral diagnostic classification from text: algorithm development and validation","volume":"4","author":"Lin","journal-title":"JMIR AI"},{"issue":"7","key":"R26","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000943","article-title":"Leveraging large language models for automated depression screening","volume":"4","author":"Teferra","journal-title":"PLOS Digit Health"},{"issue":"1","key":"R27","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01074-z","article-title":"Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI","volume":"7","author":"Abbasian","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R28","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01283-6","article-title":"Evaluation and mitigation of cognitive biases in medical language models","volume":"7","author":"Schmidgall","journal-title":"NPJ Digit Med"},{"key":"R29","unstructured":"Wu J Xie K Gu B Kr\u00fcger N Lin KJ Yang J . Why chain of thought fails in clinical text understanding. arXiv. Preprint posted online on Sep 26, 2025. [doi: 10.48550\/arXiv.2509.21933]"},{"issue":"1","key":"R30","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01010-1","article-title":"Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine","volume":"7","author":"Savage","journal-title":"NPJ Digit Med"},{"key":"R31","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.386","article-title":"Towards faithfully interpretable NLP systems: how should we define and evaluate faithfulness?","author":"Jacovi","journal-title":"Proc 58th Annu Meet Assoc Comput Linguist"},{"key":"R32","unstructured":"Editage [Article in Korean]. URL: https:\/\/www.editage.co.kr\/ [Accessed 12-08-2026]"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T18:41:22Z","timestamp":1788979282000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e91405"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,9]]},"references-count":32,"URL":"https:\/\/doi.org\/10.2196\/91405","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,9]]},"article-number":"v28i2e91405"}}