{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,23]],"date-time":"2026-08-23T16:19:56Z","timestamp":1787501996888,"version":"build-2736575974"},"reference-count":35,"publisher":"PeerJ","license":[{"start":{"date-parts":[[2024,11,28]],"date-time":"2024-11-28T00:00:00Z","timestamp":1732752000000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Research University Higher School of Economics"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"abstract":"<jats:p>Recent advancements in large language models (LLMs) have opened new possibilities for developing conversational agents (CAs) in various subfields of mental healthcare. However, this progress is hindered by limited access to high-quality training data, often due to privacy concerns and high annotation costs for low-resource languages. A potential solution is to create human-AI annotation systems that utilize extensive public domain user-to-user and user-to-professional discussions on social media. These discussions, however, are extremely noisy, necessitating the adaptation of LLMs for fully automatic cleaning and pre-classification to reduce human annotation effort. To date, research on LLM-based annotation in the mental health domain is extremely scarce. In this article, we explore the potential of zero-shot classification using four LLMs to select and pre-classify texts into topics representing psychiatric disorders, in order to facilitate the future development of CAs for disorder-specific counseling. We use 64,404 Russian-language texts from online discussion threads labeled with seven most commonly discussed disorders: depression, neurosis, paranoia, anxiety disorder, bipolar disorder, obsessive-compulsive disorder, and borderline personality disorder. Our research shows that while preliminary data filtering using zero-shot technology slightly improves classification, LLM fine-tuning makes a far larger contribution to its quality. Both standard and natural language inference (NLI) modes of fine-tuning increase classification accuracy by more than three times compared to non-fine-tuned training with preliminarily filtered data. Although NLI fine-tuning achieves slightly higher accuracy (0.64) than the standard approach, it is six times slower, indicating a need for further experimentation with NLI hypothesis engineering. Additionally, we demonstrate that lemmatization does not affect classification quality and that multilingual models using texts in their original language perform slightly better than English-only models using automatically translated texts. Finally, we introduce our dataset and model as the first openly available Russian-language resource for developing conversational agents in the domain of mental health counseling.<\/jats:p>","DOI":"10.7717\/peerj-cs.2395","type":"journal-article","created":{"date-parts":[[2024,11,28]],"date-time":"2024-11-28T05:07:39Z","timestamp":1732770459000},"page":"e2395","source":"Crossref","is-referenced-by-count":5,"title":["Using large language models for extracting and pre-annotating texts on mental health from noisy data in a low-resource language"],"prefix":"10.7717","volume":"10","author":[{"given":"Sergei","family":"Koltcov","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anton","family":"Surkov","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Olessia","family":"Koltsova","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1407-0168","authenticated-orcid":true,"given":"Vera","family":"Ignatenko","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"4443","published-online":{"date-parts":[[2024,11,28]]},"reference":[{"key":"10.7717\/peerj-cs.2395\/ref-1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2406.12687","article-title":"Using LLMs to aid annotation and collection of clinically-enriched data in bipolar disorder and schizophrenia 2024","author":"Aich","year":"2024"},{"key":"10.7717\/peerj-cs.2395\/ref-2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2201.01337","article-title":"ZeroBERTo: leveraging zero-shot text classification by topic modeling","author":"Alcoforado","year":"2022"},{"key":"10.7717\/peerj-cs.2395\/ref-3","first-page":"440","article-title":"Biographies, bollywood, boom-boxes and blenders: domain adaptation for sentiment classification","author":"Blitzer","year":"2007"},{"key":"10.7717\/peerj-cs.2395\/ref-4","doi-asserted-by":"crossref","DOI":"10.28995\/2075-7182-2022-21-81-98","article-title":"The dataset for presuicidal signals detection in text and its analysis","author":"Buyanov","year":"2022"},{"issue":"6","key":"10.7717\/peerj-cs.2395\/ref-5","doi-asserted-by":"publisher","first-page":"653","DOI":"10.1109\/JPROC.2023.3279374","article-title":"Zero-shot and few-shot learning with knowledge graphs: a comprehensive survey","volume":"111","author":"Chen","year":"2021","journal-title":"Proceedings of the IEEE"},{"key":"10.7717\/peerj-cs.2395\/ref-6","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2311.13857","article-title":"Challenges of large language models for mental health counseling","author":"Chung","year":"2023"},{"key":"10.7717\/peerj-cs.2395\/ref-7","doi-asserted-by":"publisher","first-page":"688","DOI":"10.1038\/s44159-023-00241-5","article-title":"Using large language models in psychology","volume":"2","author":"Demszky","year":"2023","journal-title":"Nature Reviews Psychology"},{"key":"10.7717\/peerj-cs.2395\/ref-8","first-page":"665","article-title":"WordNet(s)","volume-title":"Encyclopedia of Language and Linguistics","author":"Fellbaum","year":"2005","edition":"2"},{"issue":"1","key":"10.7717\/peerj-cs.2395\/ref-9","doi-asserted-by":"publisher","first-page":"112","DOI":"10.1109\/MSP.2017.2763441","article-title":"Recent advances in zero-shot recognition: toward data-efficient understanding of visual content","volume":"35","author":"Fu","year":"2018","journal-title":"IEEE Signal Processing Magazine"},{"key":"10.7717\/peerj-cs.2395\/ref-10","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2308.15192","article-title":"Enhancing psychological counseling with large language model: a multifaceted decision-support system for non-professionals","author":"Fu","year":"2023"},{"key":"10.7717\/peerj-cs.2395\/ref-11","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2403.15401","article-title":"Large language model for mental health: a systematic review","author":"Guo","year":"2024"},{"key":"10.7717\/peerj-cs.2395\/ref-12","doi-asserted-by":"publisher","first-page":"e44838","DOI":"10.2196\/44838","article-title":"An overview of chatbot-based mobile mental health apps: insights from app description and user reviews","volume":"11","author":"Haque","year":"2022","journal-title":"JMIR mHealth and uHealth"},{"issue":"4","key":"10.7717\/peerj-cs.2395\/ref-13","doi-asserted-by":"publisher","first-page":"e43862","DOI":"10.2196\/43862","article-title":"Conversational agent interventions for mental health problems: systematic review and meta-analysis of randomized controlled trials","volume":"25","author":"He","year":"2023","journal-title":"Journal of Medical Internet Research"},{"key":"10.7717\/peerj-cs.2395\/ref-14","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2401.02984v1","article-title":"Andrew beam large language models in mental health care: a scoping review","author":"Hua","year":"2024"},{"key":"10.7717\/peerj-cs.2395\/ref-15","first-page":"397","article-title":"LLMs in the loop: leveraging large language model annotations for active learning in low-resource languages","author":"Kholodna","year":"2024"},{"key":"10.7717\/peerj-cs.2395\/ref-16","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2407.11004","article-title":"The ALCHEmist: automated labeling 500x CHEaper than LLM data annotators","author":"Huang","year":"2024"},{"key":"10.7717\/peerj-cs.2395\/ref-17","article-title":"Morphological analyzer and generator for Russian and Ukrainian Languages. Analysis of images, social networks and texts","author":"Korobov","year":"2015"},{"key":"10.7717\/peerj-cs.2395\/ref-18","article-title":"GigaChat","author":"Krestnikov","year":"2024"},{"key":"10.7717\/peerj-cs.2395\/ref-19","first-page":"6525","article-title":"A comparative study on annotation quality of crowdsourcing and LLm Via label aggregation","author":"Li","year":"2024"},{"issue":"1","key":"10.7717\/peerj-cs.2395\/ref-20","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1038\/s41746-023-00979-5","article-title":"Systematic review and meta-analysis of AI-based conversational agents for promoting mental health and well-being","volume":"6","author":"Li","year":"2023","journal-title":"NPJ Digital Medicine"},{"key":"10.7717\/peerj-cs.2395\/ref-21","doi-asserted-by":"publisher","first-page":"598","DOI":"10.1016\/j.jad.2022.09.028","article-title":"Evaluation of chatbot-delivered interventions for self-management of depression: content analysis","volume":"319","author":"Martinengo","year":"2022","journal-title":"Journal of Affective Disorders"},{"key":"10.7717\/peerj-cs.2395\/ref-22","doi-asserted-by":"publisher","first-page":"71876","DOI":"10.1109\/ACCESS.2024.3402809","article-title":"ChatGPT label: comparing the quality of human-generated and LLM-generated annotations in low-resource language NLP tasks","volume":"12","author":"Nasution","year":"2024","journal-title":"IEEE Access"},{"issue":"1","key":"10.7717\/peerj-cs.2395\/ref-23","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1038\/s44277-024-00010-z","article-title":"Opportunities and risks of large language models in psychiatry","volume":"2","author":"Obradovich","year":"2024","journal-title":"NPP\u2014Digital Psychiatry and Neuroscience"},{"issue":"6","key":"10.7717\/peerj-cs.2395\/ref-24","doi-asserted-by":"publisher","first-page":"2023","DOI":"10.14569\/IJACSA.2023.0140616","article-title":"Artificial intelligence enabled mobile chatbot psychologist using AIML and cognitive behavioral therapy","volume":"14","author":"Omarov","year":"2023","journal-title":"International Journal of Advanced Computer Science and Applications"},{"issue":"1","key":"10.7717\/peerj-cs.2395\/ref-25","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1140\/epjds\/s13688-022-00333-x","article-title":"Predicting subjective well-being in a high-risk sample of Russian mental health app users","volume":"11","author":"Panicheva","year":"2022","journal-title":"EPJ Data Science"},{"issue":"4","key":"10.7717\/peerj-cs.2395\/ref-26","doi-asserted-by":"publisher","first-page":"4051","DOI":"10.1109\/TPAMI.2022.3191696","article-title":"A review of generalized zero-shot learning methods","volume":"45","author":"Pourpanah","year":"2023","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"10.7717\/peerj-cs.2395\/ref-27","volume-title":"Natural language processing with Python","author":"Steven","year":"2009"},{"key":"10.7717\/peerj-cs.2395\/ref-28","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2008.00401","article-title":"Multilingual translation with extensible multilingual pretraining and finetuning","author":"Tang","year":"2020"},{"issue":"7","key":"10.7717\/peerj-cs.2395\/ref-29","doi-asserted-by":"publisher","first-page":"116026","DOI":"10.1016\/j.psychres.2024.116026","article-title":"Large language models in psychiatry: opportunities and challenges","volume":"339","author":"Volkmer","year":"2024","journal-title":"Psychiatry Research"},{"key":"10.7717\/peerj-cs.2395\/ref-30","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3613904.3641960","article-title":"Human-LLM collaborative annotation through effective verification of LLM labels","author":"Wang","year":"2024"},{"key":"10.7717\/peerj-cs.2395\/ref-31","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2002.10957","article-title":"MiniLM: deep self-attention distillation for task-agnostic compression of pre-trained transformers","author":"Wang","year":"2020"},{"issue":"2","key":"10.7717\/peerj-cs.2395\/ref-32","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1145\/3293318","article-title":"A survey of zero-shot learning: settings, methods, and applications","volume":"10","author":"Wang","year":"2019","journal-title":"ACM Transactions on Intelligent Systems and Technology"},{"key":"10.7717\/peerj-cs.2395\/ref-33","first-page":"6056","article-title":"Towards interpretable mental health analysis with large language models","author":"Yang","year":"2023"},{"key":"10.7717\/peerj-cs.2395\/ref-34","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1909.00161","article-title":"Benchmarking zero-shot text classification: datasets, evaluation and entailment approach","author":"Yin","year":"2019"},{"key":"10.7717\/peerj-cs.2395\/ref-35","first-page":"3010","article-title":"Learning a deep embedding model for zero-shot learning","author":"Zhang","year":"2016"}],"container-title":["PeerJ Computer Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/peerj.com\/articles\/cs-2395.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-2395.xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-2395.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-2395.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,28]],"date-time":"2024-11-28T05:07:44Z","timestamp":1732770464000},"score":1,"resource":{"primary":{"URL":"https:\/\/peerj.com\/articles\/cs-2395"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,28]]},"references-count":35,"alternative-id":["10.7717\/peerj-cs.2395"],"URL":"https:\/\/doi.org\/10.7717\/peerj-cs.2395","archive":["CLOCKSS","LOCKSS","Portico"],"relation":{},"ISSN":["2376-5992"],"issn-type":[{"value":"2376-5992","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,28]]},"article-number":"e2395"}}