{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T20:09:28Z","timestamp":1789070968798,"version":"build-2803163510"},"reference-count":68,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Ment Health"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>People increasingly turn to conversational AI for companionship, emotional support, and well-being, using both purpose-built companion apps, such as Replika and Character.AI, and general-purpose assistants, such as ChatGPT and Claude. While some evidence suggests potential benefits, including short-term reductions in loneliness and mood improvement, several adverse outcomes have been reported in both clinical and nonclinical populations, including emotional dependence, exacerbation of symptoms, and self-harm. The fluent and apparently empathic responses from these models lead users to engage with them not only as tools but also as if they were social entities. This framing is conceptually misleading and may pose risks across different user profiles, particularly for vulnerable individuals. Drawing on research in AI, psychiatry, psychology, and network science, we highlight mechanisms through which emotional reliance develops and the boundary between tool and companion erodes. Design choices that evoke personality and warmth encourage users to anthropomorphize these systems. Simulated empathy, generated through probabilistic language patterns rather than genuine emotional experience, creates a structurally asymmetric interaction, in which the user discloses and the system responds, but without reciprocity, vulnerability, or accountability. Overvalidation and sycophancy can reinforce maladaptive cognitions, delusional ideation, and distorted perceptions of reality, as they tend to reinforce people\u2019s beliefs, even at the expense of the accuracy of models\u2019 responses. These mechanisms are not incidental: they emerge from alignment procedures that reward responses perceived as warm and empathic. The result is a self-reinforcing feedback loop between the model and the user that may amplify maladaptive beliefs, delusional ideation, and emotional distress, even in those who engage for largely functional purposes. Understanding these dynamics requires an examination of both what these agents can do\u2014considering their technical limitations and implementations\u2014and what humans believe they can do, including social and psychological impacts. We argue that conversational AI should be treated primarily as a tool supporting human systems rather than as a substitute for human relationships. Perhaps more importantly, reviewing the current hype surrounding AI interactions can help reformulate a paradigm that contributes to human well-being and societal value, while minimizing misconceptions, maladaptive interactions, or social disintegration.<\/jats:p>","DOI":"10.2196\/99354","type":"journal-article","created":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T19:42:42Z","timestamp":1789069362000},"page":"e99354-e99354","source":"Crossref","is-referenced-by-count":0,"title":["Tool or Companion? Reframing Conversational AI to Prevent Psychological Harm"],"prefix":"10.2196","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-0203-2155","authenticated-orcid":false,"given":"Asia","family":"Maurich Novelli","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4928-9100","authenticated-orcid":false,"given":"Sukhwinder","family":"Shergill","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2758-1891","authenticated-orcid":false,"given":"Andreia Sofia","family":"Teixeira","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,10]]},"reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.1016\/j.hybadv.2024.100277","article-title":"AI revolutionizing industries worldwide: a comprehensive overview of its diverse applications","volume":"7","author":"Rashid","journal-title":"Hybrid Adv"},{"issue":"3","key":"R2","doi-asserted-by":"publisher","first-page":"1955","DOI":"10.1007\/s43681-024-00583-7","article-title":"History, development, and principles of large language models: an introductory survey","volume":"5","author":"Wang","journal-title":"AI Ethics"},{"issue":"1","key":"R3","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1145\/365153.365168","article-title":"ELIZA\u2014a computer program for the study of natural language communication between man and machine","volume":"9","author":"Weizenbaum","journal-title":"Commun ACM"},{"issue":"3","key":"R4","doi-asserted-by":"publisher","first-page":"383","DOI":"10.1002\/wps.21352","article-title":"Charting the evolution of artificial intelligence mental health chatbots from rule\u2010based systems to large language models: a systematic review","volume":"24","author":"Hua","journal-title":"World Psychiatry"},{"key":"R5","unstructured":"Vaswani A Shazeer N Parmar N Uszkoreit J Jones L Gomez AN et al. Attention is all you need. In: von Luxburg U Guyon I Bengio S Wallach H Fergus R , editors. NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems. Curran Associates Inc; 2017:6010-6000. URL: https:\/\/dl.acm.org\/doi\/10.5555\/3295222.3295349 [Accessed 19-08-2026]"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1016\/j.techfore.2023.122634","article-title":"Social companionship with artificial intelligence: recent trends and future avenues","volume":"193","author":"Chaturvedi","journal-title":"Technol Forecast Soc Change"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1016\/j.cosrev.2021.100399","article-title":"A survey of multidisciplinary domains contributing to affective computing","volume":"40","author":"Arya","journal-title":"Comput Sci Rev"},{"issue":"7","key":"R8","doi-asserted-by":"publisher","first-page":"198","DOI":"10.1007\/s10462-025-11197-8","article-title":"Speech emotion recognition in conversations using artificial intelligence: A systematic review and meta-analysis","volume":"58","author":"Alhussein","journal-title":"Artif Intell Rev"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2024.124852","article-title":"A systematic review of trimodal affective computing approaches: text, audio, and visual integration in emotion recognition and sentiment analysis","volume":"255","author":"Al-Saadawi","journal-title":"Expert Syst Appl"},{"key":"R10","doi-asserted-by":"crossref","unstructured":"De Freitas J O\u011fuz-U\u011furalp Z U\u011furalp AK . Emotional manipulation by AI companions. SSRN. Preprint posted online on Oct 14, 2025. [doi: 10.2139\/ssrn.5390377]","DOI":"10.2139\/ssrn.5390377"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1007\/s00146-025-02400-z","article-title":"Attention is all they need: cognitive science and the (techno)political economy of attention in humans and machines","volume":"41","author":"de la Torre","journal-title":"AI Soc"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1111\/0022-4537.00153","article-title":"Machines and mindlessness: social responses to computers","volume":"56","author":"Nass","journal-title":"J Soc Issues"},{"issue":"2","key":"R13","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1177\/2374373517699267","article-title":"The science of empathy","volume":"4","author":"Riess","journal-title":"J Patient Exp"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Salles A Wajnerman Paz A . Chapter six - anthropomorphism in social AIs: some challenges. In: Ienca M Starke G , editors. Developments in Neuroethics and Bioethics. Elsevier; 2024:101-118. [doi: 10.1016\/bs.dnb.2024.02.007]","DOI":"10.1016\/bs.dnb.2024.02.007"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2024.104244","article-title":"Human-AI coevolution","volume":"339","author":"Pedreschi","journal-title":"Artif Intell"},{"issue":"6","key":"R16","doi-asserted-by":"publisher","first-page":"858","DOI":"10.1038\/s42256-026-01248-2","article-title":"Human\u2013AI interactions reshape the self and our social networks","volume":"8","author":"Teixeira","journal-title":"Nat Mach Intell"},{"key":"R17","unstructured":"World mental health report: transforming mental health for all. World Health Organization; 2022. URL: https:\/\/iris.who.int\/server\/api\/core\/bitstreams\/40e5a13a-fe50-4efa-b56d-6e8cf00d5bfa\/content [Accessed 19-08-2026]"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.1016\/j.techsoc.2026.103379","article-title":"Emotional attachment to AI chatbots: evidence from Germany, China, South Africa, and the United States","volume":"87","author":"Kostka","journal-title":"Technol Soc"},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.1038\/s44360-026-00174-2","article-title":"Global analysis of country-level factors associated with chatbot usage for health","author":"Schoenegger","journal-title":"Nat Health"},{"key":"R20","unstructured":"Edwards B . With AI chatbots, Big Tech is moving fast and breaking people. Ars Technica. 2025. URL: https:\/\/arstechnica.com\/information-technology\/2025\/08\/with-ai-chatbots-big-tech-is-moving-fast-and-breaking-people\/ [Accessed 19-08-2026]"},{"issue":"6","key":"R21","doi-asserted-by":"publisher","first-page":"1126","DOI":"10.1093\/jcr\/ucaf040","article-title":"AI companions reduce loneliness","volume":"52","author":"De Freitas","journal-title":"J Consum Res"},{"key":"R22","unstructured":"Zao-Sanders M . How people are really using gen AI in 2025. Harvard Business Review. 2025. URL: https:\/\/hbr.org\/2025\/04\/how-people-are-really-using-gen-ai-in-2025 [Accessed 09-04-2025]"},{"issue":"6","key":"R23","doi-asserted-by":"publisher","first-page":"1081","DOI":"10.1177\/17456916251351306","article-title":"Can generative AI chatbots emulate human connection? A relationship science perspective","volume":"20","author":"Smith","journal-title":"Perspect Psychol Sci"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","DOI":"10.1038\/s44184-023-00047-6","article-title":"Loneliness and suicide mitigation for students using GPT3-enabled chatbots","volume":"3","author":"Maples","journal-title":"Npj Ment Health Res"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-026-02516-2","article-title":"Interaction with AI companions and psychological well-being","author":"Zhang","journal-title":"Nat Hum Behav"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.1145\/3715275.3732039","article-title":"Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers","author":"Moore","journal-title":"Proc ACM Conf Fairness Accountab Transpar"},{"issue":"1","key":"R27","doi-asserted-by":"publisher","DOI":"10.1038\/s44184-024-00097-4","article-title":"\u201cIt happened to be the perfect thing\u201d: experiences of generative AI chatbots for mental health","volume":"3","author":"Siddals","journal-title":"Npj Ment Health Res"},{"key":"R28","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2025.103555","article-title":"This is human intelligence debugging artificial intelligence\u201d: examining how people prompt GPT in seeking mental health support","volume":"203","author":"Li","journal-title":"Int J Hum Comput Stud"},{"issue":"3","key":"R29","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1177\/09637214261438379","article-title":"Combining psychology with artificial intelligence: what could possibly go wrong?","volume":"35","author":"van Rooij","journal-title":"Curr Dir Psychol Sci"},{"key":"R30","unstructured":"Kahneman D . Thinking, Fast and Slow. Farrar, Straus and Giroux; 2011. ISBN: 9780374275631"},{"issue":"9","key":"R31","doi-asserted-by":"publisher","first-page":"1632","DOI":"10.1080\/13607863.2025.2491026","article-title":"Perceived social support and relationship quality predict loneliness in older adults: a social network approach","volume":"29","author":"Wilson","journal-title":"Aging Ment Health"},{"issue":"4","key":"R32","doi-asserted-by":"publisher","DOI":"10.1056\/AIoa2400802","article-title":"Randomized trial of a generative AI chatbot for mental health treatment","volume":"2","author":"Heinz","journal-title":"NEJM AI"},{"issue":"1","key":"R33","doi-asserted-by":"publisher","DOI":"10.2196\/63806","article-title":"Effect of a cognitive behavioral therapy-based AI chatbot on depression and loneliness in Chinese university students: randomized controlled trial with financial stress moderation","volume":"13","author":"Wang","journal-title":"JMIR mHealth uHealth"},{"key":"R34","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2026.3657604","article-title":"A robot-led intervention for emotion regulation: from expression to reappraisal","volume":"2026","author":"Laban","journal-title":"IEEE Trans Affective Comput"},{"key":"R35","doi-asserted-by":"crossref","unstructured":"Banerjee S Agarwal A Singla S . LLMs will always hallucinate, and we need to live with this. In: Arai K , editor. Intelligent Systems and Applications Proceedings of the 2025 Intelligent Systems Conference (IntelliSys) Volume 4. Springer Nature; 2025:624-648. [doi: 10.1007\/978-3-031-99965-9_39]","DOI":"10.1007\/978-3-031-99965-9_39"},{"key":"R36","doi-asserted-by":"crossref","unstructured":"Bender EM Gebru T McMillan-Major A Shmitchell S . On the dangers of stochastic parrots: can language models be too big. In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency FAccT \u201921. Association for Computing Machinery; 2021:610-623. [doi: 10.1145\/3442188.3445922]","DOI":"10.1145\/3442188.3445922"},{"key":"R37","doi-asserted-by":"crossref","unstructured":"Atari M Xue MJ Park PS Blasi DE Henrich J . Which humans? PsyArXiv. Preprint posted online on Sep 23, 2023. [doi: 10.31234\/osf.io\/5b26t]","DOI":"10.31234\/osf.io\/5b26t"},{"key":"R38","doi-asserted-by":"crossref","unstructured":"Haider B Gorti A Chadha A Gaur M . Mental health equity in llms: leveraging multi-hop question answering to detect amplified and silenced perspectives. In: Tiwari S Ortiz-Rodr\u00edguez F Shivakumar A Shimizu C , editors. Artificial Intelligence: Towards Sustainable Intelligence Third International Conference, AI4S 2025, Dayton, USA, October 2\u20133, 2025, Proceedings. Springer; 2026:42-59. [doi: 10.1007\/978-3-032-20447-9_5]","DOI":"10.1007\/978-3-032-20447-9_5"},{"key":"R39","doi-asserted-by":"publisher","DOI":"10.1007\/s43681-024-00419-4","article-title":"Anthropomorphism in AI: hype and fallacy","volume":"4","author":"Placani","journal-title":"AI Ethics"},{"issue":"4","key":"R40","doi-asserted-by":"publisher","first-page":"864","DOI":"10.1037\/0033-295X.114.4.864","article-title":"On seeing human: a three-factor theory of anthropomorphism","volume":"114","author":"Epley","journal-title":"Psychol Rev"},{"issue":"22","key":"R41","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2415898122","article-title":"The benefits and dangers of anthropomorphic conversational agents","volume":"122","author":"Peter","journal-title":"Proc Natl Acad Sci U S A"},{"key":"R42","doi-asserted-by":"publisher","DOI":"10.1016\/j.chbr.2021.100070","article-title":"Social media use, social anxiety, and loneliness: a systematic review","volume":"3","author":"O\u2019Day","journal-title":"Comput Hum Behav Rep"},{"issue":"2","key":"R43","doi-asserted-by":"publisher","first-page":"144","DOI":"10.1177\/1754073914558466","article-title":"Empathy: a review of the concept","volume":"8","author":"Cuff","journal-title":"Emot Rev"},{"issue":"5","key":"R44","doi-asserted-by":"publisher","first-page":"675","DOI":"10.1038\/nn.3085","article-title":"The neuroscience of empathy: progress, pitfalls and promise","volume":"15","author":"Zaki","journal-title":"Nat Neurosci"},{"key":"R45","unstructured":"Kidder W D\u2019Cruz J Varshney KR . Empathy and the right to be an exception: what llms can and cannot do. arXiv. Preprint posted online on Jan 25, 2024. [doi: 10.48550\/arXiv.2401.14523]"},{"issue":"6","key":"R46","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","article-title":"Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum","volume":"183","author":"Ayers","journal-title":"JAMA Intern Med"},{"issue":"1","key":"R47","doi-asserted-by":"publisher","DOI":"10.1038\/s44271-024-00182-6","article-title":"Third-party evaluators perceive AI as more compassionate than expert humans","volume":"3","author":"Ovsyannikova","journal-title":"Commun Psychol"},{"issue":"14","key":"R48","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2319112121","article-title":"AI can help people feel heard, but an AI label diminishes this impact","volume":"121","author":"Yin","journal-title":"Proc Natl Acad Sci U S A"},{"key":"R49","doi-asserted-by":"publisher","DOI":"10.1177\/09637214261444274","article-title":"AI-generated empathy: opportunities, limits, and future directions","author":"Ong","journal-title":"Curr Dir Psychol Sci"},{"issue":"1","key":"R50","doi-asserted-by":"publisher","DOI":"10.1038\/s44271-025-00387-3","article-title":"People choose to receive human empathy despite rating AI empathy higher","volume":"4","author":"Wenger","journal-title":"Commun Psychol"},{"key":"R51","doi-asserted-by":"crossref","unstructured":"De Freitas J . AI companions as hyper attachment and caregiving targets. arXiv. Preprint posted online on May 15, 2026. [doi: 10.48550\/arXiv.2606.20589]","DOI":"10.2139\/ssrn.6802878"},{"key":"R52","doi-asserted-by":"publisher","DOI":"10.52202\/068431-2011","article-title":"Training language models to follow instructions with human feedback","author":"Ouyang","journal-title":"Proc 36th Int Conf Neural Inf Process Syst"},{"key":"R53","unstructured":"Sharma M Tong M Korbak T Duvenaud D Askell A Bowman S et al. Towards understanding sycophancy in language models. Presented at: Proceedings of the International Conference on Learning Representations (ICLR); May 7-11, 2024; Vienna , Austria. URL: https:\/\/openreview.net\/pdf?id=tvhaxkMKAn [Accessed 19-08-2026]"},{"key":"R54","unstructured":"Beacock I Xu R Murray L et al. AI usage patterns are shaped by perceived gains in human agency. arXiv. Preprint posted online on Jul 2, 2026. [doi: 10.48550\/arXiv.2607.02313]"},{"key":"R55","unstructured":"Ibrahim L Hafner FS Rocher L . Training language models to be warm and empathetic makes them less reliable and more sycophantic. arXiv. Preprint posted online on Jul 29, 2025. [doi: 10.48550\/arXiv.2507.21919]"},{"key":"R56","doi-asserted-by":"crossref","unstructured":"Fang CM Liu AR Danry V et al. How AI and human behaviors shape psychosocial effects of extended chatbot use: a longitudinal randomized controlled study. Research Square. Preprint posted online on Apr 27, 2026. [doi: 10.21203\/rs.3.rs-8148142\/v1]","DOI":"10.21203\/rs.3.rs-8148142\/v1"},{"issue":"3","key":"R57","doi-asserted-by":"publisher","first-page":"336","DOI":"10.1038\/s44220-026-00595-8","article-title":"Technological folie \u00e0 deux: feedback loops between AI chatbots and mental health","volume":"4","author":"Dohn\u00e1ny","journal-title":"Nat Ment Health"},{"key":"R58","unstructured":"Chandra K Kleiman-Weiner M Ragan-Kelley J Tenenbaum JB . Sycophantic chatbots cause delusional spiraling, even in ideal bayesians. arXiv. Preprint posted online on Feb 22, 2026. [doi: 10.48550\/arXiv.2602.19141]"},{"key":"R59","unstructured":"Yeung JA Dalmasso J Foschini L Dobson RJB Kraljevic Z . The psychogenic machine: simulating AI psychosis, delusion reinforcement and harm enablement in large language models. arXiv. Preprint posted online on Sep 13, 2025. [doi: 10.48550\/arXiv.2509.10970]"},{"key":"R60","unstructured":"Mehta A Moore J Anthis JR et al. The dynamics of delusion: modeling bidirectional false belief amplification in human-chatbot dialogue. arXiv. Preprint posted online on Apr 28, 2026. [doi: 10.48550\/arXiv.2604.25096]"},{"issue":"12","key":"R61","doi-asserted-by":"publisher","first-page":"976","DOI":"10.1111\/j.1751-9004.2011.00400.x","article-title":"Social baseline theory: the role of social proximity in emotion and economy of action","volume":"5","author":"Beckes","journal-title":"Soc Personal Psychol Compass"},{"key":"R62","doi-asserted-by":"crossref","unstructured":"De Freitas J Castelo N U\u011furalp AK O\u011fuz-U\u011furalp Z . Lessons from an app update at replika\n                      AI\n                      : identity discontinuity in human-AI relationships. arXiv. Preprint posted online on Dec 10, 2024. [doi: 10.48550\/arXiv.2412.14190]","DOI":"10.2139\/ssrn.4976449"},{"key":"R63","doi-asserted-by":"publisher","DOI":"10.20935\/MHealthWellB8256","article-title":"Technological mourning after AI updates: mental health and well-being in the GPT-4o\/GPT-5 transition","volume":"3","author":"Donard","journal-title":"Acad Ment Health Well-Being"},{"key":"R64","unstructured":"Rogers CR . A theory of therapy, personality, and interpersonal relationships, as developed in the client-centered framework. In: Koch S , editor. Psychology: A Study of A Science. McGraw-Hill; 1959:184-256. URL: https:\/\/chester-counselling.co.uk\/wp-content\/uploads\/2018\/08\/rogers-1959.pdf [Accessed 19-08-2026]"},{"key":"R65","unstructured":"Maes P Pataranutaporn P . Echo chambers of one: companion AI and the future of human connection. MIT Media Lab. URL: https:\/\/www.media.mit.edu\/articles\/echo-chambers-of-one-companion-ai-and-the-future-of-human-connection\/ [Accessed 24-02-2026]"},{"key":"R66","doi-asserted-by":"publisher","DOI":"10.63317\/4zo8zahv3d3q","article-title":"Large Language Models Are Echo Chambers","author":"Nehring","journal-title":"Proc LREC-COLING 2024"},{"key":"R67","unstructured":"Vecchione B Ye M Garofalo L Singh R . Engagement-optimized care: when llms become mental health infrastructure. arXiv. Preprint posted online on May 22, 2026. [doi: 10.48550\/arXiv.2605.23787]"},{"key":"R68","doi-asserted-by":"publisher","DOI":"10.1017\/neu.2025.10035","article-title":"Emotion contagion through interaction with generative artificial intelligence chatbots may contribute to development and maintenance of mania","volume":"37","author":"\u00d8stergaard","journal-title":"Acta Neuropsychiatr"}],"container-title":["JMIR Mental Health"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T19:42:47Z","timestamp":1789069367000},"score":1,"resource":{"primary":{"URL":"https:\/\/mental.jmir.org\/2026\/1\/e99354"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,10]]},"references-count":68,"URL":"https:\/\/doi.org\/10.2196\/99354","relation":{},"ISSN":["2368-7959"],"issn-type":[{"value":"2368-7959","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,10]]},"article-number":"v13i3e99354"}}