{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,3]],"date-time":"2026-11-03T01:36:23Z","timestamp":1793669783123,"version":"4.5.1"},"reference-count":61,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Cancer"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Patients with breast cancer frequently experience significant uncertainty, prompting them to seek detailed, personalized, and reliable medical information to enhance adherence to prescribed treatments, medications, and recommended lifestyle adjustments. Although high-quality information exists within oncology guidelines and patient-oriented resources, the provision of tailored responses to individual patient queries remains challenging, especially for non\u2013English-speaking populations.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to evaluate the potential of an artificial intelligence\u2013driven chatbot, specifically leveraging ChatGPT (GPT-4; OpenAI) combined with retrieval-augmented generation, to deliver personalized answers to complex breast cancer-related patient questions in German.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We collaborated with one of Germany\u2019s largest breast cancer Patient Representation Groups to collect authentic patient inquiries, receiving a total of 118 questions. After initial screening, we selected 104 medical questions, organized into 7 distinct categories: aftercare, bone health, ductal carcinoma in situ, diagnostics, nutrition and supplements, complementary medicine, and therapy. A customized version of GPT-4 was configured with specific system prompts emphasizing empathetic, evidence-based responses and integrated with a comprehensive database comprising guidelines, recommendations, and patient information materials published by recognized German medical societies. To assess chatbot responses, we used 4 evaluation criteria: comprehensibility (clarity from a patient perspective), correctness (accuracy per current medical guidelines), completeness (inclusion of all relevant aspects), and potential harm (risk of undue patient harm or misinformation). Ratings were conducted using a 5-point Likert scale by a breast cancer expert (correctness, completeness, and potential harm) and patient representatives (comprehensibility).<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n The chatbot provided high-quality responses across multiple dimensions. Of the 499 responses evaluated for comprehensibility, 427 (85.6%) were rated as comprehensible. Among the 104 responses assessed for the remaining dimensions, 91 (87.5%) were rated as correct, 72 (69.2%) as complete, and 93 (89.4%) as nonharmful. Reasons for incomplete answers included omission of reimbursement details, updates from recent therapeutic guidelines, or nuanced recommendations regarding endocrine therapy and aftercare schedules. In addition, 6 (5.8%) of the answers were rated as potentially harmful due to outdated or contextually inappropriate recommendations. The chatbot also performed well in the nutrition and bone health categories despite occasionally incomplete document retrieval.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Our findings demonstrate that an artificial intelligence\u2013powered chatbot with GPT-4 and retrieval augmentation can effectively provide personalized, linguistically accessible, and largely accurate information to German-speaking patients with breast cancer. This approach holds considerable promise for improving patient-centered communication, empowering patients to make informed decisions. Nonetheless, observed limitations regarding response completeness and potential harm underscore the critical need for ongoing human oversight. Future research and development should prioritize regularly updated databases, advanced retrieval methods to handle complex document structures, multimodal capabilities, and clearly articulated disclaimers emphasizing the necessity of professional medical consultation. Our evaluation, along with the provided set of realistic patient questions, establishes a benchmark for future development and validation of German-language oncology chatbots.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/68426","type":"journal-article","created":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T11:20:07Z","timestamp":1750159207000},"page":"e68426-e68426","source":"Crossref","is-referenced-by-count":14,"title":["Evaluating a Chatbot as a Companion for Patients With Breast Cancer: Collaborative Pilot Study"],"prefix":"10.2196","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7993-7901","authenticated-orcid":false,"given":"Sebastian Daniel","family":"Boie","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-9987-4319","authenticated-orcid":false,"given":"Esther","family":"Glastetter","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2781-2178","authenticated-orcid":false,"given":"Michael Patrick","family":"Lux","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1575-2056","authenticated-orcid":false,"given":"Felix","family":"Balzer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9221-3297","authenticated-orcid":false,"given":"Christof","family":"von Kalle","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-0863-2219","authenticated-orcid":false,"given":"Christian","family":"Lenz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-4255-9664","authenticated-orcid":false,"given":"Ulrike","family":"M\u00fcller","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,8,13]]},"reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.1016\/j.breast.2022.08.010","article-title":"Current and future burden of breast cancer: global statistics for 2020 and 2040","volume":"66","author":"Arnold","journal-title":"Breast"},{"issue":"17","key":"R2","doi-asserted-by":"publisher","DOI":"10.3390\/cancers13174287","article-title":"Breast cancer-epidemiology, risk factors, classification, prognostic markers, and current treatment strategies-an updated review","volume":"13","author":"\u0141ukasiewicz","journal-title":"Cancers (Basel)"},{"issue":"4","key":"R3","doi-asserted-by":"publisher","DOI":"10.1258\/shorts.2010.010032","article-title":"How do the information needs of cancer patients differ at different stages of the cancer journey? A cross-sectional survey","volume":"1","author":"Mistry","journal-title":"JRSM Short Rep"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1002\/cam4.1915","article-title":"What information do patients want across their cancer journeys? A network analysis of cancer patients\u2019 information needs","volume":"8","author":"Tran","journal-title":"Cancer Med"},{"issue":"2","key":"R5","doi-asserted-by":"publisher","first-page":"408","DOI":"10.1002\/cncr.32524","article-title":"Changes in internet use and wishes of cancer survivors: A comparison between 2005 and 2017","volume":"126","author":"van Eenbergen","journal-title":"Cancer"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcpo.2022.100375","article-title":"Cancer literacy - Informing patients and implementing shared decision making","volume":"35","author":"Buyens","journal-title":"J Cancer Policy"},{"issue":"2","key":"R7","doi-asserted-by":"publisher","first-page":"130","DOI":"10.1002\/pon.3611","article-title":"Informing women with breast cancer about endocrine therapy: effects on knowledge and adherence","volume":"24","author":"Heisig","journal-title":"Psychooncology"},{"issue":"7","key":"R8","doi-asserted-by":"publisher","DOI":"10.3390\/pharmaceutics13071100","article-title":"Factors contributing to medication adherence in patients with a chronic condition: a scoping review of qualitative research","volume":"13","author":"Kvarnstr\u00f6m","journal-title":"Pharmaceutics"},{"issue":"Suppl 1","key":"R9","doi-asserted-by":"publisher","first-page":"S3","DOI":"10.3747\/co.v18i0.899","article-title":"Patient adherence to aromatase inhibitor treatment in the adjuvant setting","volume":"18 Suppl 1","author":"Verma","journal-title":"Curr Oncol"},{"issue":"8","key":"R10","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.9364","article-title":"Relationship between internet health information and patient compliance based on trust: empirical study","volume":"20","author":"Lu","journal-title":"J Med Internet Res"},{"issue":"10","key":"R11","doi-asserted-by":"publisher","DOI":"10.2196\/11515","article-title":"Nature and diffusion of gynecologic cancer\u2013related misinformation on social media: analysis of tweets","volume":"20","author":"Chen","journal-title":"J Med Internet Res"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.2196\/43749","article-title":"Exposure and reactions to cancer treatment misinformation and advice: survey study","volume":"9","author":"Lazard","journal-title":"JMIR Cancer"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1177\/20552076221116772","article-title":"Attitudes and perception of artificial intelligence in healthcare: a cross-sectional survey among patients","volume":"8","author":"Fritsch","journal-title":"Digit Health"},{"issue":"2","key":"R14","doi-asserted-by":"publisher","DOI":"10.1093\/jncics\/pkad010","article-title":"Artificial intelligence chatbots will revolutionize how cancer patients access information: ChatGPT represents a paradigm-shift","volume":"7","author":"Hopkins","journal-title":"JNCI Cancer Spectr"},{"issue":"1","key":"R15","doi-asserted-by":"publisher","first-page":"100046","DOI":"10.1016\/j.chbah.2024.100046","article-title":"Fine for others but not for me: the role of perspective in patients\u2019 perception of artificial intelligence in online medical platforms","volume":"2","author":"Hudecek","journal-title":"Computers in Human Behavior: Artificial Humans"},{"issue":"9","key":"R16","doi-asserted-by":"publisher","first-page":"e599","DOI":"10.1016\/S2589-7500(21)00132-1","article-title":"Patient and general public attitudes towards clinical artificial intelligence: a mixed methods systematic review","volume":"3","author":"Young","journal-title":"Lancet Digit Health"},{"issue":"14","key":"R17","doi-asserted-by":"publisher","first-page":"1607","DOI":"10.1200\/JCO.23.02439","article-title":"Promise and perils of large language models for cancer survivorship and supportive care","volume":"42","author":"Bitterman","journal-title":"J Clin Oncol"},{"issue":"4","key":"R18","doi-asserted-by":"publisher","first-page":"e196","DOI":"10.1016\/S2665-9913(24)00056-0","article-title":"ChatSLE: consulting ChatGPT-4 for 100 frequently asked lupus questions","volume":"6","author":"Haase","journal-title":"Lancet Rheumatol"},{"issue":"6","key":"R19","doi-asserted-by":"publisher","first-page":"1488","DOI":"10.1016\/j.pec.2021.09.027","article-title":"Ask Rosa - The making of a digital genetic conversation tool, a chatbot, about hereditary breast and ovarian cancer","volume":"105","author":"Siglen","journal-title":"Patient Educ Couns"},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.2196\/51300","article-title":"An AI dietitian for type 2 diabetes mellitus management based on large language and image recognition models: preclinical concept validation study","volume":"25","author":"Sun","journal-title":"J Med Internet Res"},{"issue":"2","key":"R21","doi-asserted-by":"publisher","DOI":"10.5001\/omj.2020.40","article-title":"Implications of language barriers for healthcare: a systematic review","volume":"35","author":"Al Shamsi","journal-title":"Oman Med J"},{"key":"R22","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2023.121598","article-title":"medBERT.de: a comprehensive German BERT model for the medical domain","volume":"237","author":"Bressem","journal-title":"Expert Syst Appl"},{"key":"R23","unstructured":"Tikkanen R Osborn R Mossialos E Djordjevic A . International Profiles of Health Care Systems. Commonwealth Fund; 2020. URL: https:\/\/www.commonwealthfund.org\/sites\/default\/files\/2020-12\/International_Profiles_of_Health_Care_Systems_Dec2020.pdf"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6963-8-198","article-title":"Country differences in the diagnosis and management of coronary heart disease - a comparison between the US, the UK and Germany","volume":"8","author":"von dem Knesebeck","journal-title":"BMC Health Serv Res"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","first-page":"160","DOI":"10.1016\/j.euo.2023.06.009","article-title":"Development of a personalized chat model based on the european association of urology oncology guidelines: harnessing the power of generative artificial intelligence in clinical practice","volume":"7","author":"Khene","journal-title":"Eur Urol Oncol"},{"key":"R26","unstructured":"Huang L Yu W Ma W et al. A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. arXiv. Preprint posted online on Nov 9, 2023. [doi: 10.48550\/arXiv.2311.05232]"},{"key":"R27","doi-asserted-by":"crossref","unstructured":"Rawte V Chakraborty S Pathak A et al. The troubling emergence of hallucination in large language models -- an extensive definition, quantification, and prescriptive remediations. arXiv. Preprint posted online on Oct 8, 2023. [doi: 10.48550\/arXiv.2310.04988]","DOI":"10.18653\/v1\/2023.emnlp-main.155"},{"issue":"1","key":"R28","doi-asserted-by":"publisher","DOI":"10.2196\/26911","article-title":"Development andearly feasibility of chatbots for educating patients with lung cancer and their caregivers in Japan: mixed methods study","volume":"7","author":"Kataoka","journal-title":"JMIR Cancer"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.1177\/20552076231173304","article-title":"An artificial intelligence-based chatbot for prostate cancer education: Design and patient evaluation study","volume":"9","author":"G\u00f6rtz","journal-title":"Digit Health"},{"issue":"5","key":"R30","doi-asserted-by":"publisher","first-page":"1158","DOI":"10.1097\/HEP.0000000000000834","article-title":"Development of a liver disease-specific large language model chat interface using retrieval-augmented generation","volume":"80","author":"Ge","journal-title":"Hepatology"},{"issue":"10","key":"R31","doi-asserted-by":"publisher","first-page":"2396","DOI":"10.1038\/s41591-023-02412-6","article-title":"Large language model AI chatbots require approval as medical devices","volume":"29","author":"Gilbert","journal-title":"Nat Med"},{"key":"R32","unstructured":"Tonmoy S Zaman SMM Jain V et al. A comprehensive survey of hallucination mitigation techniques in large language models. Jan 2, 2024. [doi: 10.48550\/arXiv.2401.01313]"},{"key":"R33","unstructured":"Lewis P Perez E Piktus A et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. May 22, 2020. [doi: 10.48550\/arXiv.2005.11401]"},{"issue":"20","key":"R34","doi-asserted-by":"publisher","first-page":"2563","DOI":"10.1200\/JCO.2012.46.8371","article-title":"Critical evaluation of oncology clinical practice guidelines","volume":"31","author":"Reames","journal-title":"J Clin Oncol"},{"issue":"2","key":"R35","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1055\/a-1463-8544","article-title":"Behandlung von Patientinnen mit fr\u00fchem Mammakarzinom: Evidenz, Kontroversen, Konsens \u2013 Meinungsbild deutscher Expert*innen zur 17. Internationalen St.-Gallen-Konsensuskonferenz [Article in German]","volume":"18","author":"Untch","journal-title":"Senologie - Zeitschrift F\u00fcr Mammadiagnostik und -therapie"},{"issue":"4","key":"R36","doi-asserted-by":"publisher","first-page":"452","DOI":"10.6004\/jnccn.2020.0016","article-title":"Breast cancer, version 3.2020, NCCN clinical practice guidelines in oncology","volume":"18","author":"Gradishar","journal-title":"J Natl Compr Canc Netw"},{"key":"R37","unstructured":"OpenAIAdler S Agarwal S et al. GPT-4 technical report. arXiv. Preprint posted online on Mar 15, 2024. [doi: 10.48550\/arXiv.2303.08774]"},{"key":"R38","unstructured":"OpenAI. Introducing GPTs. 2023. URL: https:\/\/openai.com\/index\/introducing-gpts\/"},{"key":"R39","unstructured":"Finardi P Avila L Castaldoni R et al. The chronicles of RAG: the retriever, the chunk and the generator. 2024. URL: http:\/\/arxiv.org\/abs\/2401.07883"},{"key":"R40","doi-asserted-by":"crossref","unstructured":"Popat SK Deshmukh PB Metre VA . Hierarchical document clustering based on cosine similarity measure. Presented at: 2017 1st International Conference on Intelligent Systems and Information Management (ICISIM). Oct 5-6, 2017:IEEE. 153-159; Aurangabad, India. [doi: 10.1109\/ICISIM.2017.8122166]","DOI":"10.1109\/ICISIM.2017.8122166"},{"issue":"5","key":"R41","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2023-081155","article-title":"Protocol for the development of the Chatbot Assessment Reporting Tool (CHART) for clinical advice","volume":"14","author":"The CHART Collaborative","journal-title":"BMJ Open"},{"issue":"8","key":"R42","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.30320","article-title":"Comparison of ophthalmologist and large language model chatbot responses to online patient eye care questions","volume":"6","author":"Bernstein","journal-title":"JAMA Netw Open"},{"issue":"5","key":"R43","doi-asserted-by":"publisher","first-page":"2817","DOI":"10.1007\/s00330-023-10213-1","article-title":"ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports","volume":"34","author":"Jeblick","journal-title":"Eur Radiol"},{"issue":"1","key":"R44","doi-asserted-by":"publisher","DOI":"10.1186\/s41747-024-00526-1","article-title":"Probing clarity: AI-generated simplified breast imaging reports for enhanced patient comprehension powered by ChatGPT-4o","volume":"8","author":"Maroncelli","journal-title":"Eur Radiol Exp"},{"issue":"3","key":"R45","doi-asserted-by":"publisher","first-page":"721","DOI":"10.3350\/cmh.2023.0089","article-title":"Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma","volume":"29","author":"Yeo","journal-title":"Clin Mol Hepatol"},{"key":"R46","unstructured":"Janni W M\u00fcller V . Patientenratgeber Zu Den AGO-Empfehlungen 2023 [Book in German]. Zuckschwerdt Verlag M\u00fcnchen; 2023."},{"key":"R47","unstructured":"W\u00f6ckel A Kreienberg R Janni W . Interdisziplin\u00e4re S3-leitlinie f\u00fcr die fr\u00fcherkennung, diagnostik. In: Therapie Und Nachsorge Des Mammakarzinoms [Book in German]. 2021."},{"key":"R48","unstructured":"W\u00f6rmann B Aebi S Balic M et al. Mammakarzinom der frau. In: DGHO Deutsche Gesellschaft F\u00fcr H\u00e4matologie Und Medizinische Onkologie eV [Book in German]. 2018."},{"key":"R49","doi-asserted-by":"crossref","unstructured":"Borchert F Lohr C Modersohn L et al. GGPONC: a corpus of german medical text with rich metadata based on clinical practice guidelines. arXiv. Preprint posted online on Jul 13, 2020. [doi: 10.48550\/arXiv.2007.06400]","DOI":"10.18653\/v1\/2020.louhi-1.5"},{"key":"R50","unstructured":"Lahiri AK Hu QV . AlzheimerRAG: multimodal retrieval augmented generation for PubMed articles. Dec 21, 2024. [doi: 10.48550\/arXiv.2412.16701]"},{"issue":"10","key":"R51","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.36483","article-title":"Accuracy and reliability of chatbot responses to physician questions","volume":"6","author":"Goodman","journal-title":"JAMA Netw Open"},{"key":"R52","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2023.1268915","article-title":"Applications of large language models in cancer care: current evidence and future perspectives","volume":"13","author":"Iannantuono","journal-title":"Front Oncol"},{"issue":"1","key":"R53","doi-asserted-by":"publisher","DOI":"10.1056\/AIra2400380","article-title":"RAG in health care: a novel framework for improving communication and decision-making by addressing LLM limitations","volume":"2","author":"Ng","journal-title":"NEJM AI"},{"key":"R54","doi-asserted-by":"crossref","unstructured":"Quidwai MA Lagana A . A RAG chatbot for precision medicine of multiple myeloma. Genetic and Genomic Medicine. Preprint posted online on 2024. [doi: 10.1101\/2024.03.14.24304293]","DOI":"10.1101\/2024.03.14.24304293"},{"issue":"1","key":"R55","doi-asserted-by":"publisher","DOI":"10.1186\/s12885-023-11122-8","article-title":"Importance of endocrine treatment adherence and persistence in breast cancer survivorship: a systematic review","volume":"23","author":"Eliassen","journal-title":"BMC Cancer"},{"issue":"10","key":"R56","doi-asserted-by":"publisher","first-page":"1375","DOI":"10.1001\/jamaoncol.2018.2487","article-title":"Complementary medicine, refusal of conventional cancer therapy, and survival among patients with curable cancers","volume":"4","author":"Johnson","journal-title":"JAMA Oncol"},{"key":"R57","doi-asserted-by":"publisher","DOI":"10.2196\/55318","article-title":"An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing: algorithm development and validation study","volume":"12","author":"Sivarajkumar","journal-title":"JMIR Med Inform"},{"key":"R58","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2024.100213","article-title":"Applying large language models and chain-of-thought for automatic scoring","volume":"6","author":"Lee","journal-title":"Computers and Education: Artificial Intelligence"},{"key":"R59","unstructured":"Yuan J Bao P Chen Z et al. Advanced prompting as a catalyst: empowering large language models in the management of gastrointestinal cancers."},{"key":"R60","unstructured":"Xia Y Zhou J Shi Z Chen J Huang H . Improving retrieval augmented language model with self-reasoning. Jul 29, 2024. [doi: 10.48550\/arXiv.2407.19813]"},{"key":"R61","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2404.11792","article-title":"Enhancing Q&A with domain-specific fine-tuning and iterative reasoning: a comparative study","author":"Nguyen","journal-title":"arXiv"}],"container-title":["JMIR Cancer"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/cancer.jmir.org\/2025\/1\/e68426","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:55:12Z","timestamp":1782842112000},"score":1,"resource":{"primary":{"URL":"https:\/\/cancer.jmir.org\/2025\/1\/e68426"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,13]]},"references-count":61,"URL":"https:\/\/doi.org\/10.2196\/68426","relation":{},"ISSN":["2369-1999"],"issn-type":[{"value":"2369-1999","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,13]]},"article-number":"v11i17e68426"}}