{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T03:46:45Z","timestamp":1787024805270,"version":"build-2736575974"},"reference-count":45,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR AI"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n The widespread adoption of artificial intelligence (AI)\u2013powered search engines has transformed how people access health information. Microsoft Copilot, formerly Bing Chat, offers real-time web-sourced responses to user queries, raising concerns about the reliability of its health content. This is particularly critical in the domain of dietary supplements, where scientific consensus is limited and online misinformation is prevalent. Despite the popularity of supplements in Japan, little is known about the accuracy of AI-generated advice on their effectiveness for common diseases.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n We aimed to evaluate the reliability and accuracy of Microsoft Copilot, an AI search engine, in responding to health-related queries about dietary supplements. Our findings can help consumers use large language models more safely and effectively when seeking information on dietary supplements and support developers in improving large language models\u2019 performance in this field.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We simulated typical consumer behavior by posing 180 questions (6 per supplement \u00d7 30 supplements) to Copilot\u2019s 3 response modes (creative, balanced, and precise) in Japanese. These questions addressed the effectiveness of supplements in treating 6 common conditions (cancer, diabetes, obesity, constipation, joint pain, and hypertension). We classified the AI search engine\u2019s answers as \u201ceffective,\u201d \u201cuncertain,\u201d or \u201cineffective\u201d and evaluated for accuracy against evidence-based assessments conducted by licensed physicians. We conducted a qualitative content analysis of the response texts and systematically examined the types of sources cited in all responses.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n The proportion of Copilot responses claiming supplement effectiveness was 29.4% (53\/180), 47.8% (86\/180), and 45% (81\/180) for the creative, balanced, and precise modes, respectively, whereas overall accuracy of the responses was low across all modes: 36.1% (65\/180), 31.7% (57\/180), and 31.7% (57\/180) for creative, balanced, and precise, respectively. No significant difference was observed among the 3 modes (\n P=<\/jats:italic>\n .59). Notably, 72.7% (2240\/3081) of the citations came from unverified sources such as blogs, sales websites, and social media. Of the 540 responses analyzed, 54 (10%) contained at least 1 citation in which the cited source did not include or support the claim made by Copilot, indicating hallucinated content. Only 48.5% (262\/540) of the responses included a recommendation to consult health care professionals. Among disease categories, the highest accuracy was found for cancer-related questions, likely due to lower misinformation prevalence.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n This is the first study to assess Copilot\u2019s performance on dietary supplement information. Despite its authoritative appearance, Copilot frequently cited noncredible sources and provided ambiguous or inaccurate information. Its tendency to avoid definitive stances and align with perceived user expectations poses potential risks for health misinformation. These findings highlight the need for integrating health communication principles\u2014such as transparency, audience empowerment, and informed choice\u2014into the development and regulation of AI search engines to ensure safe public use.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/78436","type":"journal-article","created":{"date-parts":[[2025,10,29]],"date-time":"2025-10-29T18:22:26Z","timestamp":1761762146000},"page":"e78436-e78436","source":"Crossref","is-referenced-by-count":4,"title":["Evaluating the Reliability and Accuracy of an AI-Powered Search Engine in Providing Responses on Dietary Supplements: Quantitative and Qualitative Evaluation"],"prefix":"10.2196","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6320-544X","authenticated-orcid":false,"given":"Mingxin","family":"Liu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6251-3587","authenticated-orcid":false,"given":"Tsuyoshi","family":"Okuhara","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6427-210X","authenticated-orcid":false,"given":"Ritsuko","family":"Shirabe","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-9004-4182","authenticated-orcid":false,"given":"Yuriko","family":"Nishiie","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7811-453X","authenticated-orcid":false,"given":"Yinghan","family":"Xu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7877-9753","authenticated-orcid":false,"given":"Hiroko","family":"Okada","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5934-0681","authenticated-orcid":false,"given":"Takahiro","family":"Kiuchi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,10,29]]},"reference":[{"issue":"4","key":"R1","doi-asserted-by":"publisher","DOI":"10.2196\/36463","article-title":"Consumers\u2019 evaluation of web-based health information quality: meta-analysis","volume":"24","author":"Zhang","journal-title":"J Med Internet Res"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.1016\/j.pec.2025.108677","article-title":"Are AI chatbots concordant with evidence-based cancer screening recommendations?","volume":"134","author":"Nickel","journal-title":"Patient Educ Couns"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.2196\/48568","article-title":"Utility of ChatGPT in clinical practice","volume":"25","author":"Liu","journal-title":"J Med Internet Res"},{"key":"R4","unstructured":"Microsoft launches the new Bing, with ChatGPT built in. TechCrunch. URL: https:\/\/techcrunch.com\/2023\/02\/07\/microsoft-launches-the-new-bing-with-chatgpt-built-in\/ [Accessed 12-01-2025]"},{"key":"R5","unstructured":"Microsoft rebrands Bing Chat to Copilot, to better compete with ChatGPT. The Verge. URL: https:\/\/www.theverge.com\/2023\/11\/15\/23960517\/microsoft-copilot-bing-chat-rebranding-chatgpt-ai?utm_source=chatgpt.com [Accessed 12-01-2025]"},{"key":"R6","unstructured":"Laukkonen J . Microsoft Copilot: what it is, where to find it, and how much it costs. Lifewire Tech for Humans. URL: https:\/\/www.lifewire.com\/what-is-bing-ai-chatbot-7371141 [Accessed 12-01-2025]"},{"issue":"1","key":"R7","doi-asserted-by":"publisher","first-page":"1007","DOI":"10.1002\/pra2.927","article-title":"Bing chat: the future of search engines?","volume":"60","author":"Kelly","journal-title":"Proc Assoc Inf Sci Technol"},{"key":"R8","unstructured":"GPT-4. OpenAI. URL: https:\/\/platform.openai.com\/docs\/models\/gpt-4 [Accessed 12-02-2025]"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1007\/s42001-024-00338-8","article-title":"In generative AI we trust: can chatbots effectively verify political information?","volume":"8","author":"Kuznetsova","journal-title":"J Comput Soc Sc"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.2196\/53086","article-title":"Search engines and generative artificial intelligence integration: public health risks and recommendations to safeguard consumers online","volume":"10","author":"Ashraf","journal-title":"JMIR Public Health Surveill"},{"key":"R11","unstructured":"Top websites ranking. Similarweb. URL: https:\/\/www.similarweb.com\/engines\/ [Accessed 12-01-2025]"},{"key":"R12","unstructured":"Generative AI & legal research. Delaware Law School, Widener University. URL: https:\/\/libguides.law.widener.edu\/c.php?g=1342893&p=10140129 [Accessed 12-09-2025]"},{"key":"R13","unstructured":"Microsoft Copilot revenue and usage statistics (2025). Business of Apps. 2025. URL: https:\/\/www.businessofapps.com\/data\/microsoft-copilot-statistics\/?utm_source=chatgpt.com [Accessed 11-09-2025]"},{"issue":"5","key":"R14","doi-asserted-by":"publisher","first-page":"1160","DOI":"10.3390\/clinpract13050104","article-title":"AI-powered renal diet support: performance of ChatGPT, Bard AI, and Bing Chat","volume":"13","author":"Qarajeh","journal-title":"Clin Pract"},{"issue":"9","key":"R15","doi-asserted-by":"publisher","first-page":"3335","DOI":"10.1108\/BFJ-02-2024-0158","article-title":"AI showdown: info accuracy on protein quality content in foods from ChatGPT 3.5, ChatGPT 4, Bard AI and Bing chat","volume":"126","author":"Bayram","journal-title":"Br Food J"},{"key":"R16","unstructured":"Dietary supplements market size and share report. Grand View Research. URL: https:\/\/www.grandviewresearch.com\/industry-analysis\/dietary-supplements-market-report [Accessed 13-10-2025]"},{"key":"R17","unstructured":"Japan: share of people taking dietary supplements 2023. Statista. URL: https:\/\/www.statista.com\/statistics\/1182707\/japan-share-of-people-taking-dietary-supplements\/ [Accessed 12-10-2025]"},{"issue":"7","key":"R18","doi-asserted-by":"publisher","first-page":"1345","DOI":"10.1017\/S1368980023000873","article-title":"Quality and accuracy of online nutrition-related information: a systematic review of content analysis studies","volume":"26","author":"Denniss","journal-title":"Public Health Nutr"},{"issue":"2","key":"R19","doi-asserted-by":"publisher","first-page":"618","DOI":"10.1038\/s41591-024-03445-1","article-title":"Medical large language models are vulnerable to data-poisoning attacks","volume":"31","author":"Alber","journal-title":"Nat Med"},{"key":"R20","unstructured":"Healthy foods market stats and prospects: market survey edition [Web page in Japanese]. Yano Research Institute Ltd. 2023. URL: https:\/\/www.yano.co.jp\/market_reports\/C64130600 [Accessed 11-09-2025]"},{"key":"R21","unstructured":"Health food [Web page in Japanese]. National Institute of Health and Nutrition. URL: https:\/\/www.nibiohn.go.jp\/eiken\/info\/hf2.html [Accessed 20-11-2023]"},{"key":"R22","unstructured":"Copilot. Microsoft. URL: https:\/\/copilot.microsoft.com\/ [Accessed 20-11-2023]"},{"key":"R23","unstructured":"What are the conversation styles of Microsoft Copilot? My FSU Service Center. URL: https:\/\/servicecenter.fsu.edu\/s\/article\/What-are-the-conversation-styles-of-Microsoft-Copilot [Accessed 11-09-2025]"},{"key":"R24","unstructured":"Introduction to Microsoft Copilot: the AI assistant for Microsoft 365. The Inform Team. URL: https:\/\/www.theinformteam.com\/blog\/introduction-to-microsoft-copilot\/ [Accessed 11-09-2025]"},{"issue":"2","key":"R25","doi-asserted-by":"publisher","first-page":"350","DOI":"10.1016\/j.ajodo.2015.05.012","article-title":"Comparison of 2 means (independent z test or independent t test)","volume":"148","author":"Pandis","journal-title":"Am J Orthod Dentofacial Orthop"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.2196\/60807","article-title":"Performance of ChatGPT across different versions in medical licensing examinations worldwide: systematic review and meta-analysis","volume":"26","author":"Liu","journal-title":"J Med Internet Res"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2024.105673","article-title":"Evaluating the effectiveness of advanced large language models in medical knowledge: a comparative study using Japanese national medical examination","volume":"193","author":"Liu","journal-title":"Int J Med Inform"},{"issue":"1","key":"R28","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1016\/j.identj.2024.10.014","article-title":"Large language models in dental licensing examinations: systematic review and meta-analysis","volume":"75","author":"Liu","journal-title":"Int Dent J"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.2196\/48002","article-title":"Performance of GPT-3.5 and GPT-4 on the Japanese medical licensing examination: comparison study","volume":"9","author":"Takagi","journal-title":"JMIR Med Educ"},{"issue":"10","key":"R30","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0312771","article-title":"Performance of ChatGPT in medical licensing examinations in countries worldwide: a systematic review and meta-analysis protocol","volume":"19","author":"Liu","journal-title":"PLoS ONE"},{"key":"R31","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.c4675","article-title":"Effects of glucosamine, chondroitin, or placebo in patients with osteoarthritis of hip or knee: network meta-analysis","volume":"341","author":"Wandel","journal-title":"BMJ"},{"issue":"3","key":"R32","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1007\/s00296-009-0969-5","article-title":"Effect of glucosamine or chondroitin sulfate on the osteoarthritis progression: a meta-analysis","volume":"30","author":"Lee","journal-title":"Rheumatol Int"},{"issue":"3","key":"R33","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1057\/s41271-018-0131-2","article-title":"Drug information, misinformation, and disinformation on social media: a content analysis study","volume":"39","author":"Al Khaja","journal-title":"J Public Health Policy"},{"key":"R34","unstructured":"AI-powered Bing chat gains three distinct personalities. ArsTechnica. URL: https:\/\/arstechnica.com\/information-technology\/2023\/03\/microsoft-equips-bing-chat-with-multiple-personalities-creative-balanced-precise\/ [Accessed 20-02-2025]"},{"key":"R35","unstructured":"Are dietary supplements safe? American Cancer Society. URL: https:\/\/www.cancer.org\/cancer\/managing-cancer\/treatment-types\/complementary-and-integrative-medicine\/dietary-supplements\/safety.html [Accessed 13-10-2025]"},{"key":"R36","unstructured":"Popular diets, supplements, and cancer. National Cancer Institute. URL: https:\/\/www.cancer.gov\/about-cancer\/treatment\/cam\/diets-supplements [Accessed 20-02-2025]"},{"issue":"10","key":"R37","doi-asserted-by":"publisher","first-page":"732","DOI":"10.1093\/jnci\/djs195","article-title":"Dietary supplements and cancer prevention: balancing potential benefits against proven harms","volume":"104","author":"Mart\u00ednez","journal-title":"J Natl Cancer Inst"},{"issue":"11","key":"R38","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2021-050898","article-title":"Japanese newspaper advertisements for dietary supplements before and after COVID-19: a content analysis","volume":"11","author":"Okuhara","journal-title":"BMJ Open"},{"key":"R39","unstructured":"Perez E Ringer S Luko\u0161i\u016bt\u0117 K et al. Discovering language model behaviors with model-written evaluations. arXiv. Preprint posted online on Dec 19, 2022. [doi: 10.48550\/arXiv.2212.09251]"},{"key":"R40","unstructured":"Liu Y Yao Y Ton JF et al. Trustworthy LLMs: a survey and guideline for evaluating large language models\u2019 alignment. arXiv. Preprint posted online on Aug 10, 2023. [doi: 10.48550\/arXiv.2308.05374]"},{"key":"R41","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajo.2023.05.024","article-title":"Performance of generative large language models on ophthalmology board-style questions","volume":"254","author":"Cai","journal-title":"Am J Ophthalmol"},{"key":"R42","doi-asserted-by":"crossref","unstructured":"Pal A Umapathi LK Sankarasubbu M . Med-halt: medical domain hallucination test for large language models. Presented at: Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL); Dec 6-7, 2023; Singapore. [doi: 10.18653\/v1\/2023.conll-1.21]","DOI":"10.18653\/v1\/2023.conll-1.21"},{"issue":"4","key":"R43","doi-asserted-by":"publisher","DOI":"10.1016\/j.advnut.2025.100398","article-title":"A scoping review of artificial intelligence for precision nutrition","volume":"16","author":"Wu","journal-title":"Adv Nutr"},{"issue":"12","key":"R44","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3571730","article-title":"Survey of hallucination in natural language generation","volume":"55","author":"Ji","journal-title":"ACM Comput Surv"},{"key":"R45","doi-asserted-by":"crossref","unstructured":"Kim SSY Liao QV Vorvoreanu M Ballard S Vaughan JW . \u201cI\u2019m not sure, but...\u201d: examining the impact of large language models\u2019 uncertainty expression on user reliance and trust. Presented at: Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT \u201924); Jun 3-6, 2024; Rio de Janeiro, Brazil. URL: https:\/\/dl.acm.org\/doi\/proceedings\/10.1145\/3630106 [Accessed 12-10-2025]","DOI":"10.1145\/3630106.3658941"}],"container-title":["JMIR AI"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ai.jmir.org\/2025\/1\/e78436","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:11:05Z","timestamp":1782843065000},"score":1,"resource":{"primary":{"URL":"https:\/\/ai.jmir.org\/2025\/1\/e78436"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,29]]},"references-count":45,"URL":"https:\/\/doi.org\/10.2196\/78436","relation":{},"ISSN":["2817-1705"],"issn-type":[{"value":"2817-1705","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,29]]},"article-number":"v4i4e78436"}}