{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T07:04:10Z","timestamp":1789369450509,"version":"build-2803163510"},"reference-count":35,"publisher":"SAGE Publications","issue":"3","license":[{"start":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T00:00:00Z","timestamp":1782864000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"},{"start":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T00:00:00Z","timestamp":1782864000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Health Informatics J"],"published-print":{"date-parts":[[2026,7]]},"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Chatbots are increasingly used in healthcare for symptom checking and care navigation. However, concerns about accuracy limit large language model (LLM) use in clinical settings.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This convergent mixed-methods study evaluated three enhancements to an existing rules-based triage chatbot. The purpose is to inform the development of tools improving patient access and system navigation.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Three artificial intelligence tools (Dialogflow, ChatGPT, and Gemini) were integrated into a rules-based workflow from December 21st 2024 to July 7th 2025. Thirteen testers completed 156 vignette-based interactions and 52 Likert-scale evaluations from July 17th 2025 to August 5th 2025. Quantitative outcomes included accuracy, conversational turns, time to completion, and tester ratings. Qualitative feedback was thematically analyzed.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The rules-based chatbot was most accurate (71.8%, 95% CI 56.2\u201383.5%), outperforming Dialogflow (30.8%, p = 0.002) and Gemini (41.0%, p = 0.031). Dialogflow, ChatGPT, and Gemini were statistically indistinguishable. Qualitative analysis revealed user preference for ChatGPT and Gemini despite equal or worse performance than the rules-based chatbot.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Rules-based triage remained most reliable, and the divergence between user preference and measured accuracy underscores that safe clinical deployment of LLM-based chatbots requires governance addressing accuracy, interpretability, and masking of poorer performance by conversational fluency.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/14604582261485150","type":"journal-article","created":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T21:30:20Z","timestamp":1789162220000},"update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":0,"title":["Comparative analysis of AI enhancements to rules-based clinical triage chatbot: Convergent mixed-methods study"],"prefix":"10.1177","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-1719-283X","authenticated-orcid":false,"given":"Aleksa","family":"Stanki\u0107","sequence":"first","affiliation":[{"name":"Union University School of Computing, Department of Computer Engineering"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Du\u0161an","family":"Vujo\u0161evi\u0107","sequence":"additional","affiliation":[{"name":"Union University School of Computing, Department of Computer Engineering"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nemanja","family":"Radosavljevi\u0107","sequence":"additional","affiliation":[{"name":"Union University School of Computing, Department of Computer Engineering"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adrijana","family":"Jeremi\u0107","sequence":"additional","affiliation":[{"name":"Union University School of Computing, Department of Computer Engineering"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2026,9,11]]},"reference":[{"key":"e_1_3_9_2_2","doi-asserted-by":"crossref","first-page":"e002236","DOI":"10.1136\/fmch-2023-002236","article-title":"Biopsy of Canada\u2019s family physician shortage","volume":"11","author":"Li K","year":"2023","unstructured":"Li K, Frumkin A, Bi WG, et al. Biopsy of Canada\u2019s family physician shortage. Fam Med Community Health 2023; 11: e002236. https:\/\/doi.org\/10.1136\/fmch-2023-002236","journal-title":"Fam Med Community Health"},{"key":"e_1_3_9_3_2","doi-asserted-by":"publisher","DOI":"10.1503\/cmaj.1096014"},{"key":"e_1_3_9_4_2","first-page":"786","article-title":"Emergency Department Overcrowding: An Environmental Scan of Contributing Factors and a Summary of Systematic Review Evidence on Interventions","volume":"3","author":"Haas R","year":"2023","unstructured":"Haas R, Brundisini F, Barbara A, et al. Emergency Department Overcrowding: An Environmental Scan of Contributing Factors and a Summary of Systematic Review Evidence on Interventions. Can J Health Technol 2023; 3: 786. https:\/\/doi.org\/10.51731\/cjht.2023.786","journal-title":"Can J Health Technol"},{"key":"e_1_3_9_5_2","volume-title":"Emergency department crowding: Beyond primary care access","author":"Canadian Institute for Health Information","year":"2024","unstructured":"Canadian Institute for Health Information. Emergency department crowding: Beyond primary care access. CIHI. https:\/\/www.cihi.ca\/en\/primary-and-virtual-care-access-emergency-department-visits-for-primary-care-conditions\/emergency-department-crowding-beyond-primary-care-access (2024, accessed 25 January 2025)."},{"key":"e_1_3_9_6_2","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.d2983"},{"key":"e_1_3_9_7_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.sapharm.2018.12.003"},{"key":"e_1_3_9_8_2","volume-title":"Correlates of Preventable Emergency Department Visits in Canada: Evidence from the Literature and the Canadian Community Health Survey","author":"Lau T","year":"2020","unstructured":"Lau T. Correlates of Preventable Emergency Department Visits in Canada: Evidence from the Literature and the Canadian Community Health Survey. Western University, 2020."},{"key":"e_1_3_9_9_2","doi-asserted-by":"publisher","DOI":"10.1017\/S1463423624000677"},{"key":"e_1_3_9_10_2","doi-asserted-by":"publisher","DOI":"10.1186\/1476-072X-11-28"},{"key":"e_1_3_9_11_2","doi-asserted-by":"publisher","DOI":"10.1080\/17434440.2021.2013200"},{"key":"e_1_3_9_12_2","doi-asserted-by":"publisher","DOI":"10.2196\/46448"},{"key":"e_1_3_9_13_2","doi-asserted-by":"publisher","DOI":"10.1136\/bmjqs-2024-017476"},{"key":"e_1_3_9_14_2","article-title":"Chatbots in Health Care: Connecting Patients to Information","volume":"4","author":"Clark M","unstructured":"Clark M, Bailey S. Chatbots in Health Care: Connecting Patients to Information. Can J Health Technol 4. Epub ahead of print 22 January 2024. https:\/\/doi.org\/10.51731\/cjht.2024.818","journal-title":"Can J Health Technol"},{"key":"e_1_3_9_15_2","volume-title":"The User Experience of Chatbots","author":"Budiu R","unstructured":"Budiu R. The User Experience of Chatbots. Nielsen Norman Group. https:\/\/www.nngroup.com\/articles\/chatbots\/ (accessed 3 August 2025)."},{"key":"e_1_3_9_16_2","volume-title":"Top Chatbot UX Tips and Best Practices for 2025","author":"Rafalski K","unstructured":"Rafalski K. Top Chatbot UX Tips and Best Practices for 2025. Netguru S.A. https:\/\/www.netguru.com\/blog\/chatbot-ux-tips (accessed 3 August 2025)."},{"key":"e_1_3_9_17_2","volume-title":"Principles of bot design","author":"Connolly E","year":"2016","unstructured":"Connolly E. Principles of bot design. The Intercom Blog, 2016. https:\/\/www.intercom.com\/blog\/principles-bot-design\/ (accessed 3 August 2025)."},{"key":"e_1_3_9_18_2","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocy072"},{"key":"e_1_3_9_19_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.57879"},{"key":"e_1_3_9_20_2","doi-asserted-by":"crossref","unstructured":"Shekar S Pataranutaporn P Sarabu C et al. People over trust AI-generated medical responses and view them to be as valid as doctors despite low accuracy. 2024. Epub ahead of print 11 August 2024. https:\/\/doi.org\/10.48550\/arXiv.2408.15266","DOI":"10.1056\/AIoa2300015"},{"key":"e_1_3_9_21_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-026-02409-8"},{"key":"e_1_3_9_22_2","doi-asserted-by":"publisher","DOI":"10.2196\/53297"},{"key":"e_1_3_9_23_2","doi-asserted-by":"publisher","DOI":"10.1177\/20552076241227132"},{"key":"e_1_3_9_24_2","doi-asserted-by":"publisher","DOI":"10.2196\/19928"},{"key":"e_1_3_9_25_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01367-3"},{"key":"e_1_3_9_26_2","doi-asserted-by":"publisher","DOI":"10.1177\/1460458214534092"},{"key":"e_1_3_9_27_2","doi-asserted-by":"publisher","DOI":"10.7326\/0003-4819-155-2-201107190-00005"},{"key":"e_1_3_9_28_2","doi-asserted-by":"publisher","DOI":"10.1177\/1715163515608399"},{"key":"e_1_3_9_29_2","first-page":"4300","article-title":"How Interaction Mechanism and Error Responses Influence Users\u2019 Responses to Customer Service Chatbots","volume":"41","author":"Klein SH","year":"2025","unstructured":"Klein SH, Papies D, Utz S. How Interaction Mechanism and Error Responses Influence Users\u2019 Responses to Customer Service Chatbots. Int J Human\u2013Computer Interact 2025; 41: 4300\u20134318.","journal-title":"Int J Human\u2013Computer Interact"},{"key":"e_1_3_9_30_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jacr.2025.02.051"},{"key":"e_1_3_9_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/1735223.1735255"},{"key":"e_1_3_9_32_2","doi-asserted-by":"publisher","DOI":"10.3758\/BF03195514"},{"key":"e_1_3_9_33_2","volume-title":"What are order effects in experiments? Definition and examples","author":"The Statsig Team","unstructured":"The Statsig Team. What are order effects in experiments? Definition and examples. Statsig, LLC. https:\/\/www.statsig.com\/perspectives\/ordereffectsinexperiments (accessed 18 August 2025)."},{"key":"e_1_3_9_34_2","unstructured":"Brooke J. SUS - A quick and dirty usability scale. In: Usability evaluation in industry. Taylor & Francis."},{"key":"e_1_3_9_35_2","doi-asserted-by":"publisher","DOI":"10.2196\/69006"},{"key":"e_1_3_9_36_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2006.03.012"}],"container-title":["Health Informatics Journal"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/14604582261485150","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/14604582261485150","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/14604582261485150","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T06:17:48Z","timestamp":1789366668000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/14604582261485150"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7]]},"references-count":35,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,7]]}},"alternative-id":["10.1177\/14604582261485150"],"URL":"https:\/\/doi.org\/10.1177\/14604582261485150","relation":{},"ISSN":["1460-4582","1741-2811"],"issn-type":[{"value":"1460-4582","type":"print"},{"value":"1741-2811","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7]]},"article-number":"14604582261485150"}}