{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,21]],"date-time":"2026-09-21T10:02:39Z","timestamp":1789984959564,"version":"4.0.1"},"reference-count":48,"publisher":"Wiley","issue":"6","license":[{"start":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T00:00:00Z","timestamp":1781049600000},"content-version":"vor","delay-in-days":9,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T00:00:00Z","timestamp":1780272000000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"}],"funder":[{"DOI":"10.13039\/501100005117","name":"Birjand University of Medical Sciences","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100005117","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Health Science Reports"],"published-print":{"date-parts":[[2026,6]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background and Aims<\/jats:title>\n                    <jats:p>The use of Large Language Model (LLM)\u2010based chatbots across various fields has yielded positive outcomes. Understanding the health service delivery system offers numerous benefits. This study aimed to analyze the performance of LLMs in answering healthcare delivery questions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>A validated questionnaire relevant to the research context was administered to a sample of LLM\u2010based chatbots. The chatbots evaluated in this study included GPT\u20104.1\u2010mini, Gemini 2.5, Copilot 2025, and Perplexity. A written prompt was provided to facilitate response generation by the chatbots. To analyze and compare the performance of the AI models in addressing the research questions, confusion matrices were constructed, and key metrics\u2014sensitivity, specificity, positive predictive value, negative predictive value, and overall accuracy\u2014were calculated.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The initial assessment of the chatbots showed perfect sensitivity (1.00), accurately identifying all true positives without false negatives. Specificity varied, with ChatGPT and Perplexity at 0.50, Gemini at 0.43, and Copilot at 0.33. Positive predictive values (PPV) ranged from 0.67 (Gemini) to 0.75 (ChatGPT and Perplexity), while negative predictive values (NPV) were uniformly perfect (1.00). Overall accuracy was highest for ChatGPT and Perplexity (0.80), with Gemini and Copilot at 0.73. In the second round, sensitivity remained perfect for all chatbots. Gemini achieved the highest specificity (0.80), followed by ChatGPT (0.67), Perplexity (0.60), and Copilot (0.50). PPVs improved, ranging from 0.75 (Copilot) to 0.91 (Gemini). NPVs remained perfect (1.00) across all models. Overall accuracy led by Gemini (0.93), with ChatGPT and Perplexity both at 0.87, and Copilot at 0.80.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>ChatGPT and Perplexity showed the highest initial performance, while the second round revealed improvements in most chatbots, especially in specificity and accuracy, with Gemini performing best. Further research is needed for deeper insights.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1002\/hsr2.72634","type":"journal-article","created":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T07:49:03Z","timestamp":1781077743000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Performance of Large Language Models in Answering Healthcare Delivery Questions: A Quantitative Cross\u2010Sectional Study"],"prefix":"10.1002","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0576-7660","authenticated-orcid":false,"given":"Mohsen","family":"Khosravi","sequence":"first","affiliation":[{"name":"Social Determinants of Health Research Center Birjand University of Medical Sciences Birjand Iran"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zahra","family":"Zamaninasab","sequence":"additional","affiliation":[{"name":"Department of Epidemiology and Biostatistics, School of Health, Social Determinants of Health Research Center Birjand University of Medical Sciences Birjand Iran"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fatemeh","family":"Khosravi","sequence":"additional","affiliation":[{"name":"School of Medicine Guilan University of Medical Sciences Rasht Iran"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Melika","family":"Attar","sequence":"additional","affiliation":[{"name":"School of Rehabilitation Sciences Babol University of Medical Sciences Babol Iran"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7223-6707","authenticated-orcid":false,"given":"Morteza","family":"Arab\u2010Zozani","sequence":"additional","affiliation":[{"name":"Social Determinants of Health Research Center Birjand University of Medical Sciences Birjand Iran"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2026,6,10]]},"reference":[{"key":"e_1_2_13_2_1","doi-asserted-by":"publisher","DOI":"10.4103\/jfmpc.jfmpc_440_19"},{"key":"e_1_2_13_3_1","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2023.1237704"},{"key":"e_1_2_13_4_1","first-page":"373","article-title":"An Overview of Chatbot Technology","volume":"584","author":"Adamopoulou E.","year":"2020","journal-title":"Artificial Intelligence Applications and Innovations"},{"key":"e_1_2_13_5_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.57879"},{"key":"e_1_2_13_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.bjao.2024.100280"},{"key":"e_1_2_13_7_1","doi-asserted-by":"publisher","DOI":"10.1093\/clinchem\/hvad106"},{"key":"e_1_2_13_8_1","doi-asserted-by":"publisher","DOI":"10.3390\/biomedinformatics4010047"},{"key":"e_1_2_13_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tacc.2021.02.007"},{"key":"e_1_2_13_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.rcsop.2024.100419"},{"key":"e_1_2_13_11_1","doi-asserted-by":"publisher","DOI":"10.3390\/info16070549"},{"key":"e_1_2_13_12_1","doi-asserted-by":"publisher","DOI":"10.2196\/15787"},{"key":"e_1_2_13_13_1","doi-asserted-by":"publisher","DOI":"10.3390\/encyclopedia1010021"},{"key":"e_1_2_13_14_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2023.1838"},{"key":"e_1_2_13_15_1","first-page":"337","article-title":"A Comparative Analysis of Responses of Artificial Intelligence Chatbots in Special Needs Dentistry","volume":"46","author":"Rokhshad R.","year":"2024","journal-title":"Pediatric Dentistry"},{"key":"e_1_2_13_16_1","doi-asserted-by":"publisher","DOI":"10.1002\/jrsm.1689"},{"key":"e_1_2_13_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00345-024-05326-1"},{"key":"e_1_2_13_18_1","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare12050534"},{"key":"e_1_2_13_19_1","doi-asserted-by":"publisher","DOI":"10.6004\/jnccn.2023.7334"},{"key":"e_1_2_13_20_1","doi-asserted-by":"publisher","DOI":"10.2196\/40789"},{"key":"e_1_2_13_21_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamaoncol.2024.0836"},{"key":"e_1_2_13_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.oooo.2024.01.015"},{"key":"e_1_2_13_23_1","volume-title":"Building a Better Delivery System: A New Engineering\/Health Care Partnership","author":"Fanjiang G.","year":"2005"},{"key":"e_1_2_13_24_1","volume-title":"Health System Performance Assessment: A Framework For Policy Analysis","author":"Papanicolas I.","year":"2022"},{"key":"e_1_2_13_25_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2020-037667"},{"key":"e_1_2_13_26_1","doi-asserted-by":"publisher","DOI":"10.7812\/TPP\/08-100"},{"key":"e_1_2_13_27_1","doi-asserted-by":"publisher","DOI":"10.2196\/84120"},{"key":"e_1_2_13_28_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13244-026-02236-1"},{"key":"e_1_2_13_29_1","doi-asserted-by":"publisher","DOI":"10.7326\/0003-4819-147-8-200710160-00010"},{"key":"e_1_2_13_30_1","unstructured":"Study.com \u201cHealth 307: Healthcare Delivery Systems Final Exam [Online] \u201d published 2025 https:\/\/study.com\/academy\/exam\/course\/health-307-healthcare-delivery-systems.html."},{"key":"e_1_2_13_31_1","doi-asserted-by":"publisher","DOI":"10.4103\/0301\u20104738.37595"},{"key":"e_1_2_13_32_1","doi-asserted-by":"publisher","DOI":"10.2147\/DHPS.S425858"},{"key":"e_1_2_13_33_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-52415-1"},{"key":"e_1_2_13_34_1","doi-asserted-by":"publisher","DOI":"10.3389\/fdsfr.2024.1379260"},{"key":"e_1_2_13_35_1","doi-asserted-by":"publisher","DOI":"10.3389\/fmed.2025.1495378"},{"key":"e_1_2_13_36_1","doi-asserted-by":"publisher","DOI":"10.12669\/pjms.41.4.11178"},{"key":"e_1_2_13_37_1","doi-asserted-by":"publisher","DOI":"10.3390\/medicina60060957"},{"key":"e_1_2_13_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10143-025-03472-7"},{"key":"e_1_2_13_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00467-024-06486-3"},{"key":"e_1_2_13_40_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-06291-2"},{"key":"e_1_2_13_41_1","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocad259"},{"key":"e_1_2_13_42_1","first-page":"429","article-title":"Utilizing Large Language Models to Generate Synthetic Data to Increase the Performance of BERT\u2010Based Neural Networks","volume":"2024","author":"Woolsey C. R.","year":"2024","journal-title":"AMIA Joint Summits on Translational Science Proceedings"},{"key":"e_1_2_13_43_1","doi-asserted-by":"publisher","DOI":"10.1080\/10872981.2025.2534065"},{"key":"e_1_2_13_44_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2024.101030"},{"key":"e_1_2_13_45_1","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230582"},{"key":"e_1_2_13_46_1","doi-asserted-by":"publisher","DOI":"10.3390\/biomedinformatics4010047"},{"key":"e_1_2_13_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2025.106019"},{"key":"e_1_2_13_48_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10389-024-02219-w"},{"key":"e_1_2_13_49_1","doi-asserted-by":"publisher","DOI":"10.2196\/64406"}],"container-title":["Health Science Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/hsr2.72634","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/hsr2.72634","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/hsr2.72634","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,21]],"date-time":"2026-09-21T09:21:33Z","timestamp":1789982493000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/hsr2.72634"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6]]},"references-count":48,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2026,6]]}},"alternative-id":["10.1002\/hsr2.72634"],"URL":"https:\/\/doi.org\/10.1002\/hsr2.72634","archive":["Portico"],"relation":{},"ISSN":["2398-8835","2398-8835"],"issn-type":[{"value":"2398-8835","type":"print"},{"value":"2398-8835","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6]]},"assertion":[{"value":"2026-01-27","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-06-01","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-06-10","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e72634"}}