{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:43:50Z","timestamp":1785545030119,"version":"3.56.0"},"reference-count":52,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Hypospadias is a common congenital malformation requiring surgery. Caregivers face substantial perioperative information needs, and large language models (LLMs) offer a potential health education channel, but their performance in pediatric urology and the relation between citation accuracy and clinical content safety lack systematic evaluation.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to evaluate 5 LLMs (ChatGPT-4o, Gemini-2.5-Pro, OpenEvidence, Zhipu Qingyan, and DeepSeek) for pediatric hypospadias perioperative consultation, and to characterize the dimensions clinicians and caregivers prioritize.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n A noninterventional cross-sectional study was conducted at a tertiary hospital in April 2025. From a 40-item bank, 10 high-priority questions were selected by an independent caregiver screening cohort (N=34, cohort A) and classified into 3 risk levels. Twenty-three pediatric urology experts (6 dimensions) and 36 primary caregivers (cohort B; 4 dimensions) evaluated responses by double-blind forced-ranking (reverse-scored, 5=best). Friedman tests with Kendall\n W<\/jats:italic>\n assessed overall differences; paired Wilcoxon tests with Bonferroni correction (adjusted \u03b1=.005) and rank-biserial\n r<\/jats:italic>\n with Hodges-Lehmann 95% CIs were used post hoc. Reference authenticity was independently verified by 2 reviewers (XH and WH) using a 5-category scheme (V\/PV\/F\/G\/NR [V: Verifiable, PV: Partially Verifiable, F: Fabricated, G: Guideline-Based, Nonspecific, and NR: No References]), with consensus after canonical-source reverification (Cohen \u03ba=0.702 preadjudication). An 8-reviewer clinical safety audit (7 senior specialists plus 1 European Association of Urology [EAU]-anchored intermediate-title clinician) applied a 4-level severity scheme (None\/Mild\/Moderate\/Severe).\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Models differed significantly (caregiver:\n \u03c7<\/jats:italic>\n \u00b2\n 4<\/jats:sub>\n =77.5,\n W<\/jats:italic>\n =0.538,\n P<\/jats:italic>\n <.001; expert:\n \u03c7<\/jats:italic>\n \u00b2\n 4<\/jats:sub>\n =62.2,\n W<\/jats:italic>\n =0.676,\n P<\/jats:italic>\n <.001). Gemini-2.5-Pro ranked first (expert median 5.0, IQR 3.0\u20105.0; caregiver 4.0, IQR 3.0\u20105.0). DeepSeek ranked second (4.0 both), with superior Empathy versus ChatGPT-4o (\n r<\/jats:italic>\n =\u22120.343;\n P<\/jats:italic>\n <.001). OpenEvidence scored lowest (2.0 both), despite high citation accuracy. Expert-caregiver agreement was strong (Spearman \u03c1=0.89;\n P<\/jats:italic>\n =.04). Citation accuracy diverged sharply: OpenEvidence was fully verifiable (\n V<\/jats:italic>\n =100%,\n F<\/jats:italic>\n =0%), whereas DeepSeek and Zhipu Qingyan showed the highest fabrication (\n F<\/jats:italic>\n of 33% and 24%, respectively); Gemini-2.5-Pro fabricated none but used nonspecific guideline citations (\n G<\/jats:italic>\n =85%). The safety audit yielded 78 flags, including 9 Severe-level flags across 5 question\u2013model combinations; OpenEvidence carried the largest Severe burden (5 of 9) and the highest severity-weighted score, whereas Gemini-2.5-Pro had the lowest. Bibliographic accuracy and clinical safety were dissociable, and the ranking held under poststratification weighting.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n High citation accuracy does not guarantee clinical safety. In the first dual-perspective evaluation, no model was uniformly best. Gemini-2.5-Pro was most comprehensive but relied on nonspecific guidelines. DeepSeek scored highest on caregiver-rated Empathy, yet it had the highest fabrication rate. OpenEvidence produced the most verifiable citations but carried the heaviest Severe-flag burden. These dimension-level priorities, the dissociation between citation quality and safety, and the portable evaluation framework can inform future pediatric medical\u2013artificial intelligence (AI) development. For perioperative use, AI should follow a tiered human-machine collaboration model with mandatory clinician oversight in high-risk scenarios.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/93393","type":"journal-article","created":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T06:25:07Z","timestamp":1782973507000},"page":"e93393-e93393","source":"Crossref","is-referenced-by-count":0,"title":["Performance of 5 Large Language Models in Perioperative Consultation for Pediatric Hypospadias: Cross-Sectional Comparative Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-0198-4194","authenticated-orcid":false,"given":"Ting","family":"Kang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-3003-7409","authenticated-orcid":false,"given":"Chi","family":"Yuan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-4311-4898","authenticated-orcid":false,"given":"Xinyu","family":"Hu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4554-5605","authenticated-orcid":false,"given":"Wenjiao","family":"Huang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,7,29]]},"reference":[{"issue":"3","key":"R1","doi-asserted-by":"publisher","first-page":"152","DOI":"10.1016\/j.jpurol.2015.12.002","article-title":"Worldwide prevalence of hypospadias","volume":"12","author":"Springer","journal-title":"J Pediatr Urol"},{"issue":"5","key":"R2","doi-asserted-by":"publisher","DOI":"10.3892\/etm.2023.11910","article-title":"Current perspectives in hypospadias research: a scoping review of articles published in 2021 (Review)","volume":"25","author":"Gozar","journal-title":"Exp Ther Med"},{"issue":"5","key":"R3","doi-asserted-by":"publisher","first-page":"504","DOI":"10.1016\/j.jpurol.2011.02.008","article-title":"Functional, cosmetic and psychosexual results in adult men who underwent hypospadias correction in childhood","volume":"7","author":"Rynja","journal-title":"J Pediatr Urol"},{"issue":"8","key":"R4","doi-asserted-by":"publisher","DOI":"10.2196\/17367","article-title":"The acceptability and impact of the Xploro digital therapeutic platform to inform and prepare children for planned procedures in a hospital: before and after evaluation study","volume":"22","author":"Bray","journal-title":"J Med Internet Res"},{"issue":"2","key":"R5","doi-asserted-by":"publisher","first-page":"242","DOI":"10.3760\/cma.j.issn.1673-4351.2020.02.016","article-title":"Preoperative psychological status of children with hypospadias and their families [in Chinese]","volume":"39","author":"Wen","journal-title":"Int J Nurs"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.2196\/41156","article-title":"When people with chronic conditions turn to peers on social media to obtain and share information: systematic review of the implications for relationships with health care professionals","volume":"25","author":"Kj\u00e6rulff","journal-title":"J Med Internet Res"},{"issue":"9","key":"R7","doi-asserted-by":"publisher","first-page":"5047","DOI":"10.1007\/s10115-024-02120-8","article-title":"An analysis of large language models: their impact and potential applications","volume":"66","author":"Bharathi Mohan","journal-title":"Knowl Inf Syst"},{"key":"R8","unstructured":"Nori H King N McKinney SM Carignan D Horvitz E . Capabilities of GPT-4 on medical challenge problems. arXiv. Preprint posted online on Mar 20, 2023. [doi: 10.48550\/arXiv.2303.13375]"},{"issue":"7","key":"R9","doi-asserted-by":"publisher","first-page":"956","DOI":"10.1001\/jamaoncol.2024.0836","article-title":"Physician and artificial intelligence chatbot responses to cancer questions from social media","volume":"10","author":"Chen","journal-title":"JAMA Oncol"},{"issue":"8","key":"R10","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.30320","article-title":"Comparison of ophthalmologist and large language model chatbot responses to online patient eye care questions","volume":"6","author":"Bernstein","journal-title":"JAMA Netw Open"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.2196\/60291","article-title":"Accuracy of prospective assessments of 4 large language model chatbot responses to patient questions about emergency care: experimental comparative study","volume":"26","author":"Yau","journal-title":"J Med Internet Res"},{"issue":"4","key":"R12","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.37432","article-title":"Exploring the boundaries of reality: investigating the phenomenon of artificial intelligence hallucination in scientific writing through ChatGPT references","volume":"15","author":"Athaluri","journal-title":"Cureus"},{"issue":"2","key":"R13","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.35179","article-title":"Artificial hallucinations in ChatGPT: implications in scientific writing","volume":"15","author":"Alkaissi","journal-title":"Cureus"},{"issue":"2","key":"R14","doi-asserted-by":"publisher","first-page":"106","DOI":"10.1093\/jbmr\/zjad007","article-title":"The performance of artificial intelligence chatbot large language models to address skeletal biology and bone health queries","volume":"39","author":"Cung","journal-title":"J Bone Miner Res"},{"issue":"1","key":"R15","doi-asserted-by":"publisher","DOI":"10.1007\/s00345-025-06104-3","article-title":"Artificial intelligence chatbots vs. YPUC pediatric urologists: performance on a Campbell Walsh urology hypospadiology questionnaire","volume":"43","author":"Faraj","journal-title":"World J Urol"},{"issue":"2","key":"R16","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.79221","article-title":"DeepSeek in healthcare: revealing opportunities and steering challenges of a new open-source artificial Intelligence frontier","volume":"17","author":"Temsah","journal-title":"Cureus"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","DOI":"10.1002\/cam4.70554","article-title":"Can large language models aid caregivers of pediatric cancer patients in information seeking? A cross-sectional investigation","volume":"14","author":"Sezgin","journal-title":"Cancer Med"},{"key":"R18","unstructured":"Statistical communiqu\u00e9 of the People\u2019s Republic of China on the 2024 national economic and social development [in Chinese]. National Bureau of Statistics of China. 2025. URL: https:\/\/www.stats.gov.cn\/xxgk\/sjfb\/zxfb2020\/202502\/t20250228_1958817.html [Accessed 01-05-2025]"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-025-04274-w","article-title":"Reporting guideline for Chatbot Health Advice studies: the CHART statement","volume":"23","author":"Huo","journal-title":"BMC Med"},{"key":"R20","unstructured":"2024-2025 China AI large model market status and development trend research report. iiMedia Research Group. 2024. URL: https:\/\/www.iimedia.cn\/c400\/104152.html [Accessed 20-03-2025]"},{"key":"R21","unstructured":"Maslej N Fattorini L Perrault R et al. The AI Index 2024 Annual Report. Stanford HAI; 2024. URL: https:\/\/aiindex.stanford.edu\/report [Accessed 20-03-2025]"},{"key":"R22","unstructured":"EAU guidelines on paediatric urology. European Association of Urology. 2025. URL: https:\/\/uroweb.org\/guidelines\/paediatric-urology [Accessed 10-05-2026]"},{"issue":"10015","key":"R23","doi-asserted-by":"publisher","first-page":"239","DOI":"10.1016\/S0140-6736(15)00608-X","article-title":"Neurodevelopmental outcome at 2 years of age after general anaesthesia and awake-regional anaesthesia in infancy (GAS): an international multicentre, randomised controlled trial","volume":"387","author":"Davidson","journal-title":"Lancet"},{"issue":"21","key":"R24","doi-asserted-by":"publisher","first-page":"2312","DOI":"10.1001\/jama.2016.6967","article-title":"Association between a single general anesthesia exposure before age 36 months and neurocognitive outcomes in later childhood","volume":"315","author":"Sun","journal-title":"JAMA"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1097\/ALN.0000000000002232","article-title":"Neuropsychological and behavioral outcomes after exposure of young children to procedures requiring general anesthesia","volume":"129","author":"Warner","journal-title":"Anesthesiology"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.1227\/neu.0000000000002856","article-title":"Usefulness and accuracy of artificial intelligence chatbot responses to patient questions for neurosurgical procedures","author":"Gajjar","journal-title":"Neurosurgery"},{"issue":"4","key":"R27","doi-asserted-by":"publisher","first-page":"1941","DOI":"10.1097\/JS9.0000000000001066","article-title":"Evaluation of large language models in breast cancer clinical scenarios: a comparative analysis based on ChatGPT-3.5, ChatGPT-4.0, and Claude2","volume":"110","author":"Deng","journal-title":"Int J Surg"},{"issue":"3","key":"R28","doi-asserted-by":"publisher","first-page":"490","DOI":"10.1016\/j.pec.2017.09.003","article-title":"Interrater reliability of the Patient Education Materials Assessment Tool (PEMAT)","volume":"101","author":"Vishnevetsky","journal-title":"Patient Educ Couns"},{"issue":"3","key":"R29","doi-asserted-by":"publisher","first-page":"395","DOI":"10.1016\/j.pec.2014.05.027","article-title":"Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information","volume":"96","author":"Shoemaker","journal-title":"Patient Educ Couns"},{"issue":"5","key":"R30","doi-asserted-by":"publisher","first-page":"527","DOI":"10.4103\/aja.aja_11_21","article-title":"Analysis of the social and clinical factors affecting the age of children when receiving surgery for hypospadias: a retrospective study of 1611 cases in a single center","volume":"23","author":"Zhang","journal-title":"Asian J Androl"},{"issue":"6","key":"R31","doi-asserted-by":"publisher","DOI":"10.1111\/cch.70173","article-title":"The role of sociodemographic factors in family resilience of Chinese children with chronic illnesses: a cross-sectional study","volume":"51","author":"Fang","journal-title":"Child Care Health Dev"},{"key":"R32","unstructured":"Cohen J . Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Lawrence Erlbaum Associates; 1988. ISBN: 0-8058-0283-5"},{"issue":"1","key":"R33","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3109\/17538157.2014.948171","article-title":"Navigating the digital divide: a systematic review of eHealth literacy in underserved populations in the United States","volume":"41","author":"Chesser","journal-title":"Inform Health Soc Care"},{"issue":"10","key":"R34","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.6349","article-title":"Health literacy and health information technology adoption: the potential for a new digital divide","volume":"18","author":"Mackert","journal-title":"J Med Internet Res"},{"issue":"9","key":"R35","doi-asserted-by":"publisher","first-page":"1681","DOI":"10.1093\/humrep\/deaf135","article-title":"Artificial intelligence-simplified information to advance reproductive genetic literacy and health equity","volume":"40","author":"Naghdi","journal-title":"Hum Reprod"},{"issue":"2","key":"R36","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.8.2.e9","article-title":"eHealth literacy: essential skills for consumer health in a networked world","volume":"8","author":"Norman","journal-title":"J Med Internet Res"},{"key":"R37","unstructured":"Masoud RI Liu Z Ferianc M Treleaven P Rodrigues M . Cultural alignment in large language models: an explanatory analysis based on Hofstede\u2019s cultural dimensions. Presented at: Proceedings of the 31st International Conference on Computational Linguistics (COLING); Jan 19-24, 2025:8474-8503; Abu Dhabi, UAE. URL: https:\/\/aclanthology.org\/2025.coling-main.567\/ [Accessed 14-07-2026]"},{"issue":"1","key":"R38","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00873-0","article-title":"The imperative for regulatory oversight of large language models (or generative AI) in healthcare","volume":"6","author":"Mesk\u00f3","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R39","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1038\/s41591-018-0300-7","article-title":"High-performance medicine: the convergence of human and artificial intelligence","volume":"25","author":"Topol","journal-title":"Nat Med"},{"issue":"13","key":"R40","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1056\/NEJMsr2214184","article-title":"Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine","volume":"388","author":"Lee","journal-title":"N Engl J Med"},{"issue":"1","key":"R41","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1055\/s-0039-1677901","article-title":"Artificial intelligence in primary health care: perceptions, issues, and challenges","volume":"28","author":"Liyanage","journal-title":"Yearb Med Inform"},{"issue":"2","key":"R42","doi-asserted-by":"publisher","DOI":"10.14503\/THIJ-21-7572","article-title":"Artificial intelligence for education, proctoring, and credentialing in cardiovascular medicine","volume":"49","author":"Krajcer","journal-title":"Tex Heart Inst J"},{"issue":"1","key":"R43","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-023-02021-3","article-title":"Evaluating the performance of different large language models on health consultation and patient education in urolithiasis","volume":"47","author":"Song","journal-title":"J Med Syst"},{"issue":"6","key":"R44","doi-asserted-by":"publisher","first-page":"788","DOI":"10.1097\/ICO.0000000000003747","article-title":"Use of online large language model chatbots in cornea clinics","volume":"44","author":"Nichani","journal-title":"Cornea"},{"issue":"4","key":"R45","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1037\/1082-989X.9.4.453","article-title":"Comparative judgments as an alternative to ratings: identifying the scale origin","volume":"9","author":"B\u00f6ckenholt","journal-title":"Psychol Methods"},{"issue":"3","key":"R46","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1002\/acp.2350050305","article-title":"Response strategies for coping with the cognitive demands of attitude measures in surveys","volume":"5","author":"Krosnick","journal-title":"Appl Cogn Psychol"},{"key":"R47","doi-asserted-by":"crossref","unstructured":"Spielberger CD Gorsuch RL Lushene R Vagg PR Jacobs GA . Manual for the State-Trait Anxiety Inventory. Consulting Psychologists Press; 1983.","DOI":"10.1037\/t06496-000"},{"issue":"2","key":"R48","doi-asserted-by":"publisher","DOI":"10.1162\/99608f92.5317da47","article-title":"How Is ChatGPT\u2019s behavior changing over time?","volume":"6","author":"Chen","journal-title":"Harvard Data Sci Rev"},{"issue":"5","key":"R49","doi-asserted-by":"publisher","first-page":"924","DOI":"10.1038\/s41591-022-01772-9","article-title":"Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI","volume":"28","author":"Vasey","journal-title":"Nat Med"},{"key":"R50","unstructured":"Ethics and governance of artificial intelligence for health. World Health Organization. 2021. URL: https:\/\/www.who.int\/publications\/i\/item\/9789240029200 [Accessed 30-06-2026]"},{"key":"R51","unstructured":"Brown TB Mann B Ryder N et al. Language models are few-shot learners. Presented at: Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS); Dec 6-12, 2020:1877-1901; Vancouver BC Canada. 2020.URL: https:\/\/arxiv.org\/abs\/2005.14165 [Accessed 15-07-2026]"},{"key":"R52","unstructured":"Lewis P Perez E Piktus A et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. Presented at: Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS); Dec 6-12, 2020:9459-9474; Vancouver BC Canada. URL: https:\/\/arxiv.org\/abs\/2005.11401 [Accessed 15-07-2026]"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2026\/1\/e93393","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:23:43Z","timestamp":1785536623000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e93393"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,29]]},"references-count":52,"URL":"https:\/\/doi.org\/10.2196\/93393","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,29]]},"article-number":"v28i3e93393"}}