{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T20:56:17Z","timestamp":1786136177949,"version":"build-2736575974"},"reference-count":43,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec sec-type=\"background\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Systematic collection of social determinants of health (SDoH) data remains inconsistent across health care settings, despite its critical impact on patient outcomes. Large language model\u2013powered chatbots offer promise for scalable SDoH data collection, but rigorous, feasible evaluation methods for patient-facing applications are lacking.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"objective\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study aimed to describe an efficient, iterative, multidisciplinary approach for developing and evaluating a patient-facing SDoH chatbot using synthetic data and case simulation, with the goal of optimizing both chatbot performance and the evaluation rubric prior to clinical deployment.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"methods\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>A 10-criterion evaluation rubric was adapted from established health care AI frameworks and applied to 27 synthetic clinical scenarios representing diverse SDoH profiles. Scenarios were role-played by a licensed clinical social worker, and chatbot-patient interactions were rated by 3 members of the research team that were multidisciplinary experts: a social worker, a nurse practitioner, and a physician. Quantitative analysis used percent agreement and Fleiss \u03ba to characterize chatbot performance and rater consensus, with percent agreement selected due to the high prevalence of ceiling effects in several domains. Qualitative analysis synthesized rater feedback to guide iterative refinement of both chatbot prompts and rubric domains.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"results\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Across 27 simulated cases, the chatbot received high proportions of positive ratings for accurate interpretation (agreement=0.98%, 95% CI 0.91\u20100.99), communication quality, and cultural sensitivity (agreement=0.99%, 95% CI 0.93\u20101.00), and appropriately adaptive questioning (agreement=0.99%, 95% CI 0.93\u20101.00). Lower performance was observed in domain focus and completeness (agreement=0.51%, 95% CI 0.40\u20100.61), completeness of data capture (agreement=0.59%, 95% CI 0.48\u20100.69; Fleiss \u03ba=0.18), and safety (agreement=0.69%, 95% CI 0.58\u20100.78; Fleiss \u03ba=\u22120.04), prompting targeted adaptations. Qualitative feedback highlighted the importance of distinguishing screening from clinical interviewing capabilities and informed the refinement of the rubric, including clarifying the definition of safety to focus on recognition of physical and mental health emergencies.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"conclusions\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>This study describes a formative feasibility approach for iterative refinement of a patient-facing SDoH chatbot and its evaluation rubric using synthetic case simulation. Future work will include independent external raters, patient stakeholders, repeated scenario testing, and prospective clinical evaluation.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/89837","type":"journal-article","created":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T10:13:45Z","timestamp":1782814425000},"page":"e89837-e89837","source":"Crossref","is-referenced-by-count":0,"title":["Iterative Multidisciplinary Development and Evaluation of a Patient-Facing Social Determinants of Health Chatbot Using Synthetic Data Simulation: Mixed Methods Study"],"prefix":"10.2196","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2829-7331","authenticated-orcid":false,"given":"Anna M","family":"Maw","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8438-5758","authenticated-orcid":false,"given":"Alexander","family":"Lupi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9989-2085","authenticated-orcid":false,"given":"Rachel","family":"Johnson-Koenke","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0741-5631","authenticated-orcid":false,"given":"Heather","family":"Coats","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3874-4833","authenticated-orcid":false,"given":"Li","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5443-6393","authenticated-orcid":false,"given":"Benjamin","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6051-2567","authenticated-orcid":false,"given":"James","family":"Mitchell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0459-6759","authenticated-orcid":false,"given":"Alexander","family":"Kotz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9686-3876","authenticated-orcid":false,"given":"Joseph","family":"Plasek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7939-7885","authenticated-orcid":false,"given":"Foster","family":"Goss","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,8,7]]},"reference":[{"issue":"2","key":"R1","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1016\/j.amepre.2015.08.024","article-title":"County health rankings: relationships between determinant factors and health outcomes","volume":"50","author":"Hood","journal-title":"Am J Prev Med"},{"key":"R2","unstructured":"Artiga S Hinton E . Beyond health care: the role of social determinants in promoting health and health equity. KFF; 2018. URL: https:\/\/www.kff.org\/racial-equity-and-health-policy\/beyond-health-care-the-role-of-social-determinants-in-promoting-health-and-health-equity\/ [Accessed 25-07-2026]"},{"issue":"7","key":"R3","doi-asserted-by":"publisher","first-page":"1196","DOI":"10.1377\/hlthaff.2012.0825","article-title":"Understanding why patients of low socioeconomic status prefer hospitals over ambulatory care","volume":"32","author":"Kangovi","journal-title":"Health Aff (Millwood)"},{"key":"R4","doi-asserted-by":"crossref","unstructured":"Institute of Medicine (IOM). Capturing Social and Behavioral Domains and Measures in Electronic Health Records: Phase 2. National Academies Press; 2014. [doi: 10.17226\/18951]","DOI":"10.17226\/18951"},{"issue":"1","key":"R5","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01074-z","article-title":"Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI","volume":"7","author":"Abbasian","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R6","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-025-02232-w","article-title":"Dissecting HealthBench: disease spectrum, clinical diversity, and data insights from multi-turn clinical AI evaluation benchmark","volume":"49","author":"Liu","journal-title":"J Med Syst"},{"issue":"3","key":"R7","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1097\/FCH.0000000000000071","article-title":"Trauma informed care in medicine: current knowledge and future research directions","volume":"38","author":"Raja","journal-title":"Fam Community Health"},{"issue":"8004","key":"R8","doi-asserted-by":"publisher","first-page":"476","DOI":"10.1038\/d41586-024-00779-1","article-title":"Chatbot AI makes racist judgements on the basis of dialect","volume":"627","author":"Gibney","journal-title":"Nature"},{"issue":"4","key":"R9","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models: a systematic review","volume":"333","author":"Bedi","journal-title":"JAMA"},{"issue":"9","key":"R10","doi-asserted-by":"publisher","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","article-title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making","volume":"30","author":"Hager","journal-title":"Nat Med"},{"key":"#cr-split#-R11.1","doi-asserted-by":"crossref","unstructured":"Papineni K Roukos S Ward T Zhu WJ . BLEU: a method for automatic evaluation of machine translation. Presented at: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Jul 6-12, 2002:Association for Computational Linguistics. 311-318","DOI":"10.3115\/1073083.1073135"},{"key":"#cr-split#-R11.2","doi-asserted-by":"crossref","unstructured":"Philadelphia, PA. 2002.[doi: 10.3115\/1073083.1073135]","DOI":"10.3115\/1073083.1073135"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.2196\/69006","article-title":"Standardizing and scaffolding health care AI-chatbot evaluation: systematic review","volume":"4","author":"Hua","journal-title":"JMIR AI"},{"issue":"8","key":"R13","doi-asserted-by":"publisher","DOI":"10.1002\/pon.70227","article-title":"Unmet social needs and trust in cancer information from health authorities: findings from the 2022 health information national trends survey","volume":"34","author":"Baeker Bispo","journal-title":"Psychooncology"},{"issue":"4","key":"R14","doi-asserted-by":"publisher","first-page":"358","DOI":"10.1093\/phr\/118.4.358","article-title":"Race and trust in the health care system","volume":"118","author":"Boulware","journal-title":"Public Health Rep"},{"issue":"4","key":"R15","doi-asserted-by":"publisher","first-page":"1126","DOI":"10.1007\/s11606-020-06568-6","article-title":"Racial, ethnic, and income-based disparities in health care-related trust","volume":"36","author":"Greene","journal-title":"J Gen Intern Med"},{"issue":"2","key":"R16","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1258\/jhsrp.2007.007074","article-title":"The quality of mixed methods studies in health services research","volume":"13","author":"O\u2019Cathain","journal-title":"J Health Serv Res Policy"},{"issue":"9","key":"R17","doi-asserted-by":"publisher","first-page":"1245","DOI":"10.1097\/ACM.0000000000000388","article-title":"Standards for reporting qualitative research: a synthesis of recommendations","volume":"89","author":"O\u2019Brien","journal-title":"Acad Med"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.1136\/bmj-2022-070904","article-title":"Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI","volume":"377","author":"Vasey","journal-title":"BMJ"},{"issue":"6","key":"R19","doi-asserted-by":"publisher","first-page":"1722","DOI":"10.1007\/s11606-025-09992-8","article-title":"Impact of centers for medicare & medicaid services screening mandate on inpatient Z-code documentation of social drivers of health","volume":"41","author":"Santiago","journal-title":"J Gen Intern Med"},{"key":"R20","unstructured":"Schulhoff S Ilie M Balepur N Kahadze K Liu A Si C et al. The prompt report: a systematic survey of prompting techniques. arXiv. Preprint posted online on Jun 6, 2024. [doi: 10.48550\/arXiv.2406.06608]"},{"issue":"2","key":"R21","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1214\/ss\/1009213286","article-title":"Interval estimation for a binomial proportion","volume":"16","author":"Brown","journal-title":"Statist Sci"},{"issue":"8","key":"R22","doi-asserted-by":"publisher","first-page":"857","DOI":"10.1002\/(sici)1097-0258(19980430)17:8<857::aid-sim777>3.0.co;2-e","article-title":"Two-sided confidence intervals for the single proportion: comparison of seven methods","volume":"17","author":"Newcombe","journal-title":"Stat Med"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-10-1","article-title":"A tutorial on pilot studies: the what, why and how","volume":"10","author":"Thabane","journal-title":"BMC Med Res Methodol"},{"issue":"2","key":"R24","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1111\/j..2002.384.doc.x","article-title":"Design and analysis of pilot studies: recommendations for good practice","volume":"10","author":"Lancaster","journal-title":"J Eval Clin Pract"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.i5239","article-title":"CONSORT 2010 statement: extension to randomised pilot and feasibility trials","volume":"355","author":"Eldridge","journal-title":"BMJ"},{"issue":"9","key":"R26","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0222916","article-title":"Why Cohen\u2019s kappa should be avoided as performance measure in classification","volume":"14","author":"Delgado","journal-title":"PLoS One"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.2174\/1874434601711010211","article-title":"High agreement and high prevalence: the paradox of Cohen\u2019s kappa","volume":"11","author":"Zec","journal-title":"Open Nurs J"},{"issue":"2","key":"R28","doi-asserted-by":"publisher","first-page":"98","DOI":"10.1016\/j.jcjq.2022.11.003","article-title":"Practical applications of rapid qualitative analysis for operations, quality improvement, and research in dynamically changing hospital environments","volume":"49","author":"Keniston","journal-title":"Jt Comm J Qual Patient Saf"},{"issue":"6","key":"R29","doi-asserted-by":"publisher","first-page":"855","DOI":"10.1177\/104973230201200611","article-title":"Matrix analysis as a complementary analytic strategy in qualitative inquiry","volume":"12","author":"Averill","journal-title":"Qual Health Res"},{"key":"R30","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-13-117","article-title":"Using the framework method for the analysis of qualitative data in multi-disciplinary health research","volume":"13","author":"Gale","journal-title":"BMC Med Res Methodol"},{"issue":"1","key":"R31","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01503-7","article-title":"Bias recognition and mitigation strategies in artificial intelligence healthcare applications","volume":"8","author":"Hasanzadeh","journal-title":"NPJ Digit Med"},{"key":"R32","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.n2183","article-title":"Accuracy of the patient health questionnaire-9 for screening to detect major depression: updated systematic review and individual participant data meta-analysis","volume":"375","author":"Negeri","journal-title":"BMJ"},{"issue":"6","key":"R33","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","article-title":"Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum","volume":"183","author":"Ayers","journal-title":"JAMA Intern Med"},{"issue":"3","key":"R34","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.3201","article-title":"Artificial intelligence-generated draft replies to patient inbox messages","volume":"7","author":"Garcia","journal-title":"JAMA Netw Open"},{"issue":"1","key":"R35","doi-asserted-by":"publisher","DOI":"10.1136\/bmjment-2023-300884","article-title":"ChatGPT and mental healthcare: balancing benefits with risks of harms","volume":"26","author":"Blease","journal-title":"BMJ Ment Health"},{"issue":"1","key":"R36","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-022-00611-y","article-title":"Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare","volume":"5","author":"Feng","journal-title":"NPJ Digit Med"},{"key":"R37","doi-asserted-by":"publisher","DOI":"10.2196\/39742","article-title":"Stakeholder perspectives of clinical artificial intelligence implementation: systematic review of qualitative evidence","volume":"25","author":"Hogg","journal-title":"J Med Internet Res"},{"key":"R38","doi-asserted-by":"publisher","DOI":"10.1016\/j.socscimed.2023.116357","article-title":"Multi-stakeholder preferences for the use of artificial intelligence in healthcare: a systematic review and thematic analysis","volume":"338","author":"Vo","journal-title":"Soc Sci Med"},{"key":"R39","doi-asserted-by":"publisher","DOI":"10.2196\/56930","article-title":"Roles, users, benefits, and limitations of chatbots in health care: rapid review","volume":"26","author":"Laymouna","journal-title":"J Med Internet Res"},{"issue":"11","key":"R40","doi-asserted-by":"publisher","DOI":"10.2196\/30704","article-title":"Mitigating patient and consumer safety risks when using conversational assistants for medical information: exploratory mixed methods experiment","volume":"23","author":"Bickmore","journal-title":"J Med Internet Res"},{"issue":"1","key":"R41","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00873-0","article-title":"The imperative for regulatory oversight of large language models (or generative AI) in healthcare","volume":"6","author":"Mesk\u00f3","journal-title":"NPJ Digit Med"},{"issue":"18","key":"R42","doi-asserted-by":"publisher","first-page":"1650","DOI":"10.1001\/jama.2025.18490","article-title":"AI, health, and health care today and tomorrow: the JAMA summit report on artificial intelligence","volume":"334","author":"Angus","journal-title":"JAMA"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T20:27:19Z","timestamp":1786134439000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e89837"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,7]]},"references-count":43,"URL":"https:\/\/doi.org\/10.2196\/89837","relation":{},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,7]]},"article-number":"v10i2e89837"}}