{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,13]],"date-time":"2026-10-13T19:50:28Z","timestamp":1791921028601,"version":"4.3.4"},"reference-count":61,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Large language models (LLMs) could assist in streamlining health care instrument validation. This is particularly relevant as traditional instrument development is time-intensive and resource-intensive. Currently, there is a significant gap in specifically developed and validated trust instruments for pediatric emergency and surgical contexts. Such tools are key because trust is foundational to the patient family\u2013physician relationship and is associated with improved care-seeking and treatment adherence.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed (1) to develop new trust instruments and conduct their preliminary content validation confirming item relevance through a Synthetic Instrument Validation approach integrating human and LLM capabilities and (2) to evaluate appropriate use cases for LLMs in psychometric assessment.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Two trust instruments were developed (patient family and physician versions). In phase 1, instruments underwent a 2-stage content validation process using parallel synthetic and human expert panels (5 patient family members, 3 physician synthetic experts, and 7 unique human experts). Synthetic panels consisted of 3 persona-prompted LLMs (Claude Sonnet 4, GPT-5, and Grok 4), with human panels serving as comparators. The scale\u2013content validity index (S-CVI) threshold was set at \u22650.80, percent agreement on item relevancy was used to compare panels. In phase 2, LLM performance in quantitative research tasks was evaluated. The patient family instrument underwent Flesch-Kincaid assessment; both instruments underwent cosine similarity analyses using algorithmic and LLM-based methods.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n In phase 1, percent agreement between panels for the patient family and physician instruments was 90.9% and 88.9% for dimensional validation and 90.9% and 100% for contextual validation. Fleiss \u03ba for within-panel agreement ranged from poor to almost perfect; 0.20 (95% CI -0.23 to 0.63) to 1.00 (95% CI 1.00-1.00). All instrument subsections met or exceeded S-CVI thresholds across both rounds and panels. Phase 2 Flesch-Kincaid metrics converged across all 3 methods to target benchmark (mean Grade Level 8.1, SD 1.1; mean Reading Ease 60.1, SD 5.6), while reference readability was near target, showing convergence across implementations of the same formula. In contrast, cosine similarity analyses revealed significant LLM performance limitations; cosine similarity analyses were, therefore, abandoned in favor of the algorithmic method alone, which achieved a maximum cosine similarity of 0.83, indicating acceptable item distinctiveness overall.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Persona-prompted LLMs effectively assisted preliminary content validation and may help reduce timelines from months to weeks, but showed limitations in quantitative computations. This suggests that, in this study, persona-prompted LLMs performed better in subjective rating tasks than in deterministic cosine similarity computations. These findings help illustrate task-dependent boundaries for LLM integration in psychometric research, necessitating selective human-LLM collaboration. While these instruments achieved preliminary content validity, they require full psychometric testing before they can be considered validated for clinical use. Synthetic Instrument Validation offers a feasible approach to preliminary health care instrument development while preserving key elements of content validation.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/96901","type":"journal-article","created":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T10:10:05Z","timestamp":1790331005000},"page":"e96901-e96901","source":"Crossref","is-referenced-by-count":0,"title":["Synthetic Content Validation of Pediatric Trust Instruments Using Persona-Driven Large Language Models: Development and Evaluation Study"],"prefix":"10.2196","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-9300-1193","authenticated-orcid":false,"given":"Ella","family":"Boone","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3691-1319","authenticated-orcid":false,"given":"Katya","family":"Loban","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-6117-1470","authenticated-orcid":false,"given":"Carmel","family":"Daskalo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4087-1890","authenticated-orcid":false,"given":"Suganthi","family":"Rajasegaran","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4616-9990","authenticated-orcid":false,"given":"Elena","family":"Guadagno","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2473-3861","authenticated-orcid":false,"given":"Jeannie","family":"Haggerty","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9202-9251","authenticated-orcid":false,"given":"Sylvie","family":"Lambert","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6267-6140","authenticated-orcid":false,"given":"Dan","family":"Poenaru","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,10,13]]},"reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2023.1225850","article-title":"Measuring what matters in healthcare: a practical guide to psychometric principles and instrument development","volume":"14","author":"Swan","journal-title":"Front Psychol"},{"issue":"4","key":"R2","doi-asserted-by":"publisher","first-page":"326","DOI":"10.1111\/ijpp.12600","article-title":"How to measure a latent construct: psychometric principles for the development and validation of measurement instruments","volume":"28","author":"El-Den","journal-title":"Int J Pharm Pract"},{"issue":"96","key":"R3","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-14-96","article-title":"Designing questionnaires: healthcare survey to compare two different response scales","volume":"14","author":"Dell-Kuster","journal-title":"BMC Med Res Methodol"},{"issue":"2","key":"R4","doi-asserted-by":"publisher","first-page":"165","DOI":"10.15171\/jcs.2015.017","article-title":"Design and implementation content validity study: development of an instrument for measuring patient-centered communication","volume":"4","author":"Zamanzadeh","journal-title":"J Caring Sci"},{"issue":"1","key":"R5","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1016\/j.ambp.2005.08.001","article-title":"Parents\u2019 trust in their child\u2019s physician: using an adapted Trust in Physician Scale","volume":"6","author":"Moseley","journal-title":"Ambul Pediatr"},{"issue":"1","key":"R6","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1038\/s41591-021-01614-0","article-title":"AI in health and medicine","volume":"28","author":"Rajpurkar","journal-title":"Nat Med"},{"issue":"1","key":"R7","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00927-3","article-title":"Harnessing the power of synthetic data in healthcare: innovation, application, and privacy","volume":"6","author":"Giuffr\u00e8","journal-title":"NPJ Digit Med"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.3389\/frsps.2025.1460277","article-title":"Validating the use of large language models for psychological text classification","volume":"3","author":"Bunt","journal-title":"Front Soc Psychol"},{"issue":"11","key":"R9","doi-asserted-by":"publisher","DOI":"10.1016\/j.isci.2022.105331","article-title":"Synthetic data as an enabler for machine learning applications in medicine","volume":"25","author":"Rajotte","journal-title":"iScience"},{"issue":"11","key":"R10","doi-asserted-by":"publisher","first-page":"987","DOI":"10.1016\/j.tics.2023.08.006","article-title":"What are large language models supposed to model?","volume":"27","author":"Blank","journal-title":"Trends Cogn Sci"},{"key":"R11","unstructured":"Kim S Jeong J Han JS Shin D . LLM-mirror: a generated-persona approach for survey pre-testing. arXiv. Preprint posted online on Dec 4, 2024. [doi: 10.48550\/arXiv.2412.03162]"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1016\/j.nlp.2023.100020","article-title":"Employing large language models in survey research","volume":"4","author":"Jansen","journal-title":"Nat Lang Process J"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1016\/j.cpa.2024.102722","article-title":"Artificial intelligence and qualitative research: the promise and perils of large language model (LLM) \u2018assistance\u2019","volume":"99","author":"Roberts","journal-title":"Crit Perspect Account"},{"issue":"5","key":"R14","doi-asserted-by":"publisher","first-page":"871","DOI":"10.1016\/j.jpedsurg.2023.01.022","article-title":"Exploring trust development in families of children towards surgical and emergency care providers: a scoping review of the literature","volume":"58","author":"Serhan","journal-title":"J Pediatr Surg"},{"issue":"3","key":"R15","doi-asserted-by":"publisher","first-page":"504","DOI":"10.1016\/j.acap.2020.09.005","article-title":"A pilot study of a measure of parental trust in pediatric emergency department care teams","volume":"21","author":"Byczkowski","journal-title":"Acad Pediatr"},{"issue":"6","key":"R16","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0305562","article-title":"A national survey of children\u2019s experiences and needs when attending Canadian pediatric emergency departments","volume":"19","author":"Ma","journal-title":"PLoS One"},{"issue":"5","key":"R17","doi-asserted-by":"publisher","first-page":"841","DOI":"10.1016\/j.surg.2013.12.011","article-title":"The effect of complications on the patient-surgeon relationship after colorectal cancer surgery","volume":"155","author":"Regenbogen","journal-title":"Surgery"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.1177\/2050312116664224","article-title":"Trust in healthcare settings: scale development, methods, and preliminary determinants","volume":"4","author":"LoCurto","journal-title":"SAGE Open Med"},{"issue":"9","key":"R19","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0106844","article-title":"Assessment of trust in physician: a systematic review of measures","volume":"9","author":"M\u00fcller","journal-title":"PLoS One"},{"issue":"7","key":"R20","doi-asserted-by":"publisher","first-page":"509","DOI":"10.1046\/j.1525-1497.2000.11002.x","article-title":"Patients\u2019 trust in physicians: many theories, few measures, and little data","volume":"15","author":"Pearson","journal-title":"J Gen Intern Med"},{"issue":"4","key":"R21","first-page":"323","volume":"50","author":"Thom","journal-title":"J Fam Pract"},{"issue":"3","key":"R22","doi-asserted-by":"publisher","first-page":"227","DOI":"10.1002\/pon.1745","article-title":"Cancer patients\u2019 trust in their physician-a review","volume":"20","author":"Hillen","journal-title":"Psychooncology"},{"issue":"4","key":"R23","doi-asserted-by":"publisher","first-page":"613","DOI":"10.1111\/1468-0009.00223","article-title":"Trust in physicians and medical institutions: what is it, can it be measured, and does it matter?","volume":"79","author":"Hall","journal-title":"Milbank Q"},{"issue":"7","key":"R24","doi-asserted-by":"publisher","DOI":"10.1007\/s00431-025-06297-0","article-title":"Trustful relationships between healthcare professionals and children: a concept analysis using Rodgers\u2019 evolutionary approach","volume":"184","author":"Dickens","journal-title":"Eur J Pediatr"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.1016\/j.chbah.2025.100144","article-title":"Harnessing the power of AI in qualitative research: exploring, using and redesigning ChatGPT","volume":"4","author":"Zhang","journal-title":"Comput Hum Behav Artif Hum"},{"issue":"3","key":"R26","doi-asserted-by":"publisher","first-page":"293","DOI":"10.1177\/1077558702059003004","article-title":"Measuring patients\u2019 trust in their primary care providers","volume":"59","author":"Hall","journal-title":"Med Care Res Rev"},{"issue":"1","key":"R27","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1177\/00914150231218929","article-title":"Psychometric properties of the Wake Forest Physician Trust Scale with older adults","volume":"100","author":"Katz","journal-title":"Int J Aging Hum Dev"},{"issue":"5","key":"R28","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0303840","article-title":"Measuring trust in one\u2019s physician: A scoping review","volume":"19","author":"Weber","journal-title":"PLoS One"},{"issue":"2","key":"R29","doi-asserted-by":"publisher","first-page":"49","DOI":"10.21315\/eimj2019.11.2.6","article-title":"ABC of content validation and content validity index calculation","volume":"11","author":"Yusoff","journal-title":"Educ Med J"},{"issue":"1","key":"R30","doi-asserted-by":"publisher","first-page":"554","DOI":"10.1186\/s12913-017-2469-8","article-title":"Development and evaluation of the content validity, practicability and feasibility of the innovative dementia-oriented assessment system for challenging behaviour in residents with dementia","volume":"17","author":"Halek","journal-title":"BMC Health Serv Res"},{"issue":"1","key":"R31","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01258-7","article-title":"A framework for human evaluation of large language models in healthcare derived from literature review","volume":"7","author":"Tam","journal-title":"NPJ Digit Med"},{"key":"R32","unstructured":"Introducing Claude 4. Anthropic. URL: https:\/\/www.anthropic.com\/news\/claude-4 [Accessed 03-11-2025]"},{"key":"R33","unstructured":"Introducing GPT-5. OpenAI. 2025. URL: https:\/\/openai.com\/index\/introducing-gpt-5\/ [Accessed 03-11-2025]"},{"key":"R34","unstructured":"Grok 4. Space X. URL: https:\/\/x.ai\/news\/grok-4 [Accessed 03-11-2025]"},{"key":"R35","unstructured":"Randolph JJ . Online Kappa calculator. Justus Randolph. 2008. URL: http:\/\/justusrandolph.net\/kappa\/ [Accessed 06-10-2026]"},{"issue":"1","key":"R36","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","volume":"33","author":"Landis","journal-title":"Biometrics"},{"key":"R37","unstructured":"Introducing Claude Sonnet 4.5. Anthropic. URL: https:\/\/www.anthropic.com\/news\/claude-sonnet-4-5 [Accessed 12-12-2025]"},{"issue":"2","key":"R38","doi-asserted-by":"publisher","DOI":"10.5812\/atr.18161","article-title":"Readability of patient education materials on the American Association for Surgery of Trauma website","volume":"3","author":"Eltorai","journal-title":"Arch Trauma Res"},{"key":"R39","unstructured":"Ease FR . Flesch Reading Ease and Flesch-Kincaid Grade Level. Readable. URL: https:\/\/readable.com\/readability\/flesch-reading-ease-flesch-kincaid-grade-level\/ [Accessed 02-12-2025]"},{"issue":"5","key":"R40","doi-asserted-by":"publisher","first-page":"293","DOI":"10.1097\/SPV.0b013e31828ab3e2","article-title":"Readability of common health-related quality-of-life instruments in female pelvic medicine","volume":"19","author":"Alas","journal-title":"Female Pelvic Med Reconstr Surg"},{"key":"R41","unstructured":"OpenAI. URL: https:\/\/platform.openai.com [Accessed 10-01-2026]"},{"issue":"5","key":"R42","doi-asserted-by":"publisher","first-page":"1147","DOI":"10.1007\/s11136-018-1798-3","article-title":"COSMIN guideline for systematic reviews of patient-reported outcome measures","volume":"27","author":"Prinsen","journal-title":"Qual Life Res"},{"issue":"2","key":"R43","doi-asserted-by":"publisher","first-page":"356","DOI":"10.1177\/0013164420988169","article-title":"On the relationship between item stem formulation and criterion validity of multiple-component measuring instruments","volume":"82","author":"Menold","journal-title":"Educ Psychol Meas"},{"issue":"1","key":"R44","doi-asserted-by":"publisher","first-page":"8","DOI":"10.51983\/ajms-2022.11.1.3087","article-title":"Designing a questionnaire for a research paper: a comprehensive guide to design and develop an effective questionnaire","volume":"11","author":"Taherdoost","journal-title":"Asian J Manag Sci"},{"issue":"2","key":"R45","doi-asserted-by":"publisher","first-page":"234","DOI":"10.1016\/j.pec.2007.09.009","article-title":"Barriers to patient involvement in health service planning and evaluation: an exploratory study","volume":"70","author":"Gagliardi","journal-title":"Patient Educ Couns"},{"issue":"8","key":"R46","doi-asserted-by":"publisher","first-page":"501","DOI":"10.1093\/pch\/pxae012","article-title":"Canadian developmental pediatrics workforce survey","volume":"29","author":"McLeod","journal-title":"Paediatr Child Health"},{"key":"R47","doi-asserted-by":"crossref","unstructured":"Croxford E Gao Y First E et al. Automating evaluation of AI text generation in healthcare with a large language model (LLM)-as-a-judge. medRxiv. Preprint posted online on May 6, 2025. [doi: 10.1101\/2025.04.22.25326219]","DOI":"10.1101\/2025.04.22.25326219"},{"issue":"4","key":"R48","doi-asserted-by":"publisher","DOI":"10.3390\/ijerph18042121","article-title":"A review on human-AI interaction in machine learning and insights for medical applications","volume":"18","author":"Maadi","journal-title":"Int J Environ Res Public Health"},{"issue":"12","key":"R49","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2022.46051","article-title":"Comparison of readability scores for written health information across formulas using automated vs manual measures","volume":"5","author":"Mac","journal-title":"JAMA Netw Open"},{"issue":"1","key":"R50","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1109\/TPC.2016.2635720","article-title":"How consistent are the best-known readability equations in estimating the readability of design standards?","volume":"60","author":"Zhou","journal-title":"IEEE Trans Profess Commun"},{"key":"R51","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2022.10.064","article-title":"Research on text classification techniques based on improved TF-IDF algorithm and LSTM inputs","volume":"208","author":"Liang","journal-title":"Procedia Comput Sci"},{"key":"R52","doi-asserted-by":"crossref","unstructured":"Shojaee P Mirzadeh I Alizadeh K Horton M Bengio S Farajtabar M . The illusion of thinking: understanding the strengths and limitations of reasoning models via the lens of problem complexity. arXiv. Preprint posted online on Jun 7, 2025. [doi: 10.48550\/arXiv.2506.06941]","DOI":"10.70777\/si.v2i6.15919"},{"issue":"8032","key":"R53","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1038\/s41586-024-07930-y","article-title":"Larger and more instructable language models become less reliable","volume":"634","author":"Zhou","journal-title":"Nature"},{"key":"R54","doi-asserted-by":"crossref","unstructured":"Williams S Huckle J . Easy problems that LLMs get wrong. arXiv. Preprint posted online on May 30, 2024. [doi: 10.1007\/978-3-031-84457-7_19]","DOI":"10.1007\/978-3-031-84457-7_19"},{"issue":"4","key":"R55","doi-asserted-by":"publisher","first-page":"401","DOI":"10.15288\/jsad.2020.81.401","article-title":"The case for cognitive interviewing in survey item validation: a useful approach for improving the measurement and assessment of substance use disorders","volume":"81","author":"Boness","journal-title":"J Stud Alcohol Drugs"},{"key":"R56","doi-asserted-by":"publisher","DOI":"10.1016\/j.nedt.2022.105603","article-title":"Instrument development and psychometric testing in nursing education research","volume":"119","author":"Mikkonen","journal-title":"Nurse Educ Today"},{"key":"R57","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijnsa.2025.100327","article-title":"Development and application of patient-reported experience measures for cancer patients: a scoping review","volume":"8","author":"Shao","journal-title":"Int J Nurs Stud Adv"},{"key":"R58","doi-asserted-by":"publisher","DOI":"10.1136\/bmj-2024-081554","article-title":"FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare","volume":"388","author":"Lekadir","journal-title":"BMJ"},{"issue":"2","key":"R59","doi-asserted-by":"publisher","first-page":"65","DOI":"10.3390\/encyclopedia5020065","article-title":"Questionnaire use and development in health research","volume":"5","author":"Koo","journal-title":"Encyclopedia"},{"issue":"1","key":"R60","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-21289-8","article-title":"Rethinking psychometrics through LLMs: how item semantics shape measurement and prediction in psychological questionnaires","volume":"15","author":"Ravenda","journal-title":"Sci Rep"},{"key":"R61","doi-asserted-by":"publisher","DOI":"10.2196\/90770","article-title":"Quantifying and disclosing the environmental footprint of AI in research: life cycle-informed framework and open-access calculator development study","volume":"5","author":"Mozafari","journal-title":"JMIR AI"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,10,13]],"date-time":"2026-10-13T19:25:18Z","timestamp":1791919518000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e96901"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,13]]},"references-count":61,"URL":"https:\/\/doi.org\/10.2196\/96901","relation":{},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,13]]},"article-number":"v10i4e96901"}}