{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,5]],"date-time":"2026-03-05T07:46:35Z","timestamp":1772696795147,"version":"3.50.1"},"reference-count":23,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,10,21]],"date-time":"2024-10-21T00:00:00Z","timestamp":1729468800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Digit. Health"],"abstract":"<jats:p>Digital health research often relies on case vignettes (descriptions of fictitious or real patients) to navigate ethical and practical challenges. Despite their utility, the quality and lack of standardization of these vignettes has often been criticized, especially in studies on symptom-assessment applications (SAAs) and self-triage decision-making. To address this, our paper introduces a method to refine an existing set of vignettes, drawing on principles from classical test theory. First, we removed any vignette with an item difficulty of zero and an item-total correlation below zero. Second, we stratified the remaining vignettes to reflect the natural base rates of symptoms that SAAs are typically approached with, selecting those vignettes with the highest item-total correlation in each quota. Although this two-step procedure reduced the size of the original vignette set by 40%, comparing self-triage performance on the reduced and the original vignette sets, we found a strong correlation (<jats:italic>r<\/jats:italic>\u2009=\u20090.747 to <jats:italic>r<\/jats:italic>\u2009=\u20090.997, <jats:italic>p<\/jats:italic>\u2009&amp;lt;\u2009.001). This indicates that using our refinement method helps identifying vignettes with high predictive power of an agent's self-triage performance while simultaneously increasing cost-efficiency of vignette-based evaluation studies. This might ultimately lead to higher research quality and more reliable results.<\/jats:p>","DOI":"10.3389\/fdgth.2024.1411924","type":"journal-article","created":{"date-parts":[[2024,10,21]],"date-time":"2024-10-21T05:10:39Z","timestamp":1729487439000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Statistical refinement of patient-centered case vignettes for digital health research"],"prefix":"10.3389","volume":"6","author":[{"given":"Marvin","family":"Kopka","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Markus A.","family":"Feufel","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2024,10,21]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1177\/20552076231194929","article-title":"Triage and diagnostic accuracy of online symptom checkers: systematic review","volume":"25","author":"Riboli-Sasco","year":"2023","journal-title":"J Med Internet Res"},{"key":"B2","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1080\/13645579.2021.1901440","article-title":"Adapting vignettes for internet-based research: eliciting realistic responses to the digital milieu","volume":"25","author":"McInroy","year":"2022","journal-title":"Int J Soc Res Methodol"},{"key":"B3","doi-asserted-by":"publisher","first-page":"283","DOI":"10.1080\/17538068.2021.1945766","article-title":"Vignettes as research tools in global health communication: a systematic review of the literature from 2000 to 2020","volume":"14","author":"Riley","year":"2021","journal-title":"J Commun Healthc"},{"key":"B4","doi-asserted-by":"publisher","first-page":"586","DOI":"10.1007\/s11606-015-3365-8","article-title":"Methods of observing variations in physicians\u2019 decisions: the opportunities of clinical vignettes","volume":"30","author":"Converse","year":"2015","journal-title":"J Gen Intern Med"},{"key":"B5","doi-asserted-by":"publisher","first-page":"812","DOI":"10.1016\/j.jval.2020.12.017","article-title":"Vignette-based utilities: usefulness, limitations, and methodological recommendations","volume":"24","author":"Matza","year":"2021","journal-title":"Value Health"},{"key":"B6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1136\/bmj.h3480","article-title":"Evaluation of symptom checkers for self diagnosis and triage: audit study","volume":"351","author":"Semigran","year":"2015","journal-title":"Br Med J"},{"key":"B7","doi-asserted-by":"publisher","first-page":"118","DOI":"10.1038\/s41746-022-00667-w","article-title":"The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review","volume":"5","author":"Wallace","year":"2022","journal-title":"NPJ Digit Med"},{"key":"B8","doi-asserted-by":"publisher","first-page":"e37408","DOI":"10.2196\/37408","article-title":"Online symptom checkers: recommendations for a vignette-based clinical evaluation standard","volume":"24","author":"Painter","year":"2022","journal-title":"J Med Internet Res"},{"key":"B9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1177\/20552076231194929","article-title":"How suitable are clinical vignettes for the evaluation of symptom checker apps? A test theoretical perspective","volume":"9","author":"Kopka","year":"2023","journal-title":"Digit Health"},{"key":"B10","doi-asserted-by":"publisher","first-page":"514","DOI":"10.5694\/mja2.50600","article-title":"The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia","volume":"212","author":"Hill","year":"2020","journal-title":"Med J Aust"},{"key":"B11","doi-asserted-by":"publisher","first-page":"e31810","DOI":"10.2196\/31810","article-title":"Triage accuracy of symptom checker apps: 5-year follow-up evaluation","volume":"24","author":"Schmieding","year":"2022","journal-title":"J Med Internet Res"},{"key":"B12","doi-asserted-by":"publisher","first-page":"e24475","DOI":"10.2196\/24475","article-title":"Benchmarking triage capability of symptom checkers against that of medical laypersons: survey study","volume":"23","author":"Schmieding","year":"2021","journal-title":"J Med Internet Res"},{"key":"B13","doi-asserted-by":"publisher","first-page":"e555","DOI":"10.1016\/S2589-7500(24)00097-9","article-title":"The diagnostic and triage accuracy of the GPT-3 artificial intelligence model: an observational study","volume":"6","author":"Levine","year":"2024","journal-title":"Lancet Digit Health"},{"key":"B14","doi-asserted-by":"publisher","first-page":"e053566","DOI":"10.1136\/bmjopen-2021-053566","article-title":"What is the suitability of clinical vignettes in benchmarking the performance of online symptom checkers? An audit study","volume":"12","author":"El-Osta","year":"2022","journal-title":"BMJ Open"},{"key":"B15","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1111\/j.1365-2923.2009.03425.x","article-title":"A primer on classical test theory and item response theory for assessments in medical education","volume":"44","author":"De Champlain","year":"2010","journal-title":"Med Educ"},{"key":"B16","first-page":"2024.04.02.24305193","article-title":"Evaluating self-triage accuracy of laypeople, symptom-assessment apps, and large language models: a framework for case vignette development using a representative design approach (RepVig). medRxiv","author":"Kopka","year":"2024"},{"key":"B17","doi-asserted-by":"publisher","first-page":"e36322","DOI":"10.2196\/36322","article-title":"Health information seeking from an intelligent web-based symptom checker: cross-sectional questionnaire study","volume":"24","author":"Carmona","year":"2022","journal-title":"J Med Internet Res"},{"key":"B18","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1007\/978-3-642-20072-4_4","article-title":"Deskriptivstatistische evaluation von items (itemanalyse) und testwertverteilungen","volume-title":"Testtheorie und Fragebogenkonstruktion","author":"Kelava","year":"2012"},{"key":"B19","first-page":"1","article-title":"Grundlegende quantitative analysen medizinischer pr\u00fcfungen","volume":"23","author":"M\u00f6ltner","year":"2006","journal-title":"GMS J Med Educ"},{"key":"B20","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1186\/s44247-024-00096-7","article-title":"Software symptom check R: an R package for analyzing and visualizing symptom checker triage performance","volume":"2","author":"Kopka","year":"2024","journal-title":"BMC Digit Health"},{"key":"B21","doi-asserted-by":"publisher","first-page":"76","DOI":"10.1177\/011542650802300176","article-title":"Biostatistics primer: part 2","volume":"23","author":"Overholser","year":"2008","journal-title":"Nutr Clin Pract"},{"key":"B22","doi-asserted-by":"publisher","first-page":"e38977","DOI":"10.2196\/38977","article-title":"The triage capability of laypersons: retrospective exploratory analysis","volume":"6","author":"Kopka","year":"2022","journal-title":"JMIR Form Res"},{"key":"B23","doi-asserted-by":"publisher","first-page":"485","DOI":"10.1007\/s11136-013-0487-5","article-title":"Is Rasch model analysis applicable in small sample size pilot studies for assessing item characteristics? An example using PROMIS pain behavior item bank data","volume":"23","author":"Chen","year":"2014","journal-title":"Qual Life Res"}],"container-title":["Frontiers in Digital Health"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2024.1411924\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,21]],"date-time":"2024-10-21T05:10:42Z","timestamp":1729487442000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2024.1411924\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,21]]},"references-count":23,"alternative-id":["10.3389\/fdgth.2024.1411924"],"URL":"https:\/\/doi.org\/10.3389\/fdgth.2024.1411924","relation":{},"ISSN":["2673-253X"],"issn-type":[{"value":"2673-253X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,21]]},"article-number":"1411924"}}