{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"medRxiv"}],"indexed":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T10:24:12Z","timestamp":1786962252487,"version":"build-2736575974"},"posted":{"date-parts":[[2026,8,14]]},"group-title":"Psychiatry and Clinical Psychology","reference-count":45,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T00:00:00Z","timestamp":1786665600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2026,8,14]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                <jats:sec>\n                  <jats:title>IMPORTANCE<\/jats:title>\n                  <jats:p>Systematic reviews and meta-analyses inform suicide-prevention policy and practice, but broad database searches are difficult to screen manually. This limits capture of upstream interventions, such as economic policies, with indirect effects on suicide. Reliable automated screening could make broader and more comprehensive evidence syntheses feasible.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>OBJECTIVE<\/jats:title>\n                  <jats:p>To develop and validate ScreenAgent, a large language model (LLM) agent for title and abstract screening, and a review-specific method for prospectively estimating screening performance.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>DESIGN, SETTING, AND PARTICIPANTS<\/jats:title>\n                  <jats:p>ScreenAgent was validated internally on a prospective meta-analysis, and externally on two published systematic reviews. The correct include and exclude decisions followed standard systematic-review screening methodology.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>EXPOSURES<\/jats:title>\n                  <jats:p>ScreenAgent, an LLM agent returning structured include-or-exclude decisions. Records it marked for inclusion were re-checked by a second, cascade pass using a higher-effort LLM. For the external reviews, the agent\u2019s prompt was tuned automatically on a small set of labeled examples.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>MAIN OUTCOMES AND MEASURES<\/jats:title>\n                  <jats:p>We calculated sensitivity, specificity, workload reduction (the percentage of records removed from human review), and agent-versus-human reliability via Cohen kappa. Sensitivity was estimated by direct comparison (internal) and 5-fold cross-validation (external).<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>RESULTS<\/jats:title>\n                  <jats:p>In the internal validation, ScreenAgent identified 43 of 44 eligible studies (sensitivity 97.7%; 95% CI, 88.2%-99.6%) with a generic prompt applied without any review-specific optimization, specificity 98.0%, and a measured full-corpus workload reduction of 99.4%. The cost was $855.91 for the full 201,064-record corpus (0.43 US cents per record). Agent-versus-human-consensus agreement exceeded human-versus-human agreement (Cohen kappa 0.75 vs 0.64; percent agreement 97.3% vs 95.4%). For two external validation studies, automatic tuning resulted in a cross-validated sensitivity of 95.9% (95% CI, 90.0%-98.4%) and 97.4% (90.9%-99.3%), with workload reductions of 97.4% and 98.4%.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>CONCLUSIONS AND RELEVANCE<\/jats:title>\n                  <jats:p>Suicide prevention efforts often require rapid consolidation of evidence because of the inherent challenges of single studies trying to prevent rare outcomes. On both internal and external validation sets, ScreenAgent identified nearly all eligible studies with human-level reliability for a fraction of a US cent per record while keeping human reviewers as the final arbiters. By making broad searches feasible and screening performance measurable beforehand, this approach can serve as a transparent methodology to strengthen the speed at which we can inform and advance suicide prevention efforts.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Key point<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Question<\/jats:title>\n                    <jats:p>Can an LLM agent screen records accurately to make broad-scale meta-analyses in suicide prevention more feasible?<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Findings<\/jats:title>\n                    <jats:p>Across a prospective review and two published reviews, ScreenAgent identified nearly all eligible studies (internal sensitivity 97.7%; external 95.9\/97.4%) with human-level reliability (Cohen kappa 0.75 agent-vs-consensus, vs 0.64 between humans) and 98.0% specificity.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Meaning<\/jats:title>\n                    <jats:p>Systematic reviews and meta-analyses are critical for evidence synthesis but are time and labor intensive. An LLM screening agent identified nearly all eligible studies with human-level reliability while reducing human workload by 99% at a fraction of a cent per record, which makes large-scale syntheses tractable.<\/jats:p>\n                  <\/jats:sec>\n                <\/jats:sec>","DOI":"10.64898\/2026.08.12.26360335","type":"posted-content","created":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T13:20:16Z","timestamp":1786713616000},"source":"Crossref","is-referenced-by-count":0,"title":["Making Broad Evidence Synthesis Feasible: An LLM Screening Agent for Meta-Analyses Applied To Suicide Prevention"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-3879-614X","authenticated-orcid":false,"given":"Daniel","family":"Dobin","sequence":"first","affiliation":[{"name":"Center for Suicide Prevention, Department of Mental Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA"},{"name":"Johns Hopkins Krieger School of Arts and Sciences, Baltimore, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-7558-9907","authenticated-orcid":false,"given":"Ashley M.","family":"Witmer","sequence":"additional","affiliation":[{"name":"Center for Suicide Prevention, Department of Mental Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2074-5611","authenticated-orcid":false,"given":"Fiona Gail","family":"Sweeney","sequence":"additional","affiliation":[{"name":"Department of Psychiatry and Behavioral Sciences, Johns Hopkins University School of Medicine, Baltimore, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7484-4749","authenticated-orcid":false,"given":"Taylor","family":"Ryan","sequence":"additional","affiliation":[{"name":"Center for Suicide Prevention, Department of Mental Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA"},{"name":"Department of Psychiatry and Behavioral Sciences, Johns Hopkins University School of Medicine, Baltimore, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7920-5943","authenticated-orcid":false,"given":"Andrea","family":"Cimino","sequence":"additional","affiliation":[{"name":"Center for Suicide Prevention, Department of Mental Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA"},{"name":"Department of Psychiatry and Behavioral Sciences, Johns Hopkins University School of Medicine, Baltimore, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1833-4925","authenticated-orcid":false,"given":"Emily E.","family":"Haroz","sequence":"additional","affiliation":[{"name":"Center for Suicide Prevention, Department of Mental Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2479-703X","authenticated-orcid":false,"given":"Paul S.","family":"Nestadt","sequence":"additional","affiliation":[{"name":"Center for Suicide Prevention, Department of Mental Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA"},{"name":"Department of Psychiatry and Behavioral Sciences, Johns Hopkins University School of Medicine, Baltimore, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2624-0654","authenticated-orcid":false,"given":"Holly C.","family":"Wilcox","sequence":"additional","affiliation":[{"name":"Center for Suicide Prevention, Department of Mental Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA"},{"name":"Department of Psychiatry and Behavioral Sciences, Johns Hopkins University School of Medicine, Baltimore, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"54368","reference":[{"key":"2026081702250688000_2026.08.12.26360335v1.1","unstructured":"World Health Organization. Suicide. Fact sheet. Geneva: World Health Organization; updated March 25, 2025. Accessed June 14, 2026. https:\/\/www.who.int\/news-room\/fact-sheets\/detail\/suicide"},{"key":"2026081702250688000_2026.08.12.26360335v1.2","doi-asserted-by":"publisher","DOI":"10.1016\/S2468-2667(25)00006-4"},{"key":"2026081702250688000_2026.08.12.26360335v1.3","unstructured":"US Department of Health and Human Services, Office of the Surgeon General. 2024 National Strategy for Suicide Prevention. Washington, DC: US Department of Health and Human Services; 2024. Accessed June 14, 2026. https:\/\/www.hhs.gov\/sites\/default\/files\/national-strategy-suicide-prevention.pdf"},{"key":"2026081702250688000_2026.08.12.26360335v1.4","doi-asserted-by":"publisher","DOI":"10.1001\/jama.294.16.2064"},{"issue":"7","key":"2026081702250688000_2026.08.12.26360335v1.5","doi-asserted-by":"crossref","first-page":"646","DOI":"10.1016\/S2215-0366(16)30030-X","article-title":"Suicide prevention strategies revisited: 10-year systematic review","volume":"3","year":"2016","journal-title":"Lancet Psychiatry"},{"key":"2026081702250688000_2026.08.12.26360335v1.6","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.eclinm.2018.10.004","article-title":"What works in youth suicide prevention? A systematic review and meta-analysis","volume":"4-5","year":"2018","journal-title":"EClinicalMedicine"},{"key":"2026081702250688000_2026.08.12.26360335v1.7","first-page":"127","article-title":"Effectiveness of suicide prevention interventions: a systematic review and meta-analysis","volume":"63","year":"2019","journal-title":"Gen Hosp Psychiatry"},{"issue":"10","key":"2026081702250688000_2026.08.12.26360335v1.8","doi-asserted-by":"crossref","first-page":"e787","DOI":"10.1016\/S2468-2667(24)00149-X","article-title":"Preventing suicide: a public health approach to a global problem","volume":"9","year":"2024","journal-title":"Lancet Public Health"},{"key":"2026081702250688000_2026.08.12.26360335v1.9","doi-asserted-by":"publisher","DOI":"10.1177\/070674370905400407"},{"key":"2026081702250688000_2026.08.12.26360335v1.10","doi-asserted-by":"publisher","DOI":"10.1521\/suli.2007.37.3.338"},{"issue":"10","key":"2026081702250688000_2026.08.12.26360335v1.11","doi-asserted-by":"crossref","first-page":"e1026","DOI":"10.1016\/S2214-109X(17)30299-1","article-title":"Prevention of suicide with regulations aimed at restricting access to highly hazardous pesticides: a systematic review of the international evidence","volume":"5","year":"2017","journal-title":"Lancet Glob Health"},{"key":"2026081702250688000_2026.08.12.26360335v1.12","doi-asserted-by":"publisher","DOI":"10.1136\/jech-2019-212981"},{"key":"2026081702250688000_2026.08.12.26360335v1.13","doi-asserted-by":"publisher","DOI":"10.1093\/aje\/kwu106"},{"key":"2026081702250688000_2026.08.12.26360335v1.14","doi-asserted-by":"publisher","DOI":"10.1016\/j.drugalcdep.2008.01.005"},{"key":"2026081702250688000_2026.08.12.26360335v1.15","doi-asserted-by":"publisher","DOI":"10.1001\/jama.292.5.594"},{"key":"2026081702250688000_2026.08.12.26360335v1.16","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2016-012545"},{"key":"2026081702250688000_2026.08.12.26360335v1.17","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1000326"},{"key":"2026081702250688000_2026.08.12.26360335v1.18","doi-asserted-by":"crossref","first-page":"112221","DOI":"10.1016\/j.jclinepi.2026.112221","article-title":"Large language models show promising performance for some systematic review tasks but call for cautious implementation: a systematic review","volume":"194","year":"2026","journal-title":"J Clin Epidemiol"},{"issue":"2","key":"2026081702250688000_2026.08.12.26360335v1.19","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1038\/s42256-020-00287-7","article-title":"An open source machine learning framework for efficient and transparent systematic reviews","volume":"3","year":"2021","journal-title":"Nat Mach Intell"},{"issue":"1","key":"2026081702250688000_2026.08.12.26360335v1.20","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1186\/s13643-018-0707-8","article-title":"Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool","volume":"7","year":"2018","journal-title":"Syst Rev"},{"key":"2026081702250688000_2026.08.12.26360335v1.21","doi-asserted-by":"publisher","DOI":"10.1186\/2046-4053-4-5"},{"key":"2026081702250688000_2026.08.12.26360335v1.22","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-019-1074-9"},{"key":"2026081702250688000_2026.08.12.26360335v1.23","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocaf050"},{"key":"2026081702250688000_2026.08.12.26360335v1.24","doi-asserted-by":"publisher","DOI":"10.2196\/48996"},{"issue":"4","key":"2026081702250688000_2026.08.12.26360335v1.25","doi-asserted-by":"crossref","first-page":"616","DOI":"10.1002\/jrsm.1715","article-title":"Can large language models replace humans in systematic reviews? Evaluating GPT-4\u2019s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages","volume":"15","year":"2024","journal-title":"Res Synth Methods"},{"key":"2026081702250688000_2026.08.12.26360335v1.26","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-024-02575-4"},{"key":"2026081702250688000_2026.08.12.26360335v1.27","doi-asserted-by":"publisher","DOI":"10.1136\/bmjment-2025-301762"},{"key":"2026081702250688000_2026.08.12.26360335v1.28","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2020.11.003"},{"key":"2026081702250688000_2026.08.12.26360335v1.29","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2017.08.010"},{"key":"2026081702250688000_2026.08.12.26360335v1.30","first-page":"CD013668","article-title":"Psychosocial interventions for self-harm in adults","volume":"4","year":"2021","journal-title":"Cochrane Database Syst Rev"},{"key":"2026081702250688000_2026.08.12.26360335v1.31","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-03425-5"},{"key":"2026081702250688000_2026.08.12.26360335v1.32","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-020-01542-z"},{"key":"2026081702250688000_2026.08.12.26360335v1.33","unstructured":"Anthropic. Claude Opus 4.8. Anthropic; 2026."},{"key":"2026081702250688000_2026.08.12.26360335v1.34","unstructured":"Higgins JPT , Thomas J , Chandler J , et al , eds. Cochrane Handbook for Systematic Reviews of Interventions. Version 6.5. Cochrane; 2024."},{"key":"2026081702250688000_2026.08.12.26360335v1.35","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.n71"},{"key":"2026081702250688000_2026.08.12.26360335v1.36","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"issue":"2","key":"2026081702250688000_2026.08.12.26360335v1.37","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1111\/j.2517-6161.1974.tb00994.x","article-title":"Cross-validatory choice and assessment of statistical predictions","volume":"36","year":"1974","journal-title":"J R Stat Soc Series B Stat Methodol"},{"key":"2026081702250688000_2026.08.12.26360335v1.38","doi-asserted-by":"publisher","DOI":"10.2307\/2276774"},{"key":"2026081702250688000_2026.08.12.26360335v1.39","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-0258(19980430)17:8<857::AID-SIM777>3.0.CO;2-E"},{"key":"2026081702250688000_2026.08.12.26360335v1.40","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2007.55"},{"key":"2026081702250688000_2026.08.12.26360335v1.41","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-2649-2"},{"key":"2026081702250688000_2026.08.12.26360335v1.42","doi-asserted-by":"publisher","DOI":"10.1016\/0895-4356(90)90158-L"},{"key":"2026081702250688000_2026.08.12.26360335v1.43","doi-asserted-by":"publisher","DOI":"10.1016\/0895-4356(93)90018-V"},{"key":"2026081702250688000_2026.08.12.26360335v1.44","doi-asserted-by":"publisher","DOI":"10.1192\/bjp.bp.116.187799"},{"key":"2026081702250688000_2026.08.12.26360335v1.45","doi-asserted-by":"crossref","unstructured":"Chen L , Zaharia M , Zou J. How is ChatGPT\u2019s behavior changing over time? Harv Data Sci Rev. 2024;6(2).","DOI":"10.1162\/99608f92.5317da47"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.64898\/2026.08.12.26360335","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T09:25:18Z","timestamp":1786958718000},"score":1,"resource":{"primary":{"URL":"http:\/\/medrxiv.org\/lookup\/doi\/10.64898\/2026.08.12.26360335"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,14]]},"references-count":45,"URL":"https:\/\/doi.org\/10.64898\/2026.08.12.26360335","relation":{},"subject":[],"published":{"date-parts":[[2026,8,14]]},"subtype":"preprint"}}