{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T12:26:23Z","timestamp":1789475183419,"version":"build-2803163510"},"reference-count":38,"publisher":"SAGE Publications","license":[{"start":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T00:00:00Z","timestamp":1777507200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"},{"start":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T00:00:00Z","timestamp":1777507200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Can J Psychiatry"],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Large language models (LLMs) may reduce the burden associated with performing systematic reviews by prescreening abstracts from a literature search for eligibility for inclusion in full-text review.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We developed an iterative, LLM-based workflow for screening abstracts: after manual specification of eligibility criteria and seed examples, an ensemble of five LLMs deliberates through a Delphi process to classify a batch of abstracts; these labels are used to train a logistic regression model that ranks the remaining abstracts and identifies a new batch of abstracts for LLM escalation until all abstracts are labelled by the LLM or probability thresholds. We tested our workflow on abstracts screened in three published systematic reviews in psychiatry. Our primary endpoint was the recall metric, and secondary endpoint was the work saved over sampling at 95% recall metric (WSS@95%).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>In a dataset on autism biomarkers, 1,655 (35%) of 4,745 retrieved abstracts were judged to be relevant by the original authors. The Delphi\u2013LLM workflow correctly identified 1,605 (97.0%) of these 1,655 abstracts (precision\u2009=\u200954.2%, WSS@95%\u2009=\u200938.1%). The performance metrics were better than non-LLM approaches (recall\u2009\u2264\u200991%, WSS@95\u2009\u2264\u200926%), and, overall, balanced these metrics optimally compared to single-LLM agents (recall\u2009=\u200984.9\u201399.9%, WSS@95%\u2009=\u200916.7\u201339.8%). The recall and work saved metrics were similarly reliable and among the top in two low-prevalence datasets on an attention-deficit hyperactivity disorder treatment review (10% of 2,891 relevant) and a posttraumatic stress disorder trajectory review (7% of 4,453 relevant). For these two datasets, recall was 100.0% and 96.4%, and the WSS@95% was 17.3% and 18.5%, respectively.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>We presented the design and validation of a novel abstract screening workflow that centres around a Delphi-style aggregation process to harness the strengths of five open-source LLMs that can be run on consumer-level workstations. This multi-LLM workflow showed acceptable and reliable performance for use as an automated prescreening method to facilitate systematic reviews.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/07067437261445767","type":"journal-article","created":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T12:45:30Z","timestamp":1777553130000},"update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":1,"title":["Novel Abstract Screening Algorithm Using Delphi-Inspired Large Language Model Consensus for Systematic Reviews in Psychiatry: Nouvel algorithme de s\u00e9lection des r\u00e9sum\u00e9s utilisant un consensus issu d\u2019un grand mod\u00e8le de langage inspir\u00e9 de la m\u00e9thode Delphi pour les revues syst\u00e9matiques en psychiatrie"],"prefix":"10.1177","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4111-0156","authenticated-orcid":false,"given":"Mirkamal","family":"Tolend","sequence":"first","affiliation":[{"name":"Centre for Addiction and Mental Health"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3784-9556","authenticated-orcid":false,"given":"Ramzi","family":"Halabi","sequence":"additional","affiliation":[{"name":"UT Southwestern Medical Center"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-6714-6501","authenticated-orcid":false,"given":"Kousai","family":"Ghaouari","sequence":"additional","affiliation":[{"name":"Centre for Addiction and Mental Health"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-1399-4541","authenticated-orcid":false,"given":"Yvonne C.Y.","family":"Lau","sequence":"additional","affiliation":[{"name":"Centre for Addiction and Mental Health"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9544-3944","authenticated-orcid":false,"given":"Martin","family":"Alda","sequence":"additional","affiliation":[{"name":"Dalhousie University"},{"name":"National Institute of Mental Health, Klecany, Czech Republic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arend","family":"Hintze","sequence":"additional","affiliation":[{"name":"Dalarna University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0303-6450","authenticated-orcid":false,"given":"Benoit H.","family":"Mulsant","sequence":"additional","affiliation":[{"name":"Centre for Addiction and Mental Health"},{"name":"Department of Psychiatry, Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada"},{"name":"The Royal Ottawa Hospital, Ottawa, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6886-767X","authenticated-orcid":false,"given":"Abigail","family":"Ortiz","sequence":"additional","affiliation":[{"name":"Centre for Addiction and Mental Health"},{"name":"UT Southwestern Medical Center"},{"name":"Department of Psychiatry, Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2026,4,30]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-024-02459-6"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-023-02448-8"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2025.111746"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2016-012545"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.conctc.2019.100443"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2023-076912"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-019-1074-9"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1093\/aje\/kwae232"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-023-02351-w"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-024-02224-3"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-020-00287-7"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-025-02583-5"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1136\/bjo-2024-326254"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.2196\/48996"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocaf030"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2512.11261"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2505.24757"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.2196\/52758"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-025-03138-w"},{"key":"e_1_3_2_21_2","doi-asserted-by":"crossref","unstructured":"Helmer-Hirschberg O Rescher NH. On the epistemology of the inexact sciences. 1 January 1960 [accessed 13 November 2025]. https:\/\/www.rand.org\/pubs\/reports\/R353.html","DOI":"10.1287\/mnsc.6.1.25"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1176\/appi.ajp.21100992"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1002\/14651858.CD012069.pub2"},{"key":"e_1_3_2_24_2","first-page":"250","article-title":"CLEF 2019 technology assisted reviews in empirical medicine overview","volume":"2380","author":"Kanoulas E","year":"2019","unstructured":"Kanoulas E, Li D, Azzopardi L, et al. CLEF 2019 technology assisted reviews in empirical medicine overview. CEUR Workshop Proc. 2019;2380:250 [accessed 21 February 2026]. https:\/\/strathprints.strath.ac.uk\/71253\/","journal-title":"CEUR Workshop Proc."},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1080\/00273171.2017.1412293"},{"key":"e_1_3_2_26_2","unstructured":"De BruinJ MaY FerdinandsG et al. asreview\/synergy-dataset. 2023 [accessed 21 February 2026]. https:\/\/github.com\/asreview\/synergy-dataset."},{"key":"e_1_3_2_27_2","unstructured":"De Bruin J Ma Y Ferdinands G et al. SYNERGY - Open machine learning dataset on study selection in systematic reviews. DataverseNL. 2023. doi:10.34894\/HE6NAQ."},{"key":"e_1_3_2_28_2","doi-asserted-by":"crossref","unstructured":"Singh A D\u2019Arcy M Cohan A et al. Scirepeval: a multi-format benchmark for scientific document representations. In: Bouamor H Pino J Bali K editors. Proceedings of the 2023 conference on empirical methods in natural language processing. Singapore: Association for Computational Linguistics 5548\u20135566. doi:https:\/\/doi.org\/10.18653\/v1\/2023.emnlp-main.338","DOI":"10.18653\/v1\/2023.emnlp-main.338"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","unstructured":"Team G Kamath A Ferret J et al. Gemma 3 technical report. March 25 2025. doi:10.48550\/arXiv.2503.19786","DOI":"10.48550\/arXiv.2503.19786"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","unstructured":"Yang A Li A Yang B et al. Qwen3 technical report. May 14 2025. doi:10.48550\/arXiv.2505.09388","DOI":"10.48550\/arXiv.2505.09388"},{"key":"e_1_3_2_31_2","unstructured":"IBM Granite 3.2: open source reasoning and vision. 2025 [accessed 30 November 2025]. https:\/\/www.ibm.com\/new\/announcements\/ibm-granite-3-2-open-source-reasoning-and-vision."},{"key":"e_1_3_2_32_2","unstructured":"meta-llama\/Llama-3.1-8B-Instruct \u00b7 Hugging Face. 2024 [accessed 30 November 2025]. https:\/\/huggingface.co\/meta-llama\/Llama-3.1-8B-Instruct."},{"key":"e_1_3_2_33_2","unstructured":"mistralai\/Ministral-8B-Instruct-2410 \u00b7 Hugging Face [accessed 30 November 2025]. https:\/\/huggingface.co\/mistralai\/Ministral-8B-Instruct-2410."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1197\/jamia.M1929"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.iswa.2023.200193"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpsychires.2014.07.011"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1352\/1944.7558-118.6.475"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00939-z"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000651"}],"container-title":["The Canadian Journal of Psychiatry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/07067437261445767","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/07067437261445767","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/07067437261445767","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T09:48:44Z","timestamp":1777888124000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/07067437261445767"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,30]]},"references-count":38,"alternative-id":["10.1177\/07067437261445767"],"URL":"https:\/\/doi.org\/10.1177\/07067437261445767","relation":{},"ISSN":["0706-7437","1497-0015"],"issn-type":[{"value":"0706-7437","type":"print"},{"value":"1497-0015","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,30]]},"article-number":"07067437261445767"}}