{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,13]],"date-time":"2026-11-13T18:40:17Z","timestamp":1794595217042,"version":"4.9.1"},"reference-count":26,"publisher":"BMJ","issue":"1","license":[{"start":{"date-parts":[[2025,7,23]],"date-time":"2025-07-23T00:00:00Z","timestamp":1753228800000},"content-version":"unspecified","delay-in-days":22,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100010269","name":"Wellcome Trust","doi-asserted-by":"publisher","award":["N\/A"],"award-info":[{"award-number":["N\/A"]}],"id":[{"id":"10.13039\/100010269","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["BMJ Ment Health"],"accepted":{"date-parts":[[2025,7,5]]},"published-print":{"date-parts":[[2025,7]]},"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Living systematic reviews (LSRs) maintain an updated summary of evidence by incorporating newly published research. While they improve review currency, repeated screening and selection of new references make them labourious and difficult to maintain. Large language models (LLMs) show promise in assisting with screening and data extraction, but more work is needed to achieve the high accuracy required for evidence that informs clinical and policy decisions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>The study evaluated the effectiveness of an LLM (GPT-4o) in title and abstract screening compared with human reviewers.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Human decisions from an LSR on prodopaminergic interventions for anhedonia served as the reference standard. The baseline search results were divided into a development and a test set. Prompts guiding the LLM\u2019s eligibility assessments were refined using the development set and evaluated on the test set and two subsequent LSR updates. Consistency of the LLM outputs was also assessed.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Prompt development required 1045 records. When applied to the remaining baseline 11\u2009939 records and two updates, the refined prompts achieved 100% sensitivity for studies ultimately included in the review after full-text screening, though sensitivity for records included by humans at the title and abstract stage varied (58\u2013100%) across updates. Simulated workload reductions of 65\u201385% were observed. Prompt decisions showed high consistency, with minimal false exclusions, satisfying established screening performance benchmarks for systematic reviews.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Refined GPT-4o prompts demonstrated high sensitivity and moderate specificity while reducing human workload. This approach shows potential for integrating LLMs into systematic review workflows to enhance efficiency.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1136\/bmjment-2025-301762","type":"journal-article","created":{"date-parts":[[2025,7,23]],"date-time":"2025-07-23T04:05:22Z","timestamp":1753243522000},"page":"e301762","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":32,"title":["Development and evaluation of prompts for a large language model to screen titles and abstracts in a living systematic review"],"prefix":"10.1136","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5772-2475","authenticated-orcid":false,"given":"Ava","family":"Homiar","sequence":"first","affiliation":[{"name":"Department of Psychiatry, University of Oxford, Oxford, UK"},{"name":"Division of Clinical Informatics, Beth Israel Deaconess Medical Center, Boston, Massachusetts, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James","family":"Thomas","sequence":"additional","affiliation":[{"name":"University College London, London, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8717-0832","authenticated-orcid":false,"given":"Edoardo G","family":"Ostinelli","sequence":"additional","affiliation":[{"name":"Department of Psychiatry, University of Oxford, Oxford, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7312-8676","authenticated-orcid":false,"given":"Jaycee","family":"Kennett","sequence":"additional","affiliation":[{"name":"Department of Psychiatry, University of Oxford, Oxford, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Claire","family":"Friedrich","sequence":"additional","affiliation":[{"name":"Department of Psychiatry, University of Oxford, Oxford, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5497-2743","authenticated-orcid":false,"given":"Pim","family":"Cuijpers","sequence":"additional","affiliation":[{"name":"Vrije Universiteit Amsterdam, Amsterdam, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7016-2687","authenticated-orcid":false,"given":"Mathias","family":"Harrer","sequence":"additional","affiliation":[{"name":"Psychology & Digital Mental Health Care, Technical University of Munich, Munchen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4934-4352","authenticated-orcid":false,"given":"Stefan","family":"Leucht","sequence":"additional","affiliation":[{"name":"Psychiatry and Psychotherapy, Technical University of Munich School of Medicine, Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5563-5896","authenticated-orcid":false,"given":"Clara","family":"Miguel","sequence":"additional","affiliation":[{"name":"Department of Clinical, Neuro and Developmental Psychology, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alessandro","family":"Rodolico","sequence":"additional","affiliation":[{"name":"Technical University of Munich, Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7982-5213","authenticated-orcid":false,"given":"Yuki","family":"Kataoka","sequence":"additional","affiliation":[{"name":"Kyoto Min-iren Asukai Hospital, Kyoto, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tomohiro","family":"Takayama","sequence":"additional","affiliation":[{"name":"Kyoto University, Kyoto, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-3985-2197","authenticated-orcid":false,"given":"Keisuke","family":"Yoshimura","sequence":"additional","affiliation":[{"name":"Kyoto University, Kyoto, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ryuhei","family":"So","sequence":"additional","affiliation":[{"name":"Okayama Psychiatric Medical Center, Okayama, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yasushi","family":"Tsujimoto","sequence":"additional","affiliation":[{"name":"Kyoto University, Kyoto, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-7688-3075","authenticated-orcid":false,"given":"Yosuke","family":"Yamagishi","sequence":"additional","affiliation":[{"name":"The University of Tokyo, Bunkyo, Tokyo, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shiro","family":"Takagi","sequence":"additional","affiliation":[{"name":"Independent Researcher, Kyoto, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Masatsugu","family":"Sakata","sequence":"additional","affiliation":[{"name":"Nagoya City University Graduate School of Medical Sciences and Medical School, Nagoya, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"\u0110or\u0111e","family":"Ba\u0161i\u0107","sequence":"additional","affiliation":[{"name":"Vrije Universiteit Amsterdam, Amsterdam, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0071-2599","authenticated-orcid":false,"given":"Eirini","family":"Karyotaki","sequence":"additional","affiliation":[{"name":"Department of Clinical, Neuro and Developmental Psychology, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-8846-9159","authenticated-orcid":false,"given":"Jennifer","family":"Potts","sequence":"additional","affiliation":[{"name":"NIHR Oxford Health Biomedical Research Centre, Oxford, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Georgia","family":"Salanti","sequence":"additional","affiliation":[{"name":"University of Bern, Bern, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2159-3776","authenticated-orcid":false,"given":"Toshi A","family":"Furukawa","sequence":"additional","affiliation":[{"name":"Kyoto University Graduate School of Medicine Faculty of Medicine, Kyoto, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5179-8321","authenticated-orcid":false,"given":"Andrea","family":"Cipriani","sequence":"additional","affiliation":[{"name":"Department of Psychiatry, University of Oxford, Oxford, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2025,7,22]]},"reference":[{"key":"2025082701124395000_28.1.e301762.1","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pmed.1001603","article-title":"Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap","volume":"11","author":"Elliott","year":"2014","journal-title":"PLoS Med"},{"key":"2025082701124395000_28.1.e301762.2","doi-asserted-by":"publisher","DOI":"10.7326\/0003-4819-151-4-200908180-00135"},{"key":"2025082701124395000_28.1.e301762.3","doi-asserted-by":"crossref","DOI":"10.1136\/bmjopen-2016-012545","article-title":"Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry","volume":"7","author":"Borah","year":"2017","journal-title":"BMJ Open"},{"key":"2025082701124395000_28.1.e301762.4","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1007\/978-1-0716-1566-9_7","article-title":"Living Systematic Reviews","volume":"2345","author":"Simmonds","year":"2022","journal-title":"Methods Mol Biol"},{"key":"2025082701124395000_28.1.e301762.5","doi-asserted-by":"crossref","DOI":"10.2196\/33219","article-title":"Web-Based Software Tools for Systematic Literature Review in Medicine: Systematic Search and Feature Analysis","volume":"10","author":"Cowie","year":"2022","journal-title":"JMIR Med Inform"},{"key":"2025082701124395000_28.1.e301762.6","doi-asserted-by":"publisher","DOI":"10.1016\/j.lindif.2023.102274"},{"key":"2025082701124395000_28.1.e301762.7","article-title":"Streamlining Systematic Reviews: Harnessing Large Language Models for Quality Assessment and Risk-of-Bias Evaluation","volume":"15","author":"Nashwan","year":"2023","journal-title":"Cureus"},{"key":"2025082701124395000_28.1.e301762.8","doi-asserted-by":"crossref","DOI":"10.2196\/48996","article-title":"Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study","volume":"26","author":"Guo","year":"2024","journal-title":"J Med Internet Res"},{"key":"2025082701124395000_28.1.e301762.9","doi-asserted-by":"crossref","unstructured":"Krag CH , Balschmidt T , Bruun F , et al . Large language models for abstract screening in systematic- and scoping reviews: a diagnostic test accuracy study. Radiol Imaging 2024. doi:10.1101\/2024.10.01.24314702","DOI":"10.1101\/2024.10.01.24314702"},{"key":"2025082701124395000_28.1.e301762.10","doi-asserted-by":"crossref","DOI":"10.1186\/s12874-024-02203-8","article-title":"Methodological insights into ChatGPT\u2019s screening performance in systematic reviews","volume":"24","author":"Issaiy","year":"2024","journal-title":"BMC Med Res Methodol"},{"key":"2025082701124395000_28.1.e301762.11","doi-asserted-by":"crossref","unstructured":"Tran V-T , Gartlehner G , Yaacoub S , et al . Sensitivity, specificity and avoidable workload of using a large language models for title and abstract screening in systematic reviews and meta-analyses. Epidemiology [Preprint] 2023. doi:10.1101\/2023.12.15.23300018","DOI":"10.1101\/2023.12.15.23300018"},{"key":"2025082701124395000_28.1.e301762.12","unstructured":"Thomas J , Graziosi S , Brunton J , et al . EPPI-reviewer: advanced software for systematic reviews, maps and evidence synthesis. EPPI-Centre; 2020."},{"key":"2025082701124395000_28.1.e301762.13","doi-asserted-by":"crossref","DOI":"10.12688\/wellcomeopenres.19870.1","article-title":"Pro-dopaminergic pharmacological interventions for anhedonia in depression: protocol for a living systematic review of human and non-human studies","volume":"8","author":"Ostinelli","year":"2023","journal-title":"Wellcome Open Res"},{"key":"2025082701124395000_28.1.e301762.14","doi-asserted-by":"publisher","DOI":"10.1038\/s41415-023-6132-y"},{"key":"2025082701124395000_28.1.e301762.15","doi-asserted-by":"publisher","DOI":"10.1136\/bmjebm-2023-112597"},{"key":"2025082701124395000_28.1.e301762.16","doi-asserted-by":"crossref","first-page":"389","DOI":"10.7326\/ANNALS-24-02189","article-title":"Development of Prompt Templates for Large Language Model-Driven Screening in Systematic Reviews","volume":"178","author":"Cao","year":"2025","journal-title":"Ann Intern Med"},{"key":"2025082701124395000_28.1.e301762.17","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2021.05.019"},{"key":"2025082701124395000_28.1.e301762.18","unstructured":"Kramer O , Baumann J . Unlocking structured thinking in language models with cognitive prompting. arXiv 2024. Available: http:\/\/arxiv.org\/abs\/2410.02953"},{"key":"2025082701124395000_28.1.e301762.19","unstructured":"Llama. Llama. 2025 Available: https:\/\/www.google.com\/url?q=https:\/\/www.llama.com\/&sa=D&source=docs&ust=1748876702915084&usg=AOvVaw210_5kgKXQNtxBjyW4yHYe"},{"key":"2025082701124395000_28.1.e301762.20","unstructured":"DeepSeek, Available: https:\/\/www.deepseek.com\/"},{"key":"2025082701124395000_28.1.e301762.21","doi-asserted-by":"crossref","first-page":"657","DOI":"10.12688\/wellcomeopenres.23278.1","article-title":"Cognitive bias modification for social anxiety: protocol for a living systematic review of human studies and meta-analysis","volume":"9","author":"Kennett","year":"2024","journal-title":"Wellcome Open Res"},{"key":"2025082701124395000_28.1.e301762.22","unstructured":"Schmidt L , Hair K , Graziozi S , et al . Exploring the use of a Large Language Model for data extraction in systematic reviews: a rapid feasibility study. arXiv 2024. Available: http:\/\/arxiv.org\/abs\/2405.14445"},{"key":"2025082701124395000_28.1.e301762.23","doi-asserted-by":"publisher","DOI":"10.1002\/jrsm.1710"},{"key":"2025082701124395000_28.1.e301762.24","doi-asserted-by":"crossref","unstructured":"Gartlehner G , Kahwati L , Nussbaumer-Streit B , et al . From promise to practice: challenges and pitfalls in the evaluation of large language models for data extraction in evidence synthesis. BMJ EBM 2024. doi:10.1136\/bmjebm-2024-113199","DOI":"10.1136\/bmjebm-2024-113199"},{"key":"2025082701124395000_28.1.e301762.25","doi-asserted-by":"crossref","DOI":"10.1136\/bmjment-2023-300759","article-title":"New living evidence resource of human and non-human studies for early intervention and research prioritisation in anxiety, depression and psychosis","volume":"26","author":"Cipriani","year":"2023","journal-title":"BMJ Ment Health"},{"key":"2025082701124395000_28.1.e301762.26","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1192\/bjp.2024.237","article-title":"Triangulating evidence from the GALENOS living systematic review on trace amine-associated receptor 1 (TAAR1) agonists in psychosis","volume":"226","author":"Smith","year":"2025","journal-title":"Br J Psychiatry"}],"container-title":["BMJ Mental Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bmjment-2025-301762","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,27]],"date-time":"2025-08-27T08:12:56Z","timestamp":1756282376000},"score":1,"resource":{"primary":{"URL":"https:\/\/mentalhealth.bmj.com\/lookup\/doi\/10.1136\/bmjment-2025-301762"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7]]},"references-count":26,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,7,22]]},"published-print":{"date-parts":[[2025,7]]}},"alternative-id":["10.1136\/bmjment-2025-301762"],"URL":"https:\/\/doi.org\/10.1136\/bmjment-2025-301762","relation":{},"ISSN":["2755-9734"],"issn-type":[{"value":"2755-9734","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7]]}}}