{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,11]],"date-time":"2026-11-11T15:29:04Z","timestamp":1794410944500,"version":"4.9.0"},"reference-count":73,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,6,9]],"date-time":"2026-06-09T00:00:00Z","timestamp":1780963200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,6,9]],"date-time":"2026-06-09T00:00:00Z","timestamp":1780963200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Medizinische Fakult\u00e4t Heidelberg der Universit\u00e4t Heidelberg"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Syst Rev"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Systematic reviews require reviewers to decide on the eligibility of large numbers of articles derived from database searches. To accelerate review conduct while continuously more literature gets published, past studies proposed automating the title\/abstract-screening step by either supervised machine learning or large language models. Because prior studies mainly compared results within the same model family, we directly compared common TF-IDF-based supervised baselines and a zero-shot, criteria-prompted, and open-weight large language model on the same data to discuss whether, and in which scenarios, they are feasible for review screening automation.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We predicted the eligibility of labeled articles by four supervised machine learning models (Na\u00efve Bayes, support vector machine, random forest, logistic regression) and one large language model (Llama-3.1\u20148B-Instruct). Articles were labeled with eligibility as decided by human reviewers in six systematic reviews. We evaluated the performance by binary confusion matrices and calculated recall, specificity, precision, F1-score, and accuracy over a thousand bootstrap samples each. We compared these results to a reported performance of 0.86 (recall) and 0.79 (specificity) in single human reviewers.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Model performance varies greatly between the data sets. Except for Na\u00efve Bayes, recall and specificity are closer aligned in the supervised machine learning models compared to llama. Averaged across all datasets, llama matches human recall and the Na\u00efve Bayes classifier exceeds it, while both fall behind human specificity. Conversely, logistic regression, random forest and support vector machine fall behind human recall while all three exceed human specificity.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Both supervised machine learning and large language models achieve recalls close to or above those of human reviewers. The supervised machine learning models achieve a higher harmonic mean of recall and specificity, while the llama model is more sensitive. Considering the reliance on training data and the all-or-nothing automation with supervised machine learning, this study\u2019s results warrant their use in the extension of pre-existing, non-critical, systematic reviews. Contrarily, as large language models decide on articles individually and as they provide comprehensive, discussable, reasoning they may be used in tandem with human reviewers while the performance of ensembles of large language models is yet to be analyzed.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s13643-026-03199-6","type":"journal-article","created":{"date-parts":[[2026,6,9]],"date-time":"2026-06-09T00:35:29Z","timestamp":1780965329000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Comparing supervised machine learning and large language models in title-abstract screening"],"prefix":"10.1186","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0433-4556","authenticated-orcid":false,"given":"Marco F.","family":"Aigner","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2716-5425","authenticated-orcid":false,"given":"Matthias","family":"Ganzinger","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0895-4015","authenticated-orcid":false,"given":"Pascal","family":"Probst","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Moritz","family":"Rinckens","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6145-3263","authenticated-orcid":false,"given":"Thomas M.","family":"Pausch","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,6,9]]},"reference":[{"issue":"7","key":"3199_CR1","doi-asserted-by":"publisher","first-page":"525","DOI":"10.7326\/0003-4819-156-7-201204030-00009","volume":"156","author":"A Qaseem","year":"2012","unstructured":"Qaseem A. Guidelines international network: toward international standards for clinical practice guidelines. Ann Intern Med. 2012;156(7):525. https:\/\/doi.org\/10.7326\/0003-4819-156-7-201204030-00009.","journal-title":"Ann Intern Med"},{"key":"3199_CR2","unstructured":"Cochrane AL. Effectiveness and efficiency: random reflections on health services. The Nuffield Provincial Hospitals Trust; 1972. Available from: https:\/\/www.nuffieldtrust.org.uk\/research\/effectiveness-and-efficiency-random-reflections-on-health-services. Cited 2024 Mar 27."},{"key":"3199_CR3","doi-asserted-by":"publisher","first-page":"n71","DOI":"10.1136\/bmj.n71","volume":"372","author":"MJ Page","year":"2021","unstructured":"Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. https:\/\/doi.org\/10.1136\/bmj.n71. PubMed PMID: 33782057.","journal-title":"BMJ"},{"key":"3199_CR4","doi-asserted-by":"publisher","first-page":"n160","DOI":"10.1136\/bmj.n160","volume":"372","author":"MJ Page","year":"2021","unstructured":"Page MJ, Moher D, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. 2021;372:n160. https:\/\/doi.org\/10.1136\/bmj.n160. PubMed PMID: 33781993.","journal-title":"BMJ"},{"issue":"6","key":"3199_CR5","doi-asserted-by":"publisher","first-page":"531","DOI":"10.1016\/j.jclinepi.2008.02.004","volume":"61","author":"M Sampson","year":"2008","unstructured":"Sampson M, Shojania KG, Garritty C, Horsley T, Ocampo M, Moher D. Systematic reviews can be produced and published faster. J Clin Epidemiol. 2008;61(6):531\u20136. https:\/\/doi.org\/10.1016\/j.jclinepi.2008.02.004.","journal-title":"J Clin Epidemiol"},{"issue":"2","key":"3199_CR6","doi-asserted-by":"publisher","first-page":"e012545","DOI":"10.1136\/bmjopen-2016-012545","volume":"7","author":"R Borah","year":"2017","unstructured":"Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO Registry. BMJ Open. 2017;7(2):e012545. https:\/\/doi.org\/10.1136\/bmjopen-2016-012545. PubMed PMID: 28242767.","journal-title":"BMJ Open"},{"issue":"4","key":"3199_CR7","doi-asserted-by":"publisher","first-page":"224","DOI":"10.7326\/0003-4819-147-4-200708210-00179","volume":"147","author":"KG Shojania","year":"2007","unstructured":"Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med. 2007;147(4):224\u201333. https:\/\/doi.org\/10.7326\/0003-4819-147-4-200708210-00179.","journal-title":"Ann Intern Med"},{"issue":"11","key":"3199_CR8","doi-asserted-by":"publisher","first-page":"2215","DOI":"10.1002\/asi.23329","volume":"66","author":"L Bornmann","year":"2015","unstructured":"Bornmann L, Mutz R. Growth rates of modern science: a bibliometric analysis based on the number of publications and cited references. J Assoc Inf Sci Technol. 2015;66(11):2215\u201322. https:\/\/doi.org\/10.1002\/asi.23329.","journal-title":"J Assoc Inf Sci Technol"},{"issue":"1","key":"3199_CR9","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1162\/qss_a_00177","volume":"3","author":"M Thelwall","year":"2022","unstructured":"Thelwall M, Sud P. Scopus 1900\u20132020: growth in articles, abstracts, countries, fields, and journals. Quant Sci Stud. 2022;3(1):37\u201350. https:\/\/doi.org\/10.1162\/qss_a_00177.","journal-title":"Quant Sci Stud"},{"issue":"2","key":"3199_CR10","doi-asserted-by":"publisher","first-page":"e1001603","DOI":"10.1371\/journal.pmed.1001603","volume":"11","author":"JH Elliott","year":"2014","unstructured":"Elliott JH, Turner T, Clavisi O, Thomas J, Higgins JPT, Mavergames C, et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med. 2014;11(2):e1001603. https:\/\/doi.org\/10.1371\/journal.pmed.1001603.","journal-title":"PLoS Med"},{"issue":"1","key":"3199_CR11","doi-asserted-by":"publisher","first-page":"132","DOI":"10.1186\/s12874-019-0782-0","volume":"19","author":"S Waffenschmidt","year":"2019","unstructured":"Waffenschmidt S, Knelangen M, Sieben W, B\u00fchn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med Res Methodol. 2019;19(1):132. https:\/\/doi.org\/10.1186\/s12874-019-0782-0.","journal-title":"BMC Med Res Methodol"},{"key":"3199_CR12","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1016\/j.jclinepi.2020.01.005","volume":"121","author":"G Gartlehner","year":"2020","unstructured":"Gartlehner G, Affengruber L, Titscher V, Noel-Storr A, Dooley G, Ballarini N, et al. Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial. J Clin Epidemiol. 2020;121:20\u20138. https:\/\/doi.org\/10.1016\/j.jclinepi.2020.01.005.","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"3199_CR13","doi-asserted-by":"publisher","first-page":"e0227742","DOI":"10.1371\/journal.pone.0227742","volume":"15","author":"Z Wang","year":"2020","unstructured":"Wang Z, Nayfeh T, Tetzlaff J, O\u2019Blenis P, Murad MH. Error rates of human reviewers during abstract screening in systematic reviews. PLoS ONE. 2020;15(1):e0227742. https:\/\/doi.org\/10.1371\/journal.pone.0227742.","journal-title":"PLoS ONE"},{"issue":"1","key":"3199_CR14","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1186\/2046-4053-4-5","volume":"4","author":"A O\u2019Mara-Eves","year":"2015","unstructured":"O\u2019Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4(1):5. https:\/\/doi.org\/10.1186\/2046-4053-4-5.","journal-title":"Syst Rev"},{"issue":"1","key":"3199_CR15","doi-asserted-by":"publisher","first-page":"e48996","DOI":"10.2196\/48996","volume":"26","author":"E Guo","year":"2024","unstructured":"Guo E, Gupta M, Deng J, Park YJ, Paget M, Naugler C. Automated paper screening for clinical reviews using large language models: data analysis study. J Med Internet Res. 2024;26(1):e48996. https:\/\/doi.org\/10.2196\/48996.","journal-title":"J Med Internet Res"},{"issue":"4","key":"3199_CR16","doi-asserted-by":"publisher","first-page":"616","DOI":"10.1002\/jrsm.1715","volume":"15","author":"Q Khraisha","year":"2024","unstructured":"Khraisha Q, Put S, Kappenberg J, Warraitch A, Hadfield K. Can large language models replace humans in systematic reviews? Evaluating GPT-4\u2019s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. Research Synthesis Methods. 2024;15(4):616\u201326. https:\/\/doi.org\/10.1002\/jrsm.1715.","journal-title":"Research Synthesis Methods"},{"issue":"1","key":"3199_CR17","doi-asserted-by":"publisher","first-page":"219","DOI":"10.1186\/s13643-024-02609-x","volume":"13","author":"M Li","year":"2024","unstructured":"Li M, Sun J, Tan X. Evaluating the effectiveness of large language models in abstract screening: a comparative analysis. Syst Rev. 2024;13(1):219. https:\/\/doi.org\/10.1186\/s13643-024-02609-x.","journal-title":"Syst Rev"},{"key":"3199_CR18","doi-asserted-by":"publisher","unstructured":"De Bruin J, Ma Y, Ferdinands G, Teijema J, Van de Schoot R. Synergy - open machine learning dataset on study selection in systematic reviews. DataverseNL; 2023. Available from:\u00a0https:\/\/dataverse.nl\/dataset.xhtml?persistentId=10.34894\/HE6NAQ. https:\/\/doi.org\/10.34894\/HE6NAQ. Cited 2024 May 27.","DOI":"10.34894\/HE6NAQ"},{"key":"3199_CR19","doi-asserted-by":"publisher","unstructured":"Priem J, Piwowar H, Orr R. OpenAlex: a fully-open index of scholarly works, authors, venues, institutions, and concepts. arXiv; 2022. Available from: http:\/\/arxiv.org\/abs\/2205.01833. https:\/\/doi.org\/10.48550\/arXiv.2205.01833. Cited 2024 May 27.","DOI":"10.48550\/arXiv.2205.01833"},{"key":"3199_CR20","unstructured":"De Bruin J. PyAlex. 2023. Available from: https:\/\/github.com\/J535D165\/pyalex. Cited 2023 Mar 3."},{"key":"3199_CR21","doi-asserted-by":"publisher","unstructured":"The Pandas Development Team. Pandas. Zenodo; 2020. Available from: https:\/\/doi.org\/10.5281\/zenodo.3509134.","DOI":"10.5281\/zenodo.3509134"},{"key":"3199_CR22","unstructured":"Polars BV. Polars [Rust]. Polars; 2024. Available from: https:\/\/github.com\/pola-rs\/polars. Cited 2024 Oct 16."},{"key":"3199_CR23","unstructured":"Yumiya233. fast-langdetect. 2024. Available from: https:\/\/pypi.org\/project\/fast-langdetect."},{"key":"3199_CR24","volume-title":"Natural language processing with Python","author":"S Bird","year":"2009","unstructured":"Bird S, Klein E, Loper E. Natural language processing with Python. Sebastopol, California: O\u2019Reilly Media; 2009."},{"key":"3199_CR25","unstructured":"Sayers E. Entrez Programming Utilities Help. 2022. A General Introduction to the E-Utilities. Available from: https:\/\/www.ncbi.nlm.nih.gov\/books\/NBK25497\/. Cited 2024 Oct 15."},{"key":"3199_CR26","unstructured":"Most N. Metapub. 2024. Available from: https:\/\/github.com\/metapub\/metapub."},{"key":"3199_CR27","unstructured":"Richardson L. beautifulsoup4: Screen-Scraping Library [Python]. 2024. Available from: https:\/\/www.crummy.com\/software\/BeautifulSoup\/bs4\/. Cited 2024 Aug 20."},{"issue":"4","key":"3199_CR28","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1136\/jamia.2010.004325","volume":"17","author":"S Matwin","year":"2010","unstructured":"Matwin S, Kouznetsov A, Inkpen D, Frunza O, O\u2019Blenis P. A new algorithm for reducing the workload of experts in performing systematic reviews. J Am Med Inform Assoc. 2010;17(4):4. https:\/\/doi.org\/10.1136\/jamia.2010.004325. PubMed PMID: 20595313; PubMed Central PMCID: PMC2995653.","journal-title":"J Am Med Inform Assoc"},{"key":"3199_CR29","doi-asserted-by":"publisher","unstructured":"Moreno\u2010Garc\u00eda CF, Chrisina Jayne, Eyad Elyan, Magaly Aceves\u2010Martins. A novel application of machine learning and zero-shot classification methods for automated abstract screening in systematic reviews. Decis Anal J. 2023;66:100162\u2013100162. https:\/\/doi.org\/10.1016\/j.dajour.2023.100162","DOI":"10.1016\/j.dajour.2023.10016"},{"issue":"5","key":"3199_CR30","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1197\/jamia.M3162","volume":"16","author":"AM Cohen","year":"2009","unstructured":"Cohen AM, Ambert K, McDonagh M. Cross-topic learning for work prioritization in systematic review creation and update. J Am Med Inform Assoc. 2009;16(5):5. https:\/\/doi.org\/10.1197\/jamia.M3162.","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"3199_CR31","doi-asserted-by":"publisher","first-page":"1","DOI":"10.4258\/hir.2012.18.1.18","volume":"18","author":"S Kim","year":"2012","unstructured":"Kim S, Choi J. Improving the performance of text categorization models used for the selection of high quality articles. Healthc Inform Res. 2012;18(1):1. https:\/\/doi.org\/10.4258\/hir.2012.18.1.18. PubMed PMID: 22509470; PubMed Central PMCID: PMC3324751.","journal-title":"Healthc Inform Res"},{"issue":"2","key":"3199_CR32","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1002\/jrsm.1589","volume":"14","author":"MM Kebede","year":"2023","unstructured":"Kebede MM, Le Cornet C, Fortner RT. In-depth evaluation of machine learning methods for semi-automating article screening in a systematic review of mechanistic literature. Res Synth Methods. 2023;14(2):2. https:\/\/doi.org\/10.1002\/jrsm.1589.","journal-title":"Res Synth Methods"},{"issue":"1","key":"3199_CR33","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1471-2105-11-55","volume":"11","author":"BC Wallace","year":"2010","unstructured":"Wallace BC, Trikalinos TA, Lau J, Brodley C, Schmid CH. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics. 2010;11(1):1. https:\/\/doi.org\/10.1186\/1471-2105-11-55.","journal-title":"BMC Bioinformatics"},{"issue":"2","key":"3199_CR34","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1197\/jamia.m1641","volume":"12","author":"Y Aphinyanaphongs","year":"2004","unstructured":"Aphinyanaphongs Y, Tsamardinos I, Statnikov A, Hardin D, Aliferis CF. Text categorization models for high-quality article retrieval in internal medicine. J Am Med Inform Assoc. 2004;12(2):2. https:\/\/doi.org\/10.1197\/jamia.m1641. PubMed PMID: 15561789.","journal-title":"J Am Med Inform Assoc"},{"issue":"Pt 1","key":"3199_CR35","first-page":"Pt 1","volume":"160","author":"T Bekhuis","year":"2010","unstructured":"Bekhuis T, Demner-Fushman D. Towards automating the initial screening phase of a systematic review. Stud Health Technol Inform. 2010;160(Pt 1):Pt 1. PubMed PMID: 20841667.","journal-title":"Stud Health Technol Inform"},{"issue":"12","key":"3199_CR36","first-page":"2825","volume":"1","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011;1(12):2825\u201330.","journal-title":"J Mach Learn Res"},{"key":"3199_CR37","unstructured":"HuggingFace. meta-llama\/Llama-3.1\u20138B-Instruct. 2024. Available from: https:\/\/huggingface.co\/meta-llama\/Llama-3.1-8B-Instruct. Cited 2024 Oct 31."},{"issue":"4","key":"3199_CR38","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1002\/jrsm.1715","volume":"15","author":"Q Khraisha","year":"2024","unstructured":"Khraisha Q, Put S, Kappenberg J, Warraitch A, Hadfield K. Can large language models replace humans in systematic reviews? Evaluating GPT-4\u2019s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. Res Synth Methods. 2024;15(4):4. https:\/\/doi.org\/10.1002\/jrsm.1715.","journal-title":"Res Synth Methods"},{"issue":"1","key":"3199_CR39","doi-asserted-by":"publisher","first-page":"1","DOI":"10.2196\/48996","volume":"26","author":"E Guo","year":"2024","unstructured":"Guo E, Gupta M, Deng J, Park YJ, Paget M, Naugler C. Automated paper screening for clinical reviews using large language models: data analysis study. J Med Internet Res. 2024;26(1):1. https:\/\/doi.org\/10.2196\/48996.","journal-title":"J Med Internet Res"},{"issue":"1","key":"3199_CR40","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13643-024-02609-x","volume":"13","author":"M Li","year":"2024","unstructured":"Li M, Sun J, Tan X. Evaluating the effectiveness of large language models in abstract screening: a comparative analysis. Syst Rev. 2024;13(1):1. https:\/\/doi.org\/10.1186\/s13643-024-02609-x.","journal-title":"Syst Rev"},{"key":"3199_CR41","unstructured":"Hugging Face. Auto Classes. 2024. Available from: https:\/\/huggingface.co\/docs\/transformers\/v4.48.2\/en\/model_doc\/auto. Cited 2025 Feb 5."},{"key":"3199_CR42","unstructured":"Hugging Face. 2024. Available from: https:\/\/huggingface.co\/. Cited 2024 Apr 26."},{"issue":"1","key":"3199_CR43","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1038\/s41591-024-03425-5","volume":"31","author":"J Gallifant","year":"2025","unstructured":"Gallifant J, Afshar M, Ameen S, Aphinyanaphongs Y, Chen S, Cacciamani G, et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nat Med. 2025;31(1):60\u20139. https:\/\/doi.org\/10.1038\/s41591-024-03425-5.","journal-title":"Nat Med"},{"key":"3199_CR44","doi-asserted-by":"publisher","unstructured":"Head T, Kumar M, Nahrstaedt H, Louppe G, Shcherbatyi I. scikit-optimize\/scikit-optimize. Zenodo; 2021. Available from: https:\/\/zenodo.org\/records\/5565057.\u00a0https:\/\/doi.org\/10.5281\/zenodo.5565057. Cited 2024 Oct 28.","DOI":"10.5281\/zenodo.5565057"},{"key":"3199_CR45","unstructured":"Hugging Face. Pipelines. 2024. Available from: https:\/\/huggingface.co\/docs\/transformers\/main_classes\/pipelines. Cited 2025 Feb 5."},{"key":"3199_CR46","doi-asserted-by":"publisher","first-page":"200193","DOI":"10.1016\/j.iswa.2023.200193","volume":"18","author":"W Kusa","year":"2023","unstructured":"Kusa W, Lipani A, Knoth P, Hanbury A. An analysis of work saved over sampling in the evaluation of automated citation screening in systematic literature reviews. Intell Syst Appl. 2023;18:200193. https:\/\/doi.org\/10.1016\/j.iswa.2023.200193.","journal-title":"Intell Syst Appl"},{"issue":"2","key":"3199_CR47","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1197\/jamia.M1929","volume":"13","author":"AM Cohen","year":"2006","unstructured":"Cohen AM, Hersh WR, Peterson K, Yen PY. Reducing workload in systematic review preparation using automated citation classification. J Am Med Inform Assoc. 2006;13(2):206\u201319. https:\/\/doi.org\/10.1197\/jamia.M1929. PubMed PMID: 16357352; PubMed Central PMCID: PMC1447545.","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"3199_CR48","doi-asserted-by":"publisher","first-page":"4961","DOI":"10.1007\/s10489-021-02635-5","volume":"52","author":"K Takahashi","year":"2022","unstructured":"Takahashi K, Yamamoto K, Kuchiba A, Koyama T. Confidence interval for micro-averaged F1 and macro-averaged F1 scores. Appl Intell. 2022;52(5):4961\u201372. https:\/\/doi.org\/10.1007\/s10489-021-02635-5.","journal-title":"Appl Intell"},{"key":"3199_CR49","unstructured":"Gildenblat J. A python library for confidence intervals. GitHub; 2023. Available from: https:\/\/github.com\/jacobgil\/confidenceinterval."},{"issue":"5","key":"3199_CR50","doi-asserted-by":"publisher","first-page":"1517","DOI":"10.1016\/j.surg.2021.04.023","volume":"170","author":"P Probst","year":"2021","unstructured":"Probst P, H\u00fcttner FJ, Meydan \u00d6, Abu Hilal M, Adham M, Barreto SG, et al. Evidence map of pancreatic surgery \u2013 a living systematic review with meta-analyses by the International Study Group of Pancreatic Surgery (ISGPS). Surgery. 2021;170(5):1517\u201324. https:\/\/doi.org\/10.1016\/j.surg.2021.04.023.","journal-title":"Surgery"},{"issue":"2","key":"3199_CR51","doi-asserted-by":"publisher","first-page":"e00024","DOI":"10.1002\/ebm2.24","volume":"3","author":"A Bannach-Brown","year":"2016","unstructured":"Bannach-Brown A, Liao J, Wegener G, Macleod M. Understanding in vivo modelling of depression in non-human animals: a systematic review protocol. Evid Based Preclin Med. 2016;3(2):e00024. https:\/\/doi.org\/10.1002\/ebm2.24.","journal-title":"Evid Based Preclin Med"},{"key":"3199_CR52","unstructured":"McDonagh MS, Peterson K, Dana T, Thakurta S. Drug class review on pharmacologic treatments for ADHD: Final Report. Portland (OR): Oregon Health & Science University; 2007. (Drug Class Reviews). Available from: http:\/\/www.ncbi.nlm.nih.gov\/books\/NBK10293\/. PubMed PMID: 20480923. Cited 2024 Jul 23."},{"key":"3199_CR53","unstructured":"McDonagh M, Peterson K, Carson S, Fu R, Thakurta S. Drug Class Review: Atypical Antipsychotic Drugs: Final Update 3 Report. Portland (OR): Oregon Health & Science University; 2010. (Drug Class Reviews). Available from: http:\/\/www.ncbi.nlm.nih.gov\/books\/NBK50583\/. PubMed PMID: 21348048. Cited 2024 Jul 23."},{"key":"3199_CR54","unstructured":"McDonagh MS, Eden KB, Peterson K. Drug Class Review: Calcium Channel Blockers: Final Report. Portland (OR): Oregon Health & Science University; 2005. (Drug Class Reviews). Available from: http:\/\/www.ncbi.nlm.nih.gov\/books\/NBK10474\/. PubMed PMID: 20496186. Cited 2024 Jul 23."},{"key":"3199_CR55","unstructured":"Chandler C, Chou R, Helfand M. Drug Class Review: Oral Hypoglycemics: Final Report. Portland, Oregon: Oregon Health & Science University; 2005. (Drug Class Reviews). Available from: http:\/\/www.ncbi.nlm.nih.gov\/books\/NBK10633\/. PubMed PMID: 20496449. Cited 2024 Jul 23."},{"issue":"4","key":"3199_CR56","doi-asserted-by":"publisher","first-page":"446","DOI":"10.1136\/jamia.2010.004325","volume":"17","author":"S Matwin","year":"2010","unstructured":"Matwin S, Kouznetsov A, Inkpen D, Frunza O, O\u2019Blenis P. A new algorithm for reducing the workload of experts in performing systematic reviews. J Am Med Inform Assoc. 2010;17(4):446\u201353. https:\/\/doi.org\/10.1136\/jamia.2010.004325. PubMed PMID: 20595313; PubMed Central PMCID: PMC2995653.","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"3199_CR57","doi-asserted-by":"publisher","first-page":"690","DOI":"10.1197\/jamia.M3162","volume":"16","author":"AM Cohen","year":"2009","unstructured":"Cohen AM, Ambert K, McDonagh M. Cross-topic learning for work prioritization in systematic review creation and update. J Am Med Inform Assoc. 2009;16(5):690\u2013704. https:\/\/doi.org\/10.1197\/jamia.M3162.","journal-title":"J Am Med Inform Assoc"},{"key":"3199_CR58","unstructured":"Sayers E. Entrez Programming Utilities Help. 2018. E-Utilities Quick Start. Available from: https:\/\/www.ncbi.nlm.nih.gov\/books\/NBK25500\/. Cited 2025 Jun 13."},{"issue":"2","key":"3199_CR59","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1002\/jrsm.42","volume":"2","author":"M Sampson","year":"2011","unstructured":"Sampson M, Tetzlaff J, Urquhart C. Precision of healthcare systematic review searches in a cross-sectional sample. Res Synth Methods. 2011;2(2):119\u201325. https:\/\/doi.org\/10.1002\/jrsm.42.","journal-title":"Res Synth Methods"},{"key":"3199_CR60","unstructured":"Mikolov T, Chen K, Corrado G, Dean J. Efficient estimation of word representations in vector space. arXiv; 2013. Available from: http:\/\/arxiv.org\/abs\/1301.3781. Cited 2024 May 30."},{"key":"3199_CR61","doi-asserted-by":"publisher","unstructured":"Pennington J, Socher R, Manning C. GloVe: Global Vectors for Word Representation. In: Moschitti A, Pang B, Daelemans W, editors. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for Computational Linguistics; 2014. p. 1532\u201343. Available from: https:\/\/aclanthology.org\/D14-1162. https:\/\/doi.org\/10.3115\/v1\/D14-1162. Cited 2024 Oct 28.","DOI":"10.3115\/v1\/D14-1162"},{"issue":"7","key":"3199_CR62","doi-asserted-by":"publisher","first-page":"2839","DOI":"10.1007\/s00521-020-05130-z","volume":"33","author":"E Elyan","year":"2021","unstructured":"Elyan E, Moreno-Garcia CF, Jayne C. CDSMOTE: Class decomposition and synthetic minority class oversampling technique for imbalanced-data classification. Neural Comput & Applic. 2021;33(7):2839\u201351. https:\/\/doi.org\/10.1007\/s00521-020-05130-z.","journal-title":"Neural Comput & Applic"},{"key":"3199_CR63","doi-asserted-by":"publisher","unstructured":"Lewis M, Liu Y, Goyal N, Ghazvininejad M, Mohamed A, Levy O, et al. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv; 2019. Available from: http:\/\/arxiv.org\/abs\/1910.13461. https:\/\/doi.org\/10.48550\/arXiv.1910.13461. Cited 2024 May 29.","DOI":"10.48550\/arXiv.1910.13461"},{"issue":"4, Part 1","key":"3199_CR64","doi-asserted-by":"publisher","first-page":"1498","DOI":"10.1016\/j.eswa.2013.08.047","volume":"41","author":"JJ Garc\u00eda Adeva","year":"2014","unstructured":"Garc\u00eda Adeva JJ, Pikatza Atxa JM, Ubeda Carrillo M, Ansuategi Zengotitabengoa E. Automatic text classification to support systematic reviews in medicine. Expert Syst Appl. 2014;41(4, Part 1):1498\u2013508. https:\/\/doi.org\/10.1016\/j.eswa.2013.08.047.","journal-title":"Expert Syst Appl"},{"issue":"3","key":"3199_CR65","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1016\/j.artmed.2012.05.002","volume":"55","author":"T Bekhuis","year":"2012","unstructured":"Bekhuis T, Demner-Fushman D. Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers. Artif Intell Med. 2012;55(3):197\u2013207. https:\/\/doi.org\/10.1016\/j.artmed.2012.05.002.","journal-title":"Artif Intell Med"},{"issue":"2","key":"3199_CR66","doi-asserted-by":"publisher","first-page":"156","DOI":"10.1002\/jrsm.1589","volume":"14","author":"MM Kebede","year":"2023","unstructured":"Kebede MM, Le Cornet C, Fortner RT. In-depth evaluation of machine learning methods for semi-automating article screening in a systematic review of mechanistic literature. Res Synth Methods. 2023;14(2):156\u201372. https:\/\/doi.org\/10.1002\/jrsm.1589.","journal-title":"Res Synth Methods"},{"issue":"1","key":"3199_CR67","doi-asserted-by":"publisher","first-page":"18","DOI":"10.4258\/hir.2012.18.1.18","volume":"18","author":"S Kim","year":"2012","unstructured":"Kim S, Choi J. Improving the performance of text categorization models used for the selection of high quality articles. Healthc Inform Res. 2012;18(1):18\u201328. https:\/\/doi.org\/10.4258\/hir.2012.18.1.18. PubMed PMID: 22509470; PubMed Central PMCID: PMC3324751.","journal-title":"Healthc Inform Res"},{"key":"3199_CR68","unstructured":"PubMed. Foundations of MeSH in MEDLINE. 2024. Available from: https:\/\/www.nlm.nih.gov\/oet\/ed\/pubmed\/mesh\/mod01\/01-000.html. Cited 2024 Oct 31."},{"issue":"1","key":"3199_CR69","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1109\/TTS.2021.3118355","volume":"3","author":"GL Liehner","year":"2022","unstructured":"Liehner GL, Brauner P, Schaar AK, Ziefle M. Delegation of moral tasks to automated agents\u2014the impact of risk and context on trusting a machine to perform a task. IEEE Trans Technol Soc. 2022;3(1):46\u201357. https:\/\/doi.org\/10.1109\/TTS.2021.3118355.","journal-title":"IEEE Trans Technol Soc"},{"issue":"2","key":"3199_CR70","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1038\/s42256-020-00287-7","volume":"3","author":"R van de Schoot","year":"2021","unstructured":"van de Schoot R, de Bruin J, Schram R, Zahedi P, de Boer J, Weijdema F, et al. An open source machine learning framework for efficient and transparent systematic reviews. Nat Mach Intell. 2021;3(2):125\u201333. https:\/\/doi.org\/10.1038\/s42256-020-00287-7.","journal-title":"Nat Mach Intell"},{"key":"3199_CR71","unstructured":"ASReview. ASReview - Active learning for Systematic Reviews. Available from: https:\/\/asreview.nl\/. Cited 2026 Mar 29."},{"issue":"9","key":"3199_CR72","doi-asserted-by":"publisher","first-page":"e032353","DOI":"10.1136\/bmjopen-2019-032353","volume":"9","author":"P Probst","year":"2019","unstructured":"Probst P, H\u00fcttner FJ, Meydan \u00d6, Kalkum E, Kretschmer R, Jensen K, et al. Evidence map of pancreatic surgery: protocol for a living systematic review and meta-analysis. BMJ Open. 2019;9(9):e032353. https:\/\/doi.org\/10.1136\/bmjopen-2019-032353.","journal-title":"BMJ Open"},{"issue":"1","key":"3199_CR73","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1186\/s13643-019-0942-7","volume":"8","author":"A Bannach-Brown","year":"2019","unstructured":"Bannach-Brown A, Przyby\u0142a P, Thomas J, Rice ASC, Ananiadou S, Liao J, et al. Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error. Syst Rev. 2019;8(1):23. https:\/\/doi.org\/10.1186\/s13643-019-0942-7.","journal-title":"Syst Rev"}],"container-title":["Systematic Reviews"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13643-026-03199-6","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13643-026-03199-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13643-026-03199-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,9]],"date-time":"2026-06-09T16:03:21Z","timestamp":1781021001000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1186\/s13643-026-03199-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,9]]},"references-count":73,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["3199"],"URL":"https:\/\/doi.org\/10.1186\/s13643-026-03199-6","relation":{},"ISSN":["2046-4053"],"issn-type":[{"value":"2046-4053","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,9]]},"assertion":[{"value":"28 September 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 May 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 June 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"EVIglance Inc. may be affected by the research reported in this article. The authors disclose that the dataset used for the study of pancreatic surgery was provided by EVIglance Inc. PP discloses that he is board chairman of EVIglance Inc. MR discloses that he is employed at EVIglance Inc. MFA, MG, and TMP have no further competing interests to disclose.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"190"}}