{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:44:22Z","timestamp":1785545062707,"version":"3.56.0"},"reference-count":30,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec sec-type=\"background\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Artificial intelligence has gained relevance due to its potential to reduce the workload in evidence synthesis or bibliometric projects. While the main focus has been lately on the use of instruction-tuned large language models, zero-shot classification models have not been tested for such task. These models are large language models trained on large datasets of labeled data able to categorize text among proposed labels, irrespective of the text domain or the topic. They are relatively small, able to run on consumer-grade computers, and almost hyperparameter-free.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"objective\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>In our study, we use abstracts of randomized clinical trials in rheumatology as a case example to evaluate the performance of openly available, generalist, zero-shot classification models in classifying types of interventions against a human gold standard.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"methods\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We classified all rheumatology RCT abstracts published between 2009 and 2022 (n=1,054) as \u201cdrug\u201d or \u201cnon-drug\u201d using two zero-shot text classification models (DeBERTa and BART) and few-shot prompting using Llama3 8B. Different labeling of categories provided to the zero-shot classification models and different prompts provided to Llama3 8B were tested. Performance was evaluated using accuracy and predictive value of both categories against a human gold standard.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"results\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Most randomized controlled trials, RCTs (452\/1054, 42.9%) assessed drug interventions. The DeBERTa model achieved the highest accuracy (929\/1054, 88.1%; 95% CI 86%\u201090%) when using the \u201cdrug\u201d and \u201cnon-drug\u201d labels. Llama3 8B and few-shot prompting had slightly higher accuracy and predictive values. Both zero-shot and Llama3 8B models had performance on par with a human without experience in evidence synthesis (905\/1054, 85.9%; 95% CI 83.6%\u201087.8% accuracy). Misclassifications occurred for trials where the intervention was harder to classify, such as procedures (eg, intra-articular injections), food compounds, vitamins, supplements, or biological treatments.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"conclusions\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>This study shows the potential of zero-shot classification models for simple classification tasks, demonstrating accuracy comparable to that of an untrained human. These models are potential tools to streamline systematic review tasks for bibliometric studies in classifying abstracts by supplementing one reviewer.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/77943","type":"journal-article","created":{"date-parts":[[2026,5,5]],"date-time":"2026-05-05T15:35:07Z","timestamp":1777995307000},"page":"e77943-e77943","source":"Crossref","is-referenced-by-count":0,"title":["Performance of Zero-Shot Classifiers for Categorizing RCT Abstracts by Intervention Type: Validation Study"],"prefix":"10.2196","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9761-206X","authenticated-orcid":false,"given":"Diana","family":"Buitrago-Garcia","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1956-2607","authenticated-orcid":false,"given":"Delphine S","family":"Courvoisier","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sami","family":"Capderou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5871-8806","authenticated-orcid":false,"given":"Michele","family":"Iudici","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4801-8395","authenticated-orcid":false,"given":"Denis","family":"Mongin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,6,18]]},"reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbusres.2021.04.070","article-title":"How to conduct a bibliometric analysis: an overview and guidelines","volume":"133","author":"Donthu","journal-title":"J Bus Res"},{"issue":"5","key":"R2","doi-asserted-by":"publisher","first-page":"539","DOI":"10.1038\/s41562-021-01131-7","article-title":"The value of evidence synthesis","volume":"5","journal-title":"Nat Hum Behav"},{"key":"R3","unstructured":"Methodological Expectations of Cochrane Intervention Reviews (MECIR) Standards for the conduct and reporting of | Policy Commons. URL: https:\/\/policycommons.net\/artifacts\/1712743\/methodological-expectations-of-cochrane-intervention-reviews-mecir-standards-for-the-conduct-and-reporting-of\/2444392\/ [Accessed 28-08-2024]"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2020.01.005","article-title":"Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial","volume":"121","author":"Gartlehner","journal-title":"J Clin Epidemiol"},{"issue":"2","key":"R5","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2016-012545","article-title":"Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry","volume":"7","author":"Borah","journal-title":"BMJ Open"},{"issue":"1","key":"R6","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-024-02320-4","article-title":"An exploration of available methods and tools to improve the efficiency of systematic review production: a scoping review","volume":"24","author":"Affengruber","journal-title":"BMC Med Res Methodol"},{"issue":"2","key":"R7","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1197\/jamia.M1929","article-title":"Reducing Workload in Systematic Review Preparation Using Automated Citation Classification","volume":"13","author":"Cohen","journal-title":"J Am Med Inform Assoc"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.2196\/48996","article-title":"Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study","volume":"26","author":"Guo","journal-title":"J Med Internet Res"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-021-01763-w","article-title":"srBERT: automatic article classification model for systematic review using BERT","volume":"10","author":"Aum","journal-title":"Syst Rev"},{"key":"R10","doi-asserted-by":"crossref","unstructured":"Yin W Hay J Roth D . Benchmarking zero-shot text classification: datasets, evaluation and entailment approach. arXiv. Preprint posted online on Aug 31, 2019. [doi: 10.48550\/arXiv.1909.00161]","DOI":"10.18653\/v1\/D19-1404"},{"key":"R11","doi-asserted-by":"crossref","unstructured":"Moreno-Garc\u00eda CF Jayne C Elyan E Aceves-Martins M . Abstract screening for systematic reviews using machine learning and zero-shot classification. SSRN. Preprint posted online on Sep 12, 2022. [doi: 10.2139\/ssrn.4210704]","DOI":"10.2139\/ssrn.4210704"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2024.111586","article-title":"Prospective registration of trials: where we are, why, and how we could get better","volume":"176","author":"Mongin","journal-title":"J Clin Epidemiol"},{"key":"R13","unstructured":"Gelijns AC Institute of Medicine (US) Committee on Technological Innovation in Medicine. 4. the development of clinical procedures. In: Technological Innovation: Comparing Development of Drugs, Devices, and Procedures in Medicine. 1986. URL: https:\/\/www.ncbi.nlm.nih.gov\/books\/NBK222716 [Accessed 08-01-2025]"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Ninot G . Defining non-pharmacological interventions (npis). In: Ninot G , editor. Non-Pharmacological Interventions: An Essential Answer to Current Demographic, Health, and Environmental Transitions. Springer International Publishing; 2021:1-46. ISBN: 978-3-030-60971-9","DOI":"10.1007\/978-3-030-60971-9_1"},{"key":"R15","unstructured":"Zero-shot classification models. HuggingFace. 2023. URL: https:\/\/huggingface.co\/models [Accessed 10-11-2025]"},{"key":"R16","unstructured":"He P Liu X Gao J Chen W . DeBERTa: decoding-enhanced BERT with disentangled attention. arXiv. Preprint posted online on 2021. URL: http:\/\/arxiv.org\/abs\/2006.03654"},{"key":"R17","unstructured":"MoritzLaurer\/deberta-v3-base-mnli-fever-anli. HuggingFace. 2023. URL: https:\/\/huggingface.co\/MoritzLaurer\/DeBERTa-v3-base-mnli-fever-anli [Accessed 10-11-2025]"},{"key":"R18","unstructured":"Bart-large model page. HuggingFace. 2024. URL: https:\/\/huggingface.co\/facebook\/bart-large-mnli [Accessed 22-05-2025]"},{"key":"R19","doi-asserted-by":"crossref","unstructured":"Lewis M Liu Y Goyal N et al. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv. Preprint posted online on 2019. [doi: 10.48550\/arXiv.1910.13461]","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"R20","unstructured":"AI@Meta. Llama 3 model card. github. 2024. URL: https:\/\/github.com\/meta-llama\/llama3\/blob\/main\/MODEL_CARD.md"},{"key":"R21","unstructured":"Sahoo P Singh AK Saha S Jain V Mondal S Chadha A . A systematic survey of prompt engineering in large language models: techniques and applications. arXiv. Preprint posted online on Feb 5, 2024. [doi: 10.48550\/arXiv.2402.07927]"},{"issue":"4","key":"R22","doi-asserted-by":"publisher","DOI":"10.1136\/rmdopen-2024-005003","article-title":"Large language models for accurate disease detection in electronic health records: the examples of crystal arthropathies","volume":"10","author":"B\u00fcrgisser","journal-title":"RMD Open"},{"key":"R23","unstructured":"Gitlab repository. GitLab. 2024. URL: https:\/\/gitlab-b.unige.ch\/trial_integrity\/ai_intervention_categorisation_public [Accessed 22-05-2025]"},{"issue":"158","key":"R24","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1080\/01621459.1927.10502953","article-title":"Probable Inference, the Law of Succession, and Statistical Inference","volume":"22","author":"Wilson","journal-title":"J Am Stat Assoc"},{"key":"R25","unstructured":"R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing; 2019. URL: https:\/\/www.R-project.org"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.1200\/CCI-24-00150","article-title":"Metastatic versus localized disease as inclusion criteria that can be automatically extracted from randomized controlled trials using natural language processing","author":"Windisch","journal-title":"JCO Clin Cancer Inform"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.1016\/j.cola.2024.101287","article-title":"Screening articles for systematic reviews with ChatGPT","volume":"80","author":"Syriani","journal-title":"Journal of Computer Languages"},{"issue":"2","key":"R28","doi-asserted-by":"publisher","first-page":"365","DOI":"10.1017\/rsm.2025.10054","article-title":"Beyond human gold standards: A multimodel framework for automated abstract classification and information extraction","volume":"17","author":"Courvoisier","journal-title":"Res Synth Methods"},{"issue":"1","key":"R29","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-023-02243-z","article-title":"Are ChatGPT and large language models \u201cthe answer\u201d to bringing us closer to systematic review automation?","volume":"12","author":"Qureshi","journal-title":"Syst Rev"},{"issue":"1","key":"R30","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0227742","article-title":"Error rates of human reviewers during abstract screening in systematic reviews","volume":"15","author":"Wang","journal-title":"PLoS ONE"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e77943","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:26:55Z","timestamp":1785536815000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e77943"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,18]]},"references-count":30,"URL":"https:\/\/doi.org\/10.2196\/77943","relation":{},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,18]]},"article-number":"v14i5e77943"}}