{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,11]],"date-time":"2026-11-11T06:46:20Z","timestamp":1794379580559,"version":"4.9.0"},"reference-count":37,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,3,27]],"date-time":"2024-03-27T00:00:00Z","timestamp":1711497600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,3,27]],"date-time":"2024-03-27T00:00:00Z","timestamp":1711497600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Res Methodol"],"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>The screening process for systematic reviews and meta-analyses in medical research is a labor-intensive and time-consuming task. While machine learning and deep learning have been applied to facilitate this process, these methods often require training data and user annotation. This study aims to assess the efficacy of ChatGPT, a large language model based on the Generative Pretrained Transformers (GPT) architecture, in automating the screening process for systematic reviews in radiology without the need for training data.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>A prospective simulation study was conducted between May 2nd and 24th, 2023, comparing ChatGPT\u2019s performance in screening abstracts against that of general physicians (GPs). A total of 1198 abstracts across three subfields of radiology were evaluated. Metrics such as sensitivity, specificity, positive and negative predictive values (PPV and NPV), workload saving, and others were employed. Statistical analyses included the Kappa coefficient for inter-rater agreement, ROC curve plotting, AUC calculation, and bootstrapping for p-values and confidence intervals.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>ChatGPT completed the screening process within an hour, while GPs took an average of 7\u201310 days. The AI model achieved a sensitivity of 95% and an NPV of 99%, slightly outperforming the GPs\u2019 sensitive consensus (i.e., including records if at least one person includes them). It also exhibited remarkably low false negative counts and high workload savings, ranging from 40 to 83%. However, ChatGPT had lower specificity and PPV compared to human raters. The average Kappa agreement between ChatGPT and other raters was 0.27.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>ChatGPT shows promise in automating the article screening phase of systematic reviews, achieving high sensitivity and workload savings. While not entirely replacing human expertise, it could serve as an efficient first-line screening tool, particularly in reducing the burden on human resources. Further studies are needed to fine-tune its capabilities and validate its utility across different medical subfields.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12874-024-02203-8","type":"journal-article","created":{"date-parts":[[2024,3,27]],"date-time":"2024-03-27T14:04:59Z","timestamp":1711548299000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":92,"title":["Methodological insights into ChatGPT\u2019s screening performance in systematic reviews"],"prefix":"10.1186","volume":"24","author":[{"given":"Mahbod","family":"Issaiy","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hossein","family":"Ghanaati","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shahriar","family":"Kolahi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Madjid","family":"Shakiba","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amir Hossein","family":"Jalali","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Diana","family":"Zarei","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sina","family":"Kazemian","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mahsa Alborzi","family":"Avanaki","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kavous","family":"Firouznia","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,3,27]]},"reference":[{"issue":"7553","key":"2203_CR1","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521(7553):436\u201344.","journal-title":"Nature"},{"key":"2203_CR2","unstructured":"Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate. arXiv pre-print server. 2016."},{"key":"2203_CR3","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention is all you need. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R, eds. 2017."},{"key":"2203_CR4","unstructured":"Devlin J, Chang M-W, Lee K, Toutanova K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:181004805. 2018."},{"key":"2203_CR5","unstructured":"Alec R, Karthik N. Improving language understanding by generative pre-training. 2018."},{"key":"2203_CR6","unstructured":"Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res. 2020;21(1)."},{"key":"2203_CR7","unstructured":"Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, et al. Language Models are few-shot learners. In: Larochelle H, Ranzato M, Hadsell R, Balcan MF, Lin H, eds. 2020. p. 1877\u2013901."},{"key":"2203_CR8","unstructured":"Alec R, Jeff W, Rewon C, David L, Dario A, Ilya S. Language models are unsupervised multitask learners. 2019."},{"key":"2203_CR9","unstructured":"Bubeck S, Chandrasekaran V, Eldan R, Gehrke J, Horvitz E, Kamar E, et al. Sparks of Artificial General Intelligence: Early experiments with GPT-4. 2023."},{"key":"2203_CR10","unstructured":"Alessandro L, Douglas GA, Jennifer Marie T, Cynthia DM, Peter Christian G, John PAI, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6."},{"key":"2203_CR11","unstructured":"Byron CW, Kevin S, Carla EB, Joseph L, Thomas AT. Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. 2012."},{"key":"2203_CR12","doi-asserted-by":"crossref","unstructured":"Ana Helena Salles dos R, Ana Luiza Miranda de O, Carolina F, James Z, Paulo F, Janaine Cunha P. Usefulness of machine learning softwares to screen titles of systematic reviews: a methodological study. Syst Rev. 2023;12.","DOI":"10.1186\/s13643-023-02231-3"},{"key":"2203_CR13","doi-asserted-by":"crossref","unstructured":"Kevin EKC, Robin LJL, Daniel FG, Leo N. Research Screener: a machine learning tool to semi-automate abstract screening for systematic reviews. Syst Rev. 2021;10.","DOI":"10.1186\/s13643-021-01635-3"},{"key":"2203_CR14","doi-asserted-by":"crossref","unstructured":"Amir V, Mana M, Amin N-A, Seyed Hossein Hosseini A, Mehrnush Saghab T, Reyhaneh A, et al. Abstract screening using the automated tool Rayyan: results of effectiveness in three diagnostic test accuracy systematic reviews. BMC Med Res Methodol. 2022;22.","DOI":"10.1186\/s12874-022-01631-8"},{"key":"2203_CR15","unstructured":"The EndNote Team. EndNote. EndNote X9 ed. Philadelphia, PA: Clarivate; 2013."},{"key":"2203_CR16","doi-asserted-by":"crossref","unstructured":"McKinney W. Data Structures for Statistical Computing in Python2010. 56\u201361 p.","DOI":"10.25080\/Majora-92bf1922-00a"},{"issue":"7825","key":"2203_CR17","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1038\/s41586-020-2649-2","volume":"585","author":"CR Harris","year":"2020","unstructured":"Harris CR, Millman KJ, van der Walt SJ, Gommers R, Virtanen P, Cournapeau D, et al. Array programming with NumPy. Nature. 2020;585(7825):357\u201362.","journal-title":"Nature"},{"key":"2203_CR18","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: machine learning in P ython. J Mach Learn Res. 2011;12:2825\u201330.","journal-title":"J Mach Learn Res"},{"key":"2203_CR19","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1109\/MCSE.2007.55","volume":"9","author":"JD Hunter","year":"2007","unstructured":"Hunter JD. Matplotlib: a 2D graphics environment. Comput Sci Eng. 2007;9:90\u20135.","journal-title":"Comput Sci Eng"},{"key":"2203_CR20","doi-asserted-by":"publisher","first-page":"3021","DOI":"10.21105\/joss.03021","volume":"6","author":"M Waskom","year":"2021","unstructured":"Waskom M. seaborn: statistical data visualization. J Open Source Softw. 2021;6:3021.","journal-title":"J Open Source Softw"},{"issue":"5","key":"2203_CR21","first-page":"360","volume":"37","author":"AJ Viera","year":"2005","unstructured":"Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37(5):360\u20133.","journal-title":"Fam Med"},{"issue":"8","key":"2203_CR22","doi-asserted-by":"publisher","first-page":"861","DOI":"10.1016\/j.patrec.2005.10.010","volume":"27","author":"T Fawcett","year":"2006","unstructured":"Fawcett T. An introduction to ROC analysis. Pattern Recogn Lett. 2006;27(8):861\u201374.","journal-title":"Pattern Recogn Lett"},{"issue":"1","key":"2203_CR23","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1097\/01.ede.0000147512.81966.ba","volume":"16","author":"EF Schisterman","year":"2005","unstructured":"Schisterman EF, Perkins NJ, Liu A, Bondell H. Optimal cut-point and its corresponding Youden Index to discriminate individuals using pooled blood samples. Epidemiology. 2005;16(1):73\u201381.","journal-title":"Epidemiology"},{"key":"2203_CR24","unstructured":"Jaccard index: Wikipedia; 2023. updated 2023, May 21. Available from: https:\/\/en.wikipedia.org\/wiki\/Jaccard_index."},{"key":"2203_CR25","doi-asserted-by":"publisher","DOI":"10.1201\/9780429246593","volume-title":"An Introduction to the Bootstrap","author":"B Efron","year":"1994","unstructured":"Efron B, Tibshirani RJ. An Introduction to the Bootstrap. 1st ed. New York: Chapman and Hall\/CRC; 1994.","edition":"1"},{"key":"2203_CR26","doi-asserted-by":"crossref","unstructured":"Rayyan\u2014a web and mobile app for systematic reviews. Syst Rev. 2016;5(1):210.","DOI":"10.1186\/s13643-016-0384-4"},{"key":"2203_CR27","doi-asserted-by":"crossref","unstructured":"Wallace BC, Small K, Brodley CE, Lau J, Trikalinos TA, editors. Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. International Health Informatics Symposium; 2012.","DOI":"10.1145\/2110363.2110464"},{"key":"2203_CR28","first-page":"523","volume":"109","author":"MK Kahili-Heede","year":"2021","unstructured":"Kahili-Heede MK, Hillgren KJ. Colandr. J Med Library Assoc. 2021;109:523\u20135.","journal-title":"J Med Library Assoc"},{"key":"2203_CR29","doi-asserted-by":"crossref","unstructured":"dos Reis AHS, de Oliveira ALM, Fritsch C, Zouch J, Ferreira P, Polese JC. Usefulness of machine learning softwares to screen titles of systematic reviews: a methodological study. Syst Rev. 2023;12.","DOI":"10.1186\/s13643-023-02231-3"},{"issue":"1","key":"2203_CR30","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1186\/s13643-019-0942-7","volume":"8","author":"A Bannach-Brown","year":"2019","unstructured":"Bannach-Brown A, Przyby\u0142a P, Thomas J, Rice ASC, Ananiadou S, Liao J, Macleod MR. Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error. Syst Rev. 2019;8(1):23.","journal-title":"Syst Rev"},{"key":"2203_CR31","doi-asserted-by":"crossref","unstructured":"Gates A, Johnson C, Hartling L. Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool. Syst Rev. 2018;7.","DOI":"10.1186\/s13643-018-0707-8"},{"key":"2203_CR32","doi-asserted-by":"crossref","unstructured":"Rathbone J, Hoffmann TC, Glasziou PP. Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Syst Rev. 2015;4.","DOI":"10.1186\/s13643-015-0067-6"},{"issue":"1","key":"2203_CR33","doi-asserted-by":"publisher","first-page":"278","DOI":"10.1186\/s13643-019-1222-2","volume":"8","author":"A Gates","year":"2019","unstructured":"Gates A, Guitard S, Pillay J, Elliott SA, Dyson MP, Newton AS, Hartling L. Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools. Syst Rev. 2019;8(1):278.","journal-title":"Syst Rev"},{"key":"2203_CR34","doi-asserted-by":"crossref","unstructured":"Methley A, Campbell SM, Chew\u2010Graham CA, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14.","DOI":"10.1186\/s12913-014-0579-0"},{"issue":"3","key":"2203_CR35","doi-asserted-by":"publisher","first-page":"355","DOI":"10.1108\/07378830610692127","volume":"24","author":"A Booth","year":"2006","unstructured":"Booth A. Clear and present questions: formulating questions for evidence based practice. Library Hi Tech. 2006;24(3):355\u201368.","journal-title":"Library Hi Tech"},{"issue":"2","key":"2203_CR36","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1046\/j.1471-1842.2002.00378.x","volume":"19","author":"V Wildridge","year":"2002","unstructured":"Wildridge V, Bell L. How CLIP became ECLIPSE: a mnemonic to assist in searching for health policy\/management information. Health Info Libr J. 2002;19(2):113\u20135.","journal-title":"Health Info Libr J"},{"key":"2203_CR37","doi-asserted-by":"crossref","unstructured":"Wang B, Deng X, Sun H, editors. Iteratively Prompt Pre-trained Language Models for Chain of Thought2022 December; Abu Dhabi, United Arab Emirates: Association for Computational Linguistics.","DOI":"10.18653\/v1\/2022.emnlp-main.174"}],"container-title":["BMC Medical Research Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12874-024-02203-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12874-024-02203-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12874-024-02203-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,27]],"date-time":"2024-03-27T14:11:16Z","timestamp":1711548676000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedresmethodol.biomedcentral.com\/articles\/10.1186\/s12874-024-02203-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,27]]},"references-count":37,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["2203"],"URL":"https:\/\/doi.org\/10.1186\/s12874-024-02203-8","relation":{},"ISSN":["1471-2288"],"issn-type":[{"value":"1471-2288","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,27]]},"assertion":[{"value":"5 September 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 March 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 March 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Since no human subjects were involved in the intervention and no sensitive data was used, this study did not require ethical approval.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"78"}}