{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,22]],"date-time":"2026-03-22T03:42:14Z","timestamp":1774150934489,"version":"3.50.1"},"reference-count":39,"publisher":"Oxford University Press (OUP)","issue":"8","license":[{"start":{"date-parts":[[2022,5,31]],"date-time":"2022-05-31T00:00:00Z","timestamp":1653955200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"Peking Union Medical College Hospital Research Grant for Young Scholar","award":["pumch201912048"],"award-info":[{"award-number":["pumch201912048"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,7,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>We aim to investigate the application and accuracy of artificial intelligence (AI) methods for automated medical literature screening for systematic reviews.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We systematically searched PubMed, Embase, and IEEE Xplore Digital Library to identify potentially relevant studies. We included studies in automated literature screening that reported study question, source of dataset, and developed algorithm models for literature screening. The literature screening results by human investigators were considered to be the reference standard. Quantitative synthesis of the accuracy was conducted using a bivariate model.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Eighty-six studies were included in our systematic review and 17 studies were further included for meta-analysis. The combined recall, specificity, and precision were 0.928 [95% confidence interval (CI), 0.878\u20130.958], 0.647 (95% CI, 0.442\u20130.809), and 0.200 (95% CI, 0.135\u20130.287) when achieving maximized recall, but were 0.708 (95% CI, 0.570\u20130.816), 0.921 (95% CI, 0.824\u20130.967), and 0.461 (95% CI, 0.375\u20130.549) when achieving maximized precision in the AI models. No significant difference was found in recall among subgroup analyses including the algorithms, the number of screened literatures, and the fraction of included literatures.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion and Conclusion<\/jats:title>\n                  <jats:p>This systematic review and meta-analysis study showed that the recall is more important than the specificity or precision in literature screening, and a recall over 0.95 should be prioritized. We recommend to report the effectiveness indices of automatic algorithms separately. At the current stage manual literature screening is still indispensable for medical systematic reviews.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocac066","type":"journal-article","created":{"date-parts":[[2022,6,1]],"date-time":"2022-06-01T01:04:11Z","timestamp":1654045451000},"page":"1425-1432","source":"Crossref","is-referenced-by-count":47,"title":["Automated medical literature screening using artificial intelligence: a systematic review and meta-analysis"],"prefix":"10.1093","volume":"29","author":[{"given":"Yunying","family":"Feng","sequence":"first","affiliation":[{"name":"Eight-year Program of Clinical Medicine, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College , Beijing, China"}]},{"given":"Siyu","family":"Liang","sequence":"additional","affiliation":[{"name":"Department of Endocrinology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College , Beijing, China"}]},{"given":"Yuelun","family":"Zhang","sequence":"additional","affiliation":[{"name":"Medical Research Center, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College , Beijing, China"},{"name":"Clinical Epidemiology Unit, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College , Beijing, China"}]},{"given":"Shi","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Endocrinology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College , Beijing, China"},{"name":"Clinical Epidemiology Unit, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College , Beijing, China"}]},{"given":"Qing","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Automation, Tsinghua University , Beijing, China"}]},{"given":"Tianze","family":"Huang","sequence":"additional","affiliation":[{"name":"Eight-year Program of Clinical Medicine, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College , Beijing, China"}]},{"given":"Feng","family":"Sun","sequence":"additional","affiliation":[{"name":"Department of Epidemiology and Biostatistics, School of Public Health, Peking University Health Science Center , Beijing, China"}]},{"given":"Xiaoqing","family":"Liu","sequence":"additional","affiliation":[{"name":"Clinical Epidemiology Unit, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College , Beijing, China"},{"name":"Department of Infectious Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College , Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5172-6870","authenticated-orcid":false,"given":"Huijuan","family":"Zhu","sequence":"additional","affiliation":[{"name":"Department of Endocrinology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College , Beijing, China"},{"name":"Clinical Epidemiology Unit, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College , Beijing, China"}]},{"given":"Hui","family":"Pan","sequence":"additional","affiliation":[{"name":"Department of Endocrinology, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College , Beijing, China"}]}],"member":"286","published-online":{"date-parts":[[2022,5,31]]},"reference":[{"key":"2022071310040865600_ocac066-B1","author":"Higgins","year":"2019"},{"issue":"1","key":"2022071310040865600_ocac066-B2","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1093\/pubmed\/fdr015","article-title":"Scoping the scope\u2019 of a Cochrane review","volume":"33","author":"Armstrong","year":"2011","journal-title":"J Public Health"},{"issue":"6","key":"2022071310040865600_ocac066-B3","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1016\/j.jclinepi.2008.02.004","article-title":"Systematic reviews can be produced and published faster","volume":"61","author":"Sampson","year":"2008","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"2022071310040865600_ocac066-B4","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1186\/1471-2288-11-92","article-title":"The global evidence mapping initiative: scoping research in broad topic areas","volume":"11","author":"Bragge","year":"2011","journal-title":"BMC Med Res Methodol"},{"issue":"1","key":"2022071310040865600_ocac066-B5","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13643-018-0856-9","article-title":"Time-to-update of systematic reviews relative to the availability of new evidence","volume":"7","author":"Bashir","year":"2018","journal-title":"Syst Rev"},{"key":"2022071310040865600_ocac066-B6","volume-title":"Pattern Recognition and Machine Learning","author":"Bishop","year":"2006"},{"issue":"1","key":"2022071310040865600_ocac066-B7","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1186\/2046-4053-3-74","article-title":"Systematic review automation technologies","volume":"3","author":"Tsafnat","year":"2014","journal-title":"Syst Rev"},{"issue":"1","key":"2022071310040865600_ocac066-B8","doi-asserted-by":"crossref","first-page":"266","DOI":"10.1093\/ije\/dyv306","article-title":"Machine learning to assist risk-of-bias assessments in systematic reviews","volume":"45","author":"Millard","year":"2016","journal-title":"Int J Epidemiol"},{"issue":"1","key":"2022071310040865600_ocac066-B9","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1093\/jamia\/ocv044","article-title":"RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials","volume":"23","author":"Marshall","year":"2016","journal-title":"J Am Med Inform Assoc"},{"key":"2022071310040865600_ocac066-B10","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/j.jclinepi.2021.02.003","article-title":"Updating guidance for reporting systematic reviews: development of the PRISMA 2020 statement","volume":"134","author":"Page","year":"2021","journal-title":"J Clin Epidemiol"},{"key":"2022071310040865600_ocac066-B11","doi-asserted-by":"crossref","first-page":"n160","DOI":"10.1136\/bmj.n160","article-title":"PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews","volume":"372","author":"Page","year":"2021","journal-title":"BMJ"},{"issue":"1","key":"2022071310040865600_ocac066-B12","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1186\/s13643-015-0066-7","article-title":"Automating data extraction in systematic reviews: a systematic review","volume":"4","author":"Jonnalagadda","year":"2015","journal-title":"Syst Rev"},{"issue":"2","key":"2022071310040865600_ocac066-B13","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1197\/jamia.M1929","article-title":"Reducing workload in systematic review preparation using automated citation classification","volume":"13","author":"Cohen","year":"2006","journal-title":"J Am Med Inform Assoc"},{"key":"2022071310040865600_ocac066-B14","year":"2016"},{"issue":"4","key":"2022071310040865600_ocac066-B15","doi-asserted-by":"crossref","first-page":"388","DOI":"10.1001\/jama.2017.19163","article-title":"Preferred reporting items for a systematic review and meta-analysis of diagnostic test accuracy studies: the PRISMA-DTA statement","volume":"319","author":"McInnes","year":"2018","journal-title":"JAMA"},{"issue":"8","key":"2022071310040865600_ocac066-B16","doi-asserted-by":"crossref","first-page":"529","DOI":"10.7326\/0003-4819-155-8-201110180-00009","article-title":"QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies","volume":"155","author":"Whiting","year":"2011","journal-title":"Ann Intern Med"},{"issue":"10","key":"2022071310040865600_ocac066-B17","doi-asserted-by":"crossref","first-page":"982","DOI":"10.1016\/j.jclinepi.2005.02.022","article-title":"Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews","volume":"58","author":"Reitsma","year":"2005","journal-title":"J Clin Epidemiol"},{"key":"2022071310040865600_ocac066-B18","author":"Doebler","year":"2015"},{"key":"2022071310040865600_ocac066-B19","year":"2014"},{"key":"2022071310040865600_ocac066-B20","author":"Ioannidis","year":"2021"},{"key":"2022071310040865600_ocac066-B21","author":"Powers","year":"2020"},{"issue":"6947","key":"2022071310040865600_ocac066-B22","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1136\/bmj.309.6947.102","article-title":"Statistics notes: diagnostic tests 2: predictive values","volume":"309","author":"Altman","year":"1994","journal-title":"BMJ"},{"issue":"6943","key":"2022071310040865600_ocac066-B23","doi-asserted-by":"crossref","first-page":"1552","DOI":"10.1136\/bmj.308.6943.1552","article-title":"Diagnostic tests. 1: sensitivity and specificity","volume":"308","author":"Altman","year":"1994","journal-title":"BMJ"},{"issue":"1","key":"2022071310040865600_ocac066-B24","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1186\/1472-6947-12-33","article-title":"Studying the potential impact of automated document classification on scheduling a systematic review update","volume":"12","author":"Cohen","year":"2012","journal-title":"BMC Med Inform Decis Making"},{"issue":"3","key":"2022071310040865600_ocac066-B25","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1177\/0272989X12457243","article-title":"A pilot study using machine learning and domain knowledge to facilitate comparative effectiveness review updating","volume":"33","author":"Dalal","year":"2013","journal-title":"Med Decis Making"},{"key":"2022071310040865600_ocac066-B26","year":"2015"},{"key":"2022071310040865600_ocac066-B27","year":"2016"},{"key":"2022071310040865600_ocac066-B28","author":"Saha","year":"2016"},{"key":"2022071310040865600_ocac066-B29","doi-asserted-by":"crossref","first-page":"103202","DOI":"10.1016\/j.jbi.2019.103202","article-title":"The use of bibliography enriched features for automatic citation screening","volume":"94","author":"Olorisade","year":"2019","journal-title":"J Biomed Inform"},{"key":"2022071310040865600_ocac066-B30","year":"2006"},{"key":"2022071310040865600_ocac066-B31","year":"2017"},{"key":"2022071310040865600_ocac066-B32","year":"2018"},{"issue":"2","key":"2022071310040865600_ocac066-B33","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1197\/jamia.M1641","article-title":"Text categorization models for high-quality article retrieval in internal medicine","volume":"12","author":"Aphinyanaphongs","year":"2005","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2022071310040865600_ocac066-B34","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1136\/jamia.2010.008177","article-title":"Performance of support-vector-machine-based classification on 15 systematic review topics evaluated with the WSS@ 95 measure","volume":"18","author":"Cohen","year":"2011","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"2022071310040865600_ocac066-B35","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1016\/j.artmed.2012.05.002","article-title":"Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers","volume":"55","author":"Bekhuis","year":"2012","journal-title":"Artif Intell Med"},{"issue":"4","key":"2022071310040865600_ocac066-B36","doi-asserted-by":"crossref","first-page":"1498","DOI":"10.1016\/j.eswa.2013.08.047","article-title":"Automatic text classification to support systematic reviews in medicine","volume":"41","author":"Adeva","year":"2014","journal-title":"Expert Syst Appl"},{"issue":"2","key":"2022071310040865600_ocac066-B37","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1007\/s10796-015-9589-7","article-title":"Advanced analytics for the automation of medical systematic reviews","volume":"18","author":"Timsina","year":"2016","journal-title":"Inf Syst Front"},{"issue":"11","key":"2022071310040865600_ocac066-B38","doi-asserted-by":"crossref","first-page":"1635","DOI":"10.1002\/sim.1190","article-title":"Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records","volume":"21","author":"Edwards","year":"2002","journal-title":"Stat Med"},{"issue":"46","key":"2022071310040865600_ocac066-B39","doi-asserted-by":"crossref","first-page":"22990","DOI":"10.1073\/pnas.1912091116","article-title":"Tuberculosis diagnosis and treatment under uncertainty","volume":"116","author":"Cassidy","year":"2019","journal-title":"Proc Natl Acad Sci USA"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/8\/1425\/44829570\/ocac066.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/8\/1425\/44829570\/ocac066.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,7,13]],"date-time":"2022-07-13T10:09:34Z","timestamp":1657706974000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/29\/8\/1425\/6596174"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,31]]},"references-count":39,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2022,5,31]]},"published-print":{"date-parts":[[2022,7,12]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocac066","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,8,1]]},"published":{"date-parts":[[2022,5,31]]}}}