{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T06:10:34Z","timestamp":1760249434698,"version":"3.41.2"},"reference-count":28,"publisher":"Oxford University Press (OUP)","license":[{"start":{"date-parts":[[2019,11,7]],"date-time":"2019-11-07T00:00:00Z","timestamp":1573084800000},"content-version":"vor","delay-in-days":310,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Union\u2019s Horizon 2020"},{"name":"Marie Sklodowska-Curie","award":["676207"],"award-info":[{"award-number":["676207"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,1,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Curated databases of scientific literature play an important role in helping researchers find relevant literature, but populating such databases is a labour intensive and time-consuming process. One such database is the freely accessible Comet Core Outcome Set database, which was originally populated using manual screening in an annually updated systematic review. In order to reduce the workload and facilitate more timely updates we are evaluating machine learning methods to reduce the number of references needed to screen. In this study we have evaluated a machine learning approach based on logistic regression to automatically rank the candidate articles. Data from the original systematic review and its four first review updates were used to train the model and evaluate performance. We estimated that using automatic screening would yield a workload reduction of at least 75% while keeping the number of missed references around 2%. We judged this to be an acceptable trade-off for this systematic review, and the method is now being used for the next round of the Comet database update.<\/jats:p>","DOI":"10.1093\/database\/baz109","type":"journal-article","created":{"date-parts":[[2019,8,17]],"date-time":"2019-08-17T19:12:11Z","timestamp":1566069131000},"source":"Crossref","is-referenced-by-count":8,"title":["Evaluation of an automatic article selection method for timelier updates of the Comet Core Outcome Set database"],"prefix":"10.1093","volume":"2019","author":[{"given":"Christopher R","family":"Norman","sequence":"first","affiliation":[{"name":"LIMSI, CNRS, Universit\u00e9 Paris-Saclay, B\u00e2t 507, rue du Belv\u00e9d\u00e8re, Campus Universitaire, F-91405 Orsay"}]},{"given":"Elizabeth","family":"Gargon","sequence":"additional","affiliation":[{"name":"MRC NWHMTR, Department of Biostatistics, University of Liverpool, Liverpool, UK"}]},{"given":"Mariska M G","family":"Leeflang","sequence":"additional","affiliation":[{"name":"Amsterdam Public Health, Amsterdam Umc, University of Amsterdam, Meibergdreef 9, 1105 az, Amsterdam, the Netherlands"}]},{"given":"Aur\u00e9lie","family":"N\u00e9v\u00e9ol","sequence":"additional","affiliation":[{"name":"LIMSI, CNRS, Universit\u00e9 Paris-Saclay, B\u00e2t 507, rue du Belv\u00e9d\u00e8re, Campus Universitaire, F-91405 Orsay"}]},{"given":"Paula R","family":"Williamson","sequence":"additional","affiliation":[{"name":"MRC NWHMTR, Department of Biostatistics, University of Liverpool, Liverpool, UK"}]}],"member":"286","published-online":{"date-parts":[[2019,11,7]]},"reference":[{"key":"2019110708261298100_ref1","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1186\/s13643-019-0942-7","article-title":"Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error","volume":"8","author":"Bannach-Brown","year":"2019","journal-title":"Syst. Rev."},{"key":"2019110708261298100_ref2","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1186\/s13643-018-0740-7","article-title":"Making progress with the automation of systematic reviews: principles of the international collaboration for the automation of systematic reviews (icasr)","volume":"7","author":"Beller","year":"2018","journal-title":"Syst. Rev."},{"key":"2019110708261298100_ref3","first-page":"81","article-title":"From ranknet to lambdarank to lambdamart: an overview","volume":"11","author":"Burges","year":"2010","journal-title":"Learning"},{"key":"2019110708261298100_ref4","article-title":"Feature generation, feature selection, classifiers, and conceptual drift for biomedical document triage","volume-title":"TREC","author":"Cohen","year":"2004"},{"key":"2019110708261298100_ref5","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0190695","article-title":"Choosing important health outcomes for comparative effectiveness research: an updated systematic review and involvement of low and middle income countries","volume":"13","author":"Davis","year":"2018","journal-title":"PLoS One"},{"key":"2019110708261298100_ref6","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1093\/comjnl\/35.3.243","article-title":"Probabilistic models in information retrieval","volume":"35","author":"Fuhr","year":"1992","journal-title":"Comput. J."},{"key":"2019110708261298100_ref7","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0209869","article-title":"Choosing important health outcomes for comparative effectiveness research: 4th annual update to a systematic review of core outcome sets for research","volume":"13","author":"Gargon","year":"2018","journal-title":"PLoS One"},{"key":"2019110708261298100_ref8","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0099111","article-title":"Choosing important health outcomes for comparative effectiveness research: a systematic review","volume":"9","author":"Gargon","year":"2014","journal-title":"PLoS One"},{"key":"2019110708261298100_ref9","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1186\/s12874-015-0019-9","article-title":"Collating the knowledge base for core outcome set development: developing and appraising the search strategy for a systematic review","volume":"15","author":"Gargon","year":"2015","journal-title":"BMC Med. Res. Methodol."},{"key":"2019110708261298100_ref10","article-title":"Choosing important health outcomes for comparative effectiveness research: an updated review and user survey","volume":"11","author":"Gorst","year":"2016","journal-title":"PLoS One"},{"key":"2019110708261298100_ref11","article-title":"Choosing important health outcomes for comparative effectiveness research: an updated review and identification of gaps","volume":"11","author":"Gorst","year":"2016","journal-title":"PLoS One"},{"key":"2019110708261298100_ref12","article-title":"Integrating text mining into the MGI biocuration workflow","volume":"2009","author":"Hill","year":"2009","journal-title":"Database"},{"key":"2019110708261298100_ref13","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1186\/s13643-016-0263-z","article-title":"Swift-review: a text-mining workbench for systematic review","volume":"5","author":"Howard","year":"2016","journal-title":"Syst. Rev."},{"key":"2019110708261298100_ref14","article-title":"Overview of the CLEF technologically assisted reviews in empirical medicine","volume-title":"Working Notes of CLEF 2017\u2014Conference and Labs of the Evaluation forum, Dublin, Ireland, September 11\u201314, 2017","author":"Kanoulas","year":"2017"},{"key":"2019110708261298100_ref15","article-title":"Clef 2018 technologically assisted reviews in empirical medicine overview","volume-title":"Working Notes of CLEF 2018\u2014Conference and Labs of the Evaluation forum, Avignon, France, September 10\u201314, 2018","author":"Kanoulas","year":"2018"},{"key":"2019110708261298100_ref16","doi-asserted-by":"crossref","first-page":"465","DOI":"10.1007\/s10994-015-5535-7","article-title":"Learning to identify relevant studies for systematic reviews using random forest and external information","volume":"102","author":"Khabsa","year":"2016","journal-title":"Mach. Lear."},{"key":"2019110708261298100_ref17","doi-asserted-by":"crossref","first-page":"S3","DOI":"10.1186\/1471-2105-12-S8-S3","article-title":"The protein\u2013protein interaction tasks of biocreative iii: classification\/ranking of articles and linking bio-ontology concepts to full text","volume":"12","author":"Krallinger","year":"2011","journal-title":"BMC Bioinform."},{"key":"2019110708261298100_ref18","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1016\/j.jclinepi.2018.12.001","article-title":"Automatic screening using word embeddings achieved high sensitivity and workload reduction for updating living network meta-analyses","volume":"108","author":"Lerner","year":"2019","journal-title":"J. Clin. Epidemiol."},{"key":"2019110708261298100_ref19","article-title":"Automating document discovery in the systematic review process: how to use chaff to extract wheat","volume-title":"International Conference on Language Resources and Evaluation","author":"Norman","year":"2018"},{"key":"2019110708261298100_ref20","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1145\/2915970.2915982","article-title":"A critical analysis of studies that address the use of text mining for citation screening in systematic reviews","volume-title":"Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering","author":"Olorisade","year":"2016"},{"key":"2019110708261298100_ref21","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1186\/2046-4053-4-5","article-title":"Using text mining for study identification in systematic reviews: a systematic review of current approaches","volume":"4","author":"O\u2019Mara-Eves","year":"2015","journal-title":"Syst. Rev."},{"key":"2019110708261298100_ref22","first-page":"2825","article-title":"Scikit-learn: machine learning in python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"2019110708261298100_ref23","doi-asserted-by":"crossref","first-page":"470","DOI":"10.1002\/jrsm.1311","article-title":"Prioritising references for systematic reviews with robotanalyst: a user study","volume":"9","author":"Przyby\u0142a","year":"2018","journal-title":"Res. Synth. Methods"},{"key":"2019110708261298100_ref24","first-page":"286","article-title":"Overview of the CLEF ehealth evaluation lab 2018","volume-title":"International Conference of the Cross-Language Evaluation Forum for European Languages","author":"Suominen","year":"2018"},{"volume-title":"Eppi-reviewer: Software for Research Synthesis","year":"2007","author":"Thomas","key":"2019110708261298100_ref25"},{"key":"2019110708261298100_ref26","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1186\/2046-4053-3-74","article-title":"Systematic review automation technologies","volume":"3","author":"Tsafnat","year":"2014","journal-title":"Syst. Rev."},{"key":"2019110708261298100_ref27","doi-asserted-by":"crossref","first-page":"819","DOI":"10.1145\/2110363.2110464","article-title":"Deploying an interactive machine learning system in an evidence-based practice center","volume-title":"Proceedings of the 2nd ACM SIGHIT symposium on International health informatics\u2014IHI\u201912","author":"Wallace","year":"2012"},{"key":"2019110708261298100_ref28","doi-asserted-by":"crossref","first-page":"280","DOI":"10.1186\/s13063-017-1978-4","article-title":"The comet handbook: version 1.0","volume":"18","author":"Williamson","year":"2017","journal-title":"Trials"}],"container-title":["Database"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/database\/article-pdf\/doi\/10.1093\/database\/baz109\/30457495\/baz109.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/database\/article-pdf\/doi\/10.1093\/database\/baz109\/30457495\/baz109.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,1,16]],"date-time":"2021-01-16T23:28:13Z","timestamp":1610839693000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/database\/article\/doi\/10.1093\/database\/baz109\/5611293"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,1,1]]},"references-count":28,"URL":"https:\/\/doi.org\/10.1093\/database\/baz109","relation":{},"ISSN":["1758-0463"],"issn-type":[{"type":"electronic","value":"1758-0463"}],"subject":[],"published-other":{"date-parts":[[2019]]},"published":{"date-parts":[[2019,1,1]]},"article-number":"baz109"}}