{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,10]],"date-time":"2026-03-10T05:21:02Z","timestamp":1773120062808,"version":"3.50.1"},"reference-count":62,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2020,9,3]],"date-time":"2020-09-03T00:00:00Z","timestamp":1599091200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.springer.com\/tdm"},{"start":{"date-parts":[[2020,9,3]],"date-time":"2020-09-03T00:00:00Z","timestamp":1599091200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.springer.com\/tdm"}],"funder":[{"name":"FONDECYT","award":["1191791"],"award-info":[{"award-number":["1191791"]}]},{"name":"IMFD Fundamentals of Data","award":["2019"],"award-info":[{"award-number":["2019"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Scientometrics"],"published-print":{"date-parts":[[2020,12]]},"DOI":"10.1007\/s11192-020-03648-6","type":"journal-article","created":{"date-parts":[[2020,9,3]],"date-time":"2020-09-03T01:02:33Z","timestamp":1599094953000},"page":"3047-3084","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":24,"title":["Automatic document screening of medical literature using word and text embeddings in an active learning setting"],"prefix":"10.1007","volume":"125","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5042-8772","authenticated-orcid":false,"given":"Andres","family":"Carvallo","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Denis","family":"Parra","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hans","family":"Lobel","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alvaro","family":"Soto","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2020,9,3]]},"reference":[{"issue":"4","key":"3648_CR1","doi-asserted-by":"publisher","first-page":"1498","DOI":"10.1016\/j.eswa.2013.08.047","volume":"41","author":"JG Adeva","year":"2014","unstructured":"Adeva, J. G., Atxa, J. P., Carrillo, M. U., & Zengotitabengoa, E. A. (2014). Automatic text classification to support systematic reviews in medicine. Expert Systems with Applications, 41(4), 1498\u20131508.","journal-title":"Expert Systems with Applications"},{"key":"3648_CR2","unstructured":"Alharbi, A., Briggs, W., & Stevenson, M. (2018). Retrieving and ranking studies for systematic reviews: University of sheffield\u2019s approach to CLEF eHealth 2018 task 2. In CEUR workshop proceedings (Vol. 2125)."},{"key":"3648_CR3","unstructured":"Alharbi, A., & Stevenson, M. (2017). Ranking abstracts to identify relevant evidence for systematic reviews: The university of sheffield\u2019s approach to CLEF eHealth 2017 task 2. In Clef (working notes)."},{"key":"3648_CR4","doi-asserted-by":"crossref","unstructured":"Alharbi, A., & Stevenson, M. (2019). Ranking studies for systematic reviews using query adaptation: University of sheffield\u2019s approach to CLEF eHealth 2019 task 2. In Clef (working notes).","DOI":"10.1007\/978-3-030-28577-7_9"},{"issue":"1","key":"3648_CR5","doi-asserted-by":"publisher","first-page":"e86277","DOI":"10.1371\/journal.pone.0086277","volume":"9","author":"T Bekhuis","year":"2014","unstructured":"Bekhuis, T., Tseytlin, E., Mitchell, K. J., & Demner-Fushman, D. (2014). Feature engineering and a proposed decision-support system for systematic reviewers of medical evidence. PloS ONE, 9(1), e86277.","journal-title":"PloS ONE"},{"key":"3648_CR6","unstructured":"Chen, J., Chen, S., Song, Y., Liu, H., Wang, Y., Hu, Q., & Yang, Y. (2017). ECNU at 2017 eHealth task 2: Technologically assisted reviews in empirical medicine. In CLEF (working notes)."},{"issue":"4","key":"3648_CR7","doi-asserted-by":"publisher","first-page":"76","DOI":"10.1016\/j.ins.2012.05.027","volume":"21","author":"S Choi","year":"2012","unstructured":"Choi, S., Ryu, B., Yoo, S., & Choi, J. (2012). Combining relevancy and methodological quality into a single ranking for evidence-based medicine. Information Sciences, 21(4), 76\u201390.","journal-title":"Information Sciences"},{"key":"3648_CR8","unstructured":"Cohen, A. M., & Smalheiser, N. R. (2018). UIC\/OHSU CLEF 2018 task 2 diagnostic test accuracy ranking using publication type cluster similarity measures. In CEUR workshop proceedings (Vol. 2125)."},{"key":"3648_CR9","unstructured":"Cormack, G. V., & Grossman, M. R. (2016). \u201c when to stop\u201d waterloo (CORMACK) participation in the TREC 2016 total recall track. In TREC."},{"key":"3648_CR10","unstructured":"Cormack, G. V., & Grossman, M. R. (2017). Technology-assisted review in empirical medicine: Waterloo participation in CLEF eHealth 2017. In CLEF (working notes)."},{"key":"3648_CR11","doi-asserted-by":"crossref","unstructured":"Dehghani, M., Zamani, H., Severyn, A., Kamps, J., & Croft, W. B. (2017). Neural ranking models with weak supervision. In Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval (pp. 65\u201374).","DOI":"10.1145\/3077136.3080832"},{"issue":"6","key":"3648_CR12","doi-asserted-by":"publisher","first-page":"10281","DOI":"10.2196\/10281","volume":"20","author":"G Del Fiol","year":"2018","unstructured":"Del Fiol, G., Michelson, M., Iorio, A., Cotoi, C., & Haynes, R. B. (2018). A deep learning method to automatically identify reports of scientifically rigorous clinical research from the biomedical literature: Comparative analytic study. Journal of Medical Internet Research, 20(6), 10281.","journal-title":"Journal of Medical Internet Research"},{"key":"3648_CR13","unstructured":"Devlin, J., Chang, M.W., Lee, K., & Toutanova, K. (2018). Bert: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805."},{"key":"3648_CR14","unstructured":"Di\u00a0Nunzio, G. M. (2019). A distributed effort approach for systematic reviews. IMS UNIPD at CLEF 2019 eHealth task 2. In Clef (working notes)."},{"key":"3648_CR15","unstructured":"Di\u00a0Nunzio, G. M., Beghini, F., Vezzani, F., & Henrot, G. (2017). An interactive two-dimensional approach to query aspects rewriting in systematic reviews. IMS UNIPD at CLEF eHealth task 2. In CLEF (working notes)."},{"key":"3648_CR16","unstructured":"Di\u00a0Nunzio, G. M., Ciuffreda, G., & Vezzani, F. (2018). Interactive sampling for systematic reviews. IMS UNIPD at CLEF 2018 eHealth task 2. In CLEF (working notes)."},{"key":"3648_CR17","doi-asserted-by":"crossref","unstructured":"Donoso-Guzm\u00e1n, I., & Parra, D. (2018). An interactive relevance feedback interface for evidence-based health care. In 23rd international conference on intelligent user interfaces (pp. 103\u2013114).","DOI":"10.1145\/3172944.3172953"},{"issue":"2","key":"3648_CR18","doi-asserted-by":"publisher","first-page":"e1001603","DOI":"10.1371\/journal.pmed.1001603","volume":"11","author":"JH Elliott","year":"2014","unstructured":"Elliott, J. H., Turner, T., Clavisi, O., Thomas, J., Higgins, J. P., Mavergames, C., et al. (2014). Living systematic reviews: An emerging opportunity to narrow the evidence-practice gap. PLoS Medicine, 11(2), e1001603.","journal-title":"PLoS Medicine"},{"issue":"5","key":"3648_CR19","doi-asserted-by":"publisher","first-page":"809","DOI":"10.1136\/amiajnl-2011-000648","volume":"19","author":"RL Figueroa","year":"2012","unstructured":"Figueroa, R. L., Zeng-Treitler, Q., Ngo, L. H., Goryachev, S., & Wiechmann, E. P. (2012). Active learning for clinical text classification: is it better than random sampling? Journal of the American Medical Informatics Association, 19(5), 809\u2013816.","journal-title":"Journal of the American Medical Informatics Association"},{"key":"3648_CR20","doi-asserted-by":"crossref","unstructured":"Goeuriot, L., Kelly, L., Suominen, H., N\u00e9v\u00e9ol, A., Robert, A., Kanoulas, E., & Zuccon, G. (2017). CLEF 2017 eHealth evaluation lab overview. In International conference of the cross-language evaluation forum for European languages (pp. 291\u2013303).","DOI":"10.1007\/978-3-319-65813-1_26"},{"key":"3648_CR21","doi-asserted-by":"crossref","unstructured":"Goodwin, T. R., & Harabagiu, S. M. (2018). Knowledge representations and inference techniques for medical question answering. In ACM transactions on intelligent systems and technology (TIST) 9214 .","DOI":"10.1145\/3106745"},{"key":"3648_CR22","unstructured":"Grossman, M. R., Cormack, G. V., & Roegiest, A. (2016). Trec 2016 total recall track overview. In TREC."},{"issue":"2","key":"3648_CR23","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1016\/j.jbi.2016.06.001","volume":"6","author":"K Hashimoto","year":"2016","unstructured":"Hashimoto, K., Kontonatsios, G., Miwa, M., & Ananiadou, S. (2016). Topic detection using paragraph vectors to support active learning in systematic reviews. Journal of Biomedical Informatics, 6(2), 59\u201365.","journal-title":"Journal of Biomedical Informatics"},{"key":"3648_CR24","unstructured":"Hollmann, N., & Eickhoff, C. (2017). Ranking and feedback-based stopping for recall-centric document retrieval. In CLEF (working notes)."},{"key":"3648_CR25","doi-asserted-by":"crossref","unstructured":"Howard, J., & Ruder, S. (2018). Universal language model fine-tuning for text classification. arXiv:1801.06146.","DOI":"10.18653\/v1\/P18-1031"},{"key":"3648_CR26","first-page":"246","volume":"235","author":"M Hughes","year":"2017","unstructured":"Hughes, M., Li, I., Kotoulas, S., & Suzumura, T. (2017). Medical text classification using convolutional neural networks. Stud Health Technol Inform, 235, 246\u201350.","journal-title":"Stud Health Technol Inform"},{"key":"3648_CR27","doi-asserted-by":"crossref","unstructured":"Joachims, T. (1998). Text categorization with support vector machines: Learning with many relevant features. In European conference on machine learning (pp. 137\u2013142).","DOI":"10.1007\/BFb0026683"},{"key":"3648_CR28","unstructured":"Kalphov, V., Georgiadis, G., & Azzopardi, L. (2017). Sis at CLEF 2017 eHealth tar task. In CEUR workshop proceedings (Vol. 1866, pp. 1\u20135)."},{"key":"3648_CR29","unstructured":"Kanoulas, E., Li, D., Azzopardi, L., & Spijker, R. (2017). CLEF 2017 technologically assisted reviews in empirical medicine overview. In CEUR workshop proceedings (Vol. 1866, pp. 1\u201329)."},{"key":"3648_CR30","unstructured":"Kanoulas, E., Li, D., Azzopardi, L., & Spijker, R. (2018). CLEF 2018 technologically assisted reviews in empirical medicine overview. CEUR workshop proceedings (Vol. 1866, pp. 1\u201334)."},{"key":"3648_CR31","unstructured":"Kanoulas, E., Li, D., Azzopardi, L., & Spijker, R. (2019). CLEF 2019 technology assisted reviews in empirical medicine overview. In CEUR workshop proceedings (Vol. 2380)."},{"issue":"6","key":"3648_CR32","doi-asserted-by":"publisher","first-page":"1151","DOI":"10.1016\/j.jbi.2012.07.012","volume":"45","author":"A Keselman","year":"2012","unstructured":"Keselman, A., & Smith, C. A. (2012). A classification of errors in lay comprehension of medical documents. Journal of Biomedical Informatics, 45(6), 1151\u20131163.","journal-title":"Journal of Biomedical Informatics"},{"key":"3648_CR33","doi-asserted-by":"crossref","unstructured":"Lagopoulos, A., Anagnostou, A., Minas, A., & Tsoumakas, G. (2018). Learning-to-rank and relevance feedback for literature appraisal in empirical medicine. In International conference of the cross-language evaluation forum for European languages (pp. 52\u201363).","DOI":"10.1007\/978-3-319-98932-7_5"},{"key":"3648_CR34","unstructured":"Lee, G. E. (2017). A study of convolutional neural networks for clinical document classification in systematic reviews: Sysreview at CLEF eHealth 2017."},{"key":"3648_CR35","doi-asserted-by":"crossref","unstructured":"Lee, G. E., & Sun, A. (2018). Seed-driven document ranking for systematic reviews in evidence-based medicine. In The 41st international ACM SIGIR conference on research & development in information retrieval (pp. 455\u2013464).","DOI":"10.1145\/3209978.3209994"},{"key":"3648_CR36","doi-asserted-by":"crossref","unstructured":"Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2019). Biobert: pre-trained biomedical language representation model for biomedical text mining. arXiv:1901.08746.","DOI":"10.1093\/bioinformatics\/btz682"},{"key":"3648_CR37","unstructured":"Li, D., Kanoulas, E. et\u00a0al. (2019). Automatic thresholding by sampling documents and estimating recall: Ilps@ uva at tar task 2.2. In CEUR workshop proceedings (Vol. 2380)."},{"key":"3648_CR38","unstructured":"Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems (pp. 3111\u20133119)."},{"key":"3648_CR39","unstructured":"Minas, A., Lagopoulos, A., & Tsoumakas, G. (2018). Aristotle university\u2019s approach to the technologically assisted reviews in empirical medicine task of the 2018 CLEF eHealth lab. In CLEF (working notes)."},{"key":"3648_CR40","doi-asserted-by":"publisher","first-page":"242","DOI":"10.1016\/j.jbi.2014.06.005","volume":"51","author":"M Miwa","year":"2014","unstructured":"Miwa, M., Thomas, J., O\u2019Mara-Eves, A., & Ananiadou, S. (2014). Reducing systematic review workload through certainty-based screening. Journal of Biomedical Informatics, 51, 242\u2013253.","journal-title":"Journal of Biomedical Informatics"},{"issue":"1","key":"3648_CR41","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1186\/s13643-015-0117-0","volume":"4","author":"Y Mo","year":"2015","unstructured":"Mo, Y., Kontonatsios, G., & Ananiadou, S. (2015). Supporting systematic reviews using LDA-based document representations. Systematic Reviews, 4(1), 172.","journal-title":"Systematic Reviews"},{"key":"3648_CR42","unstructured":"Nogueira, R., Yang, W., Cho, K., & Lin,J. (2019). Multi-stage document ranking with BERT. arXiv:1910.14424."},{"key":"3648_CR43","unstructured":"Norman, C., Leeflang, M., & N\u00e9v\u00e9ol, A. (2017). Limsi@ CLEF eHealth 2017 task 2: Logistic regression for automatic article ranking."},{"key":"3648_CR44","unstructured":"Norman, C. R., Leeflang, M. M., & N\u00e9v\u00e9ol, A. (2018). Limsi@ CLEF eHealth 2018 task 2: Technology assisted reviews by stacking active and static learning. In CLEF (working notes) 2125 (pp. 1\u201313)."},{"key":"3648_CR45","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., & Grisel, O. (2011). Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12, 2825\u20132830.","journal-title":"Journal of Machine Learning Research"},{"key":"3648_CR46","doi-asserted-by":"crossref","unstructured":"Pennington, J., Socher, R., & Manning, C. (2014). Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532\u20131543).","DOI":"10.3115\/v1\/D14-1162"},{"key":"3648_CR47","doi-asserted-by":"crossref","unstructured":"Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv:1802.05365.","DOI":"10.18653\/v1\/N18-1202"},{"key":"3648_CR48","unstructured":"Qiao, Y., Xiong, C., Liu, Z., & Liu, Z. (2019). Understanding the behaviors of bert in ranking. arXiv:1904.07531."},{"issue":"3","key":"3648_CR49","doi-asserted-by":"publisher","first-page":"269","DOI":"10.14778\/3157794.3157797","volume":"11","author":"A Ratner","year":"2017","unstructured":"Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & R\u00e9, C. (2017). Snorkel: Rapid training data creation with weak supervision. Proceedings of the VLDB Endowment, 11(3), 269\u2013282.","journal-title":"Proceedings of the VLDB Endowment"},{"key":"3648_CR50","doi-asserted-by":"crossref","unstructured":"Roy, D., Ganguly, D., Bhatia, S., Bedathur, S., & Mitra, M. (2018). Using word embeddings for information retrieval: How collection and term normalization choices affect performance. In Proceedings of the 27th ACM international conference on information and knowledge management (pp. 1835\u20131838).","DOI":"10.1145\/3269206.3269277"},{"issue":"11","key":"3648_CR51","doi-asserted-by":"publisher","first-page":"613","DOI":"10.1145\/361219.361220","volume":"18","author":"G Salton","year":"1975","unstructured":"Salton, G., Wong, A., & Yang, C. S. (1975). A vector space model for automatic indexing. Communications of the ACM, 18(11), 613\u2013620.","journal-title":"Communications of the ACM"},{"key":"3648_CR52","unstructured":"Scells, H., Zuccon, G., Deacon, A., & Koopman, B. (2017). Qut IELAB at CLEF eHealth 2017 technology assisted reviews track: initial experiments with learning to rank. In CEUR workshop proceedings: Working notes of CLEF 2017: Conference and labs of the evaluation forum (Vol. 1866, pp. Paper\u201398)."},{"issue":"1","key":"3648_CR53","doi-asserted-by":"publisher","first-page":"1","DOI":"10.2200\/S00429ED1V01Y201207AIM018","volume":"6","author":"B Settles","year":"2012","unstructured":"Settles, B. (2012). Active learning. Synthesis Lectures on Artificial Intelligence and Machine, Learning, 6(1), 1\u2013114.","journal-title":"Synthesis Lectures on Artificial Intelligence and Machine, Learning"},{"key":"3648_CR54","unstructured":"Singh, G., Marshall, I., Thomas, J., & Wallace, B. (2017). Identifying diagnostic test accuracy publications using a deep model. In CEUR workshop proceedings (Vol. 1866)."},{"key":"3648_CR55","unstructured":"Singh, J., & Thomas, L. (2017). Iiit-h at CLEF eHealth 2017 task 2: Technologically assisted reviews in empirical medicine. In CLEF (working notes)."},{"key":"3648_CR56","unstructured":"van Altena, A. J., & Olabarriaga, S. D. (2017). Predicting publication inclusion for diagnostic accuracy test reviews using random forests and topic modelling. In CLEF (working notes)."},{"key":"3648_CR57","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., & Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems (pp. 5998\u20136008)."},{"issue":"7","key":"3648_CR58","doi-asserted-by":"publisher","first-page":"663","DOI":"10.1038\/gim.2012.7","volume":"14","author":"BC Wallace","year":"2012","unstructured":"Wallace, B. C., Small, K., Brodley, C. E., Lau, J., Schmid, C. H., Bertram, L., et al. (2012). Toward modernizing the systematic review pipeline in genetics: Efficient updating via data mining. Genetics in Medicine, 14(7), 663.","journal-title":"Genetics in Medicine"},{"key":"3648_CR59","doi-asserted-by":"crossref","unstructured":"Wallace, B. C., Small, K., Brodley, C. E., & Trikalinos, T. A. (2010). Active learning for biomedical citation screening. In Proceedings of the 16th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 173\u2013182).","DOI":"10.1145\/1835804.1835829"},{"issue":"5","key":"3648_CR60","first-page":"7","volume":"4","author":"H Wu","year":"2018","unstructured":"Wu, H., Wang, T., Chen, J., Chen, S., Hu, Q., & He, L. (2018). ECNU at 2018 eHealth task 2: Technologically assisted reviews in empirical medicine. Methods, 4(5), 7.","journal-title":"Methods"},{"key":"3648_CR61","unstructured":"Yang, Y. Y., Lee, S. C., Chung, Y. A., Wu, T. E., Chen, S. A., & Lin, H. T. (2017). LIBACT: Pool-based active learning in python."},{"key":"3648_CR62","unstructured":"Yu, Z., & Menzies, T. (2017). Data balancing for technologically assisted reviews: Undersampling or reweighting. In CLEF (working notes)."}],"container-title":["Scientometrics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-020-03648-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11192-020-03648-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-020-03648-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,3]],"date-time":"2021-09-03T19:28:12Z","timestamp":1630697292000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11192-020-03648-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,3]]},"references-count":62,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["3648"],"URL":"https:\/\/doi.org\/10.1007\/s11192-020-03648-6","relation":{},"ISSN":["0138-9130","1588-2861"],"issn-type":[{"value":"0138-9130","type":"print"},{"value":"1588-2861","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,9,3]]},"assertion":[{"value":"1 October 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 September 2020","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}