{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,17]],"date-time":"2026-05-17T07:03:43Z","timestamp":1779001423764,"version":"3.51.4"},"reference-count":63,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2021,2,1]],"date-time":"2021-02-01T00:00:00Z","timestamp":1612137600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,2,1]],"date-time":"2021-02-01T00:00:00Z","timestamp":1612137600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"This project was funded by the Innovation Fund for IT in Research Projects, Utrecht University, The Netherlands."}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Nat Mach Intell"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>To help researchers conduct a systematic review or meta-analysis as efficiently and transparently as possible, we designed a tool to accelerate the step of screening titles and abstracts. For many tasks\u2014including but not limited to systematic reviews and meta-analyses\u2014the scientific literature needs to be checked systematically. Scholars and practitioners currently screen thousands of studies by hand to determine which studies to include in their review or meta-analysis. This is error prone and inefficient because of extremely imbalanced data: only a fraction of the screened studies is relevant. The future of systematic reviewing will be an interaction with machine learning algorithms to deal with the enormous increase of available text. We therefore developed an open source machine learning-aided pipeline applying active learning: ASReview. We demonstrate by means of simulation studies that active learning can yield far more efficient reviewing than manual reviewing while providing high quality. Furthermore, we describe the options of the free and open source research software and present the results from user experience tests. We invite the community to contribute to open source projects such as our own that provide measurable and reproducible improvements over current practice.<\/jats:p>","DOI":"10.1038\/s42256-020-00287-7","type":"journal-article","created":{"date-parts":[[2021,2,1]],"date-time":"2021-02-01T17:39:51Z","timestamp":1612201191000},"page":"125-133","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":892,"title":["An open source machine learning framework for efficient and transparent systematic reviews"],"prefix":"10.1038","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7736-2091","authenticated-orcid":false,"given":"Rens","family":"van de Schoot","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4297-0502","authenticated-orcid":false,"given":"Jonathan","family":"de Bruin","sequence":"additional","affiliation":[]},{"given":"Raoul","family":"Schram","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1610-3149","authenticated-orcid":false,"given":"Parisa","family":"Zahedi","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0531-3888","authenticated-orcid":false,"given":"Jan","family":"de Boer","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5150-1102","authenticated-orcid":false,"given":"Felix","family":"Weijdema","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5965-6560","authenticated-orcid":false,"given":"Bianca","family":"Kramer","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8353-0853","authenticated-orcid":false,"given":"Martijn","family":"Huijts","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1498-2052","authenticated-orcid":false,"given":"Maarten","family":"Hoogerwerf","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4998-3293","authenticated-orcid":false,"given":"Gerbrich","family":"Ferdinands","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7091-1147","authenticated-orcid":false,"given":"Albert","family":"Harkema","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7260-0828","authenticated-orcid":false,"given":"Joukje","family":"Willemsen","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4100-5468","authenticated-orcid":false,"given":"Yongchao","family":"Ma","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2689-6653","authenticated-orcid":false,"given":"Qixiang","family":"Fang","sequence":"additional","affiliation":[]},{"given":"Sybren","family":"Hindriks","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9940-9874","authenticated-orcid":false,"given":"Lars","family":"Tummers","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7467-2297","authenticated-orcid":false,"given":"Daniel L.","family":"Oberski","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,2,1]]},"reference":[{"key":"287_CR1","doi-asserted-by":"publisher","first-page":"2215","DOI":"10.1002\/asi.23329","volume":"66","author":"L Bornmann","year":"2015","unstructured":"Bornmann, L. & Mutz, R. Growth rates of modern science: a bibliometric analysis based on the number of publications and cited references. J. Assoc. Inf. Sci. Technol. 66, 2215\u20132222 (2015).","journal-title":"J. Assoc. Inf. Sci. Technol."},{"key":"287_CR2","unstructured":"Gough, D., Oliver, S. & Thomas, J. An Introduction to Systematic Reviews (Sage, 2017)."},{"key":"287_CR3","unstructured":"Cooper, H. Research Synthesis and Meta-analysis: A Step-by-Step Approach (SAGE Publications, 2015)."},{"key":"287_CR4","doi-asserted-by":"publisher","first-page":"e1","DOI":"10.1016\/j.jclinepi.2009.06.006","volume":"62","author":"A Liberati","year":"2009","unstructured":"Liberati, A. et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J. Clin. Epidemiol. 62, e1\u2013e34 (2009).","journal-title":"J. Clin. Epidemiol."},{"key":"287_CR5","unstructured":"Boaz, A. et al. Systematic Reviews: What have They Got to Offer Evidence Based Policy and Practice? (ESRC UK Centre for Evidence Based Policy and Practice London, 2002)."},{"key":"287_CR6","unstructured":"Oliver, S., Dickson, K. & Bangpan, M. Systematic Reviews: Making Them Policy Relevant. A Briefing for Policy Makers and Systematic Reviewers (UCL Institute of Education, 2015)."},{"key":"287_CR7","doi-asserted-by":"publisher","first-page":"98","DOI":"10.1136\/bmj.322.7278.98","volume":"322","author":"M Petticrew","year":"2001","unstructured":"Petticrew, M. Systematic reviews from astronomy to zoology: myths and misconceptions. Brit. Med. J. 322, 98\u2013101 (2001).","journal-title":"Brit. Med. J."},{"key":"287_CR8","doi-asserted-by":"publisher","unstructured":"Lefebvre, C., Manheimer, E. & Glanville, J. in Cochrane Handbook for Systematic Reviews of Interventions (eds. Higgins, J. P. & Green, S.) 95\u2013150 (John Wiley & Sons, 2008); https:\/\/doi.org\/10.1002\/9780470712184.ch6.","DOI":"10.1002\/9780470712184.ch6"},{"key":"287_CR9","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1002\/jrsm.42","volume":"2","author":"M Sampson","year":"2011","unstructured":"Sampson, M., Tetzlaff, J. & Urquhart, C. Precision of healthcare systematic review searches in a cross-sectional sample. Res. Synth. Methods 2, 119\u2013125 (2011).","journal-title":"Res. Synth. Methods"},{"key":"287_CR10","doi-asserted-by":"publisher","first-page":"e0227742","DOI":"10.1371\/journal.pone.0227742","volume":"15","author":"Z Wang","year":"2020","unstructured":"Wang, Z., Nayfeh, T., Tetzlaff, J., O\u2019Blenis, P. & Murad, M. H. Error rates of human reviewers during abstract screening in systematic reviews. PLoS ONE 15, e0227742 (2020).","journal-title":"PLoS ONE"},{"key":"287_CR11","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-019-1074-9","volume":"8","author":"IJ Marshall","year":"2019","unstructured":"Marshall, I. J. & Wallace, B. C. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Syst. Rev. 8, 163 (2019).","journal-title":"Syst. Rev."},{"key":"287_CR12","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-020-0897-3","volume":"20","author":"H Harrison","year":"2020","unstructured":"Harrison, H., Griffin, S. J., Kuhn, I. & Usher-Smith, J. A. Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation. BMC Med. Res. Methodol. 20, 7 (2020).","journal-title":"BMC Med. Res. Methodol."},{"key":"287_CR13","doi-asserted-by":"publisher","DOI":"10.1186\/2046-4053-4-5","volume":"4","author":"A O\u2019Mara-Eves","year":"2015","unstructured":"O\u2019Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M. & Ananiadou, S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst. Rev. 4, 5 (2015).","journal-title":"Syst. Rev."},{"key":"287_CR14","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1186\/1471-2105-11-55","volume":"11","author":"BC Wallace","year":"2010","unstructured":"Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C. & Schmid, C. H. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinf. 11, 55 (2010).","journal-title":"BMC Bioinf."},{"key":"287_CR15","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1197\/jamia.M1929","volume":"13","author":"AM Cohen","year":"2006","unstructured":"Cohen, A. M., Hersh, W. R., Peterson, K. & Yen, P.-Y. Reducing workload in systematic review preparation using automated citation classification. J. Am. Med. Inform. Assoc. 13, 206\u2013219 (2006).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"287_CR16","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1002\/widm.1132","volume":"4","author":"J Kremer","year":"2014","unstructured":"Kremer, J., Steenstrup Pedersen, K. & Igel, C. Active learning with support vector machines. WIREs Data Min. Knowl. Discov. 4, 313\u2013326 (2014).","journal-title":"WIREs Data Min. Knowl. Discov."},{"key":"287_CR17","doi-asserted-by":"publisher","first-page":"242","DOI":"10.1016\/j.jbi.2014.06.005","volume":"51","author":"M Miwa","year":"2014","unstructured":"Miwa, M., Thomas, J., O\u2019Mara-Eves, A. & Ananiadou, S. Reducing systematic review workload through certainty-based screening. J. Biomed. Inform. 51, 242\u2013253 (2014).","journal-title":"J. Biomed. Inform."},{"key":"287_CR18","unstructured":"Settles, B. Active Learning Literature Survey (Minds@UW, 2009); https:\/\/minds.wisconsin.edu\/handle\/1793\/60660"},{"key":"287_CR19","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1007\/s40708-016-0042-6","volume":"3","author":"A Holzinger","year":"2016","unstructured":"Holzinger, A. Interactive machine learning for health informatics: when do we need the human-in-the-loop? Brain Inform. 3, 119\u2013131 (2016).","journal-title":"Brain Inform."},{"key":"287_CR20","doi-asserted-by":"publisher","unstructured":"Van de Schoot, R. & De Bruin, J. Researcher-in-the-loop for Systematic Reviewing of Text Databases (Zenodo, 2020); https:\/\/doi.org\/10.5281\/zenodo.4013207","DOI":"10.5281\/zenodo.4013207"},{"key":"287_CR21","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1016\/j.ins.2018.10.006","volume":"477","author":"D Kim","year":"2019","unstructured":"Kim, D., Seo, D., Cho, S. & Kang, P. Multi-co-training for document classification using various document representations: TF\u2013IDF, LDA, and Doc2Vec. Inf. Sci. 477, 15\u201329 (2019).","journal-title":"Inf. Sci."},{"key":"287_CR22","doi-asserted-by":"publisher","first-page":"1422","DOI":"10.1126\/science.aab2374","volume":"348","author":"BA Nosek","year":"2015","unstructured":"Nosek, B. A. et al. Promoting an open research culture. Science 348, 1422\u20131425 (2015).","journal-title":"Science"},{"key":"287_CR23","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1197\/jamia.M2996","volume":"16","author":"H Kilicoglu","year":"2009","unstructured":"Kilicoglu, H., Demner-Fushman, D., Rindflesch, T. C., Wilczynski, N. L. & Haynes, R. B. Towards automatic recognition of scientifically rigorous clinical research evidence. J. Am. Med. Inform. Assoc. 16, 25\u201331 (2009).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"287_CR24","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1002\/jrsm.1378","volume":"11","author":"M Gusenbauer","year":"2020","unstructured":"Gusenbauer, M. & Haddaway, N. R. Which academic search systems are suitable for systematic reviews or meta\u2010analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Res. Synth. Methods 11, 181\u2013217 (2020).","journal-title":"Res. Synth. Methods"},{"key":"287_CR25","doi-asserted-by":"publisher","first-page":"e012545","DOI":"10.1136\/bmjopen-2016-012545","volume":"7","author":"R Borah","year":"2017","unstructured":"Borah, R., Brown, A. W., Capers, P. L. & Kaiser, K. A. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open 7, e012545 (2017).","journal-title":"BMJ Open"},{"key":"287_CR26","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1111\/padm.12209","volume":"94","author":"H de Vries","year":"2016","unstructured":"de Vries, H., Bekkers, V. & Tummers, L. Innovation in the Public Sector: a systematic review and future research agenda. Public Adm. 94, 146\u2013166 (2016).","journal-title":"Public Adm."},{"key":"287_CR27","doi-asserted-by":"publisher","unstructured":"Van de Schoot, R. et al. ASReview: Active Learning for Systematic Reviews (Zenodo, 2020); https:\/\/doi.org\/10.5281\/zenodo.3345592","DOI":"10.5281\/zenodo.3345592"},{"key":"287_CR28","doi-asserted-by":"publisher","unstructured":"De Bruin, J. et al. ASReview Software Documentation 0.14 (Zenodo, 2020); https:\/\/doi.org\/10.5281\/zenodo.4287120","DOI":"10.5281\/zenodo.4287120"},{"key":"287_CR29","unstructured":"ASReview PyPI Package (ASReview Core Development Team, 2020); https:\/\/pypi.org\/project\/asreview\/"},{"key":"287_CR30","unstructured":"Docker container for ASReview (ASReview Core Development Team, 2020); https:\/\/hub.docker.com\/r\/asreview\/asreview"},{"key":"287_CR31","doi-asserted-by":"publisher","unstructured":"Ferdinands, G. et al. Active Learning for Screening Prioritization in Systematic Reviews\u2014A Simulation Study (OSF Preprints, 2020); https:\/\/doi.org\/10.31219\/osf.io\/w6qbg","DOI":"10.31219\/osf.io\/w6qbg"},{"key":"287_CR32","doi-asserted-by":"crossref","unstructured":"Fu, J. H. & Lee, S. L. Certainty-enhanced active learning for improving imbalanced data classification. In 2011 IEEE 11th International Conference on Data Mining Workshops 405\u2013412 (IEEE, 2011).","DOI":"10.1109\/ICDMW.2011.43"},{"key":"287_CR33","unstructured":"Le, Q. V. & Mikolov, T. Distributed representations of sentences and documents. Preprint at https:\/\/arxiv.org\/abs\/1405.4053 (2014)."},{"key":"287_CR34","unstructured":"Ramos, J. Using TF\u2013IDF to determine word relevance in document queries. In Proc. 1st Instructional Conference on Machine Learning Vol. 242, 133\u2013142 (ICML, 2003)."},{"key":"287_CR35","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825\u20132830 (2011).","journal-title":"J. Mach. Learn. Res."},{"key":"287_CR36","doi-asserted-by":"crossref","unstructured":"Reimers, N. & Gurevych, I. Sentence-BERT: sentence embeddings using siamese BERT-networks Preprint at https:\/\/arxiv.org\/abs\/1908.10084 (2019).","DOI":"10.18653\/v1\/D19-1410"},{"key":"287_CR37","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-11-15","volume":"11","author":"V Smith","year":"2011","unstructured":"Smith, V., Devane, D., Begley, C. M. & Clarke, M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Med. Res. Methodol. 11, 15 (2011).","journal-title":"BMC Med. Res. Methodol."},{"key":"287_CR38","unstructured":"Wynants, L. et al. Prediction models for diagnosis and prognosis of COVID-19: systematic review and critical appraisal. Brit. Med. J. 369, 1328 (2020)."},{"key":"287_CR39","doi-asserted-by":"publisher","unstructured":"Van de Schoot, R. et al. Extension for COVID-19 Related Datasets in ASReview (Zenodo, 2020). https:\/\/doi.org\/10.5281\/zenodo.3891420.","DOI":"10.5281\/zenodo.3891420"},{"key":"287_CR40","unstructured":"Lu Wang, L. et al. CORD-19: The COVID-19 open research dataset. Preprint at https:\/\/arxiv.org\/abs\/2004.10706 (2020)."},{"key":"287_CR41","doi-asserted-by":"publisher","unstructured":"Fraser, N. & Kramer, B. Covid19_preprints (FigShare, 2020); https:\/\/doi.org\/10.6084\/m9.figshare.12033672.v18","DOI":"10.6084\/m9.figshare.12033672.v18"},{"key":"287_CR42","doi-asserted-by":"publisher","unstructured":"Ferdinands, G., Schram, R., Van de Schoot, R. & De Bruin, J. Scripts for \u2018ASReview: Open Source Software for Efficient and Transparent Active Learning for Systematic Reviews\u2019 (Zenodo, 2020); https:\/\/doi.org\/10.5281\/zenodo.4024122","DOI":"10.5281\/zenodo.4024122"},{"key":"287_CR43","doi-asserted-by":"publisher","unstructured":"Ferdinands, G., Schram, R., van de Schoot, R. & de Bruin, J. Results for \u2018ASReview: Open Source Software for Efficient and Transparent Active Learning for Systematic Reviews\u2019 (OSF, 2020); https:\/\/doi.org\/10.17605\/OSF.IO\/2JKD6","DOI":"10.17605\/OSF.IO\/2JKD6"},{"key":"287_CR44","doi-asserted-by":"publisher","first-page":"107","DOI":"10.3390\/v12010107","volume":"12","author":"KTT Kwok","year":"2020","unstructured":"Kwok, K. T. T., Nieuwenhuijse, D. F., Phan, M. V. T. & Koopmans, M. P. G. Virus metagenomics in farm animals: a systematic review. Viruses 12, 107 (2020).","journal-title":"Viruses"},{"key":"287_CR45","doi-asserted-by":"publisher","first-page":"1276","DOI":"10.1109\/TSE.2011.103","volume":"38","author":"T Hall","year":"2012","unstructured":"Hall, T., Beecham, S., Bowes, D., Gray, D. & Counsell, S. A systematic literature review on fault prediction performance in software engineering. IEEE Trans. Softw. Eng. 38, 1276\u20131304 (2012).","journal-title":"IEEE Trans. Softw. Eng."},{"key":"287_CR46","doi-asserted-by":"publisher","first-page":"451","DOI":"10.1080\/10705511.2016.1247646","volume":"24","author":"R van de Schoot","year":"2017","unstructured":"van de Schoot, R., Sijbrandij, M., Winter, S. D., Depaoli, S. & Vermunt, J. K. The GRoLTS-Checklist: guidelines for reporting on latent trajectory studies. Struct. Equ. Model. Multidiscip. J. 24, 451\u2013467 (2017).","journal-title":"Struct. Equ. Model. Multidiscip. J."},{"key":"287_CR47","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1080\/00273171.2017.1412293","volume":"53","author":"R van de Schoot","year":"2018","unstructured":"van de Schoot, R. et al. Bayesian PTSD-trajectory analysis with informed priors based on a systematic literature search and expert elicitation. Multivar. Behav. Res. 53, 267\u2013291 (2018).","journal-title":"Multivar. Behav. Res."},{"key":"287_CR48","doi-asserted-by":"crossref","unstructured":"Cohen, A. M., Bhupatiraju, R. T. & Hersh, W. R. Feature generation, feature selection, classifiers, and conceptual drift for biomedical document triage. In Proc. 13th Text Retrieval Conference (TREC, 2004).","DOI":"10.6028\/NIST.SP.500-261.genomics-ohsu.hersh"},{"key":"287_CR49","unstructured":"Vasalou, A., Ng, B. D., Wiemer-Hastings, P. & Oshlyansky, L. Human-moderated remote user testing: orotocols and applications. In 8th ERCIM Workshop, User Interfaces for All Vol. 19 (ERCIM, 2004)."},{"key":"287_CR50","unstructured":"Joffe, H. in Qualitative Research Methods in Mental Health and Psychotherapy: A Guide for Students and Practitioners (eds Harper, D. & Thompson, A. R.) Ch. 15 (Wiley, 2012)."},{"key":"287_CR51","unstructured":"NVivo v.12 (QSR International Pty, 2019)."},{"key":"287_CR52","doi-asserted-by":"publisher","unstructured":"Hindriks, S., Huijts, M. & van de Schoot, R. Data for UX-test ASReview - June 2020. OSF https:\/\/doi.org\/10.17605\/OSF.IO\/7PQNM (2020).","DOI":"10.17605\/OSF.IO\/7PQNM"},{"key":"287_CR53","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1093\/jamia\/ocv044","volume":"23","author":"IJ Marshall","year":"2016","unstructured":"Marshall, I. J., Kuiper, J. & Wallace, B. C. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. J. Am. Med. Inform. Assoc. 23, 193\u2013201 (2016).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"287_CR54","doi-asserted-by":"crossref","unstructured":"Nallapati, R., Zhou, B., dos Santos, C. N., Gulcehre, \u00c7. & Xiang, B. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proc. 20th SIGNLL Conference on Computational Natural Language Learning 280\u2013290 (Association for Computational Linguistics, 2016).","DOI":"10.18653\/v1\/K16-1028"},{"key":"287_CR55","unstructured":"Xie, Q., Dai, Z., Hovy, E., Luong, M.-T. & Le, Q. V. Unsupervised data augmentation for consistency training. Preprint at https:\/\/arxiv.org\/abs\/1904.12848 (2019)."},{"key":"287_CR56","doi-asserted-by":"publisher","first-page":"709","DOI":"10.1007\/s00778-019-00552-1","volume":"29","author":"A Ratner","year":"2020","unstructured":"Ratner, A. et al. Snorkel: rapid training data creation with weak supervision. VLDB J. 29, 709\u2013730 (2020).","journal-title":"VLDB J."},{"key":"287_CR57","unstructured":"Systematic Review Datasets (ASReview Core Development Team, 2020); https:\/\/github.com\/asreview\/systematic-review-datasets"},{"key":"287_CR58","doi-asserted-by":"crossref","unstructured":"Wallace, B. C., Small, K., Brodley, C. E., Lau, J. & Trikalinos, T. A. Deploying an interactive machine learning system in an evidence-based practice center: Abstrackr. In Proc. 2nd ACM SIGHIT International Health Informatics Symposium 819\u2013824 (Association for Computing Machinery, 2012).","DOI":"10.1145\/2110363.2110464"},{"key":"287_CR59","doi-asserted-by":"publisher","first-page":"762","DOI":"10.1111\/cobi.13117","volume":"32","author":"SH Cheng","year":"2018","unstructured":"Cheng, S. H. et al. Using machine learning to advance synthesis and use of conservation and environmental evidence. Conserv. Biol. 32, 762\u2013764 (2018).","journal-title":"Conserv. Biol."},{"key":"287_CR60","doi-asserted-by":"crossref","unstructured":"Yu, Z., Kraft, N. & Menzies, T. Finding better active learners for faster literature reviews. Empir. Softw. Eng. 23, 3161\u20133186 (2018).","DOI":"10.1007\/s10664-017-9587-0"},{"key":"287_CR61","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-016-0384-4","volume":"5","author":"M Ouzzani","year":"2016","unstructured":"Ouzzani, M., Hammady, H., Fedorowicz, Z. & Elmagarmid, A. Rayyan\u2014a web and mobile app for systematic reviews. Syst. Rev. 5, 210 (2016).","journal-title":"Syst. Rev."},{"key":"287_CR62","doi-asserted-by":"publisher","first-page":"470","DOI":"10.1002\/jrsm.1311","volume":"9","author":"P Przyby\u0142a","year":"2018","unstructured":"Przyby\u0142a, P. et al. Prioritising references for systematic reviews with RobotAnalyst: a user study. Res. Synth. Methods 9, 470\u2013488 (2018).","journal-title":"Res. Synth. Methods"},{"key":"287_CR63","unstructured":"ASReview: Active learning for Systematic Reviews (ASReview Core Development Team, 2020); https:\/\/github.com\/asreview\/asreview"}],"container-title":["Nature Machine Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s42256-020-00287-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s42256-020-00287-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s42256-020-00287-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,23]],"date-time":"2024-08-23T10:28:29Z","timestamp":1724408909000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s42256-020-00287-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,1]]},"references-count":63,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2021,2]]}},"alternative-id":["287"],"URL":"https:\/\/doi.org\/10.1038\/s42256-020-00287-7","relation":{},"ISSN":["2522-5839"],"issn-type":[{"value":"2522-5839","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,1]]},"assertion":[{"value":"4 June 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 December 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 February 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}