{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T05:13:30Z","timestamp":1784178810490,"version":"3.55.0"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2023,8,28]],"date-time":"2023-08-28T00:00:00Z","timestamp":1693180800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,8,28]],"date-time":"2023-08-28T00:00:00Z","timestamp":1693180800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Data Min Knowl Disc"],"published-print":{"date-parts":[[2024,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In high recall retrieval tasks, human experts review a large pool of documents with the goal of satisfying an information need. Documents are prioritized for review through an active learning policy, and the process is usually referred to as Technology-Assisted Review (TAR). TAR tasks also aim to stop the review process once the target recall is achieved to minimize the annotation cost. In this paper, we introduce a new stopping rule called SAL<jats:inline-formula><jats:alternatives><jats:tex-math>$$_\\tau ^R$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:msubsup><mml:mrow\/><mml:mi>\u03c4<\/mml:mi><mml:mi>R<\/mml:mi><\/mml:msubsup><\/mml:math><\/jats:alternatives><\/jats:inline-formula>(SLD for Active Learning), a modified version of the Saerens\u2013Latinne\u2013Decaestecker algorithm (SLD) that has been adapted for use in active learning. Experiments show that our algorithm stops the review well ahead of the current state-of-the-art methods, while providing the same guarantees of achieving the target recall.<\/jats:p>","DOI":"10.1007\/s10618-023-00961-5","type":"journal-article","created":{"date-parts":[[2023,8,28]],"date-time":"2023-08-28T17:06:01Z","timestamp":1693242361000},"page":"535-568","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["SAL\u03c4: efficiently stopping TAR by improving priors estimates"],"prefix":"10.1007","volume":"38","author":[{"given":"Alessio","family":"Molinari","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5725-4322","authenticated-orcid":false,"given":"Andrea","family":"Esuli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,8,28]]},"reference":[{"key":"961_CR1","unstructured":"Cormack GV, Grossman MR (2020) Systems and methods for a scalable continuous active learning approach to information classification. Google Patents. US Patent 10:671\u2013675"},{"issue":"1","key":"961_CR3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13643-020-01521-4","volume":"9","author":"MW Callaghan","year":"2020","unstructured":"Callaghan MW, M\u00fcller-Hansen F (2020) Statistical stopping criteria for automated screening in systematic reviews. Syst Rev 9(1):1\u201314","journal-title":"Syst Rev"},{"key":"961_CR4","doi-asserted-by":"crossref","unstructured":"Cormack GV, Grossman MR (2015) Autonomy and reliability of continuous active learning for technology-assisted review. CoRR abs\/1504.06868","DOI":"10.1145\/2766462.2767771"},{"key":"961_CR5","doi-asserted-by":"publisher","unstructured":"Cormack GV, Grossman MR (2016) Engineering quality and reliability in technology-assisted review. In: Proceedings of the 39th ACM conference on research and development in information retrieval (SIGIR 2016), Tokyo, JP, pp 75\u201384. https:\/\/doi.org\/10.1145\/2911451.2911510","DOI":"10.1145\/2911451.2911510"},{"key":"961_CR6","doi-asserted-by":"publisher","unstructured":"Cormack GV, Grossman MR (2016) Scalability of continuous active learning for reliable high-recall text classification. In: Proceedings of the 25th ACM conference on information and knowledge management (CIKM 2016), pp 1039\u20131048. https:\/\/doi.org\/10.1145\/2983323.2983776","DOI":"10.1145\/2983323.2983776"},{"key":"961_CR7","doi-asserted-by":"crossref","unstructured":"Cormack GV, Grossman MR, Hedin B, Oard DW (2010) Overview of the TREC 2010 legal track. In: Proceedings of the 19th text retrieval conference (TREC 2010)","DOI":"10.6028\/NIST.SP.500-294.legal-overview"},{"key":"961_CR8","doi-asserted-by":"crossref","unstructured":"Dasgupta S, Hsu D (2008) Hierarchical sampling for active learning. In: Proceedings of the 25th international conference on machine learning (ICML 2018), Stockholm, SE, pp 208\u2013215","DOI":"10.1145\/1390156.1390183"},{"key":"961_CR9","unstructured":"Esuli A, Molinari A, Sebastiani F (2022) Active learning and the Saerens\u2013Latinne\u2013Decaestecker algorithm: an evaluation. In: CIRCLE 2022: 2nd joint conference of the information retrieval communities in Europe"},{"issue":"3","key":"961_CR10","first-page":"5","volume":"17","author":"MR Grossman","year":"2011","unstructured":"Grossman MR, Cormack GV (2011) Technology-assisted review in e-discovery can be more effective and more efficient than exhaustive manual review. Richmond J Law Technol 17(3):5","journal-title":"Richmond J Law Technol"},{"issue":"10","key":"961_CR11","doi-asserted-by":"publisher","first-page":"1936","DOI":"10.1109\/TPAMI.2014.2307881","volume":"36","author":"S Huang","year":"2014","unstructured":"Huang S, Jin R, Zhou Z (2014) Active learning by querying informative and representative examples. IEEE Trans Pattern Anal Mach Intell 36(10):1936\u20131949. https:\/\/doi.org\/10.1109\/TPAMI.2014.2307881","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"961_CR12","unstructured":"Kanoulas E, Li D, Azzopardi L, Spijker R (2019) CLEF 2019 technology assisted reviews in empirical medicine overview. In: Working notes of the conference and labs of the evaluation forum (CLEF 2019), Lugano, CH"},{"key":"961_CR13","unstructured":"Konyushkova K, Sznitman R, Fua P (2017)Learning active learning from data. Adv Neural Inf Process Syst 30"},{"key":"961_CR14","unstructured":"Krishnan R, Sinha A, Ahuja N, Subedar M, Tickoo O, Iyer R (2021) Mitigating sampling bias and improving robustness in active learning. arXiv preprint arXiv:2109.06321"},{"key":"961_CR15","unstructured":"Lease M, Cormack GV, Nguyen AT, Trikalinos TA, Wallace BC (2016) Systematic review is e-discovery in doctor\u2019s clothing. In: Proceedings of the SIGIR 2016 medical information retrieval workshop (MedIR 2016), Pisa, IT"},{"key":"961_CR16","doi-asserted-by":"publisher","unstructured":"Lewis DD, Gale WA (1994) A sequential algorithm for training text classifiers. In: Proceedings of the 17th ACM international conference on research and development in information retrieval (SIGIR 1994), Dublin, IE, pp 3\u201312. https:\/\/doi.org\/10.1007\/978-1-4471-2099-5_1","DOI":"10.1007\/978-1-4471-2099-5_1"},{"key":"961_CR17","first-page":"361","volume":"5","author":"DD Lewis","year":"2004","unstructured":"Lewis DD, Yang Y, Rose TG, Li F (2004) RCV1: a new benchmark collection for text categorization research. J Mach Learn Res 5:361\u2013397","journal-title":"J Mach Learn Res"},{"key":"961_CR18","doi-asserted-by":"publisher","unstructured":"Lewis DD, Yang E, Frieder O (2021) Certifying one-phase technology-assisted reviews. In: Proceedings of the 30th ACM international conference on information and knowledge management. CIKM \u201921. Association for Computing Machinery, New York, NY, USA, pp 893\u2013902. https:\/\/doi.org\/10.1145\/3459637.3482415","DOI":"10.1145\/3459637.3482415"},{"issue":"4","key":"961_CR19","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1145\/3411755","volume":"38","author":"D Li","year":"2020","unstructured":"Li D, Kanoulas E (2020) When to stop reviewing in technology-assisted reviews: sampling from an adaptive distribution to estimate residual relevant documents. ACM Trans Inf Syst 38(4):41\u201314136. https:\/\/doi.org\/10.1145\/3411755","journal-title":"ACM Trans Inf Syst"},{"key":"961_CR20","doi-asserted-by":"publisher","DOI":"10.1016\/j.conctc.2019.100443","volume":"16","author":"M Michelson","year":"2019","unstructured":"Michelson M, Reuter K (2019) The significant cost of systematic reviews and meta-analyses: a call for greater involvement of machine learning to assess the promise of clinical trials. Contemp Clin Trials Commun 16:100443","journal-title":"Contemp Clin Trials Commun"},{"key":"961_CR21","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.7142640","author":"A Molinari","year":"2022","unstructured":"Molinari A (2022) CLEF EMED 2019 dataset. Zenodo. https:\/\/doi.org\/10.5281\/zenodo.7142640","journal-title":"Zenodo"},{"key":"961_CR22","doi-asserted-by":"crossref","unstructured":"Molinari A, Esuli A, Sebastiani F (2023) Improved risk minimization algorithms for technology-assisted review","DOI":"10.1016\/j.iswa.2023.200209"},{"issue":"1","key":"961_CR23","doi-asserted-by":"publisher","first-page":"521","DOI":"10.1016\/j.patcog.2011.06.019","volume":"45","author":"JG Moreno-Torres","year":"2012","unstructured":"Moreno-Torres JG, Raeder T, Ala\u0131z-Rodr\u0131guez R, Chawla NV, Herrera F (2012) A unifying view on dataset shift in classification. Pattern Recognit 45(1):521\u2013530. https:\/\/doi.org\/10.1016\/j.patcog.2011.06.019","journal-title":"Pattern Recognit"},{"issue":"5","key":"961_CR24","first-page":"1","volume":"4","author":"A O\u2019Mara-Eves","year":"2015","unstructured":"O\u2019Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S (2015) Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev 4(5):1\u201322","journal-title":"Syst Rev"},{"issue":"1","key":"961_CR25","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1145\/3268928","volume":"37","author":"DW Oard","year":"2018","unstructured":"Oard DW, Sebastiani F, Vinjumur JK (2018) Jointly minimizing the expected costs of review for responsiveness and privilege in e-discovery. ACM Trans Inf Syst 37(1):11\u201311135. https:\/\/doi.org\/10.1145\/3268928","journal-title":"ACM Trans Inf Syst"},{"issue":"1","key":"961_CR26","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1162\/089976602753284446","volume":"14","author":"M Saerens","year":"2002","unstructured":"Saerens M, Latinne P, Decaestecker C (2002) Adjusting the outputs of a classifier to new a priori probabilities: a simple procedure. Neural Comput 14(1):21\u201341. https:\/\/doi.org\/10.1162\/089976602753284446","journal-title":"Neural Comput"},{"key":"961_CR2","doi-asserted-by":"crossref","unstructured":"Satopaa V, Albrecht J, Irwin D, Raghavan B (2011) Finding a\" kneedle\" in a haystack: Detecting knee points in system behavior. In: 2011 31st International conference on distributed computing systems workshops, pp 166\u2013171. IEEE","DOI":"10.1109\/ICDCSW.2011.20"},{"issue":"140","key":"961_CR27","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13643-016-0315-4","volume":"5","author":"I Shemilt","year":"2016","unstructured":"Shemilt I, Khan N, Park S, Thomas J (2016) Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Syst Rev 5(140):1\u201313. https:\/\/doi.org\/10.1186\/s13643-016-0315-4","journal-title":"Syst Rev"},{"key":"961_CR28","doi-asserted-by":"crossref","unstructured":"Sneyd A, Stevenson M (2021) Stopping criteria for technology assisted reviews based on counting processes. In: Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pp 2293\u20132297","DOI":"10.1145\/3404835.3463013"},{"key":"961_CR29","doi-asserted-by":"crossref","unstructured":"Wang S, Scells H, Mourad A, Zuccon G (2022) Seed-driven document ranking for systematic reviews: a reproducibility study. In: European conference on information retrieval, pp 686\u2013700. Springer","DOI":"10.1007\/978-3-030-99736-6_46"},{"key":"961_CR30","doi-asserted-by":"publisher","unstructured":"Yang E, Lewis DD, Frieder O (2021) Heuristic stopping rules for technology-assisted review. In: Proceedings of the 21st ACM symposium on document engineering (DocEng 2021), Limerick, IE, pp 31\u201313110. https:\/\/doi.org\/10.1145\/3469096.3469873","DOI":"10.1145\/3469096.3469873"},{"key":"961_CR31","doi-asserted-by":"crossref","unstructured":"Yang E, Lewis DD, Frieder O (2021) On minimizing cost in legal document review workflows. In: Proceedings of the 21st ACM symposium on document engineering, pp 1\u201310","DOI":"10.1145\/3469096.3469872"}],"container-title":["Data Mining and Knowledge Discovery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-023-00961-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10618-023-00961-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-023-00961-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,26]],"date-time":"2024-10-26T23:50:42Z","timestamp":1729986642000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10618-023-00961-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,28]]},"references-count":31,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,3]]}},"alternative-id":["961"],"URL":"https:\/\/doi.org\/10.1007\/s10618-023-00961-5","relation":{},"ISSN":["1384-5810","1573-756X"],"issn-type":[{"value":"1384-5810","type":"print"},{"value":"1573-756X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,28]]},"assertion":[{"value":"10 November 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 July 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 August 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Authors declare to have no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}}]}}