{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,2]],"date-time":"2026-06-02T16:47:10Z","timestamp":1780418830925,"version":"3.54.1"},"reference-count":53,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2025,5,9]],"date-time":"2025-05-09T00:00:00Z","timestamp":1746748800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2025,5,31]]},"abstract":"<jats:p>Technology-Assisted Review aims to reduce the human effort required for screening processes such as abstract screening for Systematic Literature Reviews. Human reviewers label documents as relevant or irrelevant during this process, while the system incrementally updates a prediction model based on the reviewers\u2019 previous decisions. After each model update, the system proposes new documents it deems relevant, to prioritize relevant documents over irrelevant ones. A stopping criterion is necessary to guide users in stopping the review process to minimize the number of missed relevant documents and the number of read irrelevant documents. In this article, we propose and evaluate a new ensemble-based Active Learning strategy and a stopping criterion based on Chao\u2019s Population Size Estimator that estimates the prevalence of relevant documents in the dataset. Our simulation study demonstrates that this criterion performs well on several datasets and is compared to other methods presented in the literature.<\/jats:p>","DOI":"10.1145\/3724116","type":"journal-article","created":{"date-parts":[[2025,3,17]],"date-time":"2025-03-17T12:19:26Z","timestamp":1742213966000},"page":"1-51","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Using Chao\u2019s Estimator as a Stopping Criterion for Technology-Assisted Review"],"prefix":"10.1145","volume":"43","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4823-6085","authenticated-orcid":false,"given":"Michiel","family":"Bron","sequence":"first","affiliation":[{"name":"Department of Information and Computing Sciences, Faculty of Science, Utrecht University, Utrecht, The Netherlands and The Netherlands National Police, Den Haag (The Hague), The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3345-096X","authenticated-orcid":false,"given":"Peter G. M.","family":"van der Heijden","sequence":"additional","affiliation":[{"name":"Department of Methods and Statistics, Faculty of Social Sciences, Utrecht University, Utrecht, The Netherlands and University of Southampton, Southampton, United Kingdom of Great Britain and Northern Ireland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4525-1949","authenticated-orcid":false,"given":"Ad","family":"Feelders","sequence":"additional","affiliation":[{"name":"Department of Information and Computing Sciences, Faculty of Science, Utrecht University, Utrecht, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5108-7965","authenticated-orcid":false,"given":"Arno","family":"Siebes","sequence":"additional","affiliation":[{"name":"Department of Information and Computing Sciences, Faculty of Science, Utrecht University, Utrecht, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,5,9]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.5555\/69372"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF01002566"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v019.i05"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10651-020-00440-w"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.4324\/9781315151939"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.8308017"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.10887074"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.10887089"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/65.3.625"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.2307\/1936861"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-020-01521-4"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-021-01635-3"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.2307\/2531532"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1002\/0471667196.ess5051"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.2436\/20.8080.02.49"},{"key":"e_1_3_2_17_2","first-page":"1","article-title":"Die zweite form des fehlergesetzes","volume":"26","author":"Charlier Carl V. L.","year":"1905","unstructured":"Carl V. L. Charlier. 1905. Die zweite form des fehlergesetzes. Meddelanden Fran Lunds Astronomiska Observatorium Serie I 26 (1905), 1\u20138.","journal-title":"Meddelanden Fran Lunds Astronomiska Observatorium Serie I"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2005.06.023"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1197\/jamia.M1929"},{"key":"e_1_3_2_20_2","unstructured":"Gordon V. Cormack and Maura R. Grossman. 2015. Autonomy and reliability of continuous active learning for technology-assisted review. arXiv:1504.06868. Retrieved from https:\/\/arxiv.org\/abs\/1504.06868"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911510"},{"key":"e_1_3_2_22_2","volume-title":"Working Notes of CLEF 2017; Conference and Labs of the Evaluation Forum (CEUR Workshop Proceedings, Vol. 1866)","author":"Cormack Gordon V.","year":"2017","unstructured":"Gordon V. Cormack and Maura R. Grossman. 2017. Technology-assisted review in empirical medicine: Waterloo participation in CLEF eHealth 2017. In Working Notes of CLEF 2017; Conference and Labs of the Evaluation Forum (CEUR Workshop Proceedings, Vol. 1866), Linda Cappellato, Nicola Ferro, Lorraine Goeuriot, and Thomas Mandl (Eds.), CEUR-WS.org."},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.2307\/2532310"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.34894\/HE6NAQ"},{"key":"e_1_3_2_25_2","volume-title":"Likelihood: An Account of the Statistical Concept of Likelihood and Its Application to Scientific Inference","author":"Edwards A. W. F.","year":"1972","unstructured":"A. W. F. Edwards. 1972. Likelihood: An Account of the Statistical Concept of Likelihood and Its Application to Scientific Inference. University Press, Cambridge [Eng.]."},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.31219\/osf.io\/w6qbg"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.2307\/2987516"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDAR.1995.598994"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1002\/0471715816"},{"key":"e_1_3_2_30_2","first-page":"1","volume-title":"CEUR Workshop Proceedings","volume":"1866","author":"Kanoulas Evangelos","year":"2017","unstructured":"Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2017. CLEF 2017 technologically assisted reviews in empirical medicine overview. CEUR Workshop Proceedings 1866 (2017), 1\u201329."},{"key":"e_1_3_2_31_2","first-page":"1","volume-title":"CEUR Workshop Proceedings","volume":"2125","author":"Kanoulas Evangelos","year":"2018","unstructured":"Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2018. CLEF 2018 technologically assisted reviews in empirical medicine overview: 19th working notes of CLEF conference and labs of the evaluation forum, CLEF 2018. CEUR Workshop Proceedings 2125 (2018), 1\u201334."},{"key":"e_1_3_2_32_2","first-page":"9","volume-title":"CEUR Workshop Proceedings","volume":"2380","author":"Kanoulas Evangelos","year":"2019","unstructured":"Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2019. CLEF 2019 technology assisted reviews in empirical medicine overview. CEUR Workshop Proceedings 2380 (2019), 9\u201312."},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2008.06.001"},{"key":"e_1_3_2_34_2","first-page":"3149","article-title":"LightGBM: A highly efficient gradient boosting decision tree","volume":"30","author":"Ke Guolin","year":"2017","unstructured":"Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems. I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc, 3149\u20133157.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.abl7655"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.3390\/v12010107"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-14-58"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3482415"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411755"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cegh.2023.101485"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1201\/b17222"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1541-0420.2007.00779.x"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2011.03.008"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/130385.130417"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1002\/jrsm.1093"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.amjsurg.2012.11.017"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-020-00287-7"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1111\/1467-9574.00232"},{"key":"e_1_3_2_49_2","first-page":"14","volume-title":"KDD Workshop on Data Mining for Healthcare (KDDDMH\u201913)","author":"Wallace Byron C.","year":"2013","unstructured":"Byron C. Wallace, Issa J. Dahabreh, Kelly H. Moran, Carla E. Brodley, and Thomas A. Trikalinos. 2013. Active literature discovery for scoping evidence reviews. In KDD Workshop on Data Mining for Healthcare (KDDDMH\u201913), 14\u201319."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.2013.783881"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531663"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3469096.3469873"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-99736-6_34"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2018.11.021"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3724116","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3724116","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:18:59Z","timestamp":1750295939000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3724116"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,9]]},"references-count":53,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,5,31]]}},"alternative-id":["10.1145\/3724116"],"URL":"https:\/\/doi.org\/10.1145\/3724116","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,9]]},"assertion":[{"value":"2024-03-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-26","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}