{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,8]],"date-time":"2026-01-08T23:56:32Z","timestamp":1767916592839,"version":"3.49.0"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2022,10,14]],"date-time":"2022-10-14T00:00:00Z","timestamp":1665705600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,10,14]],"date-time":"2022-10-14T00:00:00Z","timestamp":1665705600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001537","name":"University of Auckland","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001537","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2024,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Active learning allows machine learning models to be trained using fewer labels while retaining similar performance to traditional supervised learning. An active learner selects the most informative data points, requests their labels, and retrains itself. While this approach is promising, it raises the question of how to determine when the model is \u2018good enough\u2019 without the additional labels required for traditional evaluation. Previously, different stopping criteria have been proposed aiming to identify the optimal stopping point. Yet, optimality can only be expressed as a domain-dependent trade-off between accuracy and the number of labels, and no criterion is superior in all applications. As a further complication, a comparison of criteria for a particular real-world application would require practitioners to collect additional labelled data they are aiming to avoid by using active learning in the first place. This work enables practitioners to employ active learning by providing actionable recommendations for which stopping criteria are best for a given real-world scenario. We contribute the first large-scale comparison of stopping criteria for pool-based active learning, using a cost measure to quantify the accuracy\/label trade-off, public implementations of all stopping criteria we evaluate, and an open-source framework for evaluating stopping criteria. Our research enables practitioners to substantially reduce labelling costs by utilizing the stopping criterion which best suits their domain.<\/jats:p>","DOI":"10.1007\/s10994-022-06253-1","type":"journal-article","created":{"date-parts":[[2022,10,14]],"date-time":"2022-10-14T21:02:42Z","timestamp":1665781362000},"page":"1529-1547","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Hitting the target: stopping active learning at the cost-based optimum"],"prefix":"10.1007","volume":"113","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8819-6129","authenticated-orcid":false,"given":"Zac","family":"Pullar-Strecker","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Katharina","family":"Dost","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eibe","family":"Frank","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"J\u00f6rg","family":"Wicker","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,10,14]]},"reference":[{"key":"6253_CR1","unstructured":"Anguita, D., Ghio, A., Oneto, L., et\u00a0al. (2013). A public domain dataset for human activity recognition using smartphones. In Proceedings of the 21th international European symposium on artificial neural networks, computational intelligence and machine learning (pp. 437\u2013442)."},{"key":"6253_CR2","doi-asserted-by":"publisher","unstructured":"Beatty, G., Kochis, E., & Bloodgood, M. (2019). The use of unlabeled data versus labeled data for stopping active learning for text classification. In 2019 IEEE 13th international conference on semantic computer (ICSC) (pp. 287\u2013294). https:\/\/doi.org\/10.1109\/ICOSC.2019.8665546","DOI":"10.1109\/ICOSC.2019.8665546"},{"key":"6253_CR3","doi-asserted-by":"crossref","unstructured":"Bloodgood, M., & Vijay-Shanker, K. (2009). A method for stopping active learning based on stabilizing predictions and the need for user-adjustable stopping. In Proceedings of the 13th international conference on natural language processing. Linguistics, USA, CoNLL \u201909 (pp. 39\u201347).","DOI":"10.3115\/1596374.1596384"},{"issue":"1","key":"6253_CR4","first-page":"23","volume":"9","author":"H Blumen","year":"2016","unstructured":"Blumen, H., Fitch, K., & Polkus, V. (2016). Comparison of treatment costs for breast cancer, by tumor stage and type of service. Amer, Health and Drug Benefits, 9(1), 23\u201332.","journal-title":"Amer, Health and Drug Benefits"},{"issue":"1","key":"6253_CR5","doi-asserted-by":"publisher","first-page":"273","DOI":"10.1186\/s13643-020-01521-4","volume":"9","author":"MW Callaghan","year":"2020","unstructured":"Callaghan, M. W., & M\u00fcller-Hansen, F. (2020). Statistical stopping criteria for automated screening in systematic reviews. Systematic Reviews, 9(1), 273. https:\/\/doi.org\/10.1186\/s13643-020-01521-4.","journal-title":"Systematic Reviews"},{"key":"6253_CR6","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1016\/j.ins.2016.10.037","volume":"379","author":"TNC Cardoso","year":"2017","unstructured":"Cardoso, T. N. C., Silva, R. M., Canuto, S., et al. (2017). Ranked batch-mode active learning. Information Science, 379, 313\u2013337. https:\/\/doi.org\/10.1016\/j.ins.2016.10.037.","journal-title":"Information Science"},{"key":"6253_CR7","doi-asserted-by":"publisher","DOI":"10.5402\/2011\/164564","author":"R Chartrand","year":"2011","unstructured":"Chartrand, R. (2011). Numerical differentiation of noisy, nonsmooth data. ISRN Applied Mathematics. https:\/\/doi.org\/10.5402\/2011\/164564.","journal-title":"ISRN Applied Mathematics"},{"issue":"1","key":"6253_CR8","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/001316446002000104","volume":"20","author":"J Cohen","year":"1960","unstructured":"Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37\u201346. https:\/\/doi.org\/10.1177\/001316446002000104.","journal-title":"Educational and Psychological Measurement"},{"key":"6253_CR9","unstructured":"Danka, T., & Horvath, P. (2018). modAL: A modular active learning framework for Python. arXiv:1805.00979"},{"issue":"34","key":"6253_CR10","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1016\/j.engappai.2018.03.023","volume":"72","author":"C DeStefano","year":"2018","unstructured":"DeStefano, C., Maniaci, M., Fontanella, F., et al. (2018). Reliable writer identification in medieval manuscripts through page layout features. Engineering Applications of Artificial Intelligence, 72(34), 99\u2013110. https:\/\/doi.org\/10.1016\/j.engappai.2018.03.023.","journal-title":"Engineering Applications of Artificial Intelligence"},{"key":"6253_CR11","doi-asserted-by":"crossref","unstructured":"Dimitrakakis, C., & Savu-Krohn, C. (2008). Cost-minimising strategies for data labelling: Optimal stopping and active learning. In: Proceedings of the 5th international conference on foundation of information and knowledge, information, FoIKS\u201908 (pp. 96\u2013111). Springer.","DOI":"10.1007\/978-3-540-77684-0_9"},{"key":"6253_CR12","unstructured":"Dua, D., & Graff, C. (2017). UCI ml repository. http:\/\/archive.ics.uci.edu\/ml"},{"key":"6253_CR13","volume-title":"Pattern classification","author":"RO Duda","year":"2001","unstructured":"Duda, R. O., Hart, P. E., & Stork, D. G. (2001). Pattern classification (2nd ed.). Wiley.","edition":"2"},{"issue":"4","key":"6253_CR14","doi-asserted-by":"publisher","first-page":"727","DOI":"10.3233\/IDA-150742","volume":"19","author":"C Fu","year":"2015","unstructured":"Fu, C., & Yang, Y. (2015). Low density separation as a stopping criterion for active learning SVM. Intelligent Data Analysis, 19(4), 727\u2013741. https:\/\/doi.org\/10.3233\/IDA-150742.","journal-title":"Intelligent Data Analysis"},{"key":"6253_CR15","unstructured":"Ghayoomi, M. (2010). Using variance as a stopping criterion for active learning of frame assignment. In Proceedings of the NAACL HLT 2010 workshop on act learning for natural language processing assosiation for computer linguistics, Los Angeles, California (pp. 1\u20139)."},{"key":"6253_CR16","unstructured":"Gissin, D., & Shalev-Shwartz, S. (2019). Discriminative active learning. arXiv:1907.06347"},{"issue":"1","key":"6253_CR17","doi-asserted-by":"publisher","first-page":"216","DOI":"10.2214\/AJR.07.3608","volume":"192","author":"TM Haygood","year":"2009","unstructured":"Haygood, T. M., Wang, J., Atkinson, E. N., et al. (2009). Timed efficiency of interpretation of digital and film-screen screening mammograms. AJR American Journal of Roentgenology, 192(1), 216\u2013220. https:\/\/doi.org\/10.2214\/AJR.07.3608.","journal-title":"AJR American Journal of Roentgenology"},{"key":"6253_CR18","unstructured":"Ishibashi, H., & Hino, H. (2020). Stopping criterion for active learning based on deterministic generalization bounds. In International PMLR: conference on artificial intelligence and statistics (pp. 386\u2013397)."},{"key":"6253_CR19","unstructured":"Ishibashi, H., & Hino, H. (2021) Stopping criterion for active learning based on error stability. arXiv:2104.01836"},{"issue":"6971","key":"6253_CR20","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1038\/nature02236","volume":"427","author":"Ross D. King","year":"2004","unstructured":"King, Ross D.., Whelan, Kenneth E.., Jones, Ffion M.., Reiser, Philip G. K.., Bryant, Christopher H.., Muggleton, Stephen H.., et al. (2004). Functional genomic hypothesis generation and experimentation by a robot scientist. Nature, 427(6971), 247\u2013252. https:\/\/doi.org\/10.1038\/nature02236.","journal-title":"Nature"},{"key":"6253_CR21","unstructured":"Lang, K., & Baum, E. B. (1992) Query learning can work poorly when a human oracle is used. In Proceedings of the international joint conference on neural networks (Vol.\u00a08, pp. 335\u2013340). IEEE Press."},{"key":"6253_CR22","doi-asserted-by":"crossref","unstructured":"Laws, F., & Sch\u00e4tze, H. (2008). Stopping criteria for active learning of named entity recognition. In Proceedings of the 22nd International Conference on Computing Linguistics. Assocation for Computer Linguistics, USA, COLING \u201908 (pp. 465\u2013472).","DOI":"10.3115\/1599081.1599140"},{"issue":"Apr","key":"6253_CR23","first-page":"361","volume":"5","author":"DD Lewis","year":"2004","unstructured":"Lewis, D. D., Yang, Y., Rose, T. G., et al. (2004). RCV1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5(Apr), 361\u2013397.","journal-title":"Journal of Machine Learning Research"},{"issue":"102","key":"6253_CR24","doi-asserted-by":"publisher","first-page":"174","DOI":"10.1016\/j.strusafe.2021.102174","volume":"96","author":"M Moustapha","year":"2022","unstructured":"Moustapha, M., Marelli, S., & Sudret, B. (2022). Active learning for structural reliability: Survey, general framework and benchmark. Structural Safety, 96(102), 174. https:\/\/doi.org\/10.1016\/j.strusafe.2021.102174.","journal-title":"Structural Safety"},{"key":"6253_CR25","unstructured":"Oliver, A., Odena, A., & Raffel, C., et\u00a0al. (2018). Realistic evaluation of deep semi-supervised learning algorithms. In Advances in neural information processing systems, NIPS\u201918 (pp. 3239\u20133250). Curran Associates Inc."},{"issue":"1","key":"6253_CR26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.2200\/S00429ED1V01Y201207AIM018","volume":"6","author":"B Settles","year":"2012","unstructured":"Settles, B. (2012). Active learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 6(1), 1\u2013114. https:\/\/doi.org\/10.2200\/S00429ED1V01Y201207AIM018.","journal-title":"Synthesis Lectures on Artificial Intelligence and Machine Learning"},{"issue":"3","key":"6253_CR27","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1016\/j.csl.2007.12.001","volume":"22","author":"A Vlachos","year":"2008","unstructured":"Vlachos, A. (2008). A stopping criterion for active learning. Computer Speech Language, 22(3), 295\u2013312. https:\/\/doi.org\/10.1016\/j.csl.2007.12.001.","journal-title":"Computer Speech Language"},{"key":"6253_CR28","doi-asserted-by":"publisher","DOI":"10.1007\/s00158-020-02622-3","author":"J Yi","year":"2020","unstructured":"Yi, J., Zhou, Q., Cheng, Y., et al. (2020). Efficient adaptive Kriging-based reliability analysis combining new learning function and error-based stopping criterion. Structural and Multidisciplinary Optimization. https:\/\/doi.org\/10.1007\/s00158-020-02622-3.","journal-title":"Structural and Multidisciplinary Optimization"},{"key":"6253_CR29","unstructured":"Zhu, J., & Hovy, E. (2007) Active learning for word sense disambiguation with methods for addressing the class imbalance problem. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and computer natural language learning (EMNLP-CoNLL). Association for Computing Linguistics, Prague, Czech Republic (pp. 783\u2013790)."},{"key":"6253_CR30","doi-asserted-by":"crossref","unstructured":"Zhu, J., Wang, H., & Hovy, E. (2008a). Learning a stopping criterion for active learning for word sense disambiguation and text classification. In Proceedings of the 3rd international joint conference on natural language processing (Vol. I).","DOI":"10.3115\/1599081.1599224"},{"key":"6253_CR31","doi-asserted-by":"crossref","unstructured":"Zhu, J., Wang H, Hovy E (2008b) Multi-criteria-based strategy to stop active learning for data annotation. In Proceedings of the 22nd international conference on computing linguistics-Volume 1. Association for computing linguistics, USA, COLING \u201908 (pp. 1129\u20131136).","DOI":"10.3115\/1599081.1599223"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-022-06253-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-022-06253-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-022-06253-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,28]],"date-time":"2024-03-28T17:08:17Z","timestamp":1711645697000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-022-06253-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,14]]},"references-count":31,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,4]]}},"alternative-id":["6253"],"URL":"https:\/\/doi.org\/10.1007\/s10994-022-06253-1","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,14]]},"assertion":[{"value":"26 May 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 August 2022","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 September 2022","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 October 2022","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"We do not declare any conflicts.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"Not applicable.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}}]}}