{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T22:04:23Z","timestamp":1767650663301,"version":"3.41.0"},"reference-count":35,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2017,10,25]],"date-time":"2017-10-25T00:00:00Z","timestamp":1508889600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2018,3,31]]},"abstract":"<jats:p>Active learning selectively labels the most informative instances, aiming to reduce the cost of data annotation. While much effort has been devoted to active sampling functions, relatively limited attention has been paid to when the learning process should stop. In this article, we focus on the stopping criterion of active learning and propose a model stability--based criterion, that is, when a model does not change with inclusion of additional training instances. The challenge lies in how to measure the model change without labeling additional instances and training new models. Inspired by the stochastic gradient update rule, we use the gradient of the loss function at each candidate example to measure its effect on model change. We propose to stop active learning when the model change brought by any of the remaining unlabeled examples is lower than a given threshold. We apply the proposed stopping criterion to two popular classifiers: logistic regression (LR) and support vector machines (SVMs). In addition, we theoretically analyze the stability and generalization ability of the model obtained by our stopping criterion. Substantial experiments on various UCI benchmark datasets and ImageNet datasets have demonstrated that the proposed approach is highly effective.<\/jats:p>","DOI":"10.1145\/3125645","type":"journal-article","created":{"date-parts":[[2017,10,26]],"date-time":"2017-10-26T14:19:33Z","timestamp":1509027573000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Stopping Criterion for Active Learning with Model Stability"],"prefix":"10.1145","volume":"9","author":[{"given":"Yexun","family":"Zhang","sequence":"first","affiliation":[{"name":"Shanghai Jiao Tong University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wenbin","family":"Cai","sequence":"additional","affiliation":[{"name":"Microsoft, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wenquan","family":"Wang","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ya","family":"Zhang","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2017,10,25]]},"reference":[{"volume-title":"Proceedings of the 15th International Conference on Machine Learning. 1--10","author":"Abe N.","key":"e_1_2_2_1_1"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-007-5019-5"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.3115\/1596374.1596384"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1093\/acprof:oso\/9780199535255.001.0001"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1162\/153244302760200704"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1009715923555"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2016.2542184"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2013.104"},{"volume-title":"Proceedings of the Conference on Computer Vision and Pattern Recognition. 248--255","author":"Deng J.","key":"e_1_2_2_9_1"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176345462"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2011.231"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2014.2307881"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSP.2004.830991"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.3115\/1599081.1599140"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4471-2099-5_1"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2339530.2339701"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2006.156"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-009-5131-9"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781107359949.008"},{"key":"e_1_2_2_20_1","first-page":"61","article-title":"Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods","volume":"10","author":"Platt J.","year":"1999","journal-title":"Advances in Large Margin Classifiers"},{"volume-title":"Proceedings of the 17th International Conference on Machine Learning. 839--846","author":"Schohn G.","key":"e_1_2_2_21_1"},{"key":"e_1_2_2_22_1","first-page":"127","article-title":"Active learning literature survey","volume":"39","author":"Settles B.","year":"2010","journal-title":"University of Wisconsin-- Madison"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/130385.130417"},{"key":"e_1_2_2_24_1","unstructured":"G. Sharma F. Jurie and P. Perez. 2014. Learning non-linear SVM in input space for image classification. HAL Archives 1--14.  G. Sharma F. Jurie and P. Perez. 2014. Learning non-linear SVM in input space for image classification. HAL Archives 1--14."},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1273496.1273598"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13042-010-0003-y"},{"volume-title":"Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. 120--127","author":"Tang M.","key":"e_1_2_2_27_1"},{"volume-title":"Proceedings of Joint Meeting of the Conference on Empirical Methods on Natural Language Processing and the Conference on Natural Language Learning. 486--495","author":"Tomanek K.","key":"e_1_2_2_28_1"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/500141.500159"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1162\/153244302760185243"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.csl.2007.12.001"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2011.2106794"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2014.99"},{"volume-title":"Proceedings of the 14th Advances in Neural Information Processing Systems. 1081--1088","author":"Zhu J.","key":"e_1_2_2_34_1"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753783.1753784"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3125645","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3125645","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T03:30:06Z","timestamp":1750217406000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3125645"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,10,25]]},"references-count":35,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2018,3,31]]}},"alternative-id":["10.1145\/3125645"],"URL":"https:\/\/doi.org\/10.1145\/3125645","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"type":"print","value":"2157-6904"},{"type":"electronic","value":"2157-6912"}],"subject":[],"published":{"date-parts":[[2017,10,25]]},"assertion":[{"value":"2017-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-10-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}