{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,15]],"date-time":"2025-07-15T03:29:24Z","timestamp":1752550164797,"version":"3.41.0"},"reference-count":46,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2012,2,1]],"date-time":"2012-02-01T00:00:00Z","timestamp":1328054400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100002920","name":"Research Grants Council, University Grants Committee, Hong Kong","doi-asserted-by":"publisher","award":["2.01E+13"],"award-info":[{"award-number":["2.01E+13"]}],"id":[{"id":"10.13039\/501100002920","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["60873091, 61073140"],"award-info":[{"award-number":["60873091, 61073140"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Speech Lang. Process."],"published-print":{"date-parts":[[2012,2]]},"abstract":"<jats:p>\n            This article deals with pool-based active learning with uncertainty sampling. While existing uncertainty sampling methods emphasize selection of instances near the decision boundary to increase the likelihood of selecting informative examples, our position is that this heuristic is a surrogate for selecting examples for which the current learning algorithm iteration is likely to misclassify. To more directly model this intuition, this article augments such uncertainty sampling methods and proposes a simple\n            <jats:italic>instability<\/jats:italic>\n            -based selective sampling approach to improving uncertainty-based active learning, in which the instability degree of each unlabeled example is estimated during the learning process. Experiments on seven evaluation datasets show that instability-based sampling methods can achieve significant improvements over the traditional uncertainty sampling method. In terms of the average percentage of actively selected examples required for the learner to achieve 99% of its performance when training on the entire dataset, instability sampling and sampling by instability and density methods achieve better effectiveness in annotation cost reduction than random sampling and traditional entropy-based uncertainty sampling. Our experimental results have also shown that instability-based methods yield no significant improvement for active learning with SVMs when a popular sigmoidal function is used to transform SVM outputs to posterior probabilities.\n          <\/jats:p>","DOI":"10.1145\/2093153.2093154","type":"journal-article","created":{"date-parts":[[2012,2,22]],"date-time":"2012-02-22T18:42:36Z","timestamp":1329936156000},"page":"1-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["Uncertainty-based active learning with instability estimation for text classification"],"prefix":"10.1145","volume":"8","author":[{"given":"Jingbo","family":"Zhu","sequence":"first","affiliation":[{"name":"Northeastern University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matthew","family":"Ma","sequence":"additional","affiliation":[{"name":"Scientific Works, Princeton, NJ"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2012,2,20]]},"reference":[{"key":"e_1_2_1_1_1","first-page":"255","article-title":"Online choice of active learning algorithms","volume":"5","author":"Yoram B.","year":"2004","unstructured":"Yoram , B. , El-Yaniv , R. , and Luz , K. 2004 . Online choice of active learning algorithms . J. Mach. Learn. Res. , 5 , 255 -- 291 . Yoram, B., El-Yaniv, R., and Luz, K. 2004. Online choice of active learning algorithms. J. Mach. Learn. Res., 5, 255--291.","journal-title":"J. Mach. Learn. Res."},{"volume-title":"Proceedings of the ACL'09 Conference.","author":"Baldridge J.","key":"e_1_2_1_2_1","unstructured":"Baldridge , J. and Palmer , A . 2009. How well does active learning actually work&quest; Time-Based evaluation of cost-reduction strategies for language documentation . In Proceedings of the ACL'09 Conference. Baldridge, J. and Palmer, A. 2009. How well does active learning actually work&quest; Time-Based evaluation of cost-reduction strategies for language documentation. In Proceedings of the ACL'09 Conference."},{"volume-title":"Proceedings of the IJCAI'05 Conference. 991--996","author":"Becker M.","key":"e_1_2_1_3_1","unstructured":"Becker , M. and Osborne , M . 2005. A two-stage method for active learning of statistical grammars . In Proceedings of the IJCAI'05 Conference. 991--996 . Becker, M. and Osborne, M. 2005. A two-stage method for active learning of statistical grammars. In Proceedings of the IJCAI'05 Conference. 991--996."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.5555\/234285.234289"},{"volume-title":"Proceedings of the 13th Conference on Computational Natural Language Learning (CoNLL), 39--47","author":"Bloodgood M.","key":"e_1_2_1_5_1","unstructured":"Bloodgood , M. and Vijay-Shanker , K . 2009. A method for stopping active learning based on stabilizing predictions and the need for user-adjustable stopping . In Proceedings of the 13th Conference on Computational Natural Language Learning (CoNLL), 39--47 . Bloodgood, M. and Vijay-Shanker, K. 2009. A method for stopping active learning based on stabilizing predictions and the need for user-adjustable stopping. In Proceedings of the 13th Conference on Computational Natural Language Learning (CoNLL), 39--47."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.3115\/981732.981752"},{"volume-title":"Proceedings of the 45th Annual Meeting on Association for Computational Linguistics. 49--56","author":"Chan Y. S.","key":"e_1_2_1_7_1","unstructured":"Chan , Y. S. and Ng , H. T . 2007. Domain adaptation with active learning for word sense disambiguation . In Proceedings of the 45th Annual Meeting on Association for Computational Linguistics. 49--56 . Chan, Y. S. and Ng, H. T. 2007. Domain adaptation with active learning for word sense disambiguation. In Proceedings of the 45th Annual Meeting on Association for Computational Linguistics. 49--56."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.3115\/1220835.1220851"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1022673506211"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.5555\/1622737.1622744"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.5555\/1619410.1619452"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.5555\/1557690.1557708"},{"volume-title":"Proceedings of NIPS'08 Workshop on Cost-Sensitive Learning.","author":"Haertel R. A.","key":"e_1_2_1_13_1","unstructured":"Haertel , R. A. , Seppi , K. D. , Ringger , E. K. , and Carroll , J. L . 2008b. Return on investment for active learning . In Proceedings of NIPS'08 Workshop on Cost-Sensitive Learning. Haertel, R. A., Seppi, K. D., Ringger, E. K., and Carroll, J. L. 2008b. Return on investment for active learning. In Proceedings of NIPS'08 Workshop on Cost-Sensitive Learning."},{"volume-title":"Proceedings of the ACL'09 Conference. 181--189","author":"Haffari G.","key":"e_1_2_1_14_1","unstructured":"Haffari , G. and Sarkar , A . 2009. Active learning for multilingual statistical machine translation . In Proceedings of the ACL'09 Conference. 181--189 . Haffari, G. and Sarkar, A. 2009. Active learning for multilingual statistical machine translation. In Proceedings of the ACL'09 Conference. 181--189."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.3115\/1117794.1117800"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.3115\/1075671.1075730"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.3115\/1118693.1118699"},{"volume-title":"Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 3--12","author":"Lewis D. D.","key":"e_1_2_1_20_1","unstructured":"Lewis , D. D. and Gale , W. A . 1994. A sequential algorithm for training text classifiers . In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 3--12 . Lewis, D. D. and Gale, W. A. 1994. A sequential algorithm for training text classifiers. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 3--12."},{"volume-title":"Proceedings of 15th International Conference on Machine Learning. 350--358","author":"McCallum A.","key":"e_1_2_1_21_1","unstructured":"McCallum , A. and Nigam , K . 1998a. Employing EM in pool-based active learning for text classification . In Proceedings of 15th International Conference on Machine Learning. 350--358 . McCallum, A. and Nigam, K. 1998a. Employing EM in pool-based active learning for text classification. In Proceedings of 15th International Conference on Machine Learning. 350--358."},{"volume-title":"Proceedings of the AAAI'98 Workshop on Learning for Text Categorization.","author":"McCallum A.","key":"e_1_2_1_22_1","unstructured":"McCallum , A. and Nigam , K . 1998b. A comparison of event models for na\u00efve bayes text classification . In Proceedings of the AAAI'98 Workshop on Learning for Text Categorization. McCallum, A. and Nigam, K. 1998b. A comparison of event models for na\u00efve bayes text classification. In Proceedings of the AAAI'98 Workshop on Learning for Text Categorization."},{"key":"e_1_2_1_23_1","unstructured":"Milgram J. Cheriet M. and Sabourin R. 2006. One against one or one against all: Which one is better for handwriting recognition with SVMs&quest; In Proceedings of the 10th International Workshop on Frontiers in Handwriting Recognition.  Milgram J. Cheriet M. and Sabourin R. 2006. One against one or one against all: Which one is better for handwriting recognition with SVMs&quest; In Proceedings of the 10 th International Workshop on Frontiers in Handwriting Recognition."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.3115\/981863.981869"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.3115\/1075218.1075234"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1015330.1015349"},{"volume-title":"Advances in Large Margin Classifiers","author":"Platt J. C.","key":"e_1_2_1_27_1","unstructured":"Platt , J. C. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods . In Advances in Large Margin Classifiers . MIT Press . Platt, J. C. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers. MIT Press."},{"volume-title":"Proceedings of the ACL'07 Linguistic Annotation Workshop (LAW).","author":"Ringger E.","key":"e_1_2_1_28_1","unstructured":"Ringger , E. , McClanahan , P. , Haertel , R. , Busby , G. , Carmen , M. , Carroll , J. , Seppi , K. , and Lonsdale , D . 2007. Active learning for part-of-speech tagging: Accelerating corpus annotation . In Proceedings of the ACL'07 Linguistic Annotation Workshop (LAW). Ringger, E., McClanahan, P., Haertel, R., Busby, G., Carmen, M., Carroll, J., Seppi, K., and Lonsdale, D. 2007. Active learning for part-of-speech tagging: Accelerating corpus annotation. In Proceedings of the ACL'07 Linguistic Annotation Workshop (LAW)."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/11871842_40"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the 18th International Conference on Machine Learning. 441--448","author":"Nicholas R.","year":"2001","unstructured":"Nicholas , R. and McCallum , A. 2001 . Toward optimal active learning through sampling estimation of error reduction . In Proceedings of the 18th International Conference on Machine Learning. 441--448 . Nicholas, R. and McCallum, A. 2001. Toward optimal active learning through sampling estimation of error reduction. In Proceedings of the 18th International Conference on Machine Learning. 441--448."},{"volume-title":"Proceedings of the 14th International Conference Machine Learning. 322--330","author":"Schapire R. E.","key":"e_1_2_1_31_1","unstructured":"Schapire , R. E. , Freund , Y. , Bartlett , P. , and Lee , W. S . 1997. Boosting the margin: A new explanation for the effectiveness of voting methods . In Proceedings of the 14th International Conference Machine Learning. 322--330 . Schapire, R. E., Freund, Y., Bartlett, P., and Lee, W. S. 1997. Boosting the margin: A new explanation for the effectiveness of voting methods. In Proceedings of the 14th International Conference Machine Learning. 322--330."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-007-5019-5"},{"volume-title":"Active learning literature survey. Computer Science Tech. rep. 1648","author":"Settles B.","key":"e_1_2_1_33_1","unstructured":"Settles , B. 2009. Active learning literature survey. Computer Science Tech. rep. 1648 , University of Wisconsin-Madison. Settles, B. 2009. Active learning literature survey. Computer Science Tech. rep. 1648, University of Wisconsin-Madison."},{"volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1070--1079","author":"Settles B.","key":"e_1_2_1_34_1","unstructured":"Settles , B. and Craven , M . 2008. An analysis of active learning strategies for sequence labeling tasks . In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1070--1079 . Settles, B. and Craven, M. 2008. An analysis of active learning strategies for sequence labeling tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1070--1079."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/130385.130417"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.3115\/1218955.1219030"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073105"},{"volume-title":"Proceedings of the 16th International Conference on Machine Learning. 406--414","author":"Thompson C. A.","key":"e_1_2_1_38_1","unstructured":"Thompson , C. A. , Califf , M. E. , and Mooney , R. J . 1999. Active learning for natural language parsing and information extraction . In Proceedings of the 16th International Conference on Machine Learning. 406--414 . Thompson, C. A., Califf, M. E., and Mooney, R. J. 1999. Active learning for natural language parsing and information extraction. In Proceedings of the 16th International Conference on Machine Learning. 406--414."},{"volume-title":"Proceedings of the Joint Meeting of the Conference on Empirical Methods on Natural Language Processing and the Conference on Natural Language Learning. 486--495","author":"Tomanek K.","key":"e_1_2_1_39_1","unstructured":"Tomanek , K. , Wermter , J. , and Hahn , U . 2007. An approach to text corpus construction which cuts annotation costs and maintains reusability of annotated data . In Proceedings of the Joint Meeting of the Conference on Empirical Methods on Natural Language Processing and the Conference on Natural Language Learning. 486--495 . Tomanek, K., Wermter, J., and Hahn, U. 2007. An approach to text corpus construction which cuts annotation costs and maintains reusability of annotated data. In Proceedings of the Joint Meeting of the Conference on Empirical Methods on Natural Language Processing and the Conference on Natural Language Learning. 486--495."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1162\/153244302760185243"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.csl.2007.12.001"},{"volume-title":"Proceedings of the AAAI Spring Symposium on New Directions in Question Answering.","author":"Wiebe J.","key":"e_1_2_1_42_1","unstructured":"Wiebe , J. , Breck , E. , and Buckley , C . 2003. Recognizing and organizing opinions expressed in the world press . In Proceedings of the AAAI Spring Symposium on New Directions in Question Answering. Wiebe, J., Breck, E., and Buckley, C. 2003. Recognizing and organizing opinions expressed in the world press. In Proceedings of the AAAI Spring Symposium on New Directions in Question Answering."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.csl.2009.03.004"},{"volume-title":"Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. 783--790","author":"Zhu J.","key":"e_1_2_1_44_1","unstructured":"Zhu , J. and Hovy , E . 2007. Active learning for word sense disambiguation with methods for addressing the class imbalance problem . In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. 783--790 . Zhu, J. and Hovy, E. 2007. Active learning for word sense disambiguation with methods for addressing the class imbalance problem. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. 783--790."},{"volume-title":"Proceedings of the 22nd International Conference on Computational Linguistics. 1137--1144","author":"Zhu J.","key":"e_1_2_1_45_1","unstructured":"Zhu , J. , Wang , H. , Yao , T. , and Tsou , B . 2008a. Active learning with sampling by uncertainty and density for word sense disambiguation and text classification . In Proceedings of the 22nd International Conference on Computational Linguistics. 1137--1144 . Zhu, J., Wang, H., Yao, T., and Tsou, B. 2008a. Active learning with sampling by uncertainty and density for word sense disambiguation and text classification. In Proceedings of the 22nd International Conference on Computational Linguistics. 1137--1144."},{"volume-title":"Proceedings of the 22nd International Conference on Computational Linguistics. 1129--1136","author":"Zhu J.","key":"e_1_2_1_46_1","unstructured":"Zhu , J. , Wang , H. , and Hovy , E . 2008b. Multi-Criteria-Based strategy to stop active learning for data annotation . In Proceedings of the 22nd International Conference on Computational Linguistics. 1129--1136 . Zhu, J., Wang, H., and Hovy, E. 2008b. Multi-Criteria-Based strategy to stop active learning for data annotation. In Proceedings of the 22nd International Conference on Computational Linguistics. 1129--1136."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753783.1753784"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-00831-3_1"}],"container-title":["ACM Transactions on Speech and Language Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2093153.2093154","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2093153.2093154","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T09:48:45Z","timestamp":1750240125000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2093153.2093154"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,2]]},"references-count":46,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2012,2]]}},"alternative-id":["10.1145\/2093153.2093154"],"URL":"https:\/\/doi.org\/10.1145\/2093153.2093154","relation":{},"ISSN":["1550-4875","1550-4883"],"issn-type":[{"type":"print","value":"1550-4875"},{"type":"electronic","value":"1550-4883"}],"subject":[],"published":{"date-parts":[[2012,2]]},"assertion":[{"value":"2011-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2011-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2012-02-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}