{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,17]],"date-time":"2026-04-17T07:29:37Z","timestamp":1776410977150,"version":"3.51.2"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,1,12]],"date-time":"2024-01-12T00:00:00Z","timestamp":1705017600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2024,4,30]]},"abstract":"<jats:p>When active learning (AL) is applied to help users develop a model on a large dataset through interactively presenting data instances for labeling, existing AL techniques often suffer from two main drawbacks: First, to reach high accuracy they may require the user to label hundreds of data instances, which is an onerous task for the user. Second, retrieving the next instance to label from a large dataset can be time-consuming, making it incompatible with the interactive nature of the human exploration process. To address these issues, we introduce a novel version-space-based active learner for kernel classifiers, which possesses strong theoretical guarantees on performance and efficient implementation in time and space. In addition, by leveraging additional insights obtained in the user labeling process, we can factorize the version space to perform active learning in a set of subspaces, which further reduces the user labeling effort. Evaluation results show that our algorithms significantly outperform state-of-the-art version space strategies, as well as a recent factorization-aware algorithm, for model development over large datasets.<\/jats:p>","DOI":"10.1145\/3637443","type":"journal-article","created":{"date-parts":[[2023,12,15]],"date-time":"2023-12-15T11:27:11Z","timestamp":1702639631000},"page":"1-49","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Efficient Version Space Algorithms for Human-in-the-loop Model Development"],"prefix":"10.1145","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2400-5602","authenticated-orcid":false,"given":"Luciano Di","family":"Palma","sequence":"first","affiliation":[{"name":"Ecole Polytechnique, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-7793-9671","authenticated-orcid":false,"given":"Yanlei","family":"Diao","sequence":"additional","affiliation":[{"name":"Ecole Polytechnique, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9166-7908","authenticated-orcid":false,"given":"Anna","family":"Liu","sequence":"additional","affiliation":[{"name":"University of Massachusetts Amherst, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,1,12]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/1102351.1102356"},{"key":"e_1_3_3_3_2","first-page":"273","volume-title":"International Conference on Machine Learning","volume":"70","author":"Bach Stephen H.","year":"2017","unstructured":"Stephen H. Bach, Bryan He, Alexander Ratner, and Christopher R\u00e9. 2017. Learning the structure of generative models without labeled data. In International Conference on Machine Learning, Vol. 70. 273\u2013282."},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1287\/moor.18.2.255"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1287\/opre.29.6.1039"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jsv.2018.08.040"},{"key":"e_1_3_3_7_2","first-page":"160","volume-title":"International Conference on Machine Learning","volume":"28","author":"Chen Yuxin","year":"2013","unstructured":"Yuxin Chen and Andreas Krause. 2013. Near-optimal batch mode active learning and adaptive submodular optimization. In International Conference on Machine Learning, Vol. 28. 160\u2013168."},{"key":"e_1_3_3_8_2","first-page":"337","volume-title":"Conference on Advances in Neural Information Processing Systems","author":"Dasgupta Sanjoy","year":"2005","unstructured":"Sanjoy Dasgupta. 2005. Analysis of a greedy active learning strategy. In Conference on Advances in Neural Information Processing Systems. 337\u2013344."},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0122670"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2019.00117"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2016.2599168"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2610523"},{"key":"e_1_3_3_13_2","first-page":"443","volume-title":"Conference on Advances in Neural Information Processing Systems 18","author":"Gilad-Bachrach Ran","year":"2006","unstructured":"Ran Gilad-Bachrach, Amir Navot, and Naftali Tishby. 2006. Query-by-committee made real. In Conference on Advances in Neural Information Processing Systems 18. 443\u2013450."},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF01583776"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.3278"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.5555\/2567709.2567744"},{"key":"e_1_3_3_17_2","doi-asserted-by":"crossref","first-page":"3951","DOI":"10.21437\/Interspeech.2017-406","volume-title":"INTERSPEECH","author":"Hantke Simone","year":"2017","unstructured":"Simone Hantke, Zixing Zhang, and Bj\u00f6rn W. Schuller. 2017. Towards intelligent crowdsourcing for audio data annotation: Integrating active learning in the real world. In INTERSPEECH. 3951\u20133955."},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.3389\/fninf.2018.00097"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.14778\/3275536.3275542"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33014039"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.2307\/2965732"},{"key":"e_1_3_3_22_2","first-page":"1915","volume-title":"International Conference on Machine Learning","volume":"70","author":"Krishnamurthy Akshay","year":"2017","unstructured":"Akshay Krishnamurthy, Alekh Agarwal, Tzu-Kuo Huang, Hal Daum\u00e9, and John Langford. 2017. Active learning for cost-sensitive classification. In International Conference on Machine Learning, Vol. 70. 1915\u20131924."},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.2041-210X.2011.00131.x"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1080\/03610926.2017.1363231"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611970203"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1007\/s101070050099"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177692644"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(82)90040-6"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.5555\/2371238"},{"key":"e_1_3_3_30_2","volume-title":"Human-in-the-Loop Machine Learning: Active Learning and Annotation for Human-centered AI","author":"Monarch Robert M.","year":"2021","unstructured":"Robert M. Monarch. 2021. Human-in-the-Loop Machine Learning: Active Learning and Annotation for Human-centered AI. Manning Publications."},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2017.10.026"},{"key":"e_1_3_3_32_2","first-page":"3567","volume-title":"Conference on Advances in Neural Information Processing Systems 29","author":"Ratner Alexander J.","year":"2016","unstructured":"Alexander J. Ratner, Christopher M. De Sa, Sen Wu, Daniel Selsam, and Christopher R\u00e9. 2016. Data programming: Creating large training sets, quickly. In Conference on Advances in Neural Information Processing Systems 29. 3567\u20133575."},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1016\/0097-3165(72)90019-2"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-01560-1"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/130385.130417"},{"key":"e_1_3_3_36_2","first-page":"911","volume-title":"International Conference on Machine Learning (ICML)","author":"Smola Alex J.","year":"2000","unstructured":"Alex J. Smola and Bernhard Sch\u00f6kopf. 2000. Sparse greedy matrix approximation for machine learning. In International Conference on Machine Learning (ICML). Morgan Kaufmann Publishers Inc., San Francisco, CA, 911\u2013918."},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2018.07.010"},{"key":"e_1_3_3_38_2","first-page":"45","article-title":"Support vector machine active learning with applications to text classification","volume":"2","author":"Tong Simon","year":"2001","unstructured":"Simon Tong and Daphne Koller. 2001. Support vector machine active learning with applications to text classification. J. Mach. Learn. Res. 2 (2001), 45\u201366.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_3_39_2","first-page":"743","volume-title":"International Conference on Artificial Intelligence and Statistics","volume":"14","author":"Trapeznikov Kirill","year":"2011","unstructured":"Kirill Trapeznikov, Venkatesh Saligrama, and David Casta\u00f1\u00f3n. 2011. Active boosted learning (ActBoost). In International Conference on Artificial Intelligence and Statistics, Vol. 14. 743\u2013751."},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.14778\/3291264.3291268"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-66179-7_46"}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637443","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3637443","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:03:25Z","timestamp":1750291405000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637443"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,12]]},"references-count":40,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,4,30]]}},"alternative-id":["10.1145\/3637443"],"URL":"https:\/\/doi.org\/10.1145\/3637443","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"value":"1556-4681","type":"print"},{"value":"1556-472X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,12]]},"assertion":[{"value":"2021-09-16","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-12-06","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}