{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,6]],"date-time":"2026-02-06T01:18:24Z","timestamp":1770340704774,"version":"3.49.0"},"reference-count":59,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2018,6,27]],"date-time":"2018-06-27T00:00:00Z","timestamp":1530057600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Research Foundation, Prime Minister's Office, Singapore under its International Research Centres in Singapore Funding Initiative"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2018,10,31]]},"abstract":"<jats:p>In literature, learning with expert advice methods usually assume that a learner always obtain the true label of every incoming training instance at the end of each trial. However, in many real-world applications, acquiring the true labels of all instances can be both costly and time consuming, especially for large-scale problems. For example, in the social media, data stream usually comes in a high speed and volume, and it is nearly impossible and highly costly to label all of the instances. In this article, we address this problem with active learning with expert advice, where the ground truth of an instance is disclosed only when it is requested by the proposed active query strategies. Our goal is to minimize the number of requests while training an online learning model without sacrificing the performance. To address this challenge, we propose a framework of active forecasters, which attempts to extend two fully supervised forecasters, Exponentially Weighted Average Forecaster and Greedy Forecaster, to tackle the task of online active learning (OAL) with expert advice. Specifically, we proposed two OAL with expert advice algorithms, named Active Exponentially Weighted Average Forecaster (AEWAF) and active greedy forecaster (AGF), by considering the difference of expert advices. To further improve the robustness of the proposed AEWAF and AGF algorithms in the noisy scenarios (where noisy experts exist), we also proposed two robust active learning with expert advice algorithms, named Robust Active Exponentially Weighted Average Forecaster and Robust Active Greedy Forecaster. We validate the efficacy of the proposed algorithms by an extensive set of experiments in both normal scenarios (where all of experts are comparably reliable) and noisy scenarios.<\/jats:p>","DOI":"10.1145\/3201604","type":"journal-article","created":{"date-parts":[[2018,6,27]],"date-time":"2018-06-27T12:14:36Z","timestamp":1530101676000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":17,"title":["Online Active Learning with Expert Advice"],"prefix":"10.1145","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3607-6317","authenticated-orcid":false,"given":"Shuji","family":"Hao","sequence":"first","affiliation":[{"name":"Institute of High Performance of Computing, Singapore"}]},{"given":"Peiying","family":"Hu","sequence":"additional","affiliation":[{"name":"Liaoning Normal University, Dalian, China"}]},{"given":"Peilin","family":"Zhao","sequence":"additional","affiliation":[{"name":"South China University of Technology, Guangzhou, China"}]},{"given":"Steven C. H.","family":"Hoi","sequence":"additional","affiliation":[{"name":"Singapore Management University, Singapore"}]},{"given":"Chunyan","family":"Miao","sequence":"additional","affiliation":[{"name":"Nanyang Technological University, Singapore"}]}],"member":"320","published-online":{"date-parts":[[2018,6,27]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.5555\/1768841.1768887"},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI\u201915)","author":"Aleksandrov Martin","year":"2015"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the 29th AAAI Conference on Artificial Intelligence","author":"Amin Kareem"},{"key":"e_1_2_1_4_1","volume-title":"Ladner","author":"Atlas Les E.","year":"1990"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1103\/RevModPhys.34.123"},{"key":"e_1_2_1_6_1","first-page":"363","article-title":"Tracking a small set of experts by mixing past posteriors","volume":"3","author":"Bousquet Olivier","year":"2002","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1137\/S0097539703432542"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/258128.258179"},{"key":"e_1_2_1_9_1","article-title":"Worst-case analysis of selective sampling for linear classification","author":"Cesa-Bianchi Nicol\u00f2","year":"2006","journal-title":"The Journal of Machine Learning Research 7"},{"key":"e_1_2_1_10_1","doi-asserted-by":"crossref","unstructured":"N. Cesa-Bianchi and G. Lugosi. 2006. Prediction Learning and Games. Cambridge University Press.   N. Cesa-Bianchi and G. Lugosi. 2006. Prediction Learning and Games. Cambridge University Press.","DOI":"10.1017\/CBO9780511546921"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.5555\/1248547.1248566"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the 22nd International Conference on Neural Information Processing Systems. 414--422","author":"Crammer Koby","year":"2009"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-013-5327-x"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390156.1390190"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.5555\/2752879.2752887"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.5555\/646944.712239"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1006\/jcss.1997.1504"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 30th International Conference on Neural Information Processing Systems. D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Eds.). 514--522","author":"Fujii Kaito","year":"2016"},{"key":"e_1_2_1_19_1","first-page":"213","article-title":"A new approximate maximal margin classification algorithm","volume":"2","author":"Gentile Claudio","year":"2001","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/2832581.2832742"},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the International Conference on Neural Information Processing Systems. 593--600","author":"Guo Yuhong","year":"2008"},{"key":"e_1_2_1_22_1","first-page":"1469","article-title":"Activized learning: Transforming passive to active with improved label complexity *","volume":"13","author":"Hanneke Steve","year":"2012","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1561\/2200000037"},{"key":"e_1_2_1_24_1","first-page":"1","article-title":"The optimal sample complexity of PAC learning","volume":"17","author":"Hanneke Steve","year":"2016","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.5555\/2789272.2912111"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2017.2778097"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2806416.2806464"},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI\u201917)","author":"Hao Shuji","year":"2017"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2016.0115"},{"key":"e_1_2_1_30_1","volume-title":"Warmuth","author":"Haussler David","year":"1995"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.5555\/2832581.2832614"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007424614876"},{"key":"e_1_2_1_33_1","unstructured":"Steven C. H. Hoi Doyen Sahoo Jing Lu and Peilin Zhao. 2018. Online learning: A comprehensive survey. arXiv:1802.02871.  Steven C. H. Hoi Doyen Sahoo Jing Lu and Peilin Zhao. 2018. Online learning: A comprehensive survey. arXiv:1802.02871."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.5555\/2627435.2627450"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.5555\/2832747.2832878"},{"key":"e_1_2_1_36_1","first-page":"2755","article-title":"Dynamic weighted majority: An ensemble method for drifting concepts","volume":"8","author":"Zico Kolter J.","year":"2007","journal-title":"The Journal of Machine Learning Research"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2010.139"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the Neural Information Processing Systems (NIPS\u201999)","author":"Li Yi"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1012435301888"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1006\/inco.1994.1009"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the Asian Conference on Machine Learning Research (ACML\u201914)","author":"Lu Jing"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-016-5555-y"},{"key":"e_1_2_1_43_1","unstructured":"Mehryar Mohri and Afshin Rostamizadeh. 2013. Perceptron mistake bounds. arxiv:1305.0208v2.  Mehryar Mohri and Afshin Rostamizadeh. 2013. Perceptron mistake bounds. arxiv:1305.0208v2."},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the 23rd International Conference on Neural Information Processing Systems. 1840--1848","author":"Orabona Francesco","year":"2010"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0042519"},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of the 28th Conference on Artificial Intelligence. AAAI Press, Qu\u00e9bec City","author":"Ruvolo Paul","year":"2014"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.5555\/3019233"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1162\/153244302760185243"},{"key":"e_1_2_1_51_1","volume-title":"Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI\u201915)","author":"Veness Joel"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.5555\/92571.92672"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2507157.2507176"},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the 29th International Conference on Machine Learning (ICML\u201912)","author":"Wang Jialei"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2013.157"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-017-5676-y"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2016.0183"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2487647"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2021051"},{"key":"e_1_2_1_60_1","volume-title":"Proceedings of the 28th International Conference on Machine Learning (ICML\u201911)","author":"Zhao Peilin"}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3201604","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3201604","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T01:08:53Z","timestamp":1750208933000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3201604"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,6,27]]},"references-count":59,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2018,10,31]]}},"alternative-id":["10.1145\/3201604"],"URL":"https:\/\/doi.org\/10.1145\/3201604","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"value":"1556-4681","type":"print"},{"value":"1556-472X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,6,27]]},"assertion":[{"value":"2017-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-06-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}