{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T19:14:06Z","timestamp":1784142846104,"version":"3.55.0"},"reference-count":81,"publisher":"Springer Science and Business Media LLC","issue":"7","license":[{"start":{"date-parts":[[2021,5,30]],"date-time":"2021-05-30T00:00:00Z","timestamp":1622332800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,5,30]],"date-time":"2021-05-30T00:00:00Z","timestamp":1622332800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004004","name":"Universit\u00e0 degli Studi di Trento","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004004","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Artif Intell Rev"],"published-print":{"date-parts":[[2021,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Training data creation is increasingly a key bottleneck for developing machine learning, especially for deep learning systems. Active learning provides a cost-effective means for creating training data by selecting the most informative instances for labeling. Labels in real applications are often collected from crowdsourcing, which engages online crowds for data labeling at scale. Despite the importance of using crowdsourced data in the active learning process, an analysis of how the existing active learning approaches behave over crowdsourced data is currently missing. This paper aims to fill this gap by reviewing the existing active learning approaches and then testing a set of benchmarking ones on crowdsourced datasets. We provide a comprehensive and systematic survey of the recent research on active learning in the hybrid human\u2013machine classification setting, where crowd workers contribute labels (often noisy) to either directly classify data instances or to train machine learning models. We identify three categories of state of the art active learning methods according to whether and how predefined queries employed for data sampling, namely fixed-strategy approaches, dynamic-strategy approaches, and strategy-free approaches. We then conduct an empirical study on their cost-effectiveness, showing that the performance of the existing active learning approaches is affected by many factors in hybrid classification contexts, such as the noise level of data, label fusion technique used, and the specific characteristics of the task. Finally, we discuss challenges and identify potential directions to design active learning strategies for hybrid classification problems.<\/jats:p>","DOI":"10.1007\/s10462-021-10021-3","type":"journal-article","created":{"date-parts":[[2021,6,4]],"date-time":"2021-06-04T14:40:26Z","timestamp":1622817626000},"page":"5283-5305","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":30,"title":["A review and experimental analysis of active learning over crowdsourced data"],"prefix":"10.1007","volume":"54","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6804-127X","authenticated-orcid":false,"given":"Burcu","family":"Sayin","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Evgeny","family":"Krivosheev","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jie","family":"Yang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrea","family":"Passerini","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fabio","family":"Casati","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,5,30]]},"reference":[{"key":"10021_CR1","unstructured":"Aggarwal CC, Kong X, Gu Q, Han J, Yu PS (2014) Chapter 22 active learning: a survey"},{"key":"10021_CR2","unstructured":"Audibert JY, Bubeck S (2009) Minimax policies for adversarial and stochastic bandits. In: Proceedings of the 22nd annual conference on learning theory (COLT), pp 217\u2013226"},{"key":"10021_CR3","doi-asserted-by":"crossref","unstructured":"Auer P, Cesa-Bianchi N, Freund Y, Schapire RE (1995) Gambling in a rigged casino: the adversarial multi-armed bandit problem. In: Proceedings of IEEE 36th annual foundations of computer science, pp 322\u2013331","DOI":"10.1109\/SFCS.1995.492488"},{"issue":"1","key":"10021_CR4","doi-asserted-by":"publisher","first-page":"48","DOI":"10.1137\/S0097539701398375","volume":"32","author":"P Auer","year":"2003","unstructured":"Auer P, Cesa-Bianchi N, Freund Y, Schapire RE (2003) The nonstochastic multiarmed bandit problem. SIAM J Comput 32(1):48\u201377","journal-title":"SIAM J Comput"},{"key":"10021_CR5","doi-asserted-by":"crossref","unstructured":"Aydin BI, Yilmaz YS, Li Y, Li Q, Gao J, Demirbas M (2014) Crowdsourcing for multiple-choice question answering. In: Proceedings of the twenty-eighth AAAI conference on artificial intelligence, pp 2946\u20132953","DOI":"10.1609\/aaai.v28i2.19016"},{"key":"10021_CR6","unstructured":"Bachman P, Sordoni A, Trischler A (2017) Learning algorithms for active learning. In: Proceedings of the 34th international conference on machine learning, vol\u00a070, pp 301\u2013310"},{"key":"10021_CR7","first-page":"255","volume":"5","author":"Y Baram","year":"2004","unstructured":"Baram Y, El-Yaniv R, Luz K (2004) Online choice of active learning algorithms. J Mach Learn Res 5:255\u2013291","journal-title":"J Mach Learn Res"},{"key":"10021_CR8","doi-asserted-by":"crossref","unstructured":"Beygelzimer A, Dasgupta S, Langford J (2009) Importance weighted active learning. In: Proceedings of the 26th annual international conference on machine learning, pp 49\u201356","DOI":"10.1145\/1553374.1553381"},{"key":"10021_CR9","unstructured":"Beygelzimer A, Hsu D, Langford J, Zhang T (2010a) Agnostic active learning without constraints. In: Proceedings of the 23rd international conference on neural information processing systems, vol 1, pp 199\u2013207"},{"key":"10021_CR10","unstructured":"Beygelzimer A, Langford J, Li L, Reyzin L, Schapire R (2010b) An optimal high probability algorithm for the contextual bandit problem. CoRR arXiv:1002.4058"},{"key":"10021_CR11","doi-asserted-by":"crossref","unstructured":"Bouguelia MR, Bela\u00efd Y, Bela\u00efd A (2016) Identifying and mitigating labelling errors in active learning. In: Pattern recognition: applications and methods, vol Lecture Notes in Computer Science. Springer, p\u00a017","DOI":"10.1007\/978-3-319-27677-9_3"},{"key":"10021_CR12","doi-asserted-by":"publisher","first-page":"1307","DOI":"10.1007\/s13042-017-0645-0","volume":"9","author":"MR Bouguelia","year":"2018","unstructured":"Bouguelia MR, Nowaczyk S, Santosh KC, Verikas A (2018) Agreeing to disagree: active learning with noisy labels without crowdsourcing. Int J Mach Learn Cybern 9:1307\u20131319","journal-title":"Int J Mach Learn Cybern"},{"key":"10021_CR13","unstructured":"Brew A, Greene D, Cunningham P (2010) Using crowdsourcing and active learning to track sentiment in online media. In: Proceedings of the 19th European conference on artificial intelligence, pp 145\u2013150"},{"key":"10021_CR14","unstructured":"Budd S, Robinson EC, Kainz B (2019) A survey on active learning and human-in-the-loop deep learning for medical image analysis. ArXiv arXiv:1910.02923"},{"key":"10021_CR15","doi-asserted-by":"crossref","unstructured":"Callaghan W, Goh J, Mohareb M, Lim A, Law E (2018) Mechanicalheart: a human\u2013machine framework for the classification of phonocardiograms. In: Proceedings of ACM Human\u2013Computer Interaction 2(CSCW)","DOI":"10.1145\/3274297"},{"key":"10021_CR16","doi-asserted-by":"crossref","unstructured":"Callison-Burch C (2009) Fast, cheap, and creative: evaluating translation quality using amazon\u2019s mechanical turk. In: Proceedings of the 2009 conference on empirical methods in natural language processing, vol 1, pp 286\u2013295","DOI":"10.3115\/1699510.1699548"},{"key":"10021_CR17","volume-title":"Semi-supervised learning","author":"O Chapelle","year":"2010","unstructured":"Chapelle O, Schlkopf B, Zien A (2010) Semi-supervised learning, 1st edn. The MIT Press, Cambridge","edition":"1"},{"key":"10021_CR18","doi-asserted-by":"crossref","unstructured":"Chu HM, Lin HT (2016) Can active learning experience be transferred? In: 2016 IEEE 16th international conference on data mining (ICDM), pp 841\u2013846","DOI":"10.1109\/ICDM.2016.0100"},{"key":"10021_CR19","first-page":"201","volume":"15","author":"D Cohn","year":"1994","unstructured":"Cohn D, Atlas L, Ladner R (1994) Improving generalization with active learning. Mach Learn 15:201\u2013221","journal-title":"Mach Learn"},{"issue":"1","key":"10021_CR20","first-page":"129","volume":"4","author":"DA Cohn","year":"1996","unstructured":"Cohn DA, Ghahramani Z, Jordan MI (1996) Active learning with statistical models. J Artif Int Res 4(1):129\u2013145","journal-title":"J Artif Int Res"},{"key":"10021_CR21","unstructured":"Contardo G, Denoyer L, Arti\u00e8res T (2017) A meta-learning approach to one-step active-learning. In: International workshop on automatic selection, configuration and composition of machine learning algorithms, vol 1998, pp 28\u201340"},{"issue":"1","key":"10021_CR22","first-page":"20","volume":"28","author":"AP Dawid","year":"1979","unstructured":"Dawid AP, Skene AM (1979) Maximum likelihood estimation of observer error-rates using the EM algorithm. J R Stat Soc Ser C Appl Stat 28(1):20\u201328","journal-title":"J R Stat Soc Ser C Appl Stat"},{"key":"10021_CR23","doi-asserted-by":"crossref","unstructured":"Demartini G, Difallah DE, Cudr\u00e9-Mauroux P (2012) Zencrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In: Proceedings of the 21st international conference on world wide web, pp 469\u2013478","DOI":"10.1145\/2187836.2187900"},{"key":"10021_CR24","doi-asserted-by":"crossref","unstructured":"Deroski S, Panov P, Kocev D, Todorovski L (2014) Probabilistic active learning: towards combining versatility, optimality and efficiency. In: Proceedings of the 17th international conference on discovery science (DS)","DOI":"10.1007\/978-3-319-11812-3"},{"key":"10021_CR25","unstructured":"Desreumaux L, Lemaire V (2020) Learning active learning at the crossroads? Evaluation and discussion. arXiv:2012.09631"},{"key":"10021_CR26","doi-asserted-by":"crossref","unstructured":"Efron B, Tibshirani RJ (1993) An introduction to the bootstrap. No.\u00a057 in Monographs on Statistics and Applied Probability. Chapman & Hall\/CRC, Boca Raton","DOI":"10.1007\/978-1-4899-4541-9"},{"key":"10021_CR27","doi-asserted-by":"crossref","unstructured":"Fan J, Li G, Ooi BC, Tan Kl, Feng J (2015) Icrowd: an adaptive crowdsourcing framework. In: Proceedings of the 2015 ACM SIGMOD international conference on management of data, pp 1015\u20131030","DOI":"10.1145\/2723372.2750550"},{"key":"10021_CR28","doi-asserted-by":"crossref","unstructured":"Fang M, Zhu X, Li B, Ding W, Wu X (2012) Self-taught active learning from crowds. In: 2012 IEEE 12th international conference on data mining, pp 858\u2013863","DOI":"10.1109\/ICDM.2012.64"},{"key":"10021_CR29","doi-asserted-by":"crossref","unstructured":"Fang M, Li Y, Cohn T (2017) Learning how to active learn: a deep reinforcement learning approach. In: Proceedings of the 2017 conference on empirical methods in natural language processing, pp 595\u2013605","DOI":"10.18653\/v1\/D17-1063"},{"key":"10021_CR30","doi-asserted-by":"crossref","unstructured":"Franklin MJ, Kossmann D, Kraska T, Ramesh S, Xin R (2011) Crowddb: answering queries with crowdsourcing. In: Proceedings of the 2011 ACM SIGMOD international conference on management of data, pp 61\u201372","DOI":"10.1145\/1989323.1989331"},{"key":"10021_CR31","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1023\/A:1007330508534","volume":"28","author":"Y Freund","year":"1997","unstructured":"Freund Y, Seung HS, Shamir E, Tishby N (1997) Selective sampling using the query by committee algorithm. Mach Learn 28:133\u2013168","journal-title":"Mach Learn"},{"key":"10021_CR32","unstructured":"Guo Y, Greiner R (2007) Optimistic active learning using mutual information. In: Proceedings of the 20th international joint conference on artifical intelligence, pp 823\u2013829"},{"key":"10021_CR33","doi-asserted-by":"crossref","unstructured":"Haussler D, Kearns M, Schapire R (1991) Bounds on the sample complexity of Bayesian learning using information theory and the VC dimension. In: Proceedings of the fourth annual workshop on computational learning theory, COLT \u201991. Morgan Kaufmann Publishers Inc., San Francisco, pp 61\u201374","DOI":"10.1016\/B978-1-55860-213-7.50010-9"},{"key":"10021_CR34","doi-asserted-by":"crossref","unstructured":"Hoi SCH, Jin R, Lyu MR (2006) Large-scale text categorization by batch mode active learning. In: Proceedings of the 15th international conference on world wide web, pp 633\u2013642","DOI":"10.1145\/1135777.1135870"},{"key":"10021_CR35","unstructured":"Hsu WN, Lin HT (2015) Active learning by learning. In: Proceedings of the twenty-ninth AAAI conference on artificial intelligence, pp 2659\u20132665"},{"key":"10021_CR36","unstructured":"Huang SJ, Jin R, Zhou ZH (2010) Active learning by querying informative and representative examples. In: Proceedings of the 23rd international conference on neural information processing systems, vol 1, pp 892\u2013900"},{"key":"10021_CR37","doi-asserted-by":"crossref","unstructured":"Imran M, Elbassuoni S, Castillo C, Diaz F, Meier P (2013) Practical extraction of disaster\u2013relevant information from social media. In: Proceedings of the 22nd international conference on world wide web, pp 1021\u20131024","DOI":"10.1145\/2487788.2488109"},{"key":"10021_CR38","doi-asserted-by":"crossref","unstructured":"Johnson M, Anderson P, Dras M, Steedman M (2018) Predicting accuracy on large datasets from smaller pilot data. In: ACL, pp 450\u2013455","DOI":"10.18653\/v1\/P18-2072"},{"key":"10021_CR39","unstructured":"Konyushkova K, Sznitman R, Fua P (2017) Learning active learning from data. In: Advances in neural information processing systems, vol 30, pp 4225\u20134235"},{"key":"10021_CR40","unstructured":"Konyushkova K, Sznitman R, Fua P (2018) Discovering general-purpose active learning strategies. CoRR arXiv:1810.04114"},{"key":"10021_CR41","doi-asserted-by":"crossref","unstructured":"Krivosheev E, Casati F, Baez M, Benatallah B (2018a) Combining crowd and machines for multi-predicate item screening. In: Proceedings of ACM Human\u2013Computer Interaction 2","DOI":"10.1145\/3274366"},{"key":"10021_CR42","doi-asserted-by":"crossref","unstructured":"Krivosheev E, Casati F, Benatallah B (2018b) Crowd-based multi-predicate screening of papers in literature reviews. In: Proceedings of the 2018 world wide web conference, pp 55\u201364","DOI":"10.1145\/3178876.3186036"},{"key":"10021_CR43","unstructured":"Krivosheev E, Casati F, Bozzon A (2021) Active hybrid classification. Computing Research Repository arXiv:2101.08854"},{"key":"10021_CR44","doi-asserted-by":"crossref","unstructured":"Lewis DD, Gale WA (1994) A sequential algorithm for training text classifiers. In: Proceedings of the 17th annual international ACM SIGIR conference on research and development in information retrieval, pp 3\u201312","DOI":"10.1007\/978-1-4471-2099-5_1"},{"key":"10021_CR45","doi-asserted-by":"publisher","unstructured":"Li L, Chu W, Langford J, Schapire RE (2010) A contextual-bandit approach to personalized news article recommendation. In: Proceedings of the 19th international conference on world wide web, WWW \u201910. Association for Computing Machinery, New York, pp 661\u2013670. https:\/\/doi.org\/10.1145\/1772690.1772758","DOI":"10.1145\/1772690.1772758"},{"issue":"4","key":"10021_CR46","doi-asserted-by":"publisher","first-page":"425","DOI":"10.14778\/2735496.2735505","volume":"8","author":"Q Li","year":"2014","unstructured":"Li Q, Li Y, Gao J, Su L, Zhao B, Demirbas M, Fan W, Han J (2014) A confidence-aware approach for truth discovery on long-tail data. Proc VLDB Endow 8(4):425\u2013436","journal-title":"Proc VLDB Endow"},{"key":"10021_CR47","doi-asserted-by":"crossref","unstructured":"Liu M, Buntine W, Haffari G (2018) Learning how to actively learn: a deep imitation learning approach. In: Proceedings of the 56th annual meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 1874\u20131883","DOI":"10.18653\/v1\/P18-1174"},{"key":"10021_CR48","unstructured":"Liu Q, Peng J, Ihler A (2012) Variational inference for crowdsourcing. In: Proceedings of the 25th international conference on neural information processing systems, vol 1, pp 692\u2013700"},{"key":"10021_CR49","doi-asserted-by":"crossref","unstructured":"Ma F, Li Y, Li Q, Qiu M, Gao J, Zhi S, Su L, Zhao B, Ji H, Han J (2015) Faitcrowd: Fine grained truth discovery for crowdsourced data aggregation. In: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp 745\u2013754","DOI":"10.1145\/2783258.2783314"},{"key":"10021_CR50","unstructured":"Marcus A, Wu E, Madden S, Miller R (2011) Crowdsourced databases: query processing with people. In: CIDR, pp 211\u2013214"},{"key":"10021_CR51","unstructured":"McCallum A, Nigam K (1998) Employing EM and pool-based active learning for text classification. In: Proceedings of the fifteenth international conference on machine learning, pp 350\u2013358"},{"key":"10021_CR52","doi-asserted-by":"publisher","first-page":"125","DOI":"10.14778\/2735471.2735474","volume":"8","author":"B Mozafari","year":"2014","unstructured":"Mozafari B, Sarkar P, Franklin MJ, Jordan MI, Madden S (2014) Scaling up crowd-sourcing to very large datasets: a case for active learning. Proc VLDB Endow 8:125\u2013136","journal-title":"Proc VLDB Endow"},{"key":"10021_CR53","doi-asserted-by":"crossref","unstructured":"Nguyen AT, Wallace BC, Lease M (2015) Combining crowd and expert labels using decision theoretic active learning. In: Proceedings of the third AAAI conference on human computation and crowdsourcing (HCOMP)","DOI":"10.1609\/hcomp.v3i1.13225"},{"key":"10021_CR54","doi-asserted-by":"crossref","unstructured":"Pang K, Dong M, Wu Y, Hospedales T (2018a) Dynamic ensemble active learning: a non-stationary bandit with expert advice. In: ICPR, pp 2269\u20132276","DOI":"10.1109\/ICPR.2018.8545422"},{"key":"10021_CR55","unstructured":"Pang K, Dong M, Wu Y, Hospedales TM (2018b) Meta-learning transferable active learning policies by deep reinforcement learning. CoRR arXiv:1806.04798"},{"key":"10021_CR56","doi-asserted-by":"crossref","unstructured":"Parameswaran A, Park H, Garcia-Molina H, Polyzotis N, Widom J (2012) Deco: Declarative crowdsourcing. In: Proceedings of the 21st ACM international conference on information and knowledge management, pp 1203\u20131212","DOI":"10.1145\/2396761.2398421"},{"key":"10021_CR57","doi-asserted-by":"crossref","unstructured":"Parker C (2011) An analysis of performance measures for binary classifiers. In: 2011 IEEE 11th international conference on data mining, pp 517\u2013526","DOI":"10.1109\/ICDM.2011.21"},{"key":"10021_CR58","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: machine learning in Python. J Mach Learn Research 12:2825\u20132830","journal-title":"J Mach Learn Research"},{"key":"10021_CR59","unstructured":"Ravi S, Larochelle H (2018) Meta-learning for batch mode active learning. In: 6th international conference on learning representations, ICLR 2018, workshop track proceedings"},{"key":"10021_CR60","unstructured":"Roy N, McCallum A (2001) Toward optimal active learning through sampling estimation of error reduction. In: ICML, pp 894\u2013905"},{"key":"10021_CR61","doi-asserted-by":"crossref","unstructured":"Rudovic O, Zhang M, Schuller BW, Picard RW (2019) Multi-modal active learning from human data: A deep reinforcement learning approach. CoRR arXiv:1906.03098","DOI":"10.1145\/3340555.3353742"},{"key":"10021_CR62","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1023\/B:MACH.0000011806.12374.c3","volume":"54","author":"M Saar-Tsechansky","year":"2004","unstructured":"Saar-Tsechansky M, Provost F (2004) Active sampling for class probability estimation and ranking. Mach Learn 54:153\u2013178","journal-title":"Mach Learn"},{"key":"10021_CR63","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1007\/s10994-007-5019-5","volume":"68","author":"AI Schein","year":"2007","unstructured":"Schein AI, Ungar LH (2007) Active learning for logistic regression: an evaluation. Mach Learn 68:235\u2013265","journal-title":"Mach Learn"},{"key":"10021_CR64","volume-title":"Active learning literature survey","author":"B Settles","year":"2010","unstructured":"Settles B (2010) Active learning literature survey, vol 52. University of Wisconsin, Madison"},{"key":"10021_CR65","doi-asserted-by":"crossref","unstructured":"Settles B, Craven M (2008) An analysis of active learning strategies for sequence labeling tasks. In: Proceedings of the conference on empirical methods in natural language processing, pp 1070\u20131079","DOI":"10.3115\/1613715.1613855"},{"key":"10021_CR66","doi-asserted-by":"crossref","unstructured":"Seung HS, Opper M, Sompolinsky H (1992) Query by committee. In: Proceedings of the fifth annual workshop on computational learning theory, pp 287\u2013294","DOI":"10.1145\/130385.130417"},{"key":"10021_CR67","doi-asserted-by":"crossref","unstructured":"Snow R, O\u2019Connor B, Jurafsky D, Ng A (2008) Cheap and fast\u2014but is it good? evaluating non-expert annotations for natural language tasks. In: Proceedings of the 2008 conference on empirical methods in natural language processing, pp 254\u2013263","DOI":"10.3115\/1613715.1613751"},{"key":"10021_CR68","unstructured":"Sun-Hosoya L, Guyon I, Sebag M (2018) Activmetal: algorithm recommendation with active meta learning. In: IAL 2018 workshop, ECML PKDD, poster"},{"key":"10021_CR69","doi-asserted-by":"crossref","unstructured":"Tsai M, Ho C, Lin C (2010) Active learning strategies using SVMs. Wiley Int Rev Data Min and Knowl Disc 313\u2013326","DOI":"10.1109\/IJCNN.2010.5596668"},{"key":"10021_CR70","doi-asserted-by":"publisher","first-page":"1341","DOI":"10.1007\/s10115-019-01386-7","volume":"62","author":"J Tu","year":"2019","unstructured":"Tu J, Yu G, Domeniconi C, Wang J, Xiao G, Guo M (2019) Multi-label crowd consensus via joint matrix factorization. Knowl Inf Syst 62:1341\u20131369","journal-title":"Knowl Inf Syst"},{"key":"10021_CR71","doi-asserted-by":"crossref","unstructured":"Vu TT, Liu M, Phung D, Haffari G (2019) Learning how to active learn by dreaming. In: Proceedings of the 57th annual meeting of the Association for Computational Linguistics, pp 4091\u20134101","DOI":"10.18653\/v1\/P19-1401"},{"key":"10021_CR72","first-page":"2269","volume":"12","author":"L Wang","year":"2011","unstructured":"Wang L (2011) Smoothness, disagreement coefficient, and the label complexity of agnostic active learning. J Mach Learn Res 12:2269\u20132292","journal-title":"J Mach Learn Res"},{"key":"10021_CR73","unstructured":"Woodward M, Finn C (2017) Active one-shot learning. In: NIPS 2016, deep reinforcement learning workshop"},{"key":"10021_CR74","unstructured":"Yan S, Chaudhuri K, Javidi T (2016) Active learning from imperfect labelers. In: Proceedings of the 30th international conference on neural information processing systems, pp 2136\u20132144"},{"key":"10021_CR75","unstructured":"Yan S, Chaudhuri K, Javidi T (2019) The label complexity of active learning from observational data. In: 33rd conference on neural information processing systems (NeurIPS 2019)"},{"key":"10021_CR76","unstructured":"Yan Y, Rosales R, Fung G, Schmidt M, Hermosillo G, Bogoni L, Moy L, Dy J (2010) Modeling annotator expertise: Learning when everybody knows a bit of something. In: Proceedings of the 13th international conference on artificial intelligence and statistics (AISTATS), vol\u00a09, pp 932\u2013939"},{"key":"10021_CR77","doi-asserted-by":"crossref","unstructured":"Yang J, Drake T, Damianou A, Maarek Y (2018) Leveraging crowdsourcing data for deep active learning an application: learning intents in Alexa. In: Proceedings of the 2018 World Wide Web conference, pp 23\u201332","DOI":"10.1145\/3178876.3186033"},{"key":"10021_CR78","doi-asserted-by":"crossref","unstructured":"Zhao L, Sukthankar GR, Sukthankar R (2011) Incremental relabeling for active learning with noisy crowdsourced annotations. In: 2011 IEEE third international conference on privacy, security, risk and trust and 2011 IEEE third international conference on social computing, pp 728\u2013733","DOI":"10.1109\/PASSAT\/SocialCom.2011.193"},{"issue":"5","key":"10021_CR79","doi-asserted-by":"publisher","first-page":"541","DOI":"10.14778\/3055540.3055547","volume":"10","author":"Y Zheng","year":"2017","unstructured":"Zheng Y, Li G, Li Y, Shan C, Cheng R (2017) Truth inference in crowdsourcing: Is the problem solved? Proc VLDB Endow 10(5):541\u2013552","journal-title":"Proc VLDB Endow"},{"key":"10021_CR80","unstructured":"Zhong J, Tang K, Zhou ZH (2015) Active learning from crowds with unsure option. In: Proceedings of the 24th international conference on artificial intelligence, pp 1061\u20131067"},{"key":"10021_CR81","unstructured":"Zhu X, Lafferty J, Ghahramani Z (2003) Combining active learning and semi-supervised learning using gaussian fields and harmonic functions. In: ICML 2003 workshop on the continuum from labeled to unlabeled data in machine learning and data mining, pp 58\u201365"}],"container-title":["Artificial Intelligence Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10462-021-10021-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10462-021-10021-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10462-021-10021-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,29]],"date-time":"2022-12-29T22:21:03Z","timestamp":1672352463000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10462-021-10021-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,30]]},"references-count":81,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2021,10]]}},"alternative-id":["10021"],"URL":"https:\/\/doi.org\/10.1007\/s10462-021-10021-3","relation":{},"ISSN":["0269-2821","1573-7462"],"issn-type":[{"value":"0269-2821","type":"print"},{"value":"1573-7462","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,30]]},"assertion":[{"value":"30 May 2021","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}