{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,31]],"date-time":"2025-10-31T07:46:32Z","timestamp":1761896792539,"version":"3.41.0"},"publisher-location":"New York, New York, USA","reference-count":50,"publisher":"ACM Press","license":[{"start":{"date-parts":[[2018,1,1]],"date-time":"2018-01-01T00:00:00Z","timestamp":1514764800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2018]]},"DOI":"10.1145\/3178876.3186033","type":"proceedings-article","created":{"date-parts":[[2018,4,13]],"date-time":"2018-04-13T15:53:48Z","timestamp":1523634828000},"page":"23-32","source":"Crossref","is-referenced-by-count":41,"title":["Leveraging Crowdsourcing Data for Deep Active Learning An Application"],"prefix":"10.1145","author":[{"given":"Jie","family":"Yang","sequence":"first","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}]},{"given":"Thomas","family":"Drake","sequence":"additional","affiliation":[{"name":"Amazon Research, Seattle, WA, USA"}]},{"given":"Andreas","family":"Damianou","sequence":"additional","affiliation":[{"name":"Amazon Research, Cambridge, United Kingdom"}]},{"given":"Yoelle","family":"Maarek","sequence":"additional","affiliation":[{"name":"Amazon Research, Haifa, Israel"}]}],"member":"320","reference":[{"key":"key-10.1145\/3178876.3186033-1","unstructured":"Sungjin Ahn, Anoop Korattikara, and Max Welling . 2012. Bayesian Posterior Sampling via Stochastic Gradient Fisher Scoring. arXiv preprint arXiv:1206.6380 (2012)."},{"key":"key-10.1145\/3178876.3186033-2","doi-asserted-by":"crossref","unstructured":"Alessandro Bozzon, Marco Brambilla, and Stefano Ceri . 2012. Answering Search Queries with Crowdsearcher. In Proceedings of the 21st International Conference on World Wide Web (WWW). ACM, 1009--1018.","DOI":"10.1145\/2187836.2187971"},{"key":"key-10.1145\/3178876.3186033-3","doi-asserted-by":"crossref","unstructured":"Alessandro Bozzon, Piero Fraternali, Luca Galli, and Roula Karam . 2014. Modeling Crowdsourcing Scenarios in Socially-enabled Human Computation Applications. Journal on Data Semantics (JoDS) Vol. 3, 3 (2014), 169--188.","DOI":"10.1007\/s13740-013-0032-2"},{"key":"key-10.1145\/3178876.3186033-4","doi-asserted-by":"crossref","unstructured":"David A Cohn, Zoubin Ghahramani, and Michael I Jordan . 1996. Active Learning with Statistical Models. Journal of Artificial Intelligence Research (JAIR) (1996).","DOI":"10.1613\/jair.295"},{"key":"key-10.1145\/3178876.3186033-5","doi-asserted-by":"crossref","unstructured":"Alexander Philip Dawid and Allan M Skene . 1979. Maximum Likelihood Estimation of Observer Error-rates Using the EM Algorithm. Applied statistics (1979), 20--28.","DOI":"10.2307\/2346806"},{"key":"key-10.1145\/3178876.3186033-6","doi-asserted-by":"crossref","unstructured":"Victor De Boer, Michiel Hildebrand, Lora Aroyo, Pieter De Leenheer, Chris Dijkshoorn, Binyam Tesfa, and Guus Schreiber Nichesourcing: Harnessing the Power of Crowds of Experts. Knowledge Engineering and Knowledge Management. Springer, 16--20.","DOI":"10.1007\/978-3-642-33876-2_3"},{"key":"key-10.1145\/3178876.3186033-7","doi-asserted-by":"crossref","unstructured":"Gianluca Demartini . 2015. Hybrid Human--machine Information Systems: Challenges and Opportunities. Computer Networks Vol. 90 (2015), 5--13.","DOI":"10.1016\/j.comnet.2015.05.018"},{"key":"key-10.1145\/3178876.3186033-8","doi-asserted-by":"crossref","unstructured":"Gianluca Demartini, Djellel Eddine Difallah, and Philippe Cudré-Mauroux . 2012. ZenCrowd: Leveraging Probabilistic Reasoning and Crowdsourcing Techniques for Large-scale Entity Linking. In Proceedings of the 21st International Conference on World Wide Web (WWW). ACM, 469--478.","DOI":"10.1145\/2187836.2187900"},{"key":"key-10.1145\/3178876.3186033-9","unstructured":"Arthur P Dempster, Nan M Laird, and Donald B Rubin . 1977. Maximum Likelihood from Incomplete Data via the EM Algorithm. Journal of the Royal Statistical Society: Series B (Methodological) (1977), 1--38."},{"key":"key-10.1145\/3178876.3186033-10","unstructured":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei . 2009. Imagenet: A Large-scale Hierarchical Image Database Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 248--255."},{"key":"key-10.1145\/3178876.3186033-11","doi-asserted-by":"crossref","unstructured":"Meng Fang, Xingquan Zhu, Bin Li, Wei Ding, and Xindong Wu . 2012. Self-taught Active Learning from Crowds. In Proceedings of the 12th IEEE International Conference on Data Mining (ICDM). IEEE, 858--863.","DOI":"10.1109\/ICDM.2012.64"},{"key":"key-10.1145\/3178876.3186033-12","unstructured":"Michael J Franklin, Donald Kossmann, Tim Kraska, Sukriti Ramesh, and Reynold Xin . 2011. CrowdDB: Answering Queries with Crowdsourcing. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data (SIGMOD). ACM, 61--72."},{"key":"key-10.1145\/3178876.3186033-13","unstructured":"Jerome Friedman, Trevor Hastie, and Robert Tibshirani . 2001. The Elements of Statistical Learning. Vol. Vol. 1. Springer Series in Statistics."},{"key":"key-10.1145\/3178876.3186033-14","doi-asserted-by":"crossref","unstructured":"Ujwal Gadiraju, Jie Yang, and Alessandro Bozzon . 2017. Clarity is a Worthwhile Quality -- On the Role of Task Clarity in Microtask Crowdsourcing Proceedings of the 28th ACM Conference on Hypertext and Social Media (HyperText). 5--14.","DOI":"10.1145\/3078714.3078715"},{"key":"key-10.1145\/3178876.3186033-15","unstructured":"Yarin Gal and Zoubin Ghahramani . 2016. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning Proceedings of the 33rd International Conference on Machine Learning (ICML). 1050--1059."},{"key":"key-10.1145\/3178876.3186033-16","unstructured":"Yarin Gal, Riashat Islam, and Zoubin Ghahramani . 2017. Deep Bayesian Active Learning with Image Data. Proceedings of the 34th International Conference on Machine Learning (ICML) (2017)."},{"key":"key-10.1145\/3178876.3186033-17","unstructured":"Ian Goodfellow, Yoshua Bengio, and Aaron Courville . 2016. Deep Learning. MIT Press. http:\/\/www.deeplearningbook.org"},{"key":"key-10.1145\/3178876.3186033-18","unstructured":"Alex Graves, Abdelrahman Mohamed, and Geoffrey Hinton . 2013. Speech Recognition with Deep Recurrent Neural Networks Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 6645--6649."},{"key":"key-10.1145\/3178876.3186033-19","unstructured":"Michael Heilman and Noah A Smith . 2010. Rating Computer-generated Questions with Mechanical Turk Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk. 35--40."},{"key":"key-10.1145\/3178876.3186033-20","unstructured":"G Hinton, N Srivastava, and K Swersky . 2012 a. RMSProp: Divide the gradient by a running average of its recent magnitude. Neural Networks for Machine Learning, Coursera Lecture 6e (2012)."},{"key":"key-10.1145\/3178876.3186033-21","unstructured":"Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov . 2012 b. Improving Neural Networks by Preventing Co-adaptation of Feature Detectors. arXiv preprint arXiv:1207.0580 (2012)."},{"key":"key-10.1145\/3178876.3186033-22","doi-asserted-by":"crossref","unstructured":"Panagiotis G Ipeirotis and Evgeniy Gabrilovich . 2014. Quizz: Targeted Crowdsourcing with a Billion (Potential) Users Proceedings of the 23rd International Conference on World Wide Web (WWW). ACM, 143--154.","DOI":"10.1145\/2566486.2567988"},{"key":"key-10.1145\/3178876.3186033-23","unstructured":"Andreas Krause, Ajit Singh, and Carlos Guestrin . 2008. Near-optimal Sensor Placements in Gaussian Processes: Theory, Efficient Algorithms and Empirical Studies. Journal of Machine Learning Research (JMLR), Vol. 9, Feb (2008), 235--284."},{"key":"key-10.1145\/3178876.3186033-24","unstructured":"Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton . 2012. Imagenet Classification with Deep Convolutional Neural Networks Advances in Neural Information Processing Systems (NIPS). 1097--1105."},{"key":"key-10.1145\/3178876.3186033-25","doi-asserted-by":"crossref","unstructured":"Edith Law and Luis von Ahn . 2011. Human Computation. Synthesis Lectures on Artificial Intelligence and Machine Learning, Vol. 5, 3 (2011), 1--121.","DOI":"10.2200\/S00371ED1V01Y201107AIM013"},{"key":"key-10.1145\/3178876.3186033-26","unstructured":"Florian Laws, Christian Scheible, and Hinrich Schütze . 2011. Active Learning with Amazon Mechanical Turk. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 1546--1556."},{"key":"key-10.1145\/3178876.3186033-27","unstructured":"Matthew Lease . 2011. On Quality Control and Machine Learning in Crowdsourcing Proceedings of the 3rd Human Computation Workshop (HCOMP). AAAI, 97--102."},{"key":"key-10.1145\/3178876.3186033-28","unstructured":"David D Lewis and William A Gale . 1994. A Sequential Algorithm for Training Text Classifiers Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM, 3--12."},{"key":"key-10.1145\/3178876.3186033-29","unstructured":"Christopher H Lin, Mausam, Daniel S Weld, and others . 2014. To Re(label), or Not to Re(label). In Proceedings of the 2nd AAAI Conference on Human Computation and Crowdsourcing (HCOMP). AAAI, 1--8."},{"key":"key-10.1145\/3178876.3186033-30","unstructured":"Bart Mellebeek, Francesc Benavent, Jens Grivolla, Joan Codina, Marta R Costa-Jussa, and Rafael Banchs . 2010. Opinion Mining of Spanish Customer Comments with Non-expert Annotations on Mechanical Turk Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk. 114--121."},{"key":"key-10.1145\/3178876.3186033-31","unstructured":"Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy . 2010. Learning from Crowds. Journal of Machine Learning Research (JMLR), Vol. 11, Apr (2010), 1297--1322."},{"key":"key-10.1145\/3178876.3186033-32","unstructured":"Nicholas Roy and Andrew McCallum . 2001. Toward Optimal Active Learning Through Monte Carlo Estimation of Error Reduction. Proceedings of the 18th International Conference on Machine Learning (ICML) (2001), 441--448."},{"key":"key-10.1145\/3178876.3186033-33","unstructured":"David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, and others . 1988. Learning Representations by Back-propagating Errors. Cognitive Modeling, Vol. 5, 3 (1988), 1."},{"key":"key-10.1145\/3178876.3186033-34","unstructured":"Burr Settles . 2010. Active Learning Literature Survey. University of Wisconsin, Madison Vol. 52, 55--66 (2010), 11."},{"key":"key-10.1145\/3178876.3186033-35","unstructured":"H Sebastian Seung, Manfred Opper, and Haim Sompolinsky . 1992. Query by Committee Proceedings of the 5th Annual Workshop on Computational Learning Theory (COLT). ACM, 287--294."},{"key":"key-10.1145\/3178876.3186033-36","unstructured":"Victor S Sheng, Foster Provost, and Panagiotis G Ipeirotis . 2008. Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). ACM, 614--622."},{"key":"key-10.1145\/3178876.3186033-37","unstructured":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov . 2014. Dropout: a Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research (JMLR), Vol. 15, 1 (2014), 1929--1958."},{"key":"key-10.1145\/3178876.3186033-38","unstructured":"Ilya Sutskever, Oriol Vinyals, and Quoc V Le . 2014. Sequence to Sequence Learning with Neural Networks Advances in Neural Information Processing Systems (NIPS). 3104--3112."},{"key":"key-10.1145\/3178876.3186033-39","unstructured":"Yuandong Tian and Jun Zhu . 2012. Learning from Crowds in the Presence of Schools of Thought Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). ACM, 226--234."},{"key":"key-10.1145\/3178876.3186033-40","doi-asserted-by":"crossref","unstructured":"Luis Von Ahn and Laura Dabbish . 2008. Designing Games with a Purpose. Commun. ACM Vol. 51, 8 (2008), 58--67.","DOI":"10.1145\/1378704.1378719"},{"key":"key-10.1145\/3178876.3186033-41","doi-asserted-by":"crossref","unstructured":"Luis Von Ahn, Benjamin Maurer, Colin McMillen, David Abraham, and Manuel Blum . 2008. Recaptcha: Human-based Character Recognition via Web Security Measures. Science, Vol. 321, 5895 (2008), 1465--1468.","DOI":"10.1126\/science.1160379"},{"key":"key-10.1145\/3178876.3186033-42","unstructured":"Max Welling and Yee W Teh . 2011. Bayesian Learning via Sochastic Gradient Langevin Dynamics Proceedings of the 28th International Conference on Machine Learning (ICML). 681--688."},{"key":"key-10.1145\/3178876.3186033-43","unstructured":"Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R Movellan, and Paul L Ruvolo . 2009. Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise Advances in Neural Information Processing Systems (NIPS). 2035--2043."},{"key":"key-10.1145\/3178876.3186033-44","unstructured":"Yan Yan, Glenn M Fung, Rómer Rosales, and Jennifer G Dy . 2011. Active Learning from Crowds. In Proceedings of the 28th International Conference on Machine Learning (ICML). 1161--1168."},{"key":"key-10.1145\/3178876.3186033-45","unstructured":"Yan Yan, Rómer Rosales, Glenn Fung, Mark W Schmidt, Gerardo H Valadez, Luca Bogoni, Linda Moy, and Jennifer G Dy . 2010. Modeling Annotator Expertise: Learning When Everybody Knows a Bit of Something International Conference on Artificial Intelligence and Statistics (AISTATS). 932--939."},{"key":"key-10.1145\/3178876.3186033-46","unstructured":"Jie Yang, Claudia Hauff, Alessandro Bozzon, and Geert-Jan Houben . 2014. Asking the Right Question in Collaborative Q&A Systems Proceedings of the 25th ACM Conference on Hypertext and Social Media (HyperText). ACM, 179--189."},{"key":"key-10.1145\/3178876.3186033-47","unstructured":"Jie Yang, Judith Redi, Gianluca Demartini, and Alessandro Bozzon . 2016. Modeling Task Complexity in Crowdsourcing. In Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP). AAAI, 249--258."},{"key":"key-10.1145\/3178876.3186033-48","unstructured":"Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals . 2016. Understanding Deep Learning Requires Rethinking Generalization International Conference on Learning Representations (ICLR)."},{"key":"key-10.1145\/3178876.3186033-49","unstructured":"Jinhong Zhong, Ke Tang, and Zhi-Hua Zhou . 2015. Active Learning from Crowds with Unsure Option.. Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI). 1061--1068."},{"key":"key-10.1145\/3178876.3186033-50","unstructured":"Denny Zhou, Sumit Basu, Yi Mao, and John C Platt . 2012. Learning from the Wisdom of Crowds by Minimax Entropy Advances in Neural Information Processing Systems (NIPS). 2195--2203."}],"event":{"number":"2018","sponsor":["SIGWEB, ACM Special Interest Group on Hypertext, Hypermedia, and Web","IW3C2, International World Wide Web Conference Committee"],"acronym":"WWW '18","name":"the 2018 World Wide Web Conference","start":{"date-parts":[[2018,4,23]]},"location":"Lyon, France","end":{"date-parts":[[2018,4,27]]}},"container-title":["Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3178876.3186033","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/dl.acm.org\/ft_gateway.cfm?id=3186033&ftid=1957471&dwn=1","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:27:00Z","timestamp":1750213620000},"score":1,"resource":{"primary":{"URL":"http:\/\/dl.acm.org\/citation.cfm?doid=3178876.3186033"}},"subtitle":["Learning Intents in Alexa"],"proceedings-subject":"World Wide Web","short-title":[],"issued":{"date-parts":[[2018]]},"references-count":50,"URL":"https:\/\/doi.org\/10.1145\/3178876.3186033","relation":{},"subject":[],"published":{"date-parts":[[2018]]}}}