{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,3]],"date-time":"2026-01-03T06:46:36Z","timestamp":1767422796933,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":40,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,4,19]],"date-time":"2021-04-19T00:00:00Z","timestamp":1618790400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,4,19]]},"DOI":"10.1145\/3442381.3449933","type":"proceedings-article","created":{"date-parts":[[2021,6,3]],"date-time":"2021-06-03T19:34:17Z","timestamp":1622748857000},"page":"1904-1915","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Online Label Aggregation: A Variational Bayesian Approach"],"prefix":"10.1145","author":[{"given":"Chi","family":"Hong","sequence":"first","affiliation":[{"name":"Delft University of Technology, Netherlands"}]},{"given":"Amirmasoud","family":"Ghiassi","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Netherlands"}]},{"given":"Yichi","family":"Zhou","sequence":"additional","affiliation":[{"name":"Tsinghua University, China"}]},{"given":"Robert","family":"Birke","sequence":"additional","affiliation":[{"name":"ABB Research, Switzerland"}]},{"given":"Lydia Y.","family":"Chen","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Netherlands"}]}],"member":"320","published-online":{"date-parts":[[2021,6,3]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"European Commission 2018. European Union\u2019s General Data Protection Regulation. European Commission. https:\/\/ec.europa.eu\/info\/law\/law-topic\/data-protection\/eu-data-protection-rules_en European Commission 2018. European Union\u2019s General Data Protection Regulation. European Commission. https:\/\/ec.europa.eu\/info\/law\/law-topic\/data-protection\/eu-data-protection-rules_en"},{"key":"e_1_3_2_1_2_1","unstructured":"Christopher\u00a0M Bishop. 2006. Pattern recognition and machine learning. (2006) 461\u2013517. Christopher\u00a0M Bishop. 2006. Pattern recognition and machine learning. (2006) 461\u2013517."},{"volume-title":"New Horizons for a Data-Driven Economy - A Roadmap for Usage and Exploitation of Big Data in Europe","author":"Cavanillas Jos\u00e9\u00a0Mar\u00eda","key":"e_1_3_2_1_3_1","unstructured":"Jos\u00e9\u00a0Mar\u00eda Cavanillas , Edward Curry , and Wolfgang Wahlster( Eds .). 2016. New Horizons for a Data-Driven Economy - A Roadmap for Usage and Exploitation of Big Data in Europe . Springer . https:\/\/doi.org\/10.1007\/978-3-319-21569-3 10.1007\/978-3-319-21569-3 Jos\u00e9\u00a0Mar\u00eda Cavanillas, Edward Curry, and Wolfgang Wahlster(Eds.). 2016. New Horizons for a Data-Driven Economy - A Roadmap for Usage and Exploitation of Big Data in Europe. Springer. https:\/\/doi.org\/10.1007\/978-3-319-21569-3"},{"key":"e_1_3_2_1_4_1","volume-title":"Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics","author":"Dawid Alexander\u00a0Philip","year":"1979","unstructured":"Alexander\u00a0Philip Dawid and Allan\u00a0 M Skene . 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics ( 1979 ), 20\u201328. Alexander\u00a0Philip Dawid and Allan\u00a0M Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics (1979), 20\u201328."},{"key":"e_1_3_2_1_5_1","unstructured":"Dheeru Dua and Casey Graff. 2017. UCI Machine Learning Repository. http:\/\/archive.ics.uci.edu\/ml Dheeru Dua and Casey Graff. 2017. UCI Machine Learning Repository. http:\/\/archive.ics.uci.edu\/ml"},{"key":"e_1_3_2_1_6_1","unstructured":"Li Fei-Fei. 2010. ImageNet: crowdsourcing benchmarking & other cool things. In CMU VASC Seminar Vol.\u00a016. 18\u201325. Li Fei-Fei. 2010. ImageNet: crowdsourcing benchmarking & other cool things. In CMU VASC Seminar Vol.\u00a016. 18\u201325."},{"key":"e_1_3_2_1_7_1","unstructured":"Alex Gaunt Diana Borsa and Yoram Bachrach. 2016. Training deep neural nets to aggregate crowdsourced responses. In UAI. 242\u2013251. Alex Gaunt Diana Borsa and Yoram Bachrach. 2016. Training deep neural nets to aggregate crowdsourced responses. In UAI. 242\u2013251."},{"key":"e_1_3_2_1_8_1","unstructured":"Amirmasoud Ghiassi Taraneh Younesian Robert Birke and Lydia\u00a0Y. Chen. 2020. TrustNet: Learning from Trusted Data Against (A)symmetric Label Noise. CoRR abs\/2007.06324(2020). Amirmasoud Ghiassi Taraneh Younesian Robert Birke and Lydia\u00a0Y. Chen. 2020. TrustNet: Learning from Trusted Data Against (A)symmetric Label Noise. CoRR abs\/2007.06324(2020)."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"crossref","unstructured":"Amirmasoud Ghiassi Taraneh Younesian Zilong Zhao Robert Birke Valerio Schiavoni and Lydia\u00a0Y. Chen. 2019. Robust (Deep) Learning Framework Against Dirty Labels and Beyond. In TPS-ISA. 236\u2013244. Amirmasoud Ghiassi Taraneh Younesian Zilong Zhao Robert Birke Valerio Schiavoni and Lydia\u00a0Y. Chen. 2019. Robust (Deep) Learning Framework Against Dirty Labels and Beyond. In TPS-ISA. 236\u2013244.","DOI":"10.1109\/TPS-ISA48467.2019.00038"},{"key":"e_1_3_2_1_10_1","volume-title":"Demographic estimation from face images: Human vs. machine performance","author":"Han Hu","year":"2015","unstructured":"Hu Han , Charles Otto , Xiaoming Liu , and Anil\u00a0 K Jain . 2015. Demographic estimation from face images: Human vs. machine performance . IEEE transactions on pattern analysis and machine intelligence 37, 6( 2015 ), 1148\u20131161. Hu Han, Charles Otto, Xiaoming Liu, and Anil\u00a0K Jain. 2015. Demographic estimation from face images: Human vs. machine performance. IEEE transactions on pattern analysis and machine intelligence 37, 6(2015), 1148\u20131161."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2567948.2577034"},{"key":"e_1_3_2_1_12_1","unstructured":"A Janosi W Steinbrunn M Pfisterer and R Detrano. 1988. Heart disease data set. In https:\/\/archive.ics.uci.edu\/ml\/datasets\/Heart+Disease. A Janosi W Steinbrunn M Pfisterer and R Detrano. 1988. Heart disease data set. In https:\/\/archive.ics.uci.edu\/ml\/datasets\/Heart+Disease."},{"key":"e_1_3_2_1_13_1","unstructured":"Hyun-Chul Kim and Zoubin Ghahramani. 2012. Bayesian classifier combination. In AISTATS. 619\u2013627. Hyun-Chul Kim and Zoubin Ghahramani. 2012. Bayesian classifier combination. In AISTATS. 619\u2013627."},{"key":"e_1_3_2_1_14_1","volume-title":"Kingma and Jimmy Ba","author":"P.","year":"2015","unstructured":"Diederik\u00a0 P. Kingma and Jimmy Ba . 2015 . Adam : A Method for Stochastic Optimization. In ICLR. Diederik\u00a0P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR."},{"volume-title":"Probabilistic graphical models: principles and techniques","author":"Koller Daphne","key":"e_1_3_2_1_15_1","unstructured":"Daphne Koller and Nir Friedman . 2009. Probabilistic graphical models: principles and techniques . MIT press . Daphne Koller and Nir Friedman. 2009. Probabilistic graphical models: principles and techniques. MIT press."},{"key":"e_1_3_2_1_16_1","unstructured":"Alex Krizhevsky Vinod Nair and Geoffrey Hinton. 2009. CIFAR-10 (Canadian Institute for Advanced Research). (2009). http:\/\/www.cs.toronto.edu\/~kriz\/cifar.html Alex Krizhevsky Vinod Nair and Geoffrey Hinton. 2009. CIFAR-10 (Canadian Institute for Advanced Research). (2009). http:\/\/www.cs.toronto.edu\/~kriz\/cifar.html"},{"key":"e_1_3_2_1_17_1","unstructured":"Kenichi Kurihara Max Welling and Yee\u00a0Whye Teh. 2007. Collapsed Variational Dirichlet Process Mixture Models.. In IJCAI Vol.\u00a07. 2796\u20132801. Kenichi Kurihara Max Welling and Yee\u00a0Whye Teh. 2007. Collapsed Variational Dirichlet Process Mixture Models.. In IJCAI Vol.\u00a07. 2796\u20132801."},{"key":"e_1_3_2_1_18_1","unstructured":"Yuan Li Benjamin Rubinstein and Trevor Cohn. 2019. Exploiting worker correlation for label aggregation in crowdsourcing. In ICML. 3886\u20133895. Yuan Li Benjamin Rubinstein and Trevor Cohn. 2019. Exploiting worker correlation for label aggregation in crowdsourcing. In ICML. 3886\u20133895."},{"key":"e_1_3_2_1_19_1","unstructured":"Qiang Liu Jian Peng and Alexander\u00a0T Ihler. 2012. Variational inference for crowdsourcing. In NeurIPS. 692\u2013700. Qiang Liu Jian Peng and Alexander\u00a0T Ihler. 2012. Variational inference for crowdsourcing. In NeurIPS. 692\u2013700."},{"key":"e_1_3_2_1_20_1","volume-title":"Bolt: Instantaneous crowdsourcing via just-in-time training. In CHI. 1\u20137.","author":"Lundgard Alan","year":"2018","unstructured":"Alan Lundgard , Yiwei Yang , Maya\u00a0 L Foster , and Walter\u00a0 S Lasecki . 2018 . Bolt: Instantaneous crowdsourcing via just-in-time training. In CHI. 1\u20137. Alan Lundgard, Yiwei Yang, Maya\u00a0L Foster, and Walter\u00a0S Lasecki. 2018. Bolt: Instantaneous crowdsourcing via just-in-time training. In CHI. 1\u20137."},{"key":"e_1_3_2_1_21_1","volume":"201","author":"McMahan Brendan","unstructured":"H.\u00a0 Brendan McMahan and Matthew\u00a0 J. Streeter. 201 0. Adaptive Bound Optimization for Online Convex Optimization. In COLT. 244\u2013256. H.\u00a0Brendan McMahan and Matthew\u00a0J. Streeter. 2010. Adaptive Bound Optimization for Online Convex Optimization. In COLT. 244\u2013256.","journal-title":"J. Streeter."},{"key":"e_1_3_2_1_22_1","unstructured":"John\u00a0W. Paisley David\u00a0M. Blei and Michael\u00a0I. Jordan. 2012. Variational Bayesian Inference with Stochastic Search. In ICML. 1363\u20131370. John\u00a0W. Paisley David\u00a0M. Blei and Michael\u00a0I. Jordan. 2012. Variational Bayesian Inference with Stochastic Search. In ICML. 1363\u20131370."},{"key":"e_1_3_2_1_23_1","volume-title":"SIGKDD workshop on human computation. 64\u201367","author":"Provost Foster","year":"2010","unstructured":"Foster Provost , Wang Jing , and Panagiotis\u00a0 G. Ipeirotis . 2010 . Quality management on amazon mechanical turk . In SIGKDD workshop on human computation. 64\u201367 . Foster Provost, Wang Jing, and Panagiotis\u00a0G. Ipeirotis. 2010. Quality management on amazon mechanical turk. In SIGKDD workshop on human computation. 64\u201367."},{"key":"e_1_3_2_1_24_1","volume-title":"Journal of Machine Learning Research","author":"Raykar C","year":"2010","unstructured":"Vikas\u00a0 C Raykar , Shipeng Yu , Linda\u00a0 H Zhao , Gerardo\u00a0Hermosillo Valadez , Charles Florin , Luca Bogoni , and Linda Moy . 2010 . Learning from crowds . Journal of Machine Learning Research 11, Apr (2010), 1297\u20131322. Vikas\u00a0C Raykar, Shipeng Yu, Linda\u00a0H Zhao, Gerardo\u00a0Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. Learning from crowds. Journal of Machine Learning Research 11, Apr (2010), 1297\u20131322."},{"key":"e_1_3_2_1_25_1","unstructured":"Pedro Savarese. 2019. On the Convergence of AdaBound and its Connection to SGD. CoRR abs\/1908.04457(2019). Pedro Savarese. 2019. On the Convergence of AdaBound and its Connection to SGD. CoRR abs\/1908.04457(2019)."},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"crossref","unstructured":"Edwin\u00a0D Simpson Matteo Venanzi Steven Reece Pushmeet Kohli John Guiver Stephen\u00a0J Roberts and Nicholas\u00a0R Jennings. 2015. Language understanding in the wild: Combining crowdsourcing and machine learning. In WWW. 992\u20131002. Edwin\u00a0D Simpson Matteo Venanzi Steven Reece Pushmeet Kohli John Guiver Stephen\u00a0J Roberts and Nicholas\u00a0R Jennings. 2015. Language understanding in the wild: Combining crowdsourcing and machine learning. In WWW. 992\u20131002.","DOI":"10.1145\/2736277.2741689"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"crossref","unstructured":"Rion Snow Brendan O\u2019Connor Daniel Jurafsky and Andrew\u00a0Y Ng. 2008. Cheap and fast\u2014but is it good?: evaluating non-expert annotations for natural language tasks. In EMNLP. 254\u2013263. Rion Snow Brendan O\u2019Connor Daniel Jurafsky and Andrew\u00a0Y Ng. 2008. Cheap and fast\u2014but is it good?: evaluating non-expert annotations for natural language tasks. In EMNLP. 254\u2013263.","DOI":"10.3115\/1613715.1613751"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Yee\u00a0W Teh David Newman and Max Welling. 2007. A collapsed variational Bayesian inference algorithm for latent Dirichlet allocation. In NeurIPS. 1353\u20131360. Yee\u00a0W Teh David Newman and Max Welling. 2007. A collapsed variational Bayesian inference algorithm for latent Dirichlet allocation. In NeurIPS. 1353\u20131360.","DOI":"10.21236\/ADA629956"},{"key":"e_1_3_2_1_29_1","volume-title":"COURSERA: Neural Networks for Machine Learning.","author":"Tieleman T.","year":"2012","unstructured":"T. Tieleman and G. Hinton . 2012 . Lecture 6.5\u2014RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning. T. Tieleman and G. Hinton. 2012. Lecture 6.5\u2014RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning."},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"crossref","unstructured":"Matteo Venanzi John Guiver Gabriella Kazai Pushmeet Kohli and Milad Shokouhi. 2014. Community-based bayesian aggregation models for crowdsourcing. In WWW. 155\u2013164. Matteo Venanzi John Guiver Gabriella Kazai Pushmeet Kohli and Milad Shokouhi. 2014. Community-based bayesian aggregation models for crowdsourcing. In WWW. 155\u2013164.","DOI":"10.1145\/2566486.2567989"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"crossref","unstructured":"Martin\u00a0J Wainwright Michael\u00a0I Jordan 2008. Graphical models exponential families and variational inference. Foundations and Trends\u00ae in Machine Learning 1 1\u20132(2008) 1\u2013305. Martin\u00a0J Wainwright Michael\u00a0I Jordan 2008. Graphical models exponential families and variational inference. Foundations and Trends\u00ae in Machine Learning 1 1\u20132(2008) 1\u2013305.","DOI":"10.1561\/2200000001"},{"key":"e_1_3_2_1_32_1","unstructured":"Jacob Whitehill Ting-fan Wu Jacob Bergsma Javier\u00a0R Movellan and Paul\u00a0L Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NeurIPS. 2035\u20132043. Jacob Whitehill Ting-fan Wu Jacob Bergsma Javier\u00a0R Movellan and Paul\u00a0L Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NeurIPS. 2035\u20132043."},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"crossref","unstructured":"Tong Xiao Tian Xia Yi Yang Chang Huang and Xiaogang Wang. 2015. Learning from massive noisy labeled data for image classification. In CVPR. 2691\u20132699. Tong Xiao Tian Xia Yi Yang Chang Huang and Xiaogang Wang. 2015. Learning from massive noisy labeled data for image classification. In CVPR. 2691\u20132699.","DOI":"10.1109\/CVPR.2015.7298885"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"crossref","unstructured":"Jie Yang Thomas Drake Andreas Damianou and Yoelle Maarek. 2018. Leveraging crowdsourcing data for deep active learning an application: Learning intents in alexa. In WWW. 23\u201332. Jie Yang Thomas Drake Andreas Damianou and Yoelle Maarek. 2018. Leveraging crowdsourcing data for deep active learning an application: Learning intents in alexa. In WWW. 23\u201332.","DOI":"10.1145\/3178876.3186033"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"crossref","unstructured":"Jie Yang Alisa Smirnova Dingqi Yang Gianluca Demartini Yuan Lu and Philippe Cudr\u00e9-Mauroux. 2019. Scalpel-cd: leveraging crowdsourcing and deep probabilistic modeling for debugging noisy training data. In WWW. 2158\u20132168. Jie Yang Alisa Smirnova Dingqi Yang Gianluca Demartini Yuan Lu and Philippe Cudr\u00e9-Mauroux. 2019. Scalpel-cd: leveraging crowdsourcing and deep probabilistic modeling for debugging noisy training data. In WWW. 2158\u20132168.","DOI":"10.1145\/3308558.3313599"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"crossref","unstructured":"Li\u2019ang Yin Jianhua Han Weinan Zhang and Yong Yu. 2017. Aggregating crowd wisdoms with label-aware autoencoders. In IJCAI. 1325\u20131331. Li\u2019ang Yin Jianhua Han Weinan Zhang and Yong Yu. 2017. Aggregating crowd wisdoms with label-aware autoencoders. In IJCAI. 1325\u20131331.","DOI":"10.24963\/ijcai.2017\/184"},{"key":"e_1_3_2_1_37_1","unstructured":"Taraneh Younesian Zilong Zhao Amirmasoud Ghiassi Robert Birke and Lydia\u00a0Y. Chen. 2020. QActor: On-line Active Learning for Noisy Labeled Stream Data. CoRR abs\/2001.10399(2020). Taraneh Younesian Zilong Zhao Amirmasoud Ghiassi Robert Birke and Lydia\u00a0Y. Chen. 2020. QActor: On-line Active Learning for Noisy Labeled Stream Data. CoRR abs\/2001.10399(2020)."},{"key":"e_1_3_2_1_38_1","unstructured":"Denny Zhou Sumit Basu Yi Mao and John\u00a0C Platt. 2012. Learning from the wisdom of crowds by minimax entropy. In NeurIPS. 2195\u20132203. Denny Zhou Sumit Basu Yi Mao and John\u00a0C Platt. 2012. Learning from the wisdom of crowds by minimax entropy. In NeurIPS. 2195\u20132203."},{"key":"e_1_3_2_1_39_1","unstructured":"Dengyong Zhou Qiang Liu John Platt and Christopher Meek. 2014. Aggregating ordinal labels from crowds by minimax conditional entropy. In ICML. 262\u2013270. Dengyong Zhou Qiang Liu John Platt and Christopher Meek. 2014. Aggregating ordinal labels from crowds by minimax conditional entropy. In ICML. 262\u2013270."},{"key":"e_1_3_2_1_40_1","unstructured":"Martin Zinkevich. 2003. Online convex programming and generalized infinitesimal gradient ascent. In ICML. 928\u2013936. Martin Zinkevich. 2003. Online convex programming and generalized infinitesimal gradient ascent. In ICML. 928\u2013936."}],"event":{"name":"WWW '21: The Web Conference 2021","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web"],"location":"Ljubljana Slovenia","acronym":"WWW '21"},"container-title":["Proceedings of the Web Conference 2021"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3442381.3449933","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3442381.3449933","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:24:32Z","timestamp":1750195472000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3442381.3449933"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,19]]},"references-count":40,"alternative-id":["10.1145\/3442381.3449933","10.1145\/3442381"],"URL":"https:\/\/doi.org\/10.1145\/3442381.3449933","relation":{},"subject":[],"published":{"date-parts":[[2021,4,19]]},"assertion":[{"value":"2021-06-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}