{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T18:49:41Z","timestamp":1775155781987,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":56,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,6,26]],"date-time":"2022-06-26T00:00:00Z","timestamp":1656201600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,6,26]]},"DOI":"10.1145\/3501247.3531580","type":"proceedings-article","created":{"date-parts":[[2022,6,24]],"date-time":"2022-06-24T22:41:09Z","timestamp":1656110469000},"page":"118-128","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["HAEM: Obtaining Higher-Quality Classification Task Results with AI Workers"],"prefix":"10.1145","author":[{"given":"Yu","family":"Yamashita","sequence":"first","affiliation":[{"name":"University of Tsukuba, Japan"}]},{"given":"Hiroyoshi","family":"Ito","sequence":"additional","affiliation":[{"name":"University of Tsukuba, Japan"}]},{"given":"Kei","family":"Wakabayashi","sequence":"additional","affiliation":[{"name":"University of Tsukuba, Japan"}]},{"given":"Masaki","family":"Kobayashi","sequence":"additional","affiliation":[{"name":"University of Tsukuba, Japan"}]},{"given":"Atsuyuki","family":"Morishima","sequence":"additional","affiliation":[{"name":"University of Tsukuba, Japan"}]}],"member":"320","published-online":{"date-parts":[[2022,6,26]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2021.106771"},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3271553.3271578"},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445736"},{"key":"e_1_3_2_2_4_1","unstructured":"Jordan\u00a0T Ash Chicheng Zhang Akshay Krishnamurthy John Langford and Alekh Agarwal. 2019. Deep batch active learning by diverse uncertain gradient lower bounds. arXiv preprint arXiv:1906.03671(2019).  Jordan\u00a0T Ash Chicheng Zhang Akshay Krishnamurthy John Langford and Alekh Agarwal. 2019. Deep batch active learning by diverse uncertain gradient lower bounds. arXiv preprint arXiv:1906.03671(2019)."},{"key":"e_1_3_2_2_5_1","volume-title":"Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems 32","author":"Barbu Andrei","year":"2019","unstructured":"Andrei Barbu , David Mayo , Julian Alverio , William Luo , Christopher Wang , Dan Gutfreund , Josh Tenenbaum , and Boris Katz . 2019 . Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems 32 (2019), 9453\u20139463. Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. 2019. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems 32 (2019), 9453\u20139463."},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00976"},{"key":"e_1_3_2_2_7_1","volume-title":"NIPS Workshop on Analyzing Networks and Learning with Graphs, Vol.\u00a04.","author":"Bilgic Mustafa","year":"2009","unstructured":"Mustafa Bilgic and Lise Getoor . 2009 . Link-based active learning . In NIPS Workshop on Analyzing Networks and Learning with Graphs, Vol.\u00a04. Mustafa Bilgic and Lise Getoor. 2009. Link-based active learning. In NIPS Workshop on Analyzing Networks and Learning with Graphs, Vol.\u00a04."},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173951"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2021.102062"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300234"},{"key":"e_1_3_2_2_11_1","volume-title":"International conference on machine learning. PMLR, 64\u201372","author":"Chen Xi","year":"2013","unstructured":"Xi Chen , Qihang Lin , and Dengyong Zhou . 2013 . Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing . In International conference on machine learning. PMLR, 64\u201372 . Xi Chen, Qihang Lin, and Dengyong Zhou. 2013. Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing. In International conference on machine learning. PMLR, 64\u201372."},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00449"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.2307\/2346806"},{"key":"e_1_3_2_2_14_1","first-page":"114","article-title":"Algorithm aversion: people erroneously avoid algorithms after seeing them err.Journal of Experimental Psychology","volume":"144","author":"Dietvorst J","year":"2015","unstructured":"Berkeley\u00a0 J Dietvorst , Joseph\u00a0 P Simmons , and Cade Massey . 2015 . Algorithm aversion: people erroneously avoid algorithms after seeing them err.Journal of Experimental Psychology : General 144 , 1 (2015), 114 . Berkeley\u00a0J Dietvorst, Joseph\u00a0P Simmons, and Cade Massey. 2015. Algorithm aversion: people erroneously avoid algorithms after seeing them err.Journal of Experimental Psychology: General 144, 1 (2015), 114.","journal-title":"General"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3302310"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2750550"},{"key":"e_1_3_2_2_17_1","volume-title":"Automated Machine Learning","author":"Feurer Matthias","unstructured":"Matthias Feurer and Frank Hutter . 2019. Hyperparameter optimization . In Automated Machine Learning . Springer , Cham , 3\u201333. Matthias Feurer and Frank Hutter. 2019. Hyperparameter optimization. In Automated Machine Learning. Springer, Cham, 3\u201333."},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10593-2_37"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.3465622"},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"crossref","unstructured":"Peter Hase and Mohit Bansal. 2020. Evaluating explainable AI: Which algorithmic explanations help users predict model behavior?arXiv preprint arXiv:2005.01831(2020).  Peter Hase and Mohit Bansal. 2020. Evaluating explainable AI: Which algorithmic explanations help users predict model behavior?arXiv preprint arXiv:2005.01831(2020).","DOI":"10.18653\/v1\/2020.acl-main.491"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00342"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567988"},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3064055"},{"key":"e_1_3_2_2_25_1","unstructured":"Ashish Khetan Zachary\u00a0C Lipton and Anima Anandkumar. 2017. Learning from noisy singly-labeled data. arXiv preprint arXiv:1712.04577(2017).  Ashish Khetan Zachary\u00a0C Lipton and Anima Anandkumar. 2017. Learning from noisy singly-labeled data. arXiv preprint arXiv:1712.04577(2017)."},{"key":"e_1_3_2_2_26_1","volume-title":"Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980(2014).","author":"Kingma P","year":"2014","unstructured":"Diederik\u00a0 P Kingma and Jimmy Ba . 2014 . Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980(2014). Diederik\u00a0P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980(2014)."},{"key":"e_1_3_2_2_27_1","unstructured":"Alex Krizhevsky Geoffrey Hinton 2009. Learning multiple layers of features from tiny images. (2009).  Alex Krizhevsky Geoffrey Hinton 2009. Learning multiple layers of features from tiny images. (2009)."},{"key":"e_1_3_2_2_28_1","volume-title":"Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25","author":"Krizhevsky Alex","year":"2012","unstructured":"Alex Krizhevsky , Ilya Sutskever , and Geoffrey\u00a0 E Hinton . 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25 ( 2012 ), 1097\u20131105. Alex Krizhevsky, Ilya Sutskever, and Geoffrey\u00a0E Hinton. 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25 (2012), 1097\u20131105."},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207678"},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300717"},{"key":"e_1_3_2_2_31_1","unstructured":"Vivian Lai Chacha Chen Q\u00a0Vera Liao Alison Smith-Renner and Chenhao Tan. 2021. Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies. arXiv preprint arXiv:2112.11471(2021).  Vivian Lai Chacha Chen Q\u00a0Vera Liao Alison Smith-Renner and Chenhao Tan. 2021. Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies. arXiv preprint arXiv:2112.11471(2021)."},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939874"},{"key":"e_1_3_2_2_33_1","unstructured":"Ya Le and Xuan Yang. 2015. Tiny imagenet visual recognition challenge. CS 231N 7(2015) 7.  Ya Le and Xuan Yang. 2015. Tiny imagenet visual recognition challenge. CS 231N 7(2015) 7."},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSTARS.2018.2874225"},{"key":"e_1_3_2_2_35_1","volume-title":"Proceedings of the 35th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol.\u00a080)","author":"Lipton Zachary","year":"2018","unstructured":"Zachary Lipton , Yu-Xiang Wang , and Alexander Smola . 2018 . Detecting and Correcting for Label Shift with Black Box Predictors . In Proceedings of the 35th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol.\u00a080) , Jennifer Dyand Andreas Krause (Eds.). PMLR, 3122\u20133130. http:\/\/proceedings.mlr.press\/v80\/lipton18a.html Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. 2018. Detecting and Correcting for Label Shift with Black Box Predictors. In Proceedings of the 35th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol.\u00a080), Jennifer Dyand Andreas Krause (Eds.). PMLR, 3122\u20133130. http:\/\/proceedings.mlr.press\/v80\/lipton18a.html"},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479552"},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSTARS.2016.2598859"},{"key":"e_1_3_2_2_38_1","unstructured":"Xuan Liu Meiyu Lu Beng\u00a0Chin Ooi Yanyan Shen Sai Wu and Meihui Zhang. 2012. Cdas: a crowdsourcing data analytics system. arXiv preprint arXiv:1207.0143(2012).  Xuan Liu Meiyu Lu Beng\u00a0Chin Ooi Yanyan Shen Sai Wu and Meihui Zhang. 2012. Cdas: a crowdsourcing data analytics system. arXiv preprint arXiv:1207.0143(2012)."},{"key":"e_1_3_2_2_39_1","unstructured":"Aditya\u00a0Krishna Menon Brendan Van\u00a0Rooyen and Nagarajan Natarajan. 2016. Learning from binary labels with instance-dependent corruption. arXiv preprint arXiv:1605.00751(2016).  Aditya\u00a0Krishna Menon Brendan Van\u00a0Rooyen and Nagarajan Natarajan. 2016. Learning from binary labels with instance-dependent corruption. arXiv preprint arXiv:1605.00751(2016)."},{"key":"e_1_3_2_2_40_1","unstructured":"Nagarajan Natarajan Inderjit\u00a0S Dhillon Pradeep Ravikumar and Ambuj Tewari. 2013. Learning with noisy labels.. In NIPS Vol.\u00a026. 1196\u20131204.  Nagarajan Natarajan Inderjit\u00a0S Dhillon Pradeep Ravikumar and Ambuj Tewari. 2013. Learning with noisy labels.. In NIPS Vol.\u00a026. 1196\u20131204."},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"crossref","unstructured":"Curtis\u00a0G Northcutt Lu Jiang and Isaac\u00a0L Chuang. 2021. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research(2021).  Curtis\u00a0G Northcutt Lu Jiang and Isaac\u00a0L Chuang. 2021. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research(2021).","DOI":"10.1613\/jair.1.12125"},{"key":"e_1_3_2_2_42_1","volume-title":"6th Italian Conference on Computational Linguistics, CLiC-it","author":"Poletto Fabio","year":"2019","unstructured":"Fabio Poletto , Valerio Basile , Cristina Bosco , Viviana Patti , and Marco Stranisci . 2019 . Annotating hate speech: Three schemes at comparison . In 6th Italian Conference on Computational Linguistics, CLiC-it 2019, Vol.\u00a02481. CEUR-WS, 1\u20138. Fabio Poletto, Valerio Basile, Cristina Bosco, Viviana Patti, and Marco Stranisci. 2019. Annotating hate speech: Three schemes at comparison. In 6th Italian Conference on Computational Linguistics, CLiC-it 2019, Vol.\u00a02481. CEUR-WS, 1\u20138."},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICIP.2017.8297020"},{"key":"e_1_3_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3447582"},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11506"},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"crossref","unstructured":"Paul R\u00f6ttger Bertie Vidgen Dirk Hovy and Janet\u00a0B Pierrehumbert. 2021. Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks. arXiv preprint arXiv:2112.07475(2021).  Paul R\u00f6ttger Bertie Vidgen Dirk Hovy and Janet\u00a0B Pierrehumbert. 2021. Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks. arXiv preprint arXiv:2112.07475(2021).","DOI":"10.18653\/v1\/2022.naacl-main.13"},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3106426.3106436"},{"key":"e_1_3_2_2_48_1","unstructured":"Christopher Schr\u00f6der and Andreas Niekler. 2020. A survey of active learning for text classification using deep neural networks. arXiv preprint arXiv:2008.07267(2020).  Christopher Schr\u00f6der and Andreas Niekler. 2020. A survey of active learning for text classification using deep neural networks. arXiv preprint arXiv:2008.07267(2020)."},{"key":"e_1_3_2_2_49_1","volume-title":"Conference on learning theory. PMLR, 489\u2013511","author":"Scott Clayton","year":"2013","unstructured":"Clayton Scott , Gilles Blanchard , and Gregory Handy . 2013 . Classification with asymmetric label noise: Consistency and maximal denoising . In Conference on learning theory. PMLR, 489\u2013511 . Clayton Scott, Gilles Blanchard, and Gregory Handy. 2013. Classification with asymmetric label noise: Consistency and maximal denoising. In Conference on learning theory. PMLR, 489\u2013511."},{"key":"e_1_3_2_2_50_1","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556(2014).  Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556(2014)."},{"key":"e_1_3_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2016.2589879"},{"key":"e_1_3_2_2_52_1","unstructured":"Peter Welinder Steve Branson Pietro Perona and Serge Belongie. 2011. The multidimensional wisdom of crowds. Neural Information Processing Systems.  Peter Welinder Steve Branson Pietro Perona and Serge Belongie. 2011. The multidimensional wisdom of crowds. Neural Information Processing Systems."},{"key":"e_1_3_2_2_53_1","volume-title":"Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in neural information processing systems 22","author":"Whitehill Jacob","year":"2009","unstructured":"Jacob Whitehill , Ting-fan Wu, Jacob Bergsma , Javier Movellan , and Paul Ruvolo . 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in neural information processing systems 22 ( 2009 ), 2035\u20132043. Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in neural information processing systems 22 (2009), 2035\u20132043."},{"key":"e_1_3_2_2_54_1","unstructured":"Zhilu Zhang and Mert\u00a0R Sabuncu. 2018. Generalized cross entropy loss for training deep neural networks with noisy labels. arXiv preprint arXiv:1805.07836(2018).  Zhilu Zhang and Mert\u00a0R Sabuncu. 2018. Generalized cross entropy loss for training deep neural networks with noisy labels. arXiv preprint arXiv:1805.07836(2018)."},{"key":"e_1_3_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2749430"},{"key":"e_1_3_2_2_56_1","unstructured":"Dengyong Zhou Qiang Liu John\u00a0C Platt Christopher Meek and Nihar\u00a0B Shah. 2015. Regularized minimax conditional entropy for crowdsourcing. arXiv preprint arXiv:1503.07240(2015).  Dengyong Zhou Qiang Liu John\u00a0C Platt Christopher Meek and Nihar\u00a0B Shah. 2015. Regularized minimax conditional entropy for crowdsourcing. arXiv preprint arXiv:1503.07240(2015)."}],"event":{"name":"WebSci '22: 14th ACM Web Science Conference 2022","location":"Barcelona Spain","acronym":"WebSci '22","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web"]},"container-title":["14th ACM Web Science Conference 2022"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3501247.3531580","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3501247.3531580","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:12:18Z","timestamp":1750191138000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3501247.3531580"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,26]]},"references-count":56,"alternative-id":["10.1145\/3501247.3531580","10.1145\/3501247"],"URL":"https:\/\/doi.org\/10.1145\/3501247.3531580","relation":{},"subject":[],"published":{"date-parts":[[2022,6,26]]},"assertion":[{"value":"2022-06-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}