{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:12:59Z","timestamp":1750219979171,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":37,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,8,4]],"date-time":"2023-08-04T00:00:00Z","timestamp":1691107200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,8,6]]},"DOI":"10.1145\/3580305.3599828","type":"proceedings-article","created":{"date-parts":[[2023,8,4]],"date-time":"2023-08-04T18:13:58Z","timestamp":1691172838000},"page":"5404-5415","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["From Labels to Decisions: A Mapping-Aware Annotator Model"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-6192-3229","authenticated-orcid":false,"given":"Evan","family":"Yao","sequence":"first","affiliation":[{"name":"Massachusetts Institute of Technology, Cambridge, MA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-3299-4613","authenticated-orcid":false,"given":"Jagdish","family":"Ramakrishnan","sequence":"additional","affiliation":[{"name":"Meta, Menlo Park, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3422-0127","authenticated-orcid":false,"given":"Xu","family":"Chen","sequence":"additional","affiliation":[{"name":"Meta, Menlo Park, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7923-0882","authenticated-orcid":false,"given":"Viet-An","family":"Nguyen","sequence":"additional","affiliation":[{"name":"Meta, Menlo Park, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6966-1945","authenticated-orcid":false,"given":"Udi","family":"Weinsberg","sequence":"additional","affiliation":[{"name":"Meta, Menlo Park, MA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,8,4]]},"reference":[{"key":"e_1_3_2_2_1_1","volume-title":"The practice of crowdsourcing. Synthesis lectures on information concepts, retrieval, and services","author":"Alonso Omar","year":"2019","unstructured":"Omar Alonso . 2019. The practice of crowdsourcing. Synthesis lectures on information concepts, retrieval, and services , Vol. 11 , 1 ( 2019 ), 1--149. Omar Alonso. 2019. The practice of crowdsourcing. Synthesis lectures on information concepts, retrieval, and services, Vol. 11, 1 (2019), 1--149."},{"key":"e_1_3_2_2_2_1","unstructured":"Vashist Avadhanula Omar Abdul Baki Hamsa Bastani Osbert Bastani Caner Gocmen Daniel Haimovich Darren Hwang Dima Karamshuk Thomas Leeper Jiayuan Ma Gregory Macnamara Jake Mullett Christopher Palow Sung Park Varun S Rajagopal Kevin Schaeffer Parikshit Shah Deeksha Sinha Nicolas Stier-Moses and Peng Xu. 2022. Bandits for Online Calibration: An Application to Content Moderation on Social Media Platforms.  Vashist Avadhanula Omar Abdul Baki Hamsa Bastani Osbert Bastani Caner Gocmen Daniel Haimovich Darren Hwang Dima Karamshuk Thomas Leeper Jiayuan Ma Gregory Macnamara Jake Mullett Christopher Palow Sung Park Varun S Rajagopal Kevin Schaeffer Parikshit Shah Deeksha Sinha Nicolas Stier-Moses and Peng Xu. 2022. Bandits for Online Calibration: An Application to Content Moderation on Social Media Platforms."},{"key":"e_1_3_2_2_3_1","volume-title":"International Conference on Learning Representations.","author":"Cao Peng","year":"2019","unstructured":"Peng Cao , Yilun Xu , Yuqing Kong , and Yizhou Wang . 2019 . Max-MIG: an Information Theoretic Approach for Joint Learning from Crowds . In International Conference on Learning Representations. Peng Cao, Yilun Xu, Yuqing Kong, and Yizhou Wang. 2019. Max-MIG: an Information Theoretic Approach for Joint Learning from Crowds. In International Conference on Learning Representations."},{"key":"e_1_3_2_2_4_1","volume-title":"Multilevel Bayesian models of categorical data annotation. Unpublished manuscript","author":"Carpenter Bob","year":"2008","unstructured":"Bob Carpenter . 2008. Multilevel Bayesian models of categorical data annotation. Unpublished manuscript , Vol. 17 , 122 ( 2008 ), 45--50. Bob Carpenter. 2008. Multilevel Bayesian models of categorical data annotation. Unpublished manuscript, Vol. 17, 122 (2008), 45--50."},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v076.i01"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i7.16730"},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3148148"},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.2307\/2346806"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313749"},{"volume-title":"Advances in Knowledge Discovery and Data Mining, Kamal Karlapalem, Hong Cheng, Naren Ramakrishnan","author":"Enomoto Masafumi","key":"e_1_3_2_2_10_1","unstructured":"Masafumi Enomoto , Kunihiro Takeoka , Yuyang Dong , Masafumi Oyamada , and Takeshi Okadome . 2021. Quality Control for Hierarchical Classification with Incomplete Annotations . In Advances in Knowledge Discovery and Data Mining, Kamal Karlapalem, Hong Cheng, Naren Ramakrishnan , R. K. Agrawal, P. Krishna Reddy, Jaideep Srivastava, and Tanmoy Chakraborty (Eds.). Springer International Publishing , Cham , 219--230. Masafumi Enomoto, Kunihiro Takeoka, Yuyang Dong, Masafumi Oyamada, and Takeshi Okadome. 2021. Quality Control for Hierarchical Classification with Incomplete Annotations. In Advances in Knowledge Discovery and Data Mining, Kamal Karlapalem, Hong Cheng, Naren Ramakrishnan, R. K. Agrawal, P. Krishna Reddy, Jaideep Srivastava, and Tanmoy Chakraborty (Eds.). Springer International Publishing, Cham, 219--230."},{"key":"e_1_3_2_2_11_1","unstructured":"Facebook. 2016. Introducing FBLearner Flow: Facebook's AI backbone. https:\/\/code.fb.com\/core-data\/introducing-fblearner-flow-facebook-s-ai-backbone\/  Facebook. 2016. Introducing FBLearner Flow: Facebook's AI backbone. https:\/\/code.fb.com\/core-data\/introducing-fblearner-flow-facebook-s-ai-backbone\/"},{"key":"e_1_3_2_2_12_1","volume-title":"Kwok","author":"Guo Xiawei","year":"2016","unstructured":"Xiawei Guo and James T . Kwok . 2016 . Aggregating Crowdsourced Ordinal Labels\u00a0via\u00a0Bayesian Clustering. In Machine Learning and Knowledge Discovery in Databases, Paolo Frasconi, Niels Landwehr, Giuseppe Manco, and Jilles Vreeken (Eds.). Springer International Publishing , Cham, 426--442. Xiawei Guo and James T. Kwok. 2016. Aggregating Crowdsourced Ordinal Labels\u00a0via\u00a0Bayesian Clustering. In Machine Learning and Knowledge Discovery in Databases, Paolo Frasconi, Niels Landwehr, Giuseppe Manco, and Jilles Vreeken (Eds.). Springer International Publishing, Cham, 426--442."},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v3i1.13238"},{"key":"e_1_3_2_2_14_1","unstructured":"Hyun-Chul Kim and Zoubin Ghahramani. 2012. Bayesian classifier combination. In Artificial Intelligence and Statistics (AISTATS). 619--627.  Hyun-Chul Kim and Zoubin Ghahramani. 2012. Bayesian classifier combination. In Artificial Intelligence and Statistics (AISTATS). 619--627."},{"key":"e_1_3_2_2_15_1","volume-title":"Beyond confusion matrix: learning from multiple annotators with awareness of instance features. Machine Learning","author":"Li Jingzheng","year":"2022","unstructured":"Jingzheng Li , Hailong Sun , and Jiyi Li. 2022. Beyond confusion matrix: learning from multiple annotators with awareness of instance features. Machine Learning ( 2022 ), 1--23. Jingzheng Li, Hailong Sun, and Jiyi Li. 2022. Beyond confusion matrix: learning from multiple annotators with awareness of instance features. Machine Learning (2022), 1--23."},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897350.2897352"},{"key":"e_1_3_2_2_17_1","unstructured":"Yuan Li Benjamin Rubinstein and Trevor Cohn. 2019. Exploiting Worker Correlation for Label Aggregation in Crowdsourcing. In ICML. 3886--3895.  Yuan Li Benjamin Rubinstein and Trevor Cohn. 2019. Exploiting Worker Correlation for Label Aggregation in Crowdsourcing. In ICML. 3886--3895."},{"key":"e_1_3_2_2_18_1","unstructured":"Qiang Liu Jian Peng and Alexander Ihler. 2012. Variational Inference for Crowdsourcing. In Advances in Neural Information Processing Systems. 692--700.  Qiang Liu Jian Peng and Alexander Ihler. 2012. Variational Inference for Crowdsourcing. In Advances in Neural Information Processing Systems. 692--700."},{"key":"e_1_3_2_2_19_1","unstructured":"Meta. 2022a. Community Standards Enforcement Report. https:\/\/transparency.fb.com\/data\/community-standards-enforcement  Meta. 2022a. Community Standards Enforcement Report. https:\/\/transparency.fb.com\/data\/community-standards-enforcement"},{"key":"e_1_3_2_2_20_1","unstructured":"Meta. 2022b. How Meta enforces its policies. https:\/\/transparency.fb.com\/enforcement\/  Meta. 2022b. How Meta enforces its policies. https:\/\/transparency.fb.com\/enforcement\/"},{"key":"e_1_3_2_2_21_1","volume-title":"Yee Whye Teh, and Fernando Perez-Cruz","author":"Moreno Pablo G.","year":"2015","unstructured":"Pablo G. Moreno , Antonio Art\u00e9s-Rodr\u00edguez , Yee Whye Teh, and Fernando Perez-Cruz . 2015 . Bayesian Nonparametric Crowdsourcing. JMLR ( 2015), 1607--1627. Pablo G. Moreno, Antonio Art\u00e9s-Rodr\u00edguez, Yee Whye Teh, and Fernando Perez-Cruz. 2015. Bayesian Nonparametric Crowdsourcing. JMLR (2015), 1607--1627."},{"key":"e_1_3_2_2_22_1","unstructured":"An T. Nguyen Byron C. Wallace and Matthew Lease. 2016. A Correlated Worker Model for Grouped Imbalanced and Multitask Data. In UAI. 537--546.  An T. Nguyen Byron C. Wallace and Matthew Lease. 2016. A Correlated Worker Model for Grouped Imbalanced and Multitask Data. In UAI. 537--546."},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539184"},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3403304"},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2016.04.009"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.2200\/S01131ED1V01Y202109HLT054"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00040"},{"key":"e_1_3_2_2_28_1","volume-title":"JMLR","volume":"11","author":"Raykar Vikas C.","year":"2010","unstructured":"Vikas C. Raykar , Shipeng Yu , Linda H. Zhao , Gerardo Hermosillo Valadez , Charles Florin , Luca Bogoni , and Linda Moy . 2010 . Learning From Crowds . JMLR , Vol. 11 (Aug. 2010), 1297--1322. Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. Learning From Crowds. JMLR, Vol. 11 (Aug. 2010), 1297--1322."},{"key":"e_1_3_2_2_29_1","volume-title":"Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing","author":"Snow Rion","year":"2008","unstructured":"Rion Snow , Brendan O'Connor , Daniel Jurafsky , and Andrew Ng . 2008 . Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks . In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing ( Honolulu, Hawaii). Association for Computational Linguistics, 254--263. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (Honolulu, Hawaii). Association for Computational Linguistics, 254--263."},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567989"},{"key":"e_1_3_2_2_31_1","volume-title":"Cornelis AW Glas, and HH De Vries","author":"Verhelst Norman D","year":"1997","unstructured":"Norman D Verhelst , Cornelis AW Glas, and HH De Vries . 1997 . A steps model to analyze partial credit. In Handbook of modern item response theory. Springer , 123--138. Norman D Verhelst, Cornelis AW Glas, and HH De Vries. 1997. A steps model to analyze partial credit. In Handbook of modern item response theory. Springer, 123--138."},{"key":"e_1_3_2_2_32_1","volume-title":"Advances in Neural Information Processing Systems","author":"Whitehill Jacob","year":"2009","unstructured":"Jacob Whitehill , Ting-fan Wu, Jacob Bergsma , Javier Movellan , and Paul Ruvolo . 2009. Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise . In Advances in Neural Information Processing Systems , Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta (Eds.), Vol. 22 . Curran Associates, Inc. https:\/\/proceedings.neurips.cc\/paper\/ 2009 \/file\/f899139df5e1059396431415e770c6dd-Paper.pdf Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. 2009. Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise. In Advances in Neural Information Processing Systems, Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta (Eds.), Vol. 22. Curran Associates, Inc. https:\/\/proceedings.neurips.cc\/paper\/2009\/file\/f899139df5e1059396431415e770c6dd-Paper.pdf"},{"key":"e_1_3_2_2_33_1","volume-title":"Crowdsourcing Truth Inference Based on Label Confidence Clustering. ACM Trans. Knowl. Discov. Data (aug","author":"Wu Gongqing","year":"2022","unstructured":"Gongqing Wu , Liangzhu Zhou , Jiazhu Xia , Lei Li , Xianyu Bao , and Xindong Wu. 2022. Crowdsourcing Truth Inference Based on Label Confidence Clustering. ACM Trans. Knowl. Discov. Data (aug 2022 ). https:\/\/doi.org\/10.1145\/3556545 Just Accepted . 10.1145\/3556545 Gongqing Wu, Liangzhu Zhou, Jiazhu Xia, Lei Li, Xianyu Bao, and Xindong Wu. 2022. Crowdsourcing Truth Inference Based on Label Confidence Clustering. ACM Trans. Knowl. Discov. Data (aug 2022). https:\/\/doi.org\/10.1145\/3556545 Just Accepted."},{"key":"e_1_3_2_2_34_1","unstructured":"Yan Yan R\u00f3mer Rosales Glenn Fung Mark Schmidt Gerardo Hermosillo Luca Bogoni Linda Moy and Jennifer Dy. 2010. Modeling annotator expertise: Learning when everybody knows a bit of something. In Artificial Intelligence and Statistics (AISTATS). 932--939.  Yan Yan R\u00f3mer Rosales Glenn Fung Mark Schmidt Gerardo Hermosillo Luca Bogoni Linda Moy and Jennifer Dy. 2010. Modeling annotator expertise: Learning when everybody knows a bit of something. In Artificial Intelligence and Statistics (AISTATS). 932--939."},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-013-5412-1"},{"key":"e_1_3_2_2_36_1","volume-title":"Spectral Methods Meet EM: A Provably Optimal Algorithm for Crowdsourcing. JMLR","volume":"17","author":"Zhang Yuchen","year":"2016","unstructured":"Yuchen Zhang , Xi Chen , Dengyong Zhou , and Michael I. Jordan . 2016 . Spectral Methods Meet EM: A Provably Optimal Algorithm for Crowdsourcing. JMLR , Vol. 17 , 1 ( Jan. 2016 ), 3537--3580. Yuchen Zhang, Xi Chen, Dengyong Zhou, and Michael I. Jordan. 2016. Spectral Methods Meet EM: A Provably Optimal Algorithm for Crowdsourcing. JMLR, Vol. 17, 1 (Jan. 2016), 3537--3580."},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.14778\/3055540.3055547"}],"event":{"name":"KDD '23: The 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data"],"location":"Long Beach CA USA","acronym":"KDD '23"},"container-title":["Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3580305.3599828","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3580305.3599828","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:49:23Z","timestamp":1750182563000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3580305.3599828"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,4]]},"references-count":37,"alternative-id":["10.1145\/3580305.3599828","10.1145\/3580305"],"URL":"https:\/\/doi.org\/10.1145\/3580305.3599828","relation":{},"subject":[],"published":{"date-parts":[[2023,8,4]]},"assertion":[{"value":"2023-08-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}