{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T16:05:45Z","timestamp":1776096345637,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":72,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,5,6]],"date-time":"2021-05-06T00:00:00Z","timestamp":1620259200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,5,6]]},"DOI":"10.1145\/3411764.3445423","type":"proceedings-article","created":{"date-parts":[[2021,5,8]],"date-time":"2021-05-08T02:46:03Z","timestamp":1620441963000},"page":"1-14","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":67,"title":["The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality"],"prefix":"10.1145","author":[{"given":"Mitchell L.","family":"Gordon","sequence":"first","affiliation":[{"name":"Computer Science Department, Stanford University, United States"}]},{"given":"Kaitlyn","family":"Zhou","sequence":"additional","affiliation":[{"name":"Computer Science Department, Stanford University, United States"}]},{"given":"Kayur","family":"Patel","sequence":"additional","affiliation":[{"name":"Apple Inc., United States"}]},{"given":"Tatsunori","family":"Hashimoto","sequence":"additional","affiliation":[{"name":"Computer Science Department, Stanford University, United States"}]},{"given":"Michael S.","family":"Bernstein","sequence":"additional","affiliation":[{"name":"Computer Science Department, Stanford University, United States"}]}],"member":"320","published-online":{"date-parts":[[2021,5,7]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300760"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00042"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v35i4.2513"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300233"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2017\/195"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0022-3999(02)00330-6"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1080\/01621459.1968.11009221","article-title":"Recent research in reinterview procedures","volume":"63","author":"Bailar A","year":"1968","unstructured":"Barbara\u00a0 A Bailar . 1968 . Recent research in reinterview procedures . J. Amer. Statist. Assoc. 63 , 321 (1968), 41 \u2013 63 . Barbara\u00a0A Bailar. 1968. Recent research in reinterview procedures. J. Amer. Statist. Assoc. 63, 321 (1968), 41\u201363.","journal-title":"J. Amer. Statist. Assoc."},{"key":"e_1_3_2_1_8_1","volume-title":"Who Moderates the Social Media Giants?Center for Business","author":"Barrett M","year":"2020","unstructured":"Paul\u00a0 M Barrett . 2020. Who Moderates the Social Media Giants?Center for Business ( 2020 ). Paul\u00a0M Barrett. 2020. Who Moderates the Social Media Giants?Center for Business (2020)."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3415164"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-67256-4_32"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308560.3317593"},{"key":"e_1_3_2_1_12_1","volume-title":"Bowker and Susan\u00a0Leigh Star","author":"C.","year":"2000","unstructured":"Geoffery\u00a0 C. Bowker and Susan\u00a0Leigh Star . 2000 . Sorting Things out: Classification and Its Consequences. MIT Press , Cambridge, MA, USA. Geoffery\u00a0C. Bowker and Susan\u00a0Leigh Star. 2000. Sorting Things out: Classification and Its Consequences. MIT Press, Cambridge, MA, USA."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3242587.3242598"},{"key":"e_1_3_2_1_14_1","volume-title":"Mechanical Turk. In Proceedings of the 2009 conference on empirical methods in natural language processing. 286\u2013295","author":"Callison-Burch Chris","year":"2009","unstructured":"Chris Callison-Burch . 2009 . Fast, cheap, and creative: Evaluating translation quality using Amazon\u2019s Mechanical Turk. In Proceedings of the 2009 conference on empirical methods in natural language processing. 286\u2013295 . Chris Callison-Burch. 2009. Fast, cheap, and creative: Evaluating translation quality using Amazon\u2019s Mechanical Turk. In Proceedings of the 2009 conference on empirical methods in natural language processing. 286\u2013295."},{"key":"e_1_3_2_1_15_1","volume-title":"Tiered governance and demonetization: The shifting terms of labor and compensation in the platform economy. Social Media+ Society 6, 2","author":"Caplan Robyn","year":"2020","unstructured":"Robyn Caplan and Tarleton Gillespie . 2020. Tiered governance and demonetization: The shifting terms of labor and compensation in the platform economy. Social Media+ Society 6, 2 ( 2020 ), 2056305120936636. Robyn Caplan and Tarleton Gillespie. 2020. Tiered governance and demonetization: The shifting terms of labor and compensation in the platform economy. Social Media+ Society 6, 2 (2020), 2056305120936636."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819963"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026044"},{"key":"e_1_3_2_1_18_1","volume-title":"Proceedings of the 27th international conference on machine learning (ICML-10)","author":"Cheng Weiwei","year":"2010","unstructured":"Weiwei Cheng , Eyke H\u00fcllermeier , and Krzysztof\u00a0 J Dembczynski . 2010 . Bayes optimal multilabel classification via probabilistic classifier chains . In Proceedings of the 27th international conference on machine learning (ICML-10) . 279\u2013286. Weiwei Cheng, Eyke H\u00fcllermeier, and Krzysztof\u00a0J Dembczynski. 2010. Bayes optimal multilabel classification via probabilistic classifier chains. In Proceedings of the 27th international conference on machine learning (ICML-10). 279\u2013286."},{"key":"e_1_3_2_1_19_1","unstructured":"Zhendong Chu Jing Ma and Hongning Wang. 2020. Learning from Crowds by Modeling Common Confusions. arxiv:cs.LG\/2012.13052 Zhendong Chu Jing Ma and Hongning Wang. 2020. Learning from Crowds by Modeling Common Confusions. arxiv:cs.LG\/2012.13052"},{"key":"e_1_3_2_1_20_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019)","author":"Joon\u00a0Young Chung John","year":"2019","unstructured":"John Joon\u00a0Young Chung , Jean\u00a0 Y Song , Sindhu Kutty , Sungsoo Hong , Juho Kim , and Walter\u00a0 S Lasecki . 2019 . Efficient elicitation approaches to estimate collective crowd answers . Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019) , 1\u201325. John Joon\u00a0Young Chung, Jean\u00a0Y Song, Sindhu Kutty, Sungsoo Hong, Juho Kim, and Walter\u00a0S Lasecki. 2019. Efficient elicitation approaches to estimate collective crowd answers. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1\u201325."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"crossref","unstructured":"J. Deng W. Dong R. Socher L.-J. Li K. Li and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09. J. Deng W. Dong R. Socher L.-J. Li K. Li and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173694"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-18818-8_43"},{"key":"e_1_3_2_1_24_1","unstructured":"Anca Dumitrache Lora Aroyo and Chris Welty. 2017. Crowdsourcing ground truth for medical relation extraction. (2017). arXiv:1701.02185 Anca Dumitrache Lora Aroyo and Chris Welty. 2017. Crowdsourcing ground truth for medical relation extraction. (2017). arXiv:1701.02185"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"crossref","unstructured":"Anca Dumitrache Lora Aroyo and Chris Welty. 2018. Capturing ambiguity in crowdsourcing frame disambiguation. (2018). arXiv:1805.00270 Anca Dumitrache Lora Aroyo and Chris Welty. 2018. Capturing ambiguity in crowdsourcing frame disambiguation. (2018). arXiv:1805.00270","DOI":"10.1609\/hcomp.v6i1.13330"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702556"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/604045.604056"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"crossref","unstructured":"G\u00f6sta Forsman and Irwin Schreiner. 2004. The design and analysis of reinterview: an overview. Measurement errors in surveys(2004) 279\u2013301. G\u00f6sta Forsman and Irwin Schreiner. 2004. The design and analysis of reinterview: an overview. Measurement errors in surveys(2004) 279\u2013301.","DOI":"10.1002\/9781118150382.ch15"},{"key":"e_1_3_2_1_29_1","unstructured":"Matt Gardner Yoav Artzi Victoria Basmova Jonathan Berant Ben Bogin Sihao Chen Pradeep Dasigi Dheeru Dua Yanai Elazar Ananth Gottumukkala 2020. Evaluating nlp models via contrast sets. (2020). arXiv:2004.02709 Matt Gardner Yoav Artzi Victoria Basmova Jonathan Berant Ben Bogin Sihao Chen Pradeep Dasigi Dheeru Dua Yanai Elazar Ananth Gottumukkala 2020. Evaluating nlp models via contrast sets. (2020). arXiv:2004.02709"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3409559"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3200947.3208069"},{"key":"e_1_3_2_1_32_1","volume-title":"Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media","author":"Gillespie Tarleton","unstructured":"Tarleton Gillespie . 2018. Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media . Yale University Press . Tarleton Gillespie. 2018. Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media. Yale University Press."},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02288892"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02288925"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/302979.303030"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.3115\/1564131.1564137"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"crossref","unstructured":"Maurice Jakesch Moran Koren Anna Evtushenko and Mor Naaman. 2019. The Role of Source and Expressive Responding in Political News Evaluation. (2019). Maurice Jakesch Moran Koren Anna Evtushenko and Mor Naaman. 2019. The Role of Source and Expressive Responding in Political News Evaluation. (2019).","DOI":"10.2139\/ssrn.3306403"},{"key":"e_1_3_2_1_38_1","unstructured":"Jogsaw. [n.d.]. Jigsaw Unintended Bias in Toxicity Classification. https:\/\/www.kaggle.com\/c\/jigsaw-unintended-bias-in-toxicity-classification\/overview Jogsaw. [n.d.]. Jigsaw Unintended Bias in Toxicity Classification. https:\/\/www.kaggle.com\/c\/jigsaw-unintended-bias-in-toxicity-classification\/overview"},{"key":"e_1_3_2_1_39_1","unstructured":"Charles\u00a0D Jones. [n.d.]. Accuracy of 1970 Census Population and Housing Characteristics as Measured by Reinterviews. Charles\u00a0D Jones. [n.d.]. Accuracy of 1970 Census Population and Housing Characteristics as Measured by Reinterviews."},{"key":"e_1_3_2_1_40_1","unstructured":"Jeremy Kahn. 2020. Can Facebook\u2019s new A.I. banish Pepe the Frog?https:\/\/fortune.com\/2020\/05\/12\/facebook-a-i-hate-speech-covid-19-misinformation\/ Jeremy Kahn. 2020. Can Facebook\u2019s new A.I. banish Pepe the Frog?https:\/\/fortune.com\/2020\/05\/12\/facebook-a-i-hate-speech-covid-19-misinformation\/"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820016"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1171"},{"key":"e_1_3_2_1_43_1","volume-title":"SIGIR 2010 workshop on crowdsourcing for search evaluation, Vol.\u00a02126","author":"Le John","year":"2010","unstructured":"John Le , Andy Edmonds , Vaughn Hester , and Lukas Biewald . 2010 . Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution . In SIGIR 2010 workshop on crowdsourcing for search evaluation, Vol.\u00a02126 . 22\u201332. John Le, Andy Edmonds, Vaughn Hester, and Lukas Biewald. 2010. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In SIGIR 2010 workshop on crowdsourcing for search evaluation, Vol.\u00a02126. 22\u201332."},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308560.3317082"},{"key":"e_1_3_2_1_45_1","volume-title":"Understanding Human and Predictive Moderation of Online Science Discourse. In 2019 IEEE Western New York Image and Signal Processing Workshop (WNYISPW). IEEE, 1\u20135.","author":"Lucas Elizabeth","year":"2019","unstructured":"Elizabeth Lucas , Cecilia\u00a0 O Alm , and Reynold Bailey . 2019 . Understanding Human and Predictive Moderation of Online Science Discourse. In 2019 IEEE Western New York Image and Signal Processing Workshop (WNYISPW). IEEE, 1\u20135. Elizabeth Lucas, Cecilia\u00a0O Alm, and Reynold Bailey. 2019. Understanding Human and Predictive Moderation of Online Science Discourse. In 2019 IEEE Western New York Image and Signal Processing Workshop (WNYISPW). IEEE, 1\u20135."},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174160"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v6i1.13338"},{"key":"e_1_3_2_1_48_1","unstructured":"Ninareh Mehrabi Fred Morstatter Nripsuta Saxena Kristina Lerman and Aram Galstyan. 2019. A survey on bias and fairness in machine learning. (2019). arXiv:1908.09635 Ninareh Mehrabi Fred Morstatter Nripsuta Saxena Kristina Lerman and Aram Galstyan. 2019. A survey on bias and fairness in machine learning. (2019). arXiv:1908.09635"},{"key":"e_1_3_2_1_49_1","volume-title":"Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems","author":"Zahra Ashktorab Narendra Nath Josh Andres","year":"2021","unstructured":"Josh Andres Zahra Ashktorab Narendra Nath Joshi Michael Desmond Aabhas Sharma Kristina Brimijoin Qian Pan Evelyn Duesterwald Casey\u00a0Dugan Michael\u00a0Muller , Christine\u00a0Wolf. 2021 . Designing Ground Truth and the Social Life of Labels . Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (2021). Josh Andres Zahra Ashktorab Narendra Nath Joshi Michael Desmond Aabhas Sharma Kristina Brimijoin Qian Pan Evelyn Duesterwald Casey\u00a0Dugan Michael\u00a0Muller, Christine\u00a0Wolf. 2021. Designing Ground Truth and the Social Life of Labels. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (2021)."},{"key":"e_1_3_2_1_50_1","volume-title":"Credbank: A large-scale social media corpus with associated credibility annotations.","author":"Mitra Tanushree","year":"2015","unstructured":"Tanushree Mitra and Eric Gilbert . 2015 . Credbank: A large-scale social media corpus with associated credibility annotations. Tanushree Mitra and Eric Gilbert. 2015. Credbank: A large-scale social media corpus with associated credibility annotations."},{"key":"e_1_3_2_1_51_1","unstructured":"Kayur Patel James Fogarty James\u00a0A Landay and Beverly\u00a0L Harrison. 2008. Examining Difficulties Software Developers Encounter in the Adoption of Statistical Machine Learning.. In AAAI. 1563\u20131566. Kayur Patel James Fogarty James\u00a0A Landay and Beverly\u00a0L Harrison. 2008. Examining Difficulties Software Developers Encounter in the Adoption of Statistical Machine Learning.. In AAAI. 1563\u20131566."},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00293"},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00971"},{"key":"e_1_3_2_1_54_1","volume-title":"Sample surveys: design, methods and applications","author":"Pfeffermann Danny","unstructured":"Danny Pfeffermann and Calyampudi\u00a0Radhakrishna Rao . 2009. Sample surveys: design, methods and applications . Elsevier . Danny Pfeffermann and Calyampudi\u00a0Radhakrishna Rao. 2009. Sample surveys: design, methods and applications. Elsevier."},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.14778\/3157794.3157797"},{"key":"e_1_3_2_1_56_1","unstructured":"Sarah\u00a0T. Roberts. 2020. Fewer Humans Are Moderating Facebook Content. That\u2019s Worrying.https:\/\/slate.com\/technology\/2020\/04\/coronavirus-facebook-content-moderation-automated.html Sarah\u00a0T. Roberts. 2020. Fewer Humans Are Moderating Facebook Content. That\u2019s Worrying.https:\/\/slate.com\/technology\/2020\/04\/coronavirus-facebook-content-moderation-automated.html"},{"key":"e_1_3_2_1_57_1","unstructured":"Bj\u00f6rn Ross Michael Rist Guillermo Carbonell Benjamin Cabrera Nils Kurowsky and Michael Wojatzki. 2017. Measuring the reliability of hate speech annotations: The case of the european refugee crisis. (2017). arXiv:1701.08118 Bj\u00f6rn Ross Michael Rist Guillermo Carbonell Benjamin Cabrera Nils Kurowsky and Michael Wojatzki. 2017. Measuring the reliability of hate speech annotations: The case of the european refugee crisis. (2017). arXiv:1701.08118"},{"key":"e_1_3_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/SNAMS.2018.8554954"},{"key":"e_1_3_2_1_59_1","unstructured":"Christian Sandvig Kevin Hamilton Karrie Karahalios and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry 22(2014). Christian Sandvig Kevin Hamilton Karrie Karahalios and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry 22(2014)."},{"key":"e_1_3_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274423"},{"key":"e_1_3_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"key":"e_1_3_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3137597.3137600"},{"key":"e_1_3_2_1_63_1","unstructured":"Harini Suresh and John\u00a0V Guttag. 2019. A framework for understanding unintended consequences of machine learning. (2019). arXiv:1901.10002 Harini Suresh and John\u00a0V Guttag. 2019. A framework for understanding unintended consequences of machine learning. (2019). arXiv:1901.10002"},{"key":"e_1_3_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/1497302.1497307"},{"key":"e_1_3_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.7717\/peerj.6762"},{"key":"e_1_3_2_1_66_1","doi-asserted-by":"crossref","unstructured":"Betty van Aken Julian Risch Ralf Krestel and Alexander L\u00f6ser. 2018. Challenges for toxic comment classification: An in-depth error analysis. (2018). arXiv:1809.07572 Betty van Aken Julian Risch Ralf Krestel and Alexander L\u00f6ser. 2018. Challenges for toxic comment classification: An in-depth error analysis. (2018). arXiv:1809.07572","DOI":"10.18653\/v1\/W18-5105"},{"key":"e_1_3_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-13739-6_25"},{"key":"e_1_3_2_1_68_1","unstructured":"Jing Wang and Xin Geng. [n.d.]. Classification with Label Distribution Learning. Jing Wang and Xin Geng. [n.d.]. Classification with Label Distribution Learning."},{"key":"e_1_3_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-5618"},{"key":"e_1_3_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/1294211.1294238"},{"key":"e_1_3_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3136755.3136792"},{"key":"e_1_3_2_1_72_1","unstructured":"Xinyi Zhou and Reza Zafarani. 2018. Fake news: A survey of research detection methods and opportunities. (2018). arXiv:1812.00315 Xinyi Zhou and Reza Zafarani. 2018. Fake news: A survey of research detection methods and opportunities. (2018). arXiv:1812.00315"}],"event":{"name":"CHI '21: CHI Conference on Human Factors in Computing Systems","location":"Yokohama Japan","acronym":"CHI '21","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411764.3445423","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3411764.3445423","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:28:35Z","timestamp":1750195715000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411764.3445423"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,6]]},"references-count":72,"alternative-id":["10.1145\/3411764.3445423","10.1145\/3411764"],"URL":"https:\/\/doi.org\/10.1145\/3411764.3445423","relation":{},"subject":[],"published":{"date-parts":[[2021,5,6]]},"assertion":[{"value":"2021-05-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}