{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T21:45:45Z","timestamp":1773870345901,"version":"3.50.1"},"reference-count":79,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2022,3,30]],"date-time":"2022-03-30T00:00:00Z","timestamp":1648598400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1544753"],"award-info":[{"award-number":["1544753"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2022,3,30]]},"abstract":"<jats:p>In this work we explore confidence elicitation methods for crowdsourcing \"soft\" labels, e.g., probability estimates, to reduce the annotation costs for domains with ambiguous data. Machine learning research has shown that such \"soft\" labels are more informative and can reduce the data requirements when training supervised machine learning models. By reducing the number of required labels, we can reduce the costs of slow annotation processes such as audio annotation. In our experiments we evaluated three confidence elicitation methods: 1) \"No Confidence\" elicitation, 2) \"Simple Confidence\" elicitation, and 3) \"Betting\" mechanism for confidence elicitation, at both individual (i.e., per participant) and aggregate (i.e., crowd) levels. In addition, we evaluated the interaction between confidence elicitation methods, annotation types (binary, probability, and z-score derived probability), and \"soft\" versus \"hard\" (i.e., binarized) aggregate labels. Our results show that both confidence elicitation mechanisms result in higher annotation quality than the \"No Confidence\" mechanism for binary annotations at both participant and recording levels. In addition, when aggregating labels at the recording level, results indicate that we can achieve comparable results to those with 10-participant aggregate annotations using fewer annotators if we aggregate \"soft\" labels instead of \"hard\" labels. These results suggest that for binary audio annotation using a confidence elicitation mechanism and aggregating continuous labels we can obtain higher annotation quality, more informative labels, with quality differences more pronounced with fewer participants. Finally, we propose a way of integrating these confidence elicitation methods into a two-stage, multi-label annotation pipeline.<\/jats:p>","DOI":"10.1145\/3512935","type":"journal-article","created":{"date-parts":[[2022,4,7]],"date-time":"2022-04-07T16:54:59Z","timestamp":1649350499000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Eliciting Confidence for Improving Crowdsourced Audio Annotations"],"prefix":"10.1145","volume":"6","author":[{"given":"Ana Elisa","family":"M\u00e9ndez M\u00e9ndez","sequence":"first","affiliation":[{"name":"New York University, New York, NY, USA"}]},{"given":"Mark","family":"Cartwright","sequence":"additional","affiliation":[{"name":"New Jersey Institute of Technology &amp; New York University, Newark, NJ, USA"}]},{"given":"Juan Pablo","family":"Bello","sequence":"additional","affiliation":[{"name":"New York University, New York, NY, USA"}]},{"given":"Oded","family":"Nov","sequence":"additional","affiliation":[{"name":"New York University, New York, NY, USA"}]}],"member":"320","published-online":{"date-parts":[[2022,4,7]]},"reference":[{"key":"e_1_2_1_1_1","first-page":"2","article-title":"Active Learning and Crowd-Sourcing for Machine Translation","volume":"1","author":"Ambati Vamshi","year":"2010","unstructured":"Vamshi Ambati , Stephan Vogel , and Jaime G Carbonell . 2010 . Active Learning and Crowd-Sourcing for Machine Translation .. In LREC , Vol. 1. 2 . Vamshi Ambati, Stephan Vogel, and Jaime G Carbonell. 2010. Active Learning and Crowd-Sourcing for Machine Translation.. In LREC, Vol. 1. 2.","journal-title":"LREC"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v36i1.2564"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1177\/0013164409332231"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2017.03.012"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2017\/195"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/FG.2018.00033"},{"key":"e_1_2_1_7_1","volume-title":"How to Grade a Test without Knowing the Answers: A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing (ICML'12)","author":"Bachrach Yoram","unstructured":"Yoram Bachrach , Tom Minka , John Guiver , and Thore Graepel . 2012. How to Grade a Test without Knowing the Answers: A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing (ICML'12) . Omnipress , Madison, WI, USA , 819--826. Yoram Bachrach, Tom Minka, John Guiver, and Thore Graepel. 2012. How to Grade a Test without Knowing the Answers: A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing (ICML'12). Omnipress, Madison, WI, USA, 819--826."},{"key":"e_1_2_1_8_1","volume-title":"Oded Nov, R. Luke DuBois, Anish Arora, Justin Salamon, Charles Mydlarz, and Harish Doraiswamy.","author":"Bello Juan Pablo","year":"2018","unstructured":"Juan Pablo Bello , Cl\u00e1 udio T. Silva , Oded Nov, R. Luke DuBois, Anish Arora, Justin Salamon, Charles Mydlarz, and Harish Doraiswamy. 2018 . SONYC : A System for the Monitoring, Analysis and Mitigation of Urban Noise Pollution. CoRR , Vol. abs\/ 1805 .00889 (2018). arxiv: 1805.00889 http:\/\/arxiv.org\/abs\/1805.00889 Juan Pablo Bello, Cl\u00e1 udio T. Silva, Oded Nov, R. Luke DuBois, Anish Arora, Justin Salamon, Charles Mydlarz, and Harish Doraiswamy. 2018. SONYC: A System for the Monitoring, Analysis and Mitigation of Urban Noise Pollution. CoRR, Vol. abs\/1805.00889 (2018). arxiv: 1805.00889 http:\/\/arxiv.org\/abs\/1805.00889"},{"key":"e_1_2_1_9_1","volume-title":"Auditory scene analysis: The perceptual organization of sound","author":"Bregman Albert S","unstructured":"Albert S Bregman . 1994. Auditory scene analysis: The perceptual organization of sound . MIT press . Albert S Bregman. 1994. Auditory scene analysis: The perceptual organization of sound .MIT press."},{"key":"e_1_2_1_10_1","volume-title":"Yu Wang, Ho-Hsiang Wu, Vincent Lostanlen, Magdalena Fuentes, Graham Dove, Charlie Mydlarz, Justin Salamon, et al.","author":"Cartwright Mark","year":"2020","unstructured":"Mark Cartwright , Jason Cramer , Ana Elisa Mendez Mendez , Yu Wang, Ho-Hsiang Wu, Vincent Lostanlen, Magdalena Fuentes, Graham Dove, Charlie Mydlarz, Justin Salamon, et al. 2020 . SONYC-UST-V2: An Urban Sound Tagging Dataset with Spatiotemporal Context . arXiv preprint arXiv:2009.05188 (2020). Mark Cartwright, Jason Cramer, Ana Elisa Mendez Mendez, Yu Wang, Ho-Hsiang Wu, Vincent Lostanlen, Magdalena Fuentes, Graham Dove, Charlie Mydlarz, Justin Salamon, et al. 2020. SONYC-UST-V2: An Urban Sound Tagging Dataset with Spatiotemporal Context. arXiv preprint arXiv:2009.05188 (2020)."},{"key":"e_1_2_1_11_1","volume-title":"Crowdsourcing Multi-Label Audio Annotation Tasks with Citizen Scientists (CHI '19)","author":"Cartwright Mark","year":"2019","unstructured":"Mark Cartwright , Graham Dove , Ana Elisa M\u00e9ndez M\u00e9ndez , Juan P. Bello , and Oded Nov . 2019 . Crowdsourcing Multi-Label Audio Annotation Tasks with Citizen Scientists (CHI '19) . Association for Computing Machinery, New York, NY, USA, 1--11. https:\/\/doi.org\/10.1145\/3290605.3300522 10.1145\/3290605.3300522 Mark Cartwright, Graham Dove, Ana Elisa M\u00e9ndez M\u00e9ndez, Juan P. Bello, and Oded Nov. 2019. Crowdsourcing Multi-Label Audio Annotation Tasks with Citizen Scientists (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--11. https:\/\/doi.org\/10.1145\/3290605.3300522"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3134664"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026044"},{"key":"e_1_2_1_14_1","volume-title":"Weld","author":"Chen Quanze","year":"2019","unstructured":"Quanze Chen , Jonathan Bragg , Lydia B. Chilton , and Dan S . Weld . 2019 . Cicero : Multi-Turn, Contextual Argumentation for Accurate Crowdsourcing (CHI '19). Association for Computing Machinery , New York, NY, USA, 1--14. https:\/\/doi.org\/10.1145\/3290605.3300761 10.1145\/3290605.3300761 Quanze Chen, Jonathan Bragg, Lydia B. Chilton, and Dan S. Weld. 2019. Cicero: Multi-Turn, Contextual Argumentation for Accurate Crowdsourcing (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--14. https:\/\/doi.org\/10.1145\/3290605.3300761"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466265"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300460"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359164"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISDA.2011.6121700"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488414"},{"key":"e_1_2_1_21_1","volume-title":"Foresight: Its Logical Laws, Its Subjective Sources","author":"de Finetti Bruno","year":"1992","unstructured":"Bruno de Finetti . 1992 . Foresight: Its Logical Laws, Its Subjective Sources . Springer New York , New York, NY , 134--174. https:\/\/doi.org\/10.1007\/978--1--4612-0919--5_10 10.1007\/978--1--4612-0919--5_10 Bruno de Finetti. 1992. Foresight: Its Logical Laws, Its Subjective Sources. Springer New York, New York, NY, 134--174. https:\/\/doi.org\/10.1007\/978--1--4612-0919--5_10"},{"key":"e_1_2_1_22_1","first-page":"2","article-title":"Resolving ambiguity in sentiment classification: The role of dependency features","volume":"8","author":"Deng Shuyuan","year":"2017","unstructured":"Shuyuan Deng , Atish P Sinha , and Huimin Zhao . 2017 . Resolving ambiguity in sentiment classification: The role of dependency features . ACM Transactions on Management Information Systems (TMIS) , Vol. 8 , 2 -- 3 (2017), 1--13. Shuyuan Deng, Atish P Sinha, and Huimin Zhao. 2017. Resolving ambiguity in sentiment classification: The role of dependency features. ACM Transactions on Management Information Systems (TMIS), Vol. 8, 2--3 (2017), 1--13.","journal-title":"ACM Transactions on Management Information Systems (TMIS)"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2017.7953161"},{"key":"e_1_2_1_24_1","volume-title":"The Semantic Web. Latest Advances and New Domains, Fabien Gandon, Marta Sabou, Harald Sack, Claudia D'Amato, Philippe Cudr\u00e9 -Mauroux","author":"Dumitrache Anca","unstructured":"Anca Dumitrache . 2015. Crowdsourcing Disagreement for Collecting Semantic Annotation . In The Semantic Web. Latest Advances and New Domains, Fabien Gandon, Marta Sabou, Harald Sack, Claudia D'Amato, Philippe Cudr\u00e9 -Mauroux , and Antoine Zimmermann (Eds.). Springer International Publishing , Cham , 701--710. Anca Dumitrache. 2015. Crowdsourcing Disagreement for Collecting Semantic Annotation. In The Semantic Web. Latest Advances and New Domains, Fabien Gandon, Marta Sabou, Harald Sack, Claudia D'Amato, Philippe Cudr\u00e9 -Mauroux, and Antoine Zimmermann (Eds.). Springer International Publishing, Cham, 701--710."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v6i1.13330"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3152889"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/IMCSIT.2008.4747229"},{"key":"e_1_2_1_28_1","volume-title":"FSD50k: an open dataset of human-labeled sound events. arXiv preprint arXiv:2010.00475","author":"Fonseca Eduardo","year":"2020","unstructured":"Eduardo Fonseca , Xavier Favory , Jordi Pons , Frederic Font , and Xavier Serra . 2020. FSD50k: an open dataset of human-labeled sound events. arXiv preprint arXiv:2010.00475 ( 2020 ). Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. 2020. FSD50k: an open dataset of human-labeled sound events. arXiv preprint arXiv:2010.00475 (2020)."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2017.2689998"},{"key":"e_1_2_1_30_1","volume-title":"International Conference on Machine Learning. 603--611","author":"Gao Chao","year":"2016","unstructured":"Chao Gao , Yu Lu , and Dengyong Zhou . 2016 . Exact exponent in optimal rates for crowdsourcing . In International Conference on Machine Learning. 603--611 . Chao Gao, Yu Lu, and Dengyong Zhou. 2016. Exact exponent in optimal rates for crowdsourcing. In International Conference on Machine Learning. 603--611."},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 776--780","author":"Gemmeke J. F.","unstructured":"J. F. Gemmeke , D. P. W. Ellis , D. Freedman , A. Jansen , W. Lawrence , R. C. Moore , M. Plakal , and M. Ritter . 2017. Audio Set: An Ontology and Human-Labeled Dataset for Audio Events . In Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 776--780 . J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter. 2017. Audio Set: An Ontology and Human-Labeled Dataset for Audio Events. In Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 776--780."},{"key":"#cr-split#-e_1_2_1_32_1.1","doi-asserted-by":"crossref","unstructured":"Geoffrey Hinton Oriol Vinyals and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. (2015) 1--9. https:\/\/doi.org\/10.1063\/1.4931082 arxiv: 1503.02531 10.1063\/1.4931082","DOI":"10.1063\/1.4931082"},{"key":"#cr-split#-e_1_2_1_32_1.2","doi-asserted-by":"crossref","unstructured":"Geoffrey Hinton Oriol Vinyals and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. (2015) 1--9. https:\/\/doi.org\/10.1063\/1.4931082 arxiv: 1503.02531","DOI":"10.1063\/1.4931082"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173640"},{"key":"e_1_2_1_34_1","volume-title":"Irina Smoke, Sangwha Sien, Janet Ng, Mark Zachry, and Juho Kim.","author":"Hong Sungsoo","year":"2019","unstructured":"Sungsoo Hong , Minhyang Suh , Tae Soo Kim , Irina Smoke, Sangwha Sien, Janet Ng, Mark Zachry, and Juho Kim. 2019 . Design for Collaborative Information-Seeking: Understanding User Challenges and Deploying Collaborative Dynamic Queries. Proceedings of the ACM on Human-Computer Interaction, Vol. 3 , CSCW ( 2019), 1--24. Sungsoo Hong, Minhyang Suh, Tae Soo Kim, Irina Smoke, Sangwha Sien, Janet Ng, Mark Zachry, and Juho Kim. 2019. Design for Collaborative Information-Seeking: Understanding User Challenges and Deploying Collaborative Dynamic Queries. Proceedings of the ACM on Human-Computer Interaction, Vol. 3, CSCW (2019), 1--24."},{"key":"e_1_2_1_35_1","volume-title":"The online laboratory: Conducting experiments in a real labor market. Experimental economics","author":"Horton John J","year":"2011","unstructured":"John J Horton , David G Rand , and Richard J Zeckhauser . 2011. The online laboratory: Conducting experiments in a real labor market. Experimental economics , Vol. 14 , 3 ( 2011 ), 399--425. John J Horton, David G Rand, and Richard J Zeckhauser. 2011. The online laboratory: Conducting experiments in a real labor market. Experimental economics, Vol. 14, 3 (2011), 399--425."},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the 19th International Society for Music Information Retrieval Conference. ISMIR","author":"Humphrey Eric","year":"2018","unstructured":"Eric Humphrey , Simon Durand , and Brian McFee . 2018 . OpenMIC-2018: An Open Data-set for Multiple Instrument Recognition . In Proceedings of the 19th International Society for Music Information Retrieval Conference. ISMIR , Paris, France, 438--444. https:\/\/doi.org\/10.5281\/zenodo.1492445 10.5281\/zenodo.1492445 Eric Humphrey, Simon Durand, and Brian McFee. 2018. OpenMIC-2018: An Open Data-set for Multiple Instrument Recognition. In Proceedings of the 19th International Society for Music Information Retrieval Conference. ISMIR, Paris, France, 438--444. https:\/\/doi.org\/10.5281\/zenodo.1492445"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-2099"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics","author":"Jiang Youxuan","year":"1865","unstructured":"Youxuan Jiang , Jonathan K. Kummerfeld , and Walter S. Lasecki . 2017. Understanding Task Design Trade-offs in Crowdsourced Paraphrase Collection . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics , Vancouver, Canada, 103--109. https:\/\/doi.org\/10. 1865 3\/v1\/P17--2017 10.18653\/v1 Youxuan Jiang, Jonathan K. Kummerfeld, and Walter S. Lasecki. 2017. Understanding Task Design Trade-offs in Crowdsourced Paraphrase Collection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Vancouver, Canada, 103--109. https:\/\/doi.org\/10.18653\/v1\/P17--2017"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 556--562","author":"Jurgens David","year":"2013","unstructured":"David Jurgens . 2013 . Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels . In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 556--562 . David Jurgens. 2013. Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 556--562."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_2_1_41_1","volume-title":"International Conference on Machine Learning. 2708--2717","author":"Pranjal Awasthi Kleindessner","year":"2018","unstructured":"Matth\"aus Kleindessner and Pranjal Awasthi . 2018 . Crowdsourcing with arbitrary adversaries . In International Conference on Machine Learning. 2708--2717 . Matth\"aus Kleindessner and Pranjal Awasthi. 2018. Crowdsourcing with arbitrary adversaries. In International Conference on Machine Learning. 2708--2717."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.5555\/2049960"},{"key":"e_1_2_1_43_1","volume-title":"Human Computation. Morgan & Claypool","author":"Law Edith","unstructured":"Edith Law and Luis Von Ahn . 2011. Human Computation. Morgan & Claypool , San Rafael, CA . Edith Law and Luis Von Ahn. 2011. Human Computation. Morgan & Claypool, San Rafael, CA."},{"key":"e_1_2_1_44_1","volume-title":"Science Advances","volume":"4","author":"Lebreton Ma\u00eb","year":"2018","unstructured":"Ma\u00eb l Lebreton , Shari Langdon , Matthijs J. Slieker , Jip S. Nooitgedacht , Anna E. Goudriaan , Damiaan Denys , Ruth J. van Holst , and Judy Luigjes . 2018 . Two Sides of the Same Coin: Monetary Incentives Concurrently Improve and Bias Confidence Judgments . Science Advances , Vol. 4 , 5 (2018). https:\/\/doi.org\/10.1126\/sciadv.aaq0668 10.1126\/sciadv.aaq0668 Ma\u00eb l Lebreton, Shari Langdon, Matthijs J. Slieker, Jip S. Nooitgedacht, Anna E. Goudriaan, Damiaan Denys, Ruth J. van Holst, and Judy Luigjes. 2018. Two Sides of the Same Coin: Monetary Incentives Concurrently Improve and Bias Confidence Judgments. Science Advances, Vol. 4, 5 (2018). https:\/\/doi.org\/10.1126\/sciadv.aaq0668"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274373"},{"key":"e_1_2_1_46_1","unstructured":"Qiang Liu Jian Peng and Alexander T Ihler. 2012. Variational inference for crowdsourcing. In Advances in neural information processing systems. 692--700.  Qiang Liu Jian Peng and Alexander T Ihler. 2012. Variational inference for crowdsourcing. In Advances in neural information processing systems. 692--700."},{"key":"e_1_2_1_47_1","volume-title":"International Conference on Machine Learning. PMLR, 3335--3344","author":"Ma Yao","year":"2018","unstructured":"Yao Ma , Alexander Olshevsky , Csaba Szepesvari , and Venkatesh Saligrama . 2018 . Gradient descent for sparse rank-one matrix completion for crowd-sourced aggregation of sparsely interacting workers . In International Conference on Machine Learning. PMLR, 3335--3344 . Yao Ma, Alexander Olshevsky, Csaba Szepesvari, and Venkatesh Saligrama. 2018. Gradient descent for sparse rank-one matrix completion for crowd-sourced aggregation of sparsely interacting workers. In International Conference on Machine Learning. PMLR, 3335--3344."},{"key":"e_1_2_1_48_1","volume-title":"Efficiency of active learning for the allocation of workers on crowdsourced classification tasks. arXiv preprint arXiv:1610.06106","author":"Manino Edoardo","year":"2016","unstructured":"Edoardo Manino , Long Tran-Thanh , and Nicholas R Jennings . 2016. Efficiency of active learning for the allocation of workers on crowdsourced classification tasks. arXiv preprint arXiv:1610.06106 ( 2016 ). Edoardo Manino, Long Tran-Thanh, and Nicholas R Jennings. 2016. Efficiency of active learning for the allocation of workers on crowdsourced classification tasks. arXiv preprint arXiv:1610.06106 (2016)."},{"key":"e_1_2_1_49_1","volume-title":"Multiple-Choice Testing Using Immediate Feedback-Assessment Technique (IF AT\u00ae) Forms: Second-Chance Guessing vs. Second-Chance Learning?","author":"Merrel Jeremy D","year":"2015","unstructured":"Jeremy D Merrel , Pier F Cirillo , Pauline M Schwartz , and Jeffrey Webb . 2015. Multiple-Choice Testing Using Immediate Feedback-Assessment Technique (IF AT\u00ae) Forms: Second-Chance Guessing vs. Second-Chance Learning? ( 2015 ). Jeremy D Merrel, Pier F Cirillo, Pauline M Schwartz, and Jeffrey Webb. 2015. Multiple-Choice Testing Using Immediate Feedback-Assessment Technique (IF AT\u00ae) Forms: Second-Chance Guessing vs. Second-Chance Learning? (2015)."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACII.2009.5349500"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2013.09.011"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/1743384.1743478"},{"key":"e_1_2_1_53_1","volume-title":"The Meaning of Meaning: A Study of the Influence of Language upon Thought and of the Science of Symbolism","author":"Ogden Charles Kay","unstructured":"Charles Kay Ogden and Ivor Armstrong Richards . 1923. The Meaning of Meaning: A Study of the Influence of Language upon Thought and of the Science of Symbolism . Vol. 29 . K. Paul, Trench, Trubner & Company , Limited. Charles Kay Ogden and Ivor Armstrong Richards. 1923. The Meaning of Meaning: A Study of the Influence of Language upon Thought and of the Science of Symbolism. Vol. 29. K. Paul, Trench, Trubner & Company, Limited."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187980.2188138"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1038\/nn1840"},{"key":"e_1_2_1_56_1","article-title":"Learning from crowds","volume":"11","author":"Raykar Vikas C","year":"2010","unstructured":"Vikas C Raykar , Shipeng Yu , Linda H Zhao , Gerardo Hermosillo Valadez , Charles Florin , Luca Bogoni , and Linda Moy . 2010 . Learning from crowds . Journal of Machine Learning Research , Vol. 11 , 4 (2010). Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. Learning from crowds. Journal of Machine Learning Research, Vol. 11, 4 (2010).","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_1_57_1","volume-title":"Multimodal speech emotion recognition and ambiguity resolution. arXiv preprint arXiv:1904.06022","author":"Sahu Gaurav","year":"2019","unstructured":"Gaurav Sahu . 2019. Multimodal speech emotion recognition and ambiguity resolution. arXiv preprint arXiv:1904.06022 ( 2019 ). Gaurav Sahu. 2019. Multimodal speech emotion recognition and ambiguity resolution. arXiv preprint arXiv:1904.06022 (2019)."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0166866"},{"key":"e_1_2_1_59_1","volume-title":"A Dataset and Taxonomy for Urban Sound Research (MM '14)","author":"Salamon Justin","unstructured":"Justin Salamon , Christopher Jacoby , and Juan Pablo Bello . 2014. A Dataset and Taxonomy for Urban Sound Research (MM '14) . ACM , New York, NY, USA , 1041--1044. https:\/\/doi.org\/10.1145\/2647868.2655045 10.1145\/2647868.2655045 Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. 2014. A Dataset and Taxonomy for Urban Sound Research (MM '14). ACM, New York, NY, USA, 1041--1044. https:\/\/doi.org\/10.1145\/2647868.2655045"},{"key":"e_1_2_1_60_1","volume-title":"2017 IEEE Workshop on. IEEE, 344--348","author":"Salamon Justin","year":"2017","unstructured":"Justin Salamon , Duncan MacConnell , Mark Cartwright , Peter Li , and Juan Pablo Bello . 2017 . Scaper: A library for soundscape synthesis and augmentation. In Applications of Signal Processing to Audio and Acoustics (WASPAA) , 2017 IEEE Workshop on. IEEE, 344--348 . Justin Salamon, Duncan MacConnell, Mark Cartwright, Peter Li, and Juan Pablo Bello. 2017. Scaper: A library for soundscape synthesis and augmentation. In Applications of Signal Processing to Audio and Acoustics (WASPAA), 2017 IEEE Workshop on. IEEE, 344--348."},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.concog.2009.12.013"},{"key":"e_1_2_1_62_1","volume-title":"Wainwright","author":"Shah Nihar B.","year":"2016","unstructured":"Nihar B. Shah , Sivaraman Balakrishnan , and Martin J . Wainwright . 2016 . A Permutation-based Model for Crowd Labeling: Optimal Estimation and Robustness. CoRR , Vol. abs\/ 1606 .09632 (2016). arxiv: 1606.09632 http:\/\/arxiv.org\/abs\/1606.09632 Nihar B. Shah, Sivaraman Balakrishnan, and Martin J. Wainwright. 2016. A Permutation-based Model for Crowd Labeling: Optimal Estimation and Robustness. CoRR, Vol. abs\/1606.09632 (2016). arxiv: 1606.09632 http:\/\/arxiv.org\/abs\/1606.09632"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.5555\/1613715.1613751"},{"key":"e_1_2_1_65_1","article-title":"FourEyes: Leveraging Tool Diversity as a Means to Improve Aggregate Accuracy in Crowdsourcing","volume":"10","author":"Song Jean Y.","year":"2019","unstructured":"Jean Y. Song , Raymond Fok , Juho Kim , and Walter S. Lasecki . 2019 . FourEyes: Leveraging Tool Diversity as a Means to Improve Aggregate Accuracy in Crowdsourcing . ACM Trans. Interact. Intell. Syst. , Vol. 10 , 1, Article 3 (Aug. 2019), 30 pages. https:\/\/doi.org\/10.1145\/3237188 10.1145\/3237188 Jean Y. Song, Raymond Fok, Juho Kim, and Walter S. Lasecki. 2019. FourEyes: Leveraging Tool Diversity as a Means to Improve Aggregate Accuracy in Crowdsourcing. ACM Trans. Interact. Intell. Syst., Vol. 10, 1, Article 3 (Aug. 2019), 30 pages. https:\/\/doi.org\/10.1145\/3237188","journal-title":"ACM Trans. Interact. Intell. Syst."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13415-011-0025-2"},{"key":"e_1_2_1_67_1","unstructured":"Tian Tian and Jun Zhu. 2015. Max-margin majority voting for learning from crowds. In Advances in neural information processing systems. 1621--1629.  Tian Tian and Jun Zhu. 2015. Max-margin majority voting for learning from crowds. In Advances in neural information processing systems. 1621--1629."},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567989"},{"key":"e_1_2_1_69_1","volume-title":"Keren","author":"Wagenaar Willem A.","year":"1986","unstructured":"Willem A. Wagenaar and Gideon B . Keren . 1986 . Does The Expert Know? The Reliability of Predictions and Confidence Ratings of Experts. In Intelligent Decision Support in Process Environments, Erik Hollnagel, Giuseppe Mancini, and David D. Woods (Eds.). Springer Berlin Heidelberg , Berlin, Heidelberg, 87--103. Willem A. Wagenaar and Gideon B. Keren. 1986. Does The Expert Know? The Reliability of Predictions and Confidence Ratings of Experts. In Intelligent Decision Support in Process Environments, Erik Hollnagel, Giuseppe Mancini, and David D. Woods (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 87--103."},{"key":"e_1_2_1_70_1","volume-title":"Effects of Feedback Types and Opportunities to Change Answers on Achievement and Ability to Solve Physics Problems. Research in Science Education","author":"Wancham Kittitas","year":"2020","unstructured":"Kittitas Wancham and Kamonwan Tangdhanakanond . 2020. Effects of Feedback Types and Opportunities to Change Answers on Achievement and Ability to Solve Physics Problems. Research in Science Education ( 2020 ), 1--18. Kittitas Wancham and Kamonwan Tangdhanakanond. 2020. Effects of Feedback Types and Opportunities to Change Answers on Achievement and Ability to Solve Physics Problems. Research in Science Education (2020), 1--18."},{"key":"e_1_2_1_71_1","volume-title":"Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 5275--5279","author":"Watanabe Shinji","unstructured":"Shinji Watanabe , Takaaki Hori , Jonathan Le Roux , and John R. Hershey . 2017. Student-Teacher Network Learning with Enhanced Features . In Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 5275--5279 . Shinji Watanabe, Takaaki Hori, Jonathan Le Roux, and John R. Hershey. 2017. Student-Teacher Network Learning with Enhanced Features. In Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 5275--5279."},{"key":"e_1_2_1_72_1","unstructured":"Peter Welinder Steve Branson Pietro Perona and Serge J Belongie. 2010. The multidimensional wisdom of crowds. In Advances in neural information processing systems. 2424--2432.  Peter Welinder Steve Branson Pietro Perona and Serge J Belongie. 2010. The multidimensional wisdom of crowds. In Advances in neural information processing systems. 2424--2432."},{"key":"e_1_2_1_73_1","unstructured":"Jacob Whitehill Ting-fan Wu Jacob Bergsma Javier R Movellan and Paul L Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in neural information processing systems. 2035--2043.  Jacob Whitehill Ting-fan Wu Jacob Bergsma Javier R Movellan and Paul L Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in neural information processing systems. 2035--2043."},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1978963"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3057270"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASL.2007.911513"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3136755.3136792"},{"key":"e_1_2_1_78_1","unstructured":"Dengyong Zhou Sumit Basu Yi Mao and John C Platt. 2012. Learning from the wisdom of crowds by minimax entropy. In Advances in neural information processing systems. 2195--2203.  Dengyong Zhou Sumit Basu Yi Mao and John C Platt. 2012. Learning from the wisdom of crowds by minimax entropy. In Advances in neural information processing systems. 2195--2203."},{"key":"e_1_2_1_79_1","volume-title":"International conference on machine learning. 262--270","author":"Zhou Dengyong","year":"2014","unstructured":"Dengyong Zhou , Qiang Liu , John Platt , and Christopher Meek . 2014 . Aggregating ordinal labels from crowds by minimax conditional entropy . In International conference on machine learning. 262--270 . Dengyong Zhou, Qiang Liu, John Platt, and Christopher Meek. 2014. Aggregating ordinal labels from crowds by minimax conditional entropy. In International conference on machine learning. 262--270."}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3512935","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3512935","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3512935","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:31:18Z","timestamp":1750188678000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3512935"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,30]]},"references-count":79,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2022,3,30]]}},"alternative-id":["10.1145\/3512935"],"URL":"https:\/\/doi.org\/10.1145\/3512935","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,30]]},"assertion":[{"value":"2022-04-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}