{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,29]],"date-time":"2026-01-29T22:22:52Z","timestamp":1769725372697,"version":"3.49.0"},"publisher-location":"New York, NY, USA","reference-count":80,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,7,21]],"date-time":"2021-07-21T00:00:00Z","timestamp":1626825600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,7,21]]},"DOI":"10.1145\/3461702.3462516","type":"proceedings-article","created":{"date-parts":[[2021,7,31]],"date-time":"2021-07-31T01:21:38Z","timestamp":1627694498000},"page":"154-165","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":31,"title":["Towards Unbiased and Accurate Deferral to Multiple Experts"],"prefix":"10.1145","author":[{"given":"Vijay","family":"Keswani","sequence":"first","affiliation":[{"name":"Yale University, New Haven, CT, USA"}]},{"given":"Matthew","family":"Lease","sequence":"additional","affiliation":[{"name":"University of Texas at Austin, Austin, TX, USA"}]},{"given":"Krishnaram","family":"Kenthapadi","sequence":"additional","affiliation":[{"name":"Amazon AWS AI, East Palo Alto, CA, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,7,30]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3148011.3148021"},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-04468-7_3"},{"key":"e_1_3_2_2_3_1","volume-title":"Optimizing AI for Teamwork. arXiv preprint arXiv:2004.13102","author":"Bansal Gagan","year":"2020","unstructured":"Gagan Bansal , Besmira Nushi , Ece Kamar , Eric Horvitz , and Daniel S Weld . 2020. Optimizing AI for Teamwork. arXiv preprint arXiv:2004.13102 ( 2020 ). Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, and Daniel S Weld. 2020. Optimizing AI for Teamwork. arXiv preprint arXiv:2004.13102 (2020)."},{"key":"e_1_3_2_2_4_1","unstructured":"Solon Barocas Moritz Hardt and Arvind Narayanan. 2019. Fairness and Machine Learning. fairmlbook.org. http:\/\/www.fairmlbook.org  Solon Barocas Moritz Hardt and Arvind Narayanan. 2019. Fairness and Machine Learning. fairmlbook.org. http:\/\/www.fairmlbook.org"},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-4408"},{"key":"e_1_3_2_2_6_1","volume-title":"Fair pipelines. arXiv preprint arXiv:1707.00391","author":"Bower Amanda","year":"2017","unstructured":"Amanda Bower , Sarah N Kitchen , Laura Niss , Martin J Strauss , Alexander Vargas , and Suresh Venkatasubramanian . 2017. Fair pipelines. arXiv preprint arXiv:1707.00391 ( 2017 ). Amanda Bower, Sarah N Kitchen, Laura Niss, Martin J Strauss, Alexander Vargas, and Suresh Venkatasubramanian. 2017. Fair pipelines. arXiv preprint arXiv:1707.00391 (2017)."},{"key":"e_1_3_2_2_7_1","volume-title":"Convex Optimization","author":"Boyd Stephen","unstructured":"Stephen Boyd , Stephen P Boyd , and Lieven Vandenberghe . 2004. Convex Optimization . Cambridge University Press . Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. 2004. Convex Optimization .Cambridge University Press."},{"key":"e_1_3_2_2_8_1","volume-title":"Conference on Fairness, Accountability and Transparency. 77--91","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru . 2018 . Gender shades: Intersectional Accuracy Disparities in Commercial Gender Classification . In Conference on Fairness, Accountability and Transparency. 77--91 . Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Conference on Fairness, Accountability and Transparency. 77--91."},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287586"},{"key":"e_1_3_2_2_10_1","unstructured":"Lingjiao Chen Matei Zaharia and James Zou. 2020. FrugalML: How to use ML Prediction APIs more accurately and cheaply. In NeurIPS.  Lingjiao Chen Matei Zaharia and James Zou. 2020. FrugalML: How to use ML Prediction APIs more accurately and cheaply. In NeurIPS."},{"key":"e_1_3_2_2_11_1","volume-title":"Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data","author":"Chouldechova Alexandra","year":"2017","unstructured":"Alexandra Chouldechova . 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data , Vol. 5 , 2 ( 2017 ), 153--163. Alexandra Chouldechova. 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, Vol. 5, 2 (2017), 153--163."},{"key":"e_1_3_2_2_12_1","volume-title":"Conference on Fairness, Accountability and Transparency. 134--148","author":"Chouldechova Alexandra","year":"2018","unstructured":"Alexandra Chouldechova , Diana Benavides-Prado , Oleksandr Fialko , and Rhema Vaithianathan . 2018 . A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions . In Conference on Fairness, Accountability and Transparency. 134--148 . Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Conference on Fairness, Accountability and Transparency. 134--148."},{"key":"e_1_3_2_2_13_1","volume-title":"International conference on machine learning. PMLR, 1059--1067","author":"Cortes Corinna","year":"2018","unstructured":"Corinna Cortes , Giulia DeSalvo , Claudio Gentile , Mehryar Mohri , and Scott Yang . 2018 . Online learning with abstention . In International conference on machine learning. PMLR, 1059--1067 . Corinna Cortes, Giulia DeSalvo, Claudio Gentile, Mehryar Mohri, and Scott Yang. 2018. Online learning with abstention. In International conference on machine learning. PMLR, 1059--1067."},{"key":"e_1_3_2_2_14_1","first-page":"1660","article-title":"Boosting with abstention","volume":"29","author":"Cortes Corinna","year":"2016","unstructured":"Corinna Cortes , Giulia DeSalvo , and Mehryar Mohri . 2016 a. Boosting with abstention . Advances in Neural Information Processing Systems , Vol. 29 (2016), 1660 -- 1668 . Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. 2016a. Boosting with abstention. Advances in Neural Information Processing Systems, Vol. 29 (2016), 1660--1668.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46379-7_5"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.2514\/6.2004-6313"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"crossref","unstructured":"Thomas Davidson Dana Warmsley Michael Macy and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Eleventh international aaai conference on web and social media.  Thomas Davidson Dana Warmsley Michael Macy and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Eleventh international aaai conference on web and social media.","DOI":"10.1609\/icwsm.v11i1.14955"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i03.5645"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376638"},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3461702.3462523"},{"key":"e_1_3_2_2_21_1","article-title":"Adaptive subgradient methods for online learning and stochastic optimization","volume":"12","author":"Duchi John","year":"2011","unstructured":"John Duchi , Elad Hazan , and Yoram Singer . 2011 . Adaptive subgradient methods for online learning and stochastic optimization . Journal of machine learning research , Vol. 12 , 7 (2011). John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, Vol. 12, 7 (2011).","journal-title":"Journal of machine learning research"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1111\/1745-9133.12270"},{"key":"e_1_3_2_2_23_1","volume-title":"Fairness Under Composition. In Innovations in Theoretical Computer Science Conference (ITCS).","author":"Dwork Cynthia","year":"2019","unstructured":"Cynthia Dwork and Christina Ilvento . 2019 . Fairness Under Composition. In Innovations in Theoretical Computer Science Conference (ITCS). Cynthia Dwork and Christina Ilvento. 2019. Fairness Under Composition. In Innovations in Theoretical Computer Science Conference (ITCS)."},{"key":"e_1_3_2_2_24_1","first-page":"18","article-title":"Slave to the algorithm: Why a right to an explanation is probably not the remedy you are looking for","volume":"16","author":"Edwards Lilian","year":"2017","unstructured":"Lilian Edwards and Michael Veale . 2017 . Slave to the algorithm: Why a right to an explanation is probably not the remedy you are looking for . Duke L. & Tech. Rev. , Vol. 16 (2017), 18 . Lilian Edwards and Michael Veale. 2017. Slave to the algorithm: Why a right to an explanation is probably not the remedy you are looking for. Duke L. & Tech. Rev., Vol. 16 (2017), 18.","journal-title":"Duke L. & Tech. Rev."},{"key":"e_1_3_2_2_25_1","article-title":"On the Foundations of Noise-free Selective Classification","volume":"11","author":"El-Yaniv Ran","year":"2010","unstructured":"Ran El-Yaniv 2010 . On the Foundations of Noise-free Selective Classification . Journal of Machine Learning Research , Vol. 11 , 5 (2010). Ran El-Yaniv et al. 2010. On the Foundations of Noise-free Selective Classification. Journal of Machine Learning Research, Vol. 11, 5 (2010).","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287589"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2011-000089"},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8527.2005.00304.x"},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287563"},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jsis.2020.101614"},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11756"},{"key":"e_1_3_2_2_32_1","unstructured":"Kilem L Gwet. 2011. On The Krippendorff's Alpha Coefficient. (2011). https:\/\/agreestat.com\/papers\/onkrippendorffalpha_rev10052015.pdf  Kilem L Gwet. 2011. On The Krippendorff's Alpha Coefficient. (2011). https:\/\/agreestat.com\/papers\/onkrippendorffalpha_rev10052015.pdf"},{"key":"e_1_3_2_2_33_1","volume-title":"ORES: Lowering Barriers with Participatory Machine Learning in Wikipedia. arXiv preprint arXiv:1909.05189","author":"Halfaker Aaron","year":"2019","unstructured":"Aaron Halfaker and R Stuart Geiger . 2019 . ORES: Lowering Barriers with Participatory Machine Learning in Wikipedia. arXiv preprint arXiv:1909.05189 (2019). Aaron Halfaker and R Stuart Geiger. 2019. ORES: Lowering Barriers with Participatory Machine Learning in Wikipedia. arXiv preprint arXiv:1909.05189 (2019)."},{"key":"e_1_3_2_2_34_1","unstructured":"Moritz Hardt Eric Price and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315--3323.  Moritz Hardt Eric Price and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315--3323."},{"key":"e_1_3_2_2_35_1","volume-title":"Convex Analysis and Minimization Algorithms I: Fundamentals","author":"Hiriart-Urruty Jean-Baptiste","unstructured":"Jean-Baptiste Hiriart-Urruty and Claude Lemar\u00e9chal . 2013. Convex Analysis and Minimization Algorithms I: Fundamentals . Vol. 305 . Springer science & business media. Jean-Baptiste Hiriart-Urruty and Claude Lemar\u00e9chal. 2013. Convex Analysis and Minimization Algorithms I: Fundamentals. Vol. 305. Springer science & business media."},{"key":"e_1_3_2_2_36_1","volume-title":"Crowdsourced Task Routing via Matrix Factorization. arXiv preprint arXiv:1310.5142","author":"Jung Hyun Joon","year":"2013","unstructured":"Hyun Joon Jung and Matthew Lease . 2013. Crowdsourced Task Routing via Matrix Factorization. arXiv preprint arXiv:1310.5142 ( 2013 ). Hyun Joon Jung and Matthew Lease. 2013. Crowdsourced Task Routing via Matrix Factorization. arXiv preprint arXiv:1310.5142 (2013)."},{"key":"e_1_3_2_2_37_1","first-page":"1","article-title":"Predicting Next Label Quality","volume":"14","author":"Jung Hyun Joon","year":"2014","unstructured":"Hyun Joon Jung , Yubin Park , and Matthew Lease . 2014 . Predicting Next Label Quality : A Time-Series Model of Crowdwork. HCOMP , Vol. 14 (2014), 1 -- 9 . Hyun Joon Jung, Yubin Park, and Matthew Lease. 2014. Predicting Next Label Quality: A Time-Series Model of Crowdwork. HCOMP, Vol. 14 (2014), 1--9.","journal-title":"A Time-Series Model of Crowdwork. HCOMP"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v3i1.13238"},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/IC4.2009.4909197"},{"key":"e_1_3_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDMW.2011.83"},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372874"},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702520"},{"key":"e_1_3_2_2_43_1","volume-title":"Trust for the doctor-in-the-loop. ERCIM news","author":"Kieseberg Peter","year":"2016","unstructured":"Peter Kieseberg , Edgar Weippl , and Andreas Holzinger . 2016. Trust for the doctor-in-the-loop. ERCIM news , Vol. 104 , 1 ( 2016 ), 32--33. Peter Kieseberg, Edgar Weippl, and Andreas Holzinger. 2016. Trust for the doctor-in-the-loop. ERCIM news, Vol. 104, 1 (2016), 32--33."},{"key":"e_1_3_2_2_44_1","volume-title":"Human Computation","volume":"11","author":"Lease Matthew","year":"2011","unstructured":"Matthew Lease . 2011 . On quality control and machine learning in crowdsourcing . Human Computation , Vol. 11 , 11 (2011). Matthew Lease. 2011. On quality control and machine learning in crowdsourcing. Human Computation, Vol. 11, 11 (2011)."},{"key":"e_1_3_2_2_45_1","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"3","author":"Li Hongwei","year":"2015","unstructured":"Hongwei Li and Qiang Liu . 2015 . Cheaper and Better: Selecting Good Workers for Crowdsourcing . In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , Vol. 3 . Hongwei Li and Qiang Liu. 2015. Cheaper and Better: Selecting Good Workers for Crowdsourcing. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 3."},{"key":"e_1_3_2_2_46_1","volume-title":"Knows what it knows: a framework for self-aware learning. Machine learning","author":"Li Lihong","year":"2011","unstructured":"Lihong Li , Michael L Littman , Thomas J Walsh , and Alexander L Strehl . 2011. Knows what it knows: a framework for self-aware learning. Machine learning , Vol. 82 , 3 ( 2011 ), 399--443. Lihong Li, Michael L Littman, Thomas J Walsh, and Alexander L Strehl. 2011. Knows what it knows: a framework for self-aware learning. Machine learning, Vol. 82, 3 (2011), 399--443."},{"key":"e_1_3_2_2_47_1","volume-title":"Delayed Impact of Fair Machine Learning. In International Conference on Machine Learning. 3150--3158","author":"Liu Lydia T","year":"2018","unstructured":"Lydia T Liu , Sarah Dean , Esther Rolf , Max Simchowitz , and Moritz Hardt . 2018 . Delayed Impact of Fair Machine Learning. In International Conference on Machine Learning. 3150--3158 . Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. 2018. Delayed Impact of Fair Machine Learning. In International Conference on Machine Learning. 3150--3158."},{"key":"e_1_3_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00622"},{"key":"e_1_3_2_2_49_1","unstructured":"Ziyin Liu Zhikang Wang Paul Pu Liang Russ R Salakhutdinov Louis-Philippe Morency and Masahito Ueda. 2019 b. Deep gamblers: Learning to abstain with portfolio theory. In Advances in Neural Information Processing Systems. 10623--10633.  Ziyin Liu Zhikang Wang Paul Pu Liang Russ R Salakhutdinov Louis-Philippe Morency and Masahito Ueda. 2019 b. Deep gamblers: Learning to abstain with portfolio theory. In Advances in Neural Information Processing Systems. 10623--10633."},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10489-019-01423-6"},{"key":"e_1_3_2_2_51_1","volume-title":"Predict Responsibly: Improving fairness and accuracy by learning to defer. In Advances in Neural Information Processing Systems. 6147--6157.","author":"Madras David","year":"2018","unstructured":"David Madras , Toni Pitassi , and Richard Zemel . 2018 . Predict Responsibly: Improving fairness and accuracy by learning to defer. In Advances in Neural Information Processing Systems. 6147--6157. David Madras, Toni Pitassi, and Richard Zemel. 2018. Predict Responsibly: Improving fairness and accuracy by learning to defer. In Advances in Neural Information Processing Systems. 6147--6157."},{"key":"e_1_3_2_2_52_1","volume-title":"International Conference on Machine Learning. PMLR, 6755--6764","author":"Martinez Natalia","year":"2020","unstructured":"Natalia Martinez , Martin Bertran , and Guillermo Sapiro . 2020 . Minimax Pareto fairness: A multi objective perspective . In International Conference on Machine Learning. PMLR, 6755--6764 . Natalia Martinez, Martin Bertran, and Guillermo Sapiro. 2020. Minimax Pareto fairness: A multi objective perspective. In International Conference on Machine Learning. PMLR, 6755--6764."},{"key":"e_1_3_2_2_53_1","volume-title":"A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635","author":"Mehrabi Ninareh","year":"2019","unstructured":"Ninareh Mehrabi , Fred Morstatter , Nripsuta Saxena , Kristina Lerman , and Aram Galstyan . 2019. A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635 ( 2019 ). Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2019. A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635 (2019)."},{"key":"e_1_3_2_2_54_1","volume-title":"Foundations of Machine Learning","author":"Mohri Mehryar","unstructured":"Mehryar Mohri , Afshin Rostamizadeh , and Ameet Talwalkar . 2018. Foundations of Machine Learning . MIT press . Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. 2018. Foundations of Machine Learning .MIT press."},{"key":"e_1_3_2_2_55_1","volume-title":"International Conference on Machine Learning. PMLR, 7076--7087","author":"Mozannar Hussein","year":"2020","unstructured":"Hussein Mozannar and David Sontag . 2020 . Consistent estimators for learning to defer to an expert . In International Conference on Machine Learning. PMLR, 7076--7087 . Hussein Mozannar and David Sontag. 2020. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning. PMLR, 7076--7087."},{"key":"e_1_3_2_2_56_1","volume-title":"How do humans understand explanations from machine learning systems? an evaluation of the human-interpretability of explanation. arXiv preprint arXiv:1802.00682","author":"Narayanan Menaka","year":"2018","unstructured":"Menaka Narayanan , Emily Chen , Jeffrey He , Been Kim , Sam Gershman , and Finale Doshi-Velez . 2018. How do humans understand explanations from machine learning systems? an evaluation of the human-interpretability of explanation. arXiv preprint arXiv:1802.00682 ( 2018 ). Menaka Narayanan, Emily Chen, Jeffrey He, Been Kim, Sam Gershman, and Finale Doshi-Velez. 2018. How do humans understand explanations from machine learning systems? an evaluation of the human-interpretability of explanation. arXiv preprint arXiv:1802.00682 (2018)."},{"key":"e_1_3_2_2_57_1","volume-title":"Third AAAI conference on human computation and crowdsourcing.","author":"Nguyen An Thanh","year":"2015","unstructured":"An Thanh Nguyen , Byron C Wallace , and Matthew Lease . 2015 . Combining crowd and expert labels using decision theoretic active learning . In Third AAAI conference on human computation and crowdsourcing. An Thanh Nguyen, Byron C Wallace, and Matthew Lease. 2015. Combining crowd and expert labels using decision theoretic active learning. In Third AAAI conference on human computation and crowdsourcing."},{"key":"e_1_3_2_2_58_1","volume-title":"Algorithms of oppression: How search engines reinforce racism","author":"Noble Safiya Umoja","unstructured":"Safiya Umoja Noble . 2018. Algorithms of oppression: How search engines reinforce racism . NYU Press . Safiya Umoja Noble. 2018. Algorithms of oppression: How search engines reinforce racism .NYU Press."},{"key":"e_1_3_2_2_59_1","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"3","author":"Nushi Besmira","year":"2015","unstructured":"Besmira Nushi , Adish Singla , Anja Gruenheid , Erfan Zamanian , Andreas Krause , and Donald Kossmann . 2015 . Crowd Access Path Optimization: Diversity Matters . In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , Vol. 3 . Besmira Nushi, Adish Singla, Anja Gruenheid, Erfan Zamanian, Andreas Krause, and Donald Kossmann. 2015. Crowd Access Path Optimization: Diversity Matters. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 3."},{"key":"e_1_3_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3091478.3098871"},{"key":"e_1_3_2_2_61_1","volume-title":"NeurIPS Workshop on Crowdsourcing: Theory, Algorithms and Applications.","author":"Patterson Genevieve","year":"2013","unstructured":"Genevieve Patterson , Grant Van Horn , Serge Belongie , Pietro Perona , and James Hays . 2013 . Bootstrapping Fine-Grained Classifiers: Active Learning with a Crowd in the Loop . In NeurIPS Workshop on Crowdsourcing: Theory, Algorithms and Applications. Genevieve Patterson, Grant Van Horn, Serge Belongie, Pietro Perona, and James Hays. 2013. Bootstrapping Fine-Grained Classifiers: Active Learning with a Crowd in the Loop. In NeurIPS Workshop on Crowdsourcing: Theory, Algorithms and Applications."},{"key":"e_1_3_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"e_1_3_2_2_63_1","unstructured":"Geoff Pleiss Manish Raghavan Felix Wu Jon Kleinberg and Kilian Q Weinberger. 2017. On fairness and calibration. In Advances in Neural Information Processing Systems. 5680--5689.  Geoff Pleiss Manish Raghavan Felix Wu Jon Kleinberg and Kilian Q Weinberger. 2017. On fairness and calibration. In Advances in Neural Information Processing Systems. 5680--5689."},{"key":"e_1_3_2_2_64_1","volume-title":"CHI Workshop on Human-Centered Approaches to Fair and Responsible AI.","author":"Poursabzi-Sangdeh Forough","year":"2020","unstructured":"Forough Poursabzi-Sangdeh , Samira Samadi , Jennifer Wortman Vaughan , and Hanna Wallach . 2020 . A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition . In CHI Workshop on Human-Centered Approaches to Fair and Responsible AI. Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, and Hanna Wallach. 2020. A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition. In CHI Workshop on Human-Centered Approaches to Fair and Responsible AI."},{"key":"e_1_3_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983830"},{"key":"e_1_3_2_2_66_1","volume-title":"2019 a. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv:1903.12220","author":"Raghu Maithra","year":"2019","unstructured":"Maithra Raghu , Katy Blumer , Greg Corrado , Jon Kleinberg , Ziad Obermeyer , and Sendhil Mullainathan . 2019 a. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv:1903.12220 ( 2019 ). Maithra Raghu, Katy Blumer, Greg Corrado, Jon Kleinberg, Ziad Obermeyer, and Sendhil Mullainathan. 2019 a. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv:1903.12220 (2019)."},{"key":"e_1_3_2_2_67_1","volume-title":"International Conference on Machine Learning. 5281--5290","author":"Raghu Maithra","year":"2019","unstructured":"Maithra Raghu , Katy Blumer , Rory Sayres , Ziad Obermeyer , Bobby Kleinberg , Sendhil Mullainathan , and Jon Kleinberg . 2019 b. Direct uncertainty prediction for medical second opinions . In International Conference on Machine Learning. 5281--5290 . Maithra Raghu, Katy Blumer, Rory Sayres, Ziad Obermeyer, Bobby Kleinberg, Sendhil Mullainathan, and Jon Kleinberg. 2019 b. Direct uncertainty prediction for medical second opinions. In International Conference on Machine Learning. 5281--5290."},{"key":"e_1_3_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3375627.3375820"},{"key":"e_1_3_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1163"},{"key":"e_1_3_2_2_70_1","unstructured":"Eliza Strickland. 2018. https:\/\/spectrum.ieee.org\/computing\/software\/aihuman-partnerships-tackle-fake-news  Eliza Strickland. 2018. https:\/\/spectrum.ieee.org\/computing\/software\/aihuman-partnerships-tackle-fake-news"},{"key":"e_1_3_2_2_71_1","volume-title":"An introduction to recursive partitioning: rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychological methods","author":"Strobl Carolin","year":"2009","unstructured":"Carolin Strobl , James Malley , and Gerhard Tutz . 2009. An introduction to recursive partitioning: rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychological methods , Vol. 14 , 4 ( 2009 ), 323. Carolin Strobl, James Malley, and Gerhard Tutz. 2009. An introduction to recursive partitioning: rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychological methods, Vol. 14, 4 (2009), 323."},{"key":"e_1_3_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/PST.2018.8514168"},{"key":"e_1_3_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-019-01386-7"},{"key":"e_1_3_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567989"},{"key":"e_1_3_2_2_75_1","doi-asserted-by":"crossref","unstructured":"Bryan Wilder Eric Horvitz and Ece Kamar. 2020. Learning to Complement Humans. (2020).  Bryan Wilder Eric Horvitz and Ece Kamar. 2020. Learning to Complement Humans. (2020).","DOI":"10.24963\/ijcai.2020\/212"},{"key":"e_1_3_2_2_76_1","volume-title":"International Conference of Machine Learning.","author":"Yan Yan","year":"2011","unstructured":"Yan Yan , Romer Rosales , Glenn Fung , and Jennifer G Dy . 2011 . Active learning from crowds . In International Conference of Machine Learning. Yan Yan, Romer Rosales, Glenn Fung, and Jennifer G Dy. 2011. Active learning from crowds. In International Conference of Machine Learning."},{"key":"e_1_3_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1144"},{"key":"e_1_3_2_2_78_1","unstructured":"Dirk A Zetzsche Douglas W Arner Ross P Buckley and Brian Tang. 2020. Artificial Intelligence in Finance: Putting the Human in the Loop. (2020).  Dirk A Zetzsche Douglas W Arner Ross P Buckley and Brian Tang. 2020. Artificial Intelligence in Finance: Putting the Human in the Loop. (2020)."},{"key":"e_1_3_2_2_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372852"},{"key":"e_1_3_2_2_80_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8640.2010.00358.x"}],"event":{"name":"AIES '21: AAAI\/ACM Conference on AI, Ethics, and Society","location":"Virtual Event USA","acronym":"AIES '21","sponsor":["SIGAI ACM Special Interest Group on Artificial Intelligence","AAAI"]},"container-title":["Proceedings of the 2021 AAAI\/ACM Conference on AI, Ethics, and Society"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3461702.3462516","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3461702.3462516","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:49:05Z","timestamp":1750193345000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3461702.3462516"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,21]]},"references-count":80,"alternative-id":["10.1145\/3461702.3462516","10.1145\/3461702"],"URL":"https:\/\/doi.org\/10.1145\/3461702.3462516","relation":{},"subject":[],"published":{"date-parts":[[2021,7,21]]},"assertion":[{"value":"2021-07-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}