{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T07:08:30Z","timestamp":1781334510344,"version":"3.54.1"},"publisher-location":"New York, NY, USA","reference-count":59,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,7,3]],"date-time":"2020-07-03T00:00:00Z","timestamp":1593734400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"JP Morgan Faculty Award"},{"DOI":"10.13039\/100014718","name":"National Science Foundation","doi-asserted-by":"publisher","award":["IIS-2001851,IIS-2000782,IIS-1939606"],"award-info":[{"award-number":["IIS-2001851,IIS-2000782,IIS-1939606"]}],"id":[{"id":"10.13039\/100014718","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,7,3]]},"DOI":"10.1145\/3357236.3395528","type":"proceedings-article","created":{"date-parts":[[2020,7,5]],"date-time":"2020-07-05T19:29:16Z","timestamp":1593977356000},"page":"1245-1257","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":40,"title":["Keeping Designers in the Loop"],"prefix":"10.1145","author":[{"given":"Bowen","family":"Yu","sequence":"first","affiliation":[{"name":"University of Minnesota - Twin Cities, Minneapolis, MN, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ye","family":"Yuan","sequence":"additional","affiliation":[{"name":"University of Minnesota - Twin Cities, Minneapolis, MN, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Loren","family":"Terveen","sequence":"additional","affiliation":[{"name":"University of Minnesota - Twin Cities, Minneapolis, MN, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhiwei Steven","family":"Wu","sequence":"additional","affiliation":[{"name":"University of Minnesota - Twin Cities, Minneapolis, MN, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jodi","family":"Forlizzi","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Haiyi","family":"Zhu","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,7,3]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"2019. Introduction - Machine Learning - Human Interface Guidelines - Apple Developer. (2019). https:\/\/developer.apple.com\/design\/human-interface-guidelines\/machine-learning\/overview\/introduction\/  2019. Introduction - Machine Learning - Human Interface Guidelines - Apple Developer. (2019). https:\/\/developer.apple.com\/design\/human-interface-guidelines\/machine-learning\/overview\/introduction\/"},{"key":"e_1_3_2_2_2_1","volume-title":"https:\/\/design.google\/ai-guidebook","author":"Guidebook People","year":"2019","unstructured":"2019. People + AI Guidebook . ( 2019 ). https:\/\/design.google\/ai-guidebook 2019. People + AI Guidebook. (2019). https:\/\/design.google\/ai-guidebook"},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174156"},{"key":"e_1_3_2_2_4_1","volume-title":"A reductions approach to fair classification. arXiv preprint arXiv:1803.02453","author":"Agarwal Alekh","year":"2018","unstructured":"Alekh Agarwal , Alina Beygelzimer , Miroslav Dud\u00edk , John Langford , and Hanna Wallach . 2018. A reductions approach to fair classification. arXiv preprint arXiv:1803.02453 ( 2018 ). Alekh Agarwal, Alina Beygelzimer, Miroslav Dud\u00edk, John Langford, and Hanna Wallach. 2018. A reductions approach to fair classification. arXiv preprint arXiv:1803.02453 (2018)."},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300233"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1177\/0011128705281756"},{"key":"e_1_3_2_2_7_1","volume-title":"IJCAI-17 workshop on explainable AI (XAI)","volume":"8","author":"Biran Or","year":"2017","unstructured":"Or Biran and Courtenay Cotton . 2017 . Explanation and justification in machine learning: A survey . In IJCAI-17 workshop on explainable AI (XAI) , Vol. 8 . 1. Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In IJCAI-17 workshop on explainable AI (XAI), Vol. 8. 1."},{"key":"e_1_3_2_2_8_1","volume-title":"Conference on fairness, accountability and transparency. 77--91","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru . 2018 . Gender shades: Intersectional accuracy disparities in commercial gender classification . In Conference on fairness, accountability and transparency. 77--91 . Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. 77--91."},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359206"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.23915\/distill.00009"},{"key":"e_1_3_2_2_11_1","unstructured":"Kathy Charmaz. 2014. Constructing grounded theory. sage.  Kathy Charmaz. 2014. Constructing grounded theory. sage."},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300789"},{"key":"e_1_3_2_2_13_1","volume-title":"Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data 5, 2","author":"Chouldechova Alexandra","year":"2017","unstructured":"Alexandra Chouldechova . 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data 5, 2 ( 2017 ), 153--163. Alexandra Chouldechova. 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data 5, 2 (2017), 153--163."},{"key":"e_1_3_2_2_14_1","volume-title":"Conference on Fairness, Accountability and Transparency. 134--148","author":"Chouldechova Alexandra","year":"2018","unstructured":"Alexandra Chouldechova , Diana Benavides-Prado , Oleksandr Fialko , and Rhema Vaithianathan . 2018 . A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions . In Conference on Fairness, Accountability and Transparency. 134--148 . Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Conference on Fairness, Accountability and Transparency. 134--148."},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1013115.1013148"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098095"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1071-5819(03)00041-7"},{"key":"e_1_3_2_2_18_1","unstructured":"Mark Craven and Jude Shavlik. 1999. Rule extraction: Where do we go from here. University of Wisconsin Machine Learning Research Group working Paper 99 (1999).  Mark Craven and Jude Shavlik. 1999. Rule extraction: Where do we go from here. University of Wisconsin Machine Learning Research Group working Paper 99 (1999)."},{"key":"e_1_3_2_2_19_1","unstructured":"Jean-Charles de Borda. 1781. M\u00e9moire sur les \u00e9lections au scrutin. Histoire de l'Acad\u00e9mie Royale des Sciences (1781).  Jean-Charles de Borda. 1781. M\u00e9moire sur les \u00e9lections au scrutin. Histoire de l'Acad\u00e9mie Royale des Sciences (1781)."},{"key":"e_1_3_2_2_20_1","volume-title":"Demonstrating accuracy equity and predictive parity","author":"Dieterich William","year":"2016","unstructured":"William Dieterich , Christina Mendoza , and Tim Brennan . 2016.com PAS risk scales : Demonstrating accuracy equity and predictive parity . Northpoint Inc ( 2016 ). William Dieterich, Christina Mendoza, and Tim Brennan. 2016.comPAS risk scales: Demonstrating accuracy equity and predictive parity. Northpoint Inc (2016)."},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025739"},{"key":"e_1_3_2_2_22_1","volume-title":"The accuracy, fairness, and limits of predicting recidivism. Science advances 4, 1","author":"Dressel Julia","year":"2018","unstructured":"Julia Dressel and Hany Farid . 2018. The accuracy, fairness, and limits of predicting recidivism. Science advances 4, 1 ( 2018 ), eaao5580. Julia Dressel and Hany Farid. 2018. The accuracy, fairness, and limits of predicting recidivism. Science advances 4, 1 (2018), eaao5580."},{"key":"e_1_3_2_2_23_1","volume-title":"Conference on Fairness, Accountability and Transparency. 119--133","author":"Dwork Cynthia","year":"2018","unstructured":"Cynthia Dwork , Nicole Immorlica , Adam Tauman Kalai , and Max Leiserson . 2018 . Decoupled classifiers for group-fair and efficient machine learning . In Conference on Fairness, Accountability and Transparency. 119--133 . Cynthia Dwork, Nicole Immorlica, Adam Tauman Kalai, and Max Leiserson. 2018. Decoupled classifiers for group-fair and efficient machine learning. In Conference on Fairness, Accountability and Transparency. 119--133."},{"key":"e_1_3_2_2_24_1","volume-title":"Proceedings of The Eleventh International Society for Music Information Retrieval Conference (ISMIR 2010)","author":"Fiebrink Rebecca","year":"2010","unstructured":"Rebecca Fiebrink and Perry R Cook . 2010 . The Wekinator: a system for real-time, interactive machine learning in music . In Proceedings of The Eleventh International Society for Music Information Retrieval Conference (ISMIR 2010) (Utrecht). Rebecca Fiebrink and Perry R Cook. 2010. The Wekinator: a system for real-time, interactive machine learning in music. In Proceedings of The Eleventh International Society for Music Information Retrieval Conference (ISMIR 2010)(Utrecht)."},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSAA.2018.00018"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1177\/1473871611416549"},{"key":"e_1_3_2_2_27_1","unstructured":"Moritz Hardt Eric Price Nati Srebro and others. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315--3323.  Moritz Hardt Eric Price Nati Srebro and others. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315--3323."},{"key":"e_1_3_2_2_28_1","volume-title":"Jessamyn Burns Wendell, and Shelley Wood","author":"Holtzblatt Karen","year":"2004","unstructured":"Karen Holtzblatt , Jessamyn Burns Wendell, and Shelley Wood . 2004 . Rapid contextual design: a how-to guide to key techniques for user-centered design. Elsevier . Karen Holtzblatt, Jessamyn Burns Wendell, and Shelley Wood. 2004. Rapid contextual design: a how-to guide to key techniques for user-centered design. Elsevier."},{"key":"e_1_3_2_2_29_1","volume-title":"Surya Mattu and Julia Angwin","author":"Jeff Larson Lauren Kirchner","year":"2016","unstructured":"Lauren Kirchner Jeff Larson , Surya Mattu and Julia Angwin . 2016 . How We Analyzed the COMPAS Recidivism Algorithm . (2016). https:\/\/www.propublica.org\/article\/how-we-analyzed-the-compas-recidivism-algorithm Lauren Kirchner Jeff Larson, Surya Mattu and Julia Angwin. 2016. How We Analyzed the COMPAS Recidivism Algorithm. (2016). https:\/\/www.propublica.org\/article\/how-we-analyzed-the-compas-recidivism-algorithm"},{"key":"e_1_3_2_2_30_1","volume-title":"Jeff Larson and Lauren Kirchner","author":"Julia Angwin Surya Mattu","year":"2016","unstructured":"Surya Mattu Julia Angwin , Jeff Larson and Lauren Kirchner . 2016 . Machine Bias . (2016). https:\/\/www.propublica.org\/article\/machine-bias-risk-assessments-in-criminal-sentencing Surya Mattu Julia Angwin, Jeff Larson and Lauren Kirchner. 2016. Machine Bias. (2016). https:\/\/www.propublica.org\/article\/machine-bias-risk-assessments-in-criminal-sentencing"},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287592"},{"key":"e_1_3_2_2_32_1","volume-title":"Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm\u00e4ssan","author":"Kearns Michael J.","year":"2018","unstructured":"Michael J. Kearns , Seth Neel , Aaron Roth , and Zhiwei Steven Wu . 2018 . Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness . In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm\u00e4ssan , Stockholm, Sweden, July 10--15 , 2018. 2569--2577. http:\/\/proceedings.mlr.press\/v80\/kearns18a.html Michael J. Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm\u00e4ssan, Stockholm, Sweden, July 10--15, 2018. 2569--2577. http:\/\/proceedings.mlr.press\/v80\/kearns18a.html"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287592"},{"key":"e_1_3_2_2_34_1","volume-title":"Aea papers and proceedings","author":"Kleinberg Jon","unstructured":"Jon Kleinberg , Jens Ludwig , Sendhil Mullainathan , and Ashesh Rambachan . 2018. Algorithmic fairness . In Aea papers and proceedings , Vol. 108 . 22--27. Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Ashesh Rambachan. 2018. Algorithmic fairness. In Aea papers and proceedings, Vol. 108. 22--27."},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.4230\/LIPIcs.ITCS.2017.43"},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1001\/archfami.7.5.451"},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858529"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939874"},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1080\/00140139208967392"},{"key":"e_1_3_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359284"},{"key":"e_1_3_2_2_41_1","volume-title":"Xinran Yuan, A. D. C. Chan, Ritesh Noothigattu, Daniel See, Siheon Lee, Christos-Alexandros Psomas, and Ariel D. Procaccia.","author":"Lee Min Kyung","year":"2018","unstructured":"Min Kyung Lee , Daniel Kusbit , Anson Kahng , Ji Seong Tae , Xinran Yuan, A. D. C. Chan, Ritesh Noothigattu, Daniel See, Siheon Lee, Christos-Alexandros Psomas, and Ariel D. Procaccia. 2018 . WeBuildAI : Participatory Framework for Fair and Efficient Algorithmic Governance . Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Seong Tae, Xinran Yuan, A. D. C. Chan, Ritesh Noothigattu, Daniel See, Siheon Lee, Christos-Alexandros Psomas, and Ariel D. Procaccia. 2018. WeBuildAI : Participatory Framework for Fair and Efficient Algorithmic Governance."},{"key":"e_1_3_2_2_42_1","volume-title":"Identifying careless responses in survey data. Psychological methods 17, 3","author":"Meade Adam W","year":"2012","unstructured":"Adam W Meade and S Bartholomew Craig . 2012. Identifying careless responses in survey data. Psychological methods 17, 3 ( 2012 ), 437. Adam W Meade and S Bartholomew Craig. 2012. Identifying careless responses in survey data. Psychological methods 17, 3 (2012), 437."},{"key":"e_1_3_2_2_43_1","volume-title":"Conference on Fairness, Accountability and Transparency. 107--118","author":"Menon Aditya Krishna","year":"2018","unstructured":"Aditya Krishna Menon and Robert C Williamson . 2018 . The cost of fairness in binary classification . In Conference on Fairness, Accountability and Transparency. 107--118 . Aditya Krishna Menon and Robert C Williamson. 2018. The cost of fairness in binary classification. In Conference on Fairness, Accountability and Transparency. 107--118."},{"key":"e_1_3_2_2_44_1","unstructured":"Geoff Pleiss Manish Raghavan Felix Wu Jon Kleinberg and Kilian Q Weinberger. 2017. On fairness and calibration. In Advances in Neural Information Processing Systems. 5680--5689.  Geoff Pleiss Manish Raghavan Felix Wu Jon Kleinberg and Kilian Q Weinberger. 2017. On fairness and calibration. In Advances in Neural Information Processing Systems. 5680--5689."},{"key":"e_1_3_2_2_45_1","volume-title":"Model-agnostic interpretability of machine learning. arXiv preprint arXiv:1606.05386","author":"Ribeiro Marco Tulio","year":"2016","unstructured":"Marco Tulio Ribeiro , Sameer Singh , and Carlos Guestrin . 2016a. Model-agnostic interpretability of machine learning. arXiv preprint arXiv:1606.05386 ( 2016 ). Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016a. Model-agnostic interpretability of machine learning. arXiv preprint arXiv:1606.05386 (2016)."},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_3_2_2_47_1","volume-title":"The sociology of georg simmel","author":"Simmel Georg","unstructured":"Georg Simmel . 1950. The sociology of georg simmel . Vol. 92892 . Simon and Schuster . Georg Simmel. 1950. The sociology of georg simmel. Vol. 92892. Simon and Schuster."},{"key":"e_1_3_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1080\/14999013.2014.922141"},{"key":"e_1_3_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1111\/1745-9125.12123"},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1518895"},{"key":"e_1_3_2_2_51_1","volume-title":"Prototyping ways of prototyping AI. interactions 25, 6","author":"van Allen Philip","year":"2018","unstructured":"Philip van Allen . 2018. Prototyping ways of prototyping AI. interactions 25, 6 ( 2018 ), 46--51. Philip van Allen. 2018. Prototyping ways of prototyping AI. interactions 25, 6 (2018), 46--51."},{"key":"e_1_3_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174014"},{"key":"e_1_3_2_2_53_1","volume-title":"Your resume vs. oblivion. The Wall Street Journal 24","author":"Weber Lauren","year":"2012","unstructured":"Lauren Weber and RE Silverman . 2012. Your resume vs. oblivion. The Wall Street Journal 24 ( 2012 ). Lauren Weber and RE Silverman. 2012. Your resume vs. oblivion. The Wall Street Journal 24 (2012)."},{"key":"e_1_3_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2648.2008.04924.x"},{"key":"e_1_3_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.5555\/3122009.3153004"},{"key":"e_1_3_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3196709.3196730"},{"key":"e_1_3_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376301"},{"key":"e_1_3_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/2901790.2901858"},{"key":"e_1_3_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274463"}],"event":{"name":"DIS '20: Designing Interactive Systems Conference 2020","location":"Eindhoven Netherlands","acronym":"DIS '20","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Proceedings of the 2020 ACM Designing Interactive Systems Conference"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3357236.3395528","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3357236.3395528","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:41:26Z","timestamp":1750200086000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3357236.3395528"}},"subtitle":["Communicating Inherent Algorithmic Trade-offs Across Multiple Objectives"],"short-title":[],"issued":{"date-parts":[[2020,7,3]]},"references-count":59,"alternative-id":["10.1145\/3357236.3395528","10.1145\/3357236"],"URL":"https:\/\/doi.org\/10.1145\/3357236.3395528","relation":{},"subject":[],"published":{"date-parts":[[2020,7,3]]},"assertion":[{"value":"2020-07-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}