{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T04:16:15Z","timestamp":1771560975278,"version":"3.50.1"},"reference-count":96,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2022,11,7]],"date-time":"2022-11-07T00:00:00Z","timestamp":1667779200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"NVIDIA GPU Grant"},{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["No. 1755850, No. 1841520, No. 2007716, No. 2007976, No. 1942594, No. 190780"],"award-info":[{"award-number":["No. 1755850, No. 1841520, No. 2007716, No. 2007976, No. 1942594, No. 190780"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Amazon Research Award"},{"name":"Jeffress Memorial Trust Award"},{"name":"Design Knowledge Company","award":["10827.002.120.04"],"award-info":[{"award-number":["10827.002.120.04"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2022,11,7]]},"abstract":"<jats:p>While Deep Neural Networks (DNNs) are deriving the major innovations through their powerful automation, we are also witnessing the peril behind automation as a form of bias, such as automated racism, gender bias, and adversarial bias. As the societal impact of DNNs grows, finding an effective way to steer DNNs to align their behavior with the human mental model has become indispensable in realizing fair and accountable models. While establishing the way to adjust DNNs to \"think like humans'' is in pressing need, there have been few approaches aiming to capture how \"humans would think'' when DNNs introduce biased reasoning in seeing a new instance. We propose Interactive Attention Alignment (IAA), a framework that uses the methods for visualizing model attention, such as saliency maps, as an interactive medium that humans can leverage to unveil the cases of DNN's biased reasoning and directly adjust the attention. To realize more effective human-steerable DNNs than state-of-the-art, IAA introduces two novel devices. First, IAA uses Reasonability Matrix to systematically identify and adjust the cases of biased attention. Second, IAA applies GRADIA, a computational pipeline designed for effectively applying the adjusted attention to jointly maximize attention quality and prediction accuracy. We evaluated Reasonability Matrix in Study 1 and GRADIA in Study 2 in the gender classification problem. In Study 1, we found applying Reasonability Matrix in bias detection can significantly improve the perceived quality of model attention from human eyes than not applying Reasonability Matrix. In Study 2, we found using GRADIA significantly improves (1) the human-assessed perceived quality of model attention and (2) model performance in scenarios where the training samples are limited. Based on our observation in the two studies, we present implications for future design in the problem space of social computing and interactive data annotation toward achieving a human-centered steerable AI.<\/jats:p>","DOI":"10.1145\/3555590","type":"journal-article","created":{"date-parts":[[2022,11,11]],"date-time":"2022-11-11T22:59:06Z","timestamp":1668207546000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":25,"title":["Aligning Eyes between Humans and Deep Neural Network through Interactive Attention Alignment"],"prefix":"10.1145","volume":"6","author":[{"given":"Yuyang","family":"Gao","sequence":"first","affiliation":[{"name":"Emory University, Atlanta, GA, USA"}]},{"given":"Tong Steven","family":"Sun","sequence":"additional","affiliation":[{"name":"George Mason University, Fairfax, VA, USA"}]},{"given":"Liang","family":"Zhao","sequence":"additional","affiliation":[{"name":"Emory University, Atlanta, GA, USA"}]},{"given":"Sungsoo Ray","family":"Hong","sequence":"additional","affiliation":[{"name":"George Mason University, Fairfax, VA, USA"}]}],"member":"320","published-online":{"date-parts":[[2022,11,11]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33011418"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2014.2346660"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702509"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209581"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539442"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3450069"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3432931"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.354"},{"key":"e_1_2_1_9_1","volume-title":"Do convolutional neural networks learn class hierarchy? IEEE transactions on visualization and computer graphics","author":"Bilal Alsallakh","year":"2017","unstructured":"Alsallakh Bilal , Amin Jourabloo , Mao Ye , Xiaoming Liu , and Liu Ren . 2017. Do convolutional neural networks learn class hierarchy? IEEE transactions on visualization and computer graphics , Vol. 24 , 1 ( 2017 ), 152--162. Alsallakh Bilal, Amin Jourabloo, Mao Ye, Xiaoming Liu, and Liu Ren. 2017. Do convolutional neural networks learn class hierarchy? IEEE transactions on visualization and computer graphics, Vol. 24, 1 (2017), 152--162."},{"key":"e_1_2_1_10_1","volume-title":"International Conference on Machine Learning. PMLR, 1014--1023","author":"Boopathy Akhilan","year":"2020","unstructured":"Akhilan Boopathy , Sijia Liu , Gaoyuan Zhang , Cynthia Liu , Pin-Yu Chen , Shiyu Chang , and Luca Daniel . 2020 . Proper network interpretability helps adversarial robustness in classification . In International Conference on Machine Learning. PMLR, 1014--1023 . Akhilan Boopathy, Sijia Liu, Gaoyuan Zhang, Cynthia Liu, Pin-Yu Chen, Shiyu Chang, and Luca Daniel. 2020. Proper network interpretability helps adversarial robustness in classification. In International Conference on Machine Learning. PMLR, 1014--1023."},{"key":"e_1_2_1_11_1","volume-title":"VINE: Visualizing Statistical Interactions in Black Box Models. arXiv preprint arXiv:1904.00561","author":"Britton Matthew","year":"2019","unstructured":"Matthew Britton . 2019 . VINE: Visualizing Statistical Interactions in Black Box Models. arXiv preprint arXiv:1904.00561 (2019). Matthew Britton. 2019. VINE: Visualizing Statistical Interactions in Black Box Models. arXiv preprint arXiv:1904.00561 (2019)."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/VAST.2015.7347637"},{"key":"e_1_2_1_13_1","volume-title":"European Conference on Computer Vision. Springer, 793--811","author":"Burns Kaylee","year":"2018","unstructured":"Kaylee Burns , Lisa Anne Hendricks , Kate Saenko , Trevor Darrell , and Anna Rohrbach . 2018 . Women also snowboard: Overcoming bias in captioning models . In European Conference on Computer Vision. Springer, 793--811 . Kaylee Burns, Lisa Anne Hendricks, Kate Saenko, Trevor Darrell, and Anna Rohrbach. 2018. Women also snowboard: Overcoming bias in captioning models. In European Conference on Computer Vision. Springer, 793--811."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/VAST47406.2019.8986948"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDMW.2009.83"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-010-0190-x"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.953"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300460"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/VAST.2010.5652443"},{"key":"e_1_2_1_20_1","volume-title":"The frontiers of fairness in machine learning. arXiv preprint arXiv:1810.08810","author":"Chouldechova Alexandra","year":"2018","unstructured":"Alexandra Chouldechova and Aaron Roth . 2018. The frontiers of fairness in machine learning. arXiv preprint arXiv:1810.08810 ( 2018 ). Alexandra Chouldechova and Aaron Roth. 2018. The frontiers of fairness in machine learning. arXiv preprint arXiv:1810.08810 (2018)."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3476037"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction CSCW","author":"Chung Chaeyeon","year":"2021","unstructured":"Chaeyeon Chung , Jung Soo Lee , Kyungmin Park , Junsoo Lee , Jaegul Choo , and Sungsoo Ray Hong . 2021 a. Understanding Human-side Impact of Sequencing Images in Batch Labeling for Subjective Tasks . Proceedings of the ACM on Human-Computer Interaction CSCW (2021). Chaeyeon Chung, Jung Soo Lee, Kyungmin Park, Junsoo Lee, Jaegul Choo, and Sungsoo Ray Hong. 2021a. Understanding Human-side Impact of Sequencing Images in Batch Labeling for Subjective Tasks. Proceedings of the ACM on Human-Computer Interaction CSCW (2021)."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2017.10.001"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2865043"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/604045.604056"},{"key":"e_1_2_1_26_1","unstructured":"Alice Feng and Shuyan Wu. 2019 (accessed August 23 2020). The Myth of the Impartial Machine. https:\/\/parametric.press\/issue-01\/the-myth-of-the-impartial-machine\/  Alice Feng and Shuyan Wu. 2019 (accessed August 23 2020). The Myth of the Impartial Machine. https:\/\/parametric.press\/issue-01\/the-myth-of-the-impartial-machine\/"},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10705--10714","author":"Fukui Hiroshi","unstructured":"Hiroshi Fukui , Tsubasa Hirakawa , Takayoshi Yamashita , and Hironobu Fujiyoshi . [n.,d.]. Attention branch network: Learning of attention mechanism for visual explanation . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10705--10714 . Hiroshi Fukui, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. [n.,d.]. Attention branch network: Learning of attention mechanism for visual explanation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10705--10714."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.3389\/fnbot.2021.567482"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM51629.2021.00023"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539419"},{"key":"e_1_2_1_31_1","volume-title":"Maximilian M\u00fchlegg, Sebastian Dorn, Tiffany Fernandez, Martin J\"anicke, Sudesh Mirashi, Chiragkumar Savani, Martin Sturm, Oleksandr Vorobiov, Martin Oelker, Sebastian Garreis, and Peter Schuberth.","author":"Geyer Jakob","year":"2020","unstructured":"Jakob Geyer , Yohannes Kassahun , Mentar Mahmudi , Xavier Ricou , Rupesh Durgesh , Andrew S. Chung , Lorenz Hauswald , Viet Hoang Pham , Maximilian M\u00fchlegg, Sebastian Dorn, Tiffany Fernandez, Martin J\"anicke, Sudesh Mirashi, Chiragkumar Savani, Martin Sturm, Oleksandr Vorobiov, Martin Oelker, Sebastian Garreis, and Peter Schuberth. 2020 . A2D2: Audi Autonomous Driving Dataset . (2020). arxiv: 2004.06320 [cs.CV] https:\/\/www.a2d2.audi Jakob Geyer, Yohannes Kassahun, Mentar Mahmudi, Xavier Ricou, Rupesh Durgesh, Andrew S. Chung, Lorenz Hauswald, Viet Hoang Pham, Maximilian M\u00fchlegg, Sebastian Dorn, Tiffany Fernandez, Martin J\"anicke, Sudesh Mirashi, Chiragkumar Savani, Martin Sturm, Oleksandr Vorobiov, Martin Oelker, Sebastian Garreis, and Peter Schuberth. 2020. A2D2: Audi Autonomous Driving Dataset. (2020). arxiv: 2004.06320 [cs.CV] https:\/\/www.a2d2.audi"},{"key":"e_1_2_1_32_1","volume-title":"A 20-Year Community Roadmap for Artificial Intelligence Research in the US. arXiv preprint arXiv:1908.02624","author":"Gil Yolanda","year":"2019","unstructured":"Yolanda Gil and Bart Selman . 2019. A 20-Year Community Roadmap for Artificial Intelligence Research in the US. arXiv preprint arXiv:1908.02624 ( 2019 ). Yolanda Gil and Bart Selman. 2019. A 20-Year Community Roadmap for Artificial Intelligence Research in the US. arXiv preprint arXiv:1908.02624 (2019)."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236009"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3403221"},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition workshops. 10--17","author":"Gupta Ritwik","year":"2019","unstructured":"Ritwik Gupta , Bryce Goodman , Nirav Patel , Ricky Hosfelt , Sandra Sajeev , Eric Heim , Jigar Doshi , Keane Lucas , Howie Choset , and Matthew Gaston . 2019 . Creating xBD: A dataset for assessing building damage from satellite imagery . In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition workshops. 10--17 . Ritwik Gupta, Bryce Goodman, Nirav Patel, Ricky Hosfelt, Sandra Sajeev, Eric Heim, Jigar Doshi, Keane Lucas, Howie Choset, and Matthew Gaston. 2019. Creating xBD: A dataset for assessing building damage from satellite imagery. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition workshops. 10--17."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1656274.1656278"},{"key":"e_1_2_1_37_1","unstructured":"Moritz Hardt Eric Price and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315--3323.  Moritz Hardt Eric Price and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315--3323."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2008.239"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_1_40_1","volume-title":"Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers","author":"Hohman Fred","year":"2018","unstructured":"Fred Hohman , Minsuk Kahng , Robert Pienta , and Duen Horng Chau . 2018. Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers . IEEE Transactions on Visualization and Computer Graphics ( 2018 ). https:\/\/doi.org\/10.1109\/TVCG.2018.2843369 10.1109\/TVCG.2018.2843369 Fred Hohman, Minsuk Kahng, Robert Pienta, and Duen Horng Chau. 2018. Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers. IEEE Transactions on Visualization and Computer Graphics (2018). https:\/\/doi.org\/10.1109\/TVCG.2018.2843369"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '19)","author":"Holstein Kenneth","unstructured":"Kenneth Holstein , Jennifer Wortman Vaughan , Hal Daum\u00e9 , III , Miro Dudik , and Hanna Wallach . [n.,d.]. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? . In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '19) . ACM, New York, NY, USA, 600:1--600:16. https:\/\/doi.org\/10.1145\/3290605.3300830 10.1145\/3290605.3300830 Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daum\u00e9, III, Miro Dudik, and Hanna Wallach. [n.,d.]. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, 600:1--600:16. https:\/\/doi.org\/10.1145\/3290605.3300830"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3392878"},{"key":"e_1_2_1_43_1","volume-title":"Harnessing deep neural networks with logic rules. arXiv preprint arXiv:1603.06318","author":"Hu Zhiting","year":"2016","unstructured":"Zhiting Hu , Xuezhe Ma , Zhengzhong Liu , Eduard Hovy , and Eric Xing . 2016. Harnessing deep neural networks with logic rules. arXiv preprint arXiv:1603.06318 ( 2016 ). Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. 2016. Harnessing deep neural networks with logic rules. arXiv preprint arXiv:1603.06318 (2016)."},{"key":"e_1_2_1_44_1","volume-title":"Aligning Faithful Interpretations with their Social Attribution. arXiv preprint arXiv:2006.01067","author":"Jacovi Alon","year":"2020","unstructured":"Alon Jacovi and Yoav Goldberg . 2020. Aligning Faithful Interpretations with their Social Attribution. arXiv preprint arXiv:2006.01067 ( 2020 ). Alon Jacovi and Yoav Goldberg. 2020. Aligning Faithful Interpretations with their Social Attribution. arXiv preprint arXiv:2006.01067 (2020)."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744718"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939502.2939503"},{"key":"e_1_2_1_47_1","volume-title":"Fernanda B Vi\u00e9gas, and Martin Wattenberg.","author":"Kahng Minsuk","year":"2018","unstructured":"Minsuk Kahng , Nikhil Thorat , Duen Horng Polo Chau , Fernanda B Vi\u00e9gas, and Martin Wattenberg. 2018 . Gan lab: Understanding complex deep generative models using interactive visual experimentation. IEEE transactions on visualization and computer graphics, Vol. 25 , 1 (2018), 1--11. Minsuk Kahng, Nikhil Thorat, Duen Horng Polo Chau, Fernanda B Vi\u00e9gas, and Martin Wattenberg. 2018. Gan lab: Understanding complex deep generative models using interactive visual experimentation. IEEE transactions on visualization and computer graphics, Vol. 25, 1 (2018), 1--11."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/IC4.2009.4909197"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2010.50"},{"key":"e_1_2_1_50_1","doi-asserted-by":"crossref","unstructured":"Alexandros Karargyris Satyananda Kashyap Ismini Lourentzou Joy T Wu Arjun Sharma Matthew Tong Shafiq Abedin David Beymer Vandana Mukherjee Elizabeth A Krupinski etal 2021. Creation and validation of a chest X-ray dataset with eye-tracking and report dictation for AI development. Scientific data Vol. 8 1 (2021) 1--18.  Alexandros Karargyris Satyananda Kashyap Ismini Lourentzou Joy T Wu Arjun Sharma Matthew Tong Shafiq Abedin David Beymer Vandana Mukherjee Elizabeth A Krupinski et al. 2021. Creation and validation of a chest X-ray dataset with eye-tracking and report dictation for AI development. Scientific data Vol. 8 1 (2021) 1--18.","DOI":"10.1038\/s41597-021-00863-5"},{"key":"e_1_2_1_51_1","unstructured":"John Kendall. 2018 (accessed August 23 2020). How to Boost Border Security While Protecting Privacy. https:\/\/www.nextgov.com\/ideas\/2018\/05\/how-boost-border-security-while-protecting-privacy\/148288\/  John Kendall. 2018 (accessed August 23 2020). How to Boost Border Security While Protecting Privacy. https:\/\/www.nextgov.com\/ideas\/2018\/05\/how-boost-border-security-while-protecting-privacy\/148288\/"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00922"},{"key":"e_1_2_1_53_1","volume-title":"Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980","author":"Kingma Diederik P","year":"2014","unstructured":"Diederik P Kingma and Jimmy Ba . 2014 . Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)."},{"key":"e_1_2_1_54_1","volume-title":"Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907","author":"Kipf Thomas N","year":"2016","unstructured":"Thomas N Kipf and Max Welling . 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 ( 2016 ). Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2017.285"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858529"},{"key":"e_1_2_1_57_1","doi-asserted-by":"crossref","unstructured":"Himabindu Lakkaraju Ece Kamar Rich Caruana and Eric Horvitz. 2017. Identifying unknown unknowns in the open world: Representations and policies for guided exploration. In Thirty-first aaai conference on artificial intelligence.  Himabindu Lakkaraju Ece Kamar Rich Caruana and Eric Horvitz. 2017. Identifying unknown unknowns in the open world: Representations and policies for guided exploration. In Thirty-first aaai conference on artificial intelligence.","DOI":"10.1609\/aaai.v31i1.10821"},{"key":"e_1_2_1_58_1","volume-title":"Grid search, random search, genetic algorithm: A big comparison for NAS. arXiv preprint arXiv:1912.06059","author":"Liashchynskyi Petro","year":"2019","unstructured":"Petro Liashchynskyi and Pavlo Liashchynskyi . 2019. Grid search, random search, genetic algorithm: A big comparison for NAS. arXiv preprint arXiv:1912.06059 ( 2019 ). Petro Liashchynskyi and Pavlo Liashchynskyi. 2019. Grid search, random search, genetic algorithm: A big comparison for NAS. arXiv preprint arXiv:1912.06059 (2019)."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380306"},{"key":"e_1_2_1_61_1","volume-title":"Incorporating priors with feature attribution on text classification. arXiv preprint arXiv:1906.08286","author":"Liu Frederick","year":"2019","unstructured":"Frederick Liu and Besim Avci . 2019. Incorporating priors with feature attribution on text classification. arXiv preprint arXiv:1906.08286 ( 2019 ). Frederick Liu and Besim Avci. 2019. Incorporating priors with feature attribution on text classification. arXiv preprint arXiv:1906.08286 (2019)."},{"key":"e_1_2_1_62_1","volume-title":"Analyzing the training processes of deep generative models","author":"Liu Mengchen","year":"2017","unstructured":"Mengchen Liu , Jiaxin Shi , Kelei Cao , Jun Zhu , and Shixia Liu . 2017. Analyzing the training processes of deep generative models . IEEE transactions on visualization and computer graphics, Vol. 24 , 1 ( 2017 ), 77--87. Mengchen Liu, Jiaxin Shi, Kelei Cao, Jun Zhu, and Shixia Liu. 2017. Analyzing the training processes of deep generative models. IEEE transactions on visualization and computer graphics, Vol. 24, 1 (2017), 77--87."},{"key":"e_1_2_1_63_1","volume-title":"Towards better analysis of deep convolutional neural networks","author":"Liu Mengchen","year":"2016","unstructured":"Mengchen Liu , Jiaxin Shi , Zhen Li , Chongxuan Li , Jun Zhu , and Shixia Liu . 2016. Towards better analysis of deep convolutional neural networks . IEEE transactions on visualization and computer graphics, Vol. 23 , 1 ( 2016 ), 91--100. Mengchen Liu, Jiaxin Shi, Zhen Li, Chongxuan Li, Jun Zhu, and Shixia Liu. 2016. Towards better analysis of deep convolutional neural networks. IEEE transactions on visualization and computer graphics, Vol. 23, 1 (2016), 91--100."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10921-015-0315-7"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00222"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/VAST.2017.8585721"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2864812"},{"key":"e_1_2_1_68_1","volume-title":"Embedding Human Knowledge into Deep Neural Network via Attention Map. arXiv preprint arXiv:1905.03540","author":"Mitsuhara Masahiro","year":"2019","unstructured":"Masahiro Mitsuhara , Hiroshi Fukui , Yusuke Sakashita , Takanori Ogata , Tsubasa Hirakawa , Takayoshi Yamashita , and Hironobu Fujiyoshi . 2019. Embedding Human Knowledge into Deep Neural Network via Attention Map. arXiv preprint arXiv:1905.03540 ( 2019 ). Masahiro Mitsuhara, Hiroshi Fukui, Yusuke Sakashita, Takanori Ogata, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. 2019. Embedding Human Knowledge into Deep Neural Network via Attention Map. arXiv preprint arXiv:1905.03540 (2019)."},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.534"},{"key":"e_1_2_1_70_1","volume-title":"Towards accountable ai: Hybrid human-machine analyses for characterizing system failure. arXiv preprint arXiv:1809.07424","author":"Nushi Besmira","year":"2018","unstructured":"Besmira Nushi , Ece Kamar , and Eric Horvitz . 2018. Towards accountable ai: Hybrid human-machine analyses for characterizing system failure. arXiv preprint arXiv:1809.07424 ( 2018 ). Besmira Nushi, Ece Kamar, and Eric Horvitz. 2018. Towards accountable ai: Hybrid human-machine analyses for characterizing system failure. arXiv preprint arXiv:1809.07424 (2018)."},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i07.6858"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401959"},{"key":"e_1_2_1_73_1","volume-title":"Boudewijn PF Lelieveldt, Elmar Eisemann, and Anna Vilanova.","author":"Pezzotti Nicola","year":"2017","unstructured":"Nicola Pezzotti , Thomas H\u00f6llt , Jan Van Gemert , Boudewijn PF Lelieveldt, Elmar Eisemann, and Anna Vilanova. 2017 . Deepeyes : Progressive visual analytics for designing deep neural networks. IEEE transactions on visualization and computer graphics, Vol. 24 , 1 (2017), 98--108. Nicola Pezzotti, Thomas H\u00f6llt, Jan Van Gemert, Boudewijn PF Lelieveldt, Elmar Eisemann, and Anna Vilanova. 2017. Deepeyes: Progressive visual analytics for designing deep neural networks. IEEE transactions on visualization and computer graphics, Vol. 24, 1 (2017), 98--108."},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.12272"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00842"},{"key":"e_1_2_1_76_1","volume-title":"Squares: Supporting interactive performance analysis for multiclass classifiers","author":"Ren Donghao","year":"2016","unstructured":"Donghao Ren , Saleema Amershi , Bongshin Lee , Jina Suh , and Jason D Williams . 2016 . Squares: Supporting interactive performance analysis for multiclass classifiers . IEEE transactions on visualization and computer graphics, Vol. 23 , 1 (2016), 61--70. https:\/\/doi.org\/10.1109\/TVCG.2016.2598828 10.1109\/TVCG.2016.2598828 Donghao Ren, Saleema Amershi, Bongshin Lee, Jina Suh, and Jason D Williams. 2016. Squares: Supporting interactive performance analysis for multiclass classifiers. IEEE transactions on visualization and computer graphics, Vol. 23, 1 (2016), 61--70. https:\/\/doi.org\/10.1109\/TVCG.2016.2598828"},{"key":"e_1_2_1_77_1","volume-title":"Right for the right reasons: Training differentiable models by constraining their explanations. arXiv preprint arXiv:1703.03717","author":"Ross Andrew Slavin","year":"2017","unstructured":"Andrew Slavin Ross , Michael C Hughes , and Finale Doshi-Velez . 2017. Right for the right reasons: Training differentiable models by constraining their explanations. arXiv preprint arXiv:1703.03717 ( 2017 ). Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Velez. 2017. Right for the right reasons: Training differentiable models by constraining their explanations. arXiv preprint arXiv:1703.03717 (2017)."},{"key":"e_1_2_1_78_1","volume-title":"International Conference on Machine Learning. PMLR, 8346--8356","author":"Sagawa Shiori","year":"2020","unstructured":"Shiori Sagawa , Aditi Raghunathan , Pang Wei Koh , and Percy Liang . 2020 . An investigation of why overparameterization exacerbates spurious correlations . In International Conference on Machine Learning. PMLR, 8346--8356 . Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. 2020. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning. PMLR, 8346--8356."},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314248"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.74"},{"key":"e_1_2_1_81_1","volume-title":"Lstmvis: A tool for visual analysis of hidden state dynamics in recurrent neural networks","author":"Strobelt Hendrik","year":"2017","unstructured":"Hendrik Strobelt , Sebastian Gehrmann , Hanspeter Pfister , and Alexander M Rush . 2017 . Lstmvis: A tool for visual analysis of hidden state dynamics in recurrent neural networks . IEEE transactions on visualization and computer graphics, Vol. 24 , 1 (2017), 667--676. Hendrik Strobelt, Sebastian Gehrmann, Hanspeter Pfister, and Alexander M Rush. 2017. Lstmvis: A tool for visual analysis of hidden state dynamics in recurrent neural networks. IEEE transactions on visualization and computer graphics, Vol. 24, 1 (2017), 667--676."},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1518895"},{"key":"e_1_2_1_83_1","unstructured":"Tesla. 2020 (accessed August 23 2020). Tesla Autopilot. https:\/\/www.tesla.com\/autopilotAI  Tesla. 2020 (accessed August 23 2020). Tesla Autopilot. https:\/\/www.tesla.com\/autopilotAI"},{"key":"e_1_2_1_84_1","volume-title":"Few-shot learning with per-sample rich supervision. arXiv preprint arXiv:1906.03859","author":"Visotsky Roman","year":"2019","unstructured":"Roman Visotsky , Yuval Atzmon , and Gal Chechik . 2019. Few-shot learning with per-sample rich supervision. arXiv preprint arXiv:1906.03859 ( 2019 ). Roman Visotsky, Yuval Atzmon, and Gal Chechik. 2019. Few-shot learning with per-sample rich supervision. arXiv preprint arXiv:1906.03859 (2019)."},{"key":"e_1_2_1_85_1","volume-title":"Follow My Eye: Using Gaze to Supervise Computer-Aided Diagnosis","author":"Wang Sheng","year":"2022","unstructured":"Sheng Wang , Xi Ouyang , Tianming Liu , Qian Wang , and Dinggang Shen . 2022. Follow My Eye: Using Gaze to Supervise Computer-Aided Diagnosis . IEEE Transactions on Medical Imaging ( 2022 ). Sheng Wang, Xi Ouyang, Tianming Liu, Qian Wang, and Dinggang Shen. 2022. Follow My Eye: Using Gaze to Supervise Computer-Aided Diagnosis. IEEE Transactions on Medical Imaging (2022)."},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00541"},{"key":"e_1_2_1_87_1","first-page":"35","article-title":"Cost-sensitive learning vs. sampling: Which is best for handling unbalanced classes with unequal error costs","volume":"7","author":"Weiss Gary M","year":"2007","unstructured":"Gary M Weiss , Kate McCarthy , and Bibi Zabar . 2007 . Cost-sensitive learning vs. sampling: Which is best for handling unbalanced classes with unequal error costs ? Dmin , Vol. 7 , 35 -- 41 (2007), 24. Gary M Weiss, Kate McCarthy, and Bibi Zabar. 2007. Cost-sensitive learning vs. sampling: Which is best for handling unbalanced classes with unequal error costs? Dmin, Vol. 7, 35--41 (2007), 24.","journal-title":"Dmin"},{"key":"e_1_2_1_88_1","volume-title":"Visualizing Dataflow Graphs of Deep learning Models in Tensorflow","author":"Wongsuphasawat Kanit","year":"2017","unstructured":"Kanit Wongsuphasawat , Daniel Smilkov , James Wexler , Jimbo Wilson , Dandelion Mane , Doug Fritz , Dilip Krishnan , Fernanda B Vi\u00e9gas , and Martin Wattenberg . 2017. Visualizing Dataflow Graphs of Deep learning Models in Tensorflow . IEEE transactions on visualization and computer graphics, Vol. 24 , 1 ( 2017 ), 1--12. https:\/\/doi.org\/10.1109\/TVCG.2017.2744878 10.1109\/TVCG.2017.2744878 Kanit Wongsuphasawat, Daniel Smilkov, James Wexler, Jimbo Wilson, Dandelion Mane, Doug Fritz, Dilip Krishnan, Fernanda B Vi\u00e9gas, and Martin Wattenberg. 2017. Visualizing Dataflow Graphs of Deep learning Models in Tensorflow. IEEE transactions on visualization and computer graphics, Vol. 24, 1 (2017), 1--12. https:\/\/doi.org\/10.1109\/TVCG.2017.2744878"},{"key":"e_1_2_1_89_1","unstructured":"Chuan Yan John Joon Young Chung Kiheon Yoon Yotam Gingold Eytan Adar and Sungsoo Ray Hong. [n. d.]. FlatMagic: Improving Flat Colorization through AI-driven Design for Digital Comic Professionals.  Chuan Yan John Joon Young Chung Kiheon Yoon Yotam Gingold Eytan Adar and Sungsoo Ray Hong. [n. d.]. FlatMagic: Improving Flat Colorization through AI-driven Design for Digital Comic Professionals."},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376447"},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1076"},{"key":"e_1_2_1_92_1","volume-title":"Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv:1707.09457","author":"Zhao Jieyu","year":"2017","unstructured":"Jieyu Zhao , Tianlu Wang , Mark Yatskar , Vicente Ordonez , and Kai-Wei Chang . 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv:1707.09457 ( 2017 ). Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv:1707.09457 (2017)."},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.319"},{"key":"e_1_2_1_94_1","volume-title":"A brief introduction to weakly supervised learning. National science review","author":"Zhou Zhi-Hua","year":"2018","unstructured":"Zhi-Hua Zhou . 2018. A brief introduction to weakly supervised learning. National science review , Vol. 5 , 1 ( 2018 ), 44--53. Zhi-Hua Zhou. 2018. A brief introduction to weakly supervised learning. National science review, Vol. 5, 1 (2018), 44--53."},{"key":"e_1_2_1_95_1","volume-title":"Gaze-Guided Class Activation Mapping: Leveraging Human Attention for Network Attention in Chest X-rays Classification. arXiv preprint arXiv:2202.07107","author":"Zhu Hongzhi","year":"2022","unstructured":"Hongzhi Zhu , Septimiu Salcudean , and Robert Rohling . 2022. Gaze-Guided Class Activation Mapping: Leveraging Human Attention for Network Attention in Chest X-rays Classification. arXiv preprint arXiv:2202.07107 ( 2022 ). Hongzhi Zhu, Septimiu Salcudean, and Robert Rohling. 2022. Gaze-Guided Class Activation Mapping: Leveraging Human Attention for Network Attention in Chest X-rays Classification. arXiv preprint arXiv:2202.07107 (2022)."},{"key":"e_1_2_1_96_1","volume-title":"A survey on measuring indirect discrimination in machine learning. arXiv preprint arXiv:1511.00148","author":"Zliobaite Indre","year":"2015","unstructured":"Indre Zliobaite . 2015. A survey on measuring indirect discrimination in machine learning. arXiv preprint arXiv:1511.00148 ( 2015 ). Indre Zliobaite. 2015. A survey on measuring indirect discrimination in machine learning. arXiv preprint arXiv:1511.00148 (2015)."}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3555590","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3555590","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3555590","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:49:03Z","timestamp":1750182543000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3555590"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,7]]},"references-count":96,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2022,11,7]]}},"alternative-id":["10.1145\/3555590"],"URL":"https:\/\/doi.org\/10.1145\/3555590","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,7]]},"assertion":[{"value":"2022-11-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}