{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,23]],"date-time":"2026-03-23T00:54:37Z","timestamp":1774227277166,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":42,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,10,18]],"date-time":"2020-10-18T00:00:00Z","timestamp":1602979200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000093","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["U54 CA193313"],"award-info":[{"award-number":["U54 CA193313"]}],"id":[{"id":"10.13039\/100000093","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,10,19]]},"DOI":"10.1145\/3412815.3416889","type":"proceedings-article","created":{"date-parts":[[2020,10,15]],"date-time":"2020-10-15T23:37:54Z","timestamp":1602805074000},"page":"47-57","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":21,"title":["Interpreting Black Box Models via Hypothesis Testing"],"prefix":"10.1145","author":[{"given":"Collin","family":"Burns","sequence":"first","affiliation":[{"name":"Columbia University, New York, NY, USA"}]},{"given":"Jesse","family":"Thomason","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}]},{"given":"Wesley","family":"Tansey","sequence":"additional","affiliation":[{"name":"Memorial Sloan Kettering Cancer Center, New York, NY, USA"}]}],"member":"320","published-online":{"date-parts":[[2020,10,18]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"Julius Adebayo Justin Gilmer Michael Muelly Ian J. Goodfellow Moritz Hardt and Been Kim. 2018. Sanity Checks for Saliency Maps. In Neural Information Processing Systems (NeurIPS) .  Julius Adebayo Justin Gilmer Michael Muelly Ian J. Goodfellow Moritz Hardt and Been Kim. 2018. Sanity Checks for Saliency Maps. In Neural Information Processing Systems (NeurIPS) ."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1214\/15-AOS1337"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-017-0189-z"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1995.tb02031.x"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1013699998"},{"key":"e_1_3_2_1_6_1","volume-title":"Rina Foygel Barber, and Richard J Samworth","author":"Berrett Thomas B","year":"2018","unstructured":"Thomas B Berrett , Yi Wang , Rina Foygel Barber, and Richard J Samworth . 2018 . The conditional permutation test. arXiv preprint arXiv:1807.05405 (2018). Thomas B Berrett, Yi Wang, Rina Foygel Barber, and Richard J Samworth. 2018. The conditional permutation test. arXiv preprint arXiv:1807.05405 (2018)."},{"key":"e_1_3_2_1_7_1","volume-title":"On Identifiability in Transformers. arXiv preprint arXiv:1908.04211","author":"Brunner Gino","year":"2019","unstructured":"Gino Brunner , Yang Liu , Dami\u00e1n Pascual , Oliver Richter , Massimiliano Ciaramita , and Roger Wattenhofer . 2019. On Identifiability in Transformers. arXiv preprint arXiv:1908.04211 ( 2019 ). Gino Brunner, Yang Liu, Dami\u00e1n Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wattenhofer. 2019. On Identifiability in Transformers. arXiv preprint arXiv:1908.04211 (2019)."},{"key":"e_1_3_2_1_8_1","volume-title":"Interactive Classification for Deep Learning Interpretation. arXiv preprint arXiv:1806.05660","author":"Cabrera Angel","year":"2018","unstructured":"Angel Cabrera , Fred Hohman , Jason Lin , and Duen Horng Chau . 2018. Interactive Classification for Deep Learning Interpretation. arXiv preprint arXiv:1806.05660 ( 2018 ). Angel Cabrera, Fred Hohman, Jason Lin, and Duen Horng Chau. 2018. Interactive Classification for Deep Learning Interpretation. arXiv preprint arXiv:1806.05660 (2018)."},{"key":"e_1_3_2_1_9_1","volume-title":"Panning for gold: \"Model-X\" knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society: Series B","author":"Candes Emmanuel","year":"2018","unstructured":"Emmanuel Candes , Yingying Fan , Lucas Janson , and Jinchi Lv. 2018. Panning for gold: \"Model-X\" knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society: Series B ( 2018 ). Emmanuel Candes, Yingying Fan, Lucas Janson, and Jinchi Lv. 2018. Panning for gold: \"Model-X\" knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society: Series B (2018)."},{"key":"e_1_3_2_1_10_1","volume-title":"International Conference on Learning Representations (ICLR) .","author":"Chang Chun-Hao","year":"2018","unstructured":"Chun-Hao Chang , Elliot Creager , Anna Goldenberg , and David Duvenaud . 2018 . Explaining Image Classifiers by Adaptive Dropout and Generative In-filling . In International Conference on Learning Representations (ICLR) . Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. 2018. Explaining Image Classifiers by Adaptive Dropout and Generative In-filling. In International Conference on Learning Representations (ICLR) ."},{"key":"e_1_3_2_1_11_1","volume-title":"International Conference on Machine Learning (ICML) .","author":"Chen Jianbo","year":"2018","unstructured":"Jianbo Chen , Le Song , Martin J Wainwright , and Michael I Jordan . 2018 . Learning to Explain: An Information-Theoretic Perspective on Model Interpretation . In International Conference on Machine Learning (ICML) . Jianbo Chen, Le Song, Martin J Wainwright, and Michael I Jordan. 2018. Learning to Explain: An Information-Theoretic Perspective on Model Interpretation. In International Conference on Machine Learning (ICML) ."},{"key":"e_1_3_2_1_12_1","volume-title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In North American","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2019 . BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In North American Chapter of the Association for Computational Linguistics (NAACL) . Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In North American Chapter of the Association for Computational Linguistics (NAACL) ."},{"key":"e_1_3_2_1_13_1","volume-title":"Randomization tests","author":"Edgington Eugene","unstructured":"Eugene Edgington and Patrick Onghena . 2007. Randomization tests . Chapman and Hall\/CRC. Eugene Edgington and Patrick Onghena. 2007. Randomization tests .Chapman and Hall\/CRC."},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.371"},{"key":"e_1_3_2_1_15_1","volume-title":"International Conference on Learning Representations (ICLR) .","author":"Geirhos Robert","year":"2019","unstructured":"Robert Geirhos , Patricia Rubisch , Claudio Michaelis , Matthias Bethge , Felix A. Wichmann , and Wieland Brendel . 2019 . ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness . In International Conference on Learning Representations (ICLR) . Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. 2019. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations (ICLR) ."},{"key":"e_1_3_2_1_16_1","volume-title":"International Conference on Learning Representations (ICLR) .","author":"Goodfellow Ian J.","year":"2015","unstructured":"Ian J. Goodfellow , Jonathon Shlens , and Christian Szegedy . 2015 . Explaining and Harnessing Adversarial Examples . In International Conference on Learning Representations (ICLR) . Ian J. Goodfellow , Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples . In International Conference on Learning Representations (ICLR) ."},{"key":"e_1_3_2_1_17_1","first-page":"127","article-title":"Image Inpainting : Overview and Recent Advances. Signal Processing Magazine","volume":"31","author":"Guillemot Christine","year":"2014","unstructured":"Christine Guillemot and Olivier Le Meur . 2014 . Image Inpainting : Overview and Recent Advances. Signal Processing Magazine , IEEE , Vol. 31 (2014), 127 -- 144 . Christine Guillemot and Olivier Le Meur. 2014. Image Inpainting : Overview and Recent Advances. Signal Processing Magazine, IEEE , Vol. 31 (2014), 127--144.","journal-title":"IEEE"},{"key":"e_1_3_2_1_18_1","volume-title":"International Conference on Learning Representations (ICLR) .","author":"Hendrycks Dan","unstructured":"Dan Hendrycks and Thomas G. Dietterich . 2019. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations . In International Conference on Learning Representations (ICLR) . Dan Hendrycks and Thomas G. Dietterich. 2019. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. In International Conference on Learning Representations (ICLR) ."},{"key":"e_1_3_2_1_19_1","volume-title":"Wallace","author":"Jain Sarthak","year":"2019","unstructured":"Sarthak Jain and Byron C . Wallace . 2019 . Attention is not Explanation. In North American Chapter of the Association for Computational Linguistics (NAACL) . Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In North American Chapter of the Association for Computational Linguistics (NAACL) ."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00300"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1214\/15-AOS1371"},{"key":"e_1_3_2_1_22_1","volume-title":"The Mythos of Model Interpretability. In ICML Workshop on Human Interpretability in Machine Learning (WHI) .","author":"Lipton Zachary Chase","year":"2016","unstructured":"Zachary Chase Lipton . 2016 . The Mythos of Model Interpretability. In ICML Workshop on Human Interpretability in Machine Learning (WHI) . Zachary Chase Lipton. 2016. The Mythos of Model Interpretability. In ICML Workshop on Human Interpretability in Machine Learning (WHI) ."},{"key":"e_1_3_2_1_23_1","unstructured":"Jiasen Lu Dhruv Batra Devi Parikh and Stefan Lee. 2019. ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. In Neural Information Processing Systems (NeurIPS) .  Jiasen Lu Dhruv Batra Devi Parikh and Stefan Lee. 2019. ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. In Neural Information Processing Systems (NeurIPS) ."},{"key":"e_1_3_2_1_24_1","unstructured":"Scott Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Neural Information Processing Systems (NeurIPS) .  Scott Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Neural Information Processing Systems (NeurIPS) ."},{"key":"e_1_3_2_1_25_1","volume-title":"Learning Word Vectors for Sentiment Analysis","author":"Maas Andrew L.","unstructured":"Andrew L. Maas , Raymond E. Daly , Peter T. Pham , Dan Huang , Andrew Y. Ng , and Christopher Potts . 2011. Learning Word Vectors for Sentiment Analysis . In Association for Computational Linguistics : Human Language Technologies (ACL). 142--150. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning Word Vectors for Sentiment Analysis. In Association for Computational Linguistics: Human Language Technologies (ACL). 142--150."},{"key":"e_1_3_2_1_26_1","volume-title":"YOLOv3: An Incremental Improvement. arXiv preprint arXiv","author":"Redmon Joseph","year":"1804","unstructured":"Joseph Redmon and Ali Farhadi . 2018. YOLOv3: An Incremental Improvement. arXiv preprint arXiv : 1804 .02767 (2018). Joseph Redmon and Ali Farhadi. 2018. YOLOv3: An Incremental Improvement. arXiv preprint arXiv: 1804.02767 (2018)."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_3_2_1_28_1","volume-title":"Mimic and Classify: A meta-algorithm for Conditional Independence Testing. arXiv preprint arXiv:1806.09708","author":"Sen Rajat","year":"2018","unstructured":"Rajat Sen , Karthikeyan Shanmugam , Himanshu Asnani , Arman Rahimzamani , and Sreeram Kannan . 2018. Mimic and Classify: A meta-algorithm for Conditional Independence Testing. arXiv preprint arXiv:1806.09708 ( 2018 ). Rajat Sen, Karthikeyan Shanmugam, Himanshu Asnani, Arman Rahimzamani, and Sreeram Kannan. 2018. Mimic and Classify: A meta-algorithm for Conditional Independence Testing. arXiv preprint arXiv:1806.09708 (2018)."},{"key":"e_1_3_2_1_29_1","volume-title":"Learning Important Features Through Propagating Activation Differences. In International Conference on Machine Learning (ICML). 3145--3153","author":"Shrikumar Avanti","year":"2017","unstructured":"Avanti Shrikumar , Peyton Greenside , and Anshul Kundaje . 2017 . Learning Important Features Through Propagating Activation Differences. In International Conference on Machine Learning (ICML). 3145--3153 . Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning Important Features Through Propagating Activation Differences. In International Conference on Machine Learning (ICML). 3145--3153."},{"key":"e_1_3_2_1_30_1","volume-title":"Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. In International Conference on Learning Representations (ICLR) Workshop .","author":"Simonyan Karen","year":"2014","unstructured":"Karen Simonyan , Andrea Vedaldi , and Andrew Zisserman . 2014 . Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. In International Conference on Learning Representations (ICLR) Workshop . Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. In International Conference on Learning Representations (ICLR) Workshop ."},{"key":"e_1_3_2_1_31_1","volume-title":"Chest X-ray Inpainting with Deep Generative Models. arXiv preprint arXiv:1809.01471","author":"Sogancioglu Ecem","year":"2018","unstructured":"Ecem Sogancioglu , Shi Hu , Davide Belli , and Bram van Ginneken . 2018. Chest X-ray Inpainting with Deep Generative Models. arXiv preprint arXiv:1809.01471 ( 2018 ). Ecem Sogancioglu, Shi Hu, Davide Belli, and Bram van Ginneken. 2018. Chest X-ray Inpainting with Deep Generative Models. arXiv preprint arXiv:1809.01471 (2018)."},{"key":"e_1_3_2_1_32_1","volume-title":"Rethinking the Inception Architecture for Computer Vision. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .","author":"Szegedy Christian","year":"2016","unstructured":"Christian Szegedy , Vincent Vanhoucke , Sergey Ioffe , Jonathon Shlens , and Zbigniew Wojna . 2016 . Rethinking the Inception Architecture for Computer Vision. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2016. Rethinking the Inception Architecture for Computer Vision. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) ."},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1257\/jep.2.1.191"},{"key":"e_1_3_2_1_34_1","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez \u0141ukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Neural Information Processing Systems (NeurIPS) .  Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez \u0141ukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Neural Information Processing Systems (NeurIPS) ."},{"key":"e_1_3_2_1_35_1","volume-title":"Bowman","author":"Wang Alex","year":"2018","unstructured":"Alex Wang , Amanpreet Singh , Julian Michael , Felix Hill , Omer Levy , and Samuel R . Bowman . 2018 a. GLUE : A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding . arXiv preprint arXiv:1804.07461 (2018). Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018a. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. arXiv preprint arXiv:1804.07461 (2018)."},{"key":"e_1_3_2_1_36_1","unstructured":"Yi Wang Xin Tao Xiaojuan Qi Xiaoyong Shen and Jiaya Jia. 2018b. Image Inpainting via Generative Multi-column Convolutional Neural Networks. In Neural Information Processing Systems (NeurIPS) .  Yi Wang Xin Tao Xiaojuan Qi Xiaoyong Shen and Jiaya Jia. 2018b. Image Inpainting via Generative Multi-column Convolutional Neural Networks. In Neural Information Processing Systems (NeurIPS) ."},{"key":"e_1_3_2_1_37_1","volume-title":"Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144","author":"Wu Yonghui","year":"2016","unstructured":"Yonghui Wu , Mike Schuster , Zhifeng Chen , Quoc V Le , Mohammad Norouzi , Wolfgang Macherey , Maxim Krikun , Yuan Cao , Qin Gao , and Klaus Macherey . 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 ( 2016 ). Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, and Klaus Macherey. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)."},{"key":"e_1_3_2_1_38_1","volume-title":"Do","author":"Yeh Raymond A.","year":"2017","unstructured":"Raymond A. Yeh , Chen Chen , Teck-Yian Lim , Mark Hasegawa-Johnson , and Minh N . Do . 2017 . Semantic Image Inpainting with Perceptual and Contextual Losses . arXiv preprint arXiv:1607.07539 (2017). Raymond A. Yeh, Chen Chen, Teck-Yian Lim, Mark Hasegawa-Johnson, and Minh N. Do. 2017. Semantic Image Inpainting with Perceptual and Contextual Losses. arXiv preprint arXiv:1607.07539 (2017)."},{"key":"e_1_3_2_1_39_1","volume-title":"Free-Form Image Inpainting with Gated Convolution. arXiv preprint arXiv:1806.03589","author":"Yu Jiahui","year":"2018","unstructured":"Jiahui Yu , Zhe Lin , Jimei Yang , Xiaohui Shen , Xin Lu , and Thomas S Huang . 2018a. Free-Form Image Inpainting with Gated Convolution. arXiv preprint arXiv:1806.03589 ( 2018 ). Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. 2018a. Free-Form Image Inpainting with Gated Convolution. arXiv preprint arXiv:1806.03589 (2018)."},{"key":"e_1_3_2_1_40_1","volume-title":"Huang","author":"Yu Jiahui","year":"2018","unstructured":"Jiahui Yu , Zhe Lin , Jimei Yang , Xiaohui Shen , Xin Lu , and Thomas S . Huang . 2018 b. Generative Image Inpainting with Contextual Attention. In Computer Vision and Pattern Recognition (CVPR) . Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. 2018b. Generative Image Inpainting with Contextual Attention. In Computer Vision and Pattern Recognition (CVPR) ."},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2909553"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.5555\/3020548.3020641"}],"event":{"name":"FODS '20: ACM-IMS Foundations of Data Science Conference","location":"Virtual Event USA","acronym":"FODS '20"},"container-title":["Proceedings of the 2020 ACM-IMS on Foundations of Data Science Conference"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3412815.3416889","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3412815.3416889","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:25:02Z","timestamp":1750195502000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3412815.3416889"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,18]]},"references-count":42,"alternative-id":["10.1145\/3412815.3416889","10.1145\/3412815"],"URL":"https:\/\/doi.org\/10.1145\/3412815.3416889","relation":{},"subject":[],"published":{"date-parts":[[2020,10,18]]},"assertion":[{"value":"2020-10-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}