{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,27]],"date-time":"2026-10-27T18:13:13Z","timestamp":1793124793155,"version":"4.5.0"},"publisher-location":"New York, NY, USA","reference-count":58,"publisher":"ACM","funder":[{"DOI":"10.13039\/100010269","name":"Wellcome Trust","doi-asserted-by":"publisher","award":["223765\/Z\/21\/Z"],"award-info":[{"award-number":["223765\/Z\/21\/Z"]}],"id":[{"id":"10.13039\/100010269","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Sloan Foundation","award":["G-2021-16779"],"award-info":[{"award-number":["G-2021-16779"]}]},{"name":"EPSRC","award":["EP\/Y019393\/1"],"award-info":[{"award-number":["EP\/Y019393\/1"]}]},{"DOI":"10.13039\/501100000276","name":"Department of Health and Social Care","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000276","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Luminate Group"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,6,23]]},"DOI":"10.1145\/3715275.3732219","type":"proceedings-article","created":{"date-parts":[[2025,6,23]],"date-time":"2025-06-23T17:01:18Z","timestamp":1750698078000},"page":"3400-3411","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Evaluating Model Explanations without Ground Truth"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1087-4599","authenticated-orcid":false,"given":"Kaivalya","family":"Rawal","sequence":"first","affiliation":[{"name":"University of Oxford, Oxford, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-9103-9305","authenticated-orcid":false,"given":"Zihao","family":"Fu","sequence":"additional","affiliation":[{"name":"University of Oxford, Oxford, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7282-8494","authenticated-orcid":false,"given":"Eoin","family":"Delaney","sequence":"additional","affiliation":[{"name":"Trinity College Dublin, Dublin, Ireland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1665-1759","authenticated-orcid":false,"given":"Chris","family":"Russell","sequence":"additional","affiliation":[{"name":"University of Oxford, Oxford, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,23]]},"reference":[{"key":"e_1_3_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.5555\/3327546.3327621"},{"key":"e_1_3_3_1_3_2","series-title":"(NIPS \u201922)","volume-title":"Proceedings of the 36th International Conference on Neural Information Processing Systems","author":"Agarwal Chirag","year":"2024","unstructured":"Chirag Agarwal, Satyapriya Krishna, Eshika Saxena, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. 2024. OpenXAI: towards a transparent evaluation of post hoc model explanations. In Proceedings of the 36th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS \u201922). Curran Associates Inc., Red Hook, NY, USA, Article 1148, 16\u00a0pages."},{"key":"e_1_3_3_1_4_2","series-title":"(NIPS \u201921)","volume-title":"Proceedings of the 35th International Conference on Neural Information Processing Systems","author":"A\u00efvodji Ulrich","year":"2024","unstructured":"Ulrich A\u00efvodji, Hiromi Arai, S\u00e9bastien Gambs, and Satoshi Hara. 2024. Characterizing the risk of fairwashing. In Proceedings of the 35th International Conference on Neural Information Processing Systems(NIPS \u201921). Curran Associates Inc., Red Hook, NY, USA, Article 1136, 13\u00a0pages."},{"key":"e_1_3_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377325.3377519"},{"key":"e_1_3_3_1_6_2","unstructured":"Brian Barr Noah Fatsi Leif Hancox-Li Peter Richter Daniel Proano and Caleb Mok. 2023. The Disagreement Problem in Faithfulness Metrics. arxiv:https:\/\/arXiv.org\/abs\/2311.07763\u00a0[cs.LG] https:\/\/arxiv.org\/abs\/2311.07763"},{"key":"e_1_3_3_1_7_2","unstructured":"Barry Becker and Ronny Kohavi. 1996. Adult. UCI Machine Learning Repository. DOI: https:\/\/doi.org\/10.24432\/C5XW20."},{"key":"e_1_3_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377325.3377498"},{"key":"e_1_3_3_1_9_2","doi-asserted-by":"publisher","unstructured":"Diogo\u00a0V. Carvalho Eduardo\u00a0M. Pereira and Jaime\u00a0S. Cardoso. 2019. Machine Learning Interpretability: A Survey on Methods and Metrics. Electronics 8 8 (2019). https:\/\/doi.org\/10.3390\/electronics8080832","DOI":"10.3390\/electronics8080832"},{"key":"e_1_3_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1128"},{"key":"e_1_3_3_1_11_2","unstructured":"Victor Chang Jozeene Bailey Qianwen\u00a0Ariel Xu and Zhili Sun. 2022. Pima Indians diabetes mellitus classification based on machine learning (ML) algorithms. Neural Comput Appl \u00a0 \u00a0 (March 2022) 1\u201317."},{"key":"e_1_3_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3593970"},{"key":"e_1_3_3_1_13_2","first-page":"2832","volume-title":"Advances in Neural Information Processing Systems","volume":"35","author":"Colin Julien","year":"2022","unstructured":"Julien Colin, Thomas FEL, Remi Cadene, and Thomas Serre. 2022. What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods. In Advances in Neural Information Processing Systems, S.\u00a0Koyejo, S.\u00a0Mohamed, A.\u00a0Agarwal, D.\u00a0Belgrave, K.\u00a0Cho, and A.\u00a0Oh (Eds.), Vol.\u00a035. Curran Associates, Inc., \u00a0, 2832\u20132845. https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2022\/file\/13113e938f2957891c0c5e8df811dd01-Paper-Conference.pdf"},{"key":"e_1_3_3_1_14_2","doi-asserted-by":"publisher","unstructured":"T. Cover and P. Hart. 1967. Nearest neighbor pattern classification. IEEE Transactions on Information Theory 13 1 (1967) 21\u201327. https:\/\/doi.org\/10.1109\/TIT.1967.1053964","DOI":"10.1109\/TIT.1967.1053964"},{"key":"e_1_3_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3514094.3534159"},{"key":"e_1_3_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_3_1_17_2","unstructured":"Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretable Machine Learning. arxiv:https:\/\/arXiv.org\/abs\/1702.08608\u00a0[stat.ML] https:\/\/arxiv.org\/abs\/1702.08608"},{"key":"e_1_3_3_1_18_2","series-title":"(NIPS \u201921)","volume-title":"Proceedings of the 35th International Conference on Neural Information Processing Systems","author":"Fel Thomas","year":"2024","unstructured":"Thomas Fel, R\u00e9mi Cad\u00e8ne, Mathieu Chalvidal, Matthieu Cord, David Vigouroux, and Thomas Serre. 2024. Look at the variance! efficient black-box explanations with sobol-based sensitivity analysis. In Proceedings of the 35th International Conference on Neural Information Processing Systems(NIPS \u201921). Curran Associates Inc., Red Hook, NY, USA, Article 1991, 10\u00a0pages."},{"key":"e_1_3_3_1_19_2","unstructured":"FICO. 2022. Explainable Machine Learning Challenge. https:\/\/community.fico.com\/s\/explainable-machine-learning-challenge?tabset-158d9=3."},{"key":"e_1_3_3_1_20_2","doi-asserted-by":"crossref","unstructured":"Evelyn Fix and J.\u00a0L. Hodges. 1989. Discriminatory Analysis. Nonparametric Discrimination: Consistency Properties. International Statistical Review \/ Revue Internationale de Statistique 57 3 (1989) 238\u2013247. http:\/\/www.jstor.org\/stable\/1403797","DOI":"10.2307\/1403797"},{"key":"e_1_3_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00304"},{"key":"e_1_3_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.371"},{"key":"e_1_3_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3630106.3658537"},{"key":"e_1_3_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSAA.2018.00018"},{"key":"e_1_3_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW50498.2020.00014"},{"key":"e_1_3_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.491"},{"key":"e_1_3_3_1_27_2","unstructured":"Dan Hendrycks Collin Burns Steven Basart Andy Zou Mantas Mazeika Dawn Song and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. Proceedings of the International Conference on Learning Representations (ICLR) \u00a0 \u00a0 (2021) \u00a0."},{"key":"e_1_3_3_1_28_2","unstructured":"Hans Hofmann. 1994. Statlog (German Credit Data). UCI Machine Learning Repository. DOI: https:\/\/doi.org\/10.24432\/C5NC77."},{"key":"e_1_3_3_1_29_2","unstructured":"Sara Hooker Dumitru Erhan Pieter jan Kindermans and Been Kim. 2018. Evaluating Feature Importance Estimates. arXiv \u00a0 \u00a0 (2018) \u00a0. https:\/\/arxiv.org\/pdf\/1806.10758.pdf"},{"key":"e_1_3_3_1_30_2","doi-asserted-by":"publisher","unstructured":"Maksims Ivanovs Roberts Kadikis and Kaspars Ozols. 2021. Perturbation-based methods for explaining deep neural networks: A survey. Pattern Recogn. Lett. 150 C (Oct. 2021) 228\u2013234. https:\/\/doi.org\/10.1016\/j.patrec.2021.06.030","DOI":"10.1016\/j.patrec.2021.06.030"},{"key":"e_1_3_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00505"},{"key":"e_1_3_3_1_32_2","doi-asserted-by":"crossref","unstructured":"Satyapriya Krishna Tessa Han Alex Gu Steven Wu Shahin Jabbari and Himabindu Lakkaraju. 2024. The Disagreement Problem in Explainable Machine Learning: A Practitioner\u2019s Perspective. Transactions on Machine Learning Research \u00a0 \u00a0 (2024) \u00a0. https:\/\/openreview.net\/forum?id=jESY2WTZCe","DOI":"10.21203\/rs.3.rs-2963888\/v1"},{"key":"e_1_3_3_1_33_2","unstructured":"Jeff Larson Surya Mattu Lauren Kirchner and Julia Angwin. 2016. How We Analyzed the COMPAS Recidivism Algorithm. https:\/\/www.propublica.org\/article\/how-we-analyzed-the-compas-recidivism-algorithm"},{"key":"e_1_3_3_1_34_2","unstructured":"Q.\u00a0Vera Liao and Kush\u00a0R. Varshney. 2021. Human-Centered Explainable AI (XAI): From Algorithms to User Experiences. CoRR abs\/2110.10790 (2021). arXiv:https:\/\/arXiv.org\/abs\/2110.10790https:\/\/arxiv.org\/abs\/2110.10790"},{"key":"e_1_3_3_1_35_2","doi-asserted-by":"publisher","unstructured":"Zachary\u00a0C. Lipton. 2018. The Mythos of Model Interpretability: In machine learning the concept of interpretability is both important and slippery. Queue 16 3 (June 2018) 31\u201357. https:\/\/doi.org\/10.1145\/3236386.3241340","DOI":"10.1145\/3236386.3241340"},{"key":"e_1_3_3_1_36_2","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","volume":"1","author":"Liu Yang","year":"2021","unstructured":"Yang Liu, Sujay Khandagale, Sujay Khandagale, Colin White, and Willie Neiswanger. 2021. Synthetic Benchmarks for Scientific Research in Explainable Machine Learning. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, J.\u00a0Vanschoren and S.\u00a0Yeung (Eds.), Vol.\u00a01. \u00a0, \u00a0. https:\/\/datasets-benchmarks-proceedings.neurips.cc\/paper_files\/paper\/2021\/file\/c16a5320fa475530d9583c34fd356ef5-Paper-round2.pdf"},{"key":"e_1_3_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.5555\/3295222.3295230"},{"key":"e_1_3_3_1_38_2","series-title":"Proceedings of Machine Learning Research","first-page":"6765","volume-title":"Proceedings of the 37th International Conference on Machine Learning","volume":"119","author":"Marx Charles","year":"2020","unstructured":"Charles Marx, Flavio Calmon, and Berk Ustun. 2020. Predictive Multiplicity in Classification. In Proceedings of the 37th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol.\u00a0119), Hal\u00a0Daum\u00e9 III and Aarti Singh (Eds.). PMLR, \u00a0, 6765\u20136774. https:\/\/proceedings.mlr.press\/v119\/marx20a.html"},{"key":"e_1_3_3_1_39_2","doi-asserted-by":"publisher","unstructured":"Chuizheng Meng Loc Trinh Nan Xu James Enouen and Yan Liu. 2022. Interpretability and fairness evaluation of deep learning models on MIMIC-IV dataset. Scientific Reports 12 1 (03 May 2022) 7166. https:\/\/doi.org\/10.1038\/s41598-022-11012-2","DOI":"10.1038\/s41598-022-11012-2"},{"key":"e_1_3_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3630106.3659043"},{"key":"e_1_3_3_1_41_2","volume-title":"Interpretable Machine Learning (2 ed.)","author":"Molnar Christoph","year":"2022","unstructured":"Christoph Molnar. 2022. Interpretable Machine Learning (2 ed.). leanpub, \u00a0. https:\/\/christophm.github.io\/interpretable-ml-book"},{"key":"e_1_3_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-65965-3_28"},{"key":"e_1_3_3_1_43_2","doi-asserted-by":"publisher","unstructured":"Meike Nauta Jan Trienes Shreyasi Pathak Elisa Nguyen Michelle Peters Yasmin Schmitt J\u00f6rg Schl\u00f6tterer Maurice van Keulen and Christin Seifert. 2023. From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI. ACM Comput. Surv. 55 13s Article 295 (July 2023) 42\u00a0pages. https:\/\/doi.org\/10.1145\/3583558","DOI":"10.1145\/3583558"},{"key":"e_1_3_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380087"},{"key":"e_1_3_3_1_45_2","unstructured":"Vitali Petsiuk Abir Das and Kate Saenko. 2018. RISE: Randomized Input Sampling for Explanation of Black-box Models. CoRR abs\/1806.07421 (2018) \u00a0. arXiv:https:\/\/arXiv.org\/abs\/1806.07421http:\/\/arxiv.org\/abs\/1806.07421"},{"key":"e_1_3_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445315"},{"key":"e_1_3_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3485447.3512254"},{"key":"e_1_3_3_1_48_2","unstructured":"Michael Redmond. 2002. Communities and Crime. UCI Machine Learning Repository. DOI: https:\/\/doi.org\/10.24432\/C53W3X."},{"key":"e_1_3_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_3_3_1_50_2","doi-asserted-by":"crossref","unstructured":"Richard\u00a0D Riley and Gary\u00a0S Collins. 2023. Stability of clinical prediction models developed using statistical or machine learning methods. Biom J 65 8 (July 2023) e2200302.","DOI":"10.1002\/bimj.202200302"},{"key":"e_1_3_3_1_51_2","doi-asserted-by":"publisher","unstructured":"Wojciech Samek Alexander Binder Gr\u00e9goire Montavon Sebastian Lapuschkin and Klaus-Robert M\u00fcller. 2017. Evaluating the Visualization of What a Deep Neural Network Has Learned. IEEE Transactions on Neural Networks and Learning Systems 28 11 (2017) 2660\u20132673. https:\/\/doi.org\/10.1109\/TNNLS.2016.2599820","DOI":"10.1109\/TNNLS.2016.2599820"},{"key":"e_1_3_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.74"},{"key":"e_1_3_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.5555\/3305890.3306006"},{"key":"e_1_3_3_1_54_2","first-page":"\u00a0","volume-title":"2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings","author":"Simonyan Karen","year":"2014","unstructured":"Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). \u00a0, \u00a0, \u00a0. http:\/\/arxiv.org\/abs\/1312.6034"},{"key":"e_1_3_3_1_55_2","unstructured":"Dylan Slack Sophie Hilgard Emily Jia Sameer Singh and Himabindu Lakkaraju. 2019. How can we fool LIME and SHAP? Adversarial Attacks on Post hoc Explanation Methods. CoRR abs\/1911.02508 (2019) \u00a0. arXiv:https:\/\/arXiv.org\/abs\/1911.02508http:\/\/arxiv.org\/abs\/1911.02508"},{"key":"e_1_3_3_1_56_2","unstructured":"Daniel Smilkov Nikhil Thorat Been Kim Fernanda\u00a0B. Vi\u00e9gas and Martin Wattenberg. 2017. SmoothGrad: removing noise by adding noise. CoRR abs\/1706.03825 (2017) \u00a0. arXiv:https:\/\/arXiv.org\/abs\/1706.03825http:\/\/arxiv.org\/abs\/1706.03825"},{"key":"e_1_3_3_1_57_2","series-title":"Proceedings of Machine Learning Research","first-page":"3319","volume-title":"Proceedings of the 34th International Conference on Machine Learning","volume":"70","author":"Sundararajan Mukund","year":"2017","unstructured":"Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic Attribution for Deep Networks. In Proceedings of the 34th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol.\u00a070), Doina Precup and Yee\u00a0Whye Teh (Eds.). PMLR, \u00a0, 3319\u20133328. https:\/\/proceedings.mlr.press\/v70\/sundararajan17a.html"},{"key":"e_1_3_3_1_58_2","doi-asserted-by":"publisher","unstructured":"Jianming Zhang Sarah\u00a0Adel Bargal Zhe Lin Jonathan Brandt Xiaohui Shen and Stan Sclaroff. 2018. Top-Down Neural Attention by Excitation Backprop. Int. J. Comput. Vision 126 10 (oct 2018) 1084\u20131102. https:\/\/doi.org\/10.1007\/s11263-017-1059-x","DOI":"10.1007\/s11263-017-1059-x"},{"key":"e_1_3_3_1_59_2","doi-asserted-by":"publisher","unstructured":"Jianlong Zhou Amir\u00a0H. Gandomi Fang Chen and Andreas Holzinger. 2021. Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics. Electronics 10 5 (2021) \u00a0. https:\/\/doi.org\/10.3390\/electronics10050593","DOI":"10.3390\/electronics10050593"}],"event":{"name":"FAccT '25: The 2025 ACM Conference on Fairness, Accountability, and Transparency","location":"Athens Greece","acronym":"FAccT '25"},"container-title":["Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3715275.3732219","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,24]],"date-time":"2025-06-24T11:07:30Z","timestamp":1750763250000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3715275.3732219"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,23]]},"references-count":58,"alternative-id":["10.1145\/3715275.3732219","10.1145\/3715275"],"URL":"https:\/\/doi.org\/10.1145\/3715275.3732219","relation":{},"subject":[],"published":{"date-parts":[[2025,6,23]]},"assertion":[{"value":"2025-06-23","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}