{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:10:15Z","timestamp":1760242215669,"version":"build-2065373602"},"publisher-location":"Cham","reference-count":18,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783032083265","type":"print"},{"value":"9783032083272","type":"electronic"}],"license":[{"start":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T00:00:00Z","timestamp":1760227200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T00:00:00Z","timestamp":1760227200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>The explanations of large language models (e.g., where each word is assigned a relevance score) have recently been shown to be sensitive to the randomness used during model training, creating a need to evaluate this sensitivity. While simple visualization tools such as box plots can provide a qualitative characterization, exploring the design space of the parameters influencing the explanation\u2019s sensitivity to the training randomness may benefit from a more quantitative approach. First attempts in this direction explored simple (word-level univariate, first-order) explanations and proposed tentative information theoretic metrics such as the explanation\u2019s signal, noise and Signal-to-Noise Ratio (SNR). They left the suitability of such metrics as an open question, which we tackle in this work. For this purpose, we start by identifying corner cases where they appear unable to capture intuitively desirable features of explanations corresponding to a different training randomness. Namely, the SNR does not reflect well the relative differences of relevance (between words). We next put forward that the correlation with a mean explanation provides a better treatment of these corner cases, at the cost of being unable to reflect absolute differences of relevance (for single words). We then discuss how to turn these observations into a consolidated approach\u00a0for analyzing the explanations\u2019 sensitivity to the training randomness. While there is no silver bullet that perfectly deals with the full complexity of this sensitivity problem, we argue that design space exploration with the correlation metric and individual model analysis with box plots provides a good tradeoff. Besides, we put forward additional desirable features of the correlation metric (e.g., unbiased estimation thanks to cross-validation and simple confidence intervals).<\/jats:p>","DOI":"10.1007\/978-3-032-08327-2_15","type":"book-chapter","created":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T18:06:44Z","timestamp":1760206004000},"page":"310-323","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Consolidating Explanation Stability Metrics"],"prefix":"10.1007","author":[{"given":"Jeremie","family":"Bogaert","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Antonin","family":"Descampe","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fran\u00e7ois-Xavier","family":"Standaert","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2025,10,12]]},"reference":[{"key":"15_CR1","doi-asserted-by":"crossref","unstructured":"Acheampong, F.A., Nunoo-Mensah, H., Chen, W.: Transformer models for text-based emotion detection: a review of BERT-based approaches. Artif. Intell. Rev. 54(8), 5789\u20135829 (2021)","DOI":"10.1007\/s10462-021-09958-2"},{"key":"15_CR2","doi-asserted-by":"crossref","unstructured":"Artstein, R.: Inter-Annotator Agreement. Handbook of linguistic annotation, pp. 297\u2013313 (2017)","DOI":"10.1007\/978-94-024-0881-2_11"},{"key":"15_CR3","unstructured":"Bogaert, J., et al.: Explanation sensitivity to the randomness of large language models: the case of journalistic text classification. CoRR, abs\/2410.05085 (2024)"},{"key":"15_CR4","unstructured":"Bogaert, J., Escouflaire, L., de Marneffe, M., Descampe, A., Standaert, F., Fairon, C.: TIPECS: a corpus cleaning method using machine learning and qualitative analysis. In: International Conference on Corpus Linguistics (JLC) (2023)"},{"key":"15_CR5","unstructured":"Bogaert, J., Standaert, F.: A question on the explainability of large language models and the word-level univariate first-order plausibility assumption. Responsible Language Models (ReLM), p. 7 (2024)"},{"key":"15_CR6","unstructured":"Bogaert, J., de Marneffe, M., Descampe, A., Escouflaire, L., Fairon, C., Standaert, F.: Sensibilit\u00e9 des Explications \u00e0 l\u2019Al\u00e9a des Grands Mod\u00e8les de Langage: le Cas de la Classification de Textes Journalistiques, TAL (Traitement Automatique des Langues), vol. 64, no. 3, pp. 15\u201340 (2024)"},{"key":"15_CR7","doi-asserted-by":"crossref","unstructured":"Chefer, H., Gur, S., Wolf, L.: Transformer interpretability beyond attention visualization. In: CVPR, pp. 782\u2013791. Computer Vision Foundation \/ IEEE (2021)","DOI":"10.1109\/CVPR46437.2021.00084"},{"key":"15_CR8","unstructured":"Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT (1), pp. 4171\u20134186. Association for Computational Linguistics (2019)"},{"key":"15_CR9","unstructured":"Brown, T.B., et\u00a0al.: Language models are few-shot learners. In: NeurIPS (2020)"},{"issue":"1","key":"15_CR10","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1080\/19312450709336664","volume":"1","author":"AF Hayes","year":"2007","unstructured":"Hayes, A.F., Krippendorff, K.: Answering the call for a standard reliability measure for coding data. Commun. Methods Meas. 1(1), 77\u201389 (2007)","journal-title":"Commun. Methods Meas."},{"key":"15_CR11","unstructured":"Herman, B.: The promise and peril of human evaluation for model interpretability. CoRR, abs\/1711.07414 (2017)"},{"key":"15_CR12","doi-asserted-by":"crossref","unstructured":"Jacovi, A., Goldberg, Y.: Towards faithfully interpretable nlp systems: how should we define and evaluate faithfulness? In: ACL, pp. 4198\u20134205. Association for Computational Linguistics (2020)","DOI":"10.18653\/v1\/2020.acl-main.386"},{"key":"15_CR13","unstructured":"Lehmann, E.L., Romano, J.P.: Testing Statistical Hypotheses, Third Edition. Springer texts in statistics. Springer (2008)"},{"key":"15_CR14","unstructured":"Lyu, Q., Apidianaki, M., Callison-Burch, C.: Towards faithful model explanation in NLP: a survey. CoRR, abs\/2209.11326 (2022)"},{"key":"15_CR15","doi-asserted-by":"crossref","unstructured":"Martin, L., et al.: CamemBERT: a tasty french language model. CoRR, abs\/1911.03894 (2019)","DOI":"10.18653\/v1\/2020.acl-main.645"},{"key":"15_CR16","doi-asserted-by":"crossref","unstructured":"Ribeiro, M., Singh, S., Guestrin, C.: \"Why Should I Trust You?\": explaining the predictions of any classifier. In: KDD, pp. 1135\u20131144. ACM (2016)","DOI":"10.1145\/2939672.2939778"},{"key":"15_CR17","doi-asserted-by":"crossref","unstructured":"Clayton Silver, N., Dunlap, W.P.: Averaging correlation coefficients: should Fisher\u2019s Z transformation be used? J. Appl. Psychol. 72(1), 146 (1987)","DOI":"10.1037\/\/0021-9010.72.1.146"},{"key":"15_CR18","unstructured":"Wu, Z., Ong, D.C.: On explaining your explanations of BERT: an empirical study with sequence classification. CoRR, abs\/2101.00196 (2021)"}],"container-title":["Communications in Computer and Information Science","Explainable Artificial Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-032-08327-2_15","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T18:06:48Z","timestamp":1760206008000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-032-08327-2_15"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,12]]},"ISBN":["9783032083265","9783032083272"],"references-count":18,"URL":"https:\/\/doi.org\/10.1007\/978-3-032-08327-2_15","relation":{},"ISSN":["1865-0929","1865-0937"],"issn-type":[{"value":"1865-0929","type":"print"},{"value":"1865-0937","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,12]]},"assertion":[{"value":"12 October 2025","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"xAI","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"World Conference on Explainable Artificial Intelligence","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Istanbul","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"T\u00fcrkiye","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2025","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"9 July 2025","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"11 July 2025","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"3","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"xai2025","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/xaiworldconference.com\/2025\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}