{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,12]],"date-time":"2026-09-12T01:40:55Z","timestamp":1789177255113,"version":"build-2803163510"},"publisher-location":"New York, NY, USA","reference-count":54,"publisher":"ACM","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,6,23]]},"DOI":"10.1145\/3696630.3728522","type":"proceedings-article","created":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T19:08:09Z","timestamp":1753729689000},"page":"676-680","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Can Hessian-Based Insights Support Fault Diagnosis in Attention-based Models?"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6024-3018","authenticated-orcid":false,"given":"Sigma","family":"Jahan","sequence":"first","affiliation":[{"name":"Dalhousie University, Halifax, Nova Scotia, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3821-5990","authenticated-orcid":false,"given":"Mohammad Masudur","family":"Rahman","sequence":"additional","affiliation":[{"name":"Dalhousie University, Halifax, Nova Scotia, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,7,28]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies. 4171\u20134186","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies. 4171\u20134186."},{"key":"e_1_3_2_1_2_1","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez \u0141ukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3285029","article-title":"Deep learning based recommender system: A survey and new perspectives","volume":"52","author":"Zhang Shuai","year":"2019","unstructured":"Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep learning based recommender system: A survey and new perspectives. Comput. Surveys 52 (2019), 1\u201338.","journal-title":"Comput. Surveys"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3505243","article-title":"A survey on deep learning for software engineering","volume":"54","author":"Yang Yanming","year":"2022","unstructured":"Yanming Yang, Xin Xia, David Lo, and John Grundy. 2022. A survey on deep learning for software engineering. Comput. Surveys 54 (2022), 1\u201373.","journal-title":"Comput. Surveys"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"crossref","first-page":"208","DOI":"10.1016\/j.aiopen.2023.08.012","article-title":"GPT understands, too","volume":"5","author":"Liu Xiao","year":"2024","unstructured":"Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2024. GPT understands, too. AI Open 5 (2024), 208\u2013215.","journal-title":"AI Open"},{"key":"e_1_3_2_1_6_1","volume-title":"Deep learning","author":"Goodfellow Ian","unstructured":"Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. 2016. Deep learning. Vol. 1. MIT press Cambridge."},{"key":"e_1_3_2_1_7_1","unstructured":"Ian Goodfellow and Oriol Vinyals. 2015. Qualitatively characterizing neural network optimization problems. In ICLR."},{"key":"e_1_3_2_1_8_1","unstructured":"Hao Li Zheng Xu Gavin Taylor Christoph Studer and Tom Goldstein. 2018. Visualizing the loss landscape of neural nets. In NeurIPS."},{"key":"e_1_3_2_1_9_1","unstructured":"Charles Guille-Escuret Hiroki Naganuma Kilian Fatras and Ioannis Mitliagkas. 2024. No wrong turns: the simple geometry of neural networks optimization paths. In ICML."},{"key":"e_1_3_2_1_10_1","unstructured":"Fengxiang He Bohan Wang and Dacheng Tao. 2020. Piecewise linear activations substantially shape the loss surfaces of neural networks. In ICLR."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"crossref","first-page":"23103","DOI":"10.1007\/s00521-023-08957-4","article-title":"Deep learning: systematic review, models, challenges, and research directions","volume":"35","author":"Khoei Tala Talaei","year":"2023","unstructured":"Tala Talaei Khoei, Hadjar Ould Slimane, and Naima Kaabouch. 2023. Deep learning: systematic review, models, challenges, and research directions. Neural Computing and Applications 35 (2023), 23103\u201323124.","journal-title":"Neural Computing and Applications"},{"key":"e_1_3_2_1_12_1","volume-title":"Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al.","author":"Dehghani Mostafa","year":"2023","unstructured":"Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. 2023. Scaling vision transformers to 22 billion parameters. In ICML. 7480\u20137512."},{"key":"e_1_3_2_1_13_1","unstructured":"Mitchell Wortsman Peter J Liu Lechao Xiao Katie E Everett Alexander A Alemi Ben Adlam John D Co-Reyes Izzeddin Gur Abhishek Kumar Roman Novak Jeffrey Pennington Jascha Sohl-Dickstein Kelvin Xu Jaehoon Lee Justin Gilmer and Simon Kornblith. 2024. Small-scale proxies for large-scale Transformer training instabilities. In ICLR."},{"key":"e_1_3_2_1_14_1","unstructured":"Shuangfei Zhai Tatiana Likhomanenko Etai Littwin Dan Busbridge Jason Ramapuram Yizhe Zhang Jiatao Gu and Joshua M Susskind. 2023. Stabilizing transformer training by preventing attention entropy collapse. In ICML. 40770\u201340803."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","unstructured":"Jared Kaplan Sam McCandlish Tom Henighan Tom B. Brown Benjamin Chess Rewon Child Scott Gray Alec Radford Jeffrey Wu and Dario Amodei. 2020. Scaling Laws for Neural Language Models. (2020). arXiv:arXiv:2001.08361 10.48550\/arXiv.2001.08361","DOI":"10.48550\/arXiv.2001.08361"},{"key":"e_1_3_2_1_16_1","unstructured":"Yihe Dong Jean-Baptiste Cordonnier and Andreas Loukas. 2021. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In ICML. 2793\u20132803."},{"key":"e_1_3_2_1_17_1","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 3543\u20133556","author":"Jain Sarthak","unstructured":"Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 3543\u20133556."},{"key":"e_1_3_2_1_18_1","volume-title":"Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 25\u201334","author":"Bai Bing","year":"2021","unstructured":"Bing Bai, Jian Liang, Guanhua Zhang, Hao Li, Kun Bai, and Fei Wang. 2021. Why Attentions May Not Be Interpretable?. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 25\u201334."},{"key":"e_1_3_2_1_19_1","volume-title":"Deeplocalize: Fault localization for deep neural networks. In ICSE. 251\u2013262.","author":"Wardat Mohammad","year":"2021","unstructured":"Mohammad Wardat, Wei Le, and Hridesh Rajan. 2021. Deeplocalize: Fault localization for deep neural networks. In ICSE. 251\u2013262."},{"key":"e_1_3_2_1_20_1","volume-title":"Autotrainer: An automatic dnn training problem detection and repair system. In ICSE. 359\u2013371.","author":"Zhang Xiaoyu","year":"2021","unstructured":"Xiaoyu Zhang, Juan Zhai, Shiqing Ma, and Chao Shen. 2021. Autotrainer: An automatic dnn training problem detection and repair system. In ICSE. 359\u2013371."},{"key":"e_1_3_2_1_21_1","volume-title":"Deepfd: Automated fault diagnosis and localization for deep learning programs. In ICSE. 573\u2013585.","author":"Cao Jialun","year":"2022","unstructured":"Jialun Cao, Meiziniu Li, Xiao Chen, Ming Wen, Yongqiang Tian, Bo Wu, and Shing-Chi Cheung. 2022. Deepfd: Automated fault diagnosis and localization for deep learning programs. In ICSE. 573\u2013585."},{"key":"e_1_3_2_1_22_1","volume-title":"CHI conference on human factors in computing systems. 1\u201316","author":"Schoop Eldon","year":"2021","unstructured":"Eldon Schoop, Forrest Huang, and Bjoern Hartmann. 2021. Umlaut: Debugging deep learning programs using program structure and model behavior. In CHI conference on human factors in computing systems. 1\u201316."},{"key":"e_1_3_2_1_23_1","unstructured":"Ian J. Goodfellow Jonathon Shlens and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In ICLR."},{"key":"e_1_3_2_1_24_1","unstructured":"Zhewei Yao Amir Gholami Qi Lei Kurt Keutzer and Michael W Mahoney. 2018. Hessian-based analysis of large batch training and robustness to adversaries. In NeurIPS."},{"key":"e_1_3_2_1_25_1","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2931\u20132951","author":"Serrano Sofia","year":"2019","unstructured":"Sofia Serrano and Noah A Smith. 2019. Is attention interpretable?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2931\u20132951."},{"key":"e_1_3_2_1_26_1","unstructured":"Sidak Pal Singh Gregor Bachmann and Thomas Hofmann. 2021. Analytic insights into structure and rank of neural network hessian maps. In NeurIPS."},{"key":"e_1_3_2_1_27_1","unstructured":"Behrooz Ghorbani Shankar Krishnan and Ying Xiao. 2019. An investigation into neural net optimization via hessian eigenvalue density. In ICML. 2232\u20132241."},{"key":"e_1_3_2_1_28_1","unstructured":"Yiding Jiang* Behnam Neyshabur* Hossein Mobahi Dilip Krishnan and Samy Bengio. 2020. Fantastic Generalization Measures and Where to Find Them. In ICLR."},{"key":"e_1_3_2_1_29_1","first-page":"735","article-title":"Deep learning via hessian-free optimization","volume":"27","author":"James Martens","year":"2010","unstructured":"James Martens et al. 2010. Deep learning via hessian-free optimization.. In ICML, Vol. 27. 735\u2013742.","journal-title":"ICML"},{"key":"e_1_3_2_1_30_1","volume-title":"Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies. 1480\u20131489","author":"Yang Zichao","year":"2016","unstructured":"Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies. 1480\u20131489."},{"key":"e_1_3_2_1_31_1","unstructured":"S. G. R. Vinod. 2023. A PyTorch Tutorial to Text Classification. https:\/\/github.com\/sgrvinod\/a-PyTorch-Tutorial-to-Text-Classification. Accessed: 2025-04-21."},{"key":"e_1_3_2_1_32_1","unstructured":"Hugging Face. 2024. Transformers Library. https:\/\/github.com\/huggingface\/transformers. Accessed: 2025-04-21."},{"key":"e_1_3_2_1_33_1","unstructured":"Kiet Nguyen. 2023. HMDB51 Recognition. https:\/\/github.com\/kietngt00\/hmdb51-recognition. Accessed: 2025-04-21."},{"key":"e_1_3_2_1_34_1","unstructured":"Nicolo Cosimo Albanese. 2023. Implementing the Steepest Descent Algorithm in Python from Scratch. https:\/\/towardsdatascience.com\/implementing-the-steepest-descent-algorithm-in-python-from-scratch-d32da2906fe2. Towards Data Science Accessed: 2025-04-21."},{"key":"e_1_3_2_1_35_1","volume-title":"Autograd: Automatic Differentiation. https:\/\/pytorch.org\/tutorials\/beginner\/blitz\/autograd_tutorial. Accessed: 2025-04-21.","author":"Team PyTorch","year":"2025","unstructured":"PyTorch Team. 2025. Autograd: Automatic Differentiation. https:\/\/pytorch.org\/tutorials\/beginner\/blitz\/autograd_tutorial. Accessed: 2025-04-21."},{"key":"e_1_3_2_1_36_1","unstructured":"James Martens and Ilya Sutskever. 2011. Learning recurrent neural networks with hessian-free optimization. In ICML. 1033\u20131040."},{"key":"e_1_3_2_1_37_1","volume-title":"Annual Conference on Learning Theory.","author":"Lei Yunwen","year":"2023","unstructured":"Yunwen Lei. 2023. Stability and generalization of stochastic optimization with nonconvex and nonsmooth problems. In Annual Conference on Learning Theory."},{"key":"e_1_3_2_1_38_1","unstructured":"Stanislaw Jastrzebski Maciej Szymczak Stanislav Fort Devansh Arpit Jacek Tabor Kyunghyun Cho and Krzysztof Geras. 2020. The Break-Even Point on Optimization Trajectories of Deep Neural Networks. In ICLR."},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","unstructured":"Borjan Geshkovski Hugo Koubbi Yury Polyanskiy and Philippe Rigollet. 2024. Dynamic Metastability in the Self-Attention Model. (2024). arXiv:arXiv:2410.06833 10.48550\/arXiv.2410.06833","DOI":"10.48550\/arXiv.2410.06833"},{"key":"e_1_3_2_1_40_1","unstructured":"Yann N. Dauphin Razvan Pascanu Caglar Gulcehre Kyunghyun Cho Surya Ganguli and Yoshua Bengio. 2014. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In NeurIPS."},{"key":"e_1_3_2_1_41_1","unstructured":"Jingjing Xu Xu Sun Zhiyuan Zhang Guangxiang Zhao and Junyang Lin. 2019. Understanding and improving layer normalization."},{"key":"e_1_3_2_1_42_1","unstructured":"Suraj Srinivas Kyle Matoba Himabindu Lakkaraju and Fran\u00e7ois Fleuret. 2022. Efficient Training of Low-Curvature Neural Networks. In NeurIPS."},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"crossref","unstructured":"Marco Ancona Enea Ceolini Cengiz \u00d6ztireli and Markus Gross. 2018. Towards better understanding of gradient-based attribution methods for Deep Neural Networks. In ICLR.","DOI":"10.1007\/978-3-030-28954-6_9"},{"key":"e_1_3_2_1_44_1","unstructured":"Julius Adebayo Justin Gilmer Michael Muelly Ian Goodfellow Moritz Hardt and Been Kim. 2018. Sanity checks for saliency maps. In NeurIPS."},{"key":"e_1_3_2_1_45_1","volume-title":"International Conference on Big Data. 581\u2013590","author":"Yao Zhewei","year":"2020","unstructured":"Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. 2020. Pyhessian: Neural networks through the lens of the hessian. In International Conference on Big Data. 581\u2013590."},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1162\/neco.1992.4.2.141","article-title":"First-and second-order methods for learning: between steepest descent and Newton's method","volume":"4","author":"Battiti Roberto","year":"1992","unstructured":"Roberto Battiti. 1992. First-and second-order methods for learning: between steepest descent and Newton's method. Neural Computation 4 (1992), 141\u2013166.","journal-title":"Neural Computation"},{"key":"e_1_3_2_1_47_1","volume-title":"AAAI Conference on Artificial Intelligence","volume":"33","author":"Ghorbani Amirata","year":"2019","unstructured":"Amirata Ghorbani, Abubakar Abid, and James Zou. 2019. Interpretation of neural networks is fragile. In AAAI Conference on Artificial Intelligence, Vol. 33. 3681\u20133688."},{"key":"e_1_3_2_1_48_1","volume-title":"Conference on Computational Natural Language Learning.","author":"Madhyastha Pranava","year":"2019","unstructured":"Pranava Madhyastha and Rishabh Jain. 2019. On Model Stability as a Function of Random Seed. In Conference on Computational Natural Language Learning."},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"crossref","first-page":"1346","DOI":"10.1152\/jn.00127.2013","article-title":"How attention extracts objects from noise","volume":"110","author":"Pratte Michael S","year":"2013","unstructured":"Michael S Pratte, Sam Ling, Jascha D Swisher, and Frank Tong. 2013. How attention extracts objects from noise. Journal of Neurophysiology 110 (2013), 1346\u20131356.","journal-title":"Journal of Neurophysiology"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"crossref","first-page":"228","DOI":"10.1016\/j.patrec.2021.06.030","article-title":"Perturbation-based methods for explaining deep neural networks: A survey","volume":"150","author":"Ivanovs Maksims","year":"2021","unstructured":"Maksims Ivanovs, Roberts Kadikis, and Kaspars Ozols. 2021. Perturbation-based methods for explaining deep neural networks: A survey. Pattern Recognition Letters 150 (2021), 228\u2013234.","journal-title":"Pattern Recognition Letters"},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"crossref","first-page":"013295","DOI":"10.1103\/PhysRevResearch.6.013295","article-title":"Effects of noise on the overparametrization of quantum neural networks","volume":"6","author":"Garc\u00eda-Mart\u00edn Diego","year":"2024","unstructured":"Diego Garc\u00eda-Mart\u00edn, Mart\u00edn Larocca, and M. Cerezo. 2024. Effects of noise on the overparametrization of quantum neural networks. Physical Review Research 6 (2024), 013295.","journal-title":"Physical Review Research"},{"key":"e_1_3_2_1_52_1","volume-title":"Fynn Firouz Faber, and Dylan Richard Muir","author":"B\u00fcchel Julian","year":"2022","unstructured":"Julian B\u00fcchel, Fynn Firouz Faber, and Dylan Richard Muir. 2022. Network insensitivity to parameter noise via parameter attack during training. In ICLR."},{"key":"e_1_3_2_1_53_1","volume-title":"Nicolas Le Roux, and Pierre-Antoine Manzagol","author":"Alain Guillaume","year":"2019","unstructured":"Guillaume Alain, Nicolas Le Roux, and Pierre-Antoine Manzagol. 2019. Negative Eigenvalues of the Hessian in Deep Neural Networks. In ICLR."},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","unstructured":"Roger Grosse Juhan Bae Cem Anil Nelson Elhage Alex Tamkin Amirhossein Tajdini Benoit Steiner Dustin Li Esin Durmus Ethan Perez et al. 2023. Studying Large Language Model Generalization with Influence Functions. (2023). arXiv:arXiv:2308.03296 10.48550\/arXiv.2308.03296","DOI":"10.48550\/arXiv.2308.03296"}],"event":{"name":"FSE Companion '25: 33rd ACM International Conference on the Foundations of Software Engineering","location":"Clarion Hotel Trondheim Trondheim Norway","acronym":"FSE Companion '25","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3696630.3728522","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T19:12:30Z","timestamp":1753729950000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3696630.3728522"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,23]]},"references-count":54,"alternative-id":["10.1145\/3696630.3728522","10.1145\/3696630"],"URL":"https:\/\/doi.org\/10.1145\/3696630.3728522","relation":{},"subject":[],"published":{"date-parts":[[2025,6,23]]},"assertion":[{"value":"2025-07-28","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}