{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T16:24:30Z","timestamp":1778603070104,"version":"3.51.4"},"reference-count":36,"publisher":"Association for Computing Machinery (ACM)","issue":"1","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGKDD Explor. Newsl."],"published-print":{"date-parts":[[2025,7,7]]},"abstract":"<jats:p>Pre-trained language models (PLMs) have achieved impressive results on various natural language processing tasks. However, recent research has revealed that these models often rely on superficial features and shortcuts instead of developing a genuine understanding of language, especially for natural language understanding (NLU) tasks. Consequently, the models struggle to generalize to out-of-domain data. In this work, we propose Divergence Based Regularization (DBR) to mitigate this shortcut learning behavior. Our method measures the divergence between the output distributions for original examples and examples where shortcut tokens have been masked. This process prevents the model's predictions from being overly influenced by shortcut features or biases. We evaluate our model on three NLU tasks and find that it improves out-of-domain performance with little loss of in-domain accuracy. Our results demonstrate that reducing the reliance on shortcuts and superficial features can enhance the generalization ability of large pretrained language models.<\/jats:p>","DOI":"10.1145\/3748239.3748241","type":"journal-article","created":{"date-parts":[[2025,7,7]],"date-time":"2025-07-07T19:19:42Z","timestamp":1751915982000},"page":"1-9","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["DBR: Divergence-Based Regularization for Debiasing Natural Language Understanding Models"],"prefix":"10.1145","volume":"27","author":[{"given":"Zihao","family":"Li","sequence":"first","affiliation":[{"name":"New Jersey Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ruixiang","family":"Tang","sequence":"additional","affiliation":[{"name":"Rutgers University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lu","family":"Cheng","sequence":"additional","affiliation":[{"name":"University of Illinois Chicago"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shuaiqiang","family":"Wang","sequence":"additional","affiliation":[{"name":"Baidu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dawei","family":"Yin","sequence":"additional","affiliation":[{"name":"Baidu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mengnan","family":"Du","sequence":"additional","affiliation":[{"name":"New Jersey Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,7,7]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1418"},{"key":"e_1_2_1_2_1","volume-title":"International Conference on Learning Representations (ICLR)","author":"Clark K.","year":"2020","unstructured":"K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. International Conference on Learning Representations (ICLR), 2020."},{"key":"e_1_2_1_3_1","volume-title":"North American Chapter of the Association for Computational Linguistics (NAACL)","author":"Devlin J.","year":"2019","unstructured":"J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. North American Chapter of the Association for Computational Linguistics (NAACL), 2019."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2024.3360306"},{"key":"e_1_2_1_5_1","volume-title":"Communications of the ACM (CACM)","author":"Du M.","year":"2023","unstructured":"M. Du, F. He, N. Zou, D. Tao, and X. Hu. Shortcut learning of large language models in natural language understanding. Communications of the ACM (CACM), 2023."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.71"},{"key":"e_1_2_1_7_1","first-page":"31","volume-title":"International symposium onInformation theory, 2004. ISIT 2004. Proceedings.","author":"Fuglede B.","unstructured":"B. Fuglede and F. Topsoe. Jensen-shannon divergence and hilbert space embedding. In International symposium onInformation theory, 2004. ISIT 2004. Proceedings., page 31. IEEE, 2004."},{"issue":"11","key":"e_1_2_1_8_1","first-page":"665","volume":"2","author":"Geirhos R.","year":"2020","unstructured":"R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665--673, 2020.","journal-title":"Shortcut learning in deep neural networks. Nature Machine Intelligence"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.72"},{"key":"e_1_2_1_10_1","volume-title":"Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.02324","author":"Gururangan S.","year":"2018","unstructured":"S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. R. Bowman, and N. A. Smith. Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.02324, 2018."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-6115"},{"key":"e_1_2_1_12_1","volume-title":"LREC-COLING 2024","author":"He Z.","year":"2024","unstructured":"Z. He, H. Deng, H. Zhao, N. Liu, and M. Du. Mitigating shortcuts in language models with soft label encoding. LREC-COLING 2024, 2024."},{"key":"e_1_2_1_13_1","unstructured":"H. Huang T. Zhou G. Long T. Shen J. Jiang and C. Zhang. Biaspad: A bias-progressive auto-debiasing framework."},{"key":"e_1_2_1_14_1","volume-title":"E. B. Hanna, F. Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088","author":"Jiang A. Q.","year":"2024","unstructured":"A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.324"},{"key":"e_1_2_1_16_1","volume-title":"Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692","author":"Liu Y.","year":"2019","unstructured":"Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-acl.170"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v37i11.26567"},{"key":"e_1_2_1_19_1","volume-title":"Endto- end bias mitigation by modelling biases in corpora. arXiv preprint arXiv:1909.06321","author":"Mahabadi R. K.","year":"2019","unstructured":"R. K. Mahabadi, Y. Belinkov, and J. Henderson. Endto- end bias mitigation by modelling biases in corpora. arXiv preprint arXiv:1909.06321, 2019."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1334"},{"key":"e_1_2_1_21_1","volume-title":"Debiasing masks: A new framework for shortcut mitigation in nlu. arXiv preprint arXiv:2210.16079","author":"Meissner J. M.","year":"2022","unstructured":"J. M. Meissner, S. Sugawara, and A. Aizawa. Debiasing masks: A new framework for shortcut mitigation in nlu. arXiv preprint arXiv:2210.16079, 2022."},{"key":"e_1_2_1_22_1","volume-title":"Hypothesis only baselines in natural language inference. arXiv preprint arXiv:1805.01042","author":"Poliak A.","year":"2018","unstructured":"A. Poliak, J. Naradowsky, A. Haldar, R. Rudinger, and B. Van Durme. Hypothesis only baselines in natural language inference. arXiv preprint arXiv:1805.01042, 2018."},{"key":"e_1_2_1_23_1","volume-title":"International Conference on Learning Representations (ICLR)","author":"Sanh V.","year":"2021","unstructured":"V. Sanh, T. Wolf, Y. Belinkov, and A. M. Rush. Learning from others' mistakes: Avoiding dataset biases without modeling them. International Conference on Learning Representations (ICLR), 2021."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1341"},{"key":"e_1_2_1_25_1","first-page":"9109","volume-title":"International Conference on Machine Learning","author":"Srivastava M.","year":"2020","unstructured":"M. Srivastava, T. Hashimoto, and P. Liang. Robustness to spurious correlations via human annotations. In International Conference on Machine Learning, pages 9109--9119. PMLR, 2020."},{"key":"e_1_2_1_26_1","volume-title":"International Conference on Machine Learning (ICML)","author":"Sundararajan M.","year":"2017","unstructured":"M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. International Conference on Machine Learning (ICML), 2017."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1074"},{"key":"e_1_2_1_28_1","volume-title":"Llama 2: Open foundation and finetuned chat models. arXiv preprint arXiv:2307.09288","author":"Touvron H.","year":"2023","unstructured":"H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and finetuned chat models. arXiv preprint arXiv:2307.09288, 2023."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.770"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.770"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.613"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.findings-naacl.130"},{"key":"e_1_2_1_33_1","volume-title":"Identifying spurious correlations for robust text classification. arXiv preprint arXiv:2010.02458","author":"Wang Z.","year":"2020","unstructured":"Z. Wang and A. Culotta. Identifying spurious correlations for robust text classification. arXiv preprint arXiv:2010.02458, 2020."},{"key":"e_1_2_1_34_1","volume-title":"Towards real world debiasing: A fine-grained analysis on spurious correlation. arXiv preprint arXiv:2405.15240","author":"Wang Z.","year":"2024","unstructured":"Z. Wang, P. Kuang, Z. Chu, J. Wang, and K. Ren. Towards real world debiasing: A fine-grained analysis on spurious correlation. arXiv preprint arXiv:2405.15240, 2024."},{"key":"e_1_2_1_35_1","volume-title":"North American Chapter of the Association for Computational Linguistics (NAACL)","author":"Williams A.","year":"2018","unstructured":"A. Williams, N. Nangia, and S. R. Bowman. A broadcoverage challenge corpus for sentence understanding through inference. North American Chapter of the Association for Computational Linguistics (NAACL), 2018."},{"key":"e_1_2_1_36_1","volume-title":"Paws: Paraphrase adversaries from word scrambling. arXiv preprint arXiv:1904.01130","author":"Zhang Y.","year":"2019","unstructured":"Y. Zhang, J. Baldridge, and L. He. Paws: Paraphrase adversaries from word scrambling. arXiv preprint arXiv:1904.01130, 2019."}],"container-title":["ACM SIGKDD Explorations Newsletter"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3748239.3748241","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,7]],"date-time":"2025-07-07T19:20:54Z","timestamp":1751916054000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3748239.3748241"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,7]]},"references-count":36,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,7,7]]}},"alternative-id":["10.1145\/3748239.3748241"],"URL":"https:\/\/doi.org\/10.1145\/3748239.3748241","relation":{},"ISSN":["1931-0145","1931-0153"],"issn-type":[{"value":"1931-0145","type":"print"},{"value":"1931-0153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,7]]},"assertion":[{"value":"2025-07-07","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}