{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:46:47Z","timestamp":1760147207036,"version":"build-2065373602"},"reference-count":58,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2023,1,18]],"date-time":"2023-01-18T00:00:00Z","timestamp":1674000000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Guangdong R&amp;D key project of China","award":["2019B010155001"],"award-info":[{"award-number":["2019B010155001"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Learning invariant (causal) features for out-of-distribution (OOD) generalization have attracted extensive attention recently, and among the proposals, invariant risk minimization (IRM) is a notable solution. In spite of its theoretical promise for linear regression, the challenges of using IRM in linear classification problems remain. By introducing the information bottleneck (IB) principle into the learning of IRM, the IB-IRM approach has demonstrated its power to solve these challenges. In this paper, we further improve IB-IRM from two aspects. First, we show that the key assumption of support overlap of invariant features used in IB-IRM guarantees OOD generalization, and it is still possible to achieve the optimal solution without this assumption. Second, we illustrate two failure modes where IB-IRM (and IRM) could fail in learning the invariant features, and to address such failures, we propose a Counterfactual Supervision-based Information Bottleneck (CSIB) learning algorithm that recovers the invariant features. By requiring counterfactual inference, CSIB works even when accessing data from a single environment. Empirical experiments on several datasets verify our theoretical results.<\/jats:p>","DOI":"10.3390\/e25020193","type":"journal-article","created":{"date-parts":[[2023,1,19]],"date-time":"2023-01-19T02:51:37Z","timestamp":1674096697000},"page":"193","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Counterfactual Supervision-Based Information Bottleneck for Out-of-Distribution Generalization"],"prefix":"10.3390","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0211-5886","authenticated-orcid":false,"given":"Bin","family":"Deng","sequence":"first","affiliation":[{"name":"School of Electronic and Information Engineering, South China University of Technology, Guangzhou 510641, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2661-5700","authenticated-orcid":false,"given":"Kui","family":"Jia","sequence":"additional","affiliation":[{"name":"School of Electronic and Information Engineering, South China University of Technology, Guangzhou 510641, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,1,18]]},"reference":[{"key":"ref_1","unstructured":"Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2013). Intriguing properties of neural networks. arXiv."},{"key":"ref_2","unstructured":"Rosenfeld, A., Zemel, R., and Tsotsos, J.K. (2018). The elephant in the room. arXiv."},{"key":"ref_3","unstructured":"Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F.A., and Brendel, W. (2019, January 6\u20139). ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Nguyen, A., Yosinski, J., and Clune, J. (2015, January 7\u201312). Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. Proceedings of the Computer Vision and Pattern Recognition Conference, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298640"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S.R., and Smith, N.A. (2018). Annotation Artifacts in Natural Language Inference Data. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), Association for Computational Linguistics.","DOI":"10.18653\/v1\/N18-2017"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"665","DOI":"10.1038\/s42256-020-00257-z","article-title":"Shortcut learning in deep neural networks","volume":"2","author":"Geirhos","year":"2020","journal-title":"Nat. Mach. Intell."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Beery, S., Van Horn, G., and Perona, P. (2018, January 8\u201314). Recognition in terra incognita. Proceedings of the European Conference on Computer Vision, Munich, Germany.","DOI":"10.1007\/978-3-030-01270-0_28"},{"key":"ref_8","unstructured":"Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D. (2019). Invariant risk minimization. arXiv."},{"key":"ref_9","unstructured":"Krueger, D., Caballero, E., Jacobsen, J.H., Zhang, A., Binas, J., Zhang, D., Le Priol, R., and Courville, A. (2021, January 18\u201324). Out-of-distribution generalization via risk extrapolation (rex). Proceedings of the International Conference on Machine Learning, PMLR, Virtual."},{"key":"ref_10","unstructured":"Ahuja, K., Shanmugam, K., Varshney, K., and Dhurandhar, A. (2020, January 13\u201318). Invariant risk minimization games. Proceedings of the International Conference on Machine Learning, PMLR, Virtual."},{"key":"ref_11","unstructured":"Pezeshki, M., Kaba, O., Bengio, Y., Courville, A.C., Precup, D., and Lajoie, G. (2021, January 6\u201314). Gradient starvation: A learning proclivity in neural networks. Proceedings of the Neural Information Processing Systems, Virtual."},{"key":"ref_12","unstructured":"Ahuja, K., Caballero, E., Zhang, D., Gagnon-Audet, J.C., Bengio, Y., Mitliagkas, I., and Rish, I. (2021, January 6\u201314). Invariance principle meets information bottleneck for out-of-distribution generalization. Proceedings of the Neural Information Processing Systems, Virtual."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Pearl, J. (2009). Causality, Cambridge University Press.","DOI":"10.1017\/CBO9780511803161"},{"key":"ref_14","unstructured":"Peters, J., Janzing, D., and Sch\u00f6lkopf, B. (2017). Elements of Causal Inference: Foundations and Learning Algorithms, The MIT Press."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"947","DOI":"10.1111\/rssb.12167","article-title":"Causal inference by using invariant prediction: Identification and confidence intervals","volume":"78","author":"Peters","year":"2016","journal-title":"J. R. Stat. Soc. Ser. B"},{"key":"ref_16","unstructured":"Tishby, N. (1999, January 22\u201324). The information bottleneck method. Proceedings of the Annual Allerton Conference on Communications, Control and Computing, Monticello, IL, USA."},{"key":"ref_17","unstructured":"Aubin, B., S\u0142owik, A., Arjovsky, M., Bottou, L., and Lopez-Paz, D. (2021). Linear unit-tests for invariance discovery. arXiv."},{"key":"ref_18","first-page":"2822","article-title":"The implicit bias of gradient descent on separable data","volume":"19","author":"Soudry","year":"2018","journal-title":"J. Mach. Learn. Res."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Heinze-Deml, C., Peters, J., and Meinshausen, N. (2018). Invariant causal prediction for nonlinear models. arXiv.","DOI":"10.1515\/jci-2017-0016"},{"key":"ref_20","first-page":"1309","article-title":"Invariant models for causal transfer learning","volume":"19","author":"Turner","year":"2018","journal-title":"J. Mach. Learn. Res."},{"key":"ref_21","unstructured":"Rosenfeld, E., Ravikumar, P.K., and Risteski, A. (2021, January 3\u20137). The Risks of Invariant Risk Minimization. Proceedings of the International Conference on Learning Representations, Virtual."},{"key":"ref_22","unstructured":"Kamath, P., Tangella, A., Sutherland, D., and Srebro, N. (2021, January 13\u201315). Does invariant risk minimization capture invariance?. Proceedings of the International Conference on Artificial Intelligence and Statistics, PMLR, San Diego, CA, USA."},{"key":"ref_23","unstructured":"Lu, C., Wu, Y., Hern\u00e1ndez-Lobato, J.M., and Sch\u00f6lkopf, B. (2022, January 25\u201329). Invariant Causal Representation Learning for Out-of-Distribution Generalization. Proceedings of the International Conference on Learning Representations, Virtual."},{"key":"ref_24","unstructured":"Liu, C., Sun, X., Wang, J., Tang, H., Li, T., Qin, T., Chen, W., and Liu, T.Y. (2021, January 6\u201314). Learning causal semantic representation for out-of-distribution prediction. Proceedings of the Neural Information Processing Systems, Virtual."},{"key":"ref_25","unstructured":"Kingma, D.P., and Welling, M. (2013). Auto-encoding variational bayes. arXiv."},{"key":"ref_26","unstructured":"Rezende, D.J., Mohamed, S., and Wierstra, D. (2014, January 21\u201326). Stochastic backpropagation and approximate inference in deep generative models. Proceedings of the International Conference on Machine Learning, PMLR, Beijing, China."},{"key":"ref_27","unstructured":"Lu, C., Wu, Y., Hern\u00e1ndez-Lobato, J.M., and Sch\u00f6lkopf, B. (2021). Nonlinear invariant risk minimization: A causal approach. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1798","DOI":"10.1109\/TPAMI.2013.50","article-title":"Representation learning: A review and new perspectives","volume":"35","author":"Bengio","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"612","DOI":"10.1109\/JPROC.2021.3058954","article-title":"Toward causal representation learning","volume":"109","author":"Locatello","year":"2021","journal-title":"Proc. IEEE"},{"key":"ref_30","unstructured":"Namkoong, H., and Duchi, J.C. (2016, January 5\u201310). Stochastic gradient methods for distributionally robust optimization with f-divergences. Proceedings of the Neural Information processing Systems, Barcelona, Spain."},{"key":"ref_31","unstructured":"Sinha, A., Namkoong, H., Volpi, R., and Duchi, J. (2017). Certifying some distributional robustness with principled adversarial training. arXiv."},{"key":"ref_32","unstructured":"Lee, J., and Raginsky, M. (2018, January 3\u20138). Minimax statistical learning with wasserstein distances. Proceedings of the Neural Information Processing Systems, Montreal, Canada."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1378","DOI":"10.1214\/20-AOS2004","article-title":"Learning models with uniform performance via distributionally robust optimization","volume":"49","author":"Duchi","year":"2021","journal-title":"Ann. Stat."},{"key":"ref_34","first-page":"404","article-title":"Invariance, causality and robustness","volume":"35","year":"2020","journal-title":"Stat. Sci."},{"key":"ref_35","unstructured":"Blanchard, G., Lee, G., and Scott, C. (2011, January 12\u201315). Generalizing from several related classification tasks to a new unlabeled sample. Proceedings of the Neural Information Processing Systems, Granada, Spain."},{"key":"ref_36","unstructured":"Muandet, K., Balduzzi, D., and Sch\u00f6lkopf, B. (2013, January 16\u201321). Domain generalization via invariant feature representation. Proceedings of the International Conference on Machine Learning, PMLR, Atlanta, GA, USA."},{"key":"ref_37","unstructured":"Deshmukh, A.A., Lei, Y., Sharma, S., Dogan, U., Cutler, J.W., and Scott, C. (2019). A generalization error bound for multi-class domain generalization. arXiv."},{"key":"ref_38","unstructured":"Ye, H., Xie, C., Cai, T., Li, R., Li, Z., and Wang, L. (2021, January 6\u201314). Towards a Theoretical Framework of Out-of-Distribution Generalization. Proceedings of the Neural Information Processing Systems, Virtual."},{"key":"ref_39","unstructured":"Xie, C., Chen, F., Liu, Y., and Li, Z. (2020). Risk variance penalization: From distributional robustness to causality. arXiv."},{"key":"ref_40","unstructured":"Jin, W., Barzilay, R., and Jaakkola, T. (2020). Domain extrapolation via regret minimization. arXiv."},{"key":"ref_41","unstructured":"Mahajan, D., Tople, S., and Sharma, A. (2021, January 18\u201324). Domain generalization using causal matching. Proceedings of the International Conference on Machine Learning, PMLR, Virtual."},{"key":"ref_42","unstructured":"Bellot, A., and van der Schaar, M. (2020). Generalization and invariances in the presence of unobserved confounding. arXiv."},{"key":"ref_43","unstructured":"Li, B., Shen, Y., Wang, Y., Zhu, W., Reed, C.J., Zhang, J., Li, D., Keutzer, K., and Zhao, H. (March, January 22). Invariant information bottleneck for domain generalization. Proceedings of the Association for the Advancement of Artificial Intelligence, Virtual."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Alesiani, F., Yu, S., and Yu, X. (Knowl. Informat. Syst., 2022). Gated information bottleneck for generalization in sequential environments, Knowl. Informat. Syst., in press.","DOI":"10.1109\/ICDM51629.2021.00010"},{"key":"ref_45","unstructured":"Wang, H., Si, H., Li, B., and Zhao, H. (2022, January 17\u201323). Provable Domain Generalization via Invariant-Feature Subspace Recovery. Proceedings of the International Conference on Machine Learning, Baltimore, MD, USA."},{"key":"ref_46","unstructured":"Ganin, Y., and Lempitsky, V. (2015, January 6\u201311). Unsupervised domain adaptation by backpropagation. Proceedings of the International conference on machine learning, PMLR, Lille, France."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Li, Y., Tian, X., Gong, M., Liu, Y., Liu, T., Zhang, K., and Tao, D. (2018, January 8\u201314). Deep domain generalization via conditional invariant adversarial networks. Proceedings of the European Conference on Computer Vision, Munich, Germany.","DOI":"10.1007\/978-3-030-01267-0_38"},{"key":"ref_48","unstructured":"Zhao, S., Gong, M., Liu, T., Fu, H., and Tao, D. (2020, January 6\u201312). Domain generalization via entropy regularization. Proceedings of the Neural Information Processing Systems, Virtual."},{"key":"ref_49","unstructured":"Ben-David, S., Blitzer, J., Crammer, K., and Pereira, F. (2006, January 3\u20136). Analysis of representations for domain adaptation. Proceedings of the Neural Information Processing Systems, Hong Kong, China."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1007\/s10994-009-5152-4","article-title":"A theory of learning from different domains","volume":"79","author":"Blitzer","year":"2010","journal-title":"Mach. Learn."},{"key":"ref_51","unstructured":"Zhao, H., Des Combes, R.T., Zhang, K., and Gordon, G. (2019, January 10\u201315). On learning invariant representations for domain adaptation. Proceedings of the International Conference on Machine Learning. PMLR, Long Beach, CA, USA."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Xu, Q., Zhang, R., Zhang, Y., Wang, Y., and Tian, Q. (2021, January 20\u201325). A fourier-based framework for domain generalization. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.01415"},{"key":"ref_53","unstructured":"Zhou, K., Yang, Y., Qiao, Y., and Xiang, T. (2021, January 3\u20137). Domain Generalization with MixStyle. Proceedings of the International Conference on Learning Representations, Vienna, Austria."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Zhang, X., Cui, P., Xu, R., Zhou, L., He, Y., and Shen, Z. (2021, January 21\u201325). Deep stable learning for out-of-distribution generalization. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00533"},{"key":"ref_55","unstructured":"Wang, H., Ge, S., Lipton, Z., and Xing, E.P. (2019, January 8\u201314). Learning robust global representations by penalizing local predictive power. Proceedings of the Neural Information Processing Systems, Vancouver, BC, Canada."},{"key":"ref_56","unstructured":"Gulrajani, I., and Lopez-Paz, D. (2020, January 2\u20134). In Search of Lost Domain Generalization. Proceedings of the International Conference on Learning Representations, Virtual."},{"key":"ref_57","unstructured":"Wiles, O., Gowal, S., Stimberg, F., Rebuffi, S.A., Ktena, I., Dvijotham, K.D., and Cemgil, A.T. (2022, January 25\u201329). A Fine-Grained Analysis on Distribution Shift. Proceedings of the International Conference on Learning Representations, Virtual."},{"key":"ref_58","unstructured":"Thomas, M., and Joy, A.T. (2006). Elements of Information Theory, Wiley-Interscience."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/2\/193\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:09:50Z","timestamp":1760119790000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/2\/193"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,18]]},"references-count":58,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,2]]}},"alternative-id":["e25020193"],"URL":"https:\/\/doi.org\/10.3390\/e25020193","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2023,1,18]]}}}