{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T02:44:22Z","timestamp":1760150662442,"version":"build-2065373602"},"reference-count":50,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2023,12,20]],"date-time":"2023-12-20T00:00:00Z","timestamp":1703030400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62166016","119MS004"],"award-info":[{"award-number":["62166016","119MS004"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Hainan Provincial Natural Science Foundation of China","award":["62166016","119MS004"],"award-info":[{"award-number":["62166016","119MS004"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>In response to the challenge of overfitting, which may lead to a decline in network generalization performance, this paper proposes a new regularization technique, called the class-based decorrelation method (CDM). Specifically, this method views the neurons in a specific hidden layer as base learners, and aims to boost network generalization as well as model accuracy by minimizing the correlation among individual base learners while simultaneously maximizing their class-conditional correlation. Intuitively, CDM not only promotes diversity among the hidden neurons, but also enhances their cohesiveness among them when processing samples from the same class. Comparative experiments conducted on various datasets using deep models demonstrate that CDM effectively reduces overfitting and improves classification performance.<\/jats:p>","DOI":"10.3390\/e26010007","type":"journal-article","created":{"date-parts":[[2023,12,20]],"date-time":"2023-12-20T07:20:31Z","timestamp":1703056831000},"page":"7","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["A Deep Neural Network Regularization Measure: The Class-Based Decorrelation Method"],"prefix":"10.3390","volume":"26","author":[{"given":"Chenguang","family":"Zhang","sequence":"first","affiliation":[{"name":"School of Mathematics and Statistics, Hainan University, Haikou 570100, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tian","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Information and Communication Engineering, Hainan University, Haikou 570100, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xuejiao","family":"Du","sequence":"additional","affiliation":[{"name":"School of Mathematics and Statistics, Hainan University, Haikou 570100, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,12,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_2","first-page":"7479","article-title":"Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning","volume":"22","author":"Martin","year":"2021","journal-title":"J. Mach. Learn. Res."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"525","DOI":"10.1109\/18.661502","article-title":"The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network","volume":"44","author":"Bartlett","year":"1998","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_4","first-page":"463","article-title":"Rademacher and Gaussian complexities: Risk bounds and structural results","volume":"3","author":"Bartlett","year":"2002","journal-title":"J. Mach. Learn. Res."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1080\/00401706.1970.10488634","article-title":"Ridge Regression: Biased Estimation for Nonorthogonal Problems","volume":"12","author":"Hoerl","year":"1970","journal-title":"Technometrics"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"266","DOI":"10.1137\/1021044","article-title":"Solutions of Ill-Posed Problems (A. N. Tikhonov and V. Y. Arsenin)","volume":"21","author":"Willoughby","year":"1979","journal-title":"SIAM Rev."},{"key":"ref_7","unstructured":"Krogh, A., and Hertz, J. (1991). A simple weight decay can improve generalization. Adv. Neural Inf. Process. Syst., 4."},{"key":"ref_8","first-page":"1929","article-title":"Dropout: A simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_9","unstructured":"Neal, B., Mittal, S., Baratin, A., Tantia, V., Scicluna, M., Lacoste-Julien, S., and Mitliagkas, I. (2019, January 15). A Modern Take on the Bias-Variance Tradeoff in Neural Networks. Proceedings of the ICML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena, Long Beach, CA, USA."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"15849","DOI":"10.1073\/pnas.1903070116","article-title":"Reconciling modern machine-learning practice and the classical bias\u2013variance trade-off","volume":"116","author":"Belkin","year":"2019","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_11","unstructured":"Cogswell, M., Ahmed, F., Girshick, R., Zitnick, L., and Batra, D. (2015). Reducing overfitting in deep networks by decorrelating representations. arXiv."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Gu, S., Hou, Y., Zhang, L., and Zhang, Y. (2018, January 13\u201319). Regularizing Deep Neural Networks with an Ensemble-based Decorrelation Method. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18), Stockholm, Sweden.","DOI":"10.24963\/ijcai.2018\/301"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Yao, S., Hou, Y., Ge, L., and Hu, Z. (2019, January 3\u20137). Regularizing Deep Neural Networks by Ensemble-based Low-Level Sample-Variances Method. Proceedings of the 28th ACM International Conference on Information and Knowledge Management, Beijing, China.","DOI":"10.1145\/3357384.3357921"},{"key":"ref_14","first-page":"1947","article-title":"Emergence of invariance and disentanglement in deep representations","volume":"19","author":"Achille","year":"2018","journal-title":"J. Mach. Learn. Res."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1399","DOI":"10.1016\/S0893-6080(99)00073-8","article-title":"Ensemble learning via negative correlation","volume":"12","author":"Liu","year":"1999","journal-title":"Neural Netw."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1016\/j.ins.2013.12.016","article-title":"Fast decorrelated neural network ensembles with random weights","volume":"264","author":"Alhamdoosh","year":"2014","journal-title":"Inf. Sci."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Wang, X., Chen, H., Tang, S., Wu, Z., and Zhu, W. (2022). Disentangled representation learning. arXiv.","DOI":"10.1145\/3581783.3613859"},{"key":"ref_18","unstructured":"Carbonneau, M.A., Zaidi, J., Boilard, J., and Gagnon, G. (2020). Measuring disentanglement: A review of metrics. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"9059","DOI":"10.1109\/TCYB.2021.3053165","article-title":"When does diversity help generalization in classification ensembles?","volume":"52","author":"Bian","year":"2021","journal-title":"IEEE Trans. Cybern."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2650","DOI":"10.1109\/TNNLS.2018.2885972","article-title":"Regularizing deep neural networks by enhancing diversity in feature extraction","volume":"30","author":"Ayinde","year":"2019","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_21","unstructured":"Kingma, D.P., and Welling, M. (2013). Auto-encoding variational bayes. arXiv."},{"key":"ref_22","unstructured":"Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2016, January 2\u20134). beta-vae: Learning basic visual concepts with a constrained variational framework. Proceedings of the International Conference on Learning Representations, San Juan, Puerto Rico."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Zhou, Z.H. (2012). Ensemble Methods: Foundations and Algorithms, CRC Press.","DOI":"10.1201\/b12207"},{"key":"ref_24","unstructured":"Zhang, C., Hou, Y., Song, D., Ge, L., and Yao, Y. (2020). Redundancy of Hidden Layers in Deep Learning: An Information Perspective. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Bartlett, P., Maiorov, V., and Meir, R. (1998). Almost linear VC dimension bounds for piecewise polynomial networks. Adv. Neural Inf. Process. Syst., 11.","DOI":"10.1162\/089976698300017016"},{"key":"ref_26","first-page":"2285","article-title":"Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks","volume":"20","author":"Bartlett","year":"2019","journal-title":"J. Mach. Learn. Res."},{"key":"ref_27","unstructured":"Yang, Y., Yang, H., and Xiang, Y. (2023). Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivatives. arXiv."},{"key":"ref_28","unstructured":"Dziugaite, G.K., and Roy, D.M. (2017). Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv."},{"key":"ref_29","unstructured":"Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. (2017). Exploring generalization in deep learning. Adv. Neural Inf. Process. Syst., 30."},{"key":"ref_30","unstructured":"Neyshabur, B., Bhojanapalli, S., and Srebro, N. (2017). A pac-bayesian approach to spectrally-normalized margin bounds for neural networks. arXiv."},{"key":"ref_31","unstructured":"Neyshabur, B., Tomioka, R., and Srebro, N. (2015, January 3\u20136). Norm-based capacity control in neural networks. Proceedings of the Conference on Learning Theory, PMLR, Paris, France."},{"key":"ref_32","unstructured":"Bartlett, P.L., Foster, D.J., and Telgarsky, M.J. (2017). Spectrally-normalized margin bounds for neural networks. Adv. Neural Inf. Process. Syst., 30."},{"key":"ref_33","unstructured":"Arora, S., Du, S.S., Hu, W., Li, Z., Salakhutdinov, R.R., and Wang, R. (2019). On exact computation with an infinitely wide neural net. Adv. Neural Inf. Process. Syst., 32."},{"key":"ref_34","unstructured":"Daniely, A., and Granot, E. (2019). Generalization bounds for neural networks via approximate description length. Adv. Neural Inf. Process. Syst., 32."},{"key":"ref_35","unstructured":"Liang, T., Poggio, T., Rakhlin, A., and Stokes, J. (2019, January 16\u201318). Fisher-rao metric, geometry, and complexity of neural networks. Proceedings of the The 22nd International Conference on Artificial Intelligence and Statistics, PMLR, Okinawa, Japan."},{"key":"ref_36","unstructured":"Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. (2019). Fantastic generalization measures and where to find them. arXiv."},{"key":"ref_37","unstructured":"Kumar, A., Sattigeri, P., and Balakrishnan, A. (2017). Variational inference of disentangled latent concepts from unlabeled observations. arXiv."},{"key":"ref_38","unstructured":"Chen, R.T., Li, X., Grosse, R.B., and Duvenaud, D.K. (2018). Isolating sources of disentanglement in variational autoencoders. Adv. Neural Inf. Process. Syst., 31."},{"key":"ref_39","unstructured":"Wan, L., Zeiler, M., Zhang, S., Le Cun, Y., and Fergus, R. (2013, January 17\u201319). Regularization of neural networks using dropconnect. Proceedings of the International Conference on Machine Learning, PMLR, Atlanta, GA, USA."},{"key":"ref_40","unstructured":"Bengio, Y., and Bergstra, J. (2009). Slow, decorrelated features for pretraining complex cell-like networks. Adv. Neural Inf. Process. Syst., 22."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Bao, Y., Jiang, H., Dai, L., and Liu, C. (2013, January 26\u201331). Incoherent training of deep neural networks to de-correlate bottleneck features for speech recognition. Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.","DOI":"10.1109\/ICASSP.2013.6639015"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27\u201330). Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.308"},{"key":"ref_43","unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"Lecun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_45","unstructured":"Krizhevsky, A., and Hinton, G. (2009). Handbook of Systemic Autoimmune Diseases, Univeristy of Toronto."},{"key":"ref_46","unstructured":"Ravi, S., and Larochelle, H. (2016, January 2\u20134). Optimization as a model for few-shot learning. Proceedings of the International Conference on Learning Representations, San Juan, Puerto Rico."},{"key":"ref_47","first-page":"17616","article-title":"Relative uncertainty learning for facial expression recognition","volume":"34","author":"Zhang","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Wang, C., Ling, X., and Deng, W. (2022). Learn From All: Erasing Attention Consistency for Noisy Label Facial Expression Recognition. arXiv.","DOI":"10.1007\/978-3-031-19809-0_24"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1145\/3065386","article-title":"ImageNet classification with deep convolutional neural networks","volume":"60","author":"Krizhevsky","year":"2017","journal-title":"Commun. ACM"},{"key":"ref_50","unstructured":"Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019). Pytorch: An imperative style, high-performance deep learning library. Adv. Neural Inf. Process. Syst., 32."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/26\/1\/7\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:42:05Z","timestamp":1760132525000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/26\/1\/7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,20]]},"references-count":50,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,1]]}},"alternative-id":["e26010007"],"URL":"https:\/\/doi.org\/10.3390\/e26010007","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2023,12,20]]}}}