{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T15:43:43Z","timestamp":1783611823430,"version":"3.55.0"},"reference-count":49,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2019,1,12]],"date-time":"2019-01-12T00:00:00Z","timestamp":1547251200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Science Foundation of China","doi-asserted-by":"crossref","award":["61773324, 61573292 and 61572406"],"award-info":[{"award-number":["61773324, 61573292 and 61572406"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2019,3,31]]},"abstract":"<jats:p>This article aims to develop a new and robust approach to feature representation. Motivated by the success of Auto-Encoders, we first theoretically analyze and summarize the general properties of all algorithms that are based on traditional Auto-Encoders: (1) The reconstruction error of the input cannot be lower than a lower bound, which can be viewed as a guiding principle for reconstructing the input. Additionally, when the input is corrupted with noises, the reconstruction error of the corrupted input also cannot be lower than a lower bound. (2) The reconstruction of a hidden representation achieving its ideal situation is the necessary condition for the reconstruction of the input to reach the ideal state. (3) Minimizing the Frobenius norm of the Jacobian matrix of the hidden representation has a deficiency and may result in a much worse local optimum value. We believe that minimizing the reconstruction error of the hidden representation is more robust than minimizing the Frobenius norm of the Jacobian matrix of the hidden representation. Based on the above analysis, we propose a new model termed<jats:italic>Double Denoising Auto-Encoders<\/jats:italic>(DDAEs), which uses corruption and reconstruction on both the input and the hidden representation. We demonstrate that the proposed model is highly flexible and extensible and has a potentially better capability to learn invariant and robust feature representations. We also show that our model is more robust than Denoising Auto-Encoders (DAEs) for dealing with noises or inessential features. Furthermore, we detail how to train DDAEs with two different pretraining methods by optimizing the objective function in a combined and separate manner, respectively. Comparative experiments illustrate that the proposed model is significantly better for representation learning than the state-of-the-art models.<\/jats:p>","DOI":"10.1145\/3284174","type":"journal-article","created":{"date-parts":[[2019,1,14]],"date-time":"2019-01-14T13:16:39Z","timestamp":1547471799000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Reconstruction of Hidden Representation for Robust Feature Extraction"],"prefix":"10.1145","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3550-3495","authenticated-orcid":false,"given":"Zeng","family":"Yu","sequence":"first","affiliation":[{"name":"Southwest Jiaotong University, Chengdu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tianrui","family":"Li","sequence":"additional","affiliation":[{"name":"Southwest Jiaotong University, Chengdu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ning","family":"Yu","sequence":"additional","affiliation":[{"name":"The College at Brockport State University of New York, Brockport, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yi","family":"Pan","sequence":"additional","affiliation":[{"name":"Georgia State University, Atlanta, GA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hongmei","family":"Chen","sequence":"additional","affiliation":[{"name":"Southwest Jiaotong University, Chengdu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bing","family":"Liu","sequence":"additional","affiliation":[{"name":"University of Illinois at Chicago, Chicago, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,1,12]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.5555\/2627435.2750359"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-39593-2_1"},{"key":"e_1_2_1_3_1","volume-title":"How auto-encoders could provide credit assignment in deep networks via target propagation. arXiv preprint arXiv:1407.7906","author":"Bengio Yoshua","year":"2014","unstructured":"Yoshua Bengio . 2014. How auto-encoders could provide credit assignment in deep networks via target propagation. arXiv preprint arXiv:1407.7906 ( 2014 ). Yoshua Bengio. 2014. How auto-encoders could provide credit assignment in deep networks via target propagation. arXiv preprint arXiv:1407.7906 (2014)."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1561\/2200000006"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2013.50"},{"key":"e_1_2_1_6_1","doi-asserted-by":"crossref","unstructured":"Yoshua Bengio Pascal Lamblin Dan Popovici and Hugo Larochelle. 2007. Greedy layer-wise training of deep networks. In Advances in Neural Information Processing Systems. 153--160. Yoshua Bengio Pascal Lamblin Dan Popovici and Hugo Larochelle. 2007. Greedy layer-wise training of deep networks. In Advances in Neural Information Processing Systems. 153--160.","DOI":"10.7551\/mitpress\/7503.003.0024"},{"key":"e_1_2_1_7_1","volume-title":"International Conference on Machine Learning. 226--234","author":"Bengio Yoshua","year":"2014","unstructured":"Yoshua Bengio , Eric Laufer , Guillaume Alain , and Jason Yosinski . 2014 . Deep generative stochastic networks trainable by backprop . In International Conference on Machine Learning. 226--234 . Yoshua Bengio, Eric Laufer, Guillaume Alain, and Jason Yosinski. 2014. Deep generative stochastic networks trainable by backprop. In International Conference on Machine Learning. 226--234."},{"key":"e_1_2_1_8_1","unstructured":"Yoshua Bengio Li Yao Guillaume Alain and Pascal Vincent. 2013. Generalized denoising auto-encoders as generative models. In Advances in Neural Information Processing Systems. 899--907. Yoshua Bengio Li Yao Guillaume Alain and Pascal Vincent. 2013. Generalized denoising auto-encoders as generative models. In Advances in Neural Information Processing Systems. 899--907."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.5555\/2188385.2188395"},{"key":"e_1_2_1_10_1","unstructured":"James S. Bergstra R\u00e9mi Bardenet Yoshua Bengio and Bal\u00e1zs K\u00e9gl. 2011. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems. 2546--2554. James S. Bergstra R\u00e9mi Bardenet Yoshua Bengio and Bal\u00e1zs K\u00e9gl. 2011. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems. 2546--2554."},{"key":"e_1_2_1_11_1","volume-title":"International Conference on Machine Learning. 1476--1484","author":"Chen Minmin","year":"2014","unstructured":"Minmin Chen , Kilian Q. Weinberger , Fei Sha , and Yoshua Bengio . 2014 . Marginalized denoising auto-encoders for nonlinear representations . In International Conference on Machine Learning. 1476--1484 . Minmin Chen, Kilian Q. Weinberger, Fei Sha, and Yoshua Bengio. 2014. Marginalized denoising auto-encoders for nonlinear representations. In International Conference on Machine Learning. 1476--1484."},{"key":"e_1_2_1_12_1","unstructured":"Djork-Arn\u00e9 Clevert Andreas Mayr Thomas Unterthiner and Sepp Hochreiter. 2015. Rectified factor networks. In Advances in Neural Information Processing Systems. 1855--1863. Djork-Arn\u00e9 Clevert Andreas Mayr Thomas Unterthiner and Sepp Hochreiter. 2015. Rectified factor networks. In Advances in Neural Information Processing Systems. 1855--1863."},{"key":"e_1_2_1_13_1","volume-title":"Hinton","author":"Deng Li","year":"2010","unstructured":"Li Deng , Michael L. Seltzer , Dong Yu , Alex Acero , Abdel-rahman Mohamed, and Geoffrey E . Hinton . 2010 . Binary coding of speech spectrograms using a deep auto-encoder. In Interspeech . 1692--1695. Li Deng, Michael L. Seltzer, Dong Yu, Alex Acero, Abdel-rahman Mohamed, and Geoffrey E. Hinton. 2010. Binary coding of speech spectrograms using a deep auto-encoder. In Interspeech. 1692--1695."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCYB.2016.2536638"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/TGRS.2016.2645226"},{"key":"e_1_2_1_16_1","volume-title":"International Conference on Machine Learning. 881--889","author":"Germain Mathieu","year":"2015","unstructured":"Mathieu Germain , Karol Gregor , Iain Murray , and Hugo Larochelle . 2015 . MADE: Masked autoencoder for distribution estimation . In International Conference on Machine Learning. 881--889 . Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. 2015. MADE: Masked autoencoder for distribution estimation. In International Conference on Machine Learning. 881--889."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2015.02.023"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2014.2362140"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206545"},{"key":"e_1_2_1_20_1","volume-title":"Kingma and Max Welling","author":"Diederik","year":"2013","unstructured":"Diederik P. Kingma and Max Welling . 2013 . Auto-encoding variational Bayes . arXiv preprint arXiv:1312.6114 (2013). Diederik P. Kingma and Max Welling. 2013. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 (2013)."},{"key":"e_1_2_1_21_1","volume-title":"Hinton","author":"Krizhevsky Alex","year":"2012","unstructured":"Alex Krizhevsky , Ilya Sutskever , and Geoffrey E . Hinton . 2012 . Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems . 1097--1105. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. 1097--1105."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1273496.1273556"},{"key":"e_1_2_1_23_1","volume-title":"Deep learning. Nature 521, 7553","author":"LeCun Yann","year":"2015","unstructured":"Yann LeCun , Yoshua Bengio , and Geoffrey Hinton . 2015. Deep learning. Nature 521, 7553 ( 2015 ), 436--444. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521, 7553 (2015), 436--444."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2016.2605010"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2017.07.005"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.sigpro.2015.01.001"},{"key":"e_1_2_1_27_1","volume-title":"Adversarial autoencoders. arXiv preprint arXiv:1511.05644","author":"Makhzani Alireza","year":"2015","unstructured":"Alireza Makhzani , Jonathon Shlens , Navdeep Jaitly , Ian Goodfellow , and Brendan Frey . 2015. Adversarial autoencoders. arXiv preprint arXiv:1511.05644 ( 2015 ). Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. 2015. Adversarial autoencoders. arXiv preprint arXiv:1511.05644 (2015)."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1093\/bib\/bbx044"},{"key":"e_1_2_1_29_1","volume-title":"International Conference on Machine Learning. 2368--2376","author":"Pezeshki Mohammad","year":"2016","unstructured":"Mohammad Pezeshki , Linxi Fan , Philemon Brakel , Aaron Courville , and Yoshua Bengio . 2016 . Deconstructing the ladder network architecture . In International Conference on Machine Learning. 2368--2376 . Mohammad Pezeshki, Linxi Fan, Philemon Brakel, Aaron Courville, and Yoshua Bengio. 2016. Deconstructing the ladder network architecture. In International Conference on Machine Learning. 2368--2376."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2016.05.055"},{"key":"e_1_2_1_31_1","unstructured":"Antti Rasmus Mathias Berglund Mikko Honkala Harri Valpola and Tapani Raiko. 2015. Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems. 3546--3554. Antti Rasmus Mathias Berglund Mikko Honkala Harri Valpola and Tapani Raiko. 2015. Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems. 3546--3554."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.5555\/2034117.2034159"},{"key":"e_1_2_1_33_1","volume-title":"International Conference on Machine Learning. 833--840","author":"Rifai Salah","year":"2011","unstructured":"Salah Rifai , Pascal Vincent , Xavier Muller , Xavier Glorot , and Yoshua Bengio . 2011 . Contractive auto-encoders: Explicit invariance during feature extraction . In International Conference on Machine Learning. 833--840 . Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. 2011. Contractive auto-encoders: Explicit invariance during feature extraction. In International Conference on Machine Learning. 833--840."},{"key":"e_1_2_1_34_1","unstructured":"Ruslan Salakhutdinov and Geoffrey Hinton. 2009. Deep Boltzmann machines. In Artificial Intelligence and Statistics. 448--455. Ruslan Salakhutdinov and Geoffrey Hinton. 2009. Deep Boltzmann machines. In Artificial Intelligence and Statistics. 448--455."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2014.09.008"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2016.12.012"},{"key":"e_1_2_1_37_1","volume-title":"Adams","author":"Snoek Jasper","year":"2012","unstructured":"Jasper Snoek , Hugo Larochelle , and Ryan P . Adams . 2012 . Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems . 2951--2959. Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. 2012. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems. 2951--2959."},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing. 151--161","author":"Socher Richard","unstructured":"Richard Socher , Jeffrey Pennington , Eric H. Huang , Andrew Y. Ng , and Christopher D. Manning . 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions . In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 151--161 . Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 151--161."},{"key":"e_1_2_1_39_1","volume-title":"S\u00f8ren Kaae S\u00f8nderby, and Ole Winther","author":"S\u00f8nderby Casper Kaae","year":"2016","unstructured":"Casper Kaae S\u00f8nderby , Tapani Raiko , Lars Maal\u00f8e , S\u00f8ren Kaae S\u00f8nderby, and Ole Winther . 2016 . Ladder variational autoencoders. In Advances in Neural Information Processing Systems . 3738--3746. Casper Kaae S\u00f8nderby, Tapani Raiko, Lars Maal\u00f8e, S\u00f8ren Kaae S\u00f8nderby, and Ole Winther. 2016. Ladder variational autoencoders. In Advances in Neural Information Processing Systems. 3738--3746."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.5555\/2627435.2670313"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1162\/NECO_a_00142"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390156.1390294"},{"key":"e_1_2_1_43_1","volume-title":"Journal of Machine Learning Research 11","author":"Vincent Pascal","year":"2010","unstructured":"Pascal Vincent , Hugo Larochelle , Isabelle Lajoie , Yoshua Bengio , and Pierre-Antoine Manzagol . 2010 . Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion . Journal of Machine Learning Research 11 , (2010), 3371--3408. Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. 2010. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research 11, (2010), 3371--3408."},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. 2725--2731","author":"Wang Shuyang","year":"2017","unstructured":"Shuyang Wang , Zhengming Ding , and Yun Fu . 2017 . Feature selection guided auto-encoder . In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. 2725--2731 . Shuyang Wang, Zhengming Ding, and Yun Fu. 2017. Feature selection guided auto-encoder. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. 2725--2731."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2015.2458702"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2014.07.009"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2017.2766438"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.26599\/BDMA.2018.9020018"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2017.03.009"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3284174","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3284174","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T01:08:01Z","timestamp":1750208881000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3284174"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,1,12]]},"references-count":49,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2019,3,31]]}},"alternative-id":["10.1145\/3284174"],"URL":"https:\/\/doi.org\/10.1145\/3284174","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,1,12]]},"assertion":[{"value":"2017-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-01-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}