{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,2]],"date-time":"2026-03-02T23:31:05Z","timestamp":1772494265708,"version":"3.50.1"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,9,10]],"date-time":"2020-09-10T00:00:00Z","timestamp":1599696000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,9,10]],"date-time":"2020-09-10T00:00:00Z","timestamp":1599696000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["EURASIP J. Adv. Signal Process."],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We design a rectified linear unit-based multilayer neural network by mapping the feature vectors to a higher dimensional space in every layer. We design the weight matrices in every layer to ensure a reduction of the training cost as the number of layers increases. Linear projection to the target in the higher dimensional space leads to a lower training cost if a convex cost is minimized. An <jats:italic>\u2113<\/jats:italic><jats:sub>2<\/jats:sub>-norm convex constraint is used in the minimization to reduce the generalization error and avoid overfitting. The regularization hyperparameters of the network are derived analytically to guarantee a monotonic decrement of the training cost, and therefore, it eliminates the need for cross-validation to find the regularization hyperparameter in each layer. We show that the proposed architecture is norm-preserving and provides an invertible feature vector and, therefore, can be used to reduce the training cost of any other learning method which employs linear projection to estimate the target.<\/jats:p>","DOI":"10.1186\/s13634-020-00695-2","type":"journal-article","created":{"date-parts":[[2020,9,10]],"date-time":"2020-09-10T19:29:46Z","timestamp":1599766186000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["High-dimensional neural feature design for layer-wise reduction of training cost"],"prefix":"10.1186","volume":"2020","author":[{"given":"Alireza M.","family":"Javid","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Arun","family":"Venkitaraman","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mikael","family":"Skoglund","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Saikat","family":"Chatterjee","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2020,9,10]]},"reference":[{"issue":"3","key":"695_CR1","first-page":"273","volume":"20","author":"C. Cortes","year":"1995","unstructured":"C. Cortes, V. Vapnik, Support-vector networks. Mach. Learn.20(3), 273\u2013297 (1995).","journal-title":"Mach. Learn."},{"issue":"5","key":"695_CR2","doi-asserted-by":"publisher","first-page":"1299","DOI":"10.1162\/089976698300017467","volume":"10","author":"B. Sch\u00f6lkopf","year":"1998","unstructured":"B. Sch\u00f6lkopf, A. Smola, K. -R. Muller, Nonlinear component analysis as a kernel eigenvalue problem. Neural Comput.10(5), 1299\u20131319 (1998).","journal-title":"Neural Comput."},{"key":"695_CR3","volume-title":"Pattern Recognition and Machine Learning (Information Science and Statistics)","author":"C. M. Bishop","year":"2006","unstructured":"C. M. Bishop, Pattern Recognition and Machine Learning (Information Science and Statistics) (Springer, Berlin, Heidelberg, 2006)."},{"issue":"1","key":"695_CR4","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1109\/MSP.2010.939038","volume":"28","author":"D. Yu","year":"2011","unstructured":"D. Yu, L. Deng, Deep learning and its applications to signal and information processing [exploratory dsp]. IEEE Signal Process. Mag.28(1), 145\u2013154 (2011).","journal-title":"IEEE Signal Process. Mag."},{"issue":"3","key":"695_CR5","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","volume":"115","author":"O. Russakovsky","year":"2015","unstructured":"O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, L. Fei-Fei, Imagenet large scale visual recognition challenge. Intl. J. Computer Vision. 115(3), 211\u2013252 (2015).","journal-title":"Intl. J. Computer Vision"},{"key":"695_CR6","unstructured":"S. F. Dodge, L. J. Karam, A study and comparison of human and deep learning recognition performance under visual distortions. ArXiv e-prints (2017). http:\/\/arxiv.org\/abs\/1705.02498. Accessed 1 Oct 2019."},{"key":"695_CR7","first-page":"1058","volume-title":"Proceedings of the 30th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 28","author":"L. Wan","year":"2013","unstructured":"L. Wan, M. Zeiler, S. Zhang, Y. L. Cun, R. Fergus, in Proceedings of the 30th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 28, ed. by S. Dasgupta, D. McAllester. Regularization of neural networks using dropconnect (PMLRAtlanta, 2013), pp. 1058\u20131066."},{"key":"695_CR8","unstructured":"D. Mishkin, J. Matas, in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, ed. by Y Bengio, Y LeCun. All you need is a good init, (2016). http:\/\/arxiv.org\/abs\/1511.06422."},{"key":"695_CR9","first-page":"464","volume-title":"Proceedings of the 19th International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, vol. 51","author":"C. -Y. Lee","year":"2016","unstructured":"C. -Y. Lee, P. W. Gallagher, Z. Tu, in Proceedings of the 19th International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, vol. 51, ed. by A. Gretton, C. C. Robert. Generalizing pooling functions in convolutional neural networks: mixed, gated, and tree (PMLRCadiz, 2016), pp. 464\u2013472."},{"key":"695_CR10","doi-asserted-by":"crossref","unstructured":"A. J. Thomas, M. Petridis, S. D. Walters, S. M. Gheytassi, R. E. Morgan, in 2015 International Conference on Computational Science and Computational Intelligence (CSCI). On predicting the optimal number of hidden nodes, (2015), pp. 565\u2013570.","DOI":"10.1109\/CSCI.2015.33"},{"key":"695_CR11","unstructured":"C. Szegedy, A. Toshev, D. Erhan, in Advances in Neural Information Processing Systems 26, ed. by C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger. Deep neural networks for object detection, (2013), pp. 2553\u20132561."},{"key":"695_CR12","first-page":"1097","volume-title":"Advances in Neural Information Processing Systems 25","author":"A. Krizhevsky","year":"2012","unstructured":"A. Krizhevsky, I. Sutskever, G. E. Hinton, in Advances in Neural Information Processing Systems 25. Imagenet classification with deep convolutional neural networks (Curran Associates Inc.Red Hook, 2012), pp. 1097\u20131105."},{"key":"695_CR13","volume-title":"Training Recurrent Neural Networks. PhD thesis","author":"I. Sutskever","year":"2013","unstructured":"I. Sutskever, Training Recurrent Neural Networks. PhD thesis (University of Toronto, Toronto, 2013). AAINS22066."},{"issue":"8","key":"695_CR14","doi-asserted-by":"publisher","first-page":"1872","DOI":"10.1109\/TPAMI.2012.230","volume":"35","author":"J. Bruna","year":"2013","unstructured":"J. Bruna, S. Mallat, Invariant scattering convolution networks. IEEE Trans. Pattern Anal. Mach. Intell.35(8), 1872\u20131886 (2013).","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"695_CR15","unstructured":"R. Vidal, J. Bruna, R. Giryes, S. Soatto, Mathematics of deep learning. ArXiv e-prints (2017). http:\/\/arxiv.org\/abs\/1712.04741. Accessed 1 Oct 2019."},{"issue":"13","key":"695_CR16","doi-asserted-by":"publisher","first-page":"3444","DOI":"10.1109\/TSP.2016.2546221","volume":"64","author":"R. Giryes","year":"2016","unstructured":"R. Giryes, G. Sapiro, A. M. Bronstein, Deep neural networks with random Gaussian weights: a universal classification strategy?IEEE Trans. Signal Process.64(13), 3444\u20133457 (2016).","journal-title":"IEEE Trans. Signal Process."},{"key":"695_CR17","unstructured":"S. Chatterjee, A. M. Javid, S. K. Mostafa Sadeghi, P. P. Mitra, M. Skoglund, SSFN: self size-estimating feed-forward network and low complexity design. ArXiv e-prints (2019). http:\/\/arxiv.org\/abs\/1905.07111. Accessed 1 Oct 2019."},{"key":"695_CR18","unstructured":"S. Chatterjee, A. M. Javid, M. Sadeghi, P. P. Mitra, M. Skoglund, Progressive learning for systematic design of large neural networks. ArXiv e-prints (2017). http:\/\/arxiv.org\/abs\/1710.08177. Accessed 1 Oct 2019."},{"issue":"2","key":"695_CR19","doi-asserted-by":"publisher","first-page":"513","DOI":"10.1109\/TSMCB.2011.2168604","volume":"42","author":"G. -B. Huang","year":"2012","unstructured":"G. -B. Huang, H. Zhou, X. Ding, R. Zhang, Extreme learning machine for regression and multiclass classification. J. Trans. Sys. Man Cyber. B. 42(2), 513\u2013529 (2012).","journal-title":"J. Trans. Sys. Man Cyber. B"},{"issue":"Supplement C","key":"695_CR20","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1016\/j.neunet.2014.10.001","volume":"61","author":"G. Huang","year":"2015","unstructured":"G. Huang, G. -B. Huang, S. Song, K. You, Trends in extreme learning machines: a review. Neural Netw.61(Supplement C), 32\u201348 (2015).","journal-title":"Neural Netw."},{"issue":"99","key":"695_CR21","first-page":"1","volume":"PP","author":"T. Hussain","year":"2017","unstructured":"T. Hussain, S. M. Siniscalchi, C. C. Lee, S. S. Wang, Y. Tsao, W. H. Liao, Experimental study on extreme learning machine applications for speech enhancement. IEEE Access. PP(99), 1 (2017).","journal-title":"IEEE Access"},{"key":"695_CR22","doi-asserted-by":"crossref","unstructured":"W. Zhu, J. Miao, L. Qing, G. Huang, in 2015 International Joint Conference on Neural Networks (IJCNN). Hierarchical extreme learning machine for unsupervised representation learning, (2015), pp. 1\u20138.","DOI":"10.1109\/IJCNN.2015.7280669"},{"key":"695_CR23","doi-asserted-by":"crossref","unstructured":"A. Rosenfeld, J. K. Tsotsos, Intriguing properties of randomly weighted networks: generalizing while learning next to nothing. ArXiv e-prints (2018). http:\/\/arxiv.org\/abs\/1802.00844. Accessed 1 Oct 2019.","DOI":"10.1109\/CRV.2019.00010"},{"key":"695_CR24","doi-asserted-by":"crossref","unstructured":"A. M. Javid, S. Chatterjee, M. Skoglund, in 2018 15th International Symposium on Wireless Communication Systems (ISWCS). Mutual information preserving analysis of a single layer feedforward network, (2018), pp. 1\u20135.","DOI":"10.1109\/ISWCS.2018.8491242"},{"issue":"4","key":"695_CR25","doi-asserted-by":"publisher","first-page":"809","DOI":"10.1109\/TNNLS.2015.2424995","volume":"27","author":"J. Tang","year":"2016","unstructured":"J. Tang, C. Deng, G. Huang, Extreme learning machine for multilayer perceptron. IEEE Trans. Neural Netw. Learn. Syst.27(4), 809\u2013821 (2016).","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"695_CR26","first-page":"1058","volume-title":"Proceedings of the 30th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 28","author":"L. Wan","year":"2013","unstructured":"L. Wan, M. Zeiler, S. Zhang, Y. L. Cun, R. Fergus, in Proceedings of the 30th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 28, ed. by S. Dasgupta, D. McAllester. Regularization of neural networks using dropconnect (PMLRAtlanta, 2013), pp. 1058\u20131066."},{"issue":"2","key":"695_CR27","first-page":"161","volume":"6","author":"P. W. Frey","year":"1991","unstructured":"P. W. Frey, D. J. Slate, Letter recognition using Holland-style adaptive classifiers. Mach. Learn.6(2), 161\u2013182 (1991). https:\/\/archive.ics.uci.edu\/ml\/datasets\/letter+recognition.","journal-title":"Mach. Learn."},{"key":"695_CR28","volume-title":"UCI Repository of Machine Learning Databases","author":"C. L. Blake","year":"1998","unstructured":"C. L. Blake, C. J. Merz, UCI Repository of Machine Learning Databases (Dept. Inf. Comput. Sci., Univ., Irvine, 1998). http:\/\/www.ics.uci.edu\/~mlearn\/MLRepository.html."},{"key":"695_CR29","doi-asserted-by":"crossref","unstructured":"Y. Lecun, L. Bottou, Y. Bengio, P. Haffner, in Proceedings of the IEEE. Gradient-based learning applied to document recognition, (1998), pp. 2278\u20132324. Available: http:\/\/yann.lecun.com\/exdb\/mnist\/.","DOI":"10.1109\/5.726791"},{"issue":"1","key":"695_CR30","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1561\/2200000016","volume":"3","author":"S. Boyd","year":"2011","unstructured":"S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn.3(1), 1\u2013122 (2011).","journal-title":"Found. Trends Mach. Learn."},{"key":"695_CR31","unstructured":"D. P. Kingma, J. Ba, in Proceedings of the 3rd International Conference on Learning Representations (ICLR). ADAM: a method for stochastic optimization, (2015)."}],"container-title":["EURASIP Journal on Advances in Signal Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13634-020-00695-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13634-020-00695-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13634-020-00695-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,10]],"date-time":"2021-09-10T00:11:55Z","timestamp":1631232715000},"score":1,"resource":{"primary":{"URL":"https:\/\/asp-eurasipjournals.springeropen.com\/articles\/10.1186\/s13634-020-00695-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,10]]},"references-count":31,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["695"],"URL":"https:\/\/doi.org\/10.1186\/s13634-020-00695-2","relation":{},"ISSN":["1687-6180"],"issn-type":[{"value":"1687-6180","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,9,10]]},"assertion":[{"value":"1 November 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 August 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 September 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 November 2020","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Correction","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"\"The original publication was missing a statement noting the funding provided by Royal Institute of Technology. The article has been updated to include this statement.\"","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Not applicable.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"40"}}