{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,11]],"date-time":"2026-02-11T18:04:22Z","timestamp":1770833062782,"version":"3.50.1"},"reference-count":21,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2023,4,18]],"date-time":"2023-04-18T00:00:00Z","timestamp":1681776000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Neural networks are usually trained with different variants of gradient descent-based optimization algorithms such as the stochastic gradient descent or the Adam optimizer. Recent theoretical work states that the critical points (where the gradient of the loss is zero) of two-layer ReLU networks with the square loss are not all local minima. However, in this work, we will explore an algorithm for training two-layer neural networks with ReLU-like activation and the square loss that alternatively finds the critical points of the loss function analytically for one layer while keeping the other layer and the neuron activation pattern fixed. Experiments indicate that this simple algorithm can find deeper optima than stochastic gradient descent or the Adam optimizer, obtaining significantly smaller training loss values on four out of the five real datasets evaluated. Moreover, the method is faster than the gradient descent methods and has virtually no tuning parameters.<\/jats:p>","DOI":"10.3390\/s23084072","type":"journal-article","created":{"date-parts":[[2023,4,18]],"date-time":"2023-04-18T05:53:07Z","timestamp":1681797187000},"page":"4072","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Training a Two-Layer ReLU Network Analytically"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9548-7872","authenticated-orcid":false,"given":"Adrian","family":"Barbu","sequence":"first","affiliation":[{"name":"Statistics Department, Florida State University, Tallahassee, FL 32306, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,4,18]]},"reference":[{"key":"ref_1","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1109\/MSP.2020.3004124","article-title":"The global landscape of neural networks: An overview","volume":"37","author":"Sun","year":"2020","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_3","unstructured":"Zhou, Y., and Liang, Y. (May, January 30). Critical points of linear neural networks: Analytical forms and landscape properties. Proceedings of the Sixth International Conference on Learning Representations (ICLR), Vancouver, BC, Canada."},{"key":"ref_4","unstructured":"Gopalani, P., and Mukherjee, A. (2022). Global Convergence of SGD On Two Layer Neural Nets. arXiv."},{"key":"ref_5","unstructured":"Villani, C. (2006). Hypocoercivity. arXiv."},{"key":"ref_6","unstructured":"Shi, B., Su, W.J., and Jordan, M.I. (2020). On Learning Rates and Schr\u00f6dinger Operators. arXiv."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Liang, P., and Wainwright, M.J. (2017, January 6\u201311). Convexified convolutional neural networks. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, Australia.","DOI":"10.1109\/ICMLC.2017.8107753"},{"key":"ref_8","unstructured":"Pilanci, M., and Ergen, T. (2020, January 13\u201318). Neural networks are convex regularizers: Exact polynomial-time convex optimization formulations for two-layer networks. Proceedings of the International Conference on Machine Learning, Virtual."},{"key":"ref_9","unstructured":"Mishkin, A., Sahiner, A., and Pilanci, M. (2022, January 17\u201323). Fast convex optimization for two-layer relu networks: Equivalent model classes and cone decompositions. Proceedings of the International Conference on Machine Learning, PMLR, Baltimore, MD, USA."},{"key":"ref_10","unstructured":"Bakshi, A., Jayaram, R., and Woodruff, D.P. (2019, January 25\u201328). Learning two layer rectified neural networks in polynomial time. Proceedings of the Conference on Learning Theory, PMLR, Phoenix, AZ, USA."},{"key":"ref_11","unstructured":"Goel, S., and Klivans, A.R. (2019, January 25\u201328). Learning neural networks with two nonlinear layers in polynomial time. Proceedings of the Conference on Learning Theory, PMLR, Phoenix, AZ, USA."},{"key":"ref_12","first-page":"13485","article-title":"Efficient algorithms for learning depth-2 neural networks with general relu activations","volume":"34","author":"Awasthi","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_13","unstructured":"Rolnick, D., and Kording, K. (2020, January 13\u201318). Reverse-engineering deep relu networks. Proceedings of the International Conference on Machine Learning, PMLR, Virtual."},{"key":"ref_14","unstructured":"Fletcher, R. (2013). Practical Methods of Optimization, John Wiley & Sons."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1007\/BF01589116","article-title":"On the limited memory BFGS method for large scale optimization","volume":"45","author":"Liu","year":"1989","journal-title":"Math. Program."},{"key":"ref_16","unstructured":"Dua, D., and Graff, C. (2017). UCI Machine Learning Repository, University of California, School of Information and Computer Science."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Park, J.J., Florence, P., Straub, J., Newcombe, R., and Lovegrove, S. (2019, January 15\u201320). Deepsdf: Learning continuous signed distance functions for shape representation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00025"},{"key":"ref_18","unstructured":"Borenstein, E., Sharon, E., and Ullman, S. (July, January 27). Combining top-down and bottom-up segmentation. Proceedings of the CVPR Workshop, Washington, DC, USA."},{"key":"ref_19","unstructured":"Martin, D., Fowlkes, C., Tal, D., and Malik, J. (2001, January 9\u201312). A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics. Proceedings of the Eighth IEEE International Conference on Computer Vision, Vancouver, BC, Canada."},{"key":"ref_20","first-page":"71","article-title":"Decision tree performance enhancement using an artificial neural network implementation","volume":"Volume 11","author":"Sethi","year":"1991","journal-title":"Machine Intelligence and Pattern Recognition"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Welbl, J. (2014, January 2\u20135). Casting random forests as artificial neural networks (and profiting from it). Proceedings of the German Conference on Pattern Recognition, M\u00fcnster, Germany.","DOI":"10.1007\/978-3-319-11752-2_66"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/8\/4072\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:18:07Z","timestamp":1760123887000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/8\/4072"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,18]]},"references-count":21,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2023,4]]}},"alternative-id":["s23084072"],"URL":"https:\/\/doi.org\/10.3390\/s23084072","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,18]]}}}