{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:57:53Z","timestamp":1760144273739,"version":"build-2065373602"},"reference-count":42,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2024,4,5]],"date-time":"2024-04-05T00:00:00Z","timestamp":1712275200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Ministry of Science and Higher Education of the Russian Federation","award":["075-15-2022-1121","7750185"],"award-info":[{"award-number":["075-15-2022-1121","7750185"]}]},{"name":"Science Fund of the Republic of Serbia","award":["075-15-2022-1121","7750185"],"award-info":[{"award-number":["075-15-2022-1121","7750185"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Axioms"],"abstract":"<jats:p>In this article, we consider the correction of metric matrices in quasi-Newton methods (QNM) from the perspective of machine learning theory. Based on training information for estimating the matrix of the second derivatives of a function, we formulate a quality functional and minimize it by using gradient machine learning algorithms. We demonstrate that this approach leads us to the well-known ways of updating metric matrices used in QNM. The learning algorithm for finding metric matrices performs minimization along a system of directions, the orthogonality of which determines the convergence rate of the learning process. The degree of learning vectors\u2019 orthogonality can be increased both by choosing a QNM and by using additional orthogonalization methods. It has been shown theoretically that the orthogonality degree of learning vectors in the Broyden\u2013Fletcher\u2013Goldfarb\u2013Shanno (BFGS) method is higher than in the Davidon\u2013Fletcher\u2013Powell (DFP) method, which determines the advantage of the BFGS method. In our paper, we discuss some orthogonalization techniques. One of them is to include iterations with orthogonalization or an exact one-dimensional descent. As a result, it is theoretically possible to detect the cumulative effect of reducing the optimization space on quadratic functions. Another way to increase the orthogonality degree of learning vectors at the initial stages of the QNM is a special choice of initial metric matrices. Our computational experiments on problems with a high degree of conditionality have confirmed the stated theoretical assumptions.<\/jats:p>","DOI":"10.3390\/axioms13040240","type":"journal-article","created":{"date-parts":[[2024,4,5]],"date-time":"2024-04-05T08:27:52Z","timestamp":1712305672000},"page":"240","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Machine Learning in Quasi-Newton Methods"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8704-1670","authenticated-orcid":false,"given":"Vladimir","family":"Krutikov","sequence":"first","affiliation":[{"name":"Laboratory \u201cHybrid Methods of Modeling and Optimization in Complex Systems\u201d, Siberian Federal University, 79 Svobodny Prospekt, 660041 Krasnoyarsk, Russia"},{"name":"Department of Applied Mathematics, Kemerovo State University, 6 Krasnaya Street, 650043 Kemerovo, Russia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8500-2050","authenticated-orcid":false,"given":"Elena","family":"Tovbis","sequence":"additional","affiliation":[{"name":"Institute of Informatics and Telecommunications, Reshetnev Siberian State University of Science and Technology, 31, Krasnoyarskii Rabochii Prospekt, 660037 Krasnoyarsk, Russia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0655-3741","authenticated-orcid":false,"given":"Predrag","family":"Stanimirovi\u0107","sequence":"additional","affiliation":[{"name":"Laboratory \u201cHybrid Methods of Modeling and Optimization in Complex Systems\u201d, Siberian Federal University, 79 Svobodny Prospekt, 660041 Krasnoyarsk, Russia"},{"name":"Faculty of Sciences and Mathematics, University of Ni\u0161, 18000 Ni\u0161, Serbia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0667-4001","authenticated-orcid":false,"given":"Lev","family":"Kazakovtsev","sequence":"additional","affiliation":[{"name":"Laboratory \u201cHybrid Methods of Modeling and Optimization in Complex Systems\u201d, Siberian Federal University, 79 Svobodny Prospekt, 660041 Krasnoyarsk, Russia"},{"name":"Institute of Informatics and Telecommunications, Reshetnev Siberian State University of Science and Technology, 31, Krasnoyarskii Rabochii Prospekt, 660037 Krasnoyarsk, Russia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5308-2503","authenticated-orcid":false,"given":"Darjan","family":"Karaba\u0161evi\u0107","sequence":"additional","affiliation":[{"name":"College of Global Business, Korea University, Sejong 30019, Republic of Korea"},{"name":"Faculty of Applied Management, Economics and Finance, University Business Academy in Novi Sad, Jevrejska 24, 11000 Belgrade, Serbia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,4,5]]},"reference":[{"key":"ref_1","unstructured":"Polyak, B.T. (1987). Introduction to Optimization, Optimization Software Inc., Publ. Division. Translated from Russian."},{"key":"ref_2","unstructured":"Nocedal, J., and Wright, S. (2006). Numerical Optimization, Series in Operations Research and Financial Engineering, Springer."},{"key":"ref_3","unstructured":"Bertsekas, D.P. (1982). Constrained Optimization and Lagrange Multiplier Methods, Academic Press."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Gill, P.E., Murray, W., and Wright, M.H. (2020). Practical Optimization, SIAM.","DOI":"10.1137\/1.9781611975604"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Dennis, J.E., and Schnabel, R.B. (1996). Numerical Methods for Unconstrained Optimization and Nonlinear Equations, SIAM.","DOI":"10.1137\/1.9781611971200"},{"key":"ref_6","unstructured":"Evtushenko, Y.G. (1982). Methods for Solving Extremal Problems and Their Application in Optimization Systems, Nauka. (In Russian)."},{"key":"ref_7","unstructured":"Polak, E. (1971). Computational Methods in Optimization: A Unified Approach, Academic Press."},{"key":"ref_8","first-page":"127312","article-title":"Iteratively regularized Gauss\u2013Newton type methods for approximating quasi\u2013solutions of irregular nonlinear operator equations in Hilbert space with an application to COVID-19 epidemic dynamics","volume":"431","author":"Kokurin","year":"2022","journal-title":"Appl. Math. Comput."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"296","DOI":"10.1016\/j.optcom.2019.07.046","article-title":"A positional misalignment correction method for Fourier ptychographic microscopy based on the quasi-Newton method with a global optimization module","volume":"452","author":"Zhang","year":"2019","journal-title":"Opt. Commun."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"114091","DOI":"10.1016\/j.cma.2021.114091","article-title":"An efficient and robust monolithic approach to phase-field quasi-static brittle fracture using a modified Newton method","volume":"386","author":"Lampron","year":"2021","journal-title":"Comput. Methods Appl."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"112810","DOI":"10.1016\/j.cma.2019.112810","article-title":"A multi-vector interface quasi-Newton method with linear complexity for partitioned fluid\u2013structure interaction","volume":"361","author":"Spenke","year":"2020","journal-title":"Comput. Methods Appl. Mech. Engrg."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"106934","DOI":"10.1016\/j.compstruc.2022.106934","article-title":"A memory-efficient MultiVector Quasi-Newton method for black-box Fluid-Structure Interaction coupling","volume":"275","author":"Zorrilla","year":"2023","journal-title":"Comput. Struct."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Davis, K., Schulte, M., and Uekermann, B. (2022). Enhancing Quasi-Newton Acceleration for Fluid-Structure Interaction. Math. Comput. Appl., 27.","DOI":"10.3390\/mca27030040"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1106647","DOI":"10.1016\/j.icheatmasstransfer.2023.106647","article-title":"Extending the inverse sequential quasi-Newton method for on-line monitoring and controlling of process conditions in the solidification of alloys","volume":"142","author":"Tourn","year":"2023","journal-title":"Int. Commun. Heat Mass Transf."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"108059","DOI":"10.1016\/j.ijthermalsci.2022.108059","article-title":"A self-scaling sequential quasi-Newton method for estimating the heat transfer coefficient distribution in the air jet impingement","volume":"185","author":"Hong","year":"2023","journal-title":"Int. J. Therm. Sci."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1668","DOI":"10.1080\/10556788.2021.1977806","article-title":"Quasi-Newton Methods for Machine Learning: Forget the Past, Just Sample","volume":"37","author":"Berahas","year":"2022","journal-title":"Optim. Methods Softw."},{"key":"ref_17","unstructured":"Rafati, J. (2024, January 11). Quasi-Newton Optimization Methods For Deep Learning Applications. Available online: https:\/\/arxiv.org\/abs\/1909.01994.pdf."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Indrapriyadarsini, S., Mahboubi, S., Ninomiya, H., Kamio, T., and Asai, H. (2022). Accelerating Symmetric Rank-1 Quasi-Newton Method with Nesterov\u2019s Gradient for Training Neural Networks. Algorithms, 15.","DOI":"10.20944\/preprints202112.0097.v1"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Davidon, W.C. (1959). Variable Metric Methods for Minimization, Argonne National Laboratory. A.E.C. Res. and Develop. Report ANL\u20135990.","DOI":"10.2172\/4252678"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1093\/comjnl\/6.2.163","article-title":"A rapidly convergent descent method for minimization","volume":"6","author":"Fletcher","year":"1963","journal-title":"Comput. J."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"845","DOI":"10.1287\/mnsc.20.5.845","article-title":"Self-scaling variable metric (SSVM) algorithms I: Criteria and sufficient conditions for scaling a class of algorithms","volume":"20","author":"Oren","year":"1974","journal-title":"Manag. Sci."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"863","DOI":"10.1287\/mnsc.20.5.863","article-title":"Self-scaling variable metric (SSVM) algorithms II: Implementation and experiments","volume":"20","author":"Oren","year":"1974","journal-title":"Manag. Sci."},{"key":"ref_23","first-page":"1","article-title":"Convergence Properties of a Class of Minimization Algorithms","volume":"Volume 2","author":"Mangasarian","year":"1975","journal-title":"Nonlinear Programming"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1007\/BF01584554","article-title":"Quasi-Newton algorithms generate identical points","volume":"2","author":"Dixon","year":"1972","journal-title":"Math. Program."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"115718","DOI":"10.1016\/j.cam.2023.115718","article-title":"An accelerated structured quasi-Newton method with a diagonal second-order Hessian approximation for nonlinear least squares problems","volume":"442","author":"Huynh","year":"2024","journal-title":"J. Comp. Appl. Math."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"108281","DOI":"10.1016\/j.patcog.2021.108281","article-title":"A Novel Quasi-Newton Method for Composite Convex Minimization","volume":"122","author":"Chai","year":"2022","journal-title":"Pattern Recognit."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1016\/j.cam.2017.06.024","article-title":"A modified quasi-Newton method for nonlinear equations","volume":"328","author":"Fang","year":"2018","journal-title":"J. Comp. Appl. Math."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"112744","DOI":"10.1016\/j.cam.2020.112744","article-title":"A modified Broyden-like quasi-Newton method for nonlinear equations","volume":"372","author":"Zhou","year":"2020","journal-title":"J. Comp. Appl. Math."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1093\/imamat\/6.1.76","article-title":"The convergence of a class of double\u2013rank minimization algorithms","volume":"6","author":"Broyden","year":"1970","journal-title":"J. Inst. Math. Appl."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1093\/comjnl\/13.3.317","article-title":"A new approach to variable metric algorithms","volume":"13","author":"Fletcher","year":"1970","journal-title":"Comput. J."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1090\/S0025-5718-1970-0258249-6","article-title":"A family of variable metric methods derived by variational means","volume":"24","author":"Goldfarb","year":"1970","journal-title":"Math. Comput."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1007\/BF01589116","article-title":"On the limited memory BFGS method for large scale optimization","volume":"45","author":"Liu","year":"1989","journal-title":"Math. Program."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1145\/279232.279236","article-title":"L-BFGS-B: Algorithm 778: L-BFGS-B, FORTRAN routines for large scale bound constrained optimization","volume":"23","author":"Zhu","year":"1997","journal-title":"ACM Trans. Math. Softw."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Tovbis, E., Krutikov, V., Stanimirovi\u0107, P., Meshechkin, V., Popov, A., and Kazakovtsev, L. (2023). A Family of Multi-Step Subgradient Minimization Methods. Mathematics, 11.","DOI":"10.3390\/math11102264"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Krutikov, V., Gutova, S., Tovbis, E., Kazakovtsev, L., and Semenkin, E. (2022). Relaxation Subgradient Algorithms with Machine Learning Procedures. Mathematics, 10.","DOI":"10.3390\/math10213959"},{"key":"ref_36","first-page":"433","article-title":"On a class of dual control learning systems","volume":"25","author":"Feldbaum","year":"1964","journal-title":"Avtomat. i Telemekh."},{"key":"ref_37","unstructured":"Aizerman, M.A., Braverman, E.M., and Rozonoer, L.I. (1970). Method of Potential Functions in Machine Learning Theory, Nauka. (In Russian)."},{"key":"ref_38","unstructured":"Tsypkin, Y.Z. (1973). Foundations of the Theory of Learning Systems, Academic Press."},{"key":"ref_39","first-page":"1239","article-title":"Approximate solution of systems of linear equations","volume":"54","author":"Kaczmarz","year":"1993","journal-title":"Internet J. Control"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1016\/S0041-5553(83)80028-7","article-title":"On the convergence rate of minimization methods along vectors of a linearly independent system","volume":"23","author":"Krutikov","year":"1983","journal-title":"USSR Comput. Math. Math. Phys."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Rao, S.S. (2009). Engineering Optimization, Wiley.","DOI":"10.1002\/9780470549124"},{"key":"ref_42","unstructured":"Andrei, N. (2024, April 01). An Unconstrained Optimization Test Functions Collection. Available online: http:\/\/www.ici.ro\/camo\/journal\/vol10\/v10a10.pdf."}],"container-title":["Axioms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-1680\/13\/4\/240\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:23:41Z","timestamp":1760106221000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-1680\/13\/4\/240"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,5]]},"references-count":42,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2024,4]]}},"alternative-id":["axioms13040240"],"URL":"https:\/\/doi.org\/10.3390\/axioms13040240","relation":{},"ISSN":["2075-1680"],"issn-type":[{"type":"electronic","value":"2075-1680"}],"subject":[],"published":{"date-parts":[[2024,4,5]]}}}