{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T22:15:44Z","timestamp":1792102544215,"version":"4.4.0"},"reference-count":43,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2023,1,15]],"date-time":"2023-01-15T00:00:00Z","timestamp":1673740800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"The Casey Family Foundation","award":["2141064"],"award-info":[{"award-number":["2141064"]}]},{"name":"The Casey Family Foundation","award":["PHY-2019786"],"award-info":[{"award-number":["PHY-2019786"]}]},{"name":"Foundational Questions Institute","award":["2141064"],"award-info":[{"award-number":["2141064"]}]},{"name":"Foundational Questions Institute","award":["PHY-2019786"],"award-info":[{"award-number":["PHY-2019786"]}]},{"name":"Rothberg Family Fund for Cognitive Science, the NSF Graduate Research Fellowship","award":["2141064"],"award-info":[{"award-number":["2141064"]}]},{"name":"Rothberg Family Fund for Cognitive Science, the NSF Graduate Research Fellowship","award":["PHY-2019786"],"award-info":[{"award-number":["PHY-2019786"]}]},{"name":"IAIFI through NSF","award":["2141064"],"award-info":[{"award-number":["2141064"]}]},{"name":"IAIFI through NSF","award":["PHY-2019786"],"award-info":[{"award-number":["PHY-2019786"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>We explore unique considerations involved in fitting machine learning (ML) models to data with very high precision, as is often required for science applications. We empirically compare various function approximation methods and study how they scale with increasing parameters and data. We find that neural networks (NNs) can often outperform classical approximation methods on high-dimensional examples, by (we hypothesize) auto-discovering and exploiting modular structures therein. However, neural networks trained with common optimizers are less powerful for low-dimensional cases, which motivates us to study the unique properties of neural network loss landscapes and the corresponding optimization challenges that arise in the high precision regime. To address the optimization issue in low dimensions, we develop training tricks which enable us to train neural networks to extremely low loss, close to the limits allowed by numerical precision.<\/jats:p>","DOI":"10.3390\/e25010175","type":"journal-article","created":{"date-parts":[[2023,1,16]],"date-time":"2023-01-16T02:29:55Z","timestamp":1673836195000},"page":"175","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":39,"title":["Precision Machine Learning"],"prefix":"10.3390","volume":"25","author":[{"given":"Eric J.","family":"Michaud","sequence":"first","affiliation":[{"name":"Department of Physics, Massachusetts Institute of Technology (MIT), Cambridge, MA 02139, USA"},{"name":"NSF AI Institute for AI and Fundamental Interactions, Cambridge, MA 02139, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ziming","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Physics, Massachusetts Institute of Technology (MIT), Cambridge, MA 02139, USA"},{"name":"NSF AI Institute for AI and Fundamental Interactions, Cambridge, MA 02139, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7670-7190","authenticated-orcid":false,"given":"Max","family":"Tegmark","sequence":"additional","affiliation":[{"name":"Department of Physics, Massachusetts Institute of Technology (MIT), Cambridge, MA 02139, USA"},{"name":"NSF AI Institute for AI and Fundamental Interactions, Cambridge, MA 02139, USA"},{"name":"Center for Brains, Minds and Machines, Massachusetts Institute of Technology, Cambridge, MA 02139, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,1,15]]},"reference":[{"key":"ref_1","first-page":"1737","article-title":"Deep Learning with Limited Numerical Precision","volume":"Volume 37","author":"Bach","year":"2015","journal-title":"Proceedings of the 32nd International Conference on Machine Learning"},{"key":"ref_2","unstructured":"Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., and Venkatesh, G. (2017). Mixed precision training. arXiv."},{"key":"ref_3","unstructured":"Kalamkar, D., Mudigere, D., Mellempudi, N., Das, D., Banerjee, K., Avancha, S., Vooturi, D.T., Jammalamadaka, N., Huang, J., and Yuen, H. (2019). A study of BFLOAT16 for deep learning training. arXiv."},{"key":"ref_4","unstructured":"Wang, Y., Lai, C.Y., G\u00f3mez-Serrano, J., and Buckmaster, T. (2022). Asymptotic self-similar blow up profile for 3-D Euler via physics-informed neural networks. arXiv."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1007\/JHEP08(2022)105","article-title":"Neural network approximations for Calabi-Yau metrics","volume":"2022","author":"Jejjala","year":"2022","journal-title":"J. High Energy Phys."},{"key":"ref_6","unstructured":"Martyn, J., Luo, D., and Najafi, K. (2023). Applying the Variational Principle to Quantum Field Theory with Neural-Networks, American Physical Society. Bulletin of the American Physical Society."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"080602","DOI":"10.1103\/PhysRevLett.122.080602","article-title":"Solving statistical mechanics using variational autoregressive networks","volume":"122","author":"Wu","year":"2019","journal-title":"Phys. Rev. Lett."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"eaay2631","DOI":"10.1126\/sciadv.aay2631","article-title":"AI Feynman: A physics-inspired method for symbolic regression","volume":"6","author":"Udrescu","year":"2020","journal-title":"Sci. Adv."},{"key":"ref_9","first-page":"4860","article-title":"AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity","volume":"33","author":"Udrescu","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_10","unstructured":"Leighton, R.B., and Sands, M. (1965). The Feynman Lectures on Physics, Addison-Wesley."},{"key":"ref_11","unstructured":"(2019). IEEE Std 754-2019 (Revision of IEEE 754-2008), IEEE. IEEE Standard for Floating-Point Arithmetic."},{"key":"ref_12","unstructured":"G\u00fchring, I., Raslan, M., and Kutyniok, G. (2020). Expressivity of deep neural networks. arXiv."},{"key":"ref_13","unstructured":"Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M., Ali, M., Yang, Y., and Zhou, Y. (2017). Deep learning scaling is predictable, empirically. arXiv."},{"key":"ref_14","unstructured":"Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv."},{"key":"ref_15","unstructured":"Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T.B., Dhariwal, P., and Gray, S. (2020). Scaling laws for autoregressive generative modeling. arXiv."},{"key":"ref_16","unstructured":"Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. (2021). Scaling laws for transfer. arXiv."},{"key":"ref_17","unstructured":"Ghorbani, B., Firat, O., Freitag, M., Bapna, A., Krikun, M., Garcia, X., Chelba, C., and Cherry, C. (2021). Scaling laws for neural machine translation. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Gordon, M.A., Duh, K., and Kaplan, J. (2021, January 7\u201311). Data and parameter scaling laws for neural machine translation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic.","DOI":"10.18653\/v1\/2021.emnlp-main.478"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. (2022, January 18\u201324). Scaling vision transformers. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01179"},{"key":"ref_20","unstructured":"Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D.d.L., Hendricks, L.A., Welbl, J., and Clark, A. (2022). Training Compute-Optimal Large Language Models. arXiv."},{"key":"ref_21","unstructured":"Clark, A., de Las Casas, D., Guy, A., Mensch, A., Paganini, M., Hoffmann, J., Damoc, B., Hechtman, B., Cai, T., and Borgeaud, S. (2022, January 17\u201323). Unified scaling laws for routed language models. Proceedings of the International Conference on Machine Learning, Baltimore, MD, USA."},{"key":"ref_22","unstructured":"Sharma, U., and Kaplan, J. (2020). A neural scaling law from the dimension of the data manifold. arXiv."},{"key":"ref_23","unstructured":"Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. (2021). Explaining neural scaling laws. arXiv."},{"key":"ref_24","unstructured":"Arora, R., Basu, A., Mianjy, P., and Mukherjee, A. (2016). Understanding deep neural networks with rectified linear units. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"232","DOI":"10.1016\/j.neunet.2018.11.005","article-title":"A comparison of deep networks with ReLU activation function and linear spline-type methods","volume":"110","author":"Eckle","year":"2019","journal-title":"Neural Netw."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1038\/s41592-019-0686-2","article-title":"SciPy 1.0: Fundamental algorithms for scientific computing in Python","volume":"17","author":"Virtanen","year":"2020","journal-title":"Nat. Methods"},{"key":"ref_27","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"LeCun, Y.A., Bottou, L., Orr, G.B., and M\u00fcller, K.R. (2012). Efficient backprop. Neural Networks: Tricks of the Trade, Springer.","DOI":"10.1007\/978-3-642-35289-8_3"},{"key":"ref_29","unstructured":"Poggio, T. (2022). Compositional Sparsity: A Framework for ML, Center for Brains, Minds and Machines (CBMM). Technical Report."},{"key":"ref_30","unstructured":"Dahmen, W. (2022). Compositional Sparsity, Approximation Classes, and Parametric Transport Equations. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1007\/s11633-017-1054-2","article-title":"Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review","volume":"14","author":"Poggio","year":"2017","journal-title":"Int. J. Autom. Comput."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"2231","DOI":"10.1214\/20-AOS2034","article-title":"On the rate of convergence of fully connected deep neural network regression estimates","volume":"49","author":"Kohler","year":"2021","journal-title":"Ann. Stat."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"2261","DOI":"10.1214\/18-AOS1747","article-title":"On deep learning as a remedy for the curse of dimensionality in nonparametric regression","volume":"47","author":"Bauer","year":"2019","journal-title":"Ann. Stat."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1002\/nme.1296","article-title":"Tricubic interpolation in three dimensions","volume":"63","author":"Lekien","year":"2005","journal-title":"Int. J. Numer. Methods Eng."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1223","DOI":"10.1007\/s10955-017-1836-5","article-title":"Why does deep and cheap learning work so well?","volume":"168","author":"Lin","year":"2017","journal-title":"J. Stat. Phys."},{"key":"ref_36","unstructured":"Sagun, L., Bottou, L., and LeCun, Y. (2016). Eigenvalues of the hessian in deep learning: Singularity and beyond. arXiv."},{"key":"ref_37","unstructured":"Sagun, L., Evci, U., Guney, V.U., Dauphin, Y., and Bottou, L. (2017). Empirical analysis of the hessian of over-parametrized neural networks. arXiv."},{"key":"ref_38","unstructured":"Gur-Ari, G., Roberts, D.A., and Dyer, E. (2018). Gradient descent happens in a tiny subspace. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Nocedal, J., and Wright, S.J. (1999). Numerical Optimization, Springer.","DOI":"10.1007\/b98874"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Chen, T., and Guestrin, C. (2016, January 13\u201317). Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939785"},{"key":"ref_41","first-page":"647","article-title":"Training methods for adaptive boosting of neural networks","volume":"10","author":"Schwenk","year":"1997","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1869","DOI":"10.1162\/089976600300015178","article-title":"Boosting neural networks","volume":"12","author":"Schwenk","year":"2000","journal-title":"Neural Comput."},{"key":"ref_43","unstructured":"Badirli, S., Liu, X., Xing, Z., Bhowmik, A., Doan, K., and Keerthi, S.S. (2020). Gradient boosting neural networks: Grownet. arXiv."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/1\/175\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:06:43Z","timestamp":1760119603000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/1\/175"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,15]]},"references-count":43,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,1]]}},"alternative-id":["e25010175"],"URL":"https:\/\/doi.org\/10.3390\/e25010175","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,15]]}}}