{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,18]],"date-time":"2026-04-18T03:56:04Z","timestamp":1776484564981,"version":"3.51.2"},"reference-count":39,"publisher":"Springer Science and Business Media LLC","issue":"11","license":[{"start":{"date-parts":[[2023,9,6]],"date-time":"2023-09-06T00:00:00Z","timestamp":1693958400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,9,6]],"date-time":"2023-09-06T00:00:00Z","timestamp":1693958400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004329","name":"Javna Agencija za Raziskovalno Dejavnost RS","doi-asserted-by":"publisher","award":["P2-0103"],"award-info":[{"award-number":["P2-0103"]}],"id":[{"id":"10.13039\/501100004329","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004329","name":"Javna Agencija za Raziskovalno Dejavnost RS","doi-asserted-by":"publisher","award":["N2-0128"],"award-info":[{"award-number":["N2-0128"]}],"id":[{"id":"10.13039\/501100004329","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2023,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We propose an approach to symbolic regression based on a novel variational autoencoder for generating hierarchical structures, HVAE. It combines simple atomic units with shared weights to recursively encode and decode the individual nodes in the hierarchy. Encoding is performed bottom-up and decoding top-down. We empirically show that HVAE can be trained efficiently with small corpora of mathematical expressions and can accurately encode expressions into a smooth low-dimensional latent space. The latter can be efficiently explored with various optimization methods to address the task of symbolic regression. Indeed, random search through the latent space of HVAE performs better than random search through expressions generated by manually crafted probabilistic grammars for mathematical expressions. Finally, EDHiE system for symbolic regression, which applies an evolutionary algorithm to the latent space of HVAE, reconstructs equations from a standard symbolic regression benchmark better than a state-of-the-art system based on a similar combination of deep learning and evolutionary algorithms.<\/jats:p>","DOI":"10.1007\/s10994-023-06400-2","type":"journal-article","created":{"date-parts":[[2023,9,6]],"date-time":"2023-09-06T19:01:28Z","timestamp":1694026888000},"page":"4563-4596","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Efficient generator of mathematical expressions for symbolic regression"],"prefix":"10.1007","volume":"112","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0469-6696","authenticated-orcid":false,"given":"Sebastian","family":"Me\u017enar","sequence":"first","affiliation":[]},{"given":"Sa\u0161o","family":"D\u017eeroski","sequence":"additional","affiliation":[]},{"given":"Ljup\u010do","family":"Todorovski","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,9,6]]},"reference":[{"key":"6400_CR1","unstructured":"Allamanis, M., Chanthirasegaran, P., Kohli, P., & Sutton, C. (2017). Learning continuous semantic representations of symbolic expressions. In Proceedings of the 34th international conference on machine learning (Vol. 70, pp. 80\u201388). JMLR.org"},{"key":"6400_CR2","unstructured":"Alvarez-Melis, D., & Jaakkola, T. (2017). Tree-structured decoding with doubly-recurrent neural networks. In ICLR."},{"key":"6400_CR3","unstructured":"Bahuleyan, H. (2018). Natural language generation with neural variational models."},{"key":"6400_CR4","doi-asserted-by":"crossref","unstructured":"Bielik, P., Raychev, V., & Vechev, M. (2016). PHOG: Probabilistic model for code. In Balcan, M. F., & Weinberger, K. Q. (eds.), Proceedings of the thirty-third international conference on machine learning. Proceedings of machine learning research, New York, New York, USA (Vol. 48, pp. 2933\u20132942).","DOI":"10.1145\/2983990.2984041"},{"key":"6400_CR5","unstructured":"Biggio, L., Bendinelli, T., Neitz, A., Lucchi, A., & Parascandolo, G. (2021). Neural symbolic regression that scales. In Proceedings of 38th international conference on machine learning (ICML). Proceedings of machine learning research (Vol. 139, pp. 936\u2013945). PMLR. https:\/\/proceedings.mlr.press\/v139\/biggio21a.html"},{"key":"6400_CR6","doi-asserted-by":"publisher","first-page":"89497","DOI":"10.1109\/ACCESS.2020.2990567","volume":"8","author":"J Blank","year":"2020","unstructured":"Blank, J., & Deb, K. (2020). pymoo: Multi-objective optimization in python. IEEE Access, 8, 89497\u201389509.","journal-title":"IEEE Access"},{"key":"6400_CR7","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2021.107077","volume":"224","author":"J Brence","year":"2021","unstructured":"Brence, J., Todorovski, L., & D\u017eeroski, S. (2021). Probabilistic grammars for equation discovery. Knowledge-Based Systems, 224, 107077. https:\/\/doi.org\/10.1016\/j.knosys.2021.107077","journal-title":"Knowledge-Based Systems"},{"issue":"1","key":"6400_CR8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10994-007-5042-6","volume":"71","author":"W Bridewell","year":"2008","unstructured":"Bridewell, W., Langley, P., Todorovski, L., & D\u017eeroski, S. (2008). Inductive process modeling. Machine Learning, 71(1), 1\u201332. https:\/\/doi.org\/10.1007\/s10994-007-5042-6","journal-title":"Machine Learning"},{"key":"6400_CR9","doi-asserted-by":"publisher","unstructured":"Cho, K., van Merrienboer, B., Bahdanau, D., & Bengio, Y. (2014a). On the properties of neural machine translation: Encoder\u2013decoder approaches. https:\/\/doi.org\/10.48550\/ARXIV.1409.1259","DOI":"10.48550\/ARXIV.1409.1259"},{"key":"6400_CR10","doi-asserted-by":"publisher","unstructured":"Cho, K., van Merri\u00ebnboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014b). Learning phrase representations using RNN encoder\u2013decoder for statistical machine translation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1724\u20131734). Association for Computational Linguistics, Doha, Qatar. https:\/\/doi.org\/10.3115\/v1\/D14-1179","DOI":"10.3115\/v1\/D14-1179"},{"key":"6400_CR11","unstructured":"Cranmer, M. (2023). Interpretable machine learning for science with PySR and SymbolicRegression.jl"},{"key":"6400_CR12","unstructured":"Dai, H., Tian, Y., Dai, B., Skiena, S., & Song, L. (2018). Syntax-directed variational autoencoder for structured data"},{"key":"6400_CR13","unstructured":"d\u2019Ascoli, S., Kamienny, P.-A., Lample, G., & Charton, F. (2022). Deep symbolic regression for recurrent sequences. arXiv preprint arXiv:2201.04600"},{"issue":"2","key":"6400_CR14","doi-asserted-by":"publisher","first-page":"268","DOI":"10.1021\/acscentsci.7b00572","volume":"4","author":"R G\u00f3mez-Bombarelli","year":"2018","unstructured":"G\u00f3mez-Bombarelli, R., Wei, J. N., Duvenaud, D., Hern\u00e1ndez-Lobato, J. M., S\u00e1nchez-Lengeling, B., Sheberla, D., Aguilera-Iparraguirre, J., Hirzel, T. D., Adams, R. P., & Aspuru-Guzik, A. (2018). Automatic chemical design using a data-driven continuous representation of molecules. ACS Central Science, 4(2), 268\u2013276. https:\/\/doi.org\/10.1021\/acscentsci.7b00572","journal-title":"ACS Central Science"},{"issue":"5","key":"6400_CR15","doi-asserted-by":"publisher","first-page":"6971","DOI":"10.1126\/sciadv.aav6971","volume":"6","author":"R Guimer\u00e0","year":"2020","unstructured":"Guimer\u00e0, R., Reichardt, I., Aguilar-Mogas, A., Massucci, F. A., Miranda, M., Pallar\u00e8s, J., & Sales-Pardo, M. (2020). A Bayesian machine scientist to aid in the solution of challenging scientific problems. Science Advances, 6(5), 6971. https:\/\/doi.org\/10.1126\/sciadv.aav6971","journal-title":"Science Advances"},{"key":"6400_CR16","doi-asserted-by":"publisher","unstructured":"Janz, D., van\u00a0der Westhuizen, J., Paige, B., Kusner, M. J., & Hern\u00e1ndez-Lobato, J. M. (2017). Learning a generative model for validity in complex discrete structures. In The sixth international conference on learning representations. https:\/\/doi.org\/10.48550\/ARXIV.1712.01664","DOI":"10.48550\/ARXIV.1712.01664"},{"key":"6400_CR17","doi-asserted-by":"publisher","unstructured":"Jin, W., Barzilay, R., & Jaakkola, T. (2021). Junction tree variational autoencoder for molecular graph generation. In Artificial intelligence in drug discovery (pp. 228\u2013249). The Royal Society of Chemistry. https:\/\/doi.org\/10.1039\/9781788016841-00228","DOI":"10.1039\/9781788016841-00228"},{"key":"6400_CR18","unstructured":"Kamienny, P.-A., d\u2019Ascoli, S., Lample, G., & Charton, F. (2022). End-to-end symbolic regression with transformers. arXiv:2204.10532"},{"key":"6400_CR19","unstructured":"Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. In Y. Bengio, Y. LeCun (eds.), 3rd international conference on learning representations, ICLR 2015, San Diego, CA, USA, May 7\u20139, 2015, conference track proceedings."},{"key":"6400_CR20","unstructured":"Kingma, D. P., & Welling, M. (2014) Auto-encoding variational Bayes. In 2nd international conference on learning representations, ICLR 2014, Banff, AB, Canada, April 14\u201316, 2014, Conference Track Proceedings."},{"issue":"2","key":"6400_CR21","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1007\/BF00175355","volume":"4","author":"JR Koza","year":"1994","unstructured":"Koza, J. R. (1994). Genetic programming as a means for programming computers by natural selection. Statistics and Computing, 4(2), 87\u2013112. https:\/\/doi.org\/10.1007\/BF00175355","journal-title":"Statistics and Computing"},{"issue":"1","key":"6400_CR22","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1214\/aoms\/1177729694","volume":"22","author":"S Kullback","year":"1951","unstructured":"Kullback, S., & Leibler, R. A. (1951). On information and sufficiency. The Annals of Mathematical Statistics, 22(1), 79\u201386. https:\/\/doi.org\/10.1214\/aoms\/1177729694","journal-title":"The Annals of Mathematical Statistics"},{"key":"6400_CR23","unstructured":"Kusner, M. J., & Hern\u00e1ndez-Lobato, J. M. (2016). GANS for sequences of discrete elements with the Gumbel-softmax distribution. arxiv:1611.04051"},{"key":"6400_CR24","unstructured":"Kusner, M. J., Paige, B., Hern\u00e1ndez-Lobato, J. M. (2017). Grammar variational autoencoder. In: Proceedings of the 34th international conference on machine learning\u2014volume 70. ICML\u201917, pp. 1945\u20131954. JMLR.org. arxiv:1703.01925"},{"key":"6400_CR25","first-page":"707","volume":"10","author":"VI Levenshtein","year":"1965","unstructured":"Levenshtein, V. I. (1965). Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10, 707\u2013710.","journal-title":"Soviet Physics Doklady"},{"key":"6400_CR26","unstructured":"Martius, G., & Lampert, C. H. (2016). Extrapolation and learning equations."},{"key":"6400_CR27","doi-asserted-by":"publisher","first-page":"103","DOI":"10.7717\/peerj-cs.103","volume":"3","author":"A Meurer","year":"2017","unstructured":"Meurer, A., Smith, C. P., Paprocki, M., \u010cert\u00edk, O., Kirpichev, S. B., Rocklin, M., Kumar, A., Ivanov, S., Moore, J. K., Singh, S., Rathnayake, T., Vig, S., Granger, B. E., Muller, R. P., Bonazzi, F., Gupta, H., Vats, S., Johansson, F., Pedregosa, F., \u2026 Scopatz, A. (2017). Sympy: Symbolic computing in python. PeerJ Computer Science, 3, 103. https:\/\/doi.org\/10.7717\/peerj-cs.103","journal-title":"PeerJ Computer Science"},{"key":"6400_CR28","unstructured":"Mundhenk, T. N., Landajuela, M., Glatt, R., Santiago, C. P., Faissol, D. M., & Petersen, B. K. (2021). Symbolic regression via neural-guided genetic programming population seeding. arXiv:2111.00053"},{"key":"6400_CR29","unstructured":"Petersen, B. K., Landajuela, M., Mundhenk, T. N., Santiago, C. P., Kim, S. K., & Kim, J. T. (2021). Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In Proceedings of the international conference on learning representations."},{"key":"6400_CR30","doi-asserted-by":"publisher","DOI":"10.1016\/j.jhydrol.2021.127077","volume":"603","author":"M Radinja","year":"2021","unstructured":"Radinja, M., \u0160kerjanec, M., \u0160raj, M., D\u017eeroski, S., Todorovski, L., & Atanasova, N. (2021). Automated modelling of urban runoff based on domain knowledge and equation discovery. Journal of Hydrology, 603, 127077. https:\/\/doi.org\/10.1016\/j.jhydrol.2021.127077","journal-title":"Journal of Hydrology"},{"issue":"5923","key":"6400_CR31","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1126\/science.1165893","volume":"324","author":"M Schmidt","year":"2009","unstructured":"Schmidt, M., & Lipson, H. (2009). Distilling free-form natural laws from experimental data. Science, 324(5923), 81\u201385. https:\/\/doi.org\/10.1126\/science.1165893","journal-title":"Science"},{"key":"6400_CR32","doi-asserted-by":"publisher","DOI":"10.1016\/j.physd.2019.132306","volume":"404","author":"A Sherstinsky","year":"2020","unstructured":"Sherstinsky, A. (2020). Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network. Physica D: Nonlinear Phenomena, 404, 132306. https:\/\/doi.org\/10.1016\/j.physd.2019.132306","journal-title":"Physica D: Nonlinear Phenomena"},{"key":"6400_CR33","doi-asserted-by":"publisher","first-page":"29930","DOI":"10.1109\/access.2020.2972076","volume":"8","author":"N Simidjievski","year":"2020","unstructured":"Simidjievski, N., Todorovski, L., Kocijan, J., & Dzeroski, S. (2020). Equation discovery for nonlinear system identification. IEEE Access, 8, 29930\u201329943. https:\/\/doi.org\/10.1109\/access.2020.2972076","journal-title":"IEEE Access"},{"key":"6400_CR34","unstructured":"Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. In Proceedings of the 27th international conference on neural information processing systems\u2014volume 2. NIPS\u201914 (pp. 3104\u20133112). MIT Press"},{"key":"6400_CR35","doi-asserted-by":"publisher","unstructured":"Tai, K. S., Socher, R., & Manning, C. D. (2015). Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd annual meeting of the Association for Computational Linguistics and the 7th international joint conference on natural language processing (Vol. 1: Long Papers, pp. 1556\u20131566). Association for Computational Linguistics, Beijing, China. https:\/\/doi.org\/10.3115\/v1\/P15-1150","DOI":"10.3115\/v1\/P15-1150"},{"key":"6400_CR36","unstructured":"Todorovski, L., & Dzeroski, S. (1997). Declarative bias in equation discovery. In Proceedings of the fourteenth international conference on machine learning. ICML \u201997 (pp. 376\u2013384). Morgan Kaufmann Publishers Inc."},{"key":"6400_CR37","doi-asserted-by":"publisher","first-page":"410","DOI":"10.1007\/978-1-4899-7687-1_258","volume-title":"Encyclopedia of machine learning and data mining, Chap. 5","author":"L Todorovski","year":"2017","unstructured":"Todorovski, L. (2017). Equation discovery. In C. Sammut & G. I. Webb (Eds.), Encyclopedia of machine learning and data mining, Chap. 5 (pp. 410\u2013414). Springer. https:\/\/doi.org\/10.1007\/978-1-4899-7687-1_258"},{"issue":"16","key":"6400_CR38","doi-asserted-by":"publisher","first-page":"2631","DOI":"10.1126\/sciadv.aay2631","volume":"6","author":"S-M Udrescu","year":"2020","unstructured":"Udrescu, S.-M., & Tegmark, M. (2020). AI Feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16), 2631.","journal-title":"Science Advances"},{"issue":"2","key":"6400_CR39","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1007\/s10710-010-9121-2","volume":"12","author":"NQ Uy","year":"2011","unstructured":"Uy, N. Q., Hoai, N. X., O\u2019Neill, M., McKay, R. I., & Galv\u00e1n-L\u00f3pez, E. (2011). Semantically-based crossover in genetic programming: Application to real-valued symbolic regression. Genetic Programming and Evolvable Machines, 12(2), 91\u2013119.","journal-title":"Genetic Programming and Evolvable Machines"}],"updated-by":[{"DOI":"10.1007\/s10994-023-06407-9","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2023,9,21]],"date-time":"2023-09-21T00:00:00Z","timestamp":1695254400000}}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-023-06400-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-023-06400-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-023-06400-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,25]],"date-time":"2023-10-25T17:13:15Z","timestamp":1698253995000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-023-06400-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,6]]},"references-count":39,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2023,11]]}},"alternative-id":["6400"],"URL":"https:\/\/doi.org\/10.1007\/s10994-023-06400-2","relation":{"correction":[{"id-type":"doi","id":"10.1007\/s10994-023-06407-9","asserted-by":"object"}]},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,6]]},"assertion":[{"value":"14 February 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 June 2023","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 August 2023","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 September 2023","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 September 2023","order":5,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Correction","order":6,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"A Correction to this paper has been published:","order":7,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"https:\/\/doi.org\/10.1007\/s10994-023-06407-9","URL":"https:\/\/doi.org\/10.1007\/s10994-023-06407-9","order":8,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors acknowledge the financial support of the Slovenian Research Agency via the research core Funding No.\u00a0P2-0103, Project No.\u00a0N2-0128, and by the ARRS Grant for young researchers (first author).","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Funding"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}},{"value":"Not applicable.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"Not applicable.","order":6,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}}]}}