{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,12]],"date-time":"2026-01-12T21:21:35Z","timestamp":1768252895414,"version":"3.49.0"},"reference-count":35,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2021,3,2]],"date-time":"2021-03-02T00:00:00Z","timestamp":1614643200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,3,2]],"date-time":"2021-03-02T00:00:00Z","timestamp":1614643200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000066","name":"National Institute of Environmental Health Sciences","doi-asserted-by":"publisher","award":["T32-ES019851"],"award-info":[{"award-number":["T32-ES019851"]}],"id":[{"id":"10.13039\/100000066","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"U.S. National Library of Medicine","doi-asserted-by":"publisher","award":["R01-LM010098"],"award-info":[{"award-number":["R01-LM010098"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"U.S. National Library of Medicine","doi-asserted-by":"publisher","award":["R01-LM012601"],"award-info":[{"award-number":["R01-LM012601"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000060","name":"National Institute of Allergy and Infectious Diseases","doi-asserted-by":"publisher","award":["R01-AI116794"],"award-info":[{"award-number":["R01-AI116794"]}],"id":[{"id":"10.13039\/100000060","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006108","name":"National Center for Advancing Translational Sciences","doi-asserted-by":"publisher","award":["UL1-TR001878"],"award-info":[{"award-number":["UL1-TR001878"]}],"id":[{"id":"10.13039\/100006108","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000062","name":"National Institute of Diabetes and Digestive and Kidney Diseases","doi-asserted-by":"publisher","award":["UC4-DK112217"],"award-info":[{"award-number":["UC4-DK112217"]}],"id":[{"id":"10.13039\/100000062","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Genet Program Evolvable Mach"],"published-print":{"date-parts":[[2021,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Automated machine learning (AutoML) and artificial neural networks (ANNs) have revolutionized the field of artificial intelligence by yielding incredibly high-performing models to solve a myriad of inductive learning tasks. In spite of their successes, little guidance exists on when to use one versus the other. Furthermore, relatively few tools exist that allow the integration of both AutoML and ANNs in the same analysis to yield results combining both of their strengths. Here, we present TPOT-NN\u2014a new extension to the tree-based AutoML software TPOT\u2014and use it to explore the behavior of automated machine learning augmented with neural network estimators (AutoML+NN), particularly when compared to non-NN AutoML in the context of simple binary classification on a number of public benchmark datasets. Our observations suggest that TPOT-NN is an effective tool that achieves greater classification accuracy than standard tree-based AutoML on some datasets, with no loss in accuracy on others. We also provide preliminary guidelines for performing AutoML+NN analyses, and recommend possible future directions for AutoML+NN methods research, especially in the context of TPOT.<\/jats:p>","DOI":"10.1007\/s10710-021-09401-z","type":"journal-article","created":{"date-parts":[[2021,3,2]],"date-time":"2021-03-02T18:03:16Z","timestamp":1614708196000},"page":"207-227","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":28,"title":["TPOT-NN: augmenting tree-based automated machine learning with neural network estimators"],"prefix":"10.1007","volume":"22","author":[{"given":"Joseph D.","family":"Romano","sequence":"first","affiliation":[]},{"given":"Trang T.","family":"Le","sequence":"additional","affiliation":[]},{"given":"Weixuan","family":"Fu","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5015-1099","authenticated-orcid":false,"given":"Jason H.","family":"Moore","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,3,2]]},"reference":[{"key":"9401_CR1","doi-asserted-by":"crossref","unstructured":"P. Auer, H. Burgsteiner, W. Maass, Reducing communication for distributed learning in neural networks, in International Conference on Artificial Neural Networks (Springer, 2002), pp.\u00a0123\u2013128","DOI":"10.1007\/3-540-46084-5_21"},{"key":"9401_CR2","doi-asserted-by":"publisher","DOI":"10.1007\/BFb0055923","volume-title":"Genetic Programming","author":"W Banzhaf","year":"1998","unstructured":"W. Banzhaf, P. Nordin, R.E. Keller, F.D. Francone, Genetic Programming (Springer, Berlin, 1998)"},{"issue":"32","key":"9401_CR3","doi-asserted-by":"publisher","first-page":"15849","DOI":"10.1073\/pnas.1903070116","volume":"116","author":"M Belkin","year":"2019","unstructured":"M. Belkin, D. Hsu, S. Ma, S. Mandal, Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proc. Nat. Acad. Sci. 116(32), 15849\u201315854 (2019)","journal-title":"Proc. Nat. Acad. Sci."},{"issue":"346","key":"9401_CR4","doi-asserted-by":"publisher","first-page":"364","DOI":"10.1080\/01621459.1974.10482955","volume":"69","author":"MB Brown","year":"1974","unstructured":"M.B. Brown, A.B. Forsythe, Robust tests for the equality of variances. J. Am. Stat. Assoc. 69(346), 364\u2013367 (1974)","journal-title":"J. Am. Stat. Assoc."},{"key":"9401_CR5","doi-asserted-by":"crossref","unstructured":"T. Chen, C. Xgboost Guestrin, A scalable tree boosting system, in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2016), pp.\u00a0785\u2013794","DOI":"10.1145\/2939672.2939785"},{"key":"9401_CR6","doi-asserted-by":"crossref","unstructured":"R. Collobert, & S. Bengio, Links between perceptrons, mlps and svms, in Proceedings of the Twenty-first International Conference on Machine Learning (2004), p.\u00a023","DOI":"10.1145\/1015330.1015415"},{"key":"9401_CR7","unstructured":"F.-M. De\u00a0Rainville, F.-A. Fortin, M.-A. Gardner, M. Parizeau, C. Gagn\u00e9, DEAP: a python framework for evolutionary algorithms, in Proceedings of the 14th Annual Conference Companion on Genetic and Evolutionary Computation (2012), pp.\u00a085\u201392"},{"issue":"2","key":"9401_CR8","doi-asserted-by":"publisher","first-page":"182","DOI":"10.1109\/4235.996017","volume":"6","author":"K Deb","year":"2002","unstructured":"K. Deb, A. Pratap, S. Agarwal, T. Meyarivan, A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182\u2013197 (2002)","journal-title":"IEEE Trans. Evol. Comput."},{"key":"9401_CR9","unstructured":"D. Dua, C. Graff, UCI machine learning repository (2017). https:\/\/archive.ics.uci.edu\/ml\/index.php"},{"key":"9401_CR10","unstructured":"N. Erickson, J. Mueller, A. Shirkov, H. Zhang, P. Larroy, M. Li, A. Smola, Autogluon-tabular: robust and accurate automl for structured data. arXiv:2003.06505 (2020)"},{"key":"9401_CR11","unstructured":"S. Fahlman, C. Lebiere, The cascade-correlation learning architecture, in Advances in Neural Information Processing Systems, D.\u00a0Touretzky, Ed., vol.\u00a02, (Morgan-Kaufmann, 1990), pp.\u00a0524\u2013532"},{"key":"9401_CR12","unstructured":"W. Fu, R. Olson, J.G. Nathan, T. Augspurger, J. Romano, P. Saha, S. Shah, S. Raschka, S. DanKoretsky, K. Jaimecclin, bartdp1, G. Bradway, J. Ortiz, J.J. Smit, J.-H. Menke, M. Ficek, A. Varik, A. Chaves, J. Myatt, Ted, A.G. Badaracco, C. Kastner, C. Jer\u00f4nimo, Hristo, M. Rocklin, R. Carnevale. Epistasislab\/tpot: v0.11.5 (2020)"},{"key":"9401_CR13","unstructured":"E. Galv\u00e1n, P. Mooney, Neuroevolution in deep neural networks: current trends and future challenges. arXiv:2006.05415 (2020)"},{"key":"9401_CR14","doi-asserted-by":"crossref","unstructured":"K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 770\u2013778","DOI":"10.1109\/CVPR.2016.90"},{"key":"9401_CR15","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-05318-5","volume-title":"Automated Machine Learning: Methods, Systems, Challenges","author":"F Hutter","year":"2019","unstructured":"F. Hutter, L. Kotthoff, J. Vanschoren, Automated Machine Learning: Methods, Systems, Challenges (Springer, Berlin, 2019)."},{"key":"9401_CR16","doi-asserted-by":"crossref","unstructured":"W. Iba, J. Wogulis, P. Langley, Trading off simplicity and coverage in incremental concept learning, in Machine Learning Proceedings 1988. (Elsevier, 1988), pp.\u00a073\u201379","DOI":"10.1016\/B978-0-934613-64-4.50013-X"},{"key":"9401_CR17","doi-asserted-by":"publisher","first-page":"274","DOI":"10.1016\/j.neucom.2013.04.005","volume":"121","author":"MM Khan","year":"2013","unstructured":"M.M. Khan, A.M. Ahmad, G.M. Khan, J.F. Miller, Fast learning neural networks using cartesian genetic programming. Neurocomputing 121, 274\u2013289 (2013)","journal-title":"Neurocomputing"},{"issue":"1","key":"9401_CR18","doi-asserted-by":"publisher","first-page":"250","DOI":"10.1093\/bioinformatics\/btz470","volume":"36","author":"TT Le","year":"2020","unstructured":"T.T. Le, W. Fu, J.H. Moore, Scaling tree-based automated machine learning to biomedical big data with a feature set selector. Bioinformatics 36(1), 250\u2013256 (2020)","journal-title":"Bioinformatics"},{"issue":"7553","key":"9401_CR19","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"Y. LeCun, Y. Bengio, G. Hinton, Deep learning. Nature 521(7553), 436\u2013444 (2015)","journal-title":"Nature"},{"issue":"6","key":"9401_CR20","doi-asserted-by":"publisher","first-page":"861","DOI":"10.1016\/S0893-6080(05)80131-5","volume":"6","author":"M Leshno","year":"1993","unstructured":"M. Leshno, V.Y. Lin, A. Pinkus, S. Schocken, Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Netw. 6(6), 861\u2013867 (1993)","journal-title":"Neural Netw."},{"issue":"3","key":"9401_CR21","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1145\/3236386.3241340","volume":"16","author":"ZC Lipton","year":"2018","unstructured":"Z.C. Lipton, The mythos of model interpretability. Queue 16(3), 31\u201357 (2018)","journal-title":"Queue"},{"key":"9401_CR22","doi-asserted-by":"crossref","unstructured":"Y. Lou, R. Caruana, J. Gehrke, Intelligible models for classification and regression, in Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2012), pp.\u00a0150\u2013158","DOI":"10.1145\/2339530.2339556"},{"key":"9401_CR23","doi-asserted-by":"crossref","unstructured":"J.F. Miller, Cartesian genetic programming, in Cartesian Genetic Programming (Springer, 2011), pp.\u00a017\u201334","DOI":"10.1007\/978-3-642-17310-3_2"},{"key":"9401_CR24","unstructured":"B. Neal, S. Mittal, A. Baratin, V. Tantia, M. Scicluna, S. Lacoste-Julien, I. Mitliagkas, A modern take on the bias-variance tradeoff in neural networks. arXiv:1810.08591 (2018)"},{"key":"9401_CR25","doi-asserted-by":"crossref","unstructured":"R.S. Olson, N. Bartley, R.J. Urbanowicz, J.H. Moore, Evaluation of a tree-based pipeline optimization tool for automating data science, in Proceedings of the Genetic and Evolutionary Computation Conference 2016, GECCO\u201916 (New York, NY, USA, ACM, 2016), pp.\u00a0485\u2013492","DOI":"10.1145\/2908812.2908918"},{"issue":"1","key":"9401_CR26","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1186\/s13040-017-0154-4","volume":"10","author":"RS Olson","year":"2017","unstructured":"R.S. Olson, W. La Cava, P. Orzechowski, R.J. Urbanowicz, J.H. Moore, PMLB: a large benchmark suite for machine learning evaluation and comparison. BioData Min. 10(1), 36 (2017)","journal-title":"BioData Min"},{"key":"9401_CR27","doi-asserted-by":"crossref","unstructured":"R.S. Olson, R.J. Urbanowicz, P.C. Andrews., N.A. Lavender, L.C. Kidd, J.H. Moore. Applications of Evolutionary Computation: 19th European Conference, EvoApplications 2016, Porto, Portugal, March 30\u2013April 1, 2016, Proceedings, Part I. Springer International Publishing, 2016, ch.\u00a0Automating Biomedical Data Science Through Tree-Based Pipeline Optimization, pp.\u00a0123\u2013137","DOI":"10.1007\/978-3-319-31204-0_9"},{"key":"9401_CR28","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825\u20132830 (2011)","journal-title":"J. Mach. Learn. Res."},{"key":"9401_CR29","unstructured":"J. Romano, Archive of code and data used to evaluate TPOT-NN.\u00a0Figshare\u00a0(2020).\u00a0https:\/\/doi.org\/10.6084\/m9.figshare.12542321.v1"},{"key":"9401_CR30","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1016\/j.neunet.2014.09.003","volume":"61","author":"J Schmidhuber","year":"2015","unstructured":"J. Schmidhuber, Deep learning in neural networks: an overview. Neural Netw. 61, 85\u2013117 (2015)","journal-title":"Neural Netw."},{"key":"9401_CR31","doi-asserted-by":"publisher","first-page":"399","DOI":"10.1146\/annurev-vision-091718-014951","volume":"5","author":"T Serre","year":"2019","unstructured":"T. Serre, Deep learning: the good, the bad, and the ugly. Ann. Rev. Vis. Sci. 5, 399\u2013426 (2019)","journal-title":"Ann. Rev. Vis. Sci."},{"key":"9401_CR32","doi-asserted-by":"crossref","unstructured":"S. Shi, Wang, Q., Xu, P., and Chu, X. Benchmarking state-of-the-art deep learning software tools, in 2016 7th International Conference on Cloud Computing and Big Data (CCBD) (IEEE, 2016), pp.\u00a099\u2013104","DOI":"10.1109\/CCBD.2016.029"},{"issue":"3","key":"9401_CR33","first-page":"262","volume":"10","author":"VG Sigillito","year":"1989","unstructured":"V.G. Sigillito, S.P. Wing, L.V. Hutton, K.B. Baker, Classification of radar returns from the ionosphere using neural networks. Johns Hopkins APL Tech. Digest 10(3), 262\u2013266 (1989)","journal-title":"Johns Hopkins APL Tech. Digest"},{"key":"9401_CR34","doi-asserted-by":"crossref","unstructured":"W.N. Street, W.H. Wolberg, O.L. Mangasarian, Nuclear feature extraction for breast tumor diagnosis, in Biomedical Image Processing and Biomedical Visualization, vol.\u00a01905 (International Society for Optics and Photonics, 1993), pp.\u00a0861\u2013870","DOI":"10.1117\/12.148698"},{"key":"9401_CR35","doi-asserted-by":"crossref","unstructured":"S. Zafeiriou, D. Kollias, M.A. Nicolaou, A. Papaioannou, G. Zhao, I. Kotsia, Aff-wild: valence and arousal\u2019in-the-wild\u2019challenge, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (2017), pp. 34\u201341","DOI":"10.1109\/CVPRW.2017.248"}],"container-title":["Genetic Programming and Evolvable Machines"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10710-021-09401-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10710-021-09401-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10710-021-09401-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,6,2]],"date-time":"2021-06-02T19:34:40Z","timestamp":1622662480000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10710-021-09401-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,2]]},"references-count":35,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,6]]}},"alternative-id":["9401"],"URL":"https:\/\/doi.org\/10.1007\/s10710-021-09401-z","relation":{},"ISSN":["1389-2576","1573-7632"],"issn-type":[{"value":"1389-2576","type":"print"},{"value":"1573-7632","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,2]]},"assertion":[{"value":"22 July 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 February 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 February 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 March 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Compliance with ethical standards"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}