{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T13:46:29Z","timestamp":1780321589082,"version":"3.54.1"},"reference-count":71,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2021,1,13]],"date-time":"2021-01-13T00:00:00Z","timestamp":1610496000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>In recent years, Automated Machine Learning (AutoML) has become increasingly important in Computer Science due to the valuable potential it offers. This is testified by the high number of works published in the academic field and the significant efforts made in the industrial sector. However, some problems still need to be resolved. In this paper, we review some Machine Learning (ML) models and methods proposed in the literature to analyze their strengths and weaknesses. Then, we propose their use\u2014alone or in combination with other approaches\u2014to provide possible valid AutoML solutions. We analyze those solutions from a theoretical point of view and evaluate them empirically on three Atari games from the Arcade Learning Environment. Our goal is to identify what, we believe, could be some promising ways to create truly effective AutoML frameworks, therefore able to replace the human expert as much as possible, thereby making easier the process of applying ML approaches to typical problems of specific domains. We hope that the findings of our study will provide useful insights for future research work in AutoML.<\/jats:p>","DOI":"10.3390\/computers10010011","type":"journal-article","created":{"date-parts":[[2021,1,13]],"date-time":"2021-01-13T11:52:32Z","timestamp":1610538752000},"page":"11","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":53,"title":["An Empirical Review of Automated Machine Learning"],"prefix":"10.3390","volume":"10","author":[{"given":"Lorenzo","family":"Vaccaro","sequence":"first","affiliation":[{"name":"Department of Engineering, Roma Tre University, 00146 Rome, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4953-1390","authenticated-orcid":false,"given":"Giuseppe","family":"Sansonetti","sequence":"additional","affiliation":[{"name":"Department of Engineering, Roma Tre University, 00146 Rome, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0495-5272","authenticated-orcid":false,"given":"Alessandro","family":"Micarelli","sequence":"additional","affiliation":[{"name":"Department of Engineering, Roma Tre University, 00146 Rome, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,1,13]]},"reference":[{"key":"ref_1","unstructured":"Caldarelli, S., Feltoni Gurini, D., Micarelli, A., and Sansonetti, G. (2016). A Signal-Based Approach to News Recommendation. CEUR Workshop Proceedings, CEUR-WS.org."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Biancalana, C., Gasparetti, F., Micarelli, A., Miola, A., and Sansonetti, G. (2011, January 27). Context-aware Movie Recommendation Based on Signal Processing and Machine Learning. Proceedings of the 2nd Challenge on Context-Aware Movie Recommendation, CAMRa \u201911, Chicago, IL, USA.","DOI":"10.1145\/2096112.2096114"},{"key":"ref_3","first-page":"55","article-title":"A Comparative Analysis of Personality-Based Music Recommender Systems","volume":"Volume 1680","author":"Onori","year":"2016","journal-title":"CEUR Workshop Proceedings"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1007\/s11257-019-09225-8","article-title":"Enhancing Cultural Recommendations through Social and Linked Open Data","volume":"29","author":"Sansonetti","year":"2019","journal-title":"User Model. User-Adapt. Interact."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1007\/s00779-019-01218-z","article-title":"Point of Interest Recommendation Based on Social and Linked Open Data","volume":"23","author":"Sansonetti","year":"2019","journal-title":"Pers. Ubiquitous Comput."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1007\/s00779-018-01189-7","article-title":"Exploiting Semantics for Context-Aware Itinerary Recommendation","volume":"23","author":"Fogli","year":"2019","journal-title":"Pers. Ubiquitous Comput."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"430","DOI":"10.1016\/j.future.2017.03.020","article-title":"Temporal People-to-people Recommendation on Social Networks with Sentiment-based Matrix Factorization","volume":"78","author":"Gasparetti","year":"2018","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1109\/4235.585893","article-title":"No free lunch theorems for optimization","volume":"1","author":"Wolpert","year":"1997","journal-title":"IEEE Trans. Evol. Comput."},{"key":"ref_9","unstructured":"Yao, Q., Wang, M., Escalante, H.J., Guyon, I., Hu, Y., Li, Y., Tu, W., Yang, Q., and Yu, Y. (2018). Taking Human out of Learning Applications: A Survey on Automated Machine Learning. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"101822","DOI":"10.1016\/j.artmed.2020.101822","article-title":"Automated machine learning: Review of the state-of-the-art and opportunities for healthcare","volume":"104","author":"Waring","year":"2020","journal-title":"Artif. Intell. Med."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Hilbert, D. (1928). Die grundlagen der mathematik. Die Grundlagen der Mathematik, Springer.","DOI":"10.1007\/978-3-663-16102-8"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"345","DOI":"10.2307\/2371045","article-title":"An Unsolvable Problem of Elementary Number Theory","volume":"58","author":"Church","year":"1936","journal-title":"Am. J. Math."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1112\/plms\/s2-42.1.230","article-title":"On computable numbers, with an application to the Entscheidungsproblem","volume":"2","author":"Turing","year":"1937","journal-title":"Proc. Lond. Math. Soc."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1038\/nrn2374","article-title":"The neural systems that mediate human perceptual decision making","volume":"9","author":"Heekeren","year":"2008","journal-title":"Nat. Rev. Neurosci."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1007\/978-3-030-58811-3_9","article-title":"Automated Machine Learning: Prospects and Challenges","volume":"Volume 12252 LNCS","author":"Vaccaro","year":"2020","journal-title":"Proceedings of the Computational Science and Its Applications\u2014ICCSA 2020"},{"key":"ref_16","unstructured":"Domingos, P. (2015). The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World, Basic Books."},{"key":"ref_17","first-page":"55:1","article-title":"Neural Architecture Search: A Survey","volume":"20","author":"Elsken","year":"2019","journal-title":"J. Mach. Learn. Res."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Fox, G.C., Glazier, J.A., Kadupitiya, J.C.S., Jadhao, V., Kim, M., Qiu, J., Sluka, J.P., Somogyi, E.T., Marathe, M., and Adiga, A. (2019, January 20\u201324). Learning Everywhere: Pervasive Machine Learning for Effective High-Performance Computation. Proceedings of the IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), Rio de Janeiro, Brazil.","DOI":"10.1109\/IPDPSW.2019.00081"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Meier, B.B., Elezi, I., Amirian, M., D\u00fcrr, O., and Stadelmann, T. (2018). Learning Neural Models for End-to-End Clustering. Artificial Neural Networks in Pattern Recognition, Springer International Publishing.","DOI":"10.1007\/978-3-319-99978-4_10"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Hutter, F., Kotthoff, L., and Vanschoren, J. (2019). Automated Machine Learning\u2014Methods, Systems, Challenges, Springer. The Springer Series on Challenges in Machine Learning.","DOI":"10.1007\/978-3-030-05318-5"},{"key":"ref_21","unstructured":"Z\u00f6ller, M.A., and Huber, M.F. (2019). Benchmark and Survey of Automated Machine Learning Frameworks. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Escalante, H.J. (2020). Automated Machine Learning\u2014A brief review at the end of the early years. arXiv.","DOI":"10.1007\/978-3-030-72069-8_2"},{"key":"ref_23","unstructured":"Liu, Z., Xu, Z., Madadi, M., Junior, J.J., Escalera, S., Rajaa, S., and Guyon, I. (2019, January 20). Overview and unifying conceptualization of automated machine learning. Proceedings of the Automating Data Science Workshop, Wurzburg, Germany."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"He, X., Zhao, K., and Chu, X. (2021). AutoML: A survey of the state-of-the-art. Knowl.-Based Syst., 212.","DOI":"10.1016\/j.knosys.2020.106622"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Vanschoren, J. (2019). Meta-Learning. Automated Machine Learning: Methods, Systems, Challenges, Springer International Publishing.","DOI":"10.1007\/978-3-030-05318-5_2"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Feurer, M., and Hutter, F. (2019). Hyperparameter Optimization. Automated Machine Learning: Methods, Systems, Challenges, Springer International Publishing.","DOI":"10.1007\/978-3-030-05318-5_1"},{"key":"ref_27","unstructured":"Shawi, R.E., Maher, M., and Sakr, S. (2019). Automated Machine Learning: State-of-The-Art and Open Challenges. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Thornton, C., Hutter, F., Hoos, H.H., and Leyton-Brown, K. (2013, January 11\u201314). Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms. Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD \u201913, Chicago, IL, USA.","DOI":"10.1145\/2487575.2487629"},{"key":"ref_29","unstructured":"Ren, P., Xiao, Y., Chang, X., Huang, P., Li, Z., Chen, X., and Wang, X. (2020). A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions. arXiv."},{"key":"ref_30","unstructured":"Wistuba, M., Rawat, A., and Pedapati, T. (2019). A Survey on Neural Architecture Search. arXiv."},{"key":"ref_31","unstructured":"Chen, Y., Song, Q., and Hu, X. (2019). Techniques for Automated Machine Learning. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Frazier, P.I. (2018). Bayesian Optimization. Recent Advances in Optimization and Modeling of Contemporary Problems, PubsOnLine. Chapter 11.","DOI":"10.1287\/educ.2018.0188"},{"key":"ref_33","unstructured":"Z\u00f6ller, M., and Huber, M.F. (2019). Survey on Automated Machine Learning. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Tuggener, L., Amirian, M., Rombach, K., Lorwald, S., Varlet, A., Westermann, C., and Stadelmann, T. (2019, January 14). Automated Machine Learning in Practice: State of the Art and Recent Results. Proceedings of the 6th Swiss Conference on Data Science (SDS), Bern, Switzerland.","DOI":"10.1109\/SDS.2019.00-11"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Chung, C., Chen, C., Shih, W., Lin, T., Yeh, R., and Wang, I. (2017, January 12\u201314). Automated machine learning for Internet of Things. Proceedings of the 2017 IEEE International Conference on Consumer Electronics, (ICCE-TW), Taipei, Taiwan.","DOI":"10.1109\/ICCE-China.2017.7991112"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"3642","DOI":"10.1109\/TII.2019.2900987","article-title":"A Blockchain and AutoML Approach for Open and Automated Customer Service","volume":"15","author":"Li","year":"2019","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"2480","DOI":"10.1109\/TNSM.2020.3024225","article-title":"Experimental Review of Neural-Based Approaches for Network Intrusion Management","volume":"17","author":"Galatro","year":"2020","journal-title":"IEEE Trans. Netw. Serv. Manag."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1109\/MS.2019.2919573","article-title":"Sentiment Classification Using N-Gram Inverse Document Frequency and Automated Machine Learning","volume":"36","author":"Maipradit","year":"2019","journal-title":"IEEE Softw."},{"key":"ref_39","unstructured":"Shi, X., Wong, Y., Chai, C., and Li, M. (2020). An Automated Machine Learning (AutoML) Method of Risk Prediction for Decision-Making of Autonomous Vehicles. IEEE Trans. Intell. Transp. Syst., 1\u201310."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Chen, T., and Guestrin, C. (2016, January 13\u201317). XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD \u201916, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939785"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1023\/B:MACH.0000015880.99707.b2","article-title":"Optimal ordered problem solver","volume":"54","author":"Schmidhuber","year":"2004","journal-title":"Mach. Learn."},{"key":"ref_42","unstructured":"Trask, A., Hill, F., Reed, S., Rae, J., Dyer, C., and Blunsom, P. (2018, January 3\u20138). Neural Arithmetic Logic Units. Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS), NIPS\u201918, Montreal, QC, Canada."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"471","DOI":"10.1038\/nature20101","article-title":"Hybrid computing using a neural network with dynamic external memory","volume":"538","author":"Graves","year":"2016","journal-title":"Nature"},{"key":"ref_44","unstructured":"Zoph, B., and Le, Q.V. (2016). Neural Architecture Search with Reinforcement Learning. arXiv."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Jin, H., Song, Q., and Hu, X. (2019, January 4\u20138). Auto-keras: An efficient neural architecture search system. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA.","DOI":"10.1145\/3292500.3330648"},{"key":"ref_46","unstructured":"Kandasamy, K., Neiswanger, W., Schneider, J., Poczos, B., and Xing, E.P. (2018). Neural Architecture Search with Bayesian Optimisation and Optimal Transport. Advances in Neural Information Processing Systems (NIPS), Curran Associates, Inc."},{"key":"ref_47","unstructured":"Liu, H., Simonyan, K., Vinyals, O., Fernando, C., and Kavukcuoglu, K. (May, January 30). Hierarchical Representations for Efficient Architecture Search. Proceedings of the 6th International Conference on Learning Representations (ICLR), Vancouver, BC, Canada."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Real, E., Aggarwal, A., Huang, Y., and Le, Q.V. (February, January 27). Regularized Evolution for Image Classifier Architecture Search. Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI), Honolulu, HI, USA.","DOI":"10.1609\/aaai.v33i01.33014780"},{"key":"ref_49","unstructured":"Pham, H., Guan, M.Y., Zoph, B., Le, Q.V., and Dean, J. (2018). Efficient neural architecture search via parameter sharing. arXiv."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.J., Fei-Fei, L., Yuille, A., Huang, J., and Murphy, K. (2018, January 8\u201314). Progressive Neural Architecture Search. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01246-5_2"},{"key":"ref_51","unstructured":"Jastrz\u0119bski, S., de Laroussilhe, Q., Tan, M., Ma, X., Houlsby, N., and Gesmundo, A. (2018). Neural Architecture Search Over a Graph Search Space. arXiv."},{"key":"ref_52","unstructured":"Chen, T.Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D.K. (2018, January 8\u201314). Neural ordinary differential equations. Proceedings of the Advances in Neural Information Processing Systems (NIPS), Munich, Germany."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Hinton, G.E., Krizhevsky, A., and Wang, S.D. (2011). Transforming auto-encoders. International Conference on Artificial Neural Networks, Espoo, Finland, 14\u201317 June 2011, Springer.","DOI":"10.1007\/978-3-642-21735-7_6"},{"key":"ref_55","unstructured":"Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8\u201313). Generative adversarial nets. Proceedings of the Advances in Neural Information Processing Systems (NIPS), Montreal, QC, Canada."},{"key":"ref_56","unstructured":"McGill, M., and Perona, P. (2017, January 6\u201311). Deciding how to decide: Dynamic routing in artificial neural networks. Proceedings of the 34th International Conference on Machine Learning, Sydney, NSW, Australia."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Hassan, H.A.M., Sansonetti, G., Gasparetti, F., and Micarelli, A. (2018, January 2\u20137). Semantic-based Tag Recommendation in Scientific Bookmarking Systems. Proceedings of the ACM RecSys 2018, Vancouver, BC, Canada.","DOI":"10.1145\/3240323.3240409"},{"key":"ref_58","unstructured":"Hahn, T., Pyeon, M., and Kim, G. (2019, January 8\u201314). Self-Routing Capsule Networks. Proceedings of the Advances in Neural Information Processing Systems (NIPS), Vancouver, BC, Canada."},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Choi, J., Seo, H., Im, S., and Kang, M. (2019, January 27\u201328). Attention routing between capsules. Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW), Seoul, Korea.","DOI":"10.1109\/ICCVW.2019.00247"},{"key":"ref_60","unstructured":"Hinton, G.E., Sabour, S., and Frosst, N. (May, January 30). Matrix capsules with EM routing. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1613\/jair.3912","article-title":"The Arcade Learning Environment: An Evaluation Platform for General Agents","volume":"47","author":"Bellemare","year":"2013","journal-title":"J. Artif. Intell. Res."},{"key":"ref_62","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M.A. (2013). Playing Atari with Deep Reinforcement Learning. arXiv."},{"key":"ref_63","unstructured":"Shao, K., Tang, Z., Zhu, Y., Li, N., and Zhao, D. (2019). A Survey of Deep Reinforcement Learning in Video Games. arXiv."},{"key":"ref_64","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, A Bradford Book."},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_66","unstructured":"Sabour, S., Frosst, N., and Hinton, G.E. (2017, January 4\u20139). Dynamic Routing between Capsules. Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS), NIPS\u201917, Long Beach, CA, USA."},{"key":"ref_67","unstructured":"Kingma, D.P., and Ba, J. (2015, January 7\u20139). Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA."},{"key":"ref_68","unstructured":"Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. (2020). On the Variance of the Adaptive Learning Rate and Beyond. arXiv."},{"key":"ref_69","first-page":"26","article-title":"Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude","volume":"4","author":"Tieleman","year":"2012","journal-title":"COURSERA Neural Netw. Mach. Learn."},{"key":"ref_70","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1007\/s10994-006-5833-1","article-title":"Markov Logic Networks","volume":"62","author":"Richardson","year":"2006","journal-title":"Mach. Learn."},{"key":"ref_71","unstructured":"Navon, A., Achituve, I., Maron, H., Chechik, G., and Fetaya, E. (2020). Auxiliary Learning by Implicit Differentiation. arXiv."}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/10\/1\/11\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:10:40Z","timestamp":1760159440000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/10\/1\/11"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,1,13]]},"references-count":71,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,1]]}},"alternative-id":["computers10010011"],"URL":"https:\/\/doi.org\/10.3390\/computers10010011","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,1,13]]}}}