{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T03:54:19Z","timestamp":1784346859273,"version":"3.55.0"},"reference-count":35,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2023,5,5]],"date-time":"2023-05-05T00:00:00Z","timestamp":1683244800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Systems"],"abstract":"<jats:p>Hyperparameter optimization is one of the most tedious yet crucial steps in training machine learning models. There are numerous methods for this vital model-building stage, ranging from domain-specific manual tuning guidelines suggested by the oracles to the utilization of general purpose black-box optimization techniques. This paper proposes an agent-based collaborative technique for finding near-optimal values for any arbitrary set of hyperparameters (or decision variables) in a machine learning model (or a black-box function optimization problem). The developed method forms a hierarchical agent-based architecture for the distribution of the searching operations at different dimensions and employs a cooperative searching procedure based on an adaptive width-based random sampling technique to locate the optima. The behavior of the presented model, specifically against changes in its design parameters, is investigated in both machine learning and global function optimization applications, and its performance is compared with that of two randomized tuning strategies that are commonly used in practice. Moreover, we have compared the performance of the proposed approach against particle swarm optimization (PSO) and simulated annealing (SA) methods in function optimization to provide additional insights into its exploration in the search space. According to the empirical results, the proposed model outperformed the compared random-based methods in almost all tasks conducted, notably in a higher number of dimensions and in the presence of limited on-device computational resources.<\/jats:p>","DOI":"10.3390\/systems11050228","type":"journal-article","created":{"date-parts":[[2023,5,5]],"date-time":"2023-05-05T03:24:32Z","timestamp":1683257072000},"page":"228","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Agent-Based Collaborative Random Search for Hyperparameter Tuning and Global Function Optimization"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0612-2351","authenticated-orcid":false,"given":"Ahmad","family":"Esmaeili","sequence":"first","affiliation":[{"name":"Department of Computer and Information Technology, Purdue University, West Lafayette, IN 47907, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zahra","family":"Ghorrati","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Technology, Purdue University, West Lafayette, IN 47907, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eric T.","family":"Matson","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Technology, Purdue University, West Lafayette, IN 47907, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,5,5]]},"reference":[{"key":"ref_1","first-page":"281","article-title":"Random search for hyper-parameter optimization","volume":"13","author":"Bergstra","year":"2012","journal-title":"J. Mach. Learn. Res."},{"key":"ref_2","unstructured":"Kohavi, R., and John, G.H. (1995). Machine Learning Proceedings 1995, Elsevier."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1162\/EVCO_a_00069","article-title":"Resampling Methods for Meta-Model Validation with Recommendations for Evolutionary Computation","volume":"20","author":"Bischl","year":"2012","journal-title":"Evol. Comput."},{"key":"ref_4","unstructured":"Montgomery, D.C. (2017). Design and Analysis of Experiments, John Wiley & Sons."},{"key":"ref_5","unstructured":"John, G.H. (1994). Cross-Validated C4.5: Using Error Estimation for Automatic Parameter Selection, Stanford University. Technical Report."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Mo\u010dkus, J. (1974, January 1\u20137). On Bayesian methods for seeking the extremum. Proceedings of the Optimization Techniques IFIP Technical Conference, Novosibirsk, Russia.","DOI":"10.1007\/978-3-662-38527-2_55"},{"key":"ref_7","unstructured":"Mockus, J. (2012). Bayesian Approach to Global Optimization: Theory and Applications, Springer Science & Business Media."},{"key":"ref_8","unstructured":"Feurer, M., and Hutter, F. (2019). Automated Machine Learning, Springer."},{"key":"ref_9","unstructured":"Simon, D. (2013). Evolutionary Optimization Algorithms, John Wiley & Sons."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Alibrahim, H., and Ludwig, S.A. (July, January 28). Hyperparameter Optimization: Comparing Genetic Algorithm against Grid Search and Bayesian Optimization. Proceedings of the 2021 IEEE Congress on Evolutionary Computation (CEC), Krak\u00f3w, Poland.","DOI":"10.1109\/CEC45853.2021.9504761"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Bellman, R.E. (1961). Adaptive Control Processes, Princeton University Press.","DOI":"10.1515\/9781400874668"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1109\/JPROC.2015.2494218","article-title":"Taking the human out of the loop: A review of Bayesian optimization","volume":"104","author":"Shahriari","year":"2015","journal-title":"Proc. IEEE"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Garcia-Barcos, J., and Martinez-Cantin, R. (2019, January 10\u201316). Fully Distributed Bayesian Optimization with Stochastic Policies. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, Macao, China.","DOI":"10.24963\/ijcai.2019\/327"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1016\/j.jpdc.2019.07.008","article-title":"Distributed Bayesian optimization of deep reinforcement learning algorithms","volume":"139","author":"Young","year":"2020","journal-title":"J. Parallel Distrib. Comput."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Frazier, P.I. (2018). A Tutorial on Bayesian Optimization. arXiv.","DOI":"10.1287\/educ.2018.0188"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1016\/j.neucom.2004.11.022","article-title":"Evolutionary tuning of multiple SVM parameters","volume":"64","author":"Friedrichs","year":"2005","journal-title":"Neurocomputing"},{"key":"ref_17","unstructured":"Loshchilov, I., and Hutter, F. (2016). CMA-ES for hyperparameter optimization of deep neural networks. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Ryzko, D. (2020). Modern Big Data Architectures: A Multi-Agent Systems Perspective, John Wiley & Sons.","DOI":"10.1002\/9781119597926"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3530191","article-title":"HAMLET: A Hierarchical Agent-Based Machine Learning Platform","volume":"16","author":"Esmaeili","year":"2022","journal-title":"ACM Trans. Auton. Adapt. Syst."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Esmaeili, A., Ghorrati, Z., and Matson, E.T. (2022, January 13\u201315). Hierarchical Collaborative Hyper-Parameter Tuning. Proceedings of the Advances in Practical Applications of Agents, Multi-Agent Systems, and Complex Systems Simulation. The PAAMS Collection, L\u2019Aquila, Italy.","DOI":"10.1007\/978-3-031-18192-4_11"},{"key":"ref_21","unstructured":"Bardenet, R., Brendel, M., K\u00e9gl, B., and Sebag, M. (2013, January 17\u201319). Collaborative hyperparameter tuning. Proceedings of the International Conference on Machine Learning, PMLR, Atlanta, GA, USA."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Swearingen, T., Drevo, W., Cyphers, B., Cuesta-Infante, A., Ross, A., and Veeramachaneni, K. (2017, January 11\u201314). ATM: A distributed, collaborative, scalable system for automated machine learning. Proceedings of the 2017 IEEE International Conference on Big Data (Big Data), Boston, MA, USA.","DOI":"10.1109\/BigData.2017.8257923"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Koch, P., Golovidov, O., Gardner, S., Wujek, B., Griffin, J., and Xu, Y. (2018, January 19\u201323). Autotune: A derivative-free optimization framework for hyperparameter tuning. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.","DOI":"10.1145\/3219819.3219837"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1034","DOI":"10.1109\/TCAD.2021.3077193","article-title":"Multi-agent reinforcement learning for hyperparameter optimization of convolutional neural networks","volume":"41","author":"Iranfar","year":"2021","journal-title":"IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst."},{"key":"ref_25","first-page":"17200","article-title":"Provably efficient online hyperparameter optimization with population-based bandits","volume":"33","author":"Nguyen","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_26","first-page":"2825","article-title":"Scikit-learn: Machine Learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1961189.1961199","article-title":"LIBSVM: A library for support vector machines","volume":"2","author":"Chang","year":"2011","journal-title":"ACM Trans. Intell. Syst. Technol. (TIST)"},{"key":"ref_28","first-page":"551","article-title":"Online passive aggressive algorithms","volume":"7","author":"Crammer","year":"2006","journal-title":"J. Mach. Learn. Res."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v033.i01","article-title":"Regularization paths for generalized linear models via coordinate descent","volume":"33","author":"Friedman","year":"2010","journal-title":"J. Stat. Softw."},{"key":"ref_30","unstructured":"(2022, November 24). Scikit-Learn API Reference. Available online: https:\/\/scikit-learn.org\/stable\/modules\/classes.html."},{"key":"ref_31","first-page":"150","article-title":"A literature survey of benchmark functions for global optimisation problems","volume":"4","author":"Jamil","year":"2013","journal-title":"Int. J. Math. Model. Numer. Optim."},{"key":"ref_32","unstructured":"Rudolph, G. (1990). Globale Optimierung Mit Parallelen Evolutionsstrategien. [Ph.D. Thesis, Diplomarbeit, Universit at Dortmund, Fachbereich Informatik]."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1016\/0893-6080(90)90029-K","article-title":"Experiments in nonconvex optimization: Stochastic approximation with function smoothing and simulated annealing","volume":"3","author":"Styblinski","year":"1990","journal-title":"Neural Netw."},{"key":"ref_34","unstructured":"Kennedy, J., and Eberhart, R. (December, January 27). Particle swarm optimization. Proceedings of the ICNN\u201995-International Conference on Neural Networks, Perth, WA, Australia."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1126\/science.220.4598.671","article-title":"Optimization by simulated annealing","volume":"220","author":"Kirkpatrick","year":"1983","journal-title":"Science"}],"container-title":["Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-8954\/11\/5\/228\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:29:33Z","timestamp":1760124573000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-8954\/11\/5\/228"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,5]]},"references-count":35,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2023,5]]}},"alternative-id":["systems11050228"],"URL":"https:\/\/doi.org\/10.3390\/systems11050228","relation":{},"ISSN":["2079-8954"],"issn-type":[{"value":"2079-8954","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,5]]}}}