{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T03:34:28Z","timestamp":1782963268912,"version":"3.54.5"},"reference-count":51,"publisher":"MIT Press","issue":"2","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,6,3]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Evolution-based neural architecture search methods have shown promising results, but they require high computational resources because these methods involve training each candidate architecture from scratch and then evaluating its fitness, which results in long search time. Covariance Matrix Adaptation Evolution Strategy (CMA-ES) has shown promising results in tuning hyperparameters of neural networks but has not been used for neural architecture search. In this work, we propose a framework called CMANAS which applies the faster convergence property of CMA-ES to the deep neural architecture search problem. Instead of training each individual architecture seperately, we used the accuracy of a trained one shot model (OSM) on the validation data as a prediction of the fitness of the architecture, resulting in reduced search time. We also used an architecture-fitness table (AF table) for keeping a record of the already evaluated architecture, thus further reducing the search time. The architectures are modeled using a normal distribution, which is updated using CMA-ES based on the fitness of the sampled population. Experimentally, CMANAS achieves better results than previous evolution-based methods while reducing the search time significantly. The effectiveness of CMANAS is shown on two different search spaces using four datasets: CIFAR-10, CIFAR-100, ImageNet, and ImageNet16-120. All the results show that CMANAS is a viable alternative to previous evolution-based methods and extends the application of CMA-ES to the deep neural architecture search field.<\/jats:p>","DOI":"10.1162\/evco_a_00331","type":"journal-article","created":{"date-parts":[[2023,5,8]],"date-time":"2023-05-08T18:20:26Z","timestamp":1683570026000},"page":"177-204","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":11,"title":["Neural Architecture Search Using Covariance Matrix Adaptation\n                    Evolution Strategy"],"prefix":"10.1162","volume":"32","author":[{"given":"Nilotpal","family":"Sinha","sequence":"first","affiliation":[{"name":"Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu City, 30010, Taiwan nilotpalsinha.cs06@nycu.edu.tw"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kuan-Wen","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu City, 30010, Taiwan kuanwen@cs.nycu.edu.tw"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2024,6,3]]},"reference":[{"key":"2024060313544883400_B1","article-title":"Designing neural network architectures\n                        using reinforcement learning","author":"Baker","year":"2017","journal-title":"International\n                            Conference on Learning\n                    Representations"},{"issue":"2","key":"2024060313544883400_B2","article-title":"Random search for hyper-parameter\n                        optimization","volume":"13","author":"Bergstra","year":"2012","journal-title":"Journal of Machine Learning\n                            Research"},{"key":"2024060313544883400_B3","first-page":"1294","article-title":"Progressive differentiable architecture\n                        search: Bridging the depth gap between search and\n                    evaluation","author":"Chen","year":"2019","journal-title":"Proceedings of the IEEE\n                            International Conference on Computer Vision"},{"key":"2024060313544883400_B4","article-title":"A downsampled variant of ImageNet as an\n                        alternative to the CIFAR datasets","author":"Chrabaszcz","year":"2017"},{"key":"2024060313544883400_B5","doi-asserted-by":"crossref","first-page":"248","DOI":"10.1109\/CVPR.2009.5206848","article-title":"ImageNet: A large-scale hierarchical\n                        image database","author":"Deng","year":"2009","journal-title":"2009 IEEE Conference on\n                            Computer Vision and Pattern Recognition"},{"key":"2024060313544883400_B6","article-title":"Improved regularization of convolutional neural networks with\n                        cutout","author":"DeVries","year":"2017"},{"key":"2024060313544883400_B7","first-page":"3681","article-title":"One-shot neural architecture search via\n                        self-evaluated template network","author":"Dong","year":"2019","journal-title":"Proceedings of the IEEE International Conference on\n                            Computer Vision"},{"key":"2024060313544883400_B8","first-page":"1761","article-title":"Searching for a robust neural\n                        architecture in four GPU hours","author":"Dong","year":"2019","journal-title":"Proceedings of the IEEE Conference on Computer Vision\n                            and Pattern Recognition"},{"key":"2024060313544883400_B9","article-title":"NAS-Bench-201: Extending the scope of\n                        reproducible neural architecture search","author":"Dong","year":"2020","journal-title":"International Conference on Learning\n                            Representations"},{"key":"2024060313544883400_B10","article-title":"Efficient multi-objective neural\n                        architecture search via Lamarckian evolution","author":"Elsken","year":"2018"},{"key":"2024060313544883400_B11","article-title":"Neural architecture search: A\n                        survey","author":"Elsken","year":"2018"},{"issue":"55","key":"2024060313544883400_B12","first-page":"1","article-title":"Neural architecture search: A\n                        survey","volume":"20","author":"Elsken","year":"2019","journal-title":"Journal of Machine Learning\n                            Research"},{"key":"2024060313544883400_B13","first-page":"1437","article-title":"BOHB: Robust and efficient\n                        hyperparameter optimization at scale","author":"Falkner","year":"2018","journal-title":"International Conference on Machine\n                        Learning"},{"key":"2024060313544883400_B14","article-title":"The CMA evolution strategy: A\n                        tutorial","author":"Hansen","year":"2016"},{"issue":"2","key":"2024060313544883400_B15","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1162\/106365601750190398","article-title":"Completely derandomized self-adaptation\n                        in evolution strategies","volume":"9","author":"Hansen","year":"2001","journal-title":"Evolutionary\n                            Computation"},{"key":"2024060313544883400_B16","first-page":"770","article-title":"Deep residual learning for image\n                        recognition","author":"He","year":"2016","journal-title":"Proceedings of the IEEE\n                            Conference on Computer Vision and Pattern Recognition"},{"key":"2024060313544883400_B17","first-page":"4700","article-title":"Densely connected convolutional networks","author":"Huang","year":"2017","journal-title":"Proceedings of the IEEE Conference on Computer Vision\n                            and Pattern Recognition"},{"issue":"5","key":"2024060313544883400_B18","doi-asserted-by":"publisher","first-page":"481","DOI":"10.1109\/TEVC.2002.800884","article-title":"A framework for evolutionary\n                        optimization with approximate fitness functions","volume":"6","author":"Jin","year":"2002","journal-title":"IEEE Transactions on Evolutionary\n                        Computation"},{"key":"2024060313544883400_B19","article-title":"Learning multiple layers of features\n                        from tiny images","author":"Krizhevsky","year":"2009"},{"key":"2024060313544883400_B20","first-page":"1097","article-title":"ImageNet classification with deep convolutional neural\n                        networks","author":"Krizhevsky","year":"2012","journal-title":"Advances in neural information\n                            processing systems"},{"key":"2024060313544883400_B21","first-page":"367","article-title":"Random search and reproducibility for\n                        neural architecture search","author":"Li","year":"2020","journal-title":"Uncertainty\n                            in Artificial Intelligence"},{"key":"2024060313544883400_B22","first-page":"19","article-title":"Progressive neural architecture\n                        search","author":"Liu","year":"2018","journal-title":"Proceedings of the European\n                            Conference on Computer Vision"},{"key":"2024060313544883400_B23","article-title":"Hierarchical representations for\n                        efficient architecture search","author":"Liu","year":"2018","journal-title":"International Conference on Learning\n                            Representations"},{"key":"2024060313544883400_B24","article-title":"DARTS: Differentiable architecture\n                        search","author":"Liu","year":"2019","journal-title":"International Conference on\n                            Learning Representations"},{"key":"2024060313544883400_B25","article-title":"CMA-ES for hyperparameter optimization\n                        of deep neural networks","author":"Loshchilov","year":"2016"},{"key":"2024060313544883400_B26","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1608.03983","article-title":"SGDR: Stochastic gradient descent with\n                        warm restarts","author":"Loshchilov","year":"2017","journal-title":"Proceedings of the 5th\n                            International Conference on Learning Representations"},{"key":"2024060313544883400_B27","article-title":"Multi-objective evolutionary design of deep convolutional\n                        neural networks for image classification","author":"Lu","year":"2020","journal-title":"IEEE Transactions on Evolutionary\n                        Computation"},{"key":"2024060313544883400_B28","first-page":"8024","article-title":"PyTorch: An imperative style,\n                        high-performance deep learning library","volume-title":"Advances in neural information\n                            processing systems","author":"Paszke","year":"2019"},{"key":"2024060313544883400_B29","first-page":"4095","article-title":"Efficient neural architecture search via\n                        parameters sharing","volume":"80","author":"Pham","year":"2018","journal-title":"Proceedings of the\n                            35th International Conference on Machine Learning"},{"key":"2024060313544883400_B30","doi-asserted-by":"publisher","first-page":"4780","DOI":"10.1609\/aaai.v33i01.33014780","article-title":"Regularized evolution for image classifier architecture\n                        search","volume":"33","author":"Real","year":"2019","journal-title":"Proceedings of the AAAI\n                            Conference on Artificial Intelligence"},{"key":"2024060313544883400_B31","first-page":"2902","article-title":"Large-scale evolution of image\n                        classifiers","volume":"70","author":"Real","year":"2017","journal-title":"Proceedings of the 34th\n                            International Conference on Machine Learning"},{"key":"2024060313544883400_B32","first-page":"4510","article-title":"MobileNetV2: Inverted residuals and\n                        linear bottlenecks","author":"Sandler","year":"2018","journal-title":"Proceedings of the\n                            IEEE Conference on Computer Vision and Pattern\n                        Recognition"},{"key":"2024060313544883400_B33","article-title":"Very deep convolutional networks for\n                        large-scale image recognition","author":"Simonyan","year":"2014"},{"key":"2024060313544883400_B34","doi-asserted-by":"crossref","first-page":"910","DOI":"10.1145\/3449639.3459275","article-title":"Evolving neural architecture using one\n                        shot model","author":"Sinha","year":"2021","journal-title":"Proceedings of the Genetic\n                            and Evolutionary Computation Conference"},{"issue":"2","key":"2024060313544883400_B35","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1162\/artl.2009.15.2.15202","article-title":"A hypercube-based encoding for evolving\n                        large-scale neural networks","volume":"15","author":"Stanley","year":"2009","journal-title":"Artificial\n                            Life"},{"issue":"2","key":"2024060313544883400_B36","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1162\/106365602320169811","article-title":"Evolving neural networks through\n                        augmenting topologies","volume":"10","author":"Stanley","year":"2002","journal-title":"Evolutionary\n                            Computation"},{"key":"2024060313544883400_B37","doi-asserted-by":"crossref","first-page":"497","DOI":"10.1145\/3071178.3071229","article-title":"A genetic programming approach to\n                        designing convolutional neural network architectures","author":"Suganuma","year":"2017","journal-title":"Proceedings of the Genetic and Evolutionary Computation\n                            Conference"},{"issue":"2","key":"2024060313544883400_B38","doi-asserted-by":"publisher","first-page":"350","DOI":"10.1109\/TEVC.2019.2924461","article-title":"Surrogate-assisted evolutionary deep\n                        learning using an end-to-end random forest-based performance\n                        predictor","volume":"24","author":"Sun","year":"2019","journal-title":"IEEE Transactions on Evolutionary\n                            Computation"},{"issue":"4","key":"2024060313544883400_B39","doi-asserted-by":"publisher","first-page":"1242","DOI":"10.1109\/TNNLS.2019.2919608","article-title":"Completely automated CNN architecture design based on\n                        blocks","volume":"31","author":"Sun","year":"2019","journal-title":"IEEE Transactions on Neural Networks\n                            and Learning Systems"},{"issue":"9","key":"2024060313544883400_B40","doi-asserted-by":"publisher","first-page":"3840","DOI":"10.1109\/TCYB.2020.2983860","article-title":"Automatically designing CNN\n                        architectures using the genetic algorithm for image\n                        classification","volume":"50","author":"Sun","year":"2020","journal-title":"IEEE Transactions on\n                            Cybernetics"},{"key":"2024060313544883400_B41","first-page":"1139","article-title":"On the importance of initialization and\n                        momentum in deep learning","author":"Sutskever","year":"2013","journal-title":"International\n                            Conference on Machine Learning"},{"key":"2024060313544883400_B42","first-page":"2818","article-title":"Rethinking the inception architecture\n                        for computer vision","author":"Szegedy","year":"2016","journal-title":"Proceedings of the\n                            IEEE Conference on Computer Vision and Pattern\n                        Recognition"},{"issue":"3\u20134","key":"2024060313544883400_B43","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1007\/BF00992696","article-title":"Simple statistical gradient-following algorithms for\n                        connectionist reinforcement learning","volume":"8","author":"Williams","year":"1992","journal-title":"Machine Learning"},{"key":"2024060313544883400_B44","first-page":"1379","article-title":"Genetic CNN","author":"Xie","year":"2017","journal-title":"Proceedings of the IEEE International Conference on\n                            Computer Vision"},{"key":"2024060313544883400_B45","article-title":"SNAS: Stochastic neural architecture\n                        search","author":"Xie","year":"2019","journal-title":"International Conference on\n                            Learning Representations"},{"key":"2024060313544883400_B46","article-title":"Understanding and robustifying\n                        differentiable architecture search","author":"Zela","year":"2020","journal-title":"International Conference on Learning\n                            Representations"},{"key":"2024060313544883400_B47","article-title":"Sampled training and node inheritance\n                        for fast evolutionary neural architecture search","author":"Zhang","year":"2020"},{"key":"2024060313544883400_B48","first-page":"6848","article-title":"Shufflenet: An extremely efficient\n                        convolutional neural network for mobile devices","author":"Zhang","year":"2018","journal-title":"Proceedings of the IEEE Conference on Computer Vision\n                            and Pattern Recognition"},{"key":"2024060313544883400_B49","first-page":"2423","article-title":"Practical block-wise neural network\n                        architecture generation","author":"Zhong","year":"2018","journal-title":"Proceedings of\n                            the IEEE Conference on Computer Vision and Pattern\n                        Recognition"},{"key":"2024060313544883400_B50","article-title":"Neural architecture search with reinforcement\n                        learning","author":"Zoph","year":"2016"},{"key":"2024060313544883400_B51","first-page":"8697","article-title":"Learning transferable architectures for scalable image\n                        recognition","author":"Zoph","year":"2018","journal-title":"Proceedings of the IEEE\n                            Conference on Computer Vision and Pattern Recognition"}],"container-title":["Evolutionary Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/evco\/article-pdf\/32\/2\/177\/2372834\/evco_a_00331.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/evco\/article-pdf\/32\/2\/177\/2372834\/evco_a_00331.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,3]],"date-time":"2024-06-03T13:55:31Z","timestamp":1717422931000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/evco\/article\/32\/2\/177\/115656\/Neural-Architecture-Search-Using-Covariance-Matrix"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":51,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,6,3]]},"published-print":{"date-parts":[[2024,6,3]]}},"URL":"https:\/\/doi.org\/10.1162\/evco_a_00331","relation":{},"ISSN":["1530-9304"],"issn-type":[{"value":"1530-9304","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}