{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,2]],"date-time":"2025-08-02T18:56:20Z","timestamp":1754160980589,"version":"3.41.2"},"reference-count":45,"publisher":"IOP Publishing","issue":"3","license":[{"start":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T00:00:00Z","timestamp":1753660800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T00:00:00Z","timestamp":1753660800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2025,9,30]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Neural architecture search (NAS) has become an essential tool for designing effective and efficient neural networks. In this paper, we investigate the geometric properties of neural architecture spaces commonly used in differentiable NAS methods, specifically NAS-Bench-201 and differentiable architecture search (DARTS). By introducing notions of flatness in architecture space such as neighborhoods and accuracy barriers along paths, we reveal locality and flatness characteristics analogous to the well-known properties of neural network loss landscapes in weight space. In particular, we unveil the detailed geometrical structure of the architecture search landscape by uncovering the absence of barriers between well-performing architectures, finding that highly accurate architectures cluster together in flat regions, while suboptimal architectures instead remain isolated, showing higher values of the barriers. Building on these insights, we propose architecture-aware minimization (A<jats:sup>2<\/jats:sup>M), a novel analytically derived algorithmic framework that <jats:italic>explicitly<\/jats:italic> biases, for the first time, the gradient of differentiable NAS methods towards flat minima <jats:italic>in architecture space<\/jats:italic>. A<jats:sup>2<\/jats:sup>M consistently improves generalization over state-of-the-art DARTS-based algorithms on benchmark datasets including CIFAR-10, CIFAR-100, and ImageNet-16-120, across both NAS-Bench-201 and DARTS search spaces. Notably, A<jats:sup>2<\/jats:sup>M is able to increase the test accuracy, on average across different differentiable NAS methods, by +3.60% on CIFAR-10, +4.60% on CIFAR-100, and +3.64% on ImageNet-16-120 - while finding architectures with low accuracy barriers. A<jats:sup>2<\/jats:sup>M can be easily integrated into existing differentiable NAS frameworks, offering a versatile tool for future research and applications in automated machine learning. We will open-source our code at <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/AI-Tech-Research-Lab\/AsquaredM\">https:\/\/github.com\/AI-Tech-Research-Lab\/AsquaredM<\/jats:ext-link>.<\/jats:p>","DOI":"10.1088\/2632-2153\/adf02e","type":"journal-article","created":{"date-parts":[[2025,7,15]],"date-time":"2025-07-15T22:55:38Z","timestamp":1752620138000},"page":"035016","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Architecture-aware minimization (A<sup>2<\/sup>M): how to find flat minima in neural architecture search"],"prefix":"10.1088","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9734-670X","authenticated-orcid":true,"given":"Matteo","family":"Gambella","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1919-6141","authenticated-orcid":true,"given":"Fabrizio","family":"Pittorino","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7828-7687","authenticated-orcid":false,"given":"Manuel","family":"Roveri","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"266","published-online":{"date-parts":[[2025,7,28]]},"reference":[{"key":"mlstadf02ebib1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2023.110052","article-title":"Neural architecture search: a contemporary literature review for computer vision applications","volume":"147","author":"Poyser","year":"2024","journal-title":"Pattern Recognit."},{"key":"mlstadf02ebib2","doi-asserted-by":"crossref","DOI":"10.1109\/SMC53654.2022.9945080","article-title":"CNAS: constrained neural architecture search","author":"Gambella","year":"2022"},{"key":"mlstadf02ebib3","first-page":"4322","article-title":"Hardware-aware neural architecture search: survey and taxonomy","author":"Benmeziane","year":"2021"},{"edition":"1st edn","year":"2010","author":"Ashlock","key":"mlstadf02ebib4"},{"key":"mlstadf02ebib5","doi-asserted-by":"publisher","first-page":"1054","DOI":"10.1109\/TNN.1998.712192","article-title":"Reinforcement learning: an introduction","volume":"9","author":"Sutton","year":"1998","journal-title":"IEEE Trans. Neural Netw."},{"article-title":"DARTS: Differentiable architecture search","year":"2019","author":"Liu","key":"mlstadf02ebib6"},{"article-title":"Understanding and robustifying differentiable architecture search","year":"2020","author":"Zela","key":"mlstadf02ebib7"},{"key":"mlstadf02ebib8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3665138","article-title":"Efficient automation of neural network design: a survey on differentiable neural architecture search","volume":"56","author":"Heuillet","year":"2024","journal-title":"ACM Comput. Surv."},{"key":"mlstadf02ebib9","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.131.227301","article-title":"Star-shaped space of solutions of the spherical negative perceptron","volume":"131","author":"Annesi","year":"2023","journal-title":"Phys. Rev. Lett."},{"key":"mlstadf02ebib10","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1073\/pnas.1908636117","article-title":"Shaping the learning landscape in neural networks around wide flat minima","volume":"117","author":"Baldassi","year":"2020","journal-title":"Proc. Natl Acad. Sci."},{"article-title":"Sharpness-aware minimization for efficiently improving generalization","year":"2021","author":"Foret","key":"mlstadf02ebib11"},{"key":"mlstadf02ebib12","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/ac7d3b","article-title":"Deep learning via message passing algorithms based on belief propagation","volume":"3","author":"Lucibello","year":"2022","journal-title":"Mach. Learn.: Sci. Technol."},{"key":"mlstadf02ebib13","doi-asserted-by":"crossref","DOI":"10.5244\/C.35.226","article-title":"Neighborhood-aware neural architecture search","author":"Wang","year":"2021"},{"article-title":"Exploring the loss landscape in neural architecture search","year":"2020","author":"White","key":"mlstadf02ebib14"},{"key":"mlstadf02ebib15","first-page":"1","article-title":"Flatnas: optimizing flatness in neural architecture search for out-of-distribution robustness","author":"Gambella","year":"2024"},{"article-title":"Robust nas under adversarial training: benchmark, theory, and beyond","year":"2024","author":"Wu","key":"mlstadf02ebib16"},{"article-title":"Nas-bench-201: extending the scope of reproducible neural architecture search","year":"2020","author":"Dong","key":"mlstadf02ebib17"},{"key":"mlstadf02ebib18","doi-asserted-by":"publisher","DOI":"10.1088\/1742-5468\/ac9832","article-title":"Deep networks on toroids: removing symmetries reveals the structure of flat regions in the landscape geometry","author":"Pittorino","year":"2022","journal-title":"J. Stat. Mech."},{"key":"mlstadf02ebib19","doi-asserted-by":"crossref","DOI":"10.1088\/1742-5468\/ac3ae8","article-title":"Entropic gradient descent algorithms and wide flat minima","author":"Pittorino","year":"2021"},{"key":"mlstadf02ebib20","doi-asserted-by":"publisher","DOI":"10.1088\/1742-5468\/ac3ae8","article-title":"Entropic gradient descent algorithms and wide flat minima","volume":"2021","author":"Pittorino","year":"2021","journal-title":"J. Stat. Mech."},{"article-title":"DARTS-: robustly stepping out of performance collapse without indicators","year":"2021","author":"Chu","key":"mlstadf02ebib21"},{"key":"mlstadf02ebib22","first-page":"465","article-title":"Fair darts: eliminating unfair advantages in differentiable architecture search","author":"Chu","year":"2020"},{"key":"mlstadf02ebib23","first-page":"10864","article-title":"\u03b2-darts: beta-decay regularization for differentiable architecture search","author":"Ye","year":"2022"},{"article-title":"\u03b2-darts++: Bi-level regularization for proxy-robust differentiable architecture search","year":"2023","author":"Ye","key":"mlstadf02ebib24"},{"article-title":"Darts+: improved differentiable architecture search with early stopping","year":"2019","author":"Liang","key":"mlstadf02ebib25"},{"article-title":"Pc-darts: partial channel connections for memory-efficient architecture search","year":"2020","author":"Xu","key":"mlstadf02ebib26"},{"article-title":"\u03bb-DARTS: mitigating performance collapse by harmonizing operation selection among cells","year":"2023","author":"Movahedi","key":"mlstadf02ebib27"},{"article-title":"Rethinking architecture selection in differentiable NAS","year":"2021","author":"Wang","key":"mlstadf02ebib28"},{"article-title":"Understanding architectures learnt by cell-based neural architecture search","year":"2020","author":"Shu","key":"mlstadf02ebib29"},{"key":"mlstadf02ebib30","doi-asserted-by":"publisher","first-page":"656","DOI":"10.1016\/j.neunet.2023.08.062","article-title":"Improving differentiable architecture search via self-distillation","volume":"167","author":"Zhu","year":"2023","journal-title":"Neural Netw."},{"key":"mlstadf02ebib31","first-page":"1554","article-title":"Stabilizing differentiable architecture search via perturbation-based regularization","volume":"vol 119","author":"Chen","year":"2020"},{"article-title":"Towards understanding sharpness-aware minimization","year":"2022","author":"Andriushchenko","key":"mlstadf02ebib32"},{"key":"mlstadf02ebib33","doi-asserted-by":"publisher","first-page":"45736","DOI":"10.1109\/ACCESS.2022.3169897","article-title":"NAS-Bench-NLP: neural architecture search benchmark for natural language processing","volume":"10","author":"Klyuchnikov","year":"2022","journal-title":"IEEE Access"},{"key":"mlstadf02ebib34","first-page":"1540","article-title":"Optimizing millions of hyperparameters by implicit differentiation","author":"Lorraine","year":"2020"},{"article-title":"Learning multiple layers of features from tiny images","year":"2009","author":"Krizhevsky","key":"mlstadf02ebib35"},{"article-title":"A downsampled variant of imagenet as an alternative to the cifar datasets","year":"2017","author":"Chrabaszcz","key":"mlstadf02ebib36"},{"article-title":"ASAM: adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks","year":"2021","author":"Kwon","key":"mlstadf02ebib37"},{"key":"mlstadf02ebib38","doi-asserted-by":"publisher","first-page":"6144","DOI":"10.1609\/aaai.v38i6.28431","article-title":"CR-SAM: curvature regularized sharpness-aware minimization","volume":"vol 38","author":"Wu","year":"2023"},{"key":"mlstadf02ebib39","doi-asserted-by":"publisher","first-page":"506","DOI":"10.1016\/j.neunet.2023.10.044","article-title":"AdaSAM: boosting sharpness-aware minimization with adaptive learning rate and momentum for training deep neural networks","volume":"169","author":"Sun","year":"2023","journal-title":"Neural Netw."},{"article-title":"An adaptive policy to employ sharpness-aware minimization","year":"2023","author":"Jiang","key":"mlstadf02ebib40"},{"key":"mlstadf02ebib41","doi-asserted-by":"publisher","first-page":"3888","DOI":"10.18653\/v1\/2022.emnlp-main.257","article-title":"GA-SAM: gradient-strength based adaptive sharpness-aware minimization for improved generalization","author":"Zhang","year":"2022"},{"article-title":"Rethink darts search space and renovate a new benchmark","year":"2023","author":"Zhang","key":"mlstadf02ebib42"},{"article-title":"Fantastic generalization measures and where to find them","year":"2020","author":"Jiang","key":"mlstadf02ebib43"},{"article-title":"Once-for-all: train one network and specialize it for efficient deployment","year":"2020","author":"Cai","key":"mlstadf02ebib44"},{"key":"mlstadf02ebib45","first-page":"1554","article-title":"Stabilizing differentiable architecture search via perturbation-based regularization","author":"Chen","year":"2020"}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adf02e","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adf02e\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adf02e","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adf02e\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adf02e\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adf02e\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adf02e\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adf02e\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T10:25:11Z","timestamp":1753698311000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adf02e"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,28]]},"references-count":45,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,7,28]]},"published-print":{"date-parts":[[2025,9,30]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/adf02e","relation":{},"ISSN":["2632-2153"],"issn-type":[{"type":"electronic","value":"2632-2153"}],"subject":[],"published":{"date-parts":[[2025,7,28]]},"assertion":[{"value":"Architecture-aware minimization (A2M): how to find flat minima in neural architecture search","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2025 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2025-03-13","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2025-07-15","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2025-07-28","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}