{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,6]],"date-time":"2026-09-06T05:27:51Z","timestamp":1788672471865,"version":"build-2803163510"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2024,12,9]],"date-time":"2024-12-09T00:00:00Z","timestamp":1733702400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,12,9]],"date-time":"2024-12-09T00:00:00Z","timestamp":1733702400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["5336"],"award-info":[{"award-number":["5336"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J Comput Vis"],"published-print":{"date-parts":[[2025,5]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Zero-cost proxies are nowadays frequently studied and used to search for neural architectures. They show an impressive ability to predict the performance of architectures by making use of their untrained weights. These techniques allow for immense search speed-ups. So far the joint search for well performing and robust architectures has received much less attention in the field of NAS. Therefore, the main focus of zero-cost proxies is the clean accuracy of architectures, whereas the model robustness should play an evenly important part. In this paper, we analyze the ability of common zero-cost proxies to serve as performance predictors for robustness in the popular NAS-Bench-201 search space. We are interested in the single prediction task for robustness and the joint multi-objective of clean and robust accuracy. We further analyze the feature importance of the proxies and show that predicting the robustness makes the prediction task from existing zero-cost proxies more challenging. As a result, the joint consideration of several proxies becomes necessary to predict a model\u2019s robustness while the clean accuracy can be regressed from a single such feature. Our code is available at <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/github.com\/jovitalukasik\/zcp_eval\" ext-link-type=\"uri\">https:\/\/github.com\/jovitalukasik\/zcp_eval<\/jats:ext-link>.<\/jats:p>","DOI":"10.1007\/s11263-024-02265-7","type":"journal-article","created":{"date-parts":[[2024,12,9]],"date-time":"2024-12-09T15:54:26Z","timestamp":1733759666000},"page":"2635-2652","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["An Evaluation of Zero-Cost Proxies - from Neural Architecture Performance Prediction to Model Robustness"],"prefix":"10.1007","volume":"133","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4243-9188","authenticated-orcid":false,"given":"Jovita","family":"Lukasik","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Moeller","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Margret","family":"Keuper","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,12,9]]},"reference":[{"key":"2265_CR1","unstructured":"Abdelfattah, MS., Mehrotra, A., Dudziak, L., & Lane, ND. (2021). Zero-cost proxies for lightweight NAS. In: Proc. of the International conference on learning representations (ICLR)"},{"key":"2265_CR2","unstructured":"Chen, W., Gong, X., & Wang, Z. (2021). Neural architecture search on imagenet in four GPU hours: A theoretically inspired perspective. In: Proc. of the international conference on learning representations (ICLR)"},{"key":"2265_CR3","unstructured":"Chrabaszcz, P., Loshchilov, I., & Hutter, F. (2017). A downsampled variant of imagenet as an alternative to the CIFAR datasets. arXiv:1707.08819"},{"key":"2265_CR4","unstructured":"Croce, F., & Hein, M. (2020). Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In: Proc. of the international conference on machine learning (ICML)"},{"key":"2265_CR5","unstructured":"Dong, M, Li, Y, Wang, Y., & Xu, C. (2020). Adversarially robust neural architectures. arXiv:2009.00902"},{"key":"2265_CR6","unstructured":"Dong X., & Yang, Y. (2020). Nas-bench-201: Extending the scope of reproducible neural architecture search. In: Proc. of the international conference on learning representations (ICLR)"},{"issue":"55","key":"2265_CR7","first-page":"1","volume":"20","author":"T Elsken","year":"2019","unstructured":"Elsken, T., Metzen, J. H., & Hutter, F. (2019). Neural architecture search: A survey. Journal of Machine Learning Research, 20(55), 1\u201321.","journal-title":"Journal of Machine Learning Research"},{"key":"2265_CR8","unstructured":"Goodfellow, IJ., Shlens, J., & Szegedy, C. (2015). Explaining and harnessing adversarial examples. In: Proc. of the international conference on learning representations (ICLR)"},{"key":"2265_CR9","doi-asserted-by":"crossref","unstructured":"Guo, M., Yang, Y., Xu, R., Liu, Z., & Lin, D. (2020). When NAS meets robustness: In search of robust architectures against adversarial attacks. In: Proc. of the IEEE conference on computer vision and pattern recognition (CVPR)","DOI":"10.1109\/CVPR42600.2020.00071"},{"key":"2265_CR10","unstructured":"Hendrycks, D., & Dietterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. In: Proc of the international conference on learning representations (ICLR)"},{"key":"2265_CR11","unstructured":"Hoffman, J., Roberts, DA., & Yaida, S. (2019). Robust learning with jacobian regularization. arXiv:1908.02729"},{"key":"2265_CR12","doi-asserted-by":"crossref","unstructured":"Hosseini, R., Yang, X., & Xie, P. (2021). DSRNA: differentiable search of robust neural architectures. In: Proc. of the IEEE conference on computer vision and pattern recognition (CVPR)","DOI":"10.1109\/CVPR46437.2021.00613"},{"key":"2265_CR13","unstructured":"Jung, S., Lukasik, J., & Keuper, M. (2023). Neural architecture design and robustness: A dataset. In: Proc. of the international conference on learning representations (ICLR)"},{"key":"2265_CR14","unstructured":"Kadlecov\u00e1, G., Lukasik, J., Pil\u00e1t, M., Vidnerov\u00e1, P., Safari M., Neruda, R., & Hutter, F. (2024). Surprisingly strong performance prediction with neural graph features. 2404.16551"},{"key":"2265_CR15","unstructured":"Koh, PW., Sagawa, S., Marklund, H., Xie, SM., Zhang, M., Balsubramani, A., Hu W., Yasunaga, M., Phillips, RL., Gao, I., Lee, T., David, E., Stavness, I., Guo, W., Earnshaw, B., Haque, I., Beery, SM., Leskovec, J., Kundaje, A., Pierson, E., Levine, S., Finn, C., & Liang, P. (2021). Wilds: A benchmark of in-the-wild distribution shifts. In: Proceedings of the 38th international conference on machine learning, PMLR, proceedings of machine learning research vol 139 pp 5637\u20135664"},{"key":"2265_CR16","unstructured":"Krishnakumar, A., White, C., Zela, A., Tu, R., Safari, M., & Hutter, F. (2022). Nas-bench-suite-zero: Accelerating research on zero cost proxies. In: Advances in Neural Information Processing Systems (NeurIPS)"},{"key":"2265_CR17","volume-title":"Learning multiple layers of features from tiny images","author":"A Krizhevsky","year":"2009","unstructured":"Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Tech. rep.: University of Toronto."},{"key":"2265_CR18","unstructured":"Kurakin, A., Goodfellow, IJ., & Bengio, S. (2017). Adversarial machine learning at scale. In: Proc. of the international conference on learning representations (ICLR)"},{"key":"2265_CR19","unstructured":"Lee, N., Ajanthan, T., & Torr, PHS. (2019). Snip: single-shot network pruning based on connection sensitivity. In: Proc. of the international conference on learning representations (ICLR)"},{"key":"2265_CR20","unstructured":"Li, Y., Vinyals, O., Dyer, C., Pascanu, R., & Battaglia, PW. (2018). Learning deep generative models of graphs. arXiv:1803.03324"},{"key":"2265_CR21","doi-asserted-by":"crossref","unstructured":"Lin, M., Wang, P., Sun, Z., Chen, H., Sun, X., Qian, Q., Li, H., & Jin, R. (2021). Zen-nas: A zero-shot NAS for high-performance image recognition. In: Proc. of the IEEE international conference on computer vision (ICCV)","DOI":"10.1109\/ICCV48922.2021.00040"},{"key":"2265_CR22","unstructured":"Liu, H., Simonyan, K., & Yang, Y. (2019). Darts: Differentiable architecture search. In: Proc. of the International Conference on Learning Representations (ICLR)"},{"key":"2265_CR23","doi-asserted-by":"crossref","unstructured":"Lopes, V., Alirezazadeh, S., & Alexandre, LA. (2021). EPE-NAS: efficient performance estimation without training for neural architecture search. In: International conference on artificial neural networks (ICANN)","DOI":"10.1007\/978-3-030-86383-8_44"},{"key":"2265_CR24","doi-asserted-by":"crossref","unstructured":"Lukasik, J., Jung, S., & Keuper, M. (2022). Learning where to look - generative NAS is surprisingly efficient. In: Proc. of the European conference on computer vision (ECCV)","DOI":"10.1007\/978-3-031-20050-2_16"},{"key":"2265_CR25","doi-asserted-by":"crossref","unstructured":"Lukasik, J., Moeller M., & Keuper, M. (2023). An evaluation of zero-cost proxies - from neural architecture performance prediction to model robustness. Proc of the German Conference on Pattern Recognition (GCPR)","DOI":"10.1007\/s11263-024-02265-7"},{"key":"2265_CR26","unstructured":"Mellor, J., Turner, J., Storkey, AJ., & Crowley, EJ. (2021). Neural architecture search without training. In: Proc. of the international conference on machine learning (ICML)"},{"key":"2265_CR27","doi-asserted-by":"crossref","unstructured":"Mok, J., Na, B., Choe, H., & Yoon, S. (2021). Advrush: Searching for adversarially robust neural architectures. In: Proc. of the IEEE international conference on computer vision (ICCV)","DOI":"10.1109\/ICCV48922.2021.01210"},{"key":"2265_CR28","unstructured":"Ning, X., Tang C., Li W., Zhou Z., Liang S., Yang H., Wang Y (2021) Evaluating efficient performance estimators of neural architectures. In: Advances in Neural Information Processing Systems (NeurIPS)"},{"key":"2265_CR29","doi-asserted-by":"crossref","unstructured":"Real E., Aggarwal A., Huang Y., Le QV (2019) Regularized evolution for image classifier architecture search. In: Proc. of the conference of artificial intelligence (AAAI)","DOI":"10.1609\/aaai.v33i01.33014780"},{"key":"2265_CR30","unstructured":"Roberts N., Khodak M., Dao T., Li L., R\u00e9 C., Talwalkar A (2021) Rethinking neural operations for diverse tasks. In: Advances in neural information processing systems (NeurIPS)"},{"key":"2265_CR31","unstructured":"Ru B., Wan X., Dong X., Osborne M (2021) Interpretable neural architecture search via bayesian optimisation with weisfeiler-lehman kernels. In: Proc. of the international conference on learning representations (ICLR)"},{"key":"2265_CR32","unstructured":"Shen, Y., Li Y., Zheng J., Zhang W., Yao P., Li J., Yang S., Liu J., & Cui, B. (2021). Proxybo: Accelerating neural architecture search via bayesian optimization with zero-cost proxies. arXiv:2110.10423"},{"key":"2265_CR33","unstructured":"Tanaka, H., Kunin, D., Yamins, DLK., & Ganguli, S. (2020). Pruning neural networks without any data by iteratively conserving synaptic flow. In: Advances in neural information processing systems (NeurIPS)"},{"key":"2265_CR34","unstructured":"Turner, J., Crowley, EJ., O\u2019Boyle, MFP., Storkey, AJ., & Gray, G. (2020). Blockswap: Fisher-guided block substitution for network compression on a budget. In: Proc. of the international conference on learning representations (ICLR)"},{"key":"2265_CR35","unstructured":"Wang, C., Zhang, G., & Grosse, RB. (2020). Picking winning tickets before training by preserving gradient flow. In: Proc. of the international conference on learning representations (ICLR)"},{"key":"2265_CR36","doi-asserted-by":"crossref","unstructured":"Wen, W., Liu, H., Chen, Y., Li, HH., Bender, G., & Kindermans, P. (2020). Neural predictor for neural architecture search. In: Proc. of the European conference on computer vision (ECCV)","DOI":"10.1007\/978-3-030-58526-6_39"},{"key":"2265_CR37","doi-asserted-by":"crossref","unstructured":"White C., Neiswanger W., & Savani, Y. (2021a). Bananas: Bayesian optimization with neural architectures for neural architecture search. In: Proc. of the conference of artificial intelligence (AAAI)","DOI":"10.1609\/aaai.v35i12.17233"},{"key":"2265_CR38","unstructured":"White, C., Zela, A., Ru, B., Liu, Y., & Hutter, F. (2021b). How powerful are performance predictors in neural architecture search? In: Advances in neural information processing systems (NeurIPS)"},{"key":"2265_CR39","unstructured":"White, C., Safari, M., Sukthanker, R., Ru, B., Elsken, T., Zela, A., Dey, D., & Hutter, F. (2023). Neural architecture search: Insights from 1000 papers. arXiv:2301.08727"},{"key":"2265_CR40","unstructured":"Wiles, O., Gowal, S., Stimberg, F., Rebuffi, SA., Ktena, I., Dvijotham, KD., & Cemgil, AT. (2022). A fine-grained analysis on distribution shift. In: International conference on learning representations (ICLR)"},{"key":"2265_CR41","unstructured":"Wu, J., Dai, X., Chen, D., Chen, Y., Liu, M., Yu, Y., Wang, Z., Liu, Z., Chen, M., & Yuan, L. (2021). Stronger nas with weaker predictors. In: Advances in neural information processing systems (NeurIPS)"},{"key":"2265_CR42","unstructured":"Wu, Y., Liu, F., Simon-Gabriel, C., Chrysos, GG., & Cevher, V. (2024). Robust NAS under adversarial training: benchmark theory, and beyond. CoRR arXiv:2403.13134"},{"key":"2265_CR43","unstructured":"Xiang, L., Dudziak, L., Abdelfattah, MS., Chau, T., Lane, ND., & Wen, H. (2021). Zero-cost proxies meet differentiable architecture search. arXiv:2106.06799"},{"key":"2265_CR44","unstructured":"Yao, H., Choi, C., Cao, B., Lee, Y., Koh, PWW., & Finn, C. (2022). Wild-time: A benchmark of in-the-wild distribution shift over time. In: Advances in neural information processing systems (NeurIPS)"},{"key":"2265_CR45","unstructured":"Zhao, P., Chen, P., Das, P., Ramamurthy, KN., & Lin, X. (2020). Bridging mode connectivity in loss landscapes and adversarial robustness. In: Proc. of the international conference on learning representations (ICLR)"},{"key":"2265_CR46","unstructured":"Zoph, B., & Le, QV. (2017). Neural architecture search with reinforcement learning. In: Proc. of the international conference on learning representations (ICLR)"}],"container-title":["International Journal of Computer Vision"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11263-024-02265-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11263-024-02265-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11263-024-02265-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,4,17]],"date-time":"2025-04-17T06:02:03Z","timestamp":1744869723000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11263-024-02265-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,9]]},"references-count":46,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,5]]}},"alternative-id":["2265"],"URL":"https:\/\/doi.org\/10.1007\/s11263-024-02265-7","relation":{},"ISSN":["0920-5691","1573-1405"],"issn-type":[{"value":"0920-5691","type":"print"},{"value":"1573-1405","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,9]]},"assertion":[{"value":"27 May 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 September 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 December 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}