{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T15:54:29Z","timestamp":1787241269220,"version":"build-2736575974"},"reference-count":57,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2025,12,23]],"date-time":"2025-12-23T00:00:00Z","timestamp":1766448000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Lightweight vision models are widely deployed in mobile and embedded systems, where strict computational and memory budgets demand compact architectures. However, their evaluation remains dominated by ImageNet\u2014a single, large natural-image dataset that requires substantial training resources. This creates a dilemma: lightweight models trained on ImageNet often reach capacity limits due to their constrained size, while scaling them to billions of parameters with specialized training tricks to achieve top-tier ImageNet accuracy does not guarantee proportional performance once the architectures are scaled back down to meet mobile constraints, particularly when re-evaluated on diverse data domains. These challenges raise two key questions: How should cross-dataset robustness be quantified in a simple and lightweight way, and which architectural elements consistently support generalization under tight resource constraints? To answer them, we introduce the Cross-Dataset Score (xScore), a simple metric that captures both average accuracy across domains and the stability of model rankings. Evaluating 11 representative lightweight models (2.5 M parameters) across seven datasets, we find that (1) ImageNet accuracy is a weak proxy for cross-domain performance, (2) xScore provides a simple and interpretable robustness metric, and (3) high-xScore models reveal architectural patterns linked to stronger generalization. Finally, the architectural insights and evaluation framework presented here provide practical guidance for measuring the xScore of future lightweight models.<\/jats:p>","DOI":"10.3390\/a19010014","type":"journal-article","created":{"date-parts":[[2025,12,23]],"date-time":"2025-12-23T10:57:18Z","timestamp":1766487438000},"page":"14","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["xScore: A Simple Metric for Cross-Domain Robustness in Lightweight Vision Models"],"prefix":"10.3390","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6180-9909","authenticated-orcid":false,"given":"Weidong","family":"Zhang","sequence":"first","affiliation":[{"name":"School of Computing and AI, Arizona State University, Tempe, AZ 85281, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pak Lun Kevin","family":"Ding","sequence":"additional","affiliation":[{"name":"School of Computing and AI, Arizona State University, Tempe, AZ 85281, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Baoxin","family":"Li","sequence":"additional","affiliation":[{"name":"School of Computing and AI, Arizona State University, Tempe, AZ 85281, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3264-7904","authenticated-orcid":false,"given":"Huan","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Computing and AI, Arizona State University, Tempe, AZ 85281, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,12,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_2","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 3\u20137). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Proceedings of the International Conference on Learning Representations (ICLR), Online."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Han, K., Wang, Y., Tian, Q., Guo, J., Xu, C., and Xu, C. (2020, January 13\u201319). GhostNet: More features from cheap operations. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00165"},{"key":"ref_4","unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv."},{"key":"ref_5","unstructured":"Mehta, S., and Rastegari, M. (2021). MobileViT: Light-weight, general-purpose, and mobile-friendly vision transformer. arXiv."},{"key":"ref_6","unstructured":"Tan, M., and Le, Q.V. (2019, January 9\u201315). EfficientNet: Rethinking model scaling for convolutional neural networks. Proceedings of the International Conference on Machine Learning (ICML), Long Beach, CA, USA. PMLR."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Vasu, P.K.A., Gabriel, J., Zhu, J., Tuzel, O., and Ranjan, A. (2023, January 18\u201322). MobileOne: An improved one millisecond mobile backbone. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.00764"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Zhang, X., Zhou, X., Lin, M., and Sun, J. (2018, January 18\u201322). ShuffleNet: An extremely efficient convolutional neural network for mobile devices. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00716"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Wu, B., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., Jia, Y., and Keutzer, K. (2019, January 16\u201320). FBNet: Hardware-aware efficient convnet design via differentiable neural architecture search. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.01099"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Zhou, D., Hou, Q., Chen, Y., Feng, J., and Yan, S. (2020, January 23\u201328). Rethinking bottleneck structure for efficient mobile network design. Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK.","DOI":"10.1007\/978-3-030-58580-8_40"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1109\/MCAS.2023.3302182","article-title":"Tiny Machine Learning: Progress and Futures","volume":"23","author":"Lin","year":"2023","journal-title":"IEEE Circuits Syst. Mag."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Liu, H.I., Galindo, M., Xie, H., Wong, L.K., Shuai, H.H., Li, Y.H., and Cheng, W.H. (2024). Lightweight Deep Learning for Resource-Constrained Environments: A Survey. arXiv.","DOI":"10.1145\/3657282"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3578938","article-title":"Efficient deep learning: A survey on making deep learning models smaller, faster, and better","volume":"55","author":"Menghani","year":"2023","journal-title":"ACM Comput. Surv."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Wang, X., and Jia, W. (2025). Optimizing Edge AI: A Comprehensive Survey on Data, Model, and System Strategies. arXiv.","DOI":"10.32388\/IZOHCH"},{"key":"ref_15","first-page":"1","article-title":"Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models","volume":"57","author":"Wang","year":"2025","journal-title":"ACM Comput. Surv."},{"key":"ref_16","unstructured":"Fang, A., Kornblith, S., and Schmidt, L. (2023, January 10\u201316). Does progress on ImageNet transfer to real-world datasets?. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Kornblith, S., Shlens, J., and Le, Q.V. (2019, January 16\u201320). Do Better ImageNet Models Transfer Better?. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00277"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Tuggener, L., Schmidhuber, J., and Stadelmann, T. (2022). Is it enough to optimize CNN architectures on ImageNet?. Front. Comput. Sci., 4.","DOI":"10.3389\/fcomp.2022.1041703"},{"key":"ref_19","unstructured":"Vishniakov, K., Shen, Z., and Liu, Z. (2023). ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy. arXiv."},{"key":"ref_20","unstructured":"Tan, M., and Le, Q.V. (2021, January 18\u201324). EfficientNetV2: Smaller models and faster training. Proceedings of the International Conference on Machine Learning (ICML), Online. PMLR."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Ma, N., Zhang, X., Zheng, H., and Sun, J. (2018, January 8\u201314). ShuffleNet V2: Practical guidelines for efficient CNN architecture design. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01264-9_8"},{"key":"ref_22","unstructured":"Tang, Y., Han, K., Guo, J., Xu, C., Xu, C., and Wang, Y. (December, January 28). GhostNetV2: Enhance cheap operation with long-range attention. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Wan, A., Dai, X., Zhang, P., He, Z., Tian, Y., Xie, S., Wu, B., Yu, M., Xu, T., and Chen, K. (2020, January 14\u201319). FBNetV2: Differentiable neural architecture search for spatial and channel dimensions. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01298"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., and Sun, J. (2020, January 23\u201328). Single path one-shot neural architecture search with uniform sampling. Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK.","DOI":"10.1007\/978-3-030-58517-4_32"},{"key":"ref_25","unstructured":"Li, Y., Yuan, G., Wen, Y., Hu, J., Evangelidis, G., Tulyakov, S., and Ren, J. (December, January 28). EfficientFormer: Vision transformers at MobileNet speed. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Shaker, A., Maaz, M., Rasheed, H., Khan, S., Yang, M.H., and Khan, F.S. (2023, January 2\u20136). SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Paris, France.","DOI":"10.1109\/ICCV51070.2023.01598"},{"key":"ref_27","unstructured":"Gulrajani, I., and Lopez-Paz, D. (2020). In search of lost domain generalization. arXiv."},{"key":"ref_28","unstructured":"Koh, P.W., Sagawa, S., Marklund, H., Xie, S.M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R.L., and Gao, I. (2021, January 18\u201324). Wilds: A benchmark of in-the-wild distribution shifts. Proceedings of the International Conference on Machine Learning (ICML), Online."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). ImageNet: A large-scale hierarchical image database. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_30","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20138). ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the 25th International Conference on Neural Information Processing Systems (NeurIPS), Lake Tahoe, NV, USA."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18\u201322). MobileNetV2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00474"},{"key":"ref_32","unstructured":"Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., and Vasudevan, V. (November, January 27). Searching for MobileNetV3. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Hu, J., Shen, L., and Sun, G. (2018, January 18\u201322). Squeeze-and-Excitation Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00745"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., and Le, Q.V. (2019, January 15\u201320). MnasNet: Platform-aware neural architecture search for mobile. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00293"},{"key":"ref_35","unstructured":"Han, K., Wang, Y., Zhang, Q., Zhang, W., Xu, C., and Zhang, T. (2020). Model Rubik\u2019s Cube: Twisting Resolution, Depth and Width for TinyNets. arXiv."},{"key":"ref_36","unstructured":"Zhang, W. (2025, October 31). xScore: Evaluation Framework with Training Wrapper and Dataset Loader. GitHub Repository. Available online: https:\/\/github.com\/javawormer\/beyond_imagenet."},{"key":"ref_37","unstructured":"Krizhevsky, A., and Hinton, G. (2009). Learning Multiple Layers of Features from Tiny Images, University of Toronto. Technical Report; CIFAR-10 and CIFAR-100 datasets."},{"key":"ref_38","unstructured":"Khosla, A., Jayadevaprakash, B., Yao, B., and Fei-Fei, L. (2011, January 20\u201325). Novel dataset for fine-grained image categorization: Stanford Dogs. Proceedings of the CVPR Workshop on Fine-Grained Visual Categorization, Colorado Springs, CO, USA."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1240","DOI":"10.1007\/s42452-025-07798-1","article-title":"Evaluating unified training optimisations for MobileNetV2: Efficiency-accuracy trade-offs in fine-grained dog breed classification","volume":"7","author":"Martins","year":"2025","journal-title":"Discov. Appl. Sci."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"180161","DOI":"10.1038\/sdata.2018.161","article-title":"The HAM10000 dataset: A large collection of multi-source dermatoscopic images of common pigmented skin lesions","volume":"5","author":"Tschandl","year":"2018","journal-title":"Sci. Data"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022, January 19\u201324). A ConvNet for the 2020s. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01167"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Ma, X., Dai, X., Bai, Y., Wang, Y., and Fu, Y. (2024, January 16\u201320). Rewrite the Stars. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Waikoloa, HI, USA.","DOI":"10.1109\/CVPR52733.2024.00544"},{"key":"ref_43","unstructured":"Howard, J. (2025, July 05). Imagenette: A Subset of ImageNet for Faster Experimentation. Available online: https:\/\/github.com\/fastai\/imagenette."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Quattoni, A., and Torralba, A. (2009, January 20\u201325). Recognizing indoor scenes. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA.","DOI":"10.1109\/CVPRW.2009.5206537"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1452","DOI":"10.1109\/TPAMI.2017.2723009","article-title":"Places: A 10 million Image Database for Scene Recognition","volume":"40","author":"Zhou","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_46","unstructured":"Kingma, D.P. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_47","unstructured":"Loshchilov, I., and Hutter, F. (2017, January 24\u201326). SGDR: Stochastic Gradient Descent with Warm Restarts. Proceedings of the International Conference on Learning Representations (ICLR), Toulon, France."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27\u201330). Rethinking the Inception Architecture for Computer Vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.308"},{"key":"ref_49","unstructured":"Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., and Yoo, Y. (November, January 27). CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea."},{"key":"ref_50","unstructured":"Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., and Dosovitskiy, A. (2021, January 6\u201314). Do Vision Transformers See Like Convolutional Neural Networks?. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Virtual."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Draper, N., and Smith, H. (1998). Applied Regression Analysis, Wiley. [3rd ed.].","DOI":"10.1002\/9781118625590"},{"key":"ref_52","unstructured":"Trockman, A., and Kolter, J.Z. (2022). Patches Are All You Need?. arXiv."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Ding, X., Wang, X., Zhao, Y., Li, Y., Tang, Y., and Sun, J. (2022, January 18\u201324). Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01166"},{"key":"ref_54","unstructured":"Zhang, L., Song, J., Gao, A., Chen, J., Bao, C., and Ma, K. (November, January 27). Be your own teacher: Improve the performance of convolutional neural networks via self distillation. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"4262","DOI":"10.1109\/TMM.2022.3192663","article-title":"CDFKD-MFS: Collaborative Data-Free Knowledge Distillation via Multi-Level Feature Sharing","volume":"24","author":"Hao","year":"2022","journal-title":"IEEE Trans. Multimed."},{"key":"ref_56","unstructured":"Liang, Y., Chongjian, G., Tong, Z., Song, Y., Wang, J., and Xie, P. (April, January 25). EViT: Expediting Vision Transformers via Token Reorganizations. Proceedings of the International Conference on Learning Representations (ICLR), Online."},{"key":"ref_57","unstructured":"Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., and Hsieh, C.J. (2021, January 6\u201314). DynamicViT: Efficient vision transformers with dynamic token sparsification. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS 2021), Virtual Conference."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/19\/1\/14\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,23]],"date-time":"2025-12-23T11:02:20Z","timestamp":1766487740000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/19\/1\/14"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,23]]},"references-count":57,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,1]]}},"alternative-id":["a19010014"],"URL":"https:\/\/doi.org\/10.3390\/a19010014","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,23]]}}}