{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T16:12:37Z","timestamp":1780675957662,"version":"3.54.1"},"reference-count":48,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2025,7,21]],"date-time":"2025-07-21T00:00:00Z","timestamp":1753056000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Suqian Sci&amp;Tech Program","award":["K202415"],"award-info":[{"award-number":["K202415"]}]},{"name":"Suqian Sci&amp;Tech Program","award":["KX202037"],"award-info":[{"award-number":["KX202037"]}]},{"name":"Guangxi Key Laboratory of Trusted Software","award":["K202415"],"award-info":[{"award-number":["K202415"]}]},{"name":"Guangxi Key Laboratory of Trusted Software","award":["KX202037"],"award-info":[{"award-number":["KX202037"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Symmetry"],"abstract":"<jats:p>With the rapid proliferation of artificial intelligence (AI) applications, an increasing number of edge devices\u2014such as smartphones, cameras, and embedded controllers\u2014are being tasked with performing AI-based inference. Due to constraints in storage capacity, computational power, and network connectivity, these devices are often categorized as operating in resource-constrained environments. In such scenarios, deploying powerful Transformer-based models like ChatGPT and Vision Transformers is highly impractical because of their large parameter sizes and intensive computational requirements. While lightweight Transformer models, such as MobileViT, offer a promising solution to meet storage and computational limitations, their robustness remains insufficient. This poses a significant security risk for AI applications, particularly in critical edge environments. To address this challenge, our research focuses on enhancing the robustness of lightweight Transformer models under resource-constrained conditions. First, we propose a comprehensive robustness evaluation framework tailored for lightweight Transformer inference. This framework assesses model robustness across three key dimensions: noise robustness, distributional robustness, and adversarial robustness. It further investigates how model size and hardware limitations affect robustness, thereby providing valuable insights for robustness-aware model design. Second, we introduce a novel adversarial robustness enhancement strategy that integrates lightweight modeling techniques. This approach leverages methods such as gradient clipping and layer-wise unfreezing, as well as decision boundary optimization techniques like TRADES and SMART. Together, these strategies effectively address challenges related to training instability and decision boundary smoothness, significantly improving model robustness. Finally, we deploy the robust lightweight Transformer models in real-world resource-constrained environments and empirically validate their inference robustness. The results confirm the effectiveness of our proposed methods in enhancing the robustness and reliability of lightweight Transformers for edge AI applications.<\/jats:p>","DOI":"10.3390\/sym17071162","type":"journal-article","created":{"date-parts":[[2025,7,21]],"date-time":"2025-07-21T10:35:31Z","timestamp":1753094131000},"page":"1162","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Integrated Robust Optimization for Lightweight Transformer Models in Low-Resource Scenarios"],"prefix":"10.3390","volume":"17","author":[{"given":"Hui","family":"Huang","sequence":"first","affiliation":[{"name":"School of Traffic Management and Engineering, Guangxi Police College, Nanning 530028, China"},{"name":"School of Computer Science and Engineering, Beihang University, Beijing 100191, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hengyu","family":"Zhang","sequence":"additional","affiliation":[{"name":"College of Liberal Arts and Sciences, University of Florida, Gainesville, FL 32611, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yusen","family":"Wang","sequence":"additional","affiliation":[{"name":"Faculty of Science, The University of Sydney, Sydney, NSW 2050, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-1282-1250","authenticated-orcid":false,"given":"Haibin","family":"Liu","sequence":"additional","affiliation":[{"name":"Institut Montpellier Management, University of Montpellier, 34960 Montpellier, France"},{"name":"College of Business Administration, Lyceum of the Philippines University, Batangas 4200, Philippines"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaojie","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Finance, Fuzhou University of International Studies and Trade, Fuzhou 350202, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yiling","family":"Chen","sequence":"additional","affiliation":[{"name":"Guangxi Institute of Scientific and Technical Information, Nanning 530022, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6313-1097","authenticated-orcid":false,"given":"Yuan","family":"Liang","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Beihang University, Beijing 100191, China"},{"name":"Guangxi Key Laboratory of Trusted Software, Guilin University of Electronic Technology, Guilin 541000, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,7,21]]},"reference":[{"key":"ref_1","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4\u20139). Attention is All you Need. Proceedings of the NeurIPS, Long Beach, CA, USA."},{"key":"ref_2","unstructured":"Ahn, K., Cheng, X., Song, M., Yun, C., Jadbabaie, A., and Sra, S. (2024, January 7\u201311). Linear attention is (maybe) all you need (to understand Transformer optimization). Proceedings of the ICLR, Vienna, Austria."},{"key":"ref_3","unstructured":"Yaslioglu, M.M. (2025). Attention is All You Need Until You Need Retention. arXiv."},{"key":"ref_4","unstructured":"Meng, F., Yao, Z., and Zhang, M. (2025). TransMLA: Multi-Head Latent Attention Is All You Need. arXiv."},{"key":"ref_5","unstructured":"Yu, L., Zhang, H., and Xu, C. (2024). Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models. arXiv."},{"key":"ref_6","unstructured":"Devlin, J., Chang, M., Lee, K., and Toutanova, K. (August, January 28). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the ACL, Florence, Italy."},{"key":"ref_7","unstructured":"Patel, A., Li, B., Rasooli, M.S., Constant, N., Raffel, C., and Callison-Burch, C. (2023, January 1\u20135). Bidirectional Language Models Are Also Few-shot Learners. Proceedings of the ICLR, Kigali, Rwanda."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Madaan, A., Zhou, S., Alon, U., Yang, Y., and Neubig, G. (2022, January 7\u201311). Language Models of Code are Few-Shot Commonsense Learners. Proceedings of the EMNLP, Abu Dhabi, United Arab Emirates.","DOI":"10.18653\/v1\/2022.emnlp-main.90"},{"key":"ref_9","unstructured":"Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020, January 6\u201312). Language Models are Few-Shot Learners. Proceedings of the NeurIPS, Virtual."},{"key":"ref_10","unstructured":"Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and J\u00e9gou, H. (2021, January 18\u201324). Training data-efficient image transformers & distillation through attention. Proceedings of the ICML, Virtual."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Heo, B., Yun, S., Han, D., Chun, S., Choe, J., and Oh, S.J. (2021, January 11\u201317). Rethinking Spatial Dimensions of Vision Transformers. Proceedings of the ICCV, Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.01172"},{"key":"ref_12","unstructured":"Mehta, S., and Rastegari, M. (2022, January 25\u201329). MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer. Proceedings of the ICLR, Virtual Event."},{"key":"ref_13","first-page":"441","article-title":"Separable Self-attention for Mobile Vision Transformers","volume":"2023","author":"Mehta","year":"2023","journal-title":"Trans. Mach. Learn. Res."},{"key":"ref_14","unstructured":"Wadekar, S.N., and Chaurasia, A. (2022). MobileViTv3: Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Features. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Wang, J., Hao, Q., Huang, W., Fan, X., Tang, Z., Wang, B., Hao, J., and Li, Y. (2024, January 25\u201329). DyPS: Dynamic Parameter Sharing in Multi-Agent Reinforcement Learning for Spatio-Temporal Resource Allocation. Proceedings of the KDD, Barcelona, Spain.","DOI":"10.1145\/3637528.3672052"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1099","DOI":"10.1109\/TIFS.2023.3331239","article-title":"Machine Unlearning via Representation Forgetting With Parameter Self-Sharing","volume":"19","author":"Wang","year":"2024","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Poppi, S., Sarto, S., Cornia, M., Baraldi, L., and Cucchiara, R. (2024, January 7\u201311). Unlearning Vision Transformers Without Retaining Data via Low-Rank Decompositions. Proceedings of the ICLR, Vienna, Austria.","DOI":"10.1007\/978-3-031-78122-3_10"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1964","DOI":"10.1109\/TITS.2024.3495697","article-title":"A Global-Local Fusion Model via Edge Enhancement and Transformer for Pavement Crack Defect Segmentation","volume":"26","author":"Yang","year":"2025","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.neucom.2022.01.081","article-title":"ACORT: A compact object relation transformer for parameter efficient image captioning","volume":"482","author":"Tan","year":"2022","journal-title":"Neurocomputing"},{"key":"ref_20","unstructured":"Moens, M., Huang, X., Specia, L., and Yih, S.W. (2021, January 7\u201311). Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers. Proceedings of the EMNLP, Virtual Event."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Zafrir, O., Boudoukh, G., Izsak, P., and Wasserblat, M. (2019, January 8\u201314). Q8BERT: Quantized 8Bit BERT. Proceedings of the NeurIPS, Vancouver, CA, Canada.","DOI":"10.1109\/EMC2-NIPS53020.2019.00016"},{"key":"ref_22","unstructured":"Wang, S., Li, B.Z., Khabsa, M., Fang, H., and Ma, H. (2020). Linformer: Self-Attention with Linear Complexity. arXiv."},{"key":"ref_23","unstructured":"Cahyawijaya, S. (2021). Greenformers: Improving Computation and Memory Efficiency in Transformer Models via Low-Rank Approximation. arXiv."},{"key":"ref_24","unstructured":"Hendrycks, D., and Dietterich, T.G. (2019, January 6\u20139). Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. Proceedings of the ICLR, New Orleans, LA, USA."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., and Guo, M. (2021, January 11\u201317). The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization. Proceedings of the ICCV, Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00823"},{"key":"ref_26","unstructured":"Goodfellow, I.J., Shlens, J., and Szegedy, C. (2015, January 7\u20139). Explaining and Harnessing Adversarial Examples. Proceedings of the ICLR, San Diego, CA, USA."},{"key":"ref_27","unstructured":"Kurakin, A., Goodfellow, I.J., and Bengio, S. (2017, January 24\u201326). Adversarial examples in the physical world. Proceedings of the ICLR, Toulon, France."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"8457","DOI":"10.1109\/TIFS.2024.3453041","article-title":"Adversarial Examples Against WiFi Fingerprint-Based Localization in the Physical World","volume":"19","author":"Wang","year":"2024","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., and Song, D. (2018, January 18\u201322). Robust Physical-World Attacks on Deep Learning Visual Classification. Proceedings of the CVPR, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00175"},{"key":"ref_30","first-page":"1","article-title":"Inversion of Magnetic Data Based on L1 Norm and Total Variation Regularization","volume":"63","author":"Peng","year":"2025","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1007\/s10878-024-01250-7","article-title":"An L2 regularization reduced quadratic surface support vector machine model","volume":"49","author":"Wang","year":"2025","journal-title":"J. Comb. Optim."},{"key":"ref_32","first-page":"1929","article-title":"Dropout: A simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_33","unstructured":"Yang, Y., Lin, C., Ji, X., Tian, Q., Li, Q., Yang, H., Wang, Z., and Shen, C. (2023). Towards Deep Learning Models Resistant to Transfer-based Adversarial Attacks via Data-centric Robust Learning. arXiv."},{"key":"ref_34","unstructured":"Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. (May, January 30). Towards Deep Learning Models Resistant to Adversarial Attacks. Proceedings of the ICLR, Vancouver, BC, Canada."},{"key":"ref_35","unstructured":"Chen, J., Wu, X., Guo, Y., Liang, Y., and Jha, S. (2022, January 25\u201329). Towards Evaluating the Robustness of Neural Networks Learned by Transduction. Proceedings of the ICLR, Online."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Carlini, N., and Wagner, D.A. (2017, January 22\u201324). Towards Evaluating the Robustness of Neural Networks. Proceedings of the IEEE Symposium on Security and Privacy, San Jose, CA, USA.","DOI":"10.1109\/SP.2017.49"},{"key":"ref_37","unstructured":"Croce, F., and Hein, M. (2020, January 13\u201318). Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. Proceedings of the ICML, Virtual Event."},{"key":"ref_38","unstructured":"Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., Sakhaee, E., Li, N., Basart, S., and Li, B. (2024, January 21\u201327). HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. Proceedings of the ICML, Vienna, Austria."},{"key":"ref_39","unstructured":"Wang, H., Ge, S., Lipton, Z.C., and Xing, E.P. (2019, January 8\u201314). Learning Robust Global Representations by Penalizing Local Predictive Power. Proceedings of the NeurIPS, Vancouver, BC, Canada."},{"key":"ref_40","unstructured":"Yu, M., and Sun, S. (2022, January 6\u201310). Natural Black-Box Adversarial Examples against Deep Reinforcement Learning. Proceedings of the AAAI, Virtual."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. (2021, January 19\u201325). Natural Adversarial Examples. Proceedings of the CVPR, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.01501"},{"key":"ref_42","unstructured":"Goodfellow, I.J., Bengio, Y., and Courville, A.C. (2016). Deep Learning, MIT Press. Adaptive Computation and Machine Learning."},{"key":"ref_43","unstructured":"Prioleau, H., and Aryal, S.K. (April, January 31). Entity Only vs. Inline Approaches: Evaluating LLMs for Adverse Drug Event Detection in Clinical Text (Student Abstract). Proceedings of the AAAI, Philadelphia, PA, USA."},{"key":"ref_44","unstructured":"Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., and Madry, A. (2019, January 6\u20139). Robustness May Be at Odds with Accuracy. Proceedings of the ICLR, New Orleans, LA, USA."},{"key":"ref_45","unstructured":"Benz, P., Zhang, C., Karjauv, A., and Kweon, I.S. (2020, January 6\u201312). Robustness May Be at Odds with Fairness: An Empirical Study on Class-wise Accuracy. Proceedings of the NeurIPS, Online."},{"key":"ref_46","unstructured":"Zhang, H., Yu, Y., Jiao, J., Xing, E.P., Ghaoui, L.E., and Jordan, M.I. (2019, January 9\u201315). Theoretically Principled Trade-off between Robustness and Accuracy. Proceedings of the ICML, Long Beach, CA, USA."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Zhao, T. (2020, January 5\u201310). SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization. Proceedings of the ACL, Online.","DOI":"10.18653\/v1\/2020.acl-main.197"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"ImageNet Large Scale Visual Recognition Challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis."}],"container-title":["Symmetry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-8994\/17\/7\/1162\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:13:10Z","timestamp":1760033590000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-8994\/17\/7\/1162"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,21]]},"references-count":48,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2025,7]]}},"alternative-id":["sym17071162"],"URL":"https:\/\/doi.org\/10.3390\/sym17071162","relation":{},"ISSN":["2073-8994"],"issn-type":[{"value":"2073-8994","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,21]]}}}