{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,7]],"date-time":"2026-11-07T00:50:06Z","timestamp":1794012606178,"version":"4.8.0"},"reference-count":35,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T00:00:00Z","timestamp":1791331200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T00:00:00Z","timestamp":1791331200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>With the accelerated proliferation of the Internet of Things (IoT) and its widespread use in sensitive areas such as healthcare, homes, and factories, security has become an increasingly critical concern. IoT remains one of the most challenging domains to secure, lacking the manageable scale, well-maintained software, and homogeneous environments of easily secured systems. Large language models (LLMs) have recently been applied to improve IoT security; however, most applications treat LLMs as \u201coff-the-shelf\u201d tools, creating a domain mismatch because model representations, tokenizers, and data collators are designed for natural language rather than network traffic. In this work, we present CyberRoBERTa, built specifically to test whether departing from off-the-shelf LLM components, through a domain-specific Unigram tokenizer and an IoT-specialized data collator, yields a genuine advantage for IoT intrusion detection. We instantiate this framework as a compact, 10.2-million-parameter model designed for fully self-contained, on-device operation. Using CICIoT2023, TON_IoT, and Edge-IIoTset, the model achieves a binary classification F1-score of 96% and a multiclass F1-score of 89.54%. We further validate these results in three ways: a comparison against a classical machine learning baseline shows comparable accuracy; isolating the tokenizer\u2019s contribution from pretraining scale shows its benefit holds under matched pretraining conditions; and evaluating class-imbalance interventions shows improved detection on the most data-scarce class.<\/jats:p>","DOI":"10.3390\/s26196326","type":"journal-article","created":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T10:13:02Z","timestamp":1791367982000},"page":"6326","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["CyberRoBERTa: A Lightweight Transformer-Based Language Model for IoT Intrusion Detection"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-8248-9917","authenticated-orcid":false,"given":"Dhai","family":"Alhafi","sequence":"first","affiliation":[{"name":"Computer Science Department, College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University (IMSIU), Riyadh 11432, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-2073-9576","authenticated-orcid":false,"given":"Fatmah","family":"Alanazi","sequence":"additional","affiliation":[{"name":"Computer Science Department, College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University (IMSIU), Riyadh 11432, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,10,7]]},"reference":[{"key":"ref_1","first-page":"1","article-title":"Internet of Things for Smart Cities: Technologies, Applications and Challenges","volume":"8","author":"Jameel","year":"2026","journal-title":"Int. J. Multidiscip. Res. (IJFMR)"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"127017","DOI":"10.1016\/j.neucom.2023.127017","article-title":"A review of IoT applications in healthcare","volume":"565","author":"Li","year":"2024","journal-title":"Neurocomputing"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"9275962","DOI":"10.1049\/ise2\/9275962","article-title":"A Review on Integrating IoT, IIoT, and Industry 4.0: A Pathway to Smart Manufacturing and Digital Transformation","volume":"2025","author":"Qiu","year":"2025","journal-title":"IET Inf. Secur."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Vardakis, G., Hatzivasilis, G., Koutsaki, E., and Papadakis, N. (2024). Review of Smart-Home Security Using the Internet of Things. Electronics, 13.","DOI":"10.3390\/electronics13163343"},{"key":"ref_5","unstructured":"IoT Analytics (2026, April 10). State of IoT 2025: Number of Connected IoT Devices Growing 14% to 21.1 Billion Globally. Available online: https:\/\/iot-analytics.com\/number-connected-iot-devices\/."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1080\/23742917.2023.2228053","article-title":"Securing IoT Devices Against Emerging Security Threats: Challenges and Mitigation Techniques","volume":"7","author":"Kabir","year":"2023","journal-title":"J. Cyber Secur. Technol."},{"key":"ref_7","first-page":"5776","article-title":"Security Challenges in IoT: A Systematic Literature Review","volume":"10","year":"2025","journal-title":"Eng. Technol. J."},{"key":"ref_8","unstructured":"Feng, Y., and Sakurai, K. (2025). Network Intrusion Detection: Evolution from Conventional Approaches to LLM Collaboration and Emerging Risks. arXiv."},{"key":"ref_9","first-page":"64","article-title":"Real-Time Intrusion Detection Leveraging Deep Learning: A Comparative Analysis of CNN, RNN, and Transformer Architectures","volume":"11","author":"Musthafa","year":"2025","journal-title":"Int. J. Adv. Eng. Manag. Sci."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Chkirbene, Z., Hamila, R., Gouissem, A., and Devrim, U. (2024). Large Language Models (LLM) in Industry: A Survey of Applications, Challenges, and Trends. Proceedings of the 2024 IEEE 21st International Conference on Smart Communities: Improving Quality of Life using AI, Robotics and IoT (HONET), IEEE.","DOI":"10.1109\/HONET63146.2024.10822885"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Jaffal, N.O., Alkhanafseh, M., and Mohaisen, D. (2025). Large Language Models in Cybersecurity: A Survey of Applications, Vulnerabilities, and Defense Techniques. AI, 6.","DOI":"10.3390\/ai6090216"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Diaf, A., Korba, A.A., Elislem Karabadji, N., and Ghamri-Doudane, Y. (2024). Beyond Detection: Leveraging Large Language Models for Cyber Attack Prediction in IoT Networks. Proceedings of the 2024 20th International Conference on Distributed Computing in Smart Systems and the Internet of Things (DCOSS-IoT), Los Alamitos, CA, USA, 2024, IEEE.","DOI":"10.1109\/DCOSS-IoT61029.2024.00026"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1909","DOI":"10.1109\/TVT.2024.3402366","article-title":"IoV-BERT-IDS: Hybrid Network Intrusion Detection System in IoV Using Large Language Models","volume":"74","author":"Fu","year":"2025","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"23733","DOI":"10.1109\/ACCESS.2024.3363469","article-title":"Revolutionizing Cyber Threat Detection With Large Language Models: A Privacy-Preserving BERT-Based Lightweight Model for IoT\/IIoT Devices","volume":"12","author":"Ferrag","year":"2024","journal-title":"IEEE Access"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Chowdhury, S., Purushotham, E., Srinivasan, A., and Sreeraman, Y. (2024). Predicting Cyber-Attacks Through the Use of Deep Learning Algorithms. Proceedings of the 2024 5th International Conference on Innovative Trends in Information Technology (ICITIIT), IEEE.","DOI":"10.1109\/ICITIIT61487.2024.10580049"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"138","DOI":"10.1007\/s10586-024-04775-y","article-title":"EBIDS: Efficient BERT-based intrusion detection system in the network and application layers of IoT","volume":"28","author":"Sattarpour","year":"2024","journal-title":"Clust. Comput."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Bommarito, M.J., Katz, D.M., and Bommarito, J. (2025). KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications. arXiv.","DOI":"10.2139\/ssrn.5188502"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"6563","DOI":"10.1038\/s41598-026-37438-6","article-title":"Medical knowledge representation enhancement in large language models through clinical tokens optimization","volume":"16","author":"Li","year":"2026","journal-title":"Sci. Rep."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"e3126","DOI":"10.7717\/peerj-cs.3126","article-title":"Transformer-based tokenization for IoT traffic classification across diverse network environments","volume":"11","author":"Afifi","year":"2025","journal-title":"PeerJ Comput. Sci."},{"key":"ref_20","unstructured":"Vlachos, A., and Augenstein, I. (2023). Should You Mask 15% in Masked Language Modeling?. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Association for Computational Linguistics."},{"key":"ref_21","unstructured":"Che, W., Nabende, J., Shutova, E., and Pilehvar, M.T. (2025). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Alrefaei, A., and Ilyas, M. (2024). Using Machine Learning Multiclass Classification Technique to Detect IoT Attacks in Real Time. Sensors, 24.","DOI":"10.3390\/s24144516"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Craciun, R., Caramihai, S.I., Mocanu, S.A., Pietraru, R.N., and Moisescu, M.A. (2025). Hybrid Machine Learning for IoT-Enabled Smart Buildings. Informatics, 12.","DOI":"10.3390\/informatics12010017"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Chen, J., Xiao, J., and Xu, J. (2024). VGGIncepNet: Enhancing Network Intrusion Detection and Network Security through Non-Image-to-Image Conversion and Deep Learning. Electronics, 13.","DOI":"10.3390\/electronics13183639"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"929","DOI":"10.26599\/TST.2023.9010033","article-title":"Enhanced IDS with Deep Learning for IoT-Based Smart Cities Security","volume":"29","author":"Hazman","year":"2024","journal-title":"Tsinghua Sci. Technol."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Adjewa, F., Esseghir, M., and Merghem-Boulahia, L. (2026). From Edge Transformer to IoT Decisions: Offloaded Embeddings for Lightweight Intrusion Detection. Sensors, 26.","DOI":"10.3390\/s26020356"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Guo, R., Li, A., and Liu, H. (2025). An Adversarial Attack Detection Method Based on Bidirectional Consistency Discrimination for Deep Learning-Based Soft Sensors. Proceedings of the 2025 CAA Symposium on Fault Detection, Supervision, and Safety for Technical Processes (SAFEPROCESS), Urumqi, China, 22\u201324 August 2025, IEEE.","DOI":"10.1109\/SAFEPROCESS67117.2025.11268032"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"6062","DOI":"10.1109\/JIOT.2025.3525494","article-title":"Transformer-Based Intrusion Detection for IoT Networks","volume":"12","author":"Akuthota","year":"2025","journal-title":"IEEE Internet Things J."},{"key":"ref_29","unstructured":"Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (2021). TABBIE: Pretrained Representations of Tabular Data. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Neto, E.C.P., Dadkhah, S., Ferreira, R., Zohourian, A., Lu, R., and Ghorbani, A.A. (2023). CICIoT2023: A Real-Time Dataset and Benchmark for Large-Scale Attacks in IoT Environment. Sensors, 23.","DOI":"10.20944\/preprints202305.0443.v1"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"165130","DOI":"10.1109\/ACCESS.2020.3022862","article-title":"TON_IoT Telemetry Dataset: A New Generation Dataset of IoT and IIoT for Data-Driven Intrusion Detection Systems","volume":"8","author":"Alsaedi","year":"2020","journal-title":"IEEE Access"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"40281","DOI":"10.1109\/ACCESS.2022.3165809","article-title":"Edge-IIoTset: A New Comprehensive Realistic Cyber Security Dataset of IoT and IIoT Applications: Centralized and Federated Learning","volume":"10","author":"Ferrag","year":"2022","journal-title":"IEEE Access"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"108719","DOI":"10.1016\/j.comnet.2021.108719","article-title":"NFStream: A flexible network data analysis framework","volume":"204","author":"Aouini","year":"2022","journal-title":"Comput. Netw."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Lin, T., Goyal, P., Girshick, R.B., He, K., and Doll\u00e1r, P. (2017). Focal Loss for Dense Object Detection. arXiv.","DOI":"10.1109\/ICCV.2017.324"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Grinsztajn, L., Oyallon, E., and Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on tabular data?. arXiv.","DOI":"10.52202\/068431-0037"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/19\/6326\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,14]],"date-time":"2026-10-14T04:14:41Z","timestamp":1791951281000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/19\/6326"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,7]]},"references-count":35,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2026,10]]}},"alternative-id":["s26196326"],"URL":"https:\/\/doi.org\/10.3390\/s26196326","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,7]]}}}