{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T16:25:53Z","timestamp":1783009553299,"version":"3.54.5"},"reference-count":48,"publisher":"MDPI AG","issue":"13","license":[{"start":{"date-parts":[[2025,6,27]],"date-time":"2025-06-27T00:00:00Z","timestamp":1750982400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Office of the Director of National Intelligence (ODNI)"},{"name":"Intelligence Advanced Research Projects Activity (IARPA)"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Radio Frequency Machine Learning (RFML) has in recent years become a popular method for performing a variety of classification tasks on received signals. Among these tasks is Specific Emitter Identification (SEI), which seeks to associate a received signal with the physical emitter that transmitted it. Many different model architectures, including individual classifiers and ensemble methods, have proven their capabilities for producing high accuracy classification results when performing SEI. Though the works studying different model architectures report on successes, there is a notable absence regarding the examination of systemic failures and negative traits associated with learned behaviors. This work studies those failure patterns for a 64-radio SEI classification problem by isolating common patterns in incorrect classification results across multiple model architectures and two distinct control variables: Signal-to-Noise Ratio (SNR) and the quantity of training data utilized. This work finds that many of the RFML-based models devolve to selecting from amongst a small subset of classes (\u224810% of classes) as SNRs decrease and that observed errors are reasonably consistent across different SEI models and architectures. Moreover, our results validate the expectation that ensemble models are generally less brittle, particularly at a low SNR, yet they appear not to be the highest-performing option at a high SNR.<\/jats:p>","DOI":"10.3390\/s25134006","type":"journal-article","created":{"date-parts":[[2025,6,27]],"date-time":"2025-06-27T05:29:58Z","timestamp":1751002198000},"page":"4006","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Evaluation of Confusion Behaviors in SEI Models"],"prefix":"10.3390","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-6930-4035","authenticated-orcid":false,"given":"Brennan","family":"Olds","sequence":"first","affiliation":[{"name":"Virginia Tech National Security Institute, Blacksburg, VA 24060, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ethan","family":"Maas","sequence":"additional","affiliation":[{"name":"Virginia Tech National Security Institute, Blacksburg, VA 24060, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2437-3410","authenticated-orcid":false,"given":"Alan J.","family":"Michaels","sequence":"additional","affiliation":[{"name":"Virginia Tech National Security Institute, Blacksburg, VA 24060, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,6,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Rehman, S.U., Sowerby, K.W., Alam, S., and Ardekani, I. (2014, January 29\u201331). Radio Frequency Fingerprinting and its Challenges. Proceedings of the 2014 IEEE Conference on Communications and Network Security, San Francisco, CA, USA.","DOI":"10.1109\/CNS.2014.6997522"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1469","DOI":"10.1109\/JSAC.2011.110812","article-title":"Identifying Wireless Users via Transmitter Imperfections","volume":"29","author":"Polak","year":"2011","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"2305","DOI":"10.1109\/JSAC.2021.3087243","article-title":"An Efficient Specific Emitter Identification Method Based on Complex-Valued Neural Networks and Network Compression","volume":"39","author":"Wang","year":"2021","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2591","DOI":"10.1109\/LCOMM.2018.2871465","article-title":"Specific Emitter Identification via Convolutional Neural Networks","volume":"22","author":"Ding","year":"2018","journal-title":"IEEE Commun. Lett."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"54425","DOI":"10.1109\/ACCESS.2019.2913759","article-title":"Specific Emitter Identification Based on Deep Residual Networks","volume":"7","author":"Pan","year":"2019","journal-title":"IEEE Access"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"3791","DOI":"10.1109\/TIFS.2020.3001721","article-title":"Cooperative Specific Emitter Identification via Multiple Distorted Receivers","volume":"15","author":"He","year":"2020","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1192","DOI":"10.1109\/TIFS.2016.2520908","article-title":"Specific Emitter Identification via Hilbert\u2013Huang Transform in Single-Hop and Relaying Scenarios","volume":"11","author":"Zhang","year":"2016","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"5264526","DOI":"10.1155\/2018\/5264526","article-title":"An Ensemble Learning Method for Wireless Multimedia Device Identification","volume":"2018","author":"Zhang","year":"2018","journal-title":"Secur. Commun. Netw."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"592","DOI":"10.1109\/LCOMM.2021.3135378","article-title":"Specific Emitter Identification Based on Complex Fourier Neural Network","volume":"26","author":"Zha","year":"2022","journal-title":"IEEE Commun. Lett."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1177\/1548512921991245","article-title":"Training Data Augmentation for Deep Learning Radio Frequency Systems","volume":"18","author":"Clark","year":"2021","journal-title":"J. Def. Model. Simul."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Lin, Y., Jia, J., Wang, S., Ge, B., and Mao, S. (2020, January 7\u201311). Wireless Device Identification Based on Radio Frequency Fingerprint Features. Proceedings of the ICC 2020\u20142020 IEEE International Conference on Communications (ICC), Dublin, Ireland.","DOI":"10.1109\/ICC40277.2020.9149226"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/00401706.1988.10488313","article-title":"Signal-to-Noise Ratios, Performance Criteria, and Transformations","volume":"30","author":"Box","year":"1988","journal-title":"Technometrics"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2091","DOI":"10.1109\/TIFS.2016.2552146","article-title":"Wireless Physical-Layer Identification: Modeling and Validation","volume":"11","author":"Wang","year":"2016","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1145\/2379776.2379782","article-title":"On Physical-Layer Identification of Wireless Devices","volume":"45","author":"Danev","year":"2012","journal-title":"ACM Comput. Surv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Torra, V., and Narukawa, Y. (2021). Multinomial-Based Decision Synthesis of ML Classification Outputs. Modeling Decisions for Artificial Intelligence, Springer. Lecture Notes in Computer Science.","DOI":"10.1007\/978-3-030-85529-1"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"2776","DOI":"10.4249\/scholarpedia.2776","article-title":"Ensemble learning","volume":"4","author":"Polikar","year":"2009","journal-title":"Scholarpedia"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"14753","DOI":"10.1007\/s00521-021-06115-2","article-title":"Improving Security of the Internet of Things via RF Fingerprinting Based Device Identification System","volume":"33","author":"Abbas","year":"2021","journal-title":"Neural Comput. Appl."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1109\/TR.2014.2372432","article-title":"Improving ZigBee Device Network Authentication Using Ensemble Decision Tree Classifiers With Radio Frequency Distinct Native Attribute Fingerprinting","volume":"64","author":"Patel","year":"2015","journal-title":"IEEE Trans. Reliab."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Wong, L.J., and Michaels, A.J. (2022). Transfer Learning for Radio Frequency Machine Learning: A Taxonomy and Survey. Sensors, 22.","DOI":"10.3390\/s22041416"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"110023","DOI":"10.1109\/ACCESS.2023.3322105","article-title":"A Real-World Dataset Generator for Specific Emitter Identification","volume":"11","author":"Muller","year":"2023","journal-title":"IEEE Access"},{"key":"ref_21","unstructured":"Great Scott Gadgets (2023, December 01). YARD Stick One. Available online: https:\/\/greatscottgadgets.com\/yardstickone\/."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Muller, B., Olds, B., Wong, L., and Michaels, A. (2024). Transferring Learned Behaviors Between Similar and Different Radios. Sensors, 24.","DOI":"10.3390\/s24113574"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Olds, B., and Michaels, A. (2024, January 9\u201313). Temperature Sensitivity of RFML Algorithms. Proceedings of the 2024 IEEE International Conference on Communications Workshop on Machine Learning and Deep Learning for Wireless Security, Denver, CO, USA.","DOI":"10.1109\/ICCWorkshops59551.2024.10615857"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Matuszewski, J. (2008, January 21\u201323). Specific Emitter Identification. Proceedings of the 2008 International Radar Symposium, Wroclaw, Poland.","DOI":"10.1109\/IRS.2008.4585772"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"80327","DOI":"10.1109\/ACCESS.2024.3409471","article-title":"Sensitivity Analysis of RFML Applications","volume":"12","author":"Muller","year":"2024","journal-title":"IEEE Access"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Brik, V., Banerjee, S., Gruteser, M., and Oh, S. (2008;, January 14\u201319). Wireless device identification with radiometric signatures. Proceedings of the 14th ACM International Conference on Mobile Computing and Networking, San Francisco, CA, USA.","DOI":"10.1145\/1409944.1409959"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"500","DOI":"10.1109\/TIFS.2023.3324394","article-title":"Overcoming Data Limitations: A Few-Shot Specific Emitter Identification Method Using Self-Supervised Learning and Adversarial Augmentation","volume":"19","author":"Liu","year":"2024","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"333","DOI":"10.23919\/JSEE.2023.000054","article-title":"Specific emitter identification based on frequency and amplitude of the signal kurtosis","volume":"36","author":"Yurui","year":"2023","journal-title":"J. Syst. Eng. Electron."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Tyler, J.H., Fadul, M.K.M., and Reising, D.R. (2023). Considerations, Advances, and Challenges Associated with the Use of Specific Emitter Identification in the Security of Internet of Things Deployments: A Survey. Information, 14.","DOI":"10.20944\/preprints202307.1165.v1"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Peng, H., Xie, K., and Zou, W. (2024). Research on an Enhanced Multimodal Network for Specific Emitter Identification. Electronics, 13.","DOI":"10.3390\/electronics13030651"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1644","DOI":"10.1109\/ACCESS.2019.2962626","article-title":"Specific Emitter Identification Techniques for the Internet of Things","volume":"8","author":"Sa","year":"2020","journal-title":"IEEE Access"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"2667","DOI":"10.1109\/LCOMM.2023.3312390","article-title":"Specific Emitter Identification Based on Temporal Convolutional Network Sequence Processing","volume":"27","author":"Zhu","year":"2023","journal-title":"IEEE Commun. Lett."},{"key":"ref_33","unstructured":"Dudczyk, J., Matuszewski, J., and Wnuk, M. (2004, January 17\u201319). Applying the radiated emission to the specific emitter identification. Proceedings of the 15th International Conference on Microwaves, Radar and Wireless Communications (IEEE Cat. No. 04EX824), Warsaw, Poland."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Wang, S., Zhang, Y., Sun, J., Ohtsuki, T., and Gui, G. (2022, January 11\u201314). Attention-CNN-Aided Specific Emitter Identification Method with Limited Radio Frequency Dataset. Proceedings of the 2022 IEEE 22nd International Conference on Communication Technology (ICCT), Nanjing, China.","DOI":"10.1109\/ICCT56141.2022.10073149"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"133601","DOI":"10.1109\/ACCESS.2022.3232463","article-title":"Physical Layer-Based IoT Security: An Investigation Into Improving Preamble-Based SEI Performance When Using Multiple Waveform Collections","volume":"10","author":"Tyler","year":"2022","journal-title":"IEEE Access"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"3132","DOI":"10.1109\/TAES.2021.3074129","article-title":"A Method for Radar Model Identification Using Time-Domain Transient Signals","volume":"57","author":"Guo","year":"2021","journal-title":"IEEE Trans. Aerosp. Electron. Syst."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Shi, M., Huang, Y., and Wang, G. (2020, January 11\u201313). Receiver Distortion Nornalization Method for Specific Emitter Identification. Proceedings of the 2020 IEEE 9th Joint International Information Technology and Artificial Intelligence Conference (ITAIC), Chongqing, China.","DOI":"10.1109\/ITAIC49862.2020.9338787"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"3194","DOI":"10.1109\/TIFS.2024.3360851","article-title":"Improving RF-DNA Fingerprinting Performance in an Indoor Multipath Environment Using Semi-Supervised Learning","volume":"19","author":"Fadul","year":"2024","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"25117","DOI":"10.1109\/JIOT.2022.3195450","article-title":"An Adaptive Specific Emitter Identification System for Dynamic Noise Domain","volume":"9","author":"Zeng","year":"2022","journal-title":"IEEE Internet Things J."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Pan, J., Guo, L., Chen, Q., Zhang, S., and Xiong, J. (2022, January 1\u20133). Specific Radar Emitter Identification Using 1D-CBAM-ResNet. Proceedings of the 2022 14th International Conference on Wireless Communications and Signal Processing (WCSP), Nanjing, China.","DOI":"10.1109\/WCSP55476.2022.10039094"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"33544","DOI":"10.1109\/ACCESS.2019.2903444","article-title":"Specific Emitter Identification Using Convolutional Neural Network-Based IQ Imbalance Estimators","volume":"7","author":"Wong","year":"2019","journal-title":"IEEE Access"},{"key":"ref_42","unstructured":"Springenberg, J.T., Dosovitskiy, A., Brox, T., and Riedmiller, M. (2015). Striving for Simplicity: The All Convolutional Net. arXiv."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-Term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","article-title":"A mathematical theory of communication","volume":"27","author":"Shannon","year":"1948","journal-title":"Bell Syst. Tech. J."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"144054","DOI":"10.1109\/ACCESS.2019.2945121","article-title":"Variational Mode Decomposition-Based Radio Frequency Fingerprinting of Bluetooth Devices","volume":"7","author":"Aghnaiya","year":"2019","journal-title":"IEEE Access"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"50524","DOI":"10.1109\/ACCESS.2019.2911452","article-title":"Assessment of Features and Classifiers for Bluetooth RF Fingerprinting","volume":"7","author":"Ali","year":"2019","journal-title":"IEEE Access"},{"key":"ref_47","unstructured":"Toonstra, J., and Kinsner, W. (1995, January 15\u201316). Transient Analysis and Genetic Algorithms for Classification. Proceedings of the IEEE WESCANEX 95. Communications, Power, and Computing, Conference Proceedings, Winnipeg, MB, Canada."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Arechiga, A.P., and Michaels, A.J. (2018, January 8\u201310). The effect of weight errors on neural networks. Proceedings of the 2018 IEEE 8th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA.","DOI":"10.1109\/CCWC.2018.8301749"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/25\/13\/4006\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:59:51Z","timestamp":1760032791000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/25\/13\/4006"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,27]]},"references-count":48,"journal-issue":{"issue":"13","published-online":{"date-parts":[[2025,7]]}},"alternative-id":["s25134006"],"URL":"https:\/\/doi.org\/10.3390\/s25134006","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,27]]}}}