{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,2]],"date-time":"2026-01-02T07:34:12Z","timestamp":1767339252036,"version":"build-2065373602"},"reference-count":49,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T00:00:00Z","timestamp":1721865600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Office of the Director of National Intelligence (ODNI)"},{"name":"Intelligence Advanced Research Projects Activity (IARPA)"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>The use of transfer learning (TL) techniques has become common practice in fields such as computer vision (CV) and natural language processing (NLP). Leveraging prior knowledge gained from data with different distributions, TL offers higher performance and reduced training time, but has yet to be fully utilized in applications of machine learning (ML) and deep learning (DL) techniques and applications related to wireless communications, a field loosely termed radio frequency machine learning (RFML). This work examines whether existing transferability metrics, used in other modalities, might be useful in the context of RFML. Results show that the two existing metrics tested, Log Expected Empirical Prediction (LEEP) and Logarithm of Maximum Evidence (LogME), correlate well with post-transfer accuracy and can therefore be used to select source models for radio frequency (RF) domain adaptation and to predict post-transfer accuracy.<\/jats:p>","DOI":"10.3390\/make6030084","type":"journal-article","created":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T13:10:28Z","timestamp":1721913028000},"page":"1699-1719","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Assessing the Value of Transfer Learning Metrics for Radio Frequency Domain Adaptation"],"prefix":"10.3390","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5896-0176","authenticated-orcid":false,"given":"Lauren J.","family":"Wong","sequence":"first","affiliation":[{"name":"Intel AI Lab, Santa Clara, CA 95054, USA"},{"name":"National Security Institute, Virginia Tech, Blacksburg, VA 24060, USA"},{"name":"Bradley Department of Electrical and Computer Engineering, Virginia Tech, Blacksburg, VA 24060, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-2040-3101","authenticated-orcid":false,"given":"Braeden P.","family":"Muller","sequence":"additional","affiliation":[{"name":"National Security Institute, Virginia Tech, Blacksburg, VA 24060, USA"},{"name":"Bradley Department of Electrical and Computer Engineering, Virginia Tech, Blacksburg, VA 24060, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6244-6038","authenticated-orcid":false,"given":"Sean","family":"McPherson","sequence":"additional","affiliation":[{"name":"Intel AI Lab, Santa Clara, CA 95054, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2437-3410","authenticated-orcid":false,"given":"Alan J.","family":"Michaels","sequence":"additional","affiliation":[{"name":"National Security Institute, Virginia Tech, Blacksburg, VA 24060, USA"},{"name":"Bradley Department of Electrical and Computer Engineering, Virginia Tech, Blacksburg, VA 24060, USA"}]}],"member":"1968","published-online":{"date-parts":[[2024,7,25]]},"reference":[{"key":"ref_1","unstructured":"Mitola, J. (2000). Cognitive Radio: An Integrated Agent Architecture for Software Defined Radio. [Ph.D. Dissertation, Royal Institute of Technology]."},{"key":"ref_2","unstructured":"Rondeau, T. (2022, March 24). Radio Frequency Machine Learning Systems (RFMLS). Available online: https:\/\/www.darpa.mil\/program\/radio-frequency-machine-learning-systems."},{"key":"ref_3","unstructured":"Kolb, P. (2022, March 24). Securing Compartmented Information with Smart Radio Systems (SCISRS), Available online: https:\/\/www.iarpa.gov\/index.php\/research-programs\/scisrs."},{"key":"ref_4","unstructured":"(2024, March 24). Conference, IEEE Communications Society. In Proceedings of the DySPAN 2021: 2020 IEEE International Symposium on Dynamic Spectrum Access Networks, Los Angeles, CA, USA, 13\u201315 December 2021. Available online: https:\/\/dyspan2021.ieee-dyspan.org\/index.html."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"137184","DOI":"10.1109\/ACCESS.2019.2942390","article-title":"Machine Learning for 5G\/B5G Mobile and Wireless Communications: Potential, Limitations, and Future Directions","volume":"7","author":"Lee","year":"2019","journal-title":"IEEE Access"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Wong, L.J., and Michaels, A.J. (2022). Transfer Learning for Radio Frequency Machine Learning: A Taxonomy and Survey. Sensors, 22.","DOI":"10.3390\/s22041416"},{"key":"ref_7","unstructured":"Hauser, S.C. (2018). Real-World Considerations for Deep Learning in Spectrum Sensing. [Master\u2019s Thesis, Virginia Tech]."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Sankhe, K., Belgiovine, M., Zhou, F., Riyaz, S., Ioannidis, S., and Chowdhury, K. (2019\u20132, January 29). ORACLE: Optimized Radio Classification through Convolutional Neural Networks. Proceedings of the IEEE INFOCOM 2019-IEEE Conference on Computer Communications, Paris, France.","DOI":"10.1109\/INFOCOM.2019.8737463"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2243","DOI":"10.1109\/OJCOMS.2021.3112939","article-title":"An RFML Ecosystem: Considerations for the Application of Deep Learning to Spectrum Situational Awareness","volume":"2","author":"Wong","year":"2021","journal-title":"IEEE Open J. Commun. Soc."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Wong, L.J., Muller, B.P., McPherson, S., and Michaels, A.J. (2024). An Analysis of Radio Frequency Transfer Learning Behavior. Mach. Learn. Knowl. Extr., 6.","DOI":"10.3390\/make6020057"},{"key":"ref_11","unstructured":"Nguyen, C., Hassner, T., Seeger, M., and Archambeau, C. (2020, January 13\u201318). LEEP: A new measure to evaluate transferability of learned representations. Proceedings of the International Conference on Machine Learning, PMLR, Online."},{"key":"ref_12","unstructured":"You, K., Liu, Y., Long, M., and Wang, J. (2021). LogME: Practical Assessment of Pre-trained Models for Transfer Learning. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1049\/iet-com:20050176","article-title":"Survey of automatic modulation classification techniques: Classical approaches and new trends","volume":"1","author":"Dobre","year":"2007","journal-title":"IET Commun."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"West, N.E., and O\u2019Shea, T. (2017, January 6\u20139). Deep architectures for modulation recognition. Proceedings of the 2017 IEEE International Symposium on Dynamic Spectrum Access Networks (DySPAN), Baltimore, MD, USA.","DOI":"10.1109\/DySPAN.2017.7920754"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"89256","DOI":"10.1109\/ACCESS.2019.2925569","article-title":"Deep Learning for Large-Scale Real-World ACARS and ADS-B Radio Signal Classification","volume":"7","author":"Chen","year":"2019","journal-title":"IEEE Access"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"164529","DOI":"10.1109\/ACCESS.2020.3022513","article-title":"A Deep Convolutional Neural Network Based Transfer Learning Method for Non-Cooperative Spectrum Sensing","volume":"8","author":"Pati","year":"2020","journal-title":"IEEE Access"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Kuzdeba, S., Robinson, J., and Carmack, J. (2021, January 9\u201312). Transfer Learning with Radio Frequency Signals. Proceedings of the 2021 IEEE 18th Annual Consumer Communications Networking Conference (CCNC), Las Vegas, NV, USA.","DOI":"10.1109\/CCNC49032.2021.9369550"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Robinson, J., and Kuzdeba, S. (2021, January 9\u201312). RiftNet: Radio Frequency Classification for Large Populations. Proceedings of the 2021 IEEE 18th Annual Consumer Communications Networking Conference (CCNC), Las Vegas, NV, USA.","DOI":"10.1109\/CCNC49032.2021.9369455"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1109\/JSTSP.2018.2797022","article-title":"Over-the-Air Deep Learning Based Radio Signal Classification","volume":"12","author":"Roy","year":"2018","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"132","DOI":"10.1109\/JSTSP.2017.2784180","article-title":"Deep Learning Based Communication Over the Air","volume":"12","author":"Cammerer","year":"2018","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"138","DOI":"10.23919\/JCC.2020.02.012","article-title":"Spectrum sensing based on deep learning classification for cognitive radios","volume":"17","author":"Zheng","year":"2020","journal-title":"China Comm."},{"key":"ref_22","unstructured":"Clark, B., Leffke, Z., Headley, C., and Michaels, A. (2019). Cyborg Phase II Final Report, Ted and Karyn Hume Center for National Security and Technology. Technical report."},{"key":"ref_23","first-page":"154851292199124","article-title":"Training data augmentation for deep learning radio frequency systems","volume":"18","author":"Hauser","year":"2020","journal-title":"J. Def. Model. Simul."},{"key":"ref_24","unstructured":"Merchant, K. (2019). Deep Neural Networks for Radio Frequency Fingerprinting. [PhD Thesis, University of Maryland]."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A Survey on Transfer Learning","volume":"22","author":"Pan","year":"2010","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Zamir, A.R., Sax, A., Shen, W., Guibas, L.J., Malik, J., and Savarese, S. (2018, January 18\u201323). Taskonomy: Disentangling task transfer learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00391"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Achille, A., Lam, M., Tewari, R., Ravichandran, A., Maji, S., Fowlkes, C.C., Soatto, S., and Perona, P. (2019\u20132, January 27). Task2Vec: Task embedding for meta-learning. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Republic of Korea.","DOI":"10.1109\/ICCV.2019.00653"},{"key":"ref_28","unstructured":"Huang, L.K., Wei, Y., Rong, Y., Yang, Q., and Huang, J. (2021). Frustratingly Easy Transferability Estimation. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Tan, Y., Li, Y., and Huang, S.L. (2021, January 20\u201325). OTCE: A Transferability Metric for Cross-Domain Cross-Task Representations. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.01552"},{"key":"ref_30","unstructured":"Tan, Y., Li, Y., and Huang, S.L. (2021). Practical Transferability Estimation for Image Classification Tasks. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"P\u00e1ndy, M., Agostinelli, A., Uijlings, J., Ferrari, V., and Mensink, T. (2021). Transferability Estimation using Bhattacharyya Class Separability. arXiv.","DOI":"10.1109\/CVPR52688.2022.00896"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Tran, A.T., Nguyen, C.V., and Hassner, T. (2019\u20132, January 27). Transferability and hardness of supervised classification tasks. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Republic of Korea.","DOI":"10.1109\/ICCV.2019.00148"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Bao, Y., Li, Y., Huang, S.L., Zhang, L., Zheng, L., Zamir, A., and Guibas, L. (2019, January 22\u201325). An Information-Theoretic Approach to Transferability in Task Transfer Learning. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.","DOI":"10.1109\/ICIP.2019.8803726"},{"key":"ref_34","unstructured":"Renggli, C., Pinto, A.S., Rimanic, L., Puigcerver, J., Riquelme, C., Zhang, C., and Lucic, M. (2020). Which model to transfer? Finding the needle in the growing haystack. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Li, Y., Jia, X., Sang, R., Zhu, Y., Green, B., Wang, L., and Gong, B. (2021, January 20\u201325). Ranking neural checkpoints. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00269"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Bhattacharjee, B., Kender, J.R., Hill, M., Dube, P., Huo, S., Glass, M.R., Belgodere, B., Pankanti, S., Codella, N., and Watson, P. (2020, January 14\u201319). P2L: Predicting transfer learning for images and semantic relations. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.","DOI":"10.1109\/CVPRW50498.2020.00388"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Ruder, S., and Plank, B. (2017). Learning to select data for transfer learning with bayesian optimization. arXiv.","DOI":"10.18653\/v1\/D17-1038"},{"key":"ref_38","unstructured":"Kashyap, A.R., Hazarika, D., Kan, M.Y., and Zimmermann, R. (2020). Domain divergences: A survey and empirical analysis. arXiv."},{"key":"ref_39","unstructured":"Van Asch, V., and Daelemans, W. (2010, January 15). Using domain similarity for performance estimation. Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing, Uppsala, Sweden."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Elsahar, H., and Gall\u00e9, M. (2019, January 3\u20137). To annotate or not? Predicting performance drop under domain shift. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.","DOI":"10.18653\/v1\/D19-1222"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Pogrebnyakov, N., and Shaghaghian, S. (2021). Predicting the Success of Domain Adaptation in Text Similarity. arXiv.","DOI":"10.18653\/v1\/2021.repl4nlp-1.21"},{"key":"ref_42","unstructured":"Wong, L., McPherson, S., and Michaels, A. (2022, March 24). Transfer Learning for RF Domain Adaptation-Synthetic Dataset. Available online: https:\/\/ieee-dataport.org\/open-access\/transfer-learning-rf-domain-adaptation-%E2%80%93-synthetic-dataset."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Clark IV, W.H., and Michaels, A.J. (2022). Quantifying and extrapolating data needs in radio frequency machine learning. arXiv.","DOI":"10.1109\/MILCOM52596.2021.9652987"},{"key":"ref_44","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_45","unstructured":"(2022, March 24). Cross Entropy Loss, PyTorch 1.10.1 Documentation. Available online: https:\/\/pytorch.org\/docs\/stable\/generated\/torch.nn.CrossEntropyLoss.html."},{"key":"ref_46","first-page":"8026","article-title":"PyTorch: An imperative style, high-performance deep learning library","volume":"32","author":"Paszke","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"1763","DOI":"10.1213\/ANE.0000000000002864","article-title":"Correlation coefficients: Appropriate use and interpretation","volume":"126","author":"Schober","year":"2018","journal-title":"Anesth. Analg."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1093\/biomet\/30.1-2.81","article-title":"A new measure of rank correlation","volume":"30","author":"Kendall","year":"1938","journal-title":"Biometrika"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"4125","DOI":"10.21037\/jtd.2017.09.14","article-title":"Using the confidence interval confidently","volume":"9","author":"Hazra","year":"2017","journal-title":"J. Thorac. Dis."}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/6\/3\/84\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:23:22Z","timestamp":1760109802000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/6\/3\/84"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,25]]},"references-count":49,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,9]]}},"alternative-id":["make6030084"],"URL":"https:\/\/doi.org\/10.3390\/make6030084","relation":{},"ISSN":["2504-4990"],"issn-type":[{"type":"electronic","value":"2504-4990"}],"subject":[],"published":{"date-parts":[[2024,7,25]]}}}