{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T11:31:17Z","timestamp":1775129477866,"version":"3.50.1"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2024,3,15]],"date-time":"2024-03-15T00:00:00Z","timestamp":1710460800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,3,15]],"date-time":"2024-03-15T00:00:00Z","timestamp":1710460800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001809","name":"the National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61973103"],"award-info":[{"award-number":["61973103"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001809","name":"the National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62303164"],"award-info":[{"award-number":["62303164"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"the Outstanding Young Scholar of Henan Province","award":["222300420039"],"award-info":[{"award-number":["222300420039"]}]},{"name":"the Key R&D Program of Henan Province","award":["222102220009"],"award-info":[{"award-number":["222102220009"]}]},{"name":"the Collaborative Innovation Special Project of Zhengzhou","award":["21ZZXTCX01"],"award-info":[{"award-number":["21ZZXTCX01"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Process Lett"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Due to the well-known domain shift problem, directly deploying a trained multi-modal classifier to a new environment usually leads to poor performance. The existing multi-modal domain adaption methods not only lack the fine-grained information of cross-modal data distribution, but also lack the cross-modal correlation research. Therefore, this paper proposes a multi-modal domain adaption method based on parameter fusion and two-step alignment (PFTS) to solve the related problems. The consistency of network parameters is used to enhance the correlation among modalities, and a higher-order moment measurement is introduced to improve the alignment of data distribution at the fine-grained level. In addition, the weighting of each modality is further carried out to achieve focused transfer. Comprehensive experiments based on multi-modal datasets with different domain adaption settings have been conducted, the results show that the precision of PFTS is 5.38% higher than state-of-the-art multi-modal domain adaption methods.<\/jats:p>","DOI":"10.1007\/s11063-024-11567-3","type":"journal-article","created":{"date-parts":[[2024,3,15]],"date-time":"2024-03-15T16:01:51Z","timestamp":1710518511000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Multi-modal Domain Adaptation Method Based on Parameter Fusion and Two-Step Alignment"],"prefix":"10.1007","volume":"56","author":[{"given":"Lan","family":"Wu","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Han","family":"Wang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lishuang","family":"Gong","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuan","family":"Yao","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xin","family":"Guo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Binquan","family":"Li","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,3,15]]},"reference":[{"key":"11567_CR1","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1109\/TPAMI.2018.2798607","volume":"41","author":"T Baltru\u0161aitis","year":"2018","unstructured":"Baltru\u0161aitis T, Ahuja C, Morency LP (2018) multi-modal machine learning: a survey and taxonomy. IEEE Trans pattern anal machine intell 41:423\u2013443. https:\/\/doi.org\/10.1109\/TPAMI.2018.2798607","journal-title":"IEEE Trans pattern anal machine intell"},{"key":"11567_CR2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2682899","volume":"47","author":"SK D\u2019mello","year":"2015","unstructured":"D\u2019mello SK, Kory J (2015) A review and meta-analysis of multi-modal affect detection system. ACM Comput Surveys 47:1\u201336. https:\/\/doi.org\/10.1145\/2682899","journal-title":"ACM Comput Surveys"},{"key":"11567_CR3","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1145\/1322192.1322216","volume":"31","author":"Z Zeng","year":"2008","unstructured":"Zeng Z, Pantic M, Roisman GI (2008) A survey of affect recognition methods: audio, visiual, and spontaneous expressions. IEEE Trans pattern analy machine intell 31:39\u201358. https:\/\/doi.org\/10.1145\/1322192.1322216","journal-title":"IEEE Trans pattern analy machine intell"},{"key":"11567_CR4","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-85099-1_8","author":"G Castellano","year":"2008","unstructured":"Castellano G, Kessous L, Caridakis G (2008) Emotion recognition through multiple modalities: face, body, gesture, speech. Affect Emotion Human-Comput Interact. https:\/\/doi.org\/10.1007\/978-3-540-85099-1_8","journal-title":"Affect Emotion Human-Comput Interact"},{"key":"11567_CR5","doi-asserted-by":"publisher","first-page":"396","DOI":"10.1007\/978-3-642-24571-8_51","volume":"9\u201312","author":"GA Ramirez","year":"2011","unstructured":"Ramirez GA, Baltru\u0161aitis T, Morency LP (2011) Modeling latent discriminative dynamic of multi-dimensional affective signals. Process Int Conf Aff Comput Intell Interact 9\u201312:396\u2013406. https:\/\/doi.org\/10.1007\/978-3-642-24571-8_51","journal-title":"Process Int Conf Aff Comput Intell Interact"},{"key":"11567_CR6","doi-asserted-by":"publisher","first-page":"333","DOI":"10.1007\/s11042-013-1391-2","volume":"71","author":"Z Lan","year":"2014","unstructured":"Lan Z, Bao L, Yu SI et al (2014) Multimedia classification and event detection using double fusion. Multimed Tools Appl 71:333\u2013347. https:\/\/doi.org\/10.1007\/s11042-013-1391-2","journal-title":"Multimed Tools Appl"},{"key":"11567_CR7","doi-asserted-by":"crossref","unstructured":"Ouyang J, Lv Q, Zhang S, et al (2023) Energy Transfer Contrast Network for Unsupervised Domain Adaption. In: International Conference on Multimedia Modeling Springer Nature Switzerland, pp 115\u2013126","DOI":"10.1007\/978-3-031-27818-1_10"},{"key":"11567_CR8","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2020.107803","volume":"112","author":"W Wang","year":"2021","unstructured":"Wang W, Chen S, Xiang Y et al (2021) Sparsely-labeled source assisted domain adaptation. Pattern Recogn 112:107803. https:\/\/doi.org\/10.1016\/j.patcog.2020.107803","journal-title":"Pattern Recogn"},{"issue":"11","key":"11567_CR9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TNNLS.2021.3073119","volume":"33","author":"A Ma","year":"2021","unstructured":"Ma A, Li J, Lu K et al (2021) Adversarial entropy optimization for unsupervised domain adaptation. IEEE Trans Neural Netw Learn Syst 33(11):1\u201312. https:\/\/doi.org\/10.1109\/TNNLS.2021.3073119","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"issue":"4","key":"11567_CR10","doi-asserted-by":"publisher","first-page":"1713","DOI":"10.1109\/TNNLS.2020.2988928","volume":"32","author":"Y Zhu","year":"2020","unstructured":"Zhu Y, Zhuang F, Wang J et al (2020) Deep subdomain adaptation network for image classification. IEEE Trans Neural Netw Learn Syst 32(4):1713\u20131722. https:\/\/doi.org\/10.1109\/TNNLS.2020.2988928","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"key":"11567_CR11","doi-asserted-by":"publisher","first-page":"8200","DOI":"10.1109\/TIP.2021.3113576","volume":"30","author":"T Jing","year":"2021","unstructured":"Jing T, Xu B, Ding Z (2021) Towards fair knowledge transfer for imbalanced domain adaptation. IEEE Trans Image Process 30:8200\u20138211. https:\/\/doi.org\/10.1109\/TIP.2021.3113576","journal-title":"IEEE Trans Image Process"},{"issue":"1","key":"11567_CR12","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1109\/TPAMI.2014.2343216","volume":"37","author":"M Xiao","year":"2014","unstructured":"Xiao M, Guo Y (2014) Feature space independent semi-supervised domain adaptation via kernel matching. IEEE Trans Pattern Analy Machine Intell 37(1):54\u201366. https:\/\/doi.org\/10.1109\/TPAMI.2014.2343216","journal-title":"IEEE Trans Pattern Analy Machine Intell"},{"issue":"12","key":"11567_CR13","doi-asserted-by":"publisher","first-page":"2539","DOI":"10.1109\/TPAMI.2016.2643667","volume":"39","author":"M Elhoseiny","year":"2016","unstructured":"Elhoseiny M, Elgammal A, Saleh B (2016) Write a classifier: predicting visual classifiers from unstructured text. IEEE Trans Pattern Analy Machine Intell 39(12):2539\u20132553. https:\/\/doi.org\/10.1109\/TPAMI.2016.2643667","journal-title":"IEEE Trans Pattern Analy Machine Intell"},{"issue":"6","key":"11567_CR14","doi-asserted-by":"publisher","first-page":"1134","DOI":"10.1109\/TPAMI.2013.167","volume":"36","author":"W Li","year":"2013","unstructured":"Li W, Duan L, Xu D et al (2013) Learning with augmented features for supervised and semi-supervised heterogeneous domain adaptation. IEEE Trans Pattern Analy Machine Intell 36(6):1134\u20131148. https:\/\/doi.org\/10.1109\/TPAMI.2013.167","journal-title":"IEEE Trans Pattern Analy Machine Intell"},{"issue":"11","key":"11567_CR15","doi-asserted-by":"publisher","first-page":"5308","DOI":"10.1109\/TIP.2019.2917867","volume":"28","author":"F Yu","year":"2019","unstructured":"Yu F, Wu X, Chen J et al (2019) Exploiting images for video recognition: Heterogeneous feature augmentation via symmetric adversarial learning. IEEE Trans Image Process 28(11):5308\u20135321. https:\/\/doi.org\/10.1109\/TIP.2019.2917867","journal-title":"IEEE Trans Image Process"},{"issue":"5","key":"11567_CR16","doi-asserted-by":"publisher","first-page":"1220","DOI":"10.1109\/TBDATA.2020.3034201","volume":"8","author":"G Song","year":"2020","unstructured":"Song G, Zhang Y, Xu L et al (2020) Domain adaptive network embedding. IEEE Trans Big Data 8(5):1220\u20131232. https:\/\/doi.org\/10.1109\/TBDATA.2020.3034201","journal-title":"IEEE Trans Big Data"},{"issue":"9","key":"11567_CR17","doi-asserted-by":"publisher","first-page":"2372","DOI":"10.1109\/TCSVT.2017.2705068","volume":"28","author":"Y Peng","year":"2017","unstructured":"Peng Y, Huang X, Zhao Y (2017) An overview of cross-media retrieval: Concepts, methodologies, benchmarks, and challenges. IEEE Trans Circuits Syst Video Technol 28(9):2372\u20132385. https:\/\/doi.org\/10.1109\/TCSVT.2017.2705068","journal-title":"IEEE Trans Circuits Syst Video Technol"},{"issue":"4","key":"11567_CR18","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1109\/TBDATA.2017.2656918","volume":"5","author":"Z Xu","year":"2017","unstructured":"Xu Z, Mei L, Lv Z et al (2017) Multi-modal description of public safety events using surveillance and social media. IEEE Trans Big Data 5(4):529\u2013539. https:\/\/doi.org\/10.1109\/TBDATA.2017.2656918","journal-title":"IEEE Trans Big Data"},{"key":"11567_CR19","doi-asserted-by":"crossref","unstructured":"Yao Y, Li Y, Zhang P et al (2018) Data fusion methods for convolutional neural network based on self-sensing motor drive system. In: Proceedings of the IECON 2018\u201444th Annual Conference of the IEEE Industrial Electronics Society, 21\u201323, pp 5371\u20135376","DOI":"10.1109\/IECON.2018.8591160"},{"issue":"21","key":"11567_CR20","doi-asserted-by":"publisher","first-page":"24540","DOI":"10.1109\/JSEN.2021.3115016","volume":"21","author":"H Cao","year":"2021","unstructured":"Cao H, Chen G, Xia J et al (2021) Fusion-based feature attention gate component for vehicle detection based on event camera. IEEE Sens J 21(21):24540\u201324548. https:\/\/doi.org\/10.1109\/JSEN.2021.3115016","journal-title":"IEEE Sens J"},{"key":"11567_CR21","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1016\/j.patrec.2014.08.005","volume":"51","author":"X Jiang","year":"2015","unstructured":"Jiang X, Wu F, Zhang Y et al (2015) The classification of multi-modal data with hidden conditional random field. Pattern Recogn Lett 51:63\u201369. https:\/\/doi.org\/10.1016\/j.patrec.2014.08.005","journal-title":"Pattern Recogn Lett"},{"issue":"2","key":"11567_CR22","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1109\/TMM.2015.2510329","volume":"18","author":"S Qian","year":"2015","unstructured":"Qian S, Zhang T, Xu C et al (2015) Multi-modal event topic model for social event analysis. IEEE Trans Multimed 18(2):233\u2013246. https:\/\/doi.org\/10.1109\/TMM.2015.2510329","journal-title":"IEEE Trans Multimed"},{"issue":"8","key":"11567_CR23","doi-asserted-by":"publisher","first-page":"3748","DOI":"10.1109\/TIP.2016.2639438","volume":"26","author":"Y Xia","year":"2016","unstructured":"Xia Y, Zhang L, Liu Z et al (2016) Weakly supervised multi-modal kernel for categorizing aerial photographs. IEEE Trans Image Process 26(8):3748\u20133758. https:\/\/doi.org\/10.1109\/TIP.2016.2639438","journal-title":"IEEE Trans Image Process"},{"key":"11567_CR24","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1016\/j.neucom.2016.08.126","volume":"259","author":"T Feng","year":"2017","unstructured":"Feng T, Mao X (2017) multi-modal data fusion for SB-JPALS status prediction under antenna motion fault mode. Neurocomputing 259:46\u201354. https:\/\/doi.org\/10.1016\/j.neucom.2016.08.126","journal-title":"Neurocomputing"},{"issue":"12","key":"11567_CR25","doi-asserted-by":"publisher","first-page":"2816","DOI":"10.1109\/TMM.2017.2713408","volume":"19","author":"H Li","year":"2017","unstructured":"Li H, Sun J, Xu Z et al (2017) multi-modal 2D+3D facial expression recognition with deep fusion convolutional neural network. IEEE Trans Multimed 19(12):2816\u20132831. https:\/\/doi.org\/10.1109\/TMM.2017.2713408","journal-title":"IEEE Trans Multimed"},{"key":"11567_CR26","doi-asserted-by":"crossref","unstructured":"Khan MNA, Heisterkamp DR (2016) Adapting instance weights for unsupervised domain adaptation using quadratic mutual information and subspace learning. In: Proceedings of the 23rd ICPR, 4\u20138, pp 1560\u20131565","DOI":"10.1109\/ICPR.2016.7899859"},{"key":"11567_CR27","doi-asserted-by":"crossref","unstructured":"Cortes C, Mohri M, Riley M et al (2008) Sample selection bias correction theory. In: Proceedings of the 19th international conference on Algorithmic Learning Theory, 13\u201316, pp 38\u201353","DOI":"10.1007\/978-3-540-87987-9_8"},{"key":"11567_CR28","doi-asserted-by":"publisher","first-page":"3209","DOI":"10.1109\/CVPR.2011.5995729","volume":"20\u201325","author":"J Liu","year":"2011","unstructured":"Liu J, Shah M, Kuipers B et al (2011) Cross-view action recognition via view knowledge transfer. Proceed CVPR 20\u201325:3209\u20133216. https:\/\/doi.org\/10.1109\/CVPR.2011.5995729","journal-title":"Proceed CVPR"},{"issue":"3","key":"11567_CR29","doi-asserted-by":"publisher","first-page":"465","DOI":"10.1109\/TPAMI.2011.114","volume":"34","author":"L Duan","year":"2021","unstructured":"Duan L, Tsang IW, Xu D (2021) Domain transfer multiple kernel learning. IEEE Trans Pattern Anal Machine Intell 34(3):465\u2013479. https:\/\/doi.org\/10.1109\/TPAMI.2011.114","journal-title":"IEEE Trans Pattern Anal Machine Intell"},{"key":"11567_CR30","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.neunet.2014.01.008","volume":"53","author":"WY Deng","year":"2014","unstructured":"Deng WY, Zheng QH, Wang ZM (2014) Cross-person activity recognition using reduced kernel extreme learning machine. Neural Netw 53:1\u20137. https:\/\/doi.org\/10.1016\/j.neunet.2014.01.008","journal-title":"Neural Netw"},{"key":"11567_CR31","unstructured":"Ganin Y, Lempitsky V (2014) Unsupervised domain adaptation by backpropagation. arXiv preprint 2014, arXiv:1409.7495"},{"key":"11567_CR32","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2021.3105868","author":"Y Yao","year":"2023","unstructured":"Yao Y, Li X, Zhang Y, Ye Y (2023) Multisource heterogeneous domain adaptation with conditional weighting adversarial network. IEEE Trans Neural Netw Learn Syst. https:\/\/doi.org\/10.1109\/TNNLS.2021.3105868","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"key":"11567_CR33","doi-asserted-by":"publisher","first-page":"1541","DOI":"10.1016\/j.jpdc.2017.06.003","volume":"16\u201322","author":"C Wang","year":"2011","unstructured":"Wang C, Mahadevan S (2011) Heterogeneous domain adaptation using manifold alignment. Proceed Int Jt Conf Artif Intell 16\u201322:1541. https:\/\/doi.org\/10.1016\/j.jpdc.2017.06.003","journal-title":"Proceed Int Jt Conf Artif Intell"},{"key":"11567_CR34","doi-asserted-by":"crossref","unstructured":"Tsai YHH, Yeh YR, Wang YCF (2016) Learning cross-domain landmarks for heterogeneous domain adaptation. In: proceedings of the IEEE conference on computer vision and pattern recognition, pp 5081\u20135090","DOI":"10.1109\/CVPR.2016.549"},{"key":"11567_CR35","doi-asserted-by":"crossref","unstructured":"Li S, Xie B, Wu J et al (2020) Simultaneous semantic alignment network for heterogeneous domain adaptation. In: Proceedings of the 28th ACM international conference on multimedia, pp 3866\u20133874","DOI":"10.1145\/3394171.3413995"},{"key":"11567_CR36","doi-asserted-by":"crossref","unstructured":"Chen WY, Hsu TMH, Tsai YHH et al (2016) Transfer neural trees for heterogeneous domain adaptation. In: Computer Vision\u2013ECCV 2016: 14th European Conference 11\u201314, pp 399\u2013414","DOI":"10.1007\/978-3-319-46454-1_25"},{"key":"11567_CR37","doi-asserted-by":"crossref","unstructured":"Yao Y, Zhang Y, Li X, Ye Y (2019) Heterogeneous domain adaptation via soft transfer network. In: Proceedings of the 27th ACM International Conference 21\u201325, pp 1578\u20131586","DOI":"10.1145\/3343031.3350955"},{"issue":"1","key":"11567_CR38","doi-asserted-by":"publisher","first-page":"1087","DOI":"10.1109\/TPAMI.2022.3146234","volume":"45","author":"Z Fang","year":"2023","unstructured":"Fang Z, Lu J, Liu F et al (2023) Semi-supervised heterogeneous domain adaptation: theory and algorithms. IEEE Trans Pattern Anal Mach Intell 45(1):1087\u20131105. https:\/\/doi.org\/10.1109\/TPAMI.2022.3146234","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"issue":"10","key":"11567_CR39","doi-asserted-by":"publisher","first-page":"10328","DOI":"10.1109\/TCYB.2021.3070545","volume":"52","author":"Q Tian","year":"2022","unstructured":"Tian Q, Sun H, Ma C et al (2022) Heterogeneous domain adaptation with structure and classification space alignment. IEEE Transact Cybernetics 52(10):10328\u201310338. https:\/\/doi.org\/10.1109\/TCYB.2021.3070545","journal-title":"IEEE Transact Cybernetics"},{"key":"11567_CR40","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2021.108362","volume":"123","author":"Q Wang","year":"2022","unstructured":"Wang Q, Breckon TP (2022) Cross-domain structure preserving projection for heterogeneous domain adaptation. Pattern Recogn 123:108362. https:\/\/doi.org\/10.1016\/j.patcog.2021.108362","journal-title":"Pattern Recogn"}],"container-title":["Neural Processing Letters"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11063-024-11567-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11063-024-11567-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11063-024-11567-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,16]],"date-time":"2024-05-16T20:36:47Z","timestamp":1715891807000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11063-024-11567-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,15]]},"references-count":40,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,4]]}},"alternative-id":["11567"],"URL":"https:\/\/doi.org\/10.1007\/s11063-024-11567-3","relation":{},"ISSN":["1573-773X"],"issn-type":[{"value":"1573-773X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,15]]},"assertion":[{"value":"11 February 2024","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 March 2024","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of interest"}}],"article-number":"107"}}