{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,22]],"date-time":"2026-01-22T01:28:50Z","timestamp":1769045330734,"version":"3.49.0"},"reference-count":34,"publisher":"World Scientific Pub Co Pte Ltd","issue":"04","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Int. J. Human. Robot."],"published-print":{"date-parts":[[2025,8]]},"abstract":"<jats:p> With the rapid development of artificial intelligence technology, the demand for applying humanoid robots in complex environments is gradually increasing. How to effectively process multimodal information and improve the efficiency of behavioral decision-making has become a hot topic in current research. In response to the low efficiency of behavior decision-making and generation for humanoid robots, single data modality, and difficulty in effectively integrating information from different modalities, this paper combines Pathways Language Model 2 (PaLM 2) and multimodal Transformer architecture to study the multimodal information processing and behavior generation of humanoid robots. The experiment is based on preprocessed visual, tactile, and auditory data. A multimodal Transformer architecture is designed to fuse the multimodal data, and a soft attention mechanism is used for preliminary feature fusion. It further processes and optimizes the fused features through a multi-head attention mechanism, outputs high-dimensional feature vectors, and then uses the PaLM 2 model to understand natural speech instructions and model the context to generate accurate task descriptions. Finally, the Deep Q-Network (DQN) algorithm can be used to create behavior for humanoid robots, and a [Formula: see text]-greedy strategy can be adopted to select actions, improving the efficiency of multimodal information processing and behavior generation for humanoid robots. The experiment is based on the public normal-robots behavior dataset and the field collection and recording data of the humanoid robot company, combined with the multimodal Transformer-Pathways Language Model 2-Deep Q-Network (Transformer-PaLM 2-DQN). The humanoid robot can more accurately understand the multimodal data in complex environments, and show higher adaptability and efficiency in behavior generation and task execution. The generation efficiency reached 92.67% and 7.38% higher than Deep Deterministic Policy Gradient (DDPG) and 10.22% higher than unimodal visual information, proving the key role of multimodal information fusion in improving robot intelligent decision-making. The results show that using the PaLM 2 model and multimodal Transformer architecture to generate humanoid robot behavior using DQN greatly improves generation efficiency and accuracy, integrates information from different modalities, has certain adaptability to the environment, and promotes the widespread application of humanoid robots in today\u2019s society. <\/jats:p>","DOI":"10.1142\/s021984362540002x","type":"journal-article","created":{"date-parts":[[2025,6,12]],"date-time":"2025-06-12T09:59:27Z","timestamp":1749722367000},"source":"Crossref","is-referenced-by-count":1,"title":["Research on Multimodal Information Processing and Behavior Generation for Humanoid Robots Based on PaLM 2 and Multimodal Transformer Architecture"],"prefix":"10.1142","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2790-0610","authenticated-orcid":false,"given":"Jie","family":"Fang","sequence":"first","affiliation":[{"name":"School of Computer and Information Technology, Anhui Vocational And Technical College, Hefei 231200, Anhui, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-2081-339X","authenticated-orcid":false,"given":"Wenliang","family":"Ju","sequence":"additional","affiliation":[{"name":"Institute of Statistics and Applied Mathematics, Anhui University of Finance and Economics, Bengbu 233000, Anhui, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0681-8404","authenticated-orcid":false,"given":"Feng","family":"Xiao","sequence":"additional","affiliation":[{"name":"School of Computer and Information Technology, Anhui Vocational And Technical College, Hefei 231200, Anhui, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4366-0458","authenticated-orcid":false,"given":"Rubing","family":"Huang","sequence":"additional","affiliation":[{"name":"School of Computer and Information Technology, Anhui Vocational And Technical College, Hefei 231200, Anhui, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3563-0558","authenticated-orcid":false,"given":"Yanjun","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Computer and Information Technology, Anhui Vocational And Technical College, Hefei 231200, Anhui, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"219","published-online":{"date-parts":[[2025,7,7]]},"reference":[{"key":"S021984362540002XBIB001","doi-asserted-by":"publisher","DOI":"10.1021\/acsnano.3c11281"},{"key":"S021984362540002XBIB002","doi-asserted-by":"publisher","DOI":"10.1007\/s43154-023-00107-x"},{"key":"S021984362540002XBIB003","doi-asserted-by":"publisher","DOI":"10.3390\/biomimetics9040248"},{"key":"S021984362540002XBIB004","doi-asserted-by":"publisher","DOI":"10.1017\/S0263574723000097"},{"key":"S021984362540002XBIB005","doi-asserted-by":"publisher","DOI":"10.3390\/electronics12010022"},{"key":"S021984362540002XBIB006","doi-asserted-by":"publisher","DOI":"10.1007\/s11042-022-14253-5"},{"key":"S021984362540002XBIB007","doi-asserted-by":"publisher","DOI":"10.3390\/app13053359"},{"key":"S021984362540002XBIB008","doi-asserted-by":"publisher","DOI":"10.1007\/s00500-023-07824-7"},{"key":"S021984362540002XBIB009","doi-asserted-by":"publisher","DOI":"10.1108\/IR-01-2022-0009"},{"key":"S021984362540002XBIB010","doi-asserted-by":"publisher","DOI":"10.1038\/s41551-023-01045-x"},{"key":"S021984362540002XBIB011","doi-asserted-by":"publisher","DOI":"10.1007\/s00521-022-07790-5"},{"issue":"1","key":"S021984362540002XBIB012","first-page":"28","volume":"2","author":"Qian J.","year":"2024","journal-title":"Int. J. Comput. Sci. Inf. Technol."},{"key":"S021984362540002XBIB013","doi-asserted-by":"publisher","DOI":"10.1002\/hcs2.61"},{"key":"S021984362540002XBIB014","doi-asserted-by":"publisher","DOI":"10.1002\/aisy.202300359"},{"issue":"4","key":"S021984362540002XBIB015","first-page":"1","volume":"11","author":"Gong T.","year":"2024","journal-title":"Front. Bioeng. Biotechnol."},{"key":"S021984362540002XBIB016","doi-asserted-by":"publisher","DOI":"10.3390\/robotics13080121"},{"issue":"4","key":"S021984362540002XBIB017","first-page":"1593","volume":"6","author":"Kumar R.","year":"2024","journal-title":"Proc. Eng."},{"key":"S021984362540002XBIB018","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2023.3275156"},{"key":"S021984362540002XBIB019","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2022.09.136"},{"issue":"1","key":"S021984362540002XBIB020","first-page":"24206","volume":"34","author":"Akbari H.","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"S021984362540002XBIB021","doi-asserted-by":"publisher","DOI":"10.1080\/0952813X.2020.1744198"},{"key":"S021984362540002XBIB022","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2021.03.091"},{"issue":"4","key":"S021984362540002XBIB023","first-page":"4303","volume":"43","author":"Xue C.","year":"2022","journal-title":"J. Intell. Fuzzy Syst."},{"issue":"240","key":"S021984362540002XBIB024","first-page":"1","volume":"24","author":"Chowdhery A.","year":"2023","journal-title":"J. Mach. Learn. Res."},{"issue":"70","key":"S021984362540002XBIB025","first-page":"1","volume":"25","author":"Chung H. W.","year":"2024","journal-title":"J. Mach. Learn. Res."},{"key":"S021984362540002XBIB026","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2023.01.046"},{"key":"S021984362540002XBIB027","doi-asserted-by":"publisher","DOI":"10.3390\/electronics12194176"},{"issue":"2","key":"S021984362540002XBIB028","first-page":"239","volume":"22","author":"Zhang J.","year":"2021","journal-title":"J. Internet Technol."},{"key":"S021984362540002XBIB029","doi-asserted-by":"publisher","DOI":"10.3390\/s23073625"},{"key":"S021984362540002XBIB030","doi-asserted-by":"publisher","DOI":"10.1109\/TCYB.2021.3053414"},{"key":"S021984362540002XBIB031","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2020.12.023"},{"key":"S021984362540002XBIB032","doi-asserted-by":"publisher","DOI":"10.3390\/math11030682"},{"key":"S021984362540002XBIB033","doi-asserted-by":"publisher","DOI":"10.1007\/s11760-023-02771-0"},{"key":"S021984362540002XBIB034","doi-asserted-by":"publisher","DOI":"10.1007\/s10772-023-10022-3"}],"container-title":["International Journal of Humanoid Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.worldscientific.com\/doi\/pdf\/10.1142\/S021984362540002X","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,3]],"date-time":"2025-09-03T09:37:11Z","timestamp":1756892231000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.worldscientific.com\/doi\/10.1142\/S021984362540002X"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,7]]},"references-count":34,"journal-issue":{"issue":"04","published-print":{"date-parts":[[2025,8]]}},"alternative-id":["10.1142\/S021984362540002X"],"URL":"https:\/\/doi.org\/10.1142\/s021984362540002x","relation":{},"ISSN":["0219-8436","1793-6942"],"issn-type":[{"value":"0219-8436","type":"print"},{"value":"1793-6942","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,7]]},"article-number":"2540002"}}