{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T15:27:15Z","timestamp":1784215635003,"version":"3.55.0"},"reference-count":42,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,9,29]],"date-time":"2023-09-29T00:00:00Z","timestamp":1695945600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:sec><jats:title>Introduction<\/jats:title><jats:p>In the context of evolving societal preferences for deeper emotional connections in art, this paper explores the emergence of multimodal robot music performance art. It investigates the fusion of music and motion in robot performances to enhance expressiveness and emotional impact. The study employs Transformer models to combine audio and video signals, enabling robots to better understand music's rhythm, melody, and emotional content. Generative Adversarial Networks (GANs) are utilized to create lifelike visual performances synchronized with music, bridging auditory and visual perception. Multimodal reinforcement learning is employed to achieve harmonious alignment between sound and motion.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>The study leverages Transformer models to process audio and video signals in robot performances. Generative Adversarial Networks are employed to generate visually appealing performances that align with the musical input. Multimodal reinforcement learning is used to synchronize robot actions with music. Diverse music styles and emotions are considered in the experiments. Performance evaluation metrics include accuracy, recall rate, and F1 score.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The proposed approach yields promising results across various music styles and emotional contexts. Performance smoothness scores exceed 94 points, demonstrating the fluidity of robot actions. An accuracy rate of 95% highlights the precision of the system in aligning robot actions with music. Notably, there is a substantial 33% enhancement in performance recall rate compared to baseline modules. The collective improvement in F1 score emphasizes the advantages of the proposed approach in the realm of robot music performance art.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>The study's findings demonstrate the potential of multimodal robot music performance art in achieving heightened emotional impact. By combining audio and visual cues, robots can better interpret and respond to music, resulting in smoother and more precise performances. The substantial improvement in recall rate suggests that the proposed approach enhances the robots' ability to accurately mirror the emotional nuances of the music. These results signify the potential of this approach to transform the landscape of artistic expression through robotics, opening new avenues for emotionally resonant performances.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fnbot.2023.1281944","type":"journal-article","created":{"date-parts":[[2023,9,29]],"date-time":"2023-09-29T13:59:54Z","timestamp":1695995994000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Multi-dimensional fusion: transformer and GANs-based multimodal audiovisual perception robot for musical performance art"],"prefix":"10.3389","volume":"17","author":[{"given":"Shiyi","family":"Lu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Panpan","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,9,29]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"100004","DOI":"10.1016\/j.jjimei.2020.100004","article-title":"Generative adversarial network: An overview of theory and applications","volume":"1","author":"Aggarwal","year":"2021","journal-title":"Int. J. Inform. Manage. Data Insights"},{"key":"B2","doi-asserted-by":"publisher","first-page":"3501","DOI":"10.1109\/LRA.2020.2977333","article-title":"Generative autoregressive networks for 3d dancing move synthesis from music","volume":"5","author":"Ahn","year":"2020","journal-title":"IEEE Robot. Autom. Lett"},{"key":"B3","doi-asserted-by":"crossref","DOI":"10.1201\/9781315140223","volume-title":"Constrained Markov Decision Processes","author":"Altman","year":"2021"},{"key":"B4","doi-asserted-by":"publisher","first-page":"315","DOI":"10.1080\/14613808.2020.1767558","article-title":"Not just composing, but programming music in group robotics","volume":"22","author":"Baek","year":"2020","journal-title":"Music Educ. Res"},{"key":"B5","doi-asserted-by":"publisher","first-page":"505","DOI":"10.1016\/j.dcan.2020.12.001","article-title":"Robot and its living space: A roadmap for robot development based on the view of living space","volume":"7","author":"Cai","year":"2021","journal-title":"Dig. Commun. Netw"},{"key":"B6","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1109\/ICRAE50850.2020.9310916","article-title":"\u201cRobot human synchronization for musical ensemble: progress and challenges,\u201d","author":"Chakraborty","year":"2020","journal-title":"2020 5th International Conference on Robotics and Automation Engineering (ICRAE)"},{"key":"B7","doi-asserted-by":"publisher","first-page":"799","DOI":"10.1007\/978-3-030-72116-9_28","article-title":"\u201cHuman-robot musical interaction,\u201d","author":"Cosentino","year":"2021","journal-title":"Handbook of Artificial Intelligence for Music: Foundations, Advanced Approaches, and Developments for Creativity"},{"key":"B8","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1243\/0954411001535309","article-title":"A review of robotics in surgery","volume":"214","author":"Davies","year":"2000","journal-title":"Proc. Instit. Mech. Eng. Part H"},{"key":"B9","article-title":"Fma: A dataset for music analysis","author":"Defferrard","year":"2016","journal-title":"arXiv preprint arXiv:1612.01840"},{"key":"B10","doi-asserted-by":"publisher","first-page":"1031","DOI":"10.1007\/s10846-020-01219-8","article-title":"Behavior models of emotion-featured robots: A survey","volume":"100","author":"Dimitrievska","year":"2020","journal-title":"J. Intell. Robot. Syst"},{"key":"B11","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1007\/978-3-030-01219-9_3","article-title":"\u201cLearning to separate object sounds by watching unlabeled video,\u201d","author":"Gao","year":"2018","journal-title":"Proceedings of the European Conference on Computer Vision (ECCV)"},{"key":"B12","first-page":"146","article-title":"\u201cAiatrack: Attention in attention for transformer visual tracking,\u201d","volume-title":"European Conference on Computer Vision","author":"Gao","year":"2022"},{"key":"B13","doi-asserted-by":"publisher","first-page":"552","DOI":"10.1109\/ACII.2019.8925444","article-title":"\u201cMultimodal and temporal perception of audio-visual cues for emotion recognition,\u201d","author":"Ghaleb","year":"2019","journal-title":"2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII)"},{"key":"B14","article-title":"Enabling factorized piano music modeling and generation with the maestro dataset","author":"Hawthorne","year":"2018","journal-title":"arXiv preprint arXiv:1810.12247"},{"key":"B15","doi-asserted-by":"publisher","first-page":"26893","DOI":"10.1109\/ACCESS.2023.3257381","article-title":"Hyperbolic music transformer for structured music generation","volume":"11","author":"Huang","year":"2023","journal-title":"IEEE Access"},{"key":"B16","doi-asserted-by":"publisher","first-page":"41","DOI":"10.17977\/um018v2i12019p41-46","article-title":"Adam optimization algorithm for wide and deep neural network","volume":"2","author":"Jais","year":"2019","journal-title":"Knowl. Eng. Data Sci"},{"key":"B17","doi-asserted-by":"publisher","first-page":"2761","DOI":"10.1109\/ICIP.2019.8803238","article-title":"\u201cUnsupervised single image deraining with self-supervised constraints,\u201d","author":"Jin","year":"2019","journal-title":"2019 IEEE International Conference on Image Processing (ICIP)"},{"key":"B18","doi-asserted-by":"publisher","first-page":"522","DOI":"10.1109\/TMM.2018.2856090","article-title":"Creating a multitrack classical music performance dataset for multimodal music analysis: Challenges, insights, and applications","volume":"21","author":"Li","year":"2018","journal-title":"IEEE Trans. Multim"},{"key":"B19","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1145\/3332169.3332264","article-title":"\u201cDesigning a musical robot for chinese bamboo flute performance,\u201d","author":"Li","year":"2019","journal-title":"Proceedings of the Seventh International Symposium of Chinese CHI"},{"key":"B20","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1007\/s12369-019-00535-w","article-title":"Multimodal information fusion for automatic aesthetics evaluation of robotic dance poses","volume":"12","author":"Li","year":"2020","journal-title":"Int. J. Soc. Robot"},{"key":"B21","doi-asserted-by":"publisher","first-page":"334","DOI":"10.1145\/3171221.3171261","article-title":"\u201cMultimodal expression of artificial emotion in social robots using color, motion and sound,\u201d","author":"L\u00f6ffler","year":"2018","journal-title":"Proceedings of the 2018 ACM\/IEEE International Conference on Human-Robot Interaction"},{"key":"B22","article-title":"\u201cFixing weight decay regularization in adam,\u201d","author":"Loshchilov","year":"2018","journal-title":"ICLR 2018 Conference Acceptance Decision"},{"key":"B23","first-page":"644","article-title":"\u201cRTN: Reinforced transformer network for coronary ct angiography vessel-level image quality assessment,\u201d","volume-title":"International Conference on Medical Image Computing and Computer-Assisted Intervention","author":"Lu","year":"2022"},{"key":"B24","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1287\/educ.2018.0191","article-title":"\u201cStochastic gradient descent: Recent trends,\u201d9D","author":"Newton","year":"2018","journal-title":"Recent Advances in Optimization and Modeling of Contemporary Problems"},{"key":"B25","doi-asserted-by":"publisher","first-page":"666702","DOI":"10.3389\/fpsyg.2021.666702","article-title":"Flourishing in resonance: Joint resilience building through music and motion","volume":"12","author":"Nijs","year":"2021","journal-title":"Front. Psychol"},{"key":"B26","doi-asserted-by":"publisher","first-page":"109216","DOI":"10.1016\/j.patcog.2022.109216","article-title":"Hyper-sausage coverage function neuron model and learning algorithm for image classification","volume":"136","author":"Ning","year":"2023","journal-title":"Patt. Recogn"},{"key":"B27","doi-asserted-by":"publisher","first-page":"1850023","DOI":"10.1142\/S0219843618500238","article-title":"A music-driven dance system of humanoid robots","volume":"15","author":"Qin","year":"2018","journal-title":"Int. J. Human. Robot"},{"key":"B28","doi-asserted-by":"publisher","first-page":"126284","DOI":"10.1016\/j.neucom.2023.126284","article-title":"3d human pose and shape estimation via de-occlusion multi-task learning","volume":"548","author":"Ran","year":"2023","journal-title":"Neurocomputing"},{"key":"B29","doi-asserted-by":"publisher","first-page":"310","DOI":"10.1162\/artl_a_00376","article-title":"The enactive and interactive dimensions of ai: Ingenuity and imagination through the lens of art and music","volume":"28","author":"Sato","year":"2022","journal-title":"Artif. Life"},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.1017\/S0140525X20000333","article-title":"Music as a coevolved system for social bonding","author":"Savage","year":"2021","journal-title":"Behav. Brain Sci"},{"key":"B31","doi-asserted-by":"publisher","first-page":"823","DOI":"10.1007\/978-3-030-72116-9_29","article-title":"\u201cShimon sings-robotic musicianship finds its voice,\u201d","author":"Savery","year":"2021","journal-title":"Handbook of Artificial Intelligence for Music: Foundations, Advanced Approaches, and Developments for Creativity"},{"key":"B32","doi-asserted-by":"publisher","first-page":"e0237826","DOI":"10.1371\/journal.pone.0237826","article-title":"Gaussian process inference modelling of dynamic robot control for expressive piano playing","volume":"15","author":"Scimeca","year":"2020","journal-title":"PLoS ONE"},{"key":"B33","doi-asserted-by":"publisher","first-page":"819","DOI":"10.1007\/s10639-020-10392-0","article-title":"Utilizing social virtual reality robot (v2r) for music education to children with high-functioning autism","volume":"27","author":"Shahab","year":"2022","journal-title":"Educ. Inf. Technol"},{"key":"B34","first-page":"4596","article-title":"\u201cAdafactor: Adaptive learning rates with sublinear memory cost,\u201d","author":"Shazeer","year":"2018","journal-title":"International Conference on Machine Learning"},{"key":"B35","doi-asserted-by":"publisher","first-page":"482","DOI":"10.1109\/RO-MAN47096.2020.9223503","article-title":"\u201cBright and dark timbre expressions with sound pressure and tempo variations by violin-playing robot,\u201d","author":"Shibuya","year":"2020","journal-title":"2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)"},{"key":"B36","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton","year":"2018"},{"key":"B37","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1109\/RO-MAN47096.2020.9223449","article-title":"\u201cHow context shapes the appropriateness of a robots voice,\u201d","author":"Torre","year":"2020","journal-title":"2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)"},{"key":"B38","doi-asserted-by":"publisher","first-page":"108498","DOI":"10.1016\/j.patcog.2021.108498","article-title":"Uncertainty estimation for stereo matching based on evidential deep learning","volume":"124","author":"Wang","year":"2022","journal-title":"Patt. Recogn"},{"key":"B39","doi-asserted-by":"publisher","first-page":"137","DOI":"10.1109\/ACCESS.2023.3287271","article-title":"Motif transformer: Generating music with motifs","volume":"11","author":"Wang","year":"2023","journal-title":"IEEE Access"},{"key":"B40","doi-asserted-by":"publisher","first-page":"468","DOI":"10.1109\/THMS.2021.3131683","article-title":"A survey of hybrid human-artificial intelligence for social computing","volume":"52","author":"Wang","year":"2021","journal-title":"IEEE Trans. Hum Mach. Syst"},{"key":"B41","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-38930-7","volume-title":"Robotic Musicianship: Embodied Artificial Creativity and Mechatronic Musical Expression","author":"Weinberg","year":"2020"},{"key":"B42","doi-asserted-by":"publisher","first-page":"6688","DOI":"10.1109\/ICCV.2019.00679","article-title":"\u201cAn empirical study of spatial attention mechanisms in deep networks","author":"Zhu","year":"2019","journal-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1281944\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,29]],"date-time":"2023-09-29T14:00:01Z","timestamp":1695996001000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1281944\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,29]]},"references-count":42,"alternative-id":["10.3389\/fnbot.2023.1281944"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2023.1281944","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,29]]},"article-number":"1281944"}}