{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,27]],"date-time":"2026-07-27T17:39:42Z","timestamp":1785173982257,"version":"3.55.0"},"reference-count":40,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,1,30]],"date-time":"2024-01-30T00:00:00Z","timestamp":1706572800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>The field of multimodal robotic musical performing arts has garnered significant interest due to its innovative potential. Conventional robots face limitations in understanding emotions and artistic expression in musical performances. Therefore, this paper explores the application of multimodal robots that integrate visual and auditory perception to enhance the quality and artistic expression in music performance. Our approach involves integrating GRU (Gated Recurrent Unit) and GoogLeNet models for sentiment analysis. The GRU model processes audio data and captures the temporal dynamics of musical elements, including long-term dependencies, to extract emotional information. The GoogLeNet model excels in image processing, extracting complex visual details and aesthetic features. This synergy deepens the understanding of musical and visual elements, aiming to produce more emotionally resonant and interactive robot performances. Experimental results demonstrate the effectiveness of our approach, showing significant improvements in music performance by multimodal robots. These robots, equipped with our method, deliver high-quality, artistic performances that effectively evoke emotional engagement from the audience. Multimodal robots that merge audio-visual perception in music performance enrich the art form and offer diverse human-machine interactions. This research demonstrates the potential of multimodal robots in music performance, promoting the integration of technology and art. It opens new realms in performing arts and human-robot interactions, offering a unique and innovative experience. Our findings provide valuable insights for the development of multimodal robots in the performing arts sector.<\/jats:p>","DOI":"10.3389\/fnbot.2023.1324831","type":"journal-article","created":{"date-parts":[[2024,1,30]],"date-time":"2024-01-30T04:15:31Z","timestamp":1706588131000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Multimodal robotic music performance art based on GRU-GoogLeNet model fusing audiovisual perception"],"prefix":"10.3389","volume":"17","author":[{"given":"Lu","family":"Wang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2024,1,30]]},"reference":[{"key":"B1","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1007\/978-3-030-22113-3_16","article-title":"\u201cExplaining interaction effects within and across levels of analysis,\u201d","volume-title":"Research Methods in International Business","author":"Andersson","year":"2020"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2301.07315","article-title":"Face recognition in the age of clip & billion image datasets","author":"Bhat","year":"2023","journal-title":"arXiv preprint arXiv:2301.07315"},{"key":"B3","doi-asserted-by":"crossref","first-page":"10452","DOI":"10.1109\/IROS45743.2020.9341165","article-title":"\u201cMultimodal material classification for robots using spectroscopy and high resolution texture imaging,\u201d","volume-title":"2020 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)","author":"Erickson","year":"2020"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.1176749","article-title":"Sonification of facial actions for musical expression","author":"Funk","year":"2020","journal-title":"arXiv preprint arXiv:2010.03223"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2023.111028","article-title":"A smoothing group lasso based interval type-2 fuzzy neural network for simultaneous feature selection and system identification","author":"Gao","year":"2023","journal-title":"Knowledge Based Syst."},{"key":"B6","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1109\/ICIP40778.2020.9191019","article-title":"\u201cMultimodal attention-mechanism for temporal emotion recognition,\u201d","volume-title":"2020 IEEE International Conference on Image Processing (ICIP)","author":"Ghaleb","year":"2020"},{"key":"B7","first-page":"309","article-title":"\u201cEvaluating rhythmic descriptors for musical genre classification,\u201d","volume-title":"ISMIR","author":"Gouyon","year":"2006"},{"key":"B8","doi-asserted-by":"publisher","first-page":"605","DOI":"10.1007\/s11760-021-02050-w","article-title":"Efficient masked face recognition method during the covid-19 pandemic","volume":"16","author":"Hariri","year":"2022","journal-title":"Signal Image Video Process."},{"key":"B9","doi-asserted-by":"publisher","first-page":"549360","DOI":"10.3389\/frobt.2021.549360","article-title":"SIGverse: a cloud-based VR platform for research on multimodal human-robot interaction","volume":"8","author":"Inamura","year":"2021","journal-title":"Front. Robot. AI"},{"key":"B10","doi-asserted-by":"publisher","first-page":"239","DOI":"10.1002\/hbm.20180","article-title":"Investigating emotion with music: an fMRI study","volume":"27","author":"Koelsch","year":"2006","journal-title":"Hum. Brain Mapp."},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1186\/s40537-021-00448-4","article-title":"Intrusion detection systems using long short-term memory (LSTM)","author":"Laghrissi","year":"2021","journal-title":"J. Big Data"},{"key":"B12","first-page":"5549","article-title":"\u201cMaskGAN: towards diverse and interactive facial image manipulation,\u201d","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Lee","year":"2020"},{"key":"B13","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1007\/s10055-018-0343-6","article-title":"Real-time manufacturing modeling and simulation framework using augmented reality and stochastic network analysis","volume":"23","author":"Lee","year":"2019","journal-title":"Virt. Real."},{"key":"B14","volume-title":"Learning to Rank for Information Retrieval and Natural Language Processing","author":"Li","year":"2022"},{"key":"B15","doi-asserted-by":"publisher","first-page":"139110","DOI":"10.1109\/ACCESS.2020.3011028","article-title":"A review of face recognition technology","volume":"8","author":"Li","year":"2020","journal-title":"IEEE Access"},{"key":"B16","article-title":"\u201cDetecting emotion in music,\u201d","volume-title":"4th International Conference on Music Information Retrieval","author":"Li","year":"2003"},{"key":"B17","first-page":"127","article-title":"Hierarchical matching algorithm of visual image for mobile robots based on deep reinforcement learning","volume":"61","author":"Li","year":"2023","journal-title":"J. Jilin Univ. Sci. Ed."},{"key":"B18","first-page":"943","article-title":"A variable step size lms speech denoising algorithm based on wavelet threshold","volume":"60","author":"Liu","year":"2022","journal-title":"J. Jilin Univ. Sci. Ed."},{"key":"B19","first-page":"6","article-title":"\u201cAudio matching via chroma-based statistical features,\u201d","volume-title":"ISMIR","author":"M\u00fcller","year":"2005"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2022.109216","article-title":"Hyper-sausage coverage function neuron model and learning algorithm for image classification","author":"Ning","year":"2023","journal-title":"Pattern Recogn"},{"key":"B21","doi-asserted-by":"publisher","first-page":"4927","DOI":"10.3390\/s21144927","article-title":"Deep-learning-based multimodal emotion classification for music videos","volume":"21","author":"Pandeya","year":"2021","journal-title":"Sensors"},{"key":"B22","first-page":"9413","article-title":"\u201cMulti-scale interactive network for salient object detection,\u201d","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Pang","year":"2020"},{"key":"B23","article-title":"Emotion recognition from speech using wav2vec 2.0 embeddings","author":"Pepino","year":"2021","journal-title":"arXiv preprint arXiv:2104.03502"},{"key":"B24","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1109\/ICEST49890.2020.9232713","article-title":"\u201cAugmented and virtual reality web applications for music stage performance,\u201d","volume-title":"2020 55th International Scientific Conference on Information, Communication and Energy Systems and Technologies (ICEST)","author":"Petrovi\u0107","year":"2020"},{"key":"B25","doi-asserted-by":"publisher","first-page":"1872","DOI":"10.1007\/s11431-020-1647-3","article-title":"Pre-trained models for natural language processing: a survey","volume":"63","author":"Qiu","year":"2020","journal-title":"Sci. China Technol. Sci."},{"key":"B26","first-page":"83","article-title":"Mir_kovic: a multimodal database infrastructure for research on audio-visual affective expression","volume":"1","author":"Schuller","year":"2010","journal-title":"IEEE Trans. Affect. Comput."},{"key":"B27","doi-asserted-by":"crossref","first-page":"2347","DOI":"10.1109\/ICACCS57279.2023.10113124","article-title":"\u201cStocastic multimodal fusion method for classifying emotions with attention mechanism using deep learning,\u201d","volume-title":"2023 9th International Conference on Advanced Computing and Communication Systems (ICACCS)","author":"Selvi","year":"2023"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1909.09586","article-title":"Understanding LSTM\u2013a tutorial into long short-term memory recurrent neural networks","author":"Staudemeyer","year":"2019","journal-title":"arXiv preprint arXiv:1909.09586"},{"key":"B29","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1121\/1.1915893","article-title":"A scale for the measurement of the psychological magnitude pitch","volume":"8","author":"Stevens","year":"1937","journal-title":"J. Acoust. Soc. Am."},{"key":"B30","doi-asserted-by":"publisher","first-page":"497","DOI":"10.1109\/TMM.2020.2981989","article-title":"Deepdance: music-to-dance motion choreography with adversarial learning","volume":"23","author":"Sun","year":"2020","journal-title":"IEEE Trans. Multimedia"},{"key":"B31","doi-asserted-by":"publisher","first-page":"2682","DOI":"10.3389\/fpsyg.2018.02682","article-title":"How live music moves us: head movement differences in audiences to live versus recorded music","volume":"9","author":"Swarbrick","year":"2019","journal-title":"Front. Psychol."},{"key":"B32","doi-asserted-by":"publisher","first-page":"e0240862","DOI":"10.1371\/journal.pone.0240862","article-title":"Effects of music therapy on depression: a meta-analysis of randomized controlled trials","volume":"15","author":"Tang","year":"2020","journal-title":"PLoS ONE"},{"key":"B33","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1007\/s12369-019-00524-z","article-title":"Multimodal integration of emotional signals from voice, body, and context: effects of (in) congruence on emotion recognition and attitudes towards robots","volume":"11","author":"Tsiourti","year":"2019","journal-title":"Int. J. Soc. Robot."},{"key":"B34","doi-asserted-by":"crossref","first-page":"4939","DOI":"10.1109\/IGARSS.2019.8898220","article-title":"\u201cImage registration of satellite imagery with deep convolutional neural networks,\u201d","volume-title":"IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium","author":"Vakalopoulou","year":"2019"},{"key":"B35","doi-asserted-by":"publisher","first-page":"2339","DOI":"10.1109\/TASE.2021.3074873","article-title":"Predicting human intentions in human\u2013robot hand-over tasks through multimodal learning","volume":"19","author":"Wang","year":"2021","journal-title":"IEEE Trans. Automat. Sci. Eng."},{"key":"B36","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1155\/2022\/7192306","article-title":"Research on face recognition classification based on improved googlenet","volume":"2022","author":"Yu","year":"2022","journal-title":"Sec. Commun. Netw."},{"key":"B37","first-page":"361","article-title":"\u201cMultimodal transformer for unaligned multimodal language sequences,\u201d","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Zadeh","year":"2018"},{"key":"B38","doi-asserted-by":"publisher","first-page":"95","DOI":"10.4236\/jbbs.2020.101005","article-title":"The positive influence of music on the human brain","volume":"10","author":"Zhang","year":"2020","journal-title":"Behav. Brain Sci."},{"key":"B39","doi-asserted-by":"publisher","first-page":"78063","DOI":"10.1109\/ACCESS.2019.2923006","article-title":"Short-term photovoltaic power forecasting based on long short term memory neural network and attention mechanism","volume":"7","author":"Zhou","year":"2019","journal-title":"IEEE Access"},{"key":"B40","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1909.11764","article-title":"FreeLB: enhanced adversarial training for natural language understanding","author":"Zhu","year":"2019","journal-title":"arXiv preprint arXiv:1909.11764"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1324831\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,30]],"date-time":"2024-01-30T04:15:43Z","timestamp":1706588143000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1324831\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,30]]},"references-count":40,"alternative-id":["10.3389\/fnbot.2023.1324831"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2023.1324831","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,30]]},"article-number":"1324831"}}