{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,6]],"date-time":"2025-08-06T13:40:43Z","timestamp":1754487643892,"version":"3.41.2"},"reference-count":36,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,8,19]],"date-time":"2024-08-19T00:00:00Z","timestamp":1724025600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>Utilizing deep features from electroencephalography (EEG) data for emotional music composition provides a novel approach for creating personalized and emotionally rich music. Compared to textual data, converting continuous EEG and music data into discrete units presents significant challenges, particularly the lack of a clear and fixed vocabulary for standardizing EEG and audio data. The lack of this standard makes the mapping relationship between EEG signals and musical elements (such as rhythm, melody, and emotion) blurry and complex. Therefore, we propose a method of using clustering to create discrete representations and using the Transformer model to reverse mapping relationships. Specifically, the model uses clustering labels to segment signals and independently encodes EEG and emotional music data to construct a vocabulary, thereby achieving discrete representation. A time series dictionary was developed using clustering algorithms, which more effectively captures and utilizes the temporal and structural relationships between EEG and audio data. In response to the insensitivity to temporal information in heterogeneous data, we adopted a multi head attention mechanism and positional encoding technology to enable the model to focus on information in different subspaces, thereby enhancing the understanding of the complex internal structure of EEG and audio data. In addition, to address the mismatch between local and global information in emotion driven music generation, we introduce an audio masking prediction loss learning method. Our method generates music that<jats:italic>Hits@<\/jats:italic>20 On the indicator, a performance of 68.19% was achieved, which improved the score by 4.9% compared to other methods, indicating the effectiveness of this method.<\/jats:p>","DOI":"10.3389\/fnbot.2024.1437737","type":"journal-article","created":{"date-parts":[[2024,8,19]],"date-time":"2024-08-19T05:10:28Z","timestamp":1724044228000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["EEG-driven automatic generation of emotive music based on transformer"],"prefix":"10.3389","volume":"18","author":[{"given":"Hui","family":"Jiang","sequence":"first","affiliation":[]},{"given":"Yu","family":"Chen","sequence":"additional","affiliation":[]},{"given":"Di","family":"Wu","sequence":"additional","affiliation":[]},{"given":"Jinlin","family":"Yan","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2024,8,19]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"4140","DOI":"10.1007\/s10489-024-05335-y","article-title":"Symfornet: application of cross-modal information correspondences based on self-supervision in symbolic music generation","volume":"54","author":"Abudukelimu","year":"2024","journal-title":"Appl. Intell"},{"key":"B2","doi-asserted-by":"publisher","first-page":"109446","DOI":"10.1109\/ACCESS.2022.3213996","article-title":"Decoding the user's movements preparation from EEG signals using vision transformer architecture","volume":"10","author":"Al-Quraishi","year":"2022","journal-title":"IEEE Access"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.1121\/1.4988940","article-title":"Musicological indices for soundscape ecological analysis","author":"Bellisario","year":"2017","journal-title":"J. Acoust. Soc. Am"},{"key":"B4","unstructured":"\u201cData augmentation strategies for music composition using generative adversarial networks,\u201d Bitaraes M. Guimaraes F. Coelho F. Congresso Brasileiro de Autom\u00e1tica-CBA, Volume 2022"},{"key":"B5","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1167\/jov.23.13.6","article-title":"Emotions of subject and object affect beauty differently for images and music","volume":"23","author":"Bruns","year":"2023","journal-title":"J. Vis"},{"key":"B6","doi-asserted-by":"publisher","first-page":"164","DOI":"10.1016\/j.ins.2023.02.088","article-title":"A review on semi-supervised clustering","volume":"632","author":"Cai","year":"2023","journal-title":"Inf. Sci"},{"key":"B7","doi-asserted-by":"publisher","first-page":"465","DOI":"10.1016\/j.procs.2020.04.049","article-title":"An improved rnn-lstm based novel approach for sheet music generation","volume":"171","author":"Dua","year":"2020","journal-title":"Procedia Comput. Sci"},{"key":"B8","unstructured":"Forsgren S. Martiros H. Riffusion-stable diffusion for real-time music generation 2022"},{"key":"B9","doi-asserted-by":"publisher","first-page":"122520","DOI":"10.1016\/j.techfore.2023.122520","article-title":"Technological forecasting based on estimation of word embedding matrix using lstm networks","volume":"191","author":"Gozuacik","year":"2023","journal-title":"Technol. Forecast. Soc. Change"},{"key":"B10","doi-asserted-by":"publisher","first-page":"531","DOI":"10.1007\/s10844-021-00658-5","article-title":"Music emotion recognition using recurrent neural networks and pretrained models","volume":"57","author":"Grekow","year":"2021","journal-title":"J. Intell. Inf. Syst"},{"key":"B11","doi-asserted-by":"publisher","first-page":"760","DOI":"10.1016\/j.jestch.2020.10.009","article-title":"Music emotion recognition using convolutional long short term memory deep neural networks","volume":"24","author":"Hizlisoy","year":"2021","journal-title":"Eng. Sci. Technol. Int. J"},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2108.01374","article-title":"Emopia: a multi-modal pop piano dataset for emotion recognition and emotion-based music generation","author":"Hung","year":"2021","journal-title":"arXiv"},{"key":"B13","doi-asserted-by":"publisher","first-page":"69","DOI":"10.48293\/IJMSTA-104","article-title":"Music generation by direct sonification and musification using EEG data","volume":"5","author":"Inoue","year":"2024","journal-title":"Int. J. Music Sci. Technol. Art"},{"key":"B14","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1109\/T-AFFC.2011.15","article-title":"Deap: a database for emotion analysis; using physiological signals","volume":"3","author":"Koelstra","year":"2011","journal-title":"IEEE Trans. Affect. Comput"},{"key":"B15","doi-asserted-by":"publisher","first-page":"107542","DOI":"10.1016\/j.patcog.2020.107542","article-title":"Bringing semantics into word image representation","volume":"108","author":"Krishnan","year":"2020","journal-title":"Pattern Recognit"},{"key":"B16","doi-asserted-by":"publisher","first-page":"1912","DOI":"10.1109\/TAFFC.2022.3167013","article-title":"Self supervised adversarial domain adaptation for cross-corpus and cross-language speech emotion recognition","volume":"14","author":"Latif","year":"2023","journal-title":"IEEE Trans. Affect. Comput"},{"key":"B17","doi-asserted-by":"publisher","first-page":"539","DOI":"10.1007\/s11390-022-2030-z","article-title":"Self-supervised music motion synchronization learning for music-driven conducting motion generation","volume":"37","author":"Liu","year":"2022","journal-title":"J. Comput. Sci. Technol"},{"key":"B18","doi-asserted-by":"publisher","first-page":"6560","DOI":"10.1007\/s11227-022-04914-5","article-title":"Literature survey of multi-track music generation model based on generative confrontation network in intelligent composition","volume":"79","author":"Liu","year":"","journal-title":"J. Supercomput"},{"key":"B19","doi-asserted-by":"crossref","unstructured":"\u201cApplication of dynamic segmentation and weighted comprehensive matching in music style recognition,\u201d 186 190 Liu Y. 10.1109\/ICEDCS60513.2023.00040. IEEE 2023 International Conference on Electronics and Devices, Computational Science (ICEDCS)","DOI":"10.1109\/ICEDCS60513.2023.00040"},{"key":"B20","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1049\/ccs2.12037","article-title":"Learning to generate emotional music correlated with music structure features","volume":"4","author":"Ma","year":"2022","journal-title":"Cogn. Comput. Syst"},{"key":"B21","doi-asserted-by":"publisher","first-page":"e0276680","DOI":"10.1371\/journal.pone.0276680","article-title":"Unique estimation in EEG analysis by the ordering ICA","volume":"17","author":"Matsuda","year":"2022","journal-title":"PLoS ONE"},{"key":"B22","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1007\/978-981-16-9873-6_30","article-title":"\u201cLSTM-RNN-based automatic music generation algorithm,\u201d","volume-title":"Intelligent and Cloud Computing: Proceedings of ICICC 2021","author":"Minu","year":"2022"},{"key":"B23","doi-asserted-by":"publisher","first-page":"1050","DOI":"10.1587\/transinf.2021EDP7171","article-title":"Online EEG-based emotion prediction and music generation for inducing affective states","volume":"105","author":"Miyamoto","year":"2022","journal-title":"IEICE Trans. Inf. Syst"},{"key":"B24","unstructured":"Mubert 2022"},{"key":"B25","doi-asserted-by":"publisher","first-page":"27074","DOI":"10.1109\/ACCESS.2020.2971600","article-title":"Learning invariant representations from EEG via adversarial inference","volume":"8","author":"\u00d6zdenizci","year":"2020","journal-title":"IEEE Access"},{"key":"B26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/ICASSP49357.2023.10095660","article-title":"\u201cPretrained transformers for seizure detection,\u201d","author":"Panchavati","year":"2023","journal-title":"ICASSP 2023"},{"key":"B27","doi-asserted-by":"publisher","first-page":"105853","DOI":"10.1016\/j.bspc.2023.105853","article-title":"Novel algorithm for detection of cognitive dysfunction using neural networks","volume":"90","author":"S\u00e1nchez-Reyes","year":"2024","journal-title":"Biomed. Signal Process. Control"},{"key":"B28","unstructured":"\u201cListen to your mind's (he) art: a system for affective music generation via brain-computer interface,\u201d Tiraboschi M. Avanzini F. Boccignone G. Proceedings of the 18th Sound and Music Computing Conference 2021"},{"key":"B29","doi-asserted-by":"publisher","first-page":"1174","DOI":"10.1609\/aaai.v33i01.33011174","article-title":"Performancenet: score-to-audio music generation with multi-band convolutional residual network","volume":"33","author":"Wang","year":"2019","journal-title":"Proc. AAAI Conf. Arti. Intell"},{"key":"B30","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1109\/ICSP51882.2021.9408813","article-title":"\u201cA memetic algorithm for Chinese traditional music composition,\u201d","volume-title":"2021 6th International Conference on Intelligent Computing and Signal Processing (ICSP)","author":"Zeng","year":"2021"},{"key":"B31","doi-asserted-by":"publisher","first-page":"2616220","DOI":"10.1155\/2022\/2616220","article-title":"Research on music emotional expression based on reinforcement learning and multimodal information","volume":"2022","author":"Zhang","year":"2022","journal-title":"Mob. Inf. Syst"},{"key":"B32","article-title":"\u201cSelf-supervised contrastive pre-training for time series via time-frequency consistency,\u201d","author":"Zhang","year":"2022","journal-title":"Advances in Neural Information Processing Systems, Vol. 35"},{"key":"B33","doi-asserted-by":"crossref","first-page":"2039","DOI":"10.1109\/ITNEC.2019.8729266","article-title":"\u201cAn emotional symbolic music generation system based on lstm networks,\u201d","volume-title":"2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC)","author":"Zhao","year":"2019"},{"key":"B34","doi-asserted-by":"publisher","first-page":"841926","DOI":"10.3389\/fpsyg.2022.841926","article-title":"Emotionbox: a music-element-driven emotional music generation system based on music psychology","volume":"13","author":"Zheng","year":"2022","journal-title":"Front. Psychol"},{"key":"B35","doi-asserted-by":"publisher","first-page":"2140","DOI":"10.1109\/TNSRE.2023.3268751","article-title":"Speech2EEG: leveraging pretrained speech model for EEG signal recognition","volume":"31","author":"Zhou","year":"","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2308.02531","article-title":"Choir transformer: generating polyphonic music with relative attention on transformer","author":"Zhou","year":"","journal-title":"arXiv"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2024.1437737\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,26]],"date-time":"2024-11-26T20:34:17Z","timestamp":1732653257000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2024.1437737\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,19]]},"references-count":36,"alternative-id":["10.3389\/fnbot.2024.1437737"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2024.1437737","relation":{},"ISSN":["1662-5218"],"issn-type":[{"type":"electronic","value":"1662-5218"}],"subject":[],"published":{"date-parts":[[2024,8,19]]},"article-number":"1437737"}}