{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T18:32:13Z","timestamp":1783362733933,"version":"3.54.6"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,4,22]],"date-time":"2024-04-22T00:00:00Z","timestamp":1713744000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,4,22]],"date-time":"2024-04-22T00:00:00Z","timestamp":1713744000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J Comput Intell Syst"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Music plays a vital role in human culture and society, serving as a universal form of expression. However, accurately classifying music emotions remains challenging due to the intricate nature of emotional expressions in music and the integration of diverse data sources. To address these challenges, we propose the Multilayered Music Decomposition and Multimodal Integration Interaction (MMD-MII) model. This model employs cross-processing to facilitate interaction between audio and lyrics, ensuring coherence in emotional representation. Additionally, we introduce a hierarchical framework based on the music theory, focusing on the main and chorus sections, with the chorus processed separately to extract precise emotional representations. Experimental results on the DEAM and FMA datasets demonstrate the effectiveness of the MMD-MII model, achieving accuracies of 49.68% and 49.54% respectively. Compared with the existing methods, our model outperforms in accuracy and <jats:italic>F<\/jats:italic>1 scores, offering promising implications for music recommendation systems, healthcare, psychology, and advertising, where accurate emotional analysis is essential.<\/jats:p>","DOI":"10.1007\/s44196-024-00489-6","type":"journal-article","created":{"date-parts":[[2024,4,22]],"date-time":"2024-04-22T08:01:58Z","timestamp":1713772918000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["MMD-MII Model: A Multilayered Analysis and Multimodal Integration Interaction Approach Revolutionizing Music Emotion Classification"],"prefix":"10.1007","volume":"17","author":[{"given":"Jingyi","family":"Wang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7110-7516","authenticated-orcid":false,"given":"Alireza","family":"Sharifi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thippa Reddy","family":"Gadekallu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Achyut","family":"Shankar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,4,22]]},"reference":[{"key":"489_CR1","doi-asserted-by":"publisher","first-page":"2887","DOI":"10.1007\/s11042-020-08836-3","volume":"80","author":"YR Pandeya","year":"2021","unstructured":"Pandeya, Y.R., Lee, J.: Deep learning-based late fusion of multimodal information for emotion classification of music video. Multimedia Tools Appl. 80, 2887\u20132905 (2021)","journal-title":"Multimedia Tools Appl."},{"key":"489_CR2","doi-asserted-by":"publisher","first-page":"27096","DOI":"10.1007\/s10489-023-04967-w","volume":"53","author":"MJ Lucia-Mulas","year":"2023","unstructured":"Lucia-Mulas, M.J., Revuelta-Sanz, P., Ruiz-Mezcua, B., Gonzalez-Carrasco, I.: Automatic music emotion classification model for movie soundtrack subtitling based on neuroscientific premises. Appl. Intell. 53, 27096\u201327109 (2023)","journal-title":"Appl. Intell."},{"key":"489_CR3","unstructured":"Hung, H., Ching, J., Doh, S., Kim, N., Nam, J., Yang, Y.: EMOPIA: a multi-modal pop piano dataset for emotion recognition and emotion-based music generation. arXiv preprint arXiv:2108.01374 (2021)"},{"key":"489_CR4","unstructured":"Chou, Y., Chen, I., Chang, C., Ching, J., Yang, Y., et al.: MidiBERT-piano: large-scale pre-training for symbolic music understanding. arXiv preprint arXiv:2107.05223 (2021)"},{"key":"489_CR5","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s40537-020-00322-9","volume":"7","author":"LJ Zheng","year":"2020","unstructured":"Zheng, L.J., Mountstephens, J., Teo, J.: Four-class emotion classification in virtual reality using pupillometry. J. Big Data 7, 1\u20139 (2020)","journal-title":"J. Big Data"},{"key":"489_CR6","doi-asserted-by":"publisher","DOI":"10.1016\/j.measurement.2019.107049","volume":"150","author":"D Jiang","year":"2020","unstructured":"Jiang, D., Wu, K., Chen, D., Tu, G., Zhou, T., Garg, A., Gao, L.: A probability and integrated learning based classification algorithm for high-level human emotion recognition problems. Measurement 150, 107049 (2020)","journal-title":"Measurement"},{"key":"489_CR7","doi-asserted-by":"publisher","first-page":"139332","DOI":"10.1109\/ACCESS.2020.3011882","volume":"8","author":"S Sheykhivand","year":"2020","unstructured":"Sheykhivand, S., Mousavi, Z., Rezaii, T.Y., Farzamnia, A.: Recognizing emotions evoked by music using CNN-LSTM networks on EEG signals. IEEE Access 8, 139332\u2013139345 (2020)","journal-title":"IEEE Access"},{"key":"489_CR8","doi-asserted-by":"publisher","first-page":"637","DOI":"10.1007\/s00779-020-01389-0","volume":"25","author":"S Cunningham","year":"2021","unstructured":"Cunningham, S., Ridley, H., Weinel, J., Picking, R.: Supervised machine learning for audio emotion recognition: Enhancing film sound design using audio features, regression models and artificial neural networks. Pers. Ubiquit. Comput. 25, 637\u2013650 (2021)","journal-title":"Pers. Ubiquit. Comput."},{"key":"489_CR9","doi-asserted-by":"publisher","first-page":"59844","DOI":"10.1109\/ACCESS.2019.2914872","volume":"7","author":"B Xing","year":"2019","unstructured":"Xing, B., Zhang, H., Zhang, K., Zhang, L., Wu, X., Shi, X., Yu, S., Zhang, S.: Exploiting EEG signals and audiovisual feature fusion for video emotion recognition. IEEE Access 7, 59844\u201359861 (2019)","journal-title":"IEEE Access"},{"key":"489_CR10","doi-asserted-by":"publisher","first-page":"93711","DOI":"10.1109\/ACCESS.2019.2927768","volume":"7","author":"Z Wang","year":"2019","unstructured":"Wang, Z., Tong, Y., Heng, X.: Phase-locking value based graph convolutional neural networks for emotion recognition. IEEE Access 7, 93711\u201393722 (2019)","journal-title":"IEEE Access"},{"key":"489_CR11","unstructured":"Wu, S., Sun, M.: Exploring the efficacy of pre-trained checkpoints in text-to-music generation task. arXiv preprint arXiv:2211.11216 (2022)"},{"key":"489_CR12","doi-asserted-by":"crossref","unstructured":"Ocampo, R., Andres, J., Schmidt, A., Pegram, C., Shave, J., Hill, C., Wright, B., Bown, O.: Using GPT-3 to achieve semantically relevant data sonification for an art installation. In: International Conference on Computational Intelligence in Music, Sound, Art and Design (Part of EvoStar), , pp. 212\u2013227. Springer (2023)","DOI":"10.1007\/978-3-031-29956-8_14"},{"key":"489_CR13","unstructured":"Chang, C., Lee, C., Yang, Y.: Variable-length music score infilling via XLNet and musically specialized positional encoding. arXiv preprint arXiv:2108.05064 (2021)"},{"key":"489_CR14","doi-asserted-by":"publisher","first-page":"135594","DOI":"10.1109\/ACCESS.2021.3113256","volume":"9","author":"A Alshanqiti","year":"2021","unstructured":"Alshanqiti, A., Namoun, A., Alsughayyir, A., Mashraqi, A.M., Gilal, A.R., Albouq, S.S.: Leveraging DistilBERT for summarizing Arabic text: an extractive dual-stage approach. IEEE Access 9, 135594\u2013135607 (2021)","journal-title":"IEEE Access"},{"issue":"1","key":"489_CR15","doi-asserted-by":"publisher","first-page":"7648","DOI":"10.1038\/s41598-021-87399-1","volume":"11","author":"H Chen","year":"2021","unstructured":"Chen, H., Zhang, Z.: Hybrid neural network based on novel audio feature for vehicle type identification. Sci. Rep. 11(1), 7648 (2021)","journal-title":"Sci. Rep."},{"issue":"1","key":"489_CR16","doi-asserted-by":"publisher","first-page":"183","DOI":"10.3390\/s20010183","volume":"20","author":"Mustaqeem","year":"2019","unstructured":"Mustaqeem, Kwon, S.: A CNN-assisted enhanced audio signal processing for speech emotion recognition. Sensors 20(1), 183 (2019)","journal-title":"Sensors"},{"key":"489_CR17","doi-asserted-by":"publisher","first-page":"169160","DOI":"10.1109\/ACCESS.2020.3024218","volume":"8","author":"H Wang","year":"2020","unstructured":"Wang, H., Gao, F., Zhao, Y., Wu, L.: WaveNet with cross-attention for audiovisual speech recognition. IEEE Access 8, 169160\u2013169168 (2020)","journal-title":"IEEE Access"},{"key":"489_CR18","doi-asserted-by":"publisher","first-page":"139438","DOI":"10.1109\/ACCESS.2019.2943492","volume":"7","author":"L Shi","year":"2019","unstructured":"Shi, L., Du, K., Zhang, C., Ma, H., Yan, W.: Lung sound recognition algorithm based on VGGish-BiGRU. IEEE Access 7, 139438\u2013139449 (2019)","journal-title":"IEEE Access"},{"key":"489_CR19","doi-asserted-by":"crossref","unstructured":"Zhang, Z., An, L., Cui, Z., Xu, A., Dong, T., Jiang, Y., Shi, J., Liu, X., Sun, X., Wang, M.: ABAW5 challenge: a facial affect recognition approach utilizing transformer encoder and audiovisual fusion. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 5724\u20135733 (2023)","DOI":"10.1109\/CVPRW59228.2023.00607"},{"key":"489_CR20","doi-asserted-by":"crossref","unstructured":"Xu, S., Li, L., Yao, Y., Chen, Z., Wu, H., Lu, Q., Tong, H.: MUSENET: multi-scenario learning for repeat-aware personalized recommendation. In: Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pp. 517\u2013525 (2023)","DOI":"10.1145\/3539597.3570414"},{"issue":"3","key":"489_CR21","first-page":"1","volume":"7","author":"R Zhu","year":"2023","unstructured":"Zhu, R., Shi, L., Song, Y., Cai, Z.: Integrating gaze and mouse via joint cross-attention fusion net for students\u2019 activity recognition in e-learning. Proc ACM Interact Mob Wear Ubiquitous Technol 7(3), 1\u201335 (2023)","journal-title":"Proc ACM Interact Mob Wear Ubiquitous Technol"},{"key":"489_CR22","doi-asserted-by":"crossref","unstructured":"Usmani, A., Alsamhi, S. H., Breslin, J., and Curry, E.: A novel framework for constructing multimodal knowledge graph from MuSe-CaR video reviews. In: 2023 IEEE 17th International Conference on Semantic Computing (ICSC), pp. 323\u2013328 (2023)","DOI":"10.1109\/ICSC56153.2023.00066"},{"key":"489_CR23","doi-asserted-by":"crossref","unstructured":"Han, W., Jiang, T., Li, Y., Schuller, B., Ruan, H.: Ordinal learning for emotion recognition in customer service calls. In: ICASSP 2020\u20132020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6494\u20136498 (2020)","DOI":"10.1109\/ICASSP40776.2020.9053648"},{"issue":"1","key":"489_CR24","doi-asserted-by":"publisher","first-page":"382","DOI":"10.3390\/s23010382","volume":"23","author":"EY Koh","year":"2022","unstructured":"Koh, E.Y., Cheuk, K.W., Heung, K.Y., Agres, K.R., Herremans, D.: MERP: a music dataset with emotion ratings and raters\u2019 profile information. Sensors 23(1), 382 (2022)","journal-title":"Sensors"},{"key":"489_CR25","unstructured":"Liu, K., DeMori, J., Abayomi, K.: Open set recognition for music genre classification. arXiv preprint arXiv:2209.07548 (2022)"},{"key":"489_CR26","doi-asserted-by":"crossref","unstructured":"Ding, Z., Qi, Y., Lin, D.: Albert-based sentiment analysis of movie review. In: 2021 4th International Conference on Advanced Electronic Materials, Computers and Software Engineering (AEMCSE), pp. 1243\u20131246 (2021)","DOI":"10.1109\/AEMCSE51986.2021.00254"},{"key":"489_CR27","unstructured":"Kim, C. D., Kim, B., Lee, H., Kim, G.: AudioCaps: generating captions for audios in the wild. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 119\u2013132 (2019)"},{"key":"489_CR28","doi-asserted-by":"crossref","unstructured":"Catharin, L. G., Ribeiro, R. P., Silla, C. N., Costa, Y. M. G., Feltrim, V. D. Multimodal classification of emotions in Latin music. In: 2020 IEEE International Symposium on Multimedia (ISM), pp. 173\u2013180 (2020)","DOI":"10.1109\/ISM.2020.00038"},{"issue":"14","key":"489_CR29","doi-asserted-by":"publisher","first-page":"4927","DOI":"10.3390\/s21144927","volume":"21","author":"YR Pandeya","year":"2021","unstructured":"Pandeya, Y.R., Bhattarai, B., Lee, J.: Deep-learning-based multimodal emotion classification for music videos. Sensors 21(14), 4927 (2021)","journal-title":"Sensors"},{"key":"489_CR30","doi-asserted-by":"crossref","unstructured":"Zhao, J., Ru, G., Yu, Y., Wu, Y., Li, D., Li, W.: Multimodal music emotion recognition with hierarchical cross-modal attention network. In: 2022 IEEE International Conference on Multimedia and Expo (ICME), pp. 1\u20136 (2022)","DOI":"10.1109\/ICME52920.2022.9859812"},{"issue":"2020","key":"489_CR31","first-page":"1","volume":"2020","author":"C Chen","year":"2020","unstructured":"Chen, C., Li, Q.: A multimodal music emotion classification method based on multifeature combined network classifier. Math. Probl. Eng. 2020(2020), 1\u201311 (2020)","journal-title":"Math. Probl. Eng."},{"issue":"4","key":"489_CR32","doi-asserted-by":"publisher","first-page":"1237","DOI":"10.1007\/s00779-020-01393-4","volume":"26","author":"YO Medina","year":"2022","unstructured":"Medina, Y.O., Beltr\u00e1n, J.R., Baldassarri, S.: Emotional classification of music using neural networks with the MediaEval dataset. Pers. Ubiquitous Comput. 26(4), 1237\u20131249 (2022)","journal-title":"Pers. Ubiquitous Comput."},{"key":"489_CR33","doi-asserted-by":"publisher","first-page":"102467","DOI":"10.1016\/j.displa.2023.102467","volume":"79","author":"E Ning","year":"2023","unstructured":"Ning, E., Zhang, C., Wang, C., Ning, X., Chen, H., Bai, X.: Pedestrian Re-ID based on feature consistency and contrast enhancement. Displays 79, 102467 (2023)","journal-title":"Displays"},{"issue":"2","key":"489_CR34","first-page":"331","volume":"61","author":"C Wan","year":"2023","unstructured":"Wan, C., Wang, Y.: Node classification algorithm based on weighted meta-learning. J. Jilin Univ. Sci. Ed. 61(2), 331\u2013337 (2023)","journal-title":"J. Jilin Univ. Sci. Ed."}],"updated-by":[{"DOI":"10.1007\/s44196-024-00541-5","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2024,6,5]],"date-time":"2024-06-05T00:00:00Z","timestamp":1717545600000}}],"container-title":["International Journal of Computational Intelligence Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44196-024-00489-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s44196-024-00489-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44196-024-00489-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,5]],"date-time":"2024-06-05T15:08:35Z","timestamp":1717600115000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s44196-024-00489-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,22]]},"references-count":34,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["489"],"URL":"https:\/\/doi.org\/10.1007\/s44196-024-00489-6","relation":{"correction":[{"id-type":"doi","id":"10.1007\/s44196-024-00541-5","asserted-by":"object"}]},"ISSN":["1875-6883"],"issn-type":[{"value":"1875-6883","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,22]]},"assertion":[{"value":"30 December 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 March 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 April 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 June 2024","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Correction","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"A Correction to this paper has been published:","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"https:\/\/doi.org\/10.1007\/s44196-024-00541-5","URL":"https:\/\/doi.org\/10.1007\/s44196-024-00541-5","order":7,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"99"}}