{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T20:10:47Z","timestamp":1789503047225,"version":"build-2803163510"},"reference-count":51,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2020,3,4]],"date-time":"2020-03-04T00:00:00Z","timestamp":1583280000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61702239"],"award-info":[{"award-number":["61702239"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61672391"],"award-info":[{"award-number":["61672391"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003410","name":"Department of Education, Fujian Province","doi-asserted-by":"publisher","award":["FBJG 20180015"],"award-info":[{"award-number":["FBJG 20180015"]}],"id":[{"id":"10.13039\/501100003410","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100009102","name":"Education Department of Jiangxi Province","doi-asserted-by":"publisher","award":["GJJ170765"],"award-info":[{"award-number":["GJJ170765"]}],"id":[{"id":"10.13039\/501100009102","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100009102","name":"Education Department of Jiangxi Province","doi-asserted-by":"publisher","award":["GJJ170798"],"award-info":[{"award-number":["GJJ170798"]}],"id":[{"id":"10.13039\/501100009102","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100010857","name":"Jiangxi Provincial Department of Science and Technology","doi-asserted-by":"publisher","award":["20161GYZD011-011"],"award-info":[{"award-number":["20161GYZD011-011"]}],"id":[{"id":"10.13039\/501100010857","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Most of the existing research papers study the emotion recognition of Minnan songs from the perspectives of music analysis theory and music appreciation. However, these investigations do not explore any possibility of carrying out an automatic emotion recognition of Minnan songs. In this paper, we propose a model that consists of four main modules to classify the emotion of Minnan songs by using the bimodal data\u2014song lyrics and audio. In the proposed model, an attention-based Long Short-Term Memory (LSTM) neural network is applied to extract lyrical features, and a Convolutional Neural Network (CNN) is used to extract the audio features from the spectrum. Then, two kinds of extracted features are concatenated by multimodal compact bilinear pooling, and finally, the concatenated features are input to the classifying module to determine the song emotion. We designed three experiment groups to investigate the classifying performance of combinations of the four main parts, the comparisons of proposed model with the current approaches and the influence of a few key parameters on the performance of emotion recognition. The results show that the proposed model exhibits better performance over all other experimental groups. The accuracy, precision and recall of the proposed model exceed 0.80 in a combination of appropriate parameters.<\/jats:p>","DOI":"10.3390\/info11030145","type":"journal-article","created":{"date-parts":[[2020,3,4]],"date-time":"2020-03-04T10:46:08Z","timestamp":1583318768000},"page":"145","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Bimodal Emotion Recognition Model for Minnan Songs"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8484-5665","authenticated-orcid":false,"given":"Zhenglong","family":"Xiang","sequence":"first","affiliation":[{"name":"School of Computer Science, Wuhan University, Wuhan 430072, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xialei","family":"Dong","sequence":"additional","affiliation":[{"name":"School of Computer Science, Wuhan University, Wuhan 430072, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuanxiang","family":"Li","sequence":"additional","affiliation":[{"name":"School of Computer Science, Wuhan University, Wuhan 430072, China"},{"name":"School of Physics and Information Engineering, Minnan Normal University, Zhangzhou 363000, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fei","family":"Yu","sequence":"additional","affiliation":[{"name":"School of Physics and Information Engineering, Minnan Normal University, Zhangzhou 363000, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xing","family":"Xu","sequence":"additional","affiliation":[{"name":"School of Physics and Information Engineering, Minnan Normal University, Zhangzhou 363000, China"},{"name":"College of Information and Engineering, Jingdezhen Ceramic Institute, Jingdezhen 333000, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0492-7471","authenticated-orcid":false,"given":"Hongrun","family":"Wu","sequence":"additional","affiliation":[{"name":"School of Physics and Information Engineering, Minnan Normal University, Zhangzhou 363000, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,3,4]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1163\/1568520952600597","article-title":"The Fu of Minnan: A Local Clan in Late Tang and Song China (9th\u201313th Centuries)","volume":"38","author":"Clark","year":"1995","journal-title":"J. Econ. Soc. Hist. Orient"},{"key":"ref_2","unstructured":"Chuimei, H. (2001). The ceramic boom in Minnan during Song and Yuan times. The Emporium of the World: Maritime Quanzhou, 1000\u20131400, Brill."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1034\/j.1399-0039.2001.057003192.x","article-title":"The origin of Minnan and Hakka, the so-called Taiwanese, inferred by HLA study","volume":"57","author":"Lin","year":"2001","journal-title":"Tissue Antigens"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1177\/0305735606067168","article-title":"Songs and emotions: Are lyrics and melodies equal partners?","volume":"34","author":"Ali","year":"2006","journal-title":"Psychol. Music"},{"key":"ref_5","first-page":"17","article-title":"Study on Choral Music in Minnan Dialect in Fujian and Taiwan","volume":"14","author":"Hu","year":"2011","journal-title":"J. Jimei Univ. (Philosophy Soc. Sci.)"},{"key":"ref_6","first-page":"136","article-title":"An Analysis of Folk Songs in Minnan Dialect","volume":"30","author":"Lihong","year":"2011","journal-title":"J. Jimei Univ. (Philosophy Soc. Sci.)"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1177\/1029864912448327","article-title":"Cognitive function, origin, and evolution of musical emotions","volume":"16","author":"Perlovsky","year":"2012","journal-title":"Music. Sci."},{"key":"ref_8","unstructured":"Li, T., and Ogihara, M. (2020, March 02). Detecting Emotion in Music. Available online: https:\/\/jscholarship.library.jhu.edu\/handle\/1774.2\/41."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"564","DOI":"10.1109\/TMM.2006.870730","article-title":"Toward intelligent music information retrieval","volume":"8","author":"Li","year":"2006","journal-title":"IEEE Trans. Multimed."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Kirandziska, V., and Ackovska, N. (2013, January 1\u20134). Finding important sound features for emotion evaluation classification. Proceedings of the Eurocon 2013, Zagreb, Croatia.","DOI":"10.1109\/EUROCON.2013.6625196"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Misron, M.M., Rosli, N., Manaf, N.A., and Halim, H.A. (2014). Music Emotion Classification (MEC): Exploiting Vocal and Instrumental Sound Features. Recent Advances on Soft Computing and Data Mining, Springer.","DOI":"10.1007\/978-3-319-07692-8_51"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Ridoean, J.A., Sarno, R., Sunaryo, D., and Wijaya, D.R. (2017, January 25\u201326). Music mood classification using audio power and audio harmonicity based on MPEG-7 audio features and Support Vector Machine. Proceedings of the 2017 3rd International Conference on Science in Information Technology (ICSITech), Bandung, Indonesia.","DOI":"10.1109\/ICSITech.2017.8257088"},{"key":"ref_13","unstructured":"Eerola, T., Lartillot, O., and Toiviainen, P. (2009, January 26\u201330). Prediction of Multidimensional Emotional Ratings in Music from Audio Using Multivariate Regression Models. Proceedings of the 10th International Society for Music Information Retrieval Conference, Kobe, Japan."},{"key":"ref_14","unstructured":"Lartillot, O., and Toiviainen, P. (2007, January 10\u201315). A Matlab toolbox for musical feature extraction from audio. Proceedings of the International Conference on Digital Audio Effects, Bordeaux, France."},{"key":"ref_15","unstructured":"G\u00f3mez, E. (2006). Tonal Description of Music Audio Signals. [Ph.D. Thesis, Universitat Pompeu Fabra]."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Stoller, D., Durand, S., and Ewert, S. (2019, January 12\u201317). End-to-end Lyrics Alignment for Polyphonic Music Using an Audio-to-character Recognition Model. Proceedings of the ICASSP 2019\u20142019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.","DOI":"10.1109\/ICASSP.2019.8683470"},{"key":"ref_17","unstructured":"Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"591","DOI":"10.1109\/TPAMI.2008.111","article-title":"Efficient visual search of videos cast as text retrieval","volume":"31","author":"Sivic","year":"2008","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/505282.505283","article-title":"Machine learning in automated text categorization","volume":"34","author":"Sebastiani","year":"2002","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Hofmann, T. (1999). Probabilistic latent semantic indexing. ACM SIGIR Forum, ACM Press.","DOI":"10.1145\/312624.312649"},{"key":"ref_21","first-page":"2","article-title":"Lyric text mining in music mood classification","volume":"183","author":"Hu","year":"2009","journal-title":"Am. Music"},{"key":"ref_22","unstructured":"Laurier, C., and Herrera, P. (2008). Mood cloud: A real-time music mood visualization tool. Computer Music Modeling and Retrieval. Sense of Sounds, Springer."},{"key":"ref_23","unstructured":"Lu, Q., Chen, X., Yang, D., and Wang, J. (2010, January 9\u201313). Boosting for Multi-Modal Music Emotion. Proceedings of the 11th International Society for Music Information and Retrieval Conference, Utrecht, The Netherlands."},{"key":"ref_24","unstructured":"Meyers, O.C. (2007). A mood-based music classification and exploration system. [Master\u2019s Thesis, Massachusetts Institute of Technology]."},{"key":"ref_25","unstructured":"Yang, D., and Lee, W.S. (2004, January 10\u201314). Disambiguating Music Emotion Using Software Agents. Proceedings of the 5th International Conference on Music Information Retrieval, Barcelona, Spain."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"448","DOI":"10.1109\/TASL.2007.911513","article-title":"A regression approach to music emotion recognition","volume":"16","author":"Yang","year":"2008","journal-title":"IEEE Trans. Audio Speech Lang. Process."},{"key":"ref_27","unstructured":"Cheng, H.T., Yang, Y.H., Lin, Y.C., Liao, I.B., and Chen, H.H. (April, January 23). Automatic chord recognition for music classification and retrieval. Proceedings of the 2008 IEEE International Conference on Multimedia and Expo, Hannover, Germany."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Dang, T.T., and Shirai, K. (2009, January 13\u201317). Machine learning approaches for mood classification of songs toward music search engine. Proceedings of the 2009 International Conference on Knowledge and Systems Engineering, Hanoi, Vietnam.","DOI":"10.1109\/KSE.2009.10"},{"key":"ref_29","first-page":"99","article-title":"Lyric-Based Song Sentiment Analysis by Sentiment Vector Space Model","volume":"24","author":"Xia","year":"2010","journal-title":"J. Chin. Inf. Process."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1037\/h0054832","article-title":"Expression in music: A discussion of experimental studies and theories","volume":"42","author":"Hevner","year":"1935","journal-title":"Psychol. Rev."},{"key":"ref_31","first-page":"45","article-title":"Sentiment Vector Space Model Based Music Emotion Tag Prediction","volume":"26","author":"Li","year":"2012","journal-title":"J. Chin. Inf. Process."},{"key":"ref_32","unstructured":"Raschka, S. (2016). MusicMood: Predicting the mood of music from song lyrics using machine learning. arXiv."},{"key":"ref_33","unstructured":"Sharma, D.M., Sangal, R., and Sherly, E. (2015). Mood classification of hindi songs based on lyrics. Proceedings of the 12th International Conference on Natural Language Processing, NLP Association of India."},{"key":"ref_34","unstructured":"Miotto, R. (2011). Content-based Music Access: Combining Audio Features and Semantic Information for Music Search Engines. [Master\u2019s Thesis, Universita degli Studi di Pavia]."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Hu, X., and Downie, J.S. (2010). Improving mood classification in music digital libraries by combining lyrics and audio. Proceedings of the 10th Annual Joint Conference on Digital Libraries, Association for Computing Machinery.","DOI":"10.1145\/1816123.1816146"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Jamdar, A., Abraham, J., Khanna, K., and Dubey, R. (2015). Emotion analysis of songs based on lyrical and audio features. arXiv.","DOI":"10.5121\/ijaia.2015.6304"},{"key":"ref_37","unstructured":"Lee, C.W., Song, K.Y., Jeong, J., and Choi, W.Y. (2018, January 15\u201320). Convolutional attention networks for multimodal emotion recognition from speech and text data. Proceedings of the first Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML), Vancouver, BC, Canada."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Lin, Y.C., Yang, Y.H., Chen, H.H., Liao, I.B., and Ho, Y.C. (July, January 28). Exploiting genre for music emotion classification. Proceedings of the 2009 IEEE International Conference on Multimedia and Expo, New York, NY, USA.","DOI":"10.1109\/ICME.2009.5202572"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"846135","DOI":"10.1155\/2008\/846135","article-title":"Tango or waltz?: Putting ballroom dance style into tempo detection","volume":"2008","author":"Schuller","year":"2008","journal-title":"EURASIP J. Audio Speech Music Process."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"735854","DOI":"10.1186\/1687-4722-2010-735854","article-title":"Determination of nonprototypical valence and arousal in popular music: Features and performances","volume":"2010","author":"Schuller","year":"2010","journal-title":"EURASIP J. Audio Speech Music Process."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Widmayer, P., Eidenbenz, S., Triguero, F., Morales, R., Conejo, R., and Hennessy, M. (2002). Finding Frequent Items in Data Streams. Automata, Languages and Programming, Springer.","DOI":"10.1007\/3-540-45465-9"},{"key":"ref_42","unstructured":"Huang, R. (2015). Taiwanese Songs Cultural Communication Studies. [Ph.D. Thesis, Huazhong Normal University]."},{"key":"ref_43","unstructured":"Durrani, N., and Hussain, S. (2010, January 2\u20134). Urdu Word Segmentation. Proceedings of the Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Los Angeles, CA, USA."},{"key":"ref_44","unstructured":"Yang, Y. (2003). Discussion on the construction technology and usage of Taiwan Chinese Online Dictionary. Proceedings of the 2003 3rd International Symposium on Chinese Online Education, Taiwan University."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Eyben, F., W\u00f6llmer, M., and Schuller, B. (2010, January 25\u201329). openSMILE\u2014The Munich Versatile and Fast Open-Source Audio Feature Extractor. Proceedings of the 9th ACM International Conference on Multimedia, Firenze, Italy.","DOI":"10.1145\/1873951.1874246"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Matsumoto, K., Sasayama, M., Yoshida, M., Kita, K., and Ren, F. (2018, January 23\u201325). Transfer Learning Based on Utterance Emotion Corpus for Lyric Emotion Estimation. Proceedings of the 2018 5th IEEE International Conference on Cloud Computing and Intelligence Systems (CCIS), Nanjing, China.","DOI":"10.1109\/CCIS.2018.8691207"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"An, Y., Sun, S., and Wang, S. (2017, January 24\u201326). Naive Bayes classifiers for music emotion classification based on lyrics. Proceedings of the 2017 IEEE\/ACIS 16th International Conference on Computer and Information Science (ICIS), Wuhan, China.","DOI":"10.1109\/ICIS.2017.7960070"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Lim, W., Jang, D., and Lee, T. (2016, January 13\u201316). Speech emotion recognition using convolutional and Recurrent Neural Networks. Proceedings of the 2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), Jeju, Korea.","DOI":"10.1109\/APSIPA.2016.7820699"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.neunet.2017.02.013","article-title":"Evaluating deep learning architectures for Speech Emotion Recognition","volume":"92","author":"Fayek","year":"2017","journal-title":"Neural Netw."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Poria, S., Chaturvedi, I., Cambria, E., and Hussain, A. (2016, January 12\u201315). Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis. Proceedings of the 2016 IEEE 16th International Conference on Data Mining (ICDM), Barcelona, Spain.","DOI":"10.1109\/ICDM.2016.0055"},{"key":"ref_51","unstructured":"Jeon, B., Kim, C., Kim, A., Kim, D., Park, J., and Ha, J.W. (2020, March 03). Music Emotion Recognition via End-to-End Multimodal Neural Networks. Available online: https:\/\/pdfs.semanticscholar.org\/ce3b\/93d715b16ab9f7b65442d37a9fedcee18071.pdf."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/11\/3\/145\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:04:06Z","timestamp":1760173446000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/11\/3\/145"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,4]]},"references-count":51,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2020,3]]}},"alternative-id":["info11030145"],"URL":"https:\/\/doi.org\/10.3390\/info11030145","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3,4]]}}}