{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,28]],"date-time":"2026-01-28T20:52:28Z","timestamp":1769633548006,"version":"3.49.0"},"reference-count":31,"publisher":"Public Library of Science (PLoS)","issue":"1","license":[{"start":{"date-parts":[[2026,1,27]],"date-time":"2026-01-27T00:00:00Z","timestamp":1769472000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"<jats:p>Music therapy has emerged as a promising, yet underutilized, treatment modality in various clinical settings. This paper presents Nightingale, a novel multimodal approach that integrates audio features with electroencephalogram (EEG) signals to predict a subject\u2019s emotional response to music. The accuracy of emotional response predictions is enhanced using audio features coupled with EEG data extracted from the Dataset for Emotion Analysis using Physiological and Audiovisual Signals (DEAP). These combined modalities are used to develop a Multilayer Perceptron and Convolutional Neural Network architecture, achieving a Mean Absolute Percentage Error (MAPE) values of 19.52 for valence, 22.16 for arousal, and an R squared of 0.67 for the two domains. This approach offers superior performance to the most recent state-of-the-art in the field. Additionally, it requires significantly less computational resources and a simple network structure, providing an evidence-based prediction technique that could make music therapy more effective and accessible.<\/jats:p>","DOI":"10.1371\/journal.pone.0341497","type":"journal-article","created":{"date-parts":[[2026,1,27]],"date-time":"2026-01-27T18:48:03Z","timestamp":1769539683000},"page":"e0341497","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Nightingale: A multimodal approach combining EEG signals and audio features to enhance music therapy"],"prefix":"10.1371","volume":"21","author":[{"given":"Zirui","family":"Chen","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6651-902X","authenticated-orcid":true,"given":"Nikhil","family":"Yadav","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"340","published-online":{"date-parts":[[2026,1,27]]},"reference":[{"issue":"11","key":"pone.0341497.ref001","article-title":"Music therapy for depression","volume":"11","author":"S Aalbers","year":"2017","journal-title":"Cochrane Database Syst Rev."},{"issue":"8","key":"pone.0341497.ref002","first-page":"859","article-title":"The clinical effects of music therapy in palliative medicine","volume":"14","author":"LM Gallagher","year":"2006","journal-title":"Support Care Cancer."},{"key":"pone.0341497.ref003","unstructured":"Kern P, Humpal M. Early childhood music therapy and autism spectrum disorders. Jessica Kingsley Publishers; 2012."},{"issue":"5","key":"pone.0341497.ref004","first-page":"300","article-title":"Music therapy and Alzheimer\u2019s disease: cognitive, psychological, and behavioural effects","volume":"32","author":"M G\u00f3mez Gallego","year":"2017","journal-title":"Neurologia."},{"issue":"2","key":"pone.0341497.ref005","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1080\/17483107.2020.1775312","article-title":"Implementing music therapy through telehealth: considerations for military populations","volume":"17","author":"R Vaudreuil","year":"2022","journal-title":"Disabil Rehabil Assist Technol."},{"key":"pone.0341497.ref006","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.aip.2018.11.012","article-title":"Using distance technology to deliver the creative arts therapies to veterans: case studies in art, dance\/movement and music therapy","volume":"62","author":"H Spooner","year":"2019","journal-title":"The Arts in Psychotherapy."},{"issue":"10","key":"pone.0341497.ref007","article-title":"Music interventions for improving psychological and physical outcomes in people with cancer","volume":"10","author":"J Bradt","year":"2021","journal-title":"Cochrane Database Syst Rev."},{"issue":"4","key":"pone.0341497.ref008","doi-asserted-by":"crossref","first-page":"430","DOI":"10.1093\/jmt\/thw012","article-title":"The effects of music on pain: a meta-analysis","volume":"53","author":"JH Lee","year":"2016","journal-title":"J Music Ther."},{"issue":"6","key":"pone.0341497.ref009","doi-asserted-by":"crossref","first-page":"1161","DOI":"10.1037\/h0077714","article-title":"A circumplex model of affect.","volume":"39","author":"JA Russell","year":"1980","journal-title":"Journal of Personality and Social Psychology."},{"key":"pone.0341497.ref010","unstructured":"Bradley MM, Lang PJ. Affective norms for English words (ANEW): instruction manual and affective ratings. Center for Research in Psychophysiology, University of Florida; 1999."},{"key":"pone.0341497.ref011","unstructured":"Aucouturier JJ, Pachet F. Music similarity measures: what\u2019s the use? In: International Society for Music Information Retrieval Conference; 2002. https:\/\/api.semanticscholar.org\/CorpusID:14783916"},{"key":"pone.0341497.ref012","unstructured":"Hu X, Downie JS, Ehmann AF. Lyric text mining in music mood classification. American Music. 2009."},{"key":"pone.0341497.ref013","unstructured":"Bischoff K, Firan CS, Paiu R, Nejdl W, Laurier C, Sordo M. Music mood and theme classification: a hybrid approach. In: Proceedings of the International Society for Music Information Retrieval Conference (ISMIR). 2009. p. 657\u201362."},{"key":"pone.0341497.ref014","doi-asserted-by":"crossref","first-page":"59844","DOI":"10.1109\/ACCESS.2019.2914872","article-title":"Exploiting EEG signals and audiovisual feature fusion for video emotion recognition","volume":"7","author":"B Xing","year":"2019","journal-title":"IEEE Access."},{"key":"pone.0341497.ref015","doi-asserted-by":"crossref","unstructured":"Sawata R, Ogawa T, Haseyama M. The extraction of individual music preference based on deep time-series CCA. In: 2019 IEEE 8th Global Conference on Consumer Electronics (GCCE). 2019. p. 15\u20136. https:\/\/doi.org\/10.1109\/gcce46687.2019.9015274","DOI":"10.1109\/GCCE46687.2019.9015274"},{"key":"pone.0341497.ref016","doi-asserted-by":"crossref","unstructured":"Thammasan N, Fukui KI, Numao M. Fusion of EEG and musical features in continuous music-emotion recognition. 2016.","DOI":"10.1609\/aaai.v31i1.11112"},{"key":"pone.0341497.ref017","unstructured":"Kim YE, Schmidt EM, Migneco R, Morton BG, Richardson P, Scott J. Music emotion recognition: A state of the art review. In: Proceedings of the International Society for Music Information Retrieval Conference (ISMIR). 2010."},{"issue":"1","key":"pone.0341497.ref018","doi-asserted-by":"crossref","first-page":"13804","DOI":"10.1038\/s41598-023-40786-2","article-title":"Improved EEG-based emotion recognition through information enhancement in connectivity feature map","volume":"13","author":"MAH Akhand","year":"2023","journal-title":"Sci Rep."},{"issue":"1","key":"pone.0341497.ref019","doi-asserted-by":"crossref","first-page":"13491","DOI":"10.1038\/s41598-024-62990-4","article-title":"TPRO-NET: an EEG-based emotion recognition method reflecting subtle changes in emotion","volume":"14","author":"X Zhang","year":"2024","journal-title":"Sci Rep."},{"issue":"1","key":"pone.0341497.ref020","doi-asserted-by":"crossref","first-page":"10371","DOI":"10.1038\/s41598-024-60977-9","article-title":"Detecting emotions through EEG signals based on modified convolutional fuzzy neural network","volume":"14","author":"N Ahmadzadeh Nobari Azar","year":"2024","journal-title":"Sci Rep."},{"issue":"10","key":"pone.0341497.ref021","doi-asserted-by":"crossref","first-page":"3414","DOI":"10.3390\/s21103414","article-title":"Predicting exact valence and arousal values from EEG","volume":"21","author":"F Galv\u00e3o","year":"2021","journal-title":"Sensors (Basel)."},{"key":"pone.0341497.ref022","doi-asserted-by":"crossref","first-page":"102979","DOI":"10.1016\/j.bspc.2021.102979","article-title":"Feature and channel selection for designing a regression-based continuous-variable emotion recognition system with two EEG channels","volume":"70","author":"M Javidan","year":"2021","journal-title":"Biomedical Signal Processing and Control."},{"issue":"1","key":"pone.0341497.ref023","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/T-AFFC.2011.15","article-title":"DEAP: a database for emotion analysis; using physiological signals","volume":"3","author":"S Koelstra","year":"2012","journal-title":"IEEE Trans Affective Comput."},{"key":"pone.0341497.ref024","doi-asserted-by":"crossref","unstructured":"McFee B, Raffel C, Liang D, Ellis DPW, McVicar M, Battenberg E, et al. Librosa: audio and music signal analysis in Python. In: Proceedings of the SciPy Conference. 2015.","DOI":"10.25080\/Majora-7b98e3ed-003"},{"issue":"2","key":"pone.0341497.ref025","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1525\/mp.2019.37.2.134","article-title":"The effects of timbre on neural responses to musical emotion","volume":"37","author":"W Zhang","year":"2019","journal-title":"Music Perception."},{"key":"pone.0341497.ref026","unstructured":"Virtanen P, Gommers R, Burovski E, Oliphant TE, Weckesser W, Cournapeau D, et al. SciPy 1.6.0; 2021. https:\/\/doi.org\/10.5281\/zenodo.4725557"},{"issue":"11","key":"pone.0341497.ref027","doi-asserted-by":"crossref","first-page":"1083","DOI":"10.1109\/10.541250","article-title":"Dominant frequency analysis of EEG reveals brain\u2019s response during injury and recovery","volume":"43","author":"V Goel","year":"1996","journal-title":"IEEE Trans Biomed Eng."},{"issue":"2","key":"pone.0341497.ref028","doi-asserted-by":"crossref","first-page":"268","DOI":"10.3390\/brainsci13020268","article-title":"Convolutional neural network with a topographic representation module for EEG-based brain-computer interfaces","volume":"13","author":"X Liang","year":"2023","journal-title":"Brain Sci."},{"issue":"13","key":"pone.0341497.ref029","doi-asserted-by":"crossref","first-page":"5796","DOI":"10.3390\/app11135796","article-title":"An ensemble of convolutional neural networks for audio classification","volume":"11","author":"L Nanni","year":"2021","journal-title":"Applied Sciences."},{"key":"pone.0341497.ref030","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1016\/j.neucom.2015.12.114","article-title":"Mean absolute percentage error for regression models","volume":"192","author":"A de Myttenaere","year":"2016","journal-title":"Neurocomputing."},{"issue":"6","key":"pone.0341497.ref031","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pbio.3002651","article-title":"A closed-loop auditory stimulation approach selectively modulates alpha oscillations and sleep onset dynamics in humans","volume":"22","author":"H Hebron","year":"2024","journal-title":"PLoS Biol."}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0341497","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,27]],"date-time":"2026-01-27T18:48:07Z","timestamp":1769539687000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0341497"}},"subtitle":[],"editor":[{"given":"Mukesh","family":"Dhamala","sequence":"first","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2026,1,27]]},"references-count":31,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,1,27]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0341497","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,27]]}}}