{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,27]],"date-time":"2026-09-27T05:43:30Z","timestamp":1790487810439,"version":"4.1.0"},"reference-count":71,"publisher":"Acoustical Society of America (ASA)","issue":"6","funder":[{"name":"Cluster of Excellence Hearing4all","award":["EXC 1077\/1"],"award-info":[{"award-number":["EXC 1077\/1"]}]}],"content-domain":{"domain":["pubs.aip.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2018,6,1]]},"abstract":"<jats:p>The severe hearing loss problems that some people suffer can be treated by providing them with a surgically implanted electrical device called cochlear implant (CI). CI users struggle to perceive complex audio signals such as music; however, previous studies show that CI recipients find music more enjoyable when the vocals are enhanced with respect to the background music. In this manuscript source separation (SS) algorithms are used to remix pop songs by applying gain to the lead singing voice. This work uses deep convolutional auto-encoders, a deep recurrent neural network, a multilayer perceptron (MLP), and non-negative matrix factorization to be evaluated objectively and subjectively through two different perceptual experiments which involve normal hearing subjects and CI recipients. The evaluation assesses the relevance of the artifacts introduced by the SS algorithms considering their computation time, as this study aims at proposing one of the algorithms for real-time implementation. Results show that the MLP performs in a robust way throughout the tested data while providing levels of distortions and artifacts which are not perceived by CI users. Thus, an MLP is proposed to be implemented for real-time monaural audio SS to remix music for CI users.<\/jats:p>","DOI":"10.1121\/1.5042056","type":"journal-article","created":{"date-parts":[[2018,6,19]],"date-time":"2018-06-19T15:46:35Z","timestamp":1529423195000},"page":"3602-3615","update-policy":"https:\/\/doi.org\/10.1063\/aip-crossmark-policy-page","source":"Crossref","is-referenced-by-count":38,"title":["Deep learning models to remix music for cochlear implant users"],"prefix":"10.1121","volume":"143","author":[{"given":"Tom","family":"Gaj\u0119cki","sequence":"first","affiliation":[{"name":"Department of Otolaryngology, Medical University Hannover and Cluster of Excellence Hearing4all , Hannover, 30625, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Waldo","family":"Nogueira","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology, Medical University Hannover and Cluster of Excellence Hearing4all , Hannover, 30625, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"231","published-online":{"date-parts":[[2018,6,19]]},"reference":[{"issue":"5","key":"2023073121522663500_c1","doi-asserted-by":"publisher","first-page":"695","DOI":"10.1682\/JRRD.2007.10.0173","article-title":"Cochlear implants: Current designs and future possibilities","volume":"45","year":"2008","journal-title":"J. Rehab. Res. Dev."},{"issue":"2","key":"2023073121522663500_c2","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1177\/108471380400800203","article-title":"Music perception with cochlear implants: A review","volume":"8","year":"2004","journal-title":"Trends Amplif."},{"key":"2023073121522663500_c3","doi-asserted-by":"publisher","first-page":"1655","DOI":"10.1121\/1.387220","article-title":"Played again SAM: Further observations on the pitch of amplitude modulated noise","volume":"70","year":"1981","journal-title":"J. Acoust. Soc. Am."},{"key":"2023073121522663500_c4","doi-asserted-by":"publisher","first-page":"518","DOI":"10.1111\/j.1749-6632.2009.04551.x","article-title":"Melodic contour identification and music perception by cochlear implant users","volume":"1169","year":"2009","journal-title":"Ann. N.Y. Acad. Sci."},{"issue":"5","key":"2023073121522663500_c5","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1121\/1.3239464","article-title":"Perceptual fusion of polyphonic pitch in cochlear implant users","volume":"126","year":"2009","journal-title":"J. Acoust. Soc. Am."},{"issue":"3","key":"2023073121522663500_c6","doi-asserted-by":"publisher","first-page":"445","DOI":"10.1109\/TASLP.2015.2511623","article-title":"Spectral complexity reduction of music signals for mitigating effects of cochlear hearing loss","volume":"24","year":"2016","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"issue":"3","key":"2023073121522663500_c7","doi-asserted-by":"publisher","first-page":"1219","DOI":"10.1121\/1.5000484","article-title":"Spectral complexity reduction of music signals based on frequency-domain reduced-rank approximations: An evaluation with cochlear implant listeners","volume":"142","year":"2017","journal-title":"J. Acoust. Soc. Am."},{"issue":"3","key":"2023073121522663500_c8","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1179\/1467010015Z.000000000269","article-title":"A preliminary report of music-based training for adult cochlear implant users: Rationales and development","volume":"16","year":"2015","journal-title":"Cochl. Implants Int."},{"issue":"5","key":"2023073121522663500_c9","doi-asserted-by":"publisher","first-page":"294","DOI":"10.3109\/14992027.2013.873955","article-title":"Music mixing preferences of cochlear implant recipients: A pilot study","volume":"53","year":"2014","journal-title":"Int. J. Audiol."},{"issue":"6","key":"2023073121522663500_c10","doi-asserted-by":"publisher","first-page":"4338","DOI":"10.1121\/1.4971424","article-title":"Remixing music using source separation algorithms to improve the musical experience of cochlear implant users","volume":"140","year":"2016","journal-title":"J. Acoust. Soc. Am."},{"issue":"10","key":"2023073121522663500_c11","doi-asserted-by":"publisher","first-page":"2434","DOI":"10.1109\/TBME.2015.2428999","article-title":"A stereo music preprocessing scheme for cochlear implant users","volume":"62","year":"2015","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"2023073121522663500_c12","article-title":"Recommendation ITU-R BS. 1534-1: Method for the subjective assessment of intermediate quality level of coding systems","author":"ITU","year":"2003"},{"issue":"4","key":"2023073121522663500_c13","doi-asserted-by":"publisher","first-page":"2379","DOI":"10.1121\/1.2839887","article-title":"Using blind source separation techniques to improve speech recognition in bilateral cochlear implant patients","volume":"123","year":"2008","journal-title":"J. Acoust. Soc. Am."},{"key":"2023073121522663500_c14","first-page":"1567","article-title":"An interactive audio source separation framework based on non-negative matrix factorization","year":"2014"},{"issue":"3","key":"2023073121522663500_c15","doi-asserted-by":"publisher","first-page":"1066","DOI":"10.1109\/TASL.2006.885253","article-title":"Monaural sound source separation by nonnegative matrix factorization with temporal continuity and sparseness criteria","volume":"15","year":"2007","journal-title":"IEEE Trans. Audio Speech Lang. Process."},{"key":"2023073121522663500_c16","first-page":"1","article-title":"Non-negative matrix factorization for irregularly-spaced transforms","year":"2013"},{"key":"2023073121522663500_c17","article-title":"Low-latency bass separation using harmonic-percussion decomposition","year":"2013"},{"issue":"7553","key":"2023073121522663500_c18","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","year":"2015","journal-title":"Nature"},{"key":"2023073121522663500_c19","article-title":"Supervised speech separation based on deep learning: An overview","year":"2017"},{"issue":"5","key":"2023073121522663500_c20","doi-asserted-by":"publisher","first-page":"1075","DOI":"10.1109\/TASLP.2017.2687104","article-title":"Deep learning based binaural speech separation in reverberant environments","volume":"25","year":"2017","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"issue":"12","key":"2023073121522663500_c21","doi-asserted-by":"publisher","first-page":"1849","DOI":"10.1109\/TASLP.2014.2352935","article-title":"On training targets for supervised speech separation","volume":"22","year":"2014","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"issue":"9","key":"2023073121522663500_c22","doi-asserted-by":"publisher","first-page":"1652","DOI":"10.1109\/TASLP.2016.2580946","article-title":"Multichannel audio source separation with deep neural networks","volume":"24","year":"2016","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"2023073121522663500_c23","article-title":"Development of a sound coding strategy based on a deep recurrent neural network for monaural source separation in cochlear implants","year":"2016"},{"issue":"12","key":"2023073121522663500_c24","doi-asserted-by":"publisher","first-page":"2136","DOI":"10.1109\/TASLP.2015.2468583","article-title":"Joint optimization of masks and deep recurrent neural networks for monaural source separation","volume":"23","year":"2015","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"2023073121522663500_c25","first-page":"52","article-title":"Stacked convolutional auto-encoders for hierarchical feature extraction","volume-title":"Artificial Neural Networks and Machine Learning\u2013ICANN 2011","author":"Honkela","year":"2011"},{"issue":"4","key":"2023073121522663500_c26","doi-asserted-by":"publisher","first-page":"1017","DOI":"10.1109\/TCYB.2016.2536638","article-title":"Stacked convolutional denoising auto-encoders for feature representation","volume":"47","year":"2017","journal-title":"IEEE Trans. Cybern."},{"key":"2023073121522663500_c27","first-page":"341","article-title":"Image denoising and inpainting with deep neural networks","volume-title":"Advances in Neural Information Processing Systems 25","author":"Pereira","year":"2012"},{"key":"2023073121522663500_c28","first-page":"3371","article-title":"Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion","volume":"11","year":"2010","journal-title":"J. Mach. Learn. Res."},{"key":"2023073121522663500_c29","first-page":"1096","volume-title":"Extracting and Composing Robust Features with Denoising Autoencoders","year":"2008"},{"issue":"5786","key":"2023073121522663500_c30","doi-asserted-by":"publisher","first-page":"504","DOI":"10.1126\/science.1127647","article-title":"Reducing the dimensionality of data with neural networks","volume":"313","year":"2006","journal-title":"Science"},{"key":"2023073121522663500_c31","first-page":"100","volume-title":"Adaptive Denoising Autoencoders: A Fine-Tuning Scheme to Learn from Test Mixtures","year":"2015"},{"key":"2023073121522663500_c32","first-page":"86","article-title":"A neural network alternative to non-negative audio models","year":"2017"},{"issue":"6","key":"2023073121522663500_c33","doi-asserted-by":"publisher","first-page":"84","DOI":"10.1145\/3065386","article-title":"Imagenet classification with deep convolutional neural networks","volume":"60","year":"2017","journal-title":"Commun. ACM"},{"issue":"4","key":"2023073121522663500_c34","doi-asserted-by":"publisher","first-page":"541","DOI":"10.1162\/neco.1989.1.4.541","article-title":"Backpropagation applied to handwritten zip code recognition","volume":"1","year":"1989","journal-title":"Neural Comput."},{"key":"2023073121522663500_c35","article-title":"Overfeat: Integrated recognition, localization and detection using convolutional networks","year":"2013"},{"key":"2023073121522663500_c36","article-title":"Visualizing and understanding convolutional networks","year":"2013"},{"key":"2023073121522663500_c37","article-title":"Return of the devil in the details: Delving deep into convolutional nets","year":"2014"},{"key":"2023073121522663500_c38","article-title":"Spatial pyramid pooling in deep convolutional networks for visual recognition","year":"2014"},{"key":"2023073121522663500_c39","article-title":"Very deep convolutional networks for large-scale image recognition","year":"2014"},{"key":"2023073121522663500_c40","article-title":"Going deeper with convolutions","year":"2014"},{"key":"2023073121522663500_c41","article-title":"Simultaneous detection and segmentation","year":"2014"},{"key":"2023073121522663500_c42","article-title":"CNN features off-the-shelf: An astounding baseline for recognition","year":"2014"},{"key":"2023073121522663500_c43","first-page":"3768","article-title":"SNR-aware convolutional neural network modeling for speech enhancement","year":"2016"},{"issue":"1","key":"2023073121522663500_c44","doi-asserted-by":"publisher","first-page":"208","DOI":"10.1109\/TASLP.2016.2632307","article-title":"Deep convolutional neural networks for predominant instrument recognition in polyphonic music","volume":"25","year":"2017","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"2023073121522663500_c45","article-title":"Improving music source separation based on DNNs through data augmentation and network blending","year":"2017"},{"key":"2023073121522663500_c46","article-title":"Discriminative enhancement for single channel audio source separation using deep neural networks","year":"2016"},{"key":"2023073121522663500_c47","first-page":"258","article-title":"Monoaural audio source separation using deep convolutional neural networks","year":"2017"},{"key":"2023073121522663500_c48","article-title":"Single channel audio source separation using convolutional denoising autoencoders","year":"2017"},{"key":"2023073121522663500_c49","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1016\/j.heares.2016.11.012","article-title":"Speech enhancement based on neural networks improves speech intelligibility in noise for cochlear implant users","volume":"344","year":"2017","journal-title":"Hear. Res."},{"key":"2023073121522663500_c50","article-title":"Algorithms for non-negative matrix factorization","year":"2001"},{"issue":"14\u201315","key":"2023073121522663500_c51","doi-asserted-by":"publisher","first-page":"2627","DOI":"10.1016\/S1352-2310(97)00447-0","article-title":"Artificial neural networks (the multilayer perceptron). A review of applications in the atmospheric sciences","volume":"32","year":"1998","journal-title":"Atmos. Environ."},{"key":"2023073121522663500_c52","unstructured":"D.\u2008Kriesel, A Brief Introduction to Neural Networks, http:\/\/www.dkriesel.com (Last viewed June 7, 2018)."},{"issue":"5","key":"2023073121522663500_c53","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1016\/0893-6080(89)90020-8","article-title":"Multilayer feedforward networks are universal approximators","volume":"2","year":"1989","journal-title":"Neural Netw."},{"issue":"5","key":"2023073121522663500_c54","doi-asserted-by":"publisher","first-page":"1353","DOI":"10.1109\/TVCG.2006.113","article-title":"A spectral analysis of function composition and its implications for sampling in direct volume visualization","volume":"12","year":"2006","journal-title":"IEEE Trans. Visual. Comput. Graph."},{"key":"2023073121522663500_c55","first-page":"92","article-title":"Evaluation of pooling operations in convolutional architectures for object recognition","volume-title":"Artificial Neural Networks\u2013ICANN 2010","author":"Diamantaras","year":"2010"},{"issue":"1","key":"2023073121522663500_c56","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1214\/aoms\/1177729694","article-title":"On information and sufficiency","volume":"22","year":"1951","journal-title":"Ann. Math. Stat."},{"key":"2023073121522663500_c57","first-page":"1562","article-title":"Deep learning for monaural speech separation","year":"2014"},{"key":"2023073121522663500_c58","volume-title":"Neural Networks for Pattern Recognition","year":"1995"},{"key":"2023073121522663500_c59","article-title":"Adadelta: An adaptive learning rate method","year":"2012"},{"key":"2023073121522663500_c60","first-page":"718","article-title":"Vocal activity informed singing voice separation with the ikala dataset","year":"2015"},{"key":"2023073121522663500_c61","unstructured":"https:\/\/hydrogenaud.io\/index.php\/topic,85536.msg736023.html#msg736023 (Last viewed June 7, 2018)."},{"issue":"4","key":"2023073121522663500_c62","doi-asserted-by":"publisher","first-page":"1462","DOI":"10.1109\/TSA.2005.858005","article-title":"Performance measurement in blind audio source separation","volume":"14","year":"2006","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"issue":"7","key":"2023073121522663500_c63","doi-asserted-by":"publisher","first-page":"2046","DOI":"10.1109\/TASL.2011.2109381","article-title":"Subjective and objective quality assessment of audio source separation","volume":"19","year":"2011","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"2023073121522663500_c64","unstructured":"http:\/\/c4dm.eecs.qmul.ac.uk\/downloads\/index.html#mushram (Last viewed November 15, 2013)."},{"key":"2023073121522663500_c65","unstructured":"http:\/\/www.music-ir.org\/mirex\/wiki\/2016:Singing\\protect\\_Voice\\protect\\_Separation\\protect\\_Results (Last viewed June 3, 2016)."},{"issue":"4","key":"2023073121522663500_c66","doi-asserted-by":"publisher","first-page":"468","DOI":"10.1097\/AUD.0b013e31820dd3f0","article-title":"The effect of different cochlear implant microphones on acoustic hearing individuals binaural benefits for speech perception in noise","volume":"32","year":"2011","journal-title":"Ear Hear."},{"key":"2023073121522663500_c67","volume-title":"RStudio: Integrated Development Environment for R","author":"RStudio Team","year":"2015"},{"issue":"2","key":"2023073121522663500_c68","doi-asserted-by":"publisher","first-page":"99","DOI":"10.2307\/3001913","article-title":"Comparing individual means in the analysis of variance","volume":"5","year":"1949","journal-title":"Biometrics"},{"key":"2023073121522663500_c69","volume-title":"Python Reference Manual","year":"1995"},{"key":"2023073121522663500_c70","volume-title":"Pattern Recognition and Machine Learning (Information Science and Statistics)","year":"2006"},{"issue":"2","key":"2023073121522663500_c71","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1109\/MSP.2013.2288990","article-title":"Compositional models for audio processing: Uncovering the structure of sound mixtures","volume":"32","year":"2015","journal-title":"IEEE Sign. Process. Mag."}],"container-title":["The Journal of the Acoustical Society of America"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pubs.aip.org\/asa\/jasa\/article-pdf\/143\/6\/3602\/15332307\/3602_1_online.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/pubs.aip.org\/asa\/jasa\/article-pdf\/143\/6\/3602\/15332307\/3602_1_online.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,7,31]],"date-time":"2023-07-31T21:52:39Z","timestamp":1690840359000},"score":1,"resource":{"primary":{"URL":"https:\/\/pubs.aip.org\/jasa\/article\/143\/6\/3602\/915174\/Deep-learning-models-to-remix-music-for-cochlear"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,6,1]]},"references-count":71,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2018,6,1]]}},"URL":"https:\/\/doi.org\/10.1121\/1.5042056","relation":{},"ISSN":["0001-4966","1520-8524"],"issn-type":[{"value":"0001-4966","type":"print"},{"value":"1520-8524","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2018,6]]},"published":{"date-parts":[[2018,6,1]]}}}