{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,24]],"date-time":"2026-01-24T02:01:57Z","timestamp":1769220117517,"version":"3.49.0"},"reference-count":19,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2025,2,16]],"date-time":"2025-02-16T00:00:00Z","timestamp":1739664000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"MICIU\/AEI\/10.13039\/50100011033","award":["CPP2021-008491"],"award-info":[{"award-number":["CPP2021-008491"]}]},{"name":"European Union NextGenerationEU\/PRTR","award":["CPP2021-008491"],"award-info":[{"award-number":["CPP2021-008491"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>This paper presents a novel approach to audio classification leveraging the latent representation generated by Meta\u2019s EnCodec neural audio codec. We hypothesize that the compressed latent space representation captures essential audio features more suitable for classification tasks than the traditional spectrogram-based approaches. We train a vanilla convolutional neural network for music genre, speech\/music, and environmental sound classification using EnCodec\u2019s encoder output as input to validate this. Then, we compare its performance training with the same network using a spectrogram-based representation as input. Our experiments demonstrate that this approach achieves comparable accuracy to state-of-the-art methods while exhibiting significantly faster convergence and reduced computational load during training. These findings suggest the potential of EnCodec\u2019s latent representation for efficient, faster, and less expensive audio classification applications. We analyze the characteristics of EnCodec\u2019s output and compare its performance against traditional spectrogram-based approaches, providing insights into this novel approach\u2019s advantages.<\/jats:p>","DOI":"10.3390\/a18020108","type":"journal-article","created":{"date-parts":[[2025,2,17]],"date-time":"2025-02-17T03:41:47Z","timestamp":1739763707000},"page":"108","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Beyond Spectrograms: Rethinking Audio Classification from EnCodec\u2019s Latent Space"],"prefix":"10.3390","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8708-4734","authenticated-orcid":false,"given":"Jorge","family":"Perianez-Pascual","sequence":"first","affiliation":[{"name":"Instituto de Investigaci\u00f3n en Tecnolog\u00edas Inform\u00e1ticas Aplicadas (INTIA), Universidad de Extremadura, Av. Universidad s\/n, 10003 C\u00e1ceres, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1024-6202","authenticated-orcid":false,"given":"Juan D.","family":"Guti\u00e9rrez","sequence":"additional","affiliation":[{"name":"Department of Electronics and Computer Science, Universidad de Santiago de Compostela, R\u00faa Benigno Ledo, 27002 Lugo, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-1499-1383","authenticated-orcid":false,"given":"Laura","family":"Escobar-Encinas","sequence":"additional","affiliation":[{"name":"Department of Computers and Telematics Systems Engineering, Universidad de Extremadura, Escuela Polit\u00e9cnica, 10003 C\u00e1ceres, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2999-4304","authenticated-orcid":false,"given":"\u00c1lvaro","family":"Rubio-Largo","sequence":"additional","affiliation":[{"name":"Department of Computers and Telematics Systems Engineering, Universidad de Extremadura, Escuela Polit\u00e9cnica, 10003 C\u00e1ceres, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6545-0913","authenticated-orcid":false,"given":"Roberto","family":"Rodriguez-Echeverria","sequence":"additional","affiliation":[{"name":"Instituto de Investigaci\u00f3n en Tecnolog\u00edas Inform\u00e1ticas Aplicadas (INTIA), Universidad de Extremadura, Av. Universidad s\/n, 10003 C\u00e1ceres, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2025,2,16]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1043","DOI":"10.1109\/TCE.2006.1706505","article-title":"Genesis of the MP3 Audio Coding Standard","volume":"52","author":"Musmann","year":"2006","journal-title":"IEEE Trans. Consum. Electron."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Giannakopoulos, T. (2015). pyAudioAnalysis: An Open-Source Python Library for Audio Signal Analysis. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0144610"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Bogdanov, D., Wack, N., Gomez, E., Gulati, S., Herrera, P., Mayor, O., Roma, G., Salamon, J., Zapata, J., and Serra, X. (2013, January 4\u20138). Essentia: An Audio Analysis Library for Music Information Retrieval. Proceedings of the 14th Conference of the International Society for Music Information Retrieval (ISMIR), Curitiba, PR, Brazil.","DOI":"10.1145\/2502081.2502229"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"McFee, B., Raffel, C., Liang, D., Ellis, D., McVicar, M., Battenberg, E., and Nieto, O. (2015, January 6\u201312). Librosa: Audio and Music Signal Analysis in Python. Proceedings of the Python in Science Conference, Austin, TX, USA.","DOI":"10.25080\/Majora-7b98e3ed-003"},{"key":"ref_5","unstructured":"D\u00e9fossez, A., Copet, J., Synnaeve, G., and Adi, Y. (2022). High Fidelity Neural Audio Compression. arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. (2021). SoundStream: An End-to-End Neural Audio Codec. arXiv.","DOI":"10.1109\/TASLP.2021.3129994"},{"key":"ref_7","unstructured":"Langman, R., Juki\u0107, A., Dhawan, K., Koluguri, N.R., and Ginsburg, B. (2024). Spectral Codecs: Spectrogram-Based Audio Codecs for High Quality Speech Synthesis. arXiv."},{"key":"ref_8","unstructured":"Kong, J., Kim, J., and Bae, J. (2020). HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. arXiv."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Wu, Y.C., Gebru, I.D., Markovi\u0107, D., and Richard, A. (2023, January 4\u201310). Audiodec: An Open-Source Streaming High-Fidelity Neural Audio Codec. Proceedings of the ICASSP 2023\u20142023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.","DOI":"10.1109\/ICASSP49357.2023.10096509"},{"key":"ref_10","unstructured":"Siuzdak, H., Gr\u00f6tschla, F., and Lanzend\u00f6rfer, L.A. (2024). SNAC: Multi-Scale Neural Audio Codec. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Tagliasacchi, M., Li, Y., Misiunas, K., and Roblek, D. (2020, January 25\u201329). SEANet: A Multi-Modal Speech Enhancement Network. Proceedings of the Interspeech 2020, Shanghai, China.","DOI":"10.21437\/Interspeech.2020-1563"},{"key":"ref_12","unstructured":"Choi, K., Joo, D., and Kim, J. (2017). Kapre: On-GPU Audio Preprocessing Layers for a Quick Implementation of Deep Neural Network Models with Keras. arXiv."},{"key":"ref_13","unstructured":"Wolf-Monheim, F. (2024). Spectral and Rhythm Features for Audio Classification with Deep Convolutional Neural Networks. arXiv."},{"key":"ref_14","unstructured":"Tzanetakis, G. (2025, February 01). GTZAN Music\/Speech Dataset. Available online: https:\/\/web.archive.org\/web\/20220330025414\/http:\/\/marsyas.info\/downloads\/datasets.html#music-speech."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1109\/TSA.2002.800560","article-title":"Musical Genre Classification of Audio Signals","volume":"10","author":"Tzanetakis","year":"2002","journal-title":"IEEE Trans. Speech Audio Process."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Piczak, K.J. (2015, January 26\u201330). ESC: Dataset for Environmental Sound Classification. Proceedings of the 23rd ACM International Conference on Multimedia, Brisbane, Australia.","DOI":"10.1145\/2733373.2806390"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"2880","DOI":"10.1109\/TASLP.2020.3030497","article-title":"PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition","volume":"28","author":"Kong","year":"2020","journal-title":"IEEE\/ACM Trans. Audio, Speech, Lang. Process."},{"key":"ref_18","unstructured":"Niizumi, D., Takeuchi, D., Ohishi, Y., Harada, N., and Kashino, K. (2022). Masked Spectrogram Modeling Using Masked Autoencoders for Learning General-purpose Audio Representation. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Koutini, K., Schl\u00fcter, J., Eghbal-zadeh, H., and Widmer, G. (2022, January 18\u201322). Efficient Training of Audio Transformers with Patchout. Proceedings of the Interspeech 2022, Incheon, Republic of Korea.","DOI":"10.21437\/Interspeech.2022-227"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/18\/2\/108\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T16:35:33Z","timestamp":1760027733000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/18\/2\/108"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,16]]},"references-count":19,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2025,2]]}},"alternative-id":["a18020108"],"URL":"https:\/\/doi.org\/10.3390\/a18020108","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,16]]}}}