{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,25]],"date-time":"2026-06-25T11:27:29Z","timestamp":1782386849287,"version":"3.54.5"},"reference-count":57,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,12,18]],"date-time":"2024-12-18T00:00:00Z","timestamp":1734480000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:sec><jats:title>Introduction<\/jats:title><jats:p>Musical instrument recognition is a critical component of music information retrieval (MIR), aimed at identifying and classifying instruments from audio recordings. This task poses significant challenges due to the complexity and variability of musical signals.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>In this study, we employed convolutional neural networks (CNNs) to analyze the contributions of various spectrogram representations\u2014STFT, Log-Mel, MFCC, Chroma, Spectral Contrast, and Tonnetz\u2014to the classification of ten different musical instruments. The NSynth database was used for training and evaluation. Visual heatmap analysis and statistical metrics, including Difference Mean, KL Divergence, JS Divergence, and Earth Mover\u2019s Distance, were utilized to assess feature importance and model interpretability.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Our findings highlight the strengths and limitations of each spectrogram type in capturing distinctive features of different instruments. MFCC and Log-Mel spectrograms demonstrated superior performance across most instruments, while others provided insights into specific characteristics.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>This analysis provides some insights into optimizing spectrogram-based approaches for musical instrument recognition, offering guidance for future model development and improving interpretability through statistical and visual analyses.<\/jats:p><\/jats:sec>","DOI":"10.3389\/frai.2024.1499913","type":"journal-article","created":{"date-parts":[[2024,12,18]],"date-time":"2024-12-18T06:49:34Z","timestamp":1734504574000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Interpreting CNN models for musical instrument recognition using multi-spectrogram heatmap analysis: a preliminary study"],"prefix":"10.3389","volume":"7","author":[{"given":"Rujia","family":"Chen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Akbar","family":"Ghobakhlou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ajit","family":"Narayanan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2024,12,18]]},"reference":[{"key":"ref1","author":"Avramidis","year":"2021"},{"key":"ref2","doi-asserted-by":"publisher","first-page":"425","DOI":"10.1007\/s10844-015-0360-9","article-title":"Automatic musical instrument classification using fractional fourier transform based-MFCC features and counter propagation neural network","volume":"46","author":"Bhalke","year":"2016","journal-title":"J. Intell. Inf. Syst."},{"key":"ref3","author":"Bosch","year":"2018"},{"key":"ref4","author":"Castel-Branco","year":"2020"},{"key":"ref5","doi-asserted-by":"publisher","first-page":"977","DOI":"10.1007\/s11265-021-01661-3","article-title":"Puremic: a new audio dataset for the classification of musical instruments based on convolutional neural networks","volume":"93","author":"Castel-Branco","year":"2021","journal-title":"J. Signal Proc. Syst."},{"key":"ref6","doi-asserted-by":"publisher","first-page":"839","DOI":"10.1109\/WACV.2018.00097","article-title":"Grad-cam++: generalized gradient-based visual explanations for deep convolutional networks","volume":"2018","author":"Chattopadhay","year":"2018","journal-title":"IEEE Winter Conf. Appl. Comput. Vision"},{"key":"ref7","author":"Choi","year":"2017"},{"key":"ref8","author":"Chong","year":"2023"},{"key":"ref9","author":"Dieleman","year":"2014"},{"key":"ref10","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1002\/aris.1440370108","article-title":"Music information retrieval","volume":"37","author":"Downie","year":"2003","journal-title":"Annu. Rev. Inf. Sci. Technol."},{"key":"ref11","doi-asserted-by":"publisher","first-page":"1068","DOI":"10.48550\/arXiv.1704.01279","article-title":"Neural audio synthesis of musical notes with wavenet autoencoders","volume":"1","author":"Engel","year":"2017","journal-title":"Int. Conf. Mach. Learn."},{"key":"ref12","author":"Ezzaidi","year":"2012"},{"key":"ref13","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1007\/978-981-13-1951-8_17","article-title":"Stratification of string instruments using Chroma-based features","volume":"755","author":"Ghosal","year":"2019","journal-title":"Emerg. Technol. Data Min. Inf. Secur."},{"key":"ref14","volume-title":"Deep Learning","author":"Goodfellow","year":"2016"},{"key":"ref15","author":"Gururani","year":"2019"},{"key":"ref16","author":"Hall","year":"2014"},{"key":"ref17","author":"Han","year":"2016"},{"key":"ref18","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1215\/00222909-8033433","article-title":"Generic sequences and the generic Tonnetz","volume":"64","author":"Hook","year":"2020","journal-title":"J. Music Theory"},{"key":"ref19","author":"Humphrey","year":"2012"},{"key":"ref20","author":"Humphrey","year":"2018"},{"key":"ref21","doi-asserted-by":"publisher","first-page":"5001","DOI":"10.17762\/ijritcc.v3i7.4778","article-title":"Classification of musical instruments sounds by using MFCC and timbral audio descriptors","volume":"3","author":"Jadhav","year":"2015","journal-title":"Int. J. Recent Innov. Trends Comput. Commun"},{"key":"ref22","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1109\/ICME.2002.1035731","article-title":"Music type classification by spectral contrast feature. Proceedings","volume":"1","author":"Jiang","year":"2002","journal-title":"IEEE Int. Conf. Multimed. Expo"},{"key":"ref23","doi-asserted-by":"publisher","first-page":"174","DOI":"10.1109\/TASL.2008.2007613","article-title":"Temporal integration for audio classification with application to musical instrument classification","volume":"17","author":"Joder","year":"2009","journal-title":"IEEE Trans. Audio Speech Lang. Process."},{"key":"ref24","author":"Karystinaios","year":"2021"},{"key":"ref25","doi-asserted-by":"publisher","first-page":"590","DOI":"10.14419\/ijet.v7i3.34.19388","article-title":"A single predominant instrument recognition of polyphonic music using CNN-based timbre analysis","volume":"7","author":"Kim","year":"2018","journal-title":"Int. J. Eng. Technol."},{"key":"ref26","doi-asserted-by":"publisher","first-page":"5069","DOI":"10.48550\/arXiv.2110.05069","article-title":"Efficient training of audio transformers with patchout","volume":"2021","author":"Koutini","year":"2021","journal-title":"arXiv"},{"key":"ref27","author":"Krakov","year":"2013"},{"key":"ref28","volume-title":"Augmentation methods on monophonic audio for instrument classification in polyphonic music. 2020 28th European signal processing conference (EUSIPCO)","author":"Kratimenos","year":"2021"},{"key":"ref29","doi-asserted-by":"publisher","first-page":"1097","DOI":"10.1145\/3065386","article-title":"Imagenet classification with deep convolutional neural networks","volume":"2012","author":"Krizhevsky","year":"2012","journal-title":"Adv. Neural Inf. Proces. Syst."},{"key":"ref30","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1214\/aoms\/1177729694","article-title":"On information and sufficiency","volume":"22","author":"Kullback","year":"1951","journal-title":"Ann. Math. Stat."},{"key":"ref31","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref32","doi-asserted-by":"publisher","first-page":"541","DOI":"10.1162\/neco.1989.1.4.541","article-title":"Backpropagation applied to handwritten zip code recognition","volume":"1","author":"LeCun","year":"1989","journal-title":"Neural Comput."},{"key":"ref33","doi-asserted-by":"publisher","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref34","first-page":"1096","article-title":"Unsupervised feature learning for audio classification using convolutional deep belief networks","volume":"2009","author":"Lee","year":"2009","journal-title":"Adv. Neural Inf. Proces. Syst."},{"key":"ref35","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1109\/18.61115","article-title":"Divergence measures based on the Shannon entropy","volume":"37","author":"Lin","year":"1991","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref36","author":"Nagawade","year":"2017"},{"key":"ref37","author":"Nair","year":"2010"},{"key":"ref38","author":"Pan","year":"2021"},{"key":"ref39","author":"Pons","year":"2017"},{"key":"ref40","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1109\/JSTSP.2019.2908700","article-title":"Deep learning for audio signal processing","volume":"13","author":"Purwins","year":"2019","journal-title":"IEEE J. Select. Top. Sig. Proc."},{"key":"ref41","doi-asserted-by":"publisher","first-page":"1","DOI":"10.48550\/arXiv.1905.00954","article-title":"Visualizing deep networks by optimizing with integrated gradients","volume":"2","author":"Qi","year":"2019","journal-title":"CVPR Workshops"},{"key":"ref42","author":"Racharla","year":"2020"},{"key":"ref43","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1023\/A:1026543900054","article-title":"The earth Mover\u2019s distance as a metric for image retrieval","volume":"40","author":"Rubner","year":"2000","journal-title":"Int. J. Comput. Vis."},{"key":"ref44","author":"Sattarzadeh","year":"2021"},{"key":"ref45","author":"Schedl","year":"2014"},{"key":"ref46","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1007\/978-3-642-15825-4_10","article-title":"Evaluation of pooling operations in convolutional architectures for object recognition","volume":"2010","author":"Scherer","year":"2010","journal-title":"Int. Conf. Artif. Neural Netw."},{"key":"ref47","author":"Schl\u00fcter","year":"2022"},{"key":"ref48","doi-asserted-by":"publisher","first-page":"2227","DOI":"10.1109\/TASLP.2024.3376984","article-title":"Dynamic convolutional neural networks as efficient pre-trained audio models","volume":"32","author":"Schmid","year":"2024","journal-title":"IEEE\/ACM Trans. Audio Speech Lang Proc"},{"key":"ref49","author":"Sigtia","year":"2016"},{"key":"ref50","doi-asserted-by":"publisher","first-page":"1556","DOI":"10.48550\/arXiv.1409.1556","article-title":"Very deep convolutional networks for large-scale image recognition","volume":"2014","author":"Simonyan","year":"2014","journal-title":"arXiv"},{"key":"ref51","author":"Sundararajan","year":"2017"},{"key":"ref52","doi-asserted-by":"publisher","first-page":"9640","DOI":"10.48550\/arXiv.2006.09640","article-title":"Visual attention for musical instrument recognition","volume":"2020","author":"Watcharasupat","year":"2020","journal-title":"arXiv"},{"key":"ref53","author":"Wei\u00df","year":"2014"},{"key":"ref54","author":"Wu","year":"2017"},{"key":"ref55","doi-asserted-by":"publisher","first-page":"852","DOI":"10.1109\/TASLP.2020.2971419","article-title":"Predominant instrument recognition based on deep neural network with auxiliary classification","volume":"28","author":"Yu","year":"2020","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Proc."},{"key":"ref56","doi-asserted-by":"publisher","first-page":"8596","DOI":"10.48550\/arXiv.2101.08596","article-title":"LEAF: A learnable frontend for audio classification","volume":"2021","author":"Zeghidour","year":"2021","journal-title":"arXiv"},{"key":"ref57","doi-asserted-by":"publisher","first-page":"818","DOI":"10.1007\/978-3-319-10590-1_53","article-title":"Visualizing and understanding convolutional networks","volume":"2014","author":"Zeiler","year":"2014","journal-title":"Eur. Conf. Comput. Vision"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2024.1499913\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,18]],"date-time":"2024-12-18T06:49:39Z","timestamp":1734504579000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2024.1499913\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,18]]},"references-count":57,"alternative-id":["10.3389\/frai.2024.1499913"],"URL":"https:\/\/doi.org\/10.3389\/frai.2024.1499913","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,18]]},"article-number":"1499913"}}