{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,15]],"date-time":"2025-12-15T15:50:08Z","timestamp":1765813808615,"version":"3.48.0"},"reference-count":40,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2025,12,15]],"date-time":"2025-12-15T00:00:00Z","timestamp":1765756800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>This paper introduces a novel framework for sensory representation of brain imaging data, combining deep learning-based segmentation with multimodal visual and auditory outputs. Structural magnetic resonance imaging (MRI) predictions are converted into color-coded maps and stereophonic\/MIDI sonifications, enabling intuitive interpretation of cortical activation patterns. High-precision U-Net models efficiently generate these outputs, supporting clinical decision-making, cognitive research, and creative applications. Spatial, intensity, and anomalous features are encoded into perceivable visual and auditory cues, facilitating early detection and introducing the concept of \u201cauditory biomarkers\u201d for potential pathological identification. Despite current limitations, including dataset size, absence of clinical validation, and heuristic-based sonification, the pipeline demonstrates technical feasibility and robustness. Future work will focus on clinical user studies, the application of functional MRI (fMRI) time-series for dynamic sonification, and the integration of real-time emotional feedback in cinematic contexts. This multisensory approach offers a promising avenue for enhancing the interpretability of complex neuroimaging data across medical, research, and artistic domains.<\/jats:p>","DOI":"10.3390\/jimaging11120449","type":"journal-article","created":{"date-parts":[[2025,12,15]],"date-time":"2025-12-15T15:15:08Z","timestamp":1765811708000},"page":"449","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Sensory Representation of Neural Networks Using Sound and Color for Medical Imaging Segmentation"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-9681-1958","authenticated-orcid":false,"given":"Irenel Lopo Da","family":"Silva","sequence":"first","affiliation":[{"name":"IT Department, Computer Engineering, School of Engineering, University of Minho, 4704-553 Braga, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5895-0880","authenticated-orcid":false,"given":"Nicolas Francisco","family":"Lori","sequence":"additional","affiliation":[{"name":"Algoritmi\/LASI, University of Minho, 4704-553 Braga, Portugal"},{"name":"Faculty of Sciences and Technology, University of Azores, 9500-321 Ponta Delgada, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4121-6169","authenticated-orcid":false,"given":"Jos\u00e9 Manuel Ferreira","family":"Machado","sequence":"additional","affiliation":[{"name":"Algoritmi\/LASI, University of Minho, 4704-553 Braga, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2025,12,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1007\/s10548-009-0132-3","article-title":"Multimodal integration of fMRI and EEG data for high spatial and temporal resolution analysis of brain networks","volume":"23","author":"Mantini","year":"2010","journal-title":"Brain Topogr."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1016\/j.zemedi.2018.11.002","article-title":"An overview of deep learning in medical imaging focusing on MRI","volume":"29","author":"Lundervold","year":"2019","journal-title":"Z. Med. Phys."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"6322","DOI":"10.1007\/s00330-023-09897-2","article-title":"Diagnostic radiology and its future: What do clinicians need","volume":"33","author":"Kwee","year":"2023","journal-title":"Eur. Radiol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1029","DOI":"10.1111\/1754-9485.13388","article-title":"Non-radiologist perception of the use of artificial intelligence (AI) in diagnostic medical imaging reports","volume":"66","author":"Lim","year":"2022","journal-title":"J. Med. Imaging Radiat. Oncol."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Chen, J., Ma, J., and Huai, X. (2025). Qualitative studies: Designing a multimodal medical visualization tool for helping patients interpret 3D medical images. Front. Physiol., 16.","DOI":"10.3389\/fphys.2025.1559801"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Sch\u00fctz, L., Matinfar, S., Schafroth, G., Navab, N., Fairhurst, M., Wagner, A., Wiestler, B., Eck, U., and Navab, N. (2024). A Framework for Multimodal Medical Image Interaction. arXiv.","DOI":"10.1109\/TVCG.2024.3456163"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.inffus.2022.12.010","article-title":"Diagnosis of brain diseases in fusion of neuroimaging modalities using deep learning: A review","volume":"93","author":"Shoeibi","year":"2023","journal-title":"Inf. Fus."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Xu, M., Zhang, J., Li, W., Chen, Y., Zhou, H., Wang, P., and Zhang, L. (2023). Deep learning aided neuroimaging and brain regulation. Sensors, 23.","DOI":"10.3390\/s23114993"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2500","DOI":"10.1038\/s41593-024-01787-0","article-title":"Integrating brainstem and cortical functional architectures","volume":"27","author":"Hansen","year":"2024","journal-title":"Nat. Neurosci."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"kvac013","DOI":"10.1093\/oons\/kvac013","article-title":"Dampened sensory representations for expected input across the ventral visual stream","volume":"1","author":"Richter","year":"2022","journal-title":"Oxf. Open Neurosci."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"3908","DOI":"10.1093\/cercor\/bhx255","article-title":"Auditory frequency representations in human somatosensory cortex","volume":"28","author":"Barnes","year":"2018","journal-title":"Cereb. Cortex"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"116860","DOI":"10.1016\/j.neuroimage.2020.116860","article-title":"Towards clinical applications of movie fMRI","volume":"220","author":"Eickhoff","year":"2020","journal-title":"NeuroImage"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Schaefer, S., and Rotte, M. (2024). How movies move us: Movie preferences are linked to differences in brain reactivity. Front. Behav. Neurosci., 18.","DOI":"10.3389\/fnbeh.2024.1396811"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"7267","DOI":"10.1038\/s41598-023-33734-7","article-title":"Sentiment analysis of fMRI using automatically generated stimuli labels under naturalistic paradigm","volume":"13","author":"Mahrukh","year":"2023","journal-title":"Sci. Rep."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Saarim\u00e4ki, H. (2021). Naturalistic stimuli in affective neuroimaging: A review. Front. Hum. Neurosci., 15.","DOI":"10.3389\/fnhum.2021.675068"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"102628","DOI":"10.1016\/j.conb.2022.102628","article-title":"Reshaping sensory representations by task-specific brain states: Toward cortical circuit mechanisms","volume":"77","author":"Zhang","year":"2022","journal-title":"Curr. Opin. Neurobiol."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"264","DOI":"10.1016\/0743-7315(89)90062-2","article-title":"Entropy driven artificial neuronal networks and sensorial representation: A proposal","volume":"6","author":"Orban","year":"1989","journal-title":"J. Parallel Distrib. Comput."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1076\/jnmr.31.3.221.14190","article-title":"Performance and interpretation","volume":"31","author":"Mazzola","year":"2002","journal-title":"J. New Music Res."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1016\/0022-510X(96)00055-X","article-title":"The cortical representation of somatosensory evoked potentials of the phrenic nerve","volume":"139","author":"Zifko","year":"1996","journal-title":"J. Neurol. Sci."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"45","DOI":"10.3171\/jns.2000.92.1.0045","article-title":"Somatosensory representation in patients who have undergone hemispherectomy: A functional magnetic resonance imaging study","volume":"92","author":"Bittar","year":"2000","journal-title":"J. Neurosurg."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"e1376","DOI":"10.7717\/peerj-cs.1376","article-title":"Decoding of the neural representation of the visual RGB color model","volume":"9","author":"Wu","year":"2023","journal-title":"PeerJ Comput. Sci."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Tamakoshi, S., Minoura, N., Katayama, J., and Yagi, A. (2016). Entire sound representations are time-compressed in sensory memory: Evidence from MMN. Front. Neurosci., 10.","DOI":"10.3389\/fnins.2016.00347"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"12684","DOI":"10.1523\/JNEUROSCI.2713-07.2007","article-title":"Hearing illusory sounds in noise: Sensory-perceptual transformations in primary auditory cortex","volume":"27","author":"Riecke","year":"2007","journal-title":"J. Neurosci."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Melchiorre, A.B., and Schedl, M. (2020). Personality correlates of music audio preferences for modelling music listeners. Front. Psychol., 11.","DOI":"10.1145\/3340631.3394874"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"37622","DOI":"10.1109\/ACCESS.2021.3062484","article-title":"Machine learning and deep learning approaches for brain disease diagnosis: Principles and recent advances","volume":"9","author":"Khan","year":"2021","journal-title":"IEEE Access"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Samee, N.A., Mahmoud, N.F., Atteia, G., Abdallah, H.A., Alabdulhafith, M., Al-Gaashani, M.S.A.M., Ahmad, S., and Muthanna, M.S.A. (2022). Classification Framework for Medical Diagnosis of Brain Tumor with an Effective Hybrid Transfer Learning Model. Diagnostics, 12.","DOI":"10.3390\/diagnostics12102541"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Dudai, Y. (2012). The cinema-cognition dialogue: A match made in brain. Front. Hum. Neurosci., 6.","DOI":"10.3389\/fnhum.2012.00248"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"100600","DOI":"10.1016\/j.dcn.2018.10.004","article-title":"Movies in the magnet: Naturalistic paradigms in developmental functional neuroimaging","volume":"36","author":"Vanderwal","year":"2019","journal-title":"Dev. Cogn. Neurosci."},{"key":"ref_29","first-page":"119941","article-title":"Functional connectivity profiles of the default mode and visual networks reflect temporal accumulative effects of sustained naturalistic emotional experience","volume":"273","author":"Zhang","year":"2023","journal-title":"NeuroImage"},{"key":"ref_30","unstructured":"Roginska, A., Mohanraj, H., Ballora, M., and Friedman, K. (2013, January 9\u201311). Immersive Sonification for Displaying Brain Scan Data. Proceedings of the International Conference on Health Informatics (HEALTHINF-2013), SCITEPRESS, Philadelphia, PA, USA. Available online: https:\/\/www.scitepress.org\/Papers\/2013\/42029\/42029.pdf."},{"key":"ref_31","unstructured":"Cadiz, R.F., de la Cuadra, P., Montoya, A., Mar\u00edn, V., Andia, M.E., Tejos, C., and Irarrazaval, P. (October, January 25). Sonification of Medical Images Based on Statistical Descriptors. Proceedings of the International Computer Music Conference (ICMC), Denton, TX, USA. Available online: https:\/\/quod.lib.umich.edu\/cgi\/p\/pod\/dod-idx\/sonification-of-medical-images-based-on-statistical.pdf."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"17711","DOI":"10.1038\/s41598-019-54080-7","article-title":"Using the Sonification for Hardly Detectable Details in Medical Images","volume":"9","author":"Chiroiu","year":"2019","journal-title":"Sci. Rep."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"103571","DOI":"10.1016\/j.media.2025.103571","article-title":"From Tissue to Sound: A New Paradigm for Medical Sonic Interaction Design","volume":"103","author":"Matinfar","year":"2025","journal-title":"Med. Image Anal."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1045","DOI":"10.1007\/s10278-013-9622-7","article-title":"The Cancer Imaging Archive (TCIA): Maintaining and operating a public information repository","volume":"26","author":"Clark","year":"2013","journal-title":"J. Digit. Imaging"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Zakharova, G., Efimov, V., Raevskiy, M., Rumiantsev, P., Gudkov, A., Belogurova-Ovchinnikova, O., Sorokin, M., and Buzdin, A. (2023). Reclassification of TCGA diffuse glioma profiles linked to transcriptomic, epigenetic, genomic and clinical data, according to the 2021 WHO CNS tumor classification. Int. J. Mol. Sci., 24.","DOI":"10.3390\/ijms24010157"},{"key":"ref_36","first-page":"154","article-title":"Automatic brain MRI tumour segmentation based on deep fusion of feature maps","volume":"588","author":"Li","year":"2025","journal-title":"Artif. Intell. Rev."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Avazov, K., Mirzakhalilov, S., Umirzakova, S., Abdusalomov, A., and Cho, Y.I. (2024). Dynamic Focus on Tumor Boundaries: A Lightweight U-Net for MRI Brain Tumor Segmentation. Bioengineering, 11.","DOI":"10.3390\/bioengineering11121302"},{"key":"ref_38","first-page":"3349","article-title":"mResU-Net: Multi-scale residual U-Net-based brain tumour segmentation","volume":"27","author":"Liu","year":"2023","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_39","unstructured":"da Silva, I.L., Lori, N.F., and Machado, J.M.F. (2025, October 05). Advanced Brain Sonification Documentation (Version 1). Available online: https:\/\/doi.org\/10.5281\/zenodo.17116504."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1469","DOI":"10.1038\/s41567-025-02975-w","article-title":"A mechanical quantum memory for microwave photons","volume":"21","author":"Bozkurt","year":"2025","journal-title":"Nat. Phys."}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/11\/12\/449\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,15]],"date-time":"2025-12-15T15:46:38Z","timestamp":1765813598000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/11\/12\/449"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,15]]},"references-count":40,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["jimaging11120449"],"URL":"https:\/\/doi.org\/10.3390\/jimaging11120449","relation":{},"ISSN":["2313-433X"],"issn-type":[{"value":"2313-433X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,15]]}}}