{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T11:27:18Z","timestamp":1778066838028,"version":"3.51.4"},"reference-count":62,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2021,6,8]],"date-time":"2021-06-08T00:00:00Z","timestamp":1623110400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003696","name":"Electronics and Telecommunications Research Institute","doi-asserted-by":"publisher","award":["21ZS1200"],"award-info":[{"award-number":["21ZS1200"]}],"id":[{"id":"10.13039\/501100003696","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IJERPH"],"abstract":"<jats:p>Visual-auditory sensory substitution has demonstrated great potential to help visually impaired and blind groups to recognize objects and to perform basic navigational tasks. However, the high latency between visual information acquisition and auditory transduction may contribute to the lack of the successful adoption of such aid technologies in the blind community; thus far, substitution methods have remained only laboratory-scale research or pilot demonstrations. This high latency for data conversion leads to challenges in perceiving fast-moving objects or rapid environmental changes. To reduce this latency, prior analysis of auditory sensitivity is necessary. However, existing auditory sensitivity analyses are subjective because they were conducted using human behavioral analysis. Therefore, in this study, we propose a cross-modal generative adversarial network-based evaluation method to find an optimal auditory sensitivity to reduce transmission latency in visual-auditory sensory substitution, which is related to the perception of visual information. We further conducted a human-based assessment to evaluate the effectiveness of the proposed model-based analysis in human behavioral experiments. We conducted experiments with three participant groups, including sighted users (SU), congenitally blind (CB) and late-blind (LB) individuals. Experimental results from the proposed model showed that the temporal length of the auditory signal for sensory substitution could be reduced by 50%. This result indicates the possibility of improving the performance of the conventional vOICe method by up to two times. We confirmed that our experimental results are consistent with human assessment through behavioral experiments. Analyzing auditory sensitivity with deep learning models has the potential to improve the efficiency of sensory substitution.<\/jats:p>","DOI":"10.3390\/ijerph18126216","type":"journal-article","created":{"date-parts":[[2021,6,8]],"date-time":"2021-06-08T12:08:55Z","timestamp":1623154135000},"page":"6216","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Analysis and Validation of Cross-Modal Generative Adversarial Network for Sensory Substitution"],"prefix":"10.3390","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4914-0584","authenticated-orcid":false,"given":"Mooseop","family":"Kim","sequence":"first","affiliation":[{"name":"Human Enhancement & Assistive Technology Research Section, Artificial Intelligence Research Lab., Electronics Telecommunications Research Institute (ETRI), Daejeon 34129, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"YunKyung","family":"Park","sequence":"additional","affiliation":[{"name":"Human Enhancement & Assistive Technology Research Section, Artificial Intelligence Research Lab., Electronics Telecommunications Research Institute (ETRI), Daejeon 34129, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7221-8303","authenticated-orcid":false,"given":"KyeongDeok","family":"Moon","sequence":"additional","affiliation":[{"name":"Human Enhancement & Assistive Technology Research Section, Artificial Intelligence Research Lab., Electronics Telecommunications Research Institute (ETRI), Daejeon 34129, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7089-2516","authenticated-orcid":false,"given":"Chi Yoon","family":"Jeong","sequence":"additional","affiliation":[{"name":"Human Enhancement & Assistive Technology Research Section, Artificial Intelligence Research Lab., Electronics Telecommunications Research Institute (ETRI), Daejeon 34129, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,6,8]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1038\/430309a","article-title":"Pitch discrimination in the early blind","volume":"430","author":"Gougoux","year":"2004","journal-title":"Nature"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"e1468","DOI":"10.1002\/wcs.1468","article-title":"Brain (re) organization following visual loss","volume":"10","author":"Voss","year":"2019","journal-title":"Wiley Interdiscip. Rev. Cogn. Sci."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Kolarik, A.J., Pardhan, S., and Moore, B.C. (2021). A framework to account for the effects of visual loss on human auditory abilities. Psychol. Rev.","DOI":"10.1037\/rev0000279"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1868","DOI":"10.1016\/S0028-3932(02)00071-4","article-title":"Vertical sound localization in blind humans","volume":"40","author":"Lewald","year":"2002","journal-title":"Neuropsychologia"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1414","DOI":"10.1177\/1535370214538740","article-title":"The blind brain: How (lack of) vision shapes the morphological and functional architecture of the human brain","volume":"239","author":"Ricciardi","year":"2014","journal-title":"Exp. Biol. Med."},{"key":"ref_6","first-page":"29","article-title":"Other ways of seeing: From behavior to neural mechanisms in the online \u201cvisual\u201d control of action with sensory substitution","volume":"34","author":"Proulx","year":"2016","journal-title":"Restor. Neurol. Neurosci."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"103072","DOI":"10.1016\/j.apergo.2020.103072","article-title":"Visual-to-auditory sensory substitution alters language asymmetry in both sighted novices and experienced visually impaired users","volume":"85","author":"Proulx","year":"2020","journal-title":"Appl. Ergon."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1109\/10.121642","article-title":"An experimental system for auditory image representations","volume":"39","author":"Meijer","year":"1992","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2427","DOI":"10.1098\/rspb.1999.0942","article-title":"The perception of visual images encoded in musical form: A study in cross-modality information transfer","volume":"266","author":"Persaud","year":"1999","journal-title":"Proc. R. Soc. London. Ser. Biol. Sci."},{"key":"ref_10","first-page":"247","article-title":"EyeMusic: Introducing a \u201cvisual\u201d colorful experience for the blind using auditory sensory substitution","volume":"32","author":"Abboud","year":"2014","journal-title":"Restor. Neurol. Neurosci."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1177\/0145482X0509900404","article-title":"Personal guidance system for people with visual impairment: A comparison of spatial displays for route guidance","volume":"99","author":"Loomis","year":"2005","journal-title":"J. Vis. Impair. Blind."},{"key":"ref_12","first-page":"269","article-title":"The Vibe: A versatile vision-to-audition sensory substitution device","volume":"7","author":"Hanneton","year":"2010","journal-title":"Appl. Bionics Biomech."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"857","DOI":"10.1068\/p5219","article-title":"The Ponzo illusion with auditory substitution of vision in sighted and early-blind subjects","volume":"34","author":"Renier","year":"2005","journal-title":"Perception"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"416","DOI":"10.1068\/p5631","article-title":"Learning to perceive with a visuo\u2014Auditory substitution system: Localisation and object recognition with \u2018The Voice\u2019","volume":"36","author":"Auvray","year":"2007","journal-title":"Perception"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"498","DOI":"10.1162\/jocn_a_01210","article-title":"Hearing Shapes: Event-related Potentials Reveal the Time Course of Auditory\u2013Visual Sensory Substitution","volume":"30","author":"Graulty","year":"2018","journal-title":"J. Cogn. Neurosci."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-019-50912-8","article-title":"Cross-modal size-contrast illusion: Acoustic increases in intensity and bandwidth modulate haptic representation of object size","volume":"9","author":"Uesaki","year":"2019","journal-title":"Sci. Rep."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Spagnol, S., Baldan, S., and Unnthorsson, R. (2017, January 16\u201318). Auditory depth map representations with a sensory substitution scheme based on synthetic fluid sounds. Proceedings of the 2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP), Luton, UK.","DOI":"10.1109\/MMSP.2017.8122220"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Hamilton-Fletcher, G., Alvarez, J., Obrist, M., and Ward, J. (2019). SoundSight: A Mobile Sensory Substitution Device that Sonifies Colour, Distance, and Temperature. PsyArXiv, 1\u201320.","DOI":"10.31234\/osf.io\/xsjwr"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Neugebauer, A., Rifai, K., Getzlaff, M., and Wahl, S. (2020). Navigation aid for blind persons by visual-to-auditory sensory substitution: A pilot study. PLoS ONE, 15.","DOI":"10.1371\/journal.pone.0237344"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1443","DOI":"10.3389\/fpsyg.2020.01443","article-title":"Efficiency of sensory substitution devices alone and in combination with self-motion for spatial navigation in sighted and visually impaired","volume":"11","author":"Jicol","year":"2020","journal-title":"Front. Psychol."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Stoll, C., Palluel-Germain, R., Fristot, V., Pellerin, D., Alleysson, D., and Graff, C. (2015). Navigating from a depth image converted into sound. Appl. Bionics Biomech., 2015.","DOI":"10.1155\/2015\/543492"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.neubiorev.2013.11.007","article-title":"Sensory substitution: Closing the gap between basic research and widespread practical visual rehabilitation","volume":"41","author":"Maidenbaum","year":"2014","journal-title":"Neurosci. Biobehav. Rev."},{"key":"ref_23","first-page":"769","article-title":"Designing sensory-substitution devices: Principles, pitfalls and potential 1","volume":"34","author":"Moldoveanu","year":"2016","journal-title":"Restor. Neurol. Neurosci."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"240","DOI":"10.1016\/j.neubiorev.2016.08.019","article-title":"From science to technology: Orientation and mobility in blind children and adults","volume":"71","author":"Cuturi","year":"2016","journal-title":"Neurosci. Biobehav. Rev."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"492","DOI":"10.1016\/j.concog.2009.10.006","article-title":"Visual experiences in the blind induced by an auditory sensory substitution device","volume":"19","author":"Ward","year":"2010","journal-title":"Conscious. Cogn."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Hu, D., Wang, D., Li, X., Nie, F., and Wang, Q. (2019, January 16\u201320). Listen to the Image. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00816"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Sudol, J., Dialameh, O., Blanchard, C., and Dorcey, T. (2010, January 13\u201318). Looktel\u2014A comprehensive platform for computer-aided visual assistance. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA.","DOI":"10.1109\/CVPRW.2010.5543725"},{"key":"ref_28","unstructured":"Rao, V. (2021, April 16). Seeing AI: Microsoft Project for Blind People That Lets Them \u201cSee\u201d. Available online: https:\/\/assistivetechnologyblog.com\/2016\/03\/seeing-ai-microsoft-project-for-blind.html."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Tapu, R., Mocanu, B., and Zaharia, T. (2017). DEEP-SEE: Joint object detection, tracking and recognition with application to visually impaired navigational assistance. Sensors, 17.","DOI":"10.3390\/s17112473"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1007\/s13748-019-00203-0","article-title":"Convolutional neural network: A review of models, methodologies and applications to object detection","volume":"9","author":"Dhillon","year":"2020","journal-title":"Prog. Artif. Intell."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1016\/j.aei.2019.02.009","article-title":"Convolutional neural network: Deep learning-based classification of building quality problems","volume":"40","author":"Zhong","year":"2019","journal-title":"Adv. Eng. Inform."},{"key":"ref_32","unstructured":"V\u00e4ljam\u00e4e, A., and Kleiner, M. (2006, January 20\u201323). Spatial sound in auditory vision substitution systems. Proceedings of the Audio Engineering Society Convention 120, Paris, France. Paper 6795."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1279","DOI":"10.1109\/10.720206","article-title":"A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution","volume":"45","author":"Capelle","year":"1998","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_34","unstructured":"Durette, B., Louveton, N., Alleysson, D., and H\u00e9rault, J. (2008, January 12\u201318). Visuo-auditory sensory substitution for mobility assistance: Testing TheVIBE. Proceedings of the Workshop on Computer Vision Applications for the Visually Impaired, Marseille, France."},{"key":"ref_35","unstructured":"T\u00f3th, V., and Parkkonen, L. (2019). Autoencoding sensory substitution. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Leglaive, S., Alameda-Pineda, X., Girin, L., and Horaud, R. (2020, January 4\u20138). A recurrent variational autoencoder for speech enhancement. Proceedings of the ICASSP 2020\u20132020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.","DOI":"10.1109\/ICASSP40776.2020.9053164"},{"key":"ref_37","unstructured":"Gregor, K., Danihelka, I., Graves, A., Rezende, D., and Wierstra, D. (2015, January 6\u201311). Draw: A recurrent neural network for image generation. Proceedings of the International Conference on Machine Learning, Lille, France. PMLR 37:1462\u20131471."},{"key":"ref_38","unstructured":"Lee, S.B., Jeong, C.Y., Moon, K., and Kim, C.K. (2019, January 15\u201316). A study on applicability of Autoencoder-based sensory substitution method using dynamic time warping. Proceedings of the Annual Fall Conference of Korean Multimedia Society, Seoul, Korea."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"640","DOI":"10.1016\/j.neuron.2012.08.026","article-title":"Reading with sounds: Sensory substitution selectively activates the visual word form area in the blind","volume":"76","author":"Cohen","year":"2012","journal-title":"Neuron"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Chan, K.C., Murphy, M.C., Bang, J.W., Sims, J., Kashkoush, J., and Nau, A.C. (2018, January 18\u201321). Functional MRI of sensory substitution in the blind. Proceedings of the 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, USA.","DOI":"10.1109\/EMBC.2018.8513622"},{"key":"ref_41","unstructured":"Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., and Ng, A.Y. (July, January 28). Multimodal Deep Learning. Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML\u201911, Bellevue, WA, USA."},{"key":"ref_42","unstructured":"Owens, A., Isola, P., McDermott, J., Torralba, A., Adelson, E.H., and Freeman, W.T. (July, January 26). Visually indicated sounds. Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA."},{"key":"ref_43","unstructured":"Goodfellow, I. (2016). Nips 2016 tutorial: Generative adversarial networks. arXiv."},{"key":"ref_44","unstructured":"Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., and Aila, T. (2020). Training generative adversarial networks with limited data. arXiv."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Chen, L., Srivastava, S., Duan, Z., and Xu, C. (2017, January 23\u201317). Deep cross-modal audio-visual generation. Proceedings of the on Thematic Workshops of ACM Multimedia, Mountain View, CA, USA.","DOI":"10.1145\/3126686.3126723"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Hao, W., Zhang, Z., and Guan, H. (2018, January 2\u20137). Cmcgan: A uniform framework for cross-modal visual-audio mutual generation. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.12329"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Baldominos, A., Saez, Y., and Isasi, P. (2019). A survey of handwritten character recognition with mnist and emnist. Appl. Sci., 9.","DOI":"10.3390\/app9153169"},{"key":"ref_48","unstructured":"Odena, A., Olah, C., and Shlens, J. (2017, January 6\u201311). Conditional image synthesis with auxiliary classifier gans. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, Australia."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1007\/s11045-020-00731-2","article-title":"Label-preserving data augmentation for mobile sensor data","volume":"32","author":"Kim","year":"2020","journal-title":"Multidimens. Syst. Signal Process."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"575","DOI":"10.1016\/S0022-5371(75)80045-4","article-title":"Word length and the structure of short-term memory","volume":"14","author":"Baddeley","year":"1975","journal-title":"J. Verbal Learn. Verbal Behav."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Moolayil, J., Moolayil, J., and John, S. (2019). Learn Keras for Deep Neural Networks, Springer.","DOI":"10.1007\/978-1-4842-4240-7"},{"key":"ref_52","unstructured":"Gulli, A., and Pal, S. (2017). Deep Learning with Keras, Packt Publishing Ltd."},{"key":"ref_53","unstructured":"Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. (2016). Improved techniques for training gans. arXiv."},{"key":"ref_54","unstructured":"Barratt, S., and Sharma, R. (2018). A note on the inception score. arXiv."},{"key":"ref_55","unstructured":"Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017). Gans trained by a two time-scale update rule converge to a local nash equilibrium. arXiv."},{"key":"ref_56","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (July, January 26). Rethinking the inception architecture for computer vision. Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"573","DOI":"10.1016\/j.neuroimage.2005.01.047","article-title":"Cross-modal activation of visual cortex during depth perception using auditory substitution of vision","volume":"26","author":"Renier","year":"2005","journal-title":"NeuroImage"},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Kolarik, A.J., Scarfe, A.C., Moore, B.C., and Pardhan, S. (2016). Echoic sensory substitution information in a single obstacle circumvention task. PLoS ONE, 11.","DOI":"10.1371\/journal.pone.0160872"},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Kolarik, A.J., Scarfe, A.C., Moore, B.C., and Pardhan, S. (2017). Blindness enhances auditory obstacle circumvention: Assessing echolocation, sensory substitution, and visual-based navigation. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0175750"},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Martinez-Garcia, M., Kalawsky, R.S., Gordon, T., Smith, T., Meng, Q., and Flemisch, F. (2020). Communication and interaction with semiautonomous ground vehicles by force control steering. IEEE Trans. Cybern.","DOI":"10.1109\/TCYB.2020.3020217"},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Mart\u00ednez-Garc\u00eda, M., and Gordon, T. (2018, January 7\u201310). A new model of human steering using far-point error perception and multiplicative control. Proceedings of the 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Miyazaki, Japan.","DOI":"10.1109\/SMC.2018.00218"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1016\/j.ijhcs.2019.05.008","article-title":"Human augmentation: Past, present and future","volume":"131","author":"Raisamo","year":"2019","journal-title":"Int. J. Hum. Comput. Stud."}],"container-title":["International Journal of Environmental Research and Public Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1660-4601\/18\/12\/6216\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:12:09Z","timestamp":1760163129000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1660-4601\/18\/12\/6216"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,8]]},"references-count":62,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2021,6]]}},"alternative-id":["ijerph18126216"],"URL":"https:\/\/doi.org\/10.3390\/ijerph18126216","relation":{},"ISSN":["1660-4601"],"issn-type":[{"value":"1660-4601","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,6,8]]}}}