{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,10]],"date-time":"2026-02-10T17:42:52Z","timestamp":1770745372749,"version":"3.49.0"},"reference-count":38,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2020,6,5]],"date-time":"2020-06-05T00:00:00Z","timestamp":1591315200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Zhejiang Provincial Public Fund","award":["2016C33136"],"award-info":[{"award-number":["2016C33136"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In recent years, with the development of depth cameras and scene detection algorithms, a wide variety of electronic travel aids for visually impaired people have been proposed. However, it is still challenging to convey scene information to visually impaired people efficiently. In this paper, we propose three different auditory-based interaction methods, i.e., depth image sonification, obstacle sonification as well as path sonification, which convey raw depth images, obstacle information and path information respectively to visually impaired people. Three sonification methods are compared comprehensively through a field experiment attended by twelve visually impaired participants. The results show that the sonification of high-level scene information, such as the direction of pathway, is easier to learn and adapt, and is more suitable for point-to-point navigation. In contrast, through the sonification of low-level scene information, such as raw depth images, visually impaired people can understand the surrounding environment more comprehensively. Furthermore, there is no interaction method that is best suited for all participants in the experiment, and visually impaired individuals need a period of time to find the most suitable interaction method. Our findings highlight the features and the differences of three scene detection algorithms and the corresponding sonification methods. The results provide insights into the design of electronic travel aids, and the conclusions can also be applied in other fields, such as the sound feedback of virtual reality applications.<\/jats:p>","DOI":"10.3390\/s20113222","type":"journal-article","created":{"date-parts":[[2020,6,9]],"date-time":"2020-06-09T05:16:14Z","timestamp":1591679774000},"page":"3222","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":22,"title":["A Comparative Study in Real-Time Scene Sonification for Visually Impaired People"],"prefix":"10.3390","volume":"20","author":[{"given":"Weijian","family":"Hu","sequence":"first","affiliation":[{"name":"National Engineering Research Center of Optical Instrumentation, Zhejiang University, Hangzhou 310058, China"}]},{"given":"Kaiwei","family":"Wang","sequence":"additional","affiliation":[{"name":"National Engineering Research Center of Optical Instrumentation, Zhejiang University, Hangzhou 310058, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1090-667X","authenticated-orcid":false,"given":"Kailun","family":"Yang","sequence":"additional","affiliation":[{"name":"Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7951-196X","authenticated-orcid":false,"given":"Ruiqi","family":"Cheng","sequence":"additional","affiliation":[{"name":"National Engineering Research Center of Optical Instrumentation, Zhejiang University, Hangzhou 310058, China"}]},{"given":"Yaozu","family":"Ye","sequence":"additional","affiliation":[{"name":"National Engineering Research Center of Optical Instrumentation, Zhejiang University, Hangzhou 310058, China"}]},{"given":"Lei","family":"Sun","sequence":"additional","affiliation":[{"name":"National Engineering Research Center of Optical Instrumentation, Zhejiang University, Hangzhou 310058, China"}]},{"given":"Zhijie","family":"Xu","sequence":"additional","affiliation":[{"name":"School of Computing and Engineering, University of Huddersfield, Huddersfield HD1 3DH, UK"}]}],"member":"1968","published-online":{"date-parts":[[2020,6,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1007\/s12193-016-0235-6","article-title":"An insight into assistive technology for the visually impaired and blind people: State-of-the-art and future trends","volume":"11","author":"Bhowmick","year":"2017","journal-title":"J. Multimodal User Interfaces"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1109\/10.81565","article-title":"A blind mobility aid modeled after echolocation of bats","volume":"38","author":"Ifukube","year":"1991","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_3","unstructured":"Shoval, S., Borenstein, J., and Koren, Y. (1994, January 8\u201313). Mobile robot obstacle avoidance in a computerized travel aid for the blind. Proceedings of the 1994 IEEE International Conference on Robotics and Automation, San Diego, CA, USA."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1016\/j.cviu.2018.04.007","article-title":"RGB-D-based human motion recognition with deep learning: A survey","volume":"171","author":"Wang","year":"2018","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Huang, A.S., Bachrach, A., Henry, P., Krainin, M., Maturana, D., Fox, D., and Roy, N. (2017). Visual odometry and mapping for autonomous flight using an RGB-D camera. Robotics Research, Springer.","DOI":"10.1007\/978-3-319-29363-9_14"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Mueller, F., Mehta, D., Sotnychenko, O., Sridhar, S., Casas, D., and Theobalt, C. (2017, January 22\u201329). Real-time hand tracking under occlusion from an egocentric rgb-d sensor. Proceedings of the IEEE International Conference on Computer Vision Workshops, Venice, Italy.","DOI":"10.1109\/ICCVW.2017.82"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohi, P., Shotton, J., Hodges, S., and Fitzgibbon, A. (2011, January 26\u201329). KinectFusion: Real-time dense surface mapping and tracking. Proceedings of the 10th IEEE International Symposium on Mixed and Augmented Reality, Basel, Switzerland.","DOI":"10.1109\/ISMAR.2011.6092378"},{"key":"ref_8","first-page":"1","article-title":"Real-time 3D reconstruction at scale using voxel hashing","volume":"32","author":"Izadi","year":"2013","journal-title":"ACM Trans. Graph. ToG"},{"key":"ref_9","unstructured":"Hermann, T., Hunt, A., and Neuhoff, J.G. (2011). The Sonification Handbook, Logos Verlag."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"2274","DOI":"10.1109\/TPAMI.2012.120","article-title":"SLIC superpixels compared to state-of-the-art superpixel methods","volume":"34","author":"Achanta","year":"2012","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Yang, K., Wang, K., Hu, W., and Bai, J. (2016). Expanding the detection of traversable area with RealSense for the visually impaired. Sensors, 16.","DOI":"10.3390\/s16111954"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Kumar, A., Patra, R., Manjunatha, M., Mukhopadhyay, J., and Majumdar, A.K. (2011, January 4\u20138). An electronic travel aid for navigation of visually impaired persons. Proceedings of the 2011 Third International Conference on Communication Systems and Networks (COMSNETS 2011), Bangalore, India.","DOI":"10.1109\/COMSNETS.2011.5716517"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"641","DOI":"10.1109\/THMS.2015.2419256","article-title":"A haptic solution to assist visually impaired in mobility tasks","volume":"45","author":"Baglio","year":"2015","journal-title":"IEEE Trans. Hum. Mach. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Sendra, S., Lloret, J., Romero, P., and Parra, L. (2018). Low-Cost System for Travel Aid and Obstacles Detection for the Visually Impaired People. The International Conference on Healthcare Science and Engineering, Springer.","DOI":"10.1007\/978-981-13-6837-0_21"},{"key":"ref_15","unstructured":"Lee, Y.H., and Medioni, G. (2011, January 27\u201330). RGB-D camera based navigation for the visually impaired. Proceedings of the Robotics: Science and Systems (RSS) 2011, Los Angeles, CA, USA."},{"key":"ref_16","first-page":"427","article-title":"Form perception with a 49-point electrotactile stimulus array on the tongue: A technical note","volume":"35","author":"Kaczmarek","year":"1998","journal-title":"J. Rehabil. Res. Dev."},{"key":"ref_17","unstructured":"Wicab Inc. (2018, April 24). BrainPort V100 Vision Aid|BrainPort V100. Available online: https:\/\/www.wicab.com\/brainport-v100."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Dubus, G., and Bresin, R. (2013). A systematic review of mapping strategies for the sonification of physical quantities. PLoS ONE, 8.","DOI":"10.1371\/journal.pone.0082491"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"17476","DOI":"10.3390\/s121217476","article-title":"Assisting the visually impaired: Obstacle detection and warning system by acoustic feedback","volume":"12","author":"Yebes","year":"2012","journal-title":"Sensors"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"922","DOI":"10.1109\/JSYST.2014.2320639","article-title":"Navigation assistance for the visually impaired using RGB-D sensor with range expansion","volume":"10","author":"Aladren","year":"2014","journal-title":"IEEE Syst. J."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1109\/10.121642","article-title":"An experimental system for auditory image representations","volume":"39","author":"Meijer","year":"1992","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"924","DOI":"10.1109\/JSTSP.2016.2543678","article-title":"Audio\u2013vision substitution for blind individuals: Addressing human information processing capacity limitations","volume":"10","author":"Brown","year":"2016","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"498","DOI":"10.1162\/jocn_a_01210","article-title":"Hearing Shapes: Event-related Potentials Reveal the Time Course of Auditory\u2013Visual Sensory Substitution","volume":"30","author":"Graulty","year":"2018","journal-title":"J. Cogn. Neurosci."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1016\/j.ijhcs.2018.02.002","article-title":"Blind wayfinding with physically-based liquid sounds","volume":"115","author":"Spagnol","year":"2018","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1016\/j.ijhcs.2015.08.003","article-title":"Sonification of guidance data during road crossing for people with visual impairments or blindness","volume":"85","author":"Mascetti","year":"2016","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"ref_26","unstructured":"Grond, F., and Berger, J. (2011). Parameter mapping sonification. The Sonification Handbook, Logos Publishing House."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"19","DOI":"10.3109\/01050399109070785","article-title":"Central auditory skills in blind and sighted subjects","volume":"20","author":"Muchnik","year":"1991","journal-title":"Scand. Audiol."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"515","DOI":"10.1007\/s00221-005-2329-3","article-title":"Enhanced sensitivity to echo cues in blind subjects","volume":"165","author":"Dufour","year":"2005","journal-title":"Exp. Brain Res."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1155\/2018\/3918284","article-title":"Current use and future perspectives of spatial audio technologies in electronic travel aids","volume":"2018","author":"Spagnol","year":"2018","journal-title":"Wirel. Commun. Mob. Comput."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1145\/358669.358692","article-title":"Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography","volume":"24","author":"Fischler","year":"1981","journal-title":"Commun. ACM"},{"key":"ref_31","unstructured":"Cheng, R., Wang, K., Yang, K., and Zhao, X. (2015, January 19). A Ground and Obstacle Detection Algorithm for the Visually Impaired. Proceedings of the IET International Conference on Biomedical Image and Signal Processing (ICBISP2015), Beijing, China."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27\u201330). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Yang, K., Wang, K., Cheng, R., Hu, W., Huang, X., and Bai, J. (2017). Detecting traversable area and water hazards for the visually impaired with a pRGB-D sensor. Sensors, 17.","DOI":"10.3390\/s17081890"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Keselman, L., Woodfill, J.I., Grunnet-Jepsen, A., and Bhowmik, A. (2017, January 21\u201326). Intel (r) realsense (tm) stereoscopic depth cameras. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA.","DOI":"10.1109\/CVPRW.2017.167"},{"key":"ref_35","first-page":"113","article-title":"An efficient orientation filter for inertial and inertial\/magnetic sensor arrays","volume":"25","author":"Madgwick","year":"2010","journal-title":"Rep. X-Io Univ. Bristol"},{"key":"ref_36","unstructured":"University of Iowa Electronic Music Studios (2018, December 04). Musical Instrument Samples Database. Available online: http:\/\/theremin.music.uiowa.edu\/index.html."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Rai, A., Chintalapudi, K.K., Padmanabhan, V.N., and Sen, R. (2012, January 22\u201326). Zee: Zero-effort crowdsourcing for indoor localization. Proceedings of the 18th Annual International Conference on Mobile Computing and Networking, Istanbul, Turkey.","DOI":"10.1145\/2348543.2348580"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Yang, K., Wang, K., Bergasa, L., Romera, E., Hu, W., Sun, D., Sun, J., Cheng, R., Chen, T., and L\u00f3pez, E. (2018). Unifying terrain awareness for the visually impaired through real-time semantic segmentation. Sensors, 18.","DOI":"10.3390\/s18051506"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/11\/3222\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:36:11Z","timestamp":1760175371000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/11\/3222"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,5]]},"references-count":38,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2020,6]]}},"alternative-id":["s20113222"],"URL":"https:\/\/doi.org\/10.3390\/s20113222","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,5]]}}}