{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T22:32:19Z","timestamp":1776119539151,"version":"3.50.1"},"reference-count":41,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2021,5,20]],"date-time":"2021-05-20T00:00:00Z","timestamp":1621468800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"ZJU-Sunny Photonics Innovation Center","award":["No. 2020-03"],"award-info":[{"award-number":["No. 2020-03"]}]},{"name":"AccessibleMaps project by the Federal Ministry 627 of Labor and Social Affairs (BMAS)","award":["No. 01KM151112"],"award-info":[{"award-number":["No. 01KM151112"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Scene sonification is a powerful technique to help Visually Impaired People (VIP) understand their surroundings. Existing methods usually perform sonification on the entire images of the surrounding scene acquired by a standard camera or on the priori static obstacles acquired by image processing algorithms on the RGB image of the surrounding scene. However, if all the information in the scene are delivered to VIP simultaneously, it will cause information redundancy. In fact, biological vision is more sensitive to moving objects in the scene than static objects, which is also the original intention of the event-based camera. In this paper, we propose a real-time sonification framework to help VIP understand the moving objects in the scene. First, we capture the events in the scene using an event-based camera and cluster them into multiple moving objects without relying on any prior knowledge. Then, sonification based on MIDI is enabled on these objects synchronously. Finally, we conduct comprehensive experiments on the scene video with sonification audio attended by 20 VIP and 20 Sighted People (SP). The results show that our method allows both participants to clearly distinguish the number, size, motion speed, and motion trajectories of multiple objects. The results show that our method is more comfortable to hear than existing methods in terms of aesthetics.<\/jats:p>","DOI":"10.3390\/s21103558","type":"journal-article","created":{"date-parts":[[2021,5,20]],"date-time":"2021-05-20T11:45:57Z","timestamp":1621511157000},"page":"3558","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Seeing through Events: Real-Time Moving Object Sonification for Visually Impaired People Using Event-Based Camera"],"prefix":"10.3390","volume":"21","author":[{"given":"Zihao","family":"Ji","sequence":"first","affiliation":[{"name":"National Engineering Research Center of Optical Instrumentation, Zhejiang University, Hangzhou 310058, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Weijian","family":"Hu","sequence":"additional","affiliation":[{"name":"National Engineering Research Center of Optical Instrumentation, Zhejiang University, Hangzhou 310058, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ze","family":"Wang","sequence":"additional","affiliation":[{"name":"National Engineering Research Center of Optical Instrumentation, Zhejiang University, Hangzhou 310058, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1090-667X","authenticated-orcid":false,"given":"Kailun","family":"Yang","sequence":"additional","affiliation":[{"name":"Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kaiwei","family":"Wang","sequence":"additional","affiliation":[{"name":"National Engineering Research Center of Optical Instrumentation, Zhejiang University, Hangzhou 310058, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,5,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s12193-016-0235-6","article-title":"An insight into assistive technology for the visually impaired and blind people: State-of-the-art and future trends","volume":"11","author":"Bhowmick","year":"2017","journal-title":"J. Multimodal User Interfaces"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1109\/TSMCC.2009.2021255","article-title":"Wearable obstacle avoidance electronic travel aids for blind: A survey","volume":"40","author":"Dakopoulos","year":"2009","journal-title":"IEEE Trans. Syst. Man Cybern. Part C Appl. Rev."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Hu, W., Wang, K., Yang, K., Cheng, R., and Xu, Z.J.S. (2020). A Comparative Study in Real-Time Scene Sonification for Visually Impaired People. Sensors, 20.","DOI":"10.3390\/s20113222"},{"key":"ref_4","unstructured":"Welsh, R. (1981). Foundations of Orientation and Mobility, American Foundation for the Blind."},{"key":"ref_5","unstructured":"Posner, M.I. (1989). Visual attention. Foundations of Cognitive Science, MIT Press. Available online: https:\/\/www.researchgate.net\/profile\/Gordon-Bower\/publication\/232500375_Experimental_methods_in_cognitive_science\/links\/0deec51980ae7df1a3000000\/Experimental-methods-in-cognitive-science.pdf."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Mahowald, M. (1994). The silicon retina. An Analog VLSI System for Stereoscopic Vision, Springer.","DOI":"10.1007\/978-1-4615-2724-4"},{"key":"ref_7","unstructured":"Gallego, G., Delbruck, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A., Conradt, J., and Daniilidis, K. (2019). Event-based vision: A survey. arXiv."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Khan, W., Ansell, D., Kuru, K., and Amina, M. (2016, January 4\u20136). Automated aircraft instrument reading using real time video analysis. Proceedings of the 2016 IEEE 8th International Conference on Intelligent Systems (IS), Sofia, Bulgaria.","DOI":"10.1109\/IS.2016.7737454"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1081","DOI":"10.1109\/TRO.2012.2198930","article-title":"Asynchronous Event-Based Visual Shape Tracking for Stable Haptic Feedback in Microrobotics","volume":"28","author":"Ni","year":"2012","journal-title":"IEEE Trans. Robot."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"925","DOI":"10.1162\/NECO_a_00720","article-title":"Visual tracking using neuromorphic asynchronous event-based cameras","volume":"27","author":"Ni","year":"2015","journal-title":"Neural Comput."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Litzenberger, M., Posch, C., Bauer, D., Belbachir, A.N., Schon, P., Kohn, B., and Garn, H. (2006, January 24\u201327). Embedded vision system for real-time object tracking using an asynchronous transient vision sensor. Proceedings of the 2006 IEEE 12th Digital Signal Processing Workshop & 4th IEEE Signal Processing Education Workshop, Teton National Park, WY, USA.","DOI":"10.1109\/DSPWS.2006.265448"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1710","DOI":"10.1109\/TNNLS.2014.2352401","article-title":"Asynchronous event-based multikernel algorithm for high-speed visual features tracking","volume":"26","author":"Lagorce","year":"2014","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Pi\u0105tkowska, E., Belbachir, A.N., Schraml, S., and Gelautz, M. (2012, January 16\u201321). Spatiotemporal multiple persons tracking using dynamic vision sensor. Proceedings of the 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Providence, RI, USA.","DOI":"10.1109\/CVPRW.2012.6238892"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Glover, A., and Bartolozzi, C. (2017, January 24\u201328). Robust visual tracking with a freely-moving event camera. Proceedings of the 2017 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.","DOI":"10.1109\/IROS.2017.8206226"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Stoffregen, T., Gallego, G., Drummond, T., Kleeman, L., and Scaramuzza, D. (2019, January 27\u201328). Event-based motion segmentation by motion compensation. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Korea.","DOI":"10.1109\/ICCV.2019.00734"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Mitrokhin, A., Ferm\u00fcller, C., Parameshwara, C., and Aloimonos, Y. (2018, January 1\u20135). Event-based moving object detection and tracking. Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.","DOI":"10.1109\/IROS.2018.8593805"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Mitrokhin, A., Ye, C., Fermuller, C., Aloimonos, Y., and Delbruck, T. (2019). EV-IMO: Motion segmentation dataset and learning pipeline for event cameras. arXiv.","DOI":"10.1109\/IROS40897.2019.8968520"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1109\/10.121642","article-title":"An experimental system for auditory image representations","volume":"39","author":"Meijer","year":"1992","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_19","unstructured":"Nagarajan, R., Yaacob, S., and Sainarayanan, G. (2003, January 15\u201317). Role of object identification in sonification system for visually impaired. Proceedings of the TENCON 2003 Conference on Convergent Technologies for Asia-Pacific Region, Bangalore, India."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1048","DOI":"10.1016\/j.protcy.2013.12.117","article-title":"Color sonification for the visually impaired","volume":"9","author":"Cavaco","year":"2013","journal-title":"Procedia Technol."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Yoshida, T., Kitani, K.M., Koike, H., Belongie, S., and Schlei, K. (2011, January 13). EdgeSonic: Image feature sonification for the visually impaired. Proceedings of the 2nd Augmented Human International Conference, Tokyo, Japan.","DOI":"10.1145\/1959826.1959837"},{"key":"ref_22","unstructured":"Banf, M., and Blanz, V. (2012, January 18\u201322). A modular computer vision sonification model for the visually impaired. Proceedings of the The 18th International Conference on Auditory Display (ICAD2012), Atlanta, Georgia."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1007\/s005300050108","article-title":"Using sonification","volume":"7","author":"Barrass","year":"1999","journal-title":"Multimed. Syst."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Dubus, G., and Bresin, R. (2013). A systematic review of mapping strategies for the sonification of physical quantities. PLoS ONE, 8.","DOI":"10.1371\/journal.pone.0082491"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Khan, W., Hussain, A., Khan, B., Nawaz, R., and Baker, T. (2019, January 7\u201310). Novel framework for outdoor mobility assistance and auditory display for visually impaired people. Proceedings of the 2019 12th International Conference on Developments in eSystems Engineering (DeSE), Kazan, Russia.","DOI":"10.1109\/DeSE.2019.00183"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Constantinescu, A., M\u00fcller, K., Haurilet, M., Petrausch, V., and Stiefelhagen, R. (2020, January 25\u201329). Bring the Environment to Life: A Sonification Module for People with Visual Impairments to Improve Situation Awareness. Proceedings of the 2020 International Conference on Multimodal Interaction, Utrecht, The Netherlands.","DOI":"10.1145\/3382507.3418874"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Wang, J., Yang, K., Hu, W., and Wang, K. (2018, January 7\u201310). An environmental perception and navigational assistance system for visually impaired persons based on semantic stixels and sound interaction. Proceedings of the 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Miyazaki, Japan.","DOI":"10.1109\/SMC.2018.00332"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Presti, G., Ahmetovic, D., Ducci, M., Bernareggi, C., Ludovico, L., Barat\u00e8, A., Avanzini, F., and Mascetti, S. (2019, January 28\u201330). WatchOut: Obstacle sonification for people with visual impairment or blindness. Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility, Pittsburgh, PA, USA.","DOI":"10.1145\/3308561.3353779"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Yang, K., Wang, K., Bergasa, L.M., Romera, E., Hu, W., Sun, D., Sun, J., Cheng, R., Chen, T., and L\u00f3pez, E. (2018). Unifying terrain awareness for the visually impaired through real-time semantic segmentation. Sensors, 18.","DOI":"10.3390\/s18051506"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Kaul, O.B., Behrens, K., and Rohs, M. (2021, January 8\u201313). Mobile Recognition and Tracking of Objects in the Environment through Augmented Reality and 3D Audio Cues for People with Visual Impairments. Proceedings of the Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (CHI EA \u201921), New York, NY, USA. Article 394.","DOI":"10.1145\/3411763.3451611"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Ahmetovic, D., Avanzini, F., Barat\u00e8, A., Bernareggi, C., Galimberti, G., Ludovico, L.A., Mascetti, S., and Presti, G. (2018, January 22\u201324). Sonification of pathways for people with visual impairments. Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility, Galway, Ireland.","DOI":"10.1145\/3234695.3241005"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1007\/s12193-018-0281-3","article-title":"Interactive sonification of U-depth images in a navigation aid for the visually impaired","volume":"13","author":"Skulimowski","year":"2019","journal-title":"J. Multimodal User Interfaces"},{"key":"ref_33","unstructured":"Jeon, M., Winton, R.J., Yim, J.-B., Bruce, C.M., and Walker, B.N. (2012, January 18\u201321). Aquarium fugue: Interactive sonification for children and visually impaired audience in informal learning environments. Proceedings of the 18th International Conference on Auditory Display, Atlanta, GA, USA."},{"key":"ref_34","unstructured":"Walker, B.N., Godfrey, M.T., Orlosky, J.E., Bruce, C., and Sanford, J. (2006, January 20\u201323). Aquarium sonification: Soundscapes for accessible dynamic informal learning environments. Proceedings of the 12th International Conference on Auditory Display, London, UK."},{"key":"ref_35","unstructured":"Walker, B.N., Kim, J., and Pendse, A. (2007, January 27\u201331). Musical soundscapes for an accessible aquarium: Bringing dynamic exhibits to the visually impaired. Proceedings of the ICMC, Copenhagen, Denmark."},{"key":"ref_36","first-page":"659","article-title":"Gaussian Mixture Models","volume":"741","author":"Reynolds","year":"2009","journal-title":"Encycl. Biom."},{"key":"ref_37","unstructured":"Christensen, T. (2006). The Cambridge History of Western Music Theory, Cambridge University Press."},{"key":"ref_38","unstructured":"Hermann, T., Hunt, A., and Neuhoff, J.G. (2011). The Sonification Handbook, Logos."},{"key":"ref_39","unstructured":"Kramer, G. (2000). Auditory Display: Sonification, Audification and Auditory Interfaces, Addison-Wesley Longman Publishing Co., Inc."},{"key":"ref_40","first-page":"394","article-title":"Midi: Musical instrument digital interface","volume":"34","author":"Moog","year":"1986","journal-title":"J. Audio Eng. Soc."},{"key":"ref_41","unstructured":"Sethares, W.A. (2005). Tuning, Timbre, Spectrum, Scale, Springer Science & Business Media."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/10\/3558\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:04:38Z","timestamp":1760162678000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/10\/3558"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,20]]},"references-count":41,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2021,5]]}},"alternative-id":["s21103558"],"URL":"https:\/\/doi.org\/10.3390\/s21103558","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,20]]}}}