{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,12]],"date-time":"2025-11-12T06:27:18Z","timestamp":1762928838987,"version":"build-2065373602"},"reference-count":58,"publisher":"MDPI AG","issue":"18","license":[{"start":{"date-parts":[[2020,9,12]],"date-time":"2020-09-12T00:00:00Z","timestamp":1599868800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Federal Ministry of Labor and Social Affairs (BMAS)","award":["01KM151112."],"award-info":[{"award-number":["01KM151112."]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The current COVID-19 pandemic is having a major impact on our daily lives. Social distancing is one of the measures that has been implemented with the aim of slowing the spread of the disease, but it is difficult for blind people to comply with this. In this paper, we present a system that helps blind people to maintain physical distance to other persons using a combination of RGB and depth cameras. We use a real-time semantic segmentation algorithm on the RGB camera to detect where persons are and use the depth camera to assess the distance to them; then, we provide audio feedback through bone-conducting headphones if a person is closer than 1.5 m. Our system warns the user only if persons are nearby but does not react to non-person objects such as walls, trees or doors; thus, it is not intrusive, and it is possible to use it in combination with other assistive devices. We have tested our prototype system on one blind and four blindfolded persons, and found that the system is precise, easy to use, and amounts to low cognitive load.<\/jats:p>","DOI":"10.3390\/s20185202","type":"journal-article","created":{"date-parts":[[2020,9,13]],"date-time":"2020-09-13T21:11:32Z","timestamp":1600031492000},"page":"5202","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":45,"title":["Helping the Blind to Get through COVID-19: Social Distancing Assistant Using Real-Time Semantic Segmentation on RGB-D Video"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6020-7618","authenticated-orcid":false,"given":"Manuel","family":"Martinez","sequence":"first","affiliation":[{"name":"Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1090-667X","authenticated-orcid":false,"given":"Kailun","family":"Yang","sequence":"additional","affiliation":[{"name":"Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Angela","family":"Constantinescu","sequence":"additional","affiliation":[{"name":"Study Centre for the Visually Impaired, Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rainer","family":"Stiefelhagen","sequence":"additional","affiliation":[{"name":"Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany"},{"name":"Study Centre for the Visually Impaired, Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,9,12]]},"reference":[{"key":"ref_1","unstructured":"(2020, July 07). KR-Vision Technology. Available online: http:\/\/krvision.cn."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Yang, K., Hu, X., Chen, H., Xiang, K., Wang, K., and Stiefelhagen, R. (2019). Ds-pass: Detail-sensitive panoramic annular semantic segmentation through swaftnet for surrounding sensing. arXiv.","DOI":"10.1109\/IV47402.2020.9304706"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Neuhold, G., Ollmann, T., Bul\u00f2, S.R., and Kontschieder, P. (2017, January 22\u201329). The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes. Proceedings of the International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.534"},{"key":"ref_4","unstructured":"Ren, S., He, K., Girshick, R., and Sun, J. (2016). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Trans. Pattern Anal. Mach. Intell., 91\u201399."},{"key":"ref_5","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (July, January 26). You Only Look Once: Unified, Real-Time Object Detection. Proceedings of the Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., and Ng, A.Y. (2009, January 12\u201317). ROS: An open-source Robot Operating System. Proceedings of the International Conference of Robotics and Automation Workshop, Kobe, Japan.","DOI":"10.1109\/MRA.2010.936956"},{"key":"ref_7","first-page":"1727","article-title":"How high is high? A metanalysis of NASA TLX global workload scores","volume":"59","author":"Grier","year":"2015","journal-title":"Hum. Factors Ergon. Soc."},{"key":"ref_8","unstructured":"NASA Ames Research Center, Human Performance Research Group (2020, July 17). NASA Task Load Index, Available online: https:\/\/humansystems.arc.nasa.gov\/groups\/TLX\/downloads\/TLX.pdf."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"17476","DOI":"10.3390\/s121217476","article-title":"Assisting the visually impaired: obstacle detection and warning system by acoustic feedback","volume":"12","author":"Yebes","year":"2012","journal-title":"Sensors"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Schauerte, B., Koester, D., Martinez, M., and Stiefelhagen, R. (2014, January 6\u201312). Way to go! Detecting open areas ahead of a walking person. Proceedings of the European Conference on Computer Vision Workshops, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-16199-0_25"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Elmannai, W., and Elleithy, K. (2017). Sensor-based assistive devices for visually-impaired people: Current status, challenges, and future directions. Sensors, 17.","DOI":"10.3390\/s17030565"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"922","DOI":"10.1109\/JSYST.2014.2320639","article-title":"Navigation assistance for the visually impaired using RGB-D sensor with range expansion","volume":"10","author":"Aladren","year":"2014","journal-title":"IEEE Syst. J."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Yang, K., Wang, K., Hu, W., and Bai, J. (2016). Expanding the detection of traversable area with RealSense for the visually impaired. Sensors, 16.","DOI":"10.3390\/s16111954"},{"key":"ref_14","unstructured":"Wang, H.C., Katzschmann, R.K., Teng, S., Araki, B., Giarr\u00e9, L., and Rus, D. (June, January 29). Enabling independent navigation for visually impaired people through a wearable vision-based feedback system. Proceedings of the International Conference on Robotics and Automation, Singapore."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"258","DOI":"10.1109\/TCE.2017.014980","article-title":"Smart guiding glasses for visually impaired people in indoor environment","volume":"63","author":"Bai","year":"2017","journal-title":"IEEE Trans. Consum. Electron."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"013028","DOI":"10.1117\/1.JEI.28.1.013028","article-title":"Assisting the visually impaired: multitarget warning through millimeter wave radar and RGB-depth sensors","volume":"28","author":"Long","year":"2019","journal-title":"J. Electron. Imaging"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Yang, K., Wang, K., Cheng, R., Hu, W., Huang, X., and Bai, J. (2017). Detecting traversable area and water hazards for the visually impaired with a pRGB-D sensor. Sensors, 17.","DOI":"10.3390\/s17081890"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Yang, K., Wang, K., Lin, S., Bai, J., Bergasa, L.M., and Arroyo, R. (2018, January 27\u201329). Long-range traversability awareness and low-lying obstacle negotiation with RealSense for the visually impaired. Proceedings of the International Conference on Information Science and Systems, Jeju Island, Korea.","DOI":"10.1145\/3209914.3209943"},{"key":"ref_19","unstructured":"Hua, M., Nan, Y., and Lian, S. (November, January 27). Small Obstacle Avoidance Based on RGB-D Semantic Segmentation. Proceedings of the International Conference on Computer Vision Workshop, Seoul, Korea."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Martinez, M., Roitberg, A., Koester, D., Stiefelhagen, R., and Schauerte, B. (2017, January 22\u201329). Using Technology Developed for Autonomous Cars to Help Navigate Blind People. Proceedings of the International Conference on Computer Vision Workshops, Venice, Italy.","DOI":"10.1109\/ICCVW.2017.169"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Badino, H., Franke, U., and Pfeiffer, D. (2009, January 9\u201311). The stixel world-a compact medium level representation of the 3d-world. Proceedings of the Joint Pattern Recognition Symposium, Jena, Germany.","DOI":"10.1007\/978-3-642-03798-6_6"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Wang, J., Yang, K., Hu, W., and Wang, K. (2018, January 7\u201310). An environmental perception and navigational assistance system for visually impaired persons based on semantic stixels and sound interaction. Proceedings of the International Conference on Systems, Man, and Cybernetics, Miyazaki, Japan.","DOI":"10.1109\/SMC.2018.00332"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Bai, J., Liu, Z., Lin, Y., Li, Y., Lian, S., and Liu, D. (2019). Wearable travel aid for environment perception and navigation of visually impaired people. Electronics, 8.","DOI":"10.3390\/electronics8060697"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Kajiwara, Y., and Kimura, H. (2019). Object identification and safe route recommendation based on human flow for the visually impaired. Sensors, 19.","DOI":"10.3390\/s19245343"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Cao, Z., Simon, T., Wei, S.E., and Sheikh, Y. (2017, January 21\u201326). Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields. Proceedings of the Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.143"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Dimas, G., Diamantis, D.E., Kalozoumis, P., and Iakovidis, D.K. (2020). Uncertainty-Aware Visual Perception System for Outdoor Navigation of the Visually Challenged. Sensors, 20.","DOI":"10.3390\/s20082385"},{"key":"ref_27","unstructured":"(2020, July 28). Bat Orientation Guide. Available online: http:\/\/www.synphon.de\/en\/fledermaus-orientierungshilfe.html."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Long, J., Shelhamer, E., and Darrell, T. (2015, January 7\u201312). Fully convolutional networks for semantic segmentation. Proceedings of the Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Or\u0161ic, M., Kre\u0161o, I., Bevandic, P., and \u0160egvic, S. (2019, January 16\u201320). In Defense of Pre-Trained ImageNet Architectures for Real-Time Semantic Segmentation of Road-Driving Images. Proceedings of the Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.01289"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Feng, D., Haase-Sch\u00fctz, C., Rosenbaum, L., Hertlein, H., Glaeser, C., Timm, F., Wiesbeck, W., and Dietmayer, K. (2020). Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Trans. Intell. Transp. Syst.","DOI":"10.1109\/TITS.2020.2972974"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Yang, K., Wang, K., Bergasa, L.M., Romera, E., Hu, W., Sun, D., Sun, J., Cheng, R., Chen, T., and L\u00f3pez, E. (2018). Unifying terrain awareness for the visually impaired through real-time semantic segmentation. Sensors, 18.","DOI":"10.3390\/s18051506"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Yang, K., Bergasa, L.M., Romera, E., Sun, D., Wang, K., and Barea, R. (2018, January 12\u201314). Semantic perception of curbs beyond traversability for real-world navigation assistance systems. Proceedings of the International Conference on Vehicular Electronics and Safety, Madrid, Spain.","DOI":"10.1109\/ICVES.2018.8519526"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Cao, Z., Xu, X., Hu, B., and Zhou, M. (2020). Rapid Detection of Blind Roads and Crosswalks by Using a Lightweight Semantic Segmentation Network. IEEE Trans. Intell. Transp. Syst.","DOI":"10.1109\/TITS.2020.2989129"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Yang, K., Cheng, R., Bergasa, L.M., Romera, E., Wang, K., and Long, N. (2018, January 12\u201315). Intersection perception through real-time semantic segmentation to assist navigation of visually impaired pedestrians. Proceedings of the International Conference on Robotics and Biomimetics, Kuala Lumpur, Malaysia.","DOI":"10.1109\/ROBIO.2018.8665211"},{"key":"ref_35","unstructured":"Mehta, S., Hajishirzi, H., and Shapiro, L. (2017). Identifying most walkable direction for navigation in an outdoor environment. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Watson, J., Firman, M., Monszpart, A., and Brostow, G.J. (2020, January 9\u201311). Footprints and Free Space from a Single Color Image. Proceedings of the Conference on Computer Vision and Pattern Recognition, Bangkok, Thailand.","DOI":"10.1109\/CVPR42600.2020.00009"},{"key":"ref_37","unstructured":"Lin, Y., Wang, K., Yi, W., and Lian, S. (November, January 27). Deep Learning Based Wearable Assistive System for Visually Impaired People. Proceedings of the International Conference on Computer Vision Workshop, Seoul, Korea."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"044102","DOI":"10.1063\/1.5093279","article-title":"Unifying obstacle detection, recognition, and fusion based on millimeter wave radar and RGB-depth sensors for the visually impaired","volume":"90","author":"Long","year":"2019","journal-title":"Rev. Sci. Instrum."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Yohannes, E., Shih, T.K., and Lin, C.Y. (2019, January 19\u201321). Content-Aware Video Analysis to Guide Visually Impaired Walking on the Street. Proceedings of the International Visual Informatics Conference, Bangi, Malaysia.","DOI":"10.1007\/978-3-030-34032-2_1"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Doll\u00e1r, P., and Girshick, R. (2017, January 22\u201329). Mask R-CNN. Proceedings of the International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_41","unstructured":"Mao, W., Zhang, J., Yang, K., and Stiefelhagen, R. (2020). Can we cover navigational perception needs of the visually impaired by panoptic segmentation?. arXiv."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Porzi, L., Bul\u00f2, S.R., Colovic, A., and Kontschieder, P. (2019, January 16\u201320). Seamless Scene Segmentation. Proceedings of the Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00847"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Cristani, M., Del Bue, A., Murino, V., Setti, F., and Vinciarelli, A. (2020). The Visual Social Distancing Problem. arXiv.","DOI":"10.1109\/ACCESS.2020.3008370"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Keselman, L., Woodfill, J.I., Grunnet-Jepsen, A., and Bhowmik, A. (2017, January 21\u201326). Intel (R) RealSense (TM) Stereoscopic Depth Cameras. Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.","DOI":"10.1109\/CVPRW.2017.167"},{"key":"ref_45","unstructured":"Nvidia (2020, July 07). Jetson AGX Xavier Developer Kit. Available online: https:\/\/developer.nvidia.com\/embedded\/jetson-agx-xavier-developer-kit."},{"key":"ref_46","unstructured":"Intel (2020, July 07). RealSense Technology. Available online: https:\/\/github.com\/IntelRealSense\/librealsense."},{"key":"ref_47","unstructured":"Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019, January 8\u201314). Pytorch: An imperative style, high-performance deep learning library. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada."},{"key":"ref_48","unstructured":"(2020, July 13). Open Source Computer Vision Library. Available online: https:\/\/github.com\/opencv\/opencv."},{"key":"ref_49","unstructured":"(2020, July 13). Open Audio Library. Available online: https:\/\/www.openal.org."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Hu, J., Shen, L., and Sun, G. (2018, January 18\u201323). Squeeze-and-Excitation Networks. Proceedings of the Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00745"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Yang, K., Hu, X., Bergasa, L.M., Romera, E., and Wang, K. (2019). Pass: Panoramic annular semantic segmentation. IEEE Trans. Intell. Transp. Syst.","DOI":"10.1109\/TITS.2019.2938965"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Nielsen, J., and Landauer, T.K. (1993, January 24\u201329). A Mathematical Model of the Finding of Usability Problems. Proceedings of the INTERACT \u201993 and CHI \u201993 Conference on Human Factors in Computing Systems, Amsterdam, The Netherlands.","DOI":"10.1145\/169059.169166"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Martinez, M., Constantinescu, A., Schauerte, B., Koester, D., and Stiefelhagen, R. (2014, January 9\u201311). Cognitive evaluation of haptic and audio feedback in short range navigation tasks. Proceedings of the International Conference on Computers for Handicapped Persons, Paris, France.","DOI":"10.1007\/978-3-319-08599-9_20"},{"key":"ref_54","unstructured":"Brooke, J. (1996). SUS: A quick and dirty usability scale. Usability Evaluation in Industry, Taylor & Francis Group."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"574","DOI":"10.1080\/10447310802205776","article-title":"An Empirical Evaluation of the System Usability Scale","volume":"24","author":"Bangor","year":"2008","journal-title":"Int. J. Human Comput. Interact."},{"key":"ref_56","unstructured":"(2020, September 09). TensorFlow Lite, an Open Source Deep Learning Framework for On-Device Inference. Available online: http:\/\/www.tensorflow.org\/lite."},{"key":"ref_57","unstructured":"(2020, September 09). Coral: A Complete Toolkit to Build Products with Local AI. Available online: http:\/\/coral.ai."},{"key":"ref_58","unstructured":"(2020, September 09). Intel\u00ae Movidius\u2122 Vision Processing Units (VPUs). Available online: https:\/\/www.intel.com\/content\/www\/us\/en\/products\/processors\/movidius-vpu.html."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/18\/5202\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:09:21Z","timestamp":1760177361000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/18\/5202"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,12]]},"references-count":58,"journal-issue":{"issue":"18","published-online":{"date-parts":[[2020,9]]}},"alternative-id":["s20185202"],"URL":"https:\/\/doi.org\/10.3390\/s20185202","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2020,9,12]]}}}