{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T16:13:59Z","timestamp":1779898439517,"version":"3.53.1"},"reference-count":41,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2022,1,5]],"date-time":"2022-01-05T00:00:00Z","timestamp":1641340800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["2021R1F1A1059665"],"award-info":[{"award-number":["2021R1F1A1059665"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The demand for wheelchairs has increased recently as the population of the elderly and patients with disorders increases. However, society still pays less attention to infrastructure that can threaten the wheelchair user, such as sidewalks with cracks\/potholes. Although various studies have been proposed to recognize such challenges, they mainly depend on RGB images or IMU sensors, which are sensitive to outdoor conditions such as low illumination, bad weather, and unavoidable vibrations, resulting in unsatisfactory and unstable performance. In this paper, we introduce a novel system based on various convolutional neural networks (CNNs) to automatically classify the condition of sidewalks using images captured with depth and infrared modalities. Moreover, we compare the performance of training CNNs from scratch and the transfer learning approach, where the weights learned from the natural image domain (e.g., ImageNet) are fine-tuned to the depth and infrared image domain. In particular, we propose applying the ResNet-152 model pre-trained with self-supervised learning during transfer learning to leverage better image representations. Performance evaluation on the classification of the sidewalk condition was conducted with 100% and 10% of training data. The experimental results validate the effectiveness and feasibility of the proposed approach and bring future research directions.<\/jats:p>","DOI":"10.3390\/s22010380","type":"journal-article","created":{"date-parts":[[2022,1,9]],"date-time":"2022-01-09T23:08:26Z","timestamp":1641769706000},"page":"380","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Classification of the Sidewalk Condition Using Self-Supervised Transfer Learning for Wheelchair Safety Driving"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0911-2052","authenticated-orcid":false,"given":"Ha-Yeong","family":"Yoon","sequence":"first","affiliation":[{"name":"Department of Data Science, Seoul National University of Science and Technology, Seoul 01811, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2768-4687","authenticated-orcid":false,"given":"Jung-Hwa","family":"Kim","sequence":"additional","affiliation":[{"name":"Research Center for Data Science, Seoul National University of Science and Technology, Seoul 01811, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9313-6860","authenticated-orcid":false,"given":"Jin-Woo","family":"Jeong","sequence":"additional","affiliation":[{"name":"Department of Data Science, Seoul National University of Science and Technology, Seoul 01811, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,1,5]]},"reference":[{"key":"ref_1","unstructured":"Newswire, G. (2021). Wheelchair Market Forecast to 2028. Yahoo Financ., 6124481. Available online: https:\/\/finance.yahoo.com\/news\/wheelchair-market-forecast-2028-covid-121200871.html."},{"key":"ref_2","unstructured":"Miyasaka, A., and Ito, K. (2020). Many Schools in Japan Still not Accessible for Wheelchair Users. Asahi Shimbun, Available online: https:\/\/www.asahi.com\/ajw\/articles\/14010639."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Favey, C., Farcy, R., Donnez, J., Villanueva, J., and Zogaghi, A. (2021). Development of a New Negative Obstacle Sensor for Augmented Electric Wheelchair. Sensors, 21.","DOI":"10.3390\/s21196341"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"3799","DOI":"10.1109\/JSEN.2018.2815566","article-title":"Laser and Optical Flow Fusion for a Non-Intrusive Obstacle Detection System on an Intelligent Wheelchair","volume":"18","author":"Arnay","year":"2018","journal-title":"IEEE Sens. J."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Sun, J. (2020, January 29\u201331). A novel design of the intelligent stair-climbing wheelchair. Proceedings of the 2020 6th International Conference on Mechanical Engineering and Automation Science (ICMEAS), Moscow, Russia.","DOI":"10.1109\/ICMEAS51739.2020.00047"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Reddy Avutu, S., Paul, S., Prasad, V.A., and Verma, J.K. (2020, January 2\u20134). Modelling of Brushless DC Hub Motor to Control the Speed of Indigenous Powered Wheelchair. Proceedings of the 2020 International Conference on Computational Performance Evaluation (ComPE), Shillong, India.","DOI":"10.1109\/ComPE49325.2020.9200192"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"28356","DOI":"10.1109\/ACCESS.2018.2839729","article-title":"An Uphill Safety Controller With Deep Learning-Based Ramp Detection for Intelligent Wheelchairs","volume":"6","author":"Wu","year":"2018","journal-title":"IEEE Access"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Schabron, B., Desai, J., and Yihun, Y. (2021). Wheelchair-Mounted Upper Limb Robotic Exoskeleton with Adaptive Controller for Activities of Daily Living. Sensors, 21.","DOI":"10.3390\/s21175738"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ahmad, J., Sid\u00e9n, J., and Andersson, H. (2021). A Proposal of Implementation of Sitting Posture Monitoring System for Wheelchair Utilizing Machine Learning Methods. Sensors, 21.","DOI":"10.3390\/s21196349"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Vogel, J., Hagengruber, A., Iskandar, M., Quere, G., Leipscher, U., Bustamante, S., Dietrich, A., H\u00f6ppner, H., Leidner, D., and Albu-Sch\u00e4ffer, A. (January, January 24). EDAN: An EMG-controlled Daily Assistant to Help People With Physical Disabilities. Proceedings of the 2020 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.","DOI":"10.1109\/IROS45743.2020.9341156"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"654","DOI":"10.1109\/TNSRE.2016.2597854","article-title":"Commanding a Brain-Controlled Wheelchair Using Steady-State Somatosensory Evoked Potentials","volume":"26","author":"Kim","year":"2018","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Ho, M.T., Nguyen, T.T., Dao, H., and Tran, M.T. (2019, January 4\u20136). Visual Assistant for Crowdsourced Anomaly Event Recognition in Smart City. Proceedings of the Tenth International Symposium on Information and Communication Technology, Hanoi, Vietnam. SoICT 2019.","DOI":"10.1145\/3368926.3369734"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Edwan, E., Sarsour, N., and Alatrash, M. (2019, January 23\u201324). Mobile Application for Bumps Detection and Warning Utilizing Smartphone Sensors. Proceedings of the 2019 International Conference on Promising Electronic Technologies (ICPET), Gaza, Palestine.","DOI":"10.1109\/ICPET.2019.00017"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Wang, L., Yang, C., Yu, Z., Liu, Y., Wang, Z., and Guo, B. (2019, January 19\u201323). CrackSense: A CrowdSourcing Based Urban Road Crack Detection System. Proceedings of the 2019 IEEE SmartWorld, Ubiquitous Intelligence Computing, Advanced Trusted Computing, Scalable Computing Communications, Cloud Big Data Computing, Internet of People and Smart City Innovation (SmartWorld\/SCALCOM\/UIC\/ATC\/CBDCom\/IOP\/SCI), Leicester, UK.","DOI":"10.1109\/SmartWorld-UIC-ATC-SCALCOM-IOP-SCI.2019.00188"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Bongestu, D.R., and Warnars, H.L.H.S. (2021, January 25\u201327). Jakarta Smart City Mobile Application for Problem Reporting. Proceedings of the 2021 International Conference on Artificial Intelligence and Smart Systems (ICAIS), Coimbatore, India.","DOI":"10.1109\/ICAIS50930.2021.9395879"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"226","DOI":"10.1016\/j.protcy.2013.12.025","article-title":"A Web & Mobile City Maintenance Reporting Solution","volume":"9","author":"Santos","year":"2013","journal-title":"Procedia Technol."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"2635","DOI":"10.1109\/JSEN.2019.2952857","article-title":"A Machine Learning Approach to Road Surface Anomaly Assessment Using Smartphone Sensors","volume":"20","author":"Basavaraju","year":"2020","journal-title":"IEEE Sens. J."},{"key":"ref_18","unstructured":"Dharneeshkar, J., Aniruthan, S.A., Karthika, R., and Parameswaran, L. (2020, January 26\u201328). Deep Learning based Detection of potholes in Indian roads using YOLO. Proceedings of the 2020 International Conference on Inventive Computation Technologies (ICICT), Coimbatore, India."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Wu, C., Wang, Z., Hu, S., Lepine, J., Na, X., Ainalis, D., and Stettler, M. (2020). An Automated Machine-Learning Approach for Road Pothole Detection Using Smartphone Sensor Data. Sensors, 20.","DOI":"10.3390\/s20195564"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"8515213","DOI":"10.1155\/2020\/8515213","article-title":"Pavement Crack Detection and Segmentation Method Based on Improved Deep Learning Fusion Model","volume":"2020","author":"Feng","year":"2020","journal-title":"Math. Probl. Eng."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Watanabe, T., Takahashi, H., Iwasawa, Y., Matsuo, Y., and Eguchi Yairi, I. (2019). Weakly Supervised Learning for Evaluating Road Surface Condition from Wheelchair Driving Data. Information, 11.","DOI":"10.3390\/info11010002"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1153","DOI":"10.1587\/transinf.2015EDP7278","article-title":"Combining human action sensing of wheelchair users and machine learning for autonomous accessibility data collection","volume":"99","author":"Iwasawa","year":"2016","journal-title":"IEICE Trans. Inf. Syst."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"Imagenet large scale visual recognition challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis."},{"key":"ref_25","unstructured":"Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. (2020). Big self-supervised models are strong semi-supervised learners. arXiv."},{"key":"ref_26","first-page":"5015741","article-title":"Evaluation of Image Enhancement Techniques for Vision-Based Navigation under Low Illumination","volume":"2019","author":"Aladem","year":"2019","journal-title":"J. Robot."},{"key":"ref_27","first-page":"060403-1","article-title":"Pedestrian Detection at Night Using Deep Neural Networks and Saliency Maps","volume":"17","author":"Heo","year":"2018","journal-title":"J. Electron. Imaging"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"125459","DOI":"10.1109\/ACCESS.2020.3007481","article-title":"Thermal Object Detection in Difficult Weather Conditions Using YOLO","volume":"8","author":"Pobar","year":"2020","journal-title":"IEEE Access"},{"key":"ref_29","unstructured":"Michaelis, C., Mitzkus, B., Geirhos, R., Rusak, E., Bringmann, O., Ecker, A.S., Bethge, M., and Brendel, W. (2020, December 31). Benchmarking Robustness in Object Detection: Autonomous Driving When Winter Is Coming. Available online: https:\/\/arxiv.org\/abs\/1907.07484."},{"key":"ref_30","first-page":"2379685","article-title":"A Simple Outdoor Environment Obstacle Detection Method Based on Information Fusion of Depth and Infrared","volume":"2016","author":"Zhu","year":"2016","journal-title":"J. Robot."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"K\u016frkov\u00e1, V., Manolopoulos, Y., Hammer, B., Iliadis, L., and Maglogiannis, I. (2018). A Survey on Deep Transfer Learning. Artificial Neural Networks and Machine Learning\u2013ICANN 2018, Springer International Publishing.","DOI":"10.1007\/978-3-030-01418-6"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Misra, I., and Maaten, L.V.D. (2020, January 13\u201319). Self-supervised learning of pretext-invariant representations. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00674"},{"key":"ref_33","unstructured":"Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020, January 13\u201318). A simple framework for contrastive learning of visual representations. Proceedings of the International Conference on Machine Learning, Atlanta, GA, USA."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Noroozi, M., and Favaro, P. (2016). Unsupervised learning of visual representations by solving jigsaw puzzles. European Conference on Computer Vision, Springer.","DOI":"10.1007\/978-3-319-46466-4_5"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Doersch, C., Gupta, A., and Efros, A.A. (2015, January 11\u201318). Unsupervised visual representation learning by context prediction. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.167"},{"key":"ref_36","unstructured":"Zhao, N., Wu, Z., Lau, R.W.H., and Lin, S. (2021, January 3\u20137). What Makes Instance Discrimination Good for Transfer Learning?. Proceedings of the International Conference on Learning Representations, Virtual only."},{"key":"ref_37","unstructured":"You, Y., Gitman, I., and Ginsburg, B. (2017). Large batch training of convolutional networks. arXiv."},{"key":"ref_38","first-page":"1297","article-title":"The Effect of Training Set Size for the Performance of Neural Networks of Classification","volume":"9","author":"Sug","year":"2010","journal-title":"WSEAS Trans. Comput."},{"key":"ref_39","first-page":"5107","article-title":"Multi-View Multi-Modal Head-Gaze Estimation for Advanced Indoor User Interaction","volume":"70","author":"Kim","year":"2022","journal-title":"Comput. Mater. Contin."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Konno, T., Amma, A., and Kanezaki, A. (2021, January 7\u20139). Incremental Multi-View Object Detection from a Moving Camera. Proceedings of the 2nd ACM International Conference on Multimedia in Asia, Singapore.","DOI":"10.1145\/3444685.3446257"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Lin, J., and Lee, G.H. (2021, January 19\u201325). Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.01171"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/1\/380\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T14:00:23Z","timestamp":1760364023000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/1\/380"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,5]]},"references-count":41,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,1]]}},"alternative-id":["s22010380"],"URL":"https:\/\/doi.org\/10.3390\/s22010380","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,5]]}}}