{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T15:09:17Z","timestamp":1788880157836,"version":"build-2803163510"},"reference-count":58,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2022,2,4]],"date-time":"2022-02-04T00:00:00Z","timestamp":1643932800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001804","name":"Canada Research Chairs","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001804","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>Robotic leg prostheses and exoskeletons can provide powered locomotor assistance to older adults and\/or persons with physical disabilities. However, the current locomotion mode recognition systems being developed for automated high-level control and decision-making rely on mechanical, inertial, and\/or neuromuscular sensors, which inherently have limited prediction horizons (i.e., analogous to walking blindfolded). Inspired by the human vision-locomotor control system, we developed an environment classification system powered by computer vision and deep learning to predict the oncoming walking environments prior to physical interaction, therein allowing for more accurate and robust high-level control decisions. In this study, we first reviewed the development of our \u201cExoNet\u201d database\u2014the largest and most diverse open-source dataset of wearable camera images of indoor and outdoor real-world walking environments, which were annotated using a hierarchical labeling architecture. We then trained and tested over a dozen state-of-the-art deep convolutional neural networks (CNNs) on the ExoNet database for image classification and automatic feature engineering, including: EfficientNetB0, InceptionV3, MobileNet, MobileNetV2, VGG16, VGG19, Xception, ResNet50, ResNet101, ResNet152, DenseNet121, DenseNet169, and DenseNet201. Finally, we quantitatively compared the benchmarked CNN architectures and their environment classification predictions using an operational metric called \u201cNetScore,\u201d which balances the image classification accuracy with the computational and memory storage requirements (i.e., important for onboard real-time inference with mobile computing devices). Our comparative analyses showed that the EfficientNetB0 network achieves the highest test accuracy; VGG16 the fastest inference time; and MobileNetV2 the best NetScore, which can inform the optimal architecture design or selection depending on the desired performance. Overall, this study provides a large-scale benchmark and reference for next-generation environment classification systems for robotic leg prostheses and exoskeletons.<\/jats:p>","DOI":"10.3389\/fnbot.2021.730965","type":"journal-article","created":{"date-parts":[[2022,2,4]],"date-time":"2022-02-04T03:05:48Z","timestamp":1643943948000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":77,"title":["Environment Classification for Robotic Leg Prostheses and Exoskeletons Using Deep Convolutional Neural Networks"],"prefix":"10.3389","volume":"15","author":[{"given":"Brokoslaw","family":"Laschowski","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"William","family":"McNally","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexander","family":"Wong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John","family":"McPhee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2022,2,4]]},"reference":[{"key":"B1","volume-title":"An Analysis of Deep Neural Network Models for Practical Applications","author":"Canziani","year":"2016"},{"key":"B2","volume-title":"Xception: Deep Learning With Depthwise Separable Convolutions.","author":"Chollet","year":"2016"},{"key":"B3","volume-title":"Evaluation of Embedded Platforms for Lower Limb Prosthesis With Visual Sensing Capabilities","author":"Da Silva","year":"2020"},{"key":"B4","first-page":"248","article-title":"ImageNet: A large-scale hierarchical image database,","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Deng","year":"2009"},{"key":"B5","first-page":"1817","article-title":"Visual terrain identification and surface inclination estimation for improving human locomotion with a lower-limb prosthetic,","volume-title":"Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)","author":"Diaz","year":"2018"},{"key":"B6","volume-title":"RepVGG: Making VGG-style ConvNets Great Again","author":"Ding","year":"2021"},{"key":"B7","doi-asserted-by":"publisher","first-page":"2716","DOI":"10.1109\/TBME.2012.2208641","article-title":"Toward design of an environment-aware adaptive locomotion-mode-recognition system","volume":"59","author":"Du","year":"2012","journal-title":"IEEE Transact. Biomed. Eng."},{"key":"B8","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1186\/s12984-018-0458-8","article-title":"Mobility related physical and functional losses due to aging and disease\u2014a motivation for lower limb exoskeletons","volume":"16","author":"Grimmer","year":"2019","journal-title":"J. Neuroeng. Rehabil."},{"key":"B9","volume-title":"Deep Residual Learning for Image Recognition","author":"He","year":"2015"},{"key":"B10","volume-title":"MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications","author":"Howard","year":"2017"},{"key":"B11","first-page":"807","article-title":"A novel method for bilateral gait segmentation using a single thigh-mounted depth sensor and IMU,","volume-title":"IEEE International Conference on Biomedical Robotics and Biomechatronics (BIOROB)","author":"Hu","year":"2018"},{"key":"B12","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2017.243","article-title":"Densely connected convolutional networks,","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Huang","year":"2017"},{"key":"B13","first-page":"4255","article-title":"Improving the performance of a neural-machine interface for artificial legs using prior knowledge of walking environment,","volume-title":"Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)","author":"Huang","year":""},{"key":"B14","doi-asserted-by":"publisher","first-page":"2867","DOI":"10.1109\/TBME.2011.2161671","article-title":"Continuous locomotion-mode identification for prosthetic legs based on neuromuscular-mechanical fusion","volume":"58","author":"Huang","year":"","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"B15","doi-asserted-by":"crossref","DOI":"10.1109\/BioRob49111.2020.9224407","article-title":"An environment recognition and parameterization system for shared-control of a powered lower-limb exoskeleton,","volume-title":"IEEE RAS\/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob)","author":"Karacan","year":"2020"},{"key":"B16","article-title":"Convolutional neural networks for environmentally aware locomotion mode recognition of lower-limb amputees,","volume-title":"ASME Dynamic Systems and Control Conference (DSCC)","author":"Khademi","year":"2019"},{"key":"B17","volume-title":"Adam: A Method for Stochastic Optimization","author":"Kingma","year":"2015"},{"key":"B18","first-page":"105","article-title":"A radar-based terrain mapping approach for stair detection towards enhanced prosthetic foot control,","volume-title":"IEEE International Conference on Biomedical Robotics and Biomechatronics (BIOROB)","author":"Kleiner","year":"2018"},{"key":"B19","first-page":"615","article-title":"Recognition of ascending stairs from 2D images for control of powered lower limb prostheses,","volume-title":"International IEEE\/EMBS Conference on Neural Engineering (NER)","author":"Krausz","year":"2015"},{"key":"B20","doi-asserted-by":"publisher","first-page":"5238","DOI":"10.3390\/s19235238","article-title":"A survey of teleceptive sensing for wearable assistive robotic devices","volume":"19","author":"Krausz","year":"2019","journal-title":"Sensors"},{"key":"B21","doi-asserted-by":"publisher","first-page":"2206","DOI":"10.1109\/TMRB.2021.3082206","article-title":"Sensor fusion of vision, kinetics and kinematics for forward prediction during walking with a transfemoral prosthesis","volume":"21","author":"Krausz","year":"2021","journal-title":"IEEE Trans. Med. Robot. Bionics."},{"key":"B22","doi-asserted-by":"publisher","first-page":"4887","DOI":"10.3390\/s19224887","article-title":"Subject- and environment-based sensor variability for wearable lower-limb assistive devices","volume":"19","author":"Krausz","year":"2019","journal-title":"Sensors"},{"key":"B23","doi-asserted-by":"publisher","first-page":"2576","DOI":"10.1109\/TBME.2015.2448457","article-title":"Depth sensing for improved control of lower limb prostheses","volume":"62","author":"Krausz","year":"2015","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"B24","doi-asserted-by":"crossref","DOI":"10.1115\/DETC2018-85234","article-title":"Electromechanical design of robotic transfemoral prostheses,","volume-title":"ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference (IDETC-CIE)","author":"Laschowski","year":"2018"},{"key":"B25","first-page":"868","article-title":"Preliminary design of an environment recognition system for controlling robotic lower-limb prostheses and exoskeletons,","volume-title":"IEEE International Conference on Rehabilitation Robotics (ICORR)","author":"Laschowski","year":""},{"key":"B26","doi-asserted-by":"crossref","DOI":"10.1109\/BioRob49111.2020.9224364","article-title":"Comparative analysis of environment recognition systems for control of lower-limb exoskeletons and prostheses,","volume-title":"IEEE International Conference on Biomedical Robotics and Biomechatronics (BIOROB)","author":"Laschowski","year":""},{"key":"B27","doi-asserted-by":"publisher","first-page":"562061","DOI":"10.3389\/frobt.2020.562061","article-title":"ExoNet database: Wearable camera images of human locomotion environments","volume":"7","author":"Laschowski","year":"","journal-title":"Front. Robot. AI"},{"key":"B28","doi-asserted-by":"crossref","DOI":"10.1109\/EMBC46164.2021.9630064","article-title":"Computer vision and deep learning for environment-adaptive control of robotic lower-limb exoskeletons,","volume-title":"Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)","author":"Laschowski","year":""},{"key":"B29","doi-asserted-by":"publisher","first-page":"040801","DOI":"10.1115\/1.4043460","article-title":"Lower-limb prostheses and exoskeletons with energy regeneration: Mechatronic design and optimization review","volume":"11","author":"Laschowski","year":"","journal-title":"ASME J. Mechan. Robot."},{"key":"B30","doi-asserted-by":"publisher","first-page":"455","DOI":"10.1109\/TMRB.2021.3058323","article-title":"Simulation of stand-to-sit biomechanics for robotic exoskeletons and prostheses with energy regeneration","volume":"3","author":"Laschowski","year":"","journal-title":"IEEE Trans. Med. Robot. Bionics"},{"key":"B31","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"B32","volume-title":"Computer Vision for Assistive Healthcare","author":"Leo","year":"2018"},{"key":"B33","doi-asserted-by":"crossref","DOI":"10.1109\/EMBC.2019.8857388","article-title":"Gaze fixation comparisons between amputees and able-bodied individuals in approaching stairs and level-ground transitions: a pilot study,","volume-title":"Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)","author":"Li","year":"2019"},{"key":"B34","doi-asserted-by":"publisher","first-page":"434","DOI":"10.1109\/TNSRE.2015.2420539","article-title":"Development of an environment-aware locomotion mode recognition system for powered lower limb prostheses","volume":"24","author":"Liu","year":"2016","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"B35","volume-title":"SGDR: Stochastic Gradient Descent With Warm Restarts","author":"Loshchilov","year":"2016"},{"key":"B36","doi-asserted-by":"publisher","first-page":"1759","DOI":"10.1109\/TBME.2017.2776157","article-title":"User-independent intent recognition for lower limb prostheses using depth sensing","volume":"65","author":"Massalin","year":"2018","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"B37","doi-asserted-by":"crossref","DOI":"10.1115\/DETC2021-69203","article-title":"Myoelectric control of robotic leg prostheses and exoskeletons: a review","volume-title":"ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference (IDETC-CIE) (Virtual: ASME) Accepted","author":"Nasr","year":"2021"},{"key":"B38","first-page":"3360","author":"Novo-Torres","year":"2019"},{"key":"B39","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1016\/S0966-6362(96)01109-5","article-title":"Understanding the roles of vision in the control of human locomotion","volume":"5","author":"Patla","year":"1997","journal-title":"Posture"},{"key":"B40","first-page":"5135","article-title":"Evaluation of a visual localization system for environment awareness in assistive devices,","volume-title":"Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)","author":"Rai","year":"2018"},{"key":"B41","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2018.00474","article-title":"MobileNetV2: inverted residuals and linear bottlenecks,","volume-title":"IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Sandler","year":"2018"},{"key":"B42","volume-title":"Very deep convolutional networks for large-scale image recognition","author":"Simonyan","year":"2014"},{"key":"B43","volume-title":"Rethinking the Inception architecture for computer vision.","author":"Szegedy","year":"2015"},{"key":"B44","volume-title":"EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks","author":"Tan","year":"2019"},{"key":"B45","doi-asserted-by":"publisher","first-page":"2139","DOI":"10.1007\/s00371-021-02138-x","article-title":"Real-time limb tracking in single depth images based on circle matching and line fitting","volume":"21","author":"Tschiedel","year":"2021","journal-title":"Visual Comput."},{"key":"B46","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1743-0003-12-1","article-title":"Control strategies for active lower extremity prosthetics and orthotics: A review","volume":"12","author":"Tucker","year":"2015","journal-title":"J. NeuroEng. Rehabil."},{"key":"B47","first-page":"5055","article-title":"A feasibility study of depth image based intent recognition for lower limb prostheses,","volume-title":"Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)","author":"Varol","year":"2016"},{"key":"B48","volume-title":"Attention is all you Need","author":"Vaswani","year":"2017"},{"key":"B49","first-page":"87","article-title":"Terrain recognition improves the performance of neural-machine interface for locomotion mode recognition,","volume-title":"IEEE International Conference on Computing, Networking and Communications (ICNC)","author":"Wang","year":"2013"},{"key":"B50","author":"Wong","year":"2018"},{"key":"B51","doi-asserted-by":"publisher","first-page":"171","DOI":"10.1109\/TNSRE.2016.2521160","article-title":"State of the art and future directions for lower limb robotic exoskeletons","volume":"25","author":"Young","year":"2017","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"B52","first-page":"5452","author":"Zhang","year":"2011"},{"key":"B53","volume-title":"Sensor Fusion for Predictive Control of Human-prosthesis-Environment Dynamics in Assistive Walking: A Survey","author":"Zhang","year":""},{"key":"B54","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1109\/TCYB.2020.2978216","article-title":"A subvision system for enhancing the environmental adaptability of the powered transfemoral prosthesis","volume":"20","author":"Zhang","year":"2020","journal-title":"IEEE Trans. Cybernetics."},{"key":"B55","volume-title":"Directional PointNet: 3D Environmental Classification for Wearable Robotics","author":"Zhang","year":""},{"key":"B56","doi-asserted-by":"publisher","first-page":"465","DOI":"10.1109\/TNSRE.2019.2895221","article-title":"Environmental features recognition for lower limb prostheses toward predictive walking","volume":"27","author":"Zhang","year":"","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"B57","doi-asserted-by":"publisher","first-page":"1780","DOI":"10.1109\/TNSRE.2019.2935765","article-title":"Sequential decision fusion for environmental classification in assistive walking","volume":"27","author":"Zhang","year":"","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"B58","doi-asserted-by":"publisher","first-page":"399","DOI":"10.1109\/TASE.2020.2993399","article-title":"Environmental context prediction for lower limb prostheses with uncertainty quantification","volume":"20","author":"Zhong","year":"2020","journal-title":"IEEE Trans. Autom. Sci. Engi."}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2021.730965\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,2,4]],"date-time":"2022-02-04T03:06:33Z","timestamp":1643943993000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2021.730965\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,4]]},"references-count":58,"alternative-id":["10.3389\/fnbot.2021.730965"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2021.730965","relation":{"has-review":[{"id-type":"doi","id":"10.32388\/RPVV1A","asserted-by":"object"}],"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.06.24.449600","asserted-by":"object"}]},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,4]]},"article-number":"730965"}}