{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T15:11:04Z","timestamp":1775574664790,"version":"3.50.1"},"reference-count":44,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2021,8,26]],"date-time":"2021-08-26T00:00:00Z","timestamp":1629936000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003151","name":"Fonds de recherche du Qu\u00e9bec \u2013 Nature et technologies","doi-asserted-by":"publisher","award":["2016-PR-188869"],"award-info":[{"award-number":["2016-PR-188869"]}],"id":[{"id":"10.13039\/501100003151","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003151","name":"Fonds de recherche du Qu\u00e9bec \u2013 Nature et technologies","doi-asserted-by":"publisher","award":["2020-CO-275043"],"award-info":[{"award-number":["2020-CO-275043"]}],"id":[{"id":"10.13039\/501100003151","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000196","name":"Canada Foundation for Innovation","doi-asserted-by":"publisher","award":["35395"],"award-info":[{"award-number":["35395"]}],"id":[{"id":"10.13039\/501100000196","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Advances in robotics are part of reducing the burden associated with manufacturing tasks in workers. For example, the cobot could be used as a \u201cthird-arm\u201d during the assembling task. Thus, the necessity of designing new intuitive control modalities arises. This paper presents a foot gesture approach centered on robot control constraints to switch between four operating modalities. This control scheme is based on raw data acquired by an instrumented insole located at a human\u2019s foot. It is composed of an inertial measurement unit (IMU) and four force sensors. Firstly, a gesture dictionary was proposed and, from data acquired, a set of 78 features was computed with a statistical approach, and later reduced to 3 via variance analysis ANOVA. Then, the time series collected data were converted into a 2D image and provided as an input for a 2D convolutional neural network (CNN) for the recognition of foot gestures. Every gesture was assimilated to a predefined cobot operating mode. The offline recognition rate appears to be highly dependent on the features to be considered and their spatial representation in 2D image. We achieve a higher recognition rate for a specific representation of features by sets of triangular and rectangular forms. These results were encouraging in the use of CNN to recognize foot gestures, which then will be associated with a command to control an industrial robot.<\/jats:p>","DOI":"10.3390\/s21175743","type":"journal-article","created":{"date-parts":[[2021,8,31]],"date-time":"2021-08-31T22:58:15Z","timestamp":1630450695000},"page":"5743","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Image Generation for 2D-CNN Using Time-Series Signal Features from Foot Gesture Applied to Select Cobot Operating Mode"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6197-3899","authenticated-orcid":false,"given":"Fadwa El","family":"Aswad","sequence":"first","affiliation":[{"name":"Laboratory of Automation and Robotic interaction (LAR.i), Department of Applied Sciences, Universit\u00e9 du Qu\u00e9bec \u00e0 Chicoutimi (UQAC), 555 Boulevard de l\u2019Universit\u00e9, Chicoutimi, QC G7H 2B1, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gilde Vanel Tchane","family":"Djogdom","sequence":"additional","affiliation":[{"name":"Laboratory of Automation and Robotic interaction (LAR.i), Department of Applied Sciences, Universit\u00e9 du Qu\u00e9bec \u00e0 Chicoutimi (UQAC), 555 Boulevard de l\u2019Universit\u00e9, Chicoutimi, QC G7H 2B1, Canada"},{"name":"Technological Institute of Industrial Maintenance (ITMI), Sept-Iles College, 175 Rue de la V\u00e9rendrye, Sept-Iles, QC G4R 5B7, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8763-0536","authenticated-orcid":false,"given":"Martin J.-D.","family":"Otis","sequence":"additional","affiliation":[{"name":"Laboratory of Automation and Robotic interaction (LAR.i), Department of Applied Sciences, Universit\u00e9 du Qu\u00e9bec \u00e0 Chicoutimi (UQAC), 555 Boulevard de l\u2019Universit\u00e9, Chicoutimi, QC G7H 2B1, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Johannes C.","family":"Ayena","sequence":"additional","affiliation":[{"name":"Communications and Microelectronic Integration Laboratory (LACIME), Department of Electrical Engineering, \u00c9cole de Technologie Sup\u00e9rieure, 1100 Rue Notre-Dame Ouest, Montr\u00e9al, QC H3C1K3, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ramy","family":"Meziane","sequence":"additional","affiliation":[{"name":"Laboratory of Automation and Robotic interaction (LAR.i), Department of Applied Sciences, Universit\u00e9 du Qu\u00e9bec \u00e0 Chicoutimi (UQAC), 555 Boulevard de l\u2019Universit\u00e9, Chicoutimi, QC G7H 2B1, Canada"},{"name":"Technological Institute of Industrial Maintenance (ITMI), Sept-Iles College, 175 Rue de la V\u00e9rendrye, Sept-Iles, QC G4R 5B7, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,8,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"628","DOI":"10.1016\/j.cirp.2009.09.009","article-title":"Cooperation of human and machines in assembly lines","volume":"58","author":"Lien","year":"2009","journal-title":"CIRP Ann."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Matthias, B., Kock, S., Jerregard, H., K\u00e4llman, M., and Lundberg, I. (2011, January 25\u201327). Safety of collaborative industrial robots: Certification possibilities for a collaborative assembly robot concept. Proceedings of the 2011 IEEE International Symposium on Assembly and Manufacturing (ISAM), Tampere, Finland.","DOI":"10.1109\/ISAM.2011.5942307"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"278","DOI":"10.1016\/j.robot.2019.07.013","article-title":"On-line collision avoidance for collaborative robot manipulators by adjusting off-line generated paths: An industrial use case","volume":"119","author":"Safeea","year":"2019","journal-title":"Robot. Auton. Syst."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Ende, T., Haddadin, S., Parusel, S., W\u00fcsthoff, T., Hassenzahl, M., and Albu-Sch\u00e4ffer, A. (2011, January 25\u201330). A human-centered approach to robot gesture based communication within collaborative working processes. Proceedings of the 2011 IEEE\/RSJ International Conference on Intelligent Robots and Systems, San Francisco, CA, USA.","DOI":"10.1109\/IROS.2011.6094592"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"851","DOI":"10.3390\/app5040851","article-title":"Visual recognition and its application to robot arm control","volume":"5","author":"Juang","year":"2015","journal-title":"Appl. Sci."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"132","DOI":"10.3934\/mbe.2021007","article-title":"Wearable on-device deep learning system for hand gesture recognition based on FPGA accelerator","volume":"18","author":"Jiang","year":"2020","journal-title":"Math. Biosci. Eng."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Crossan, A., Brewster, S., and Ng, A. (2010, January 6\u201310). Foot tapping for mobile interaction. Proceedings of the 24th BCS Interaction Specialist Group Conference (HCI 2010 24), Dundee, UK.","DOI":"10.14236\/ewic\/HCI2010.49"},{"key":"ref_8","unstructured":"Valkov, D., Steinicke, F., Bruder, G., and Hinrichs, K.H. (2010, January 7\u20139). Traveling in 3d virtual environments with foot gestures and a multi-touch enabled wim. Proceedings of the Virtual reality International Conference (VRIC 2010), Laval, France."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/LSENS.2020.2984152","article-title":"A Customized Convolutional Neural Network Model Integrated with Acceleration-Based Smart Insole Toward Personalized Foot Gesture Recognition","volume":"4","author":"Hua","year":"2020","journal-title":"IEEE Sens. Lett."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"377","DOI":"10.1109\/70.954751","article-title":"Cobot architecture","volume":"17","author":"Peshkin","year":"2001","journal-title":"IEEE Trans. Robot. Autom."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Meziane, R., Li, P., Otis, M.J.-D., Ezzaidi, H., and Cardou, P. (2014, January 16\u201318). Safer Hybrid Workspace Using Human-Robot Interaction While Sharing Production Activities. Proceedings of the 2014 IEEE International Symposium on Robotic and Sensors Environments (ROSE), Timisoara, Romania.","DOI":"10.1109\/ROSE.2014.6952980"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1007\/s00170-018-2788-x","article-title":"Gesture-based human-robot interaction for human assistance in manufacturing","volume":"101","author":"Neto","year":"2019","journal-title":"Int. J. Adv. Manuf. Technol."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"593","DOI":"10.1007\/s10514-016-9556-2","article-title":"Probabilistic movement primitives for coordination of multiple human\u2013robot collaborative tasks","volume":"41","author":"Maeda","year":"2017","journal-title":"Auton. Robot."},{"key":"ref_14","unstructured":"Lopes, M., Peters, J., Piater, J., Toussaint, M., Baisero, A., Busch, B., Erkent, O., Kroemer, O., Lioutikov, R., and Maeda, G. (2015). Semi-Autonomous 3rd-Hand Robot. Robot. Future Manuf. Scenar., 3."},{"key":"ref_15","unstructured":"Bischoff, R., Kurth, J., Schreiber, G., Koeppe, R., Albu-Sch\u00e4ffer, A., Beyer, A., Eiberger, O., Haddadin, S., Stemmer, A., and Grunwald, G. (2010, January 7\u20139). The KUKA-DLR Lightweight Robot arm-a new reference platform for robotics research and manufacturing. Proceedings of the Robotics (ISR), 2010 41st International Symposium on and 2010 6th German Conference on Robotics (ROBOTIK), Munich, Germany."},{"key":"ref_16","unstructured":"Sasaki, T., Saraiji, M., Fernando, C.L., Minamizawa, K., and Inami, M. (August, January 30). MetaLimbs: Multiple arms interaction metamorphism. Proceedings of the ACM SIGGRAPH, Emerging Technologies, Los Angeles, CA, USA."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Fleming, I., Balicki, M., Koo, J., Iordachita, I., Mitchell, B., Handa, J., Hager, G., and Taylor, R. (2008, January 6\u201310). Cooperative robot assistant for retinal microsurgery. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, New York, NY, USA.","DOI":"10.1007\/978-3-540-85990-1_65"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Faria, D.R., Vieira, M., Faria, F.C., and Premebida, C. (September, January 28). Affective facial expressions recognition for human-robot interaction. Proceedings of the 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Lisbon, Portugal.","DOI":"10.1109\/ROMAN.2017.8172395"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Putro, M.D., and Jo, K.H. (2018, January 6\u20138). Real-time face tracking for human-robot interaction. Proceedings of the 2018 International Conference on Information and Communication Technology Robotics (ICT-ROBOT), Busan, Korea.","DOI":"10.1109\/ICT-ROBOT.2018.8549902"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Lakomkin, E., Zamani, M.A., Weber, C., Magg, S., and Wermter, S. (2018, January 1\u20135). On the robustness of speech emotion recognition for human-robot interaction with deep neural networks. Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.","DOI":"10.1109\/IROS.2018.8593571"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Fan, M., Ding, Y., Shen, F., You, Y., and Yu, Z. (2017, January 11\u201315). An empirical study of foot gestures for hands-occupied mobile interaction. Proceedings of the 2017 ACM International Symposium on Wearable Computers, Maui, HI, USA.","DOI":"10.1145\/3123021.3123043"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Kim, T., Blum, J.R., Alirezaee, P., Arnold, A.G., Fortin, P.E., and Cooperstock, J.R. (2019). Usability of foot-based interaction techniques for mobile solutions. Mobile Solutions and Their Usefulness in Everyday Life, Springer.","DOI":"10.1007\/978-3-319-93491-4_16"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"10187","DOI":"10.1109\/JSEN.2019.2931715","article-title":"Foot gesture recognition through dual channel wearable EMG System","volume":"19","author":"Maragliulo","year":"2019","journal-title":"IEEE Sens. J."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"518","DOI":"10.1016\/j.promfg.2015.07.216","article-title":"Gesture Recognition Using a Depth Camera for Human Robot Collaboration on Assembly Line","volume":"3","author":"Moutarde","year":"2015","journal-title":"Procedia Manuf."},{"key":"ref_25","unstructured":"Calinon, S., and Billard, A. (October, January 28). Stochastic gesture production and recognition model for a humanoid robot. Proceedings of the 2004 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Sendai, Japan."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Georgi, M., Amma, C., and Schultz, T. (2015, January 12\u201315). Recognizing Hand and Finger Gestures with IMU based Motion and EMG based Muscle Activity Sensing. Proceedings of the Biosignals 2015-International Conference on Bio-Inspired Systems and Signal Processing, Lisbon, Portugal.","DOI":"10.5220\/0005276900990108"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1109\/TSMCA.2010.2093883","article-title":"Wearable sensor-based hand gesture and daily activity recognition for robot-assisted living","volume":"41","author":"Zhu","year":"2011","journal-title":"IEEE Trans. Syst. Man Cybern. Part A Syst. Hum."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1109\/TSMCC.2007.893280","article-title":"Gesture recognition: A survey","volume":"37","author":"Mitra","year":"2007","journal-title":"IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.)"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Hartmann, B., and Link, N. (2010, January 10\u201313). Gesture recognition with inertial sensors and optimized DTW prototypes. Proceedings of the 2010 IEEE International Conference on Systems, Man and Cybernetics, Istanbul, Turkey.","DOI":"10.1109\/ICSMC.2010.5641703"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"845","DOI":"10.12733\/jics20102412","article-title":"Localization and recognition of digit-writing hand gestures for smart TV systems","volume":"11","author":"Wu","year":"2014","journal-title":"J. Inf. Comput. Sci."},{"key":"ref_31","unstructured":"Buyssens, P., and Elmoataz, A. (July, January 27). R\u00e9seaux de neurones convolutionnels multi-\u00e9chelle pour la classification cellulaire. Proceedings of the RFIA, Clermond-Ferand, France."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Cho, H., and Yoon, S.M. (2018). Divide and conquer-based 1D CNN human activity recognition using test data sharpening. Sensors, 18.","DOI":"10.3390\/s18041055"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Kiranyaz, S., Ince, T., Abdeljaber, O., Avci, O., and Gabbouj, M. (2019, January 12\u201317). 1-d convolutional neural networks for signal processing applications. Proceedings of the ICASSP 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.","DOI":"10.1109\/ICASSP.2019.8682194"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Fakhrulddin, A.H., Fei, X., and Li, H. (2017, January 11\u201313). Convolutional neural networks (CNN) based human fall detection on Body Sensor Networks (BSN) sensor data. Proceedings of the 2017 4th International Conference on Systems and Informatics (ICSAI), Hangzhou, China.","DOI":"10.1109\/ICSAI.2017.8248516"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Wang, L., Peng, M., and Zhou, Q.F. (2019, January 19\u201322). Fall detection based on convolutional neural networks using smart insole. Proceedings of the 2019 5th International Conference on Control, Automation and Robotics (ICCAR), Beijing, China.","DOI":"10.1109\/ICCAR.2019.8813332"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"2719","DOI":"10.1007\/s10586-017-1435-x","article-title":"Hand gesture recognition based on convolution neural network","volume":"22","author":"Li","year":"2019","journal-title":"Clust. Comput."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1109\/TPAMI.2012.59","article-title":"3D convolutional neural networks for human action recognition","volume":"35","author":"Ji","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Ha, S., and Choi, S. (2016, January 24\u201329). Convolutional neural networks for human activity recognition using multiple accelerometer and gyroscope sensors. Proceedings of the 2016 International Joint Conference on Neural Networks, Vancouver, BC, Canada.","DOI":"10.1109\/IJCNN.2016.7727224"},{"key":"ref_39","unstructured":"(2017, March 02). Datasheet Mpu9250. Available online: https:\/\/www.invensense.com\/wp-content\/uploads\/2015\/02\/PS-MPU-9250A-01-v1.1.pdf."},{"key":"ref_40","unstructured":"(2017, March 11). Datasheet ADS1115. Available online: https:\/\/cdn-shop.adafruit.com\/datasheets\/ads1115.pdf."},{"key":"ref_41","unstructured":"(2017, March 11). Datasheet ESP-12E. Available online: https:\/\/www.kloppenborg.net\/images\/blog\/esp8266\/esp8266-esp12e-specs.pdf."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Barkallah, E., Freulard, J., Otis, M.J.D., Ngomo, S., Ayena, J.C., and Desrosiers, C. (2017). Wearable Devices for Classification of Inadequate Posture at Work Using Neural Networks. Sensors, 17.","DOI":"10.3390\/s17092003"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1016\/S0169-7439(01)00198-8","article-title":"Pattern recognition of jet fuels: Comprehensive GC \u00d7 GC with ANOVA-based feature selection and principal component analysis","volume":"60","author":"Johnson","year":"2002","journal-title":"Chemom. Intell. Lab. Syst."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"56290","DOI":"10.1109\/ACCESS.2020.2982405","article-title":"sEMG measurement position and feature optimization strategy for gesture recognition based on ANOVA and neural networks","volume":"8","author":"Wu","year":"2020","journal-title":"IEEE Access"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/17\/5743\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:52:29Z","timestamp":1760165549000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/17\/5743"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,26]]},"references-count":44,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2021,9]]}},"alternative-id":["s21175743"],"URL":"https:\/\/doi.org\/10.3390\/s21175743","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8,26]]}}}