{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,5]],"date-time":"2025-11-05T14:30:58Z","timestamp":1762353058226,"version":"build-2065373602"},"reference-count":40,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2021,6,5]],"date-time":"2021-06-05T00:00:00Z","timestamp":1622851200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100010418","name":"Institute for Information and Communications Technology Promotion","doi-asserted-by":"publisher","award":["2019-0-00056, 2020-0-00201"],"award-info":[{"award-number":["2019-0-00056, 2020-0-00201"]}],"id":[{"id":"10.13039\/501100010418","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Recently, as technology has advanced, the use of in-vehicle infotainment systems has increased, providing many functions. However, if the driver\u2019s attention is diverted to control these systems, it can cause a fatal accident, and thus human\u2013vehicle interaction is becoming more important. Therefore, in this paper, we propose a human\u2013vehicle interaction system to reduce driver distraction during driving. We used voice and continuous-wave radar sensors that require low complexity for application to vehicle environments as resource-constrained platforms. The proposed system applies sensor fusion techniques to improve the limit of single-sensor monitoring. In addition, we used a binarized convolutional neural network algorithm, which significantly reduces the computational workload of the convolutional neural network in command classification. As a result of performance evaluation in noisy and cluttered environments, the proposed system showed a recognition accuracy of 96.4%, an improvement of 7.6% compared to a single voice sensor-based system, and 9.0% compared to a single radar sensor-based system.<\/jats:p>","DOI":"10.3390\/s21113906","type":"journal-article","created":{"date-parts":[[2021,6,7]],"date-time":"2021-06-07T01:56:40Z","timestamp":1623031000000},"page":"3906","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Command Recognition Using Binarized Convolutional Neural Network with Voice and Radar Sensors for Human-Vehicle Interaction"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4207-9418","authenticated-orcid":false,"given":"Seunghyun","family":"Oh","sequence":"first","affiliation":[{"name":"Department of Smart Drone Convergence, Korea Aerospace University, Goyang-si 10540, Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8100-4430","authenticated-orcid":false,"given":"Chanhee","family":"Bae","sequence":"additional","affiliation":[{"name":"Department of Smart Drone Convergence, Korea Aerospace University, Goyang-si 10540, Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8208-0419","authenticated-orcid":false,"given":"Jaechan","family":"Cho","sequence":"additional","affiliation":[{"name":"School of Electronics and Information Engineering, Korea Aerospace University, Goyang-si 10540, Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9344-7052","authenticated-orcid":false,"given":"Seongjoo","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Information and Communication Engineering and Convergence Engineering for Intelligent Drone, Sejong University, Seoul 05006, Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2299-9911","authenticated-orcid":false,"given":"Yunho","family":"Jung","sequence":"additional","affiliation":[{"name":"Department of Smart Drone Convergence, Korea Aerospace University, Goyang-si 10540, Korea"},{"name":"School of Electronics and Information Engineering, Korea Aerospace University, Goyang-si 10540, Korea"}]}],"member":"1968","published-online":{"date-parts":[[2021,6,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2662","DOI":"10.1109\/COMST.2017.2705027","article-title":"A survey on compressed sensing in vehicular infotainment systems","volume":"19","author":"Guo","year":"2017","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"8408","DOI":"10.1109\/TVT.2019.2930601","article-title":"Infotainment enabled smart cars: A joint communication, caching, and computation approach","volume":"68","author":"Kazmi","year":"2019","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"2368","DOI":"10.1109\/TITS.2014.2337331","article-title":"Hand gesture recognition in real time for automotive interfaces: A multimodal vision-based approach and evaluations","volume":"15","author":"Trivedi","year":"2014","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"May, K.R., Gable, T.M., and Walker, B.N. (2014, January 17\u201319). A multimodal air gesture interface for in vehicle menu navigation. Proceedings of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Seattle, WA, USA.","DOI":"10.1145\/2667239.2667280"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Deo, N., Rangesh, A., and Trivedi, M. (2016, January 1\u20134). In-vehicle hand gesture recognition using hidden markov models. Proceedings of the IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), Rio de Janeiro, Brazil.","DOI":"10.1109\/ITSC.2016.7795908"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Wang, H., Ye, Z., and Chen, J. (2018, January 26\u201329). A Front-End Speech Enhancement System for Robust Automotive Speech Recognition. Proceedings of the 2018 11th International Symposium on Chinese Spoken Language Processing (ISCSLP), Taipei, Taiwan.","DOI":"10.1109\/ISCSLP.2018.8706649"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Loh, C.Y., Boey, K.L., and Hong, K.S. (2017, January 10\u201312). Speech recognition interactive system for vehicle. Proceedings of the 2017 IEEE 13th International Colloquium on Signal Processing & Its Applications (CSPA), Batu Ferringhi, Malaysia.","DOI":"10.1109\/CSPA.2017.8064929"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Feng, X., Richardson, B., Amman, S., and Glass, J. (2015, January 19\u201324). On using heterogeneous data for vehicle-based speech recognition: A DNN-based approach. Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, Australia.","DOI":"10.1109\/ICASSP.2015.7178799"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/LSENS.2018.2810093","article-title":"Gesture recognition using mm-wave sensor for human-car interface","volume":"2","author":"Smith","year":"2018","journal-title":"IEEE Sens. Lett."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"167264","DOI":"10.1109\/ACCESS.2020.3023187","article-title":"A novel detection and recognition method for continuous hand gesture using fmcw radar","volume":"8","author":"Wang","year":"2020","journal-title":"IEEE Access"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Sun, Y., Fei, T., Schliep, F., and Pohl, N. (2018, January 15\u201317). Gesture classification with handcrafted micro-Doppler features using a FMCW radar. Proceedings of the 2018 IEEE MTT-S International Conference on Microwaves for Intelligent Mobility (ICMIM), Munich, Germany.","DOI":"10.1109\/ICMIM.2018.8443507"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Kopinski, T., Geisler, S., and Handmann, U. (2015, January 8\u201310). Gesture-based human-machine interaction for assistance systems. Proceedings of the 2015 IEEE International Conference on Information and Automation, Lijiang, China.","DOI":"10.1109\/ICInfA.2015.7279341"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Ne\u00dfelrath, R., Moniri, M.M., and Feld, M. (2016, January 14\u201316). Combining speech, gaze, and micro-gestures for the multimodal control of in-car functions. Proceedings of the 2016 12th International Conference on Intelligent Environments (IE), London, UK.","DOI":"10.1109\/IE.2016.42"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Tateno, S., Zhu, Y., and Meng, F. (2019, January 10\u201313). Hand gesture recognition system for in-car device control based on infrared array sensor. Proceedings of the 2019 58th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE), Hiroshima, Japan.","DOI":"10.23919\/SICE.2019.8859832"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Ahmed, S., Khan, F., Ghaffar, A., Hussain, F., and Cho, S.H. (2019). Finger-counting-based gesture recognition within cars using impulse radar with convolutional neural network. Sensors, 19.","DOI":"10.3390\/s19061429"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Khan, F., Leem, S.K., and Cho, S.H. (2017). Hand-Based Gesture Recognition for Vehicular Applications Using IR-UWB Radar. Sensors, 17.","DOI":"10.3390\/s17040833"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"7125","DOI":"10.1109\/ACCESS.2016.2617282","article-title":"Hand gesture recognition using micro-Doppler signatures with convolutional neural network","volume":"4","author":"Kim","year":"2016","journal-title":"IEEE Access"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Molchanov, P., Gupta, S., Kim, K., and Pulli, K. (2015, January 4\u20138). Multi-sensor system for driver\u2019s hand-gesture recognition. Proceedings of the 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Ljubljana, Slovenia.","DOI":"10.1109\/FG.2015.7163132"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"M\u00fcnzner, S., Schmidt, P., Reiss, A., Hanselmann, M., Stiefelhagen, R., and D\u00fcrichen, R. (2017, January 11\u201315). CNN-based sensor fusion techniques for multimodal human activity recognition. Proceedings of the 2017 ACM International Symposium on Wearable Computers (ISWC\u201917), Maui, HI, USA.","DOI":"10.1145\/3123021.3123046"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Alay, N., and Al-Baity, H.H. (2020). Deep Learning Approach for Multimodal Biometric Recognition System Based on Fusion of Iris, Face, and Finger Vein Traits. Sensors, 20.","DOI":"10.3390\/s20195523"},{"key":"ref_21","first-page":"777","article-title":"Design and Implementation of CNN-based HMI System using Doppler Radar and Voice Sensor","volume":"24","author":"Oh","year":"2020","journal-title":"J. IKEEE"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Nayak, P., Zhang, D., and Chai, S. (2019). Bit efficient quantization for deep neural networks. arXiv.","DOI":"10.1109\/EMC2-NIPS53020.2019.00020"},{"key":"ref_23","unstructured":"Jain, A., Bhattacharya, S., Masuda, M., Sharma, V., and Wang, Y. (2020). Efficient execution of quantized deep learning models: A compiler approach. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"102994","DOI":"10.1016\/j.micpro.2020.102994","article-title":"Towards resource-frugal deep convolutional neural networks for hyperspectral image segmentation","volume":"73","author":"Nalepa","year":"2020","journal-title":"Microprocess. Microsyst."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Simons, T., and Lee, D.J. (2019). A Review of Binarized Neural Networks. Electronics, 8.","DOI":"10.3390\/electronics8060661"},{"key":"ref_26","unstructured":"Lin, X., Zhao, C., and Pan, W. (2017, January 4\u20139). Towards accurate binary convolutional neural network. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. (2016, January 11\u201314). Xnor-net: Imagenet classification using binary convolutional neural networks. Proceedings of the European Conference on Computer Vision (ECCV 2016), Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46493-0_32"},{"key":"ref_28","unstructured":"Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., and Bengio, Y. (2016). Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or \u22121. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Cho, J., Jung, Y., Lee, S., and Jung, Y. (2021). Reconfigurable Binary Neural Network Accelerator with Adaptive Parallelism Scheme. Electronics, 10.","DOI":"10.3390\/electronics10030230"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Yin, Y., Liu, L., and Sun, X. (2011, January 3\u20134). SDUMLA-HMT: A multimodal biometric database. Proceedings of the Chinese Conference on Biometric Recognition, Beijing, China.","DOI":"10.1007\/978-3-642-25449-9_33"},{"key":"ref_31","unstructured":"(2021, March 02). MVL Lavalier Microphone for Smartphone or Tablet. Available online: https:\/\/www.shure.com\/en-US\/products\/microphones\/mvl."},{"key":"ref_32","unstructured":"(2021, March 02). 24 GHz Transceiver: BGT24LTR11. Available online: https:\/\/www.infineon.com\/dgdl\/Infineon-AN598_Sense2GOL_Pulse-ApplicationNotes-v01_00-EN.pdf?fileId=5546d4626e651a41016e82b630bc1571."},{"key":"ref_33","unstructured":"O\u2019Shea, K., and Nash, R. (2015). An introduction to convolutional neural networks. arXiv."},{"key":"ref_34","first-page":"1097","article-title":"Imagenet classification with deep convolutional neural networks","volume":"25","author":"Krizhevsky","year":"2012","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7\u201312). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_36","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Hu, J., Shen, L., and Sun, G. (2018, January 18\u201321). Squeeze-and-excitation networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA.","DOI":"10.1109\/CVPR.2018.00745"},{"key":"ref_38","unstructured":"Ioffe, S., and Szegedy, C. (2015, January 6\u201311). Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the International Conference on Machine Learning, Lille, France."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Suganuma, M., Shirakawa, S., and Nagao, T. (2017, January 15\u201319). A genetic programming approach to designing convolutional neural network architectures. Proceedings of the Genetic and Evolutionary Computation Conference, Berlin, Germany.","DOI":"10.1145\/3071178.3071229"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Lorenzo, P.R., Nalepa, J., Kawulok, M., Ramos, L.S., and Ranilla Pastor, J. (2017, January 15\u201319). Particle swarm optimization for hyper-parameter selection in deep neural networks. Proceedings of the Genetic and Evolutionary Computation Conference, Berlin, Germany.","DOI":"10.1145\/3071178.3071208"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/11\/3906\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:11:14Z","timestamp":1760163074000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/11\/3906"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,5]]},"references-count":40,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2021,6]]}},"alternative-id":["s21113906"],"URL":"https:\/\/doi.org\/10.3390\/s21113906","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2021,6,5]]}}}