{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,25]],"date-time":"2026-01-25T13:30:05Z","timestamp":1769347805397,"version":"3.49.0"},"reference-count":46,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2021,12,22]],"date-time":"2021-12-22T00:00:00Z","timestamp":1640131200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Cranfield\u2019s EPSRC Impact Acceleration Account","award":["EP\/R511511\/1"],"award-info":[{"award-number":["EP\/R511511\/1"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In conditionally automated driving, the engagement of non-driving activities (NDAs) can be regarded as the main factor that affects the driver\u2019s take-over performance, the investigation of which is of great importance to the design of an intelligent human\u2013machine interface for a safe and smooth control transition. This paper introduces a 3D convolutional neural network-based system to recognize six types of driver behaviour (four types of NDAs and two types of driving activities) through two video feeds based on head and hand movement. Based on the interaction of driver and object, the selected NDAs are divided into active mode and passive mode. The proposed recognition system achieves 85.87% accuracy for the classification of six activities. The impact of NDAs on the perspective of the driver\u2019s situation awareness and take-over quality in terms of both activity type and interaction mode is further investigated. The results show that at a similar level of achieved maximum lateral error, the engagement of NDAs demands more time for drivers to accomplish the control transition, especially for the active mode NDAs engagement, which is more mentally demanding and reduces drivers\u2019 sensitiveness to the driving situation change. Moreover, the haptic feedback torque from the steering wheel could help to reduce the time of the transition process, which can be regarded as a productive assistance system for the take-over process.<\/jats:p>","DOI":"10.3390\/s22010042","type":"journal-article","created":{"date-parts":[[2021,12,23]],"date-time":"2021-12-23T02:02:57Z","timestamp":1640224977000},"page":"42","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["The Identification of Non-Driving Activities with Associated Implication on the Take-Over Process"],"prefix":"10.3390","volume":"22","author":[{"given":"Lichao","family":"Yang","sequence":"first","affiliation":[{"name":"School of Aerospace, Transport and Manufacturing, Cranfield University, Bedford MK43 0AL, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mahdi","family":"Babayi Semiromi","sequence":"additional","affiliation":[{"name":"School of Aerospace, Transport and Manufacturing, Cranfield University, Bedford MK43 0AL, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3786-2865","authenticated-orcid":false,"given":"Yang","family":"Xing","sequence":"additional","affiliation":[{"name":"School of Aerospace, Transport and Manufacturing, Cranfield University, Bedford MK43 0AL, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chen","family":"Lv","sequence":"additional","affiliation":[{"name":"School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"James","family":"Brighton","sequence":"additional","affiliation":[{"name":"School of Aerospace, Transport and Manufacturing, Cranfield University, Bedford MK43 0AL, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2383-5724","authenticated-orcid":false,"given":"Yifan","family":"Zhao","sequence":"additional","affiliation":[{"name":"School of Aerospace, Transport and Manufacturing, Cranfield University, Bedford MK43 0AL, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,12,22]]},"reference":[{"key":"ref_1","unstructured":"(2018). Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles (Standard No. SAE International Standard J3016_201806)."},{"key":"ref_2","unstructured":"National Transportation Safety Board (2017). Collision Between a Car Operating with Automated Vehicle Control Systems and a Tractor-Semitrailer Truck, National Transportation Safety Board."},{"key":"ref_3","unstructured":"National Transportation Safety Board (2018). Collision Between a Sport Utility Vehicle Operating with Partial Driving Automation and a Crash Attenuator, Mountain View, California, March 23, 2018, National Transportation Safety Board."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"196","DOI":"10.1016\/j.trf.2014.06.016","article-title":"Effects of adaptive cruise control and highly automated driving on workload and situation awareness: A review of the empirical evidence","volume":"27","author":"Happee","year":"2014","journal-title":"Transp. Res. Part F Traffic Psychol. Behav."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1016\/j.trf.2017.01.012","article-title":"Transition of control in a partially automated vehicle: Effects of anticipation and non-driving-related task involvement","volume":"46","author":"Dogan","year":"2017","journal-title":"Transp. Res. Part F Traffic Psychol. Behav."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"620","DOI":"10.1016\/j.trf.2018.11.015","article-title":"Non-driving-related tasks, workload, and takeover performance in highly automated driving contexts","volume":"60","author":"Yoon","year":"2019","journal-title":"Transp. Res. Part F Traffic Psychol. Behav."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"2048","DOI":"10.1109\/TITS.2018.2857222","article-title":"Detection and Evaluation of Driver Distraction Using Machine Learning and Fuzzy Logic","volume":"20","author":"Aksjonov","year":"2019","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"274","DOI":"10.1016\/j.aap.2018.12.005","article-title":"A hierarchical machine learning classification approach for secondary task identification from observed driving behavior data","volume":"123","author":"Osman","year":"2019","journal-title":"Accid. Anal. Prev."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Martin, M., Popp, J., Anneken, M., Voit, M., and Stiefelhagen, R. (2018, January 26\u201330). Body Pose and Context Information for Driver Secondary Task Detection. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China.","DOI":"10.1109\/IVS.2018.8500523"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1109\/TCSS.2017.2766884","article-title":"Identification and analysis of driver postures for in-vehicle driving activities and secondary tasks recognition","volume":"5","author":"Xing","year":"2018","journal-title":"IEEE Trans. Comput. Soc. Syst."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"980","DOI":"10.1109\/TITS.2015.2493451","article-title":"Detecting drivers\u2019 mirror-checking actions and its application to maneuver and secondary task recognition","volume":"17","author":"Li","year":"2016","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"65772","DOI":"10.1109\/ACCESS.2018.2878150","article-title":"Study on the impact degrees of several driving behaviors when driving while performing secondary tasks","volume":"6","author":"Jin","year":"2018","journal-title":"IEEE Access"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"4318","DOI":"10.1109\/TITS.2019.2939676","article-title":"A dual-cameras-based driver gaze mapping system with an application on non-driving activities monitoring","volume":"21","author":"Yang","year":"2020","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1016\/j.aap.2018.11.018","article-title":"The effects of takeover request modalities on highly automated car control transitions","volume":"123","author":"Yoon","year":"2019","journal-title":"Accid. Anal. Prev."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"468","DOI":"10.1016\/j.aap.2016.08.027","article-title":"Age differences in the takeover of vehicle control and engagement in non-driving-related activities in simulated driving with conditional automation","volume":"106","author":"Clark","year":"2017","journal-title":"Accid. Anal. Prev."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"470","DOI":"10.1080\/03081060.2019.1609221","article-title":"Investigating the effects of age and disengagement in driving on driver\u2019s takeover control performance in highly automated vehicles","volume":"42","author":"Li","year":"2019","journal-title":"Transp. Plan. Technol."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"2063","DOI":"10.1177\/1541931214581434","article-title":"How traffic situations and non-driving related tasks affect the take-over quality in highly automated driving","volume":"58","author":"Radlmayr","year":"2014","journal-title":"Proc. Hum. Factors Ergon. Soc. Annu. Meet."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Scharfe, M.S.L., Zeeb, K., and Russwinkel, N. (2020). The impact of situational complexity and familiarity on takeover quality in uncritical highly automated driving scenarios. Information, 11.","DOI":"10.3390\/info11020115"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"136924","DOI":"10.1109\/ACCESS.2019.2914864","article-title":"Take-over performance and safety analysis under different scenarios and secondary tasks in conditionally automated driving","volume":"7","author":"Wu","year":"2019","journal-title":"IEEE Access"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"870","DOI":"10.1177\/0018720818768199","article-title":"Effects of non-driving related task modalities on takeover performance in highly automated driving","volume":"60","author":"Wandtner","year":"2018","journal-title":"Hum. Factors J. Hum. Factors Ergon. Soc."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1016\/j.trf.2018.10.015","article-title":"Effects of non-driving-related-task modality and road geometry on eye movements, lane-keeping performance, and workload while driving","volume":"60","author":"Jeong","year":"2019","journal-title":"Transp. Res. Part F Traffic Psychol. Behav."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Kim, J., Kim, H.-S., Kim, W., and Yoon, D. (2018, January 17\u201319). Take-over performance analysis depending on the drivers\u2019 non-driving secondary tasks in automated vehicles. Proceedings of the 2018 International Conference on Information and Communication Technology Convergence (ICTC), Jeju Island, Korea.","DOI":"10.1109\/ICTC.2018.8539431"},{"key":"ref_23","first-page":"230","article-title":"Is take-over time all that matters?","volume":"92","author":"Zeeb","year":"2016","journal-title":"The impact of visual-cognitive load on driver take-over quality after conditionally automated driving. Accid. Anal. Prev."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Petermeijer, S., Doubek, F., and de Winter, J. (2017, January 5\u20138). Driver response times to auditory, visual, and tactile take-over requests: A simulator study with 101 participants. Proceedings of the 2017 IEEE International Conference on Systems, Man and Cybernetics (SMC),  Banff, AB, Canada.","DOI":"10.1109\/SMC.2017.8122827"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Kim, H., Kim, W., Kim, J., and Yoon, D. (2019, January 5\u20137). A study on the control authority transition characteristics by driver information. Proceedings of the 2019 International Conference on Computational Science and Computational Intelligence (CSCI), Las Vegas, NV, USA.","DOI":"10.1109\/CSCI49370.2019.00297"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"103074","DOI":"10.1016\/j.apergo.2020.103074","article-title":"Effects of cognitive and visual loads on driving performance after take-over request (TOR) in automated driving","volume":"85","author":"Choi","year":"2020","journal-title":"Appl. Ergon."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Bueno, M., Dogan, E., Hadj Selem, F., Monacelli, E., Boverie, S., and Guillaume, A. (2016, January 1\u20134). How different mental workload levels affect the take-over control after automated driving. Proceedings of the 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), Rio de Janeiro, Brazil.","DOI":"10.1109\/ITSC.2016.7795886"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"129","DOI":"10.4271\/12-02-02-0009","article-title":"Situational awareness, driver\u2019s trust in automated driving systems and secondary task performance","volume":"2","author":"Petersen","year":"2019","journal-title":"SAE Int. J. Connect. Autom. Veh."},{"key":"ref_29","unstructured":"Wandtner, B., Schmidt, G., Schoemig, N., and Kunde, W. (2018, January 7\u20138). Non-driving related tasks in highly automated driving\u2014Effects of task modalities and cognitive workload on take-over performance. Proceedings of the AmE 2018\u2014Automotive meets Electronics, 9th GMM-Symposium, Dortmund, Germany."},{"key":"ref_30","unstructured":"Sivak, M., and Schoettle, B. (2015). Motion Sickness in Self-Driving Vehicles, University of Michigan, Transportation Research Institute."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1109\/TOH.2015.2437871","article-title":"The effect of haptic support systems on driver performance: A literature survey","volume":"8","author":"Petermeijer","year":"2015","journal-title":"IEEE Trans. Haptics"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"987","DOI":"10.1080\/10447318.2017.1404778","article-title":"The effects of vibration patterns of take-over request and non-driving tasks on taking-over control of automated vehicles","volume":"34","author":"Wan","year":"2018","journal-title":"Int. J. Hum. Comput. Interact."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"2558","DOI":"10.1109\/TMECH.2018.2812643","article-title":"Characterization of driver neuromuscular dynamics for human\u2013automation collaboration design of automated vehicles","volume":"23","author":"Lv","year":"2018","journal-title":"IEEE\/ASME Trans. Mechatron."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"897","DOI":"10.1109\/TITS.2015.2494873","article-title":"Vibrotactile Displays: A survey with a view on highly automated driving","volume":"17","author":"Petermeijer","year":"2016","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Lv, C., Wang, H., Cao, D., Zhao, Y., Sullman, M., Auger, D.J., Brighton, J., Matthias, R., Skrypchuk, L., and Mouzakitis, A. (2018, January 26\u201330). A novel control framework of haptic take-over system for automated vehicles. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China.","DOI":"10.1109\/IVS.2018.8500480"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Lv, C., Li, Y., Xing, Y., Huang, C., Cao, D., Zhao, Y., and Liu, Y. (2020). Human-Machine Collaboration for Automated Vehicles via an Intelligent Two-Phase Haptic Interface. arXiv.","DOI":"10.1002\/aisy.202170040"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"2014","DOI":"10.1109\/TITS.2015.2396031","article-title":"Driver gaze tracking and eyes off the road detection system","volume":"16","author":"Vicente","year":"2015","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1109\/MIS.2016.47","article-title":"Driver gaze region estimation without use of eye movement","volume":"31","author":"Fridman","year":"2016","journal-title":"IEEE Intell. Syst."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Naqvi, R.A., Arsalan, M., Batchuluun, G., Yoon, H.S., and Park, K.R. (2018). Deep learning-based gaze detection system for automobile drivers using a NIR camera sensor. Sensors, 18.","DOI":"10.3390\/s18020456"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"107955","DOI":"10.1016\/j.patcog.2021.107955","article-title":"Recognition of visual-related non-driving activities using a dual-camera monitoring system","volume":"116","author":"Yang","year":"2021","journal-title":"Pattern Recognit."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"15574","DOI":"10.1109\/JSEN.2020.3005810","article-title":"A refined non-driving activity classification using a two-stream convolutional neural network","volume":"21","author":"Yang","year":"2020","journal-title":"IEEE Sens. J."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"5379","DOI":"10.1109\/TVT.2019.2908425","article-title":"Driver activity recognition for intelligent vehicles: A deep learning approach","volume":"68","author":"Xing","year":"2019","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1109\/TPAMI.2012.59","article-title":"3D Convolutional neural networks for human action recognition","volume":"35","author":"Ji","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Hara, K., Kataoka, H., and Satoh, Y. (2017, January 22\u201329). Learning spatio-temporal features with 3D residual networks for action recognition. Proceedings of the 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy.","DOI":"10.1109\/ICCVW.2017.373"},{"key":"ref_45","unstructured":"Du, X., Li, Y., Cui, Y., Qian, R., Li, J., and Bello, I. (2021). Revisiting 3D ResNets for video recognition. arXiv."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2015, January 7\u201313). Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.123"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/1\/42\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:51:12Z","timestamp":1760169072000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/1\/42"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,22]]},"references-count":46,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,1]]}},"alternative-id":["s22010042"],"URL":"https:\/\/doi.org\/10.3390\/s22010042","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,22]]}}}