{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,5]],"date-time":"2025-11-05T21:15:57Z","timestamp":1762377357862,"version":"build-2065373602"},"reference-count":38,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2023,4,20]],"date-time":"2023-04-20T00:00:00Z","timestamp":1681948800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Sleep-deprived fatigued person is likely to commit more errors that may even prove to be fatal. Thus, it is necessary to recognize this fatigue. The novelty of the proposed research work for the detection of this fatigue is that it is nonintrusive and based on multimodal feature fusion. In the proposed methodology, fatigue is detected by obtaining features from four domains: visual images, thermal images, keystroke dynamics, and voice features. In the proposed methodology, the samples of a volunteer (subject) are obtained from all four domains for feature extraction, and empirical weights are assigned to the four different domains. Young, healthy volunteers (n = 60) between the age group of 20 to 30 years participated in the experimental study. Further, they abstained from the consumption of alcohol, caffeine, or other drugs impacting their sleep pattern during the study. Through this multimodal technique, appropriate weights are given to the features obtained from the four domains. The results are compared with k-nearest neighbors (kNN), support vector machines (SVM), random tree, random forest, and multilayer perceptron classifiers. The proposed nonintrusive technique has obtained an average detection accuracy of 93.33% in 3-fold cross-validation.<\/jats:p>","DOI":"10.3390\/s23084129","type":"journal-article","created":{"date-parts":[[2023,4,20]],"date-time":"2023-04-20T05:35:46Z","timestamp":1681968946000},"page":"4129","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["A Multimodal Feature Fusion Framework for Sleep-Deprived Fatigue Detection to Prevent Accidents"],"prefix":"10.3390","volume":"23","author":[{"given":"Jitender Singh","family":"Virk","sequence":"first","affiliation":[{"name":"EIE Department, Thapar Institute of Engineering and Technology, Patiala 147001, India"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8134-5075","authenticated-orcid":false,"given":"Mandeep","family":"Singh","sequence":"additional","affiliation":[{"name":"EIE Department, Thapar Institute of Engineering and Technology, Patiala 147001, India"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4474-4581","authenticated-orcid":false,"given":"Mandeep","family":"Singh","sequence":"additional","affiliation":[{"name":"EIE Department, Thapar Institute of Engineering and Technology, Patiala 147001, India"}]},{"given":"Usha","family":"Panjwani","sequence":"additional","affiliation":[{"name":"DIPAS, Defence Research and Development Organisation, Delhi 110054, India"}]},{"given":"Koushik","family":"Ray","sequence":"additional","affiliation":[{"name":"DIPAS, Defence Research and Development Organisation, Delhi 110054, India"}]}],"member":"1968","published-online":{"date-parts":[[2023,4,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"396","DOI":"10.1016\/j.physbeh.2005.11.009","article-title":"Effects of sleep deprivation and exercise on cognitive, motor performance and mood","volume":"87","author":"Scott","year":"2006","journal-title":"Physiol. Behav."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/j.jsr.2005.11.003","article-title":"Vigilance monitoring for operator safety: A simulation study on highway driving","volume":"37","author":"Desai","year":"2006","journal-title":"J. Saf. Res."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"862","DOI":"10.1016\/j.aap.2005.04.004","article-title":"Sleepiness\/fatigue and distraction\/inattention as factors for fatal versus nonfatal commercial motor vehicle driver injuries","volume":"37","author":"Bunn","year":"2005","journal-title":"Accid. Anal. Prev."},{"key":"ref_4","first-page":"297","article-title":"Driver Drowsiness Measurement Technologies: Current Research, Market Solutions, and Challenges","volume":"18","author":"Doudou","year":"2019","journal-title":"Int. J. Intell. Transp. Syst. Res."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"898","DOI":"10.1016\/j.aap.2009.06.001","article-title":"Monitoring drivers\u2019 mental workload in driving simulators using physiological measures","volume":"42","author":"Brookhuis","year":"2009","journal-title":"Accid. Anal. Prev."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1007\/978-3-540-75773-3_2","article-title":"Drowsy Driver Detection Through Facial Movement Analysis","volume":"Volume 4796","author":"Vural","year":"2007","journal-title":"Proceedings of the Human\u2013Computer Interaction: IEEE International Workshop, HCI 2007"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Flores, M.J., Armingol, J.M., and de la Escalera, A. (2008, January 4\u20136). Real-time drowsiness detection system for an intelligent vehicle. Proceedings of the 2008 IEEE Intelligent Vehicles Symposium, Eindhoven, The Netherlands.","DOI":"10.1109\/IVS.2008.4621125"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"234","DOI":"10.1016\/j.patrec.2009.08.014","article-title":"Gabor-based dynamic representation for human fatigue monitoring in facial image sequences","volume":"31","author":"Fan","year":"2010","journal-title":"Pattern Recognit. Lett."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Sharma, N., Dhall, A., Gedeon, T., and Goecke, R. (2013, January 2\u20135). Modeling Stress Using Thermal Facial Patterns: A Spatio-Temporal Approach. Proceedings of the 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, Geneva, Switzerland.","DOI":"10.1109\/ACII.2013.70"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1007\/978-3-540-74200-5_6","article-title":"Speech Under Stress: Analysis, Modeling and Recognition","volume":"4343","author":"Hansen","year":"2007","journal-title":"Speak. Classif. I. Lect. Notes Comput. Sci."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"502","DOI":"10.1109\/TMM.2010.2058095","article-title":"Speech Emotion Analysis: Exploring the Role of Context","volume":"12","author":"Tawari","year":"2010","journal-title":"IEEE Trans. Multimed."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1088","DOI":"10.1016\/j.specom.2011.05.003","article-title":"Recognizing affect from speech prosody using hierarchical graphical models","volume":"53","author":"Fernandez","year":"2011","journal-title":"Speech Commun."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1186","DOI":"10.1016\/j.specom.2011.04.003","article-title":"Formant position based weighted spectral features for emotion recognition","volume":"53","author":"Bozkurt","year":"2011","journal-title":"Speech Commun."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1016\/j.neucom.2011.12.021","article-title":"Applying multiple classifiers and non-linear dynamics features for detecting sleepiness from speech","volume":"84","author":"Krajewski","year":"2012","journal-title":"Neurocomputing"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1016\/j.csl.2012.12.003","article-title":"Vocal fatigue induced by prolonged oral reading: Analysis and detection","volume":"28","author":"Caraty","year":"2014","journal-title":"Comput. Speech Lang."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"116","DOI":"10.1016\/j.cviu.2006.10.019","article-title":"Multimodal human-computer interaction: A survey","volume":"108","author":"Jaimes","year":"2007","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"870","DOI":"10.1016\/j.ijhcs.2009.07.005","article-title":"Automated stress detection using keystroke and linguistic features: An exploratory study","volume":"67","author":"Vizer","year":"2009","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Lim, Y.M., Ayesh, A., and Stacey, M. (2014, January 27\u201329). Detecting cognitive stress from keyboard and mouse dynamics during mental arithmetic. Proceedings of the 2014 Science and Information Conference, London, UK.","DOI":"10.1109\/SAI.2014.6918183"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"987","DOI":"10.1080\/0144929X.2014.907343","article-title":"Identifying emotion by keystroke dynamics and text pattern analysis","volume":"33","author":"Alam","year":"2014","journal-title":"Behav. Inf. Technol."},{"key":"ref_20","first-page":"26","article-title":"A comprehensive review of approaches to detect fatigue using machine learning techniques","volume":"8","author":"Hooda","year":"2021","journal-title":"Chronic Dis. Transl. Med."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1016\/j.inffus.2017.11.005","article-title":"A sensor fusion approach for drowsiness detection in wearable ultra-low-power systems","volume":"43","author":"Kartsch","year":"2018","journal-title":"Inf. Fusion"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"103360","DOI":"10.1016\/j.bspc.2021.103360","article-title":"Identification of mental fatigue levels in a language understanding task based on multi-domain EEG features and an ensemble convolutional neural network","volume":"72","author":"Ye","year":"2022","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Wu, N., and Sun, J. (2022). Fatigue Detection of Air Traffic Controllers Based on Radiotelephony Communications and Self-Adaption Quantum Genetic Algorithm Optimization Ensemble Learning. Appl. Sci., 12.","DOI":"10.3390\/app122010252"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"8032673","DOI":"10.1155\/2022\/8032673","article-title":"Deploying Machine Learning Techniques for Human Emotion Detection","volume":"2022","author":"Siam","year":"2022","journal-title":"Comput. Intell. Neurosci."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"7463091","DOI":"10.1155\/2022\/7463091","article-title":"Human-Computer Interaction with Detection of Speaker Emotions Using Convolution Neural Networks","volume":"2022","author":"Alnuaim","year":"2022","journal-title":"Comput. Intell. Neurosci."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1007\/s11571-020-09601-w","article-title":"A dynamic center and multi threshold point based stable feature extraction network for driver fatigue detection utilizing EEG signals","volume":"15","author":"Tuncer","year":"2021","journal-title":"Cogn. Neurodyn."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"456","DOI":"10.1080\/08839514.2020.1723875","article-title":"Driver Fatigue Detection Using Viola Jones and Principal Component Analysis","volume":"34","author":"Fatima","year":"2020","journal-title":"Appl. Artif. Intell."},{"key":"ref_28","first-page":"1","article-title":"A comparison of methods used for inducing mental fatigue in performance research: Individualised, dual-task and short duration cognitive tests are most effective","volume":"63","author":"Hodder","year":"2020","journal-title":"Ergon. Taylor Fr."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1190","DOI":"10.1080\/13854046.2020.1787522","article-title":"Predicting changes in performance due to cognitive fatigue: A multimodal approach based on speech motor coordination and electrodermal activity","volume":"34","author":"Heaton","year":"2020","journal-title":"Clin. Neuropsychol."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"2339","DOI":"10.1109\/TITS.2018.2868499","article-title":"Driver Fatigue Detection Systems: A Review","volume":"20","author":"Sikander","year":"2019","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.future.2022.08.009","article-title":"FedSup: A communication-efficient federated learning fatigue driving behaviors supervision approach","volume":"138","author":"Zhao","year":"2023","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"104237","DOI":"10.1016\/j.bspc.2022.104237","article-title":"Recognising drivers\u2019 mental fatigue based on EEG multi-dimensional feature selection and fusion","volume":"79","author":"Zhang","year":"2022","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Chen, J., Yan, M., Zhu, F., Xu, J., Li, H., and Sun, X. (2022). Fatigue Driving Detection Method Based on Combination of BP Neural Network and Time Cumulative Effect. Sensors, 22.","DOI":"10.3390\/s22134717"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Li, Y., Zhang, S., Zhu, G., Huang, Z., Wang, R., Duan, X., and Wang, Z. (2023). A CNN-Based Wearable System for Driver Drowsiness Detection. Sensors, 23.","DOI":"10.3390\/s23073475"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/j.inffus.2020.01.011","article-title":"Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review","volume":"59","author":"Zhang","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1016\/j.inffus.2021.12.004","article-title":"Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network","volume":"82","author":"Tang","year":"2022","journal-title":"Inf. Fusion"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1016\/j.inffus.2013.05.007","article-title":"An information fusion based method for liver classification using texture analysis of ultrasound images","volume":"19","author":"Singh","year":"2014","journal-title":"Inf. Fusion"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Dai, L., Li, Y., and Zhang, M. (2023). Detection of Operator Fatigue in the Main Control Room of a Nuclear Power Plant Based on Eye Blink Rate, PERCLOS and Mouse Velocity. Appl. Sci., 13.","DOI":"10.3390\/app13042718"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/8\/4129\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:19:43Z","timestamp":1760123983000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/8\/4129"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,20]]},"references-count":38,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2023,4]]}},"alternative-id":["s23084129"],"URL":"https:\/\/doi.org\/10.3390\/s23084129","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2023,4,20]]}}}