{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T13:42:55Z","timestamp":1773841375856,"version":"3.50.1"},"reference-count":40,"publisher":"MDPI AG","issue":"13","license":[{"start":{"date-parts":[[2022,6,22]],"date-time":"2022-06-22T00:00:00Z","timestamp":1655856000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["52005435"],"award-info":[{"award-number":["52005435"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["BK20190873"],"award-info":[{"award-number":["BK20190873"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["JGLX2021_002"],"award-info":[{"award-number":["JGLX2021_002"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["SJCX21_1557"],"award-info":[{"award-number":["SJCX21_1557"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004608","name":"Natural Science Foundation of Jiangsu Province","doi-asserted-by":"publisher","award":["52005435"],"award-info":[{"award-number":["52005435"]}],"id":[{"id":"10.13039\/501100004608","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004608","name":"Natural Science Foundation of Jiangsu Province","doi-asserted-by":"publisher","award":["BK20190873"],"award-info":[{"award-number":["BK20190873"]}],"id":[{"id":"10.13039\/501100004608","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004608","name":"Natural Science Foundation of Jiangsu Province","doi-asserted-by":"publisher","award":["JGLX2021_002"],"award-info":[{"award-number":["JGLX2021_002"]}],"id":[{"id":"10.13039\/501100004608","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004608","name":"Natural Science Foundation of Jiangsu Province","doi-asserted-by":"publisher","award":["SJCX21_1557"],"award-info":[{"award-number":["SJCX21_1557"]}],"id":[{"id":"10.13039\/501100004608","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Graduate Education Reform and Practice Project of Yangzhou University","award":["52005435"],"award-info":[{"award-number":["52005435"]}]},{"name":"Graduate Education Reform and Practice Project of Yangzhou University","award":["BK20190873"],"award-info":[{"award-number":["BK20190873"]}]},{"name":"Graduate Education Reform and Practice Project of Yangzhou University","award":["JGLX2021_002"],"award-info":[{"award-number":["JGLX2021_002"]}]},{"name":"Graduate Education Reform and Practice Project of Yangzhou University","award":["SJCX21_1557"],"award-info":[{"award-number":["SJCX21_1557"]}]},{"name":"Postgraduate Research and Practice Innovation Program of Jiangsu Province, China","award":["52005435"],"award-info":[{"award-number":["52005435"]}]},{"name":"Postgraduate Research and Practice Innovation Program of Jiangsu Province, China","award":["BK20190873"],"award-info":[{"award-number":["BK20190873"]}]},{"name":"Postgraduate Research and Practice Innovation Program of Jiangsu Province, China","award":["JGLX2021_002"],"award-info":[{"award-number":["JGLX2021_002"]}]},{"name":"Postgraduate Research and Practice Innovation Program of Jiangsu Province, China","award":["SJCX21_1557"],"award-info":[{"award-number":["SJCX21_1557"]}]},{"name":"Lvyang Jinfeng Plan for Excellent Doctors of Yangzhou City","award":["52005435"],"award-info":[{"award-number":["52005435"]}]},{"name":"Lvyang Jinfeng Plan for Excellent Doctors of Yangzhou City","award":["BK20190873"],"award-info":[{"award-number":["BK20190873"]}]},{"name":"Lvyang Jinfeng Plan for Excellent Doctors of Yangzhou City","award":["JGLX2021_002"],"award-info":[{"award-number":["JGLX2021_002"]}]},{"name":"Lvyang Jinfeng Plan for Excellent Doctors of Yangzhou City","award":["SJCX21_1557"],"award-info":[{"award-number":["SJCX21_1557"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Fatigue driving has always received a lot of attention, but few studies have focused on the fact that human fatigue is a cumulative process over time, and there are no models available to reflect this phenomenon. Furthermore, the problem of incorrect detection due to facial expression is still not well addressed. In this article, a model based on BP neural network and time cumulative effect was proposed to solve these problems. Experimental data were used to carry out this work and validate the proposed method. Firstly, the Adaboost algorithm was applied to detect faces, and the Kalman filter algorithm was used to trace the face movement. Then, a cascade regression tree-based method was used to detect the 68 facial landmarks and an improved method combining key points and image processing was adopted to calculate the eye aspect ratio (EAR). After that, a BP neural network model was developed and trained by selecting three characteristics: the longest period of continuous eye closure, number of yawns, and percentage of eye closure time (PERCLOS), and then the detection results without and with facial expressions were discussed and analyzed. Finally, by introducing the Sigmoid function, a fatigue detection model considering the time accumulation effect was established, and the drivers\u2019 fatigue state was identified segment by segment through the recorded video. Compared with the traditional BP neural network model, the detection accuracies of the proposed model without and with facial expressions increased by 3.3% and 8.4%, respectively. The number of incorrect detections in the awake state also decreased obviously. The experimental results show that the proposed model can effectively filter out incorrect detections caused by facial expressions and truly reflect that driver fatigue is a time accumulating process.<\/jats:p>","DOI":"10.3390\/s22134717","type":"journal-article","created":{"date-parts":[[2022,6,22]],"date-time":"2022-06-22T23:11:19Z","timestamp":1655939479000},"page":"4717","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":24,"title":["Fatigue Driving Detection Method Based on Combination of BP Neural Network and Time Cumulative Effect"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2097-5740","authenticated-orcid":false,"given":"Jian","family":"Chen","sequence":"first","affiliation":[{"name":"School of Mechanical Engineering, Yangzhou University, Yangzhou 225127, China"}]},{"given":"Ming","family":"Yan","sequence":"additional","affiliation":[{"name":"School of Mechanical Engineering, Yangzhou University, Yangzhou 225127, China"}]},{"given":"Feng","family":"Zhu","sequence":"additional","affiliation":[{"name":"School of Mechanical Engineering, Yangzhou University, Yangzhou 225127, China"}]},{"given":"Jing","family":"Xu","sequence":"additional","affiliation":[{"name":"School of Mechanical Engineering, Yangzhou University, Yangzhou 225127, China"}]},{"given":"Hai","family":"Li","sequence":"additional","affiliation":[{"name":"School of Mechanical Engineering, Yangzhou University, Yangzhou 225127, China"}]},{"given":"Xiaoguang","family":"Sun","sequence":"additional","affiliation":[{"name":"School of Mechanical Engineering, Yangzhou University, Yangzhou 225127, China"}]}],"member":"1968","published-online":{"date-parts":[[2022,6,22]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"16937","DOI":"10.3390\/s121216937","article-title":"Detecting Driver Drowsiness Based on Sensors: A Review","volume":"12","author":"Sahayadhas","year":"2012","journal-title":"Sensors"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Zhao, Z., Zhou, N., Zhang, L., Yan, H., Xu, Y., and Zhang, Z. (2022, May 20). Driver Fatigue Detection Based on Convolutional Neural Networks Using EM-CNN. Available online: https:\/\/www.hindawi.com\/journals\/cin\/2020\/7251280\/.","DOI":"10.1155\/2020\/7251280"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"24191","DOI":"10.3390\/s150924191","article-title":"A Self-Adaptive Dynamic Recognition Model for Fatigue Driving Based on Multi-Source Information and Two Levels of Fusion","volume":"15","author":"Sun","year":"2015","journal-title":"Sensors"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"5383573","DOI":"10.1155\/2021\/5383573","article-title":"Driver Fatigue Detection Based on Facial Key Points and LSTM","volume":"2021","author":"Chen","year":"2021","journal-title":"Secur. Commun. Netw."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"729","DOI":"10.1007\/978-981-10-8506-2_49","article-title":"Driver Fatigue Detection Based on SVM and Steering Wheel Angle Characteristics","volume":"Volume 486","author":"Zhang","year":"2017","journal-title":"SAE-China 2017: Proceedings of the 19th Asia Pacific Automotive Engineering Conference & SAE-China Congress 2017: Selected Papers"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Li, F., Wang, X.W., and Lu, B.L. (2013). Detection of Driving Fatigue Based on Grip Force on Steering Wheel with Wavelet Transformation and Support Vector Machine. ICONIP 2013: Neural Information Processing, Springer.","DOI":"10.1007\/978-3-642-42051-1_19"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"He, Q., Li, W., and Fan, X. (2011). Estimation of Driver\u2019s Fatigue Based on Steering Wheel Angle. EPCE 2011: Engineering Psychology and Cognitive Ergonomics, Springer.","DOI":"10.1007\/978-3-642-21741-8_17"},{"key":"ref_8","unstructured":"Papadelis, C., Kourtidou-Papadeli, C., Bamidis, P.D., Chouvarda, I., Koufogiannis, D., Bekiaris, E., and Maglaveras, N. (September, January 30). Indicators of Sleepiness in an Ambulatory EEG Study of Night Driving. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, New York, NY, USA."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"996","DOI":"10.1016\/j.eswa.2010.07.109","article-title":"Comparing Combinations of EEG Activity in Train Drivers During Monotonous Driving","volume":"38","author":"Jap","year":"2011","journal-title":"Expert Syst. Appl."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1582","DOI":"10.1109\/TBME.2008.918566","article-title":"Development of Wireless Brain Computer Interface with Embedded Multitask Scheduling and its Application on Real-time Driver\u2019s Drowsiness Detection and Warning","volume":"55","author":"Lin","year":"2008","journal-title":"IEEE Trans. Bio-Med. Eng."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"8669","DOI":"10.1016\/j.eswa.2015.07.021","article-title":"Physiological Signal based Detection of Driver Hypovigilance using Higher Order Spectra","volume":"42","author":"Sahayadhas","year":"2015","journal-title":"Expert Syst. Appl."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1942","DOI":"10.1016\/j.ins.2010.01.011","article-title":"A Driver Fatigue Recognition Model Based on Information Fusion and Dynamic Bayesian Network","volume":"180","author":"Yang","year":"2010","journal-title":"Inf. Sci."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1109\/TCDS.2018.2889223","article-title":"A Regression Method with Subnetwork Neurons for Vigilance Estimation Using EOG and EEG","volume":"13","author":"Wu","year":"2018","journal-title":"IEEE Trans. Cogn. Dev. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Qi, M., Yang, W., Xie, P., Liu, Z., Zhang, Y., and Cheng, S. (December, January 30). Driver fatigue Assessment Based on the Feature Fusion and Transfer Learning of EEG and EMG. Proceedings of the 2018 Chinese Automation Congress, Xi\u2019an, China.","DOI":"10.1109\/CAC.2018.8623087"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"790292","DOI":"10.3389\/fphys.2021.790292","article-title":"Fatigue Monitoring Through Wearables: A State-of-the-Art Review","volume":"12","author":"Annaheim","year":"2021","journal-title":"Front. Physiol."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Dziuda, \u0141., Baran, P., Zieli\u0144ski, P., Murawski, K., Dziwosz, M., Krej, M., Piotrowski, M., Stablewski, R., Wojdas, A., and Strus, W. (2021). Evaluation of a Fatigue Detector Using Eye Closure-Associated Indicators Acquired from Truck Drivers in a Simulator Study. Sensors, 21.","DOI":"10.3390\/s21196449"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"25495","DOI":"10.1007\/s11042-021-10930-z","article-title":"Real-time Detection Method of Driver Fatigue State Based on Deep Learning of Face Video","volume":"80","author":"Cui","year":"2021","journal-title":"Multimed. Tools Appl."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"162805","DOI":"10.1109\/ACCESS.2021.3131601","article-title":"Early Identification and Detection of Driver Drowsiness by Hybrid Machine Learning","volume":"9","author":"Altameem","year":"2021","journal-title":"IEEE Access"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"179396","DOI":"10.1109\/ACCESS.2019.2958667","article-title":"A Real-time Driving Drowsiness Detection Algorithm with Individual Differences Consideration","volume":"7","author":"You","year":"2019","journal-title":"IEEE Access"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"64136","DOI":"10.1109\/ACCESS.2019.2917382","article-title":"Fatigue State Detection Based on Multi-Index Fusion and State Recognition Network","volume":"7","author":"Ji","year":"2019","journal-title":"IEEE Access"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Zhang, F., Su, J., Geng, L., and Xiao, Z. (2017, January 17\u201319). Driver Fatigue Detection Based on Eye State Recognition. Proceedings of the 2017 International Conference on Machine Vision and Information Technology (CMVIT), Singapore.","DOI":"10.1109\/CMVIT.2017.25"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Jie, Z., Mahmoud, M., Stafford-Fraser, Q., Robinson, P., Dias, E., and Skrypchuk, L. (2018, January 15\u201319). Analysis of Yawning Behaviour in Spontaneous Expressions of Drowsy Drivers. Proceedings of the 13th IEEE International Conference on Automatic Face and Gesture Recognition, Xi\u2019an, China.","DOI":"10.1109\/FG.2018.00091"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Sava\u015f, B.K., and Becerikli, Y. (2018, January 25\u201327). Real Time Driver Fatigue Detection Based on SVM Algorithm. Proceedings of the 6th International Conference on Control Engineering and Information Technology (CEIT), Istanbul, Turkey.","DOI":"10.1109\/CEIT.2018.8751886"},{"key":"ref_24","first-page":"1282","article-title":"Real-Time Fatigue Detection System using Computer Vision","volume":"9","author":"Tiwari","year":"2020","journal-title":"Int. J. Eng. Res. Technol."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"102723","DOI":"10.1016\/j.jvcir.2019.102723","article-title":"Driver Fatigue Detection Based on Deeply-learned Facial Expression Representation","volume":"71","author":"Liu","year":"2020","journal-title":"J. Vis. Commun. Image Represent."},{"key":"ref_26","unstructured":"Zhang, W., Murphey, Y.L., Wang, T., and Xu, Q. (2015, January 12\u201316). Driver Yawning Detection Based on Deep Convolutional Neural Learning and Robust Nose Tracking. Proceedings of the 2015 International Joint Conference on Neural Networks, Killarney, Ireland."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"624","DOI":"10.1109\/TIM.2016.2640458","article-title":"A Driver State Detection System-Combining a Capacitive Hand Detection Sensor with Physiological Sensors","volume":"66","author":"Mosa","year":"2017","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"29375","DOI":"10.1007\/s11042-020-09479-0","article-title":"Analysis of RGB-D Camera Technologies for Supporting Different Facial Usage Scenarios","volume":"79","author":"Ulrich","year":"2020","journal-title":"Multimed. Tools Appl."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"3089","DOI":"10.1109\/TITS.2020.2979527","article-title":"Vision-Based Fatigue Driving Recognition Method Integrating Heart Rate and Facial Features","volume":"22","author":"Du","year":"2021","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Abtahi, S., Omidyeganeh, M., Shirmohammadi, S., and Hariri, B. (2014, January 19\u201321). YawDD: A Yawning Detection Dataset. Proceedings of the 5th ACM Multimedia Systems Conference, Singapore.","DOI":"10.1145\/2557642.2563678"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Belhumeur, P.N., Jacobs, D.W., Kriegman, D.J., and Kumar, N. (2011, January 20\u201325). Localizing Parts of Faces Using a Consensus of Exemplars. Proceedings of the 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, USA.","DOI":"10.1109\/CVPR.2011.5995602"},{"key":"ref_32","unstructured":"(2022, May 27). labelImg 1.8.6. Pip Install Labelimg\u2014Project Description. Available online: https:\/\/pypi.org\/project\/labelImg\/."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Viola, P., and Jones, M.J. (2001, January 13). Robust Real-time Object Detection. Proceedings of the 2nd International Workshop on Statistical and Computational Theories of Vision-Modelling, Learning, Computing, and Sampling, Vancouver, BC, Canada.","DOI":"10.1109\/ICCV.2001.937709"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1023\/B:VISI.0000013087.49260.fb","article-title":"Robust Real-Time Face Detection","volume":"57","author":"Viola","year":"2004","journal-title":"Int. J. Comput. Vis."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1006\/jcss.1997.1504","article-title":"A Decision-Theoretic Generalization of on-Line Learning and an Application to Boosting","volume":"55","author":"Freund","year":"1997","journal-title":"J. Comput. Syst. Sci."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1115\/1.3662552","article-title":"A New Approach to Linear Filtering and Prediction Problems","volume":"82","author":"Kalman","year":"1960","journal-title":"J. Basic Eng."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Kazemi, V., and Sullivan, J. (2014, January 23\u201328). One Millisecond Face Alignment with an Ensemble of Regression Trees. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.241"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1109\/TSMC.1979.4310076","article-title":"A Threshold Selection Method from Gray-Level Histograms","volume":"9","author":"Otsu","year":"2007","journal-title":"IEEE Trans. Syst. Man Cybern."},{"key":"ref_39","unstructured":"Wierwille, W.W., Steven, S.W., Chris, K., Ellsworth, L.A., and Rollin, J.F. (1994). Research on Vehicle-Based Driver Status\/Performance Monitoring; Development, Validation, and Refinement of Algorithms for Detection of Driver Drowsiness, Virginia Polytechnic Institute and State University, Blacksburg, Department of Agronomy. Final Report."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"533","DOI":"10.1038\/323533a0","article-title":"Learning representations by back-propagating errors","volume":"323","author":"Rumelhart","year":"1986","journal-title":"Nature"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/13\/4717\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:37:48Z","timestamp":1760139468000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/13\/4717"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,22]]},"references-count":40,"journal-issue":{"issue":"13","published-online":{"date-parts":[[2022,7]]}},"alternative-id":["s22134717"],"URL":"https:\/\/doi.org\/10.3390\/s22134717","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,6,22]]}}}