{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T07:33:21Z","timestamp":1767598401271,"version":"build-2065373602"},"reference-count":24,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2022,3,19]],"date-time":"2022-03-19T00:00:00Z","timestamp":1647648000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002261","name":"Russian Foundation for Basic Research","doi-asserted-by":"publisher","award":["19-29-06099"],"award-info":[{"award-number":["19-29-06099"]}],"id":[{"id":"10.13039\/501100002261","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Russian State Research","award":["FFZF-2022-0005"],"award-info":[{"award-number":["FFZF-2022-0005"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This paper presents an approach and a case study for threat detection during human\u2013computer interaction, using the example of driver\u2013vehicle interaction. We analyzed a driver monitoring system and identified two types of users: the driver and the operator. The proposed approach detects possible threats for the driver. We present a method for threat detection during human\u2013system interactions that generalizes potential threats, as well as approaches for their detection. The originality of the method is that we frame the problem of threat detection in a holistic way: we build on the driver\u2013ITS system analysis and generalize existing methods for driver state analysis into a threat detection method covering the identified threats. The developed reference model of the operator\u2013computer interaction interface shows how the driver monitoring process is organized, and what information can be processed automatically, and what information related to the driver behavior has to be processed manually. In addition, the interface reference model includes mechanisms for operator behavior monitoring. We present experiments that included 14 drivers, as a case study. The experiments illustrated how the operator monitors and processes the information from the driver monitoring system. Based on the case study, we clarified that when the driver monitoring system detected the threats in the cabin and notified drivers about them, the number of threats was significantly decreased.<\/jats:p>","DOI":"10.3390\/s22062380","type":"journal-article","created":{"date-parts":[[2022,3,20]],"date-time":"2022-03-20T21:37:17Z","timestamp":1647812237000},"page":"2380","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Threats Detection during Human-Computer Interaction in Driver Monitoring Systems"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6503-1447","authenticated-orcid":false,"given":"Alexey","family":"Kashevnik","sequence":"first","affiliation":[{"name":"St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), St. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences, 199178 St. Petersburg, Russia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andrew","family":"Ponomarev","sequence":"additional","affiliation":[{"name":"St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), St. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences, 199178 St. Petersburg, Russia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9264-9127","authenticated-orcid":false,"given":"Nikolay","family":"Shilov","sequence":"additional","affiliation":[{"name":"St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), St. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences, 199178 St. Petersburg, Russia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7056-6972","authenticated-orcid":false,"given":"Andrey","family":"Chechulin","sequence":"additional","affiliation":[{"name":"St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), St. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences, 199178 St. Petersburg, Russia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"e6475","DOI":"10.1002\/cpe.6475","article-title":"Computer Vision-based Recognition of Driver Distraction: A Review","volume":"33","author":"Moslemi","year":"2021","journal-title":"Concurr. Comput. Pract. Exp."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"102319","DOI":"10.1016\/j.sysarc.2021.102319","article-title":"A Survey on Vision-Based Driver Distraction Analysis","volume":"121","author":"Li","year":"2021","journal-title":"J. Syst. Archit."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Arakawa, T. (2021). Trends and Future Prospects of the Drowsiness Detection and Estimation Technology. Sensors, 21.","DOI":"10.3390\/s21237921"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Kashevnik, A., Ponomarev, A., Shilov, N., and Chechulin, A. (2020). In-Vehicle Situation Monitoring for Potential Threats Detection Based on Smartphone Sensors. Sensors, 20.","DOI":"10.3390\/s20185049"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Singh, H.V.P., and Mahmoud, Q.H. (May, January 30). EYE-on-HMI: A Framework for Monitoring Human Machine Interfaces in Control Rooms. Proceedings of the 2017 IEEE 30th Canadian Conference on Electrical and Computer Engineering (CCECE), Windsor, ON, Canada.","DOI":"10.1109\/CCECE.2017.7946695"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Singh, H.V.P., and Mahmoud, Q. (2019, January 11\u201312). LSTM-Based Approach to Monitor Operator Situation Awareness via HMI State Prediction. Proceedings of the 2019 IEEE International Conference on Industrial Internet (ICII), Orlando, FL, USA.","DOI":"10.1109\/ICII.2019.00064"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"361","DOI":"10.1111\/j.1539-6924.2005.00595.x","article-title":"A Screening Methodology for the Identification and Ranking of Infrastructure Vulnerabilities Due to Terrorism","volume":"25","author":"Apostolakis","year":"2005","journal-title":"Risk Anal."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"60063","DOI":"10.1109\/ACCESS.2021.3073599","article-title":"Driver Distraction Detection Methods: A Literature Review and Framework","volume":"9","author":"Kashevnik","year":"2021","journal-title":"IEEE Access"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Bulygin, A., and Kashevnik, A. (2021, January 27\u201329). Image-Based Fatigue Detection of Vehicle Driver: State-of-the-Art and Reference Model. Proceedings of the 2021 30th Conference of Open Innovations Association FRUCT, Oulu, Finland.","DOI":"10.23919\/FRUCT53335.2021.9599990"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Shilov, N., and Kashevnik, A. (2021, January 12\u201314). An Effort to Detect Vehicle Driver\u2019s Drowsy State Based on the Speed Analysis. Proceedings of the 2021 29th Conference of Open Innovations Association (FRUCT), Tampere, Finland.","DOI":"10.23919\/FRUCT52173.2021.9435466"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Amershi, S., Begel, A., Bird, C., DeLIne, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., and Zimmermann, T. (2019, January 25\u201331). Software Engineering for Machine Learning: A Case Study. Proceedings of the International Conference on Software Engineering (ICSE 2019)\u2014Software Engineering in Practice Track, Montreal, QC, Canada.","DOI":"10.1109\/ICSE-SEIP.2019.00042"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Studer, S., Bui, T.B., Drescher, C., Hanuschkin, A., Winkler, L., Peters, S., and M\u00fcller, K.-R. (2021). Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology. Mach. Learn. Knowl. Extr., 3.","DOI":"10.20944\/preprints202103.0135.v1"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3453444","article-title":"Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges","volume":"54","author":"Ashmore","year":"2021","journal-title":"ACM Comput. Surv."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Hasan, F., and Kashevnik, A. (2021, January 12\u201314). State-of-the-Art Analysis of Modern Drowsiness Detection Algorithms Based on Computer Vision. Proceedings of the 2021 29th Conference of Open Innovations Association (FRUCT), Tampere, Finland.","DOI":"10.23919\/FRUCT52173.2021.9435480"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"103810","DOI":"10.1016\/j.compbiomed.2020.103810","article-title":"Proposing a Convolutional Neural Network for Stress Assessment by Means of Derived Heart Rate from Functional near Infrared Spectroscopy","volume":"121","author":"Hakimi","year":"2020","journal-title":"Comput. Biol. Med."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Khan, M.Q., and Lee, S. (2019). A Comprehensive Survey of Driving Monitoring and Assistance Systems. Sensors, 19.","DOI":"10.3390\/s19112574"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Borghini, G., Vecchiato, G., Toppi, J., Astolfi, L., Maglione, A., Isabella, R., Caltagirone, C., Kong, W., Wei, D., and Zhou, Z. (September, January 28). Assessment of Mental Fatigue during Car Driving by Using High Resolution EEG Activity and Neurophysiologic Indices. Proceedings of the 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Diego, CA, USA.","DOI":"10.1109\/EMBC.2012.6347469"},{"key":"ref_18","first-page":"263983","article-title":"A Driver Face Monitoring System for Fatigue and Distraction Detection","volume":"2013","author":"Sigari","year":"2013","journal-title":"Int. J. Veh. Technol."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"102723","DOI":"10.1016\/j.jvcir.2019.102723","article-title":"Driver Fatigue Detection Based on Deeply-Learned Facial Expression Representation","volume":"71","author":"Liu","year":"2020","journal-title":"J. Vis. Commun. Image Represent."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"901","DOI":"10.1007\/978-3-030-90321-3_75","article-title":"Driver\u2019s State Identification Based on the Vehicle Speed Analysis Taking into Account the Driving Context","volume":"231","author":"Shilov","year":"2021","journal-title":"Lect. Notes Netw. Syst."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"2427","DOI":"10.1109\/TITS.2019.2918328","article-title":"Methodology and Mobile Application for Driver Behavior Analysis and Accident Prevention","volume":"21","author":"Kashevnik","year":"2019","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Kashevnik, A., Ali, A., Lashkov, I., and Shilov, N. (2020, January 20\u201324). Seat Belt Fastness Detection Based on Image Analysis from Vehicle In-Abin Camera. Proceedings of the 2020 26th Conference of Open Innovations Association (FRUCT), Yaroslavl, Russia.","DOI":"10.23919\/FRUCT48808.2020.9087474"},{"key":"ref_23","unstructured":"Kashevnik, A., Ali, A., Lashkov, I., and Zubok, D. Human Head Angle Detection Based on Image Analysis. Proceedings of the Advances in Intelligent Systems and Computing, Springer Science and Business Media Deutschland GmbH."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"6701","DOI":"10.1109\/JSEN.2020.2975382","article-title":"Cloud-Based Driver Monitoring System Using a Smartphone","volume":"20","author":"Kashevnik","year":"2020","journal-title":"IEEE Sens. J."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/6\/2380\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:39:34Z","timestamp":1760135974000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/6\/2380"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,19]]},"references-count":24,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2022,3]]}},"alternative-id":["s22062380"],"URL":"https:\/\/doi.org\/10.3390\/s22062380","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2022,3,19]]}}}