{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T14:18:06Z","timestamp":1781187486401,"version":"3.54.1"},"reference-count":60,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2022,6,8]],"date-time":"2022-06-08T00:00:00Z","timestamp":1654646400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Brain Pool Program through the National Research Foundation of Korea (NRF)","award":["NRF-2019H1D3A1A01071115"],"award-info":[{"award-number":["NRF-2019H1D3A1A01071115"]}]},{"name":"Brain Pool Program through the National Research Foundation of Korea (NRF)","award":["2022-0-00966"],"award-info":[{"award-number":["2022-0-00966"]}]},{"name":"Institute of Information and Communications Technology Planning and Evaluation (IITP)","award":["NRF-2019H1D3A1A01071115"],"award-info":[{"award-number":["NRF-2019H1D3A1A01071115"]}]},{"name":"Institute of Information and Communications Technology Planning and Evaluation (IITP)","award":["2022-0-00966"],"award-info":[{"award-number":["2022-0-00966"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In this paper, we have demonstrated a robust in-cabin monitoring system (IMS) for safety, security, surveillance, and monitoring, including privacy concerns for personal and shared autonomous vehicles (AVs). It consists of a set of monitoring cameras and an onboard device (OBD) equipped with artificial intelligence (AI). Hereafter, this combination of a camera and an OBD is referred to as the AI camera. We have investigated the issues for mobility services in higher levels of autonomous driving, what needs to be monitored, how to monitor, etc. Our proposed IMS is an on-device AI system that indigenously has improved the privacy of the users. Furthermore, we have enlisted the essential actions to be considered in an IMS and developed an appropriate database (DB). Our DB consists of multifaced scenarios important for monitoring the in-cabin of the higher-level AVs. Moreover, we have compared popular AI models applied for object and occupant recognition. In addition, our DB is available on request to support the research on the development of seamless monitoring of the in-cabin higher levels of autonomous driving for the assurance of safety and security.<\/jats:p>","DOI":"10.3390\/s22124360","type":"journal-article","created":{"date-parts":[[2022,6,13]],"date-time":"2022-06-13T02:01:44Z","timestamp":1655085704000},"page":"4360","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":30,"title":["In-Cabin Monitoring System for Autonomous Vehicles"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8579-5583","authenticated-orcid":false,"given":"Ashutosh","family":"Mishra","sequence":"first","affiliation":[{"name":"School of Integrated Technology, Yonsei University, Incheon 21983, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sangho","family":"Lee","sequence":"additional","affiliation":[{"name":"School of Integrated Technology, Yonsei University, Incheon 21983, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dohyun","family":"Kim","sequence":"additional","affiliation":[{"name":"School of Integrated Technology, Yonsei University, Incheon 21983, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9935-1721","authenticated-orcid":false,"given":"Shiho","family":"Kim","sequence":"additional","affiliation":[{"name":"School of Integrated Technology, Yonsei University, Incheon 21983, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,6,8]]},"reference":[{"key":"ref_1","unstructured":"SAE (2021, December 14). International Releases Updated Visual Chart for Its \u201cLevels of Driving Automation\u201d Standard for Self-Driving Vehicles. Available online: https:\/\/www.sae.org\/news\/press-room\/2018\/12\/sae-international-releases-updated-visual-chart-for-its-%E2%80%9Clevels-of-driving-automation%E2%80%9D-standard-for-self-driving-vehicles."},{"key":"ref_2","unstructured":"(2021, December 14). Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles J3016_202104. Available online: https:\/\/www.sae.org\/standards\/content\/j3016_202104\/."},{"key":"ref_3","unstructured":"(2021, December 14). Automated Vehicles for Safety, Available online: https:\/\/www.nhtsa.gov\/technology-innovation\/automated-vehicles-safety#topic-road-self-driving."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1109\/MITS.2016.2583491","article-title":"Autonomous vehicle safety: An interdisciplinary challenge","volume":"9","author":"Koopman","year":"2017","journal-title":"IEEE Intell. Transp. Syst. Mag."},{"key":"ref_5","first-page":"380","article-title":"HCI Based In-Cabin Monitoring System for Irregular Situations with Occupants Facial Anonymization","volume":"Volume 12616","author":"Singh","year":"2021","journal-title":"Intelligent Human Computer Interaction, Proceedings of the IHCI 2020, Daegu, Korea, 24\u201326 November 2020"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Mishra, A., Kim, J., Kim, D., Cha, J., and Kim, S. (2020, January 21\u201324). An intelligent in-cabin monitoring system in fully autonomous vehicles. Proceedings of the 2020 International SoC Design Conference (ISOCC), Yeosu, Korea.","DOI":"10.1109\/ISOCC50952.2020.9333062"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Bosch, E., Oehl, M., Jeon, M., Alvarez, I., Healey, J., Ju, W., and Jallais, C. (2018, January 23\u201325). Emotional garage: A workshop on in-car emotion recognition and regulation. Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Toronto, ON, Canada.","DOI":"10.1145\/3239092.3239098"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2971482","article-title":"Connected car: Technologies, issues, future trends","volume":"49","author":"Coppola","year":"2016","journal-title":"ACM Comput. Surv."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"132","DOI":"10.1109\/MITS.2021.3050883","article-title":"Artificial Intelligence Methods in In-Cabin Use Cases: A Survey","volume":"14","author":"Rong","year":"2021","journal-title":"IEEE Intell. Transp. Syst. Mag."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Khan, M.Q., and Lee, S. (2019). A Comprehensive Survey of Driving Monitoring and Assistance Systems. Sensors, 19.","DOI":"10.3390\/s19112574"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"102021","DOI":"10.1109\/ACCESS.2019.2926040","article-title":"MIT advanced vehicle technology study: Large-scale naturalistic driving study of driver behavior and interaction with automation","volume":"7","author":"Fridman","year":"2019","journal-title":"IEEE Access"},{"key":"ref_12","first-page":"211","article-title":"In-Vehicle Violence Detection in Carpooling: A Brief Survey Towards a General Surveillance System","volume":"Volume 1237","author":"Dong","year":"2021","journal-title":"Distributed Computing and Artificial Intelligence, Proceedings of the 17th International Conference, DCAI 2020, L\u2019Aquila, Italy, 16\u201319 June 2020"},{"key":"ref_13","unstructured":"Nandi, P., Mishra, A., Kedia, P., and Rao, M. (November, January 19). Design of a real-time autonomous in-cabin sensory system to detect passenger anomaly. Proceedings of the IEEE Intelligent Vehicles Symposium (IV), Las Vegas, NV, USA."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1016\/j.jsr.2016.12.008","article-title":"Evaluation of an in-vehicle monitoring system (IVMS) to reduce risky driving behaviors in commercial drivers: Comparison of in-cab warning lights and supervisory coaching with videos of driving behavior","volume":"60","author":"Bell","year":"2017","journal-title":"J. Saf. Res."},{"key":"ref_15","unstructured":"Szawarski, H., Le, J., and Rao, M.K. (2019). Monitoring a Vehicle Cabin. (10,252,688), U.S. Patent."},{"key":"ref_16","unstructured":"Song, X. (2019). Safety and Clean Vehicle Monitoring System. (10,196,070), U.S. Patent."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Ribas, L.G.T., Cocron, M.P., Da Silva, J.L., Zimmer, A., and Brandmeier, T. (2021, January 11\u201317). In-Cabin vehicle synthetic data to test Deep Learning based human pose estimation models. Proceedings of the IEEE Intelligent Vehicles Symposium (IV), Nagoya, Japan.","DOI":"10.1109\/IV48863.2021.9576020"},{"key":"ref_18","unstructured":"(2021, December 15). Viisights In-Cabin Monitoring System. Available online: https:\/\/www.viisights.com\/products\/in-cabin\/."},{"key":"ref_19","unstructured":"(2021, December 15). The Importance of a Passenger Monitoring System. Available online: https:\/\/visagetechnologies.com\/passenger-monitoring-system\/."},{"key":"ref_20","unstructured":"(2021, December 15). In-Cabin Monitoring System (ICMS). Available online: https:\/\/www.infineon.com\/cms\/en\/applications\/automotive\/chassis-safety-and-adas\/in-cabin-sensing\/."},{"key":"ref_21","unstructured":"(2021, December 16). Cabin Sensing. Available online: https:\/\/www.continental-automotive.com\/en-gl\/Passenger-Cars\/Information-Management\/ICAM-Campaign."},{"key":"ref_22","unstructured":"(2021, December 16). Interior Monitoring Systems. Available online: https:\/\/www.bosch-mobility-solutions.com\/en\/solutions\/interior\/interior-monitoring-systems\/."},{"key":"ref_23","unstructured":"(2021, December 16). Driver Monitoring: From Essential Safety to Passenger Wellness. Available online: https:\/\/www.futurebridge.com\/blog\/driver-monitoring-from-essential-safety-to-passenger-wellness\/."},{"key":"ref_24","unstructured":"(2021, December 17). PathPartner Technology to Showcase In-Cabin Monitoring Solutions Based on Radar and Camera at CES 2020. Available online: https:\/\/www.pathpartnertech.com\/pathpartner-technology-to-showcase-in-cabin-monitoring-solutions-based-on-radar-and-camera-at-ces-2020\/."},{"key":"ref_25","unstructured":"(2021, December 17). INCABIN RADAR MONITORING. Available online: https:\/\/www.innosent.de\/automotive\/incabin-radar-monitoring\/."},{"key":"ref_26","unstructured":"(2021, December 17). Automotive: In-Cabin Applications for Vehicles. Available online: https:\/\/www.smartradarsystem.com\/in-cabin."},{"key":"ref_27","unstructured":"(2021, December 20). Passenger Monitoring. Available online: https:\/\/www.osram.com\/os\/applications\/automotive-applications\/sensing_passenger_monitoring.jsp."},{"key":"ref_28","unstructured":"(2021, December 20). Driver and Passenger Monitoring Brings Zero Cabin Privacy. Available online: https:\/\/www.tu-auto.com\/driver-and-passenger-monitoring-brings-zero-cabin-privacy\/."},{"key":"ref_29","unstructured":"(2021, December 20). Vayyar Announces Its 4D Imaging Radar Sensor on a Chip for In-Cabin Monitoring & Vehicle Safety Systems. Available online: https:\/\/www.futurecar.com\/4313\/Vayyar-Announces-its-4D-Imaging-Radar-Sensor-on-a-Chip-for-In-Cabin-Monitoring-&-Vehicle-Safety-Systems."},{"key":"ref_30","unstructured":"(2021, December 21). Vehicle In-Cabin Monitoring. Available online: https:\/\/www.iav.com\/en\/what-moves-us\/vehicle-in-cabin-monitoring\/."},{"key":"ref_31","unstructured":"(2021, December 21). In-Cabin Monitoring Systems. Available online: https:\/\/car.harman.com\/solutions\/adas\/in-cabin-monitoring-systems."},{"key":"ref_32","unstructured":"(2021, December 21). Xperi Develops World-First In-Cabin Monitoring Technologies on Neuromorphic Vision Systems. Available online: https:\/\/www.prophesee.ai\/2021\/04\/21\/xperi-develops-world-first-in-cabin-monitoring-technologies\/."},{"key":"ref_33","unstructured":"(2021, December 21). CoDriver for Driver Monitoring. Available online: https:\/\/www.jungo.com\/st\/codriver-segments\/codriver-driver-monitoring\/."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Poon, Y.-S., Lin, C.C., Liu, Y.-H., and Fan, C.-P. (2022, January 7\u20139). YOLO-Based Deep Learning Design for In-Cabin Monitoring System with Fisheye-Lens Camera. Proceedings of the IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA.","DOI":"10.1109\/ICCE53296.2022.9730235"},{"key":"ref_35","unstructured":"Katrolia, J.S., Mirbach, B., El-Sherif, A., Feld, H., Rambach, J., and Stricker, D. (2021). Ticam: A time-of-flight in-car cabin monitoring dataset. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Feld, H., Mirbach, B., Katrolia, J., Selim, M., Wasenm\u00fcller, O., and Stricker, D. (2021). Dfki cabin simulator: A test platform for visual in-cabin monitoring functions. Commercial Vehicle Technology 2020\/2021, Springer Vieweg.","DOI":"10.1007\/978-3-658-29717-6_28"},{"key":"ref_37","unstructured":"Mayank, M., Mihir, M., Kedar, C., Piyali, G., Tarkesh, P., Shashank, D., Shyam, J., Stefan, H., Gang, H., and Hrushikesh, G. (2022, January 17). Efficient in-cabin monitoring solution using TI TDA2PxSOCs. Proceedings of the IS&T International Symposium on Electronic Imaging: Autonomous Vehicles and Machines, San Francisco, CA, USA."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Pappalardo, G., Cafiso, S., and di Graziano, A. (2021). A severino, decision tree method to analyze the performance of lane support systems. Sustainability, 13.","DOI":"10.3390\/su13020846"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Othman, W., Kashevnik, A., Ali, A., and Shilov, N. (2022). DriverMVT: In-Cabin Dataset for Driver Monitoring including Video and Vehicle Telemetry Information. Data, 7.","DOI":"10.3390\/data7050062"},{"key":"ref_40","unstructured":"Deng, Q. (2021). Vehicle Occupant Monitoring with mmWave Wideband Planar Array Radar. [Ph.D. Thesis, Purdue University Graduate School]. Available online: https:\/\/hammer.purdue.edu\/articles\/thesis\/Vehicle_Occupant_Monitoring_with_mmWave_Wideband_Planar_Array_Radar\/14156987\/1."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Kashevnik, A., Ponomarev, A., Shilov, N., and Chechulin, A. (2022). Threats Detection during Human-Computer Interaction in Driver Monitoring Systems. Sensors, 22.","DOI":"10.3390\/s22062380"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Vennam, P., Pramod, T.C., Thippeswamy, B.M., Kim, Y.G., and Kumar, P. (2021). Attacks and preventive measures on video surveillance systems: A review. Appl. Sci., 11.","DOI":"10.3390\/app11125571"},{"key":"ref_43","unstructured":"(2022, April 05). UK\u2019s Facial Recognition Technology \u2018Breaches Privacy Rights\u2019. Available online: https:\/\/www.theguardian.com\/technology\/2020\/jun\/23\/uks-facial-recognition-technology-breaches-privacy-rights."},{"key":"ref_44","unstructured":"(2022, April 05). Facial Recognition Technology: Privacy and Accuracy Issues Related to Commercial Uses, Available online: https:\/\/www.gao.gov\/assets\/710\/708045.pdf."},{"key":"ref_45","unstructured":"(2022, April 05). Facial Recognition Technology: Fundamental Rights Considerations in the Context of Law Enforcement. Available online: https:\/\/fra.europa.eu\/en\/publication\/2019\/facial-recognition-technology-fundamental-rights-considerations-context-law."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"23649","DOI":"10.1007\/s11042-020-10249-1","article-title":"Protection of visual privacy in videos acquired with RGB cameras for active and assisted living applications","volume":"80","year":"2021","journal-title":"Multimed. Tools Appl."},{"key":"ref_47","first-page":"2020","article-title":"Schrems II: The right to privacy and the new illiberalism","volume":"29","author":"Bignami","year":"2020","journal-title":"Verfassungsblog"},{"key":"ref_48","unstructured":"Dushi, D. (2020). The use of facial recognition technology in EU law enforcement: Fundamental rights implications. Global Campus Open Knowledge Repository, Global Campus of Human Rights."},{"key":"ref_49","unstructured":"Mekrani, A. (2020). The Future of Facial Recognition in Relation to Privacy. [Master\u2019s Thesis, Tilburg University]."},{"key":"ref_50","unstructured":"Naranjo, D. (2020). Your Face Rings a Bell: How Facial Recognition Poses a Threat for Human Rights, Global Campus Open Knowledge Repository."},{"key":"ref_51","unstructured":"(2022, April 05). How Facial Recognition Technology Threatens Basic Privacy Rights. Available online: https:\/\/www.computerweekly.com\/feature\/How-facial-recognition-technology-threatens-basic-privacy-rights."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"5389359","DOI":"10.1155\/2022\/5389359","article-title":"Privacy-Preserved In-Cabin Monitoring System for Autonomous Vehicles","volume":"2022","author":"Mishra","year":"2022","journal-title":"Comput. Intell. Neurosci."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 26\u201330). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23\u201328). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.81"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Redmon, J., and Farhadi, A. (2017, January 21\u201326). YOLO9000: Better, Faster, Stronger. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.690"},{"key":"ref_56","unstructured":"Redmon, J., and Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1016\/j.ijleo.2019.02.038","article-title":"An improved tiny-yolov3 pedestrian detection algorithm","volume":"183","author":"Zhang","year":"2019","journal-title":"Optik"},{"key":"ref_58","unstructured":"Zou, Z., Shi, Z., Guo, Y., and Ye, J. (2019). Object detection in 20 years: A survey. arXiv."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1007\/s11263-019-01247-4","article-title":"Deep learning for generic object detection: A survey","volume":"128","author":"Liu","year":"2020","journal-title":"Int. J. Comput. Vis."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"128837","DOI":"10.1109\/ACCESS.2019.2939201","article-title":"A survey of deep learning-based object detection","volume":"7","author":"Jiao","year":"2019","journal-title":"IEEE Access"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/12\/4360\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:26:31Z","timestamp":1760138791000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/12\/4360"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,8]]},"references-count":60,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2022,6]]}},"alternative-id":["s22124360"],"URL":"https:\/\/doi.org\/10.3390\/s22124360","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,6,8]]}}}