{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,28]],"date-time":"2026-07-28T04:03:52Z","timestamp":1785211432805,"version":"3.55.0"},"reference-count":85,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2020,9,4]],"date-time":"2020-09-04T00:00:00Z","timestamp":1599177600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2020,9,4]]},"abstract":"<jats:p>Road safety is a major public health issue across the globe and over two-thirds of the road accidents occur at nighttime under low-light conditions or darkness. The state of the driver and her\/his actions are the key factors impacting road safety. How can we monitor these in a cost-effective manner and in low-light conditions? RGB cameras present in smartphones perform poorly in low-lighting conditions due to lack of information captured. Hence, existing monitoring solutions rely upon specialized hardware such as infrared cameras or thermal cameras in low-light conditions, but are limited to only high-end vehicles owing to the cost of the hardware. We present InSight, a windshield-mounted smartphone-based system that can be retrofitted to the vehicle to monitor the state of the driver, specifically driver fatigue (based on frequent yawning and eye closure) and driver distraction (based on their direction of gaze). Challenges arise from designing an accurate, yet low-cost and non-intrusive system to continuously monitor the state of the driver.<\/jats:p>\n          <jats:p>In this paper, we present two novel and practical approaches for continuous driver monitoring in low-light conditions: (i) Image synthesis: enabling monitoring in low-light conditions using just the smartphone RGB camera by synthesizing a thermal image from RGB with a Generative Adversarial Network, and (ii) Near-IR LED: using a low-cost near-IR (NIR) LED attachment to the smartphone, where the NIR LED acts as a light source to illuminate the driver's face, which is not visible to the human eyes, but can be captured by standard smartphone cameras without any specialized hardware. We show that the proposed techniques can capture the driver's face accurately in low-lighting conditions to monitor driver's state. Further, since NIR and thermal imagery is significantly different than RGB images, we present a systematic approach to generate labelled data, which is used to train existing computer vision models. We present an extensive evaluation of both the approaches with data collected from 15 drivers in controlled basement area and on real roads in low-light conditions. The proposed NIR LED setup has an accuracy (Fl-score) of 85% and 93.8% in detecting driver fatigue and distraction, respectively in low-light.<\/jats:p>","DOI":"10.1145\/3411819","type":"journal-article","created":{"date-parts":[[2020,9,4]],"date-time":"2020-09-04T21:39:45Z","timestamp":1599255585000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":25,"title":["InSight"],"prefix":"10.1145","volume":"4","author":[{"given":"Ishani","family":"Janveja","sequence":"first","affiliation":[{"name":"Microsoft Research, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Akshay","family":"Nambi","sequence":"additional","affiliation":[{"name":"Microsoft Research, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shruthi","family":"Bannur","sequence":"additional","affiliation":[{"name":"Microsoft Research, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sanchit","family":"Gupta","sequence":"additional","affiliation":[{"name":"Microsoft Research, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Venkat","family":"Padmanabhan","sequence":"additional","affiliation":[{"name":"Microsoft Research, India"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,9,4]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2009. Exposure compensation. https:\/\/en.wikipedia.org\/wiki\/Exposure_compensation.  2009. Exposure compensation. https:\/\/en.wikipedia.org\/wiki\/Exposure_compensation."},{"key":"e_1_2_1_2_1","unstructured":"2009. Mirror scan duration. https:\/\/www.thesafedriver.ca\/2009\/10\/08\/how-often-should-you-check-your-mirror\/.  2009. Mirror scan duration. https:\/\/www.thesafedriver.ca\/2009\/10\/08\/how-often-should-you-check-your-mirror\/."},{"key":"e_1_2_1_3_1","unstructured":"2017. Honda CR-V SUV. https:\/\/venturebeat.com\/2017\/03\/09\/this-small-suv-knows-when-you-get-sleepy-and-can-wake-you-up\/  2017. Honda CR-V SUV. https:\/\/venturebeat.com\/2017\/03\/09\/this-small-suv-knows-when-you-get-sleepy-and-can-wake-you-up\/"},{"key":"e_1_2_1_4_1","unstructured":"2017. Nighttime Car Accident Statistics. https:\/\/seriousaccidents.com\/legal-advice\/top-causes-of-car-accidents\/nighttime-driving\/  2017. Nighttime Car Accident Statistics. https:\/\/seriousaccidents.com\/legal-advice\/top-causes-of-car-accidents\/nighttime-driving\/"},{"key":"e_1_2_1_5_1","unstructured":"2017. Receive Warnings About Your Level of Alertness While Driving With Honda's Driver Attention Monitor. http:\/\/www.hiltonheadhonda.com\/blog\/how-does-the-honda-driver-attention-monitor-work\/  2017. Receive Warnings About Your Level of Alertness While Driving With Honda's Driver Attention Monitor. http:\/\/www.hiltonheadhonda.com\/blog\/how-does-the-honda-driver-attention-monitor-work\/"},{"key":"e_1_2_1_6_1","unstructured":"2018. Night Sight: Seeing in the Dark on Pixel Phones. https:\/\/ai.googleblog.com\/2018\/11\/night-sight-seeing-in-dark-on-pixel.html.  2018. Night Sight: Seeing in the Dark on Pixel Phones. https:\/\/ai.googleblog.com\/2018\/11\/night-sight-seeing-in-dark-on-pixel.html."},{"key":"e_1_2_1_7_1","unstructured":"2018. WHO Global Health Observatory (GHO) data. http:\/\/www.who.int\/gho\/road_safety\/mortality\/en\/.  2018. WHO Global Health Observatory (GHO) data. http:\/\/www.who.int\/gho\/road_safety\/mortality\/en\/."},{"key":"e_1_2_1_8_1","unstructured":"2018. WHO Road traffic injuries. http:\/\/www.who.int\/mediacentre\/factsheets\/fs358\/en\/.  2018. WHO Road traffic injuries. http:\/\/www.who.int\/mediacentre\/factsheets\/fs358\/en\/."},{"key":"e_1_2_1_9_1","unstructured":"2019. FLIR one for Android. http:\/\/www.flir.com\/flirone\/pro\/  2019. FLIR one for Android. http:\/\/www.flir.com\/flirone\/pro\/"},{"key":"e_1_2_1_10_1","unstructured":"2019. Flir one for android. https:\/\/www.flir.com\/flir-one\/  2019. Flir one for android. https:\/\/www.flir.com\/flir-one\/"},{"key":"e_1_2_1_11_1","unstructured":"2019. Infrared cameras? https:\/\/en.wikipedia.org\/wiki\/Thermographic_camera  2019. Infrared cameras? https:\/\/en.wikipedia.org\/wiki\/Thermographic_camera"},{"key":"e_1_2_1_12_1","unstructured":"2019. SFH 4787S-OSRAM 810nm IR LED. https:\/\/www.mouser.com\/ProductDetail\/OSRAM-Opto-Semiconductors\/SFH-4787S  2019. SFH 4787S-OSRAM 810nm IR LED. https:\/\/www.mouser.com\/ProductDetail\/OSRAM-Opto-Semiconductors\/SFH-4787S"},{"key":"e_1_2_1_13_1","unstructured":"2019. VANTRUE: High-end Dash Cam for Your Drive. https:\/\/www.vantrue.net\/  2019. VANTRUE: High-end Dash Cam for Your Drive. https:\/\/www.vantrue.net\/"},{"key":"e_1_2_1_14_1","unstructured":"2019. What is an Infrared Illuminator? https:\/\/www.flir.in\/discover\/ots\/what-is-an-infrared-illuminator\/  2019. What is an Infrared Illuminator? https:\/\/www.flir.in\/discover\/ots\/what-is-an-infrared-illuminator\/"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2557642.2563678"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01228-1_8"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2006.869598"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3276774.3276791"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.116"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1186\/1687-5281-2013-13"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00347"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ITST.2013.6685546"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/IVS.2011.5940406"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.927467"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.5244\/C.20.95"},{"key":"e_1_2_1_26_1","volume-title":"International Symposium on Signals, Circuits and Systems, 2005. ISSCS 2005.","volume":"1","author":"Cudalbu C."},{"key":"e_1_2_1_27_1","unstructured":"Navneet Dalal and Bill Triggs. 2005. Histograms of oriented gradients for human detection.  Navneet Dalal and Bill Triggs. 2005. Histograms of oriented gradients for human detection."},{"key":"e_1_2_1_28_1","volume-title":"Style Aggregated Network for Facial Landmark Detection. In 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 379--388","author":"Dong Xuanyi","year":"2018"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/FG.2019.8756620"},{"key":"e_1_2_1_30_1","volume-title":"A visual approach for driver inattention detection. Pattern recognition 40, 8","author":"D'Orazio Tiziana","year":"2007"},{"key":"e_1_2_1_31_1","volume-title":"Yawn duration predicts brain weight and cortical neuron number in mammals. Biology letters 12, 10","author":"Gallup Andrew C","year":"2016"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-93000-8_49"},{"key":"e_1_2_1_33_1","unstructured":"Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680.  Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2980254"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3307334.3326107"},{"key":"e_1_2_1_36_1","volume-title":"Efficient contrast enhancement using adaptive gamma correction with weighting distribution","author":"Huang Shih-Chia","year":"2012"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCE.2007.4429280"},{"key":"e_1_2_1_38_1","unstructured":"IEC IEC. 2006. 62471: 2006\/CIE S 009\/E: 2002 Photobiological safety of lamps and lamp systems.  IEC IEC. 2006. 62471: 2006\/CIE S 009\/E: 2002 Photobiological safety of lamps and lamp systems."},{"key":"e_1_2_1_39_1","unstructured":"IR Illumination and Eye Safety. [n.d.]. https:\/\/medium.com\/@alex.kilpatrick\/ir-illumination-and-eye-safety-f0804673ca7  IR Illumination and Eye Safety. [n.d.]. https:\/\/medium.com\/@alex.kilpatrick\/ir-illumination-and-eye-safety-f0804673ca7"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.632"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVT.2004.830974"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1049\/iet-its.2012.0032"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.241"},{"key":"e_1_2_1_44_1","volume-title":"Super-realtime facial landmark detection and shape fitting by deep regression of shape model parameters. arXiv preprint arXiv:1902.03459","author":"Kopaczka Marcin","year":"2019"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC.2019.8917460"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2018.2381129"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2012.41"},{"key":"e_1_2_1_48_1","volume-title":"International Conference on Biometrics. Springer, 151--158","author":"Li Stan Z","year":"2006"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.1999.786928"},{"key":"e_1_2_1_50_1","volume-title":"How much time do drivers need to obtain situation awareness? A laboratory-based study of automated driving. Applied ergonomics 60","author":"Lu Zhenji","year":"2017"},{"key":"e_1_2_1_51_1","volume-title":"International Conference on Computer Recognition Systems. Springer, 295--304","author":"Ma\u0142ecki Krzysztof","year":"2019"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2016.2582900"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3241539.3267723"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3356250.3360037"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287059"},{"key":"e_1_2_1_56_1","volume-title":"Camera calibration and 3d reconstruction. URL https:\/\/docs. opencv.org\/2.4\/modules\/calib3d\/doc\/camera_calibration_and_3d_reconstruction. html","author":"CV.","year":"2014"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISIP.2010.116"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISIP.2010.116"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13640-016-0138-1"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1088\/0034-4885\/79\/4\/046501"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCVW.2013.59"},{"key":"e_1_2_1_62_1","doi-asserted-by":"crossref","unstructured":"Arun Sahayadhas Kenneth Sundaraj and M Murugappan. 2012. Detecting driver drowsiness based on sensors-a review. (2012).  Arun Sahayadhas Kenneth Sundaraj and M Murugappan. 2012. Detecting driver drowsiness based on sensors-a review. (2012).","DOI":"10.3390\/s121216937"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2018.2868499"},{"key":"e_1_2_1_64_1","volume-title":"21st Computer Vision Winter Workshop","author":"Soukupov\u00e1 Tereza","year":"2016"},{"key":"e_1_2_1_65_1","unstructured":"Tereza Soukupov\u00e1 and Jan Cech. 2016. Real-Time Eye Blink Detection using Facial Landmarks.  Tereza Soukupov\u00e1 and Jan Cech. 2016. Real-Time Eye Blink Detection using Facial Landmarks."},{"key":"e_1_2_1_66_1","unstructured":"FLIR Systems. 2019. Flir one for android. https:\/\/www.flir.com\/flir-one\/  FLIR Systems. 2019. Flir one for android. https:\/\/www.flir.com\/flir-one\/"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISPA.2001.938625"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2001.990517"},{"key":"e_1_2_1_69_1","volume-title":"Education and Training in Optics and Photonics","author":"Vollmer Michael"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/30.754419"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1002\/joc.658"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-018-1097-z"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368305"},{"key":"e_1_2_1_75_1","volume-title":"WIDER FACE: A Face Detection Benchmark. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Yang Shuo","year":"2016"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2018.2381131"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-64698-5_4"},{"key":"e_1_2_1_78_1","volume-title":"Yuting Cheng, Mu Lin, Lorenzo Torresani, and Andrew T. Campbell.","author":"You Chuang-Wen","year":"2013"},{"key":"e_1_2_1_79_1","volume-title":"Yuting Cheng, Mu Lin, Lorenzo Torresani, et al.","author":"You Chuang-Wen","year":"2013"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCVW.2017.296"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.888718"},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11768-010-8043-0"},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.244"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.244"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.23"},{"key":"e_1_2_1_86_1","volume-title":"Proceedings. The 7th International IEEE Conference on Intelligent Transportation Systems (IEEE Cat. No. 04TH8749)","author":"Zhu Zhiwei","year":"2004"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411819","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3411819","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:24:48Z","timestamp":1750195488000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411819"}},"subtitle":["Monitoring the State of the Driver in Low-Light Using Smartphones"],"short-title":[],"issued":{"date-parts":[[2020,9,4]]},"references-count":85,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2020,9,4]]}},"alternative-id":["10.1145\/3411819"],"URL":"https:\/\/doi.org\/10.1145\/3411819","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,9,4]]},"assertion":[{"value":"2020-09-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}