{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,8]],"date-time":"2026-06-08T18:28:05Z","timestamp":1780943285788,"version":"3.54.1"},"reference-count":45,"publisher":"Association for Computing Machinery (ACM)","issue":"EICS","license":[{"start":{"date-parts":[[2020,6,18]],"date-time":"2020-06-18T00:00:00Z","timestamp":1592438400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2020,6,18]]},"abstract":"<jats:p>The increasing number and complexity of advanced driver assistance systems (ADAS) pave the way for fully automated driving. Automated vehicles are said to increase road safety and prevent human-made (fatal) accidents, amongst others. In the lower levels of automation, however, the driver is still responsible as a fallback authority. As a consequence, systems that reliably monitor the driver's state, especially the risk factor drowsiness, become increasingly essential to ensure the driver's ability to take over control from the vehicle on time. In research, the use of supervised machine learning for drowsiness detection is the prevalent method. As the ground truth for drowsiness is both application- and user-dependent, and no golden standard exists for its definition, measures are usually applied in the form of observer ratings. Also, in this work, observer ratings were investigated with regard to the required level of detail\/complexity. To this end, video data, recorded within a simulator study (N = 30) comprised of each 45-minute manual and automated driving sessions, were evaluated by trained raters. Correlation analysis results show that - depending on the number of drowsiness levels - a comparable ground truth can be generated by reducing the rating frequency and thus the rating complexity by a factor of five. The knowledge gained can be used in future studies in this research area, the collection of a reliable and valid ground truth of drowsiness, as well as for improving the process in developing interactive drowsiness detection systems.<\/jats:p>","DOI":"10.1145\/3394980","type":"journal-article","created":{"date-parts":[[2020,6,18]],"date-time":"2020-06-18T21:54:38Z","timestamp":1592517278000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Towards a Reliable Ground Truth for Drowsiness: A Complexity Analysis on the Example of Driver Fatigue"],"prefix":"10.1145","volume":"4","author":[{"given":"Thomas","family":"Kundinger","sequence":"first","affiliation":[{"name":"AUDI AG &amp; Technische Hochschule Ingolstadt (THI) &amp; Johannes Kepler University (JKU), Ingolstadt, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Celena","family":"Mayr","sequence":"additional","affiliation":[{"name":"AUDI AG &amp; Technische Hochschule Ingolstadt (THI), Ingolstadt, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andreas","family":"Riener","sequence":"additional","affiliation":[{"name":"Technische Hochschule Ingolstadt (THI) &amp; Johannes Kepler University (JKU), Ingolstadt, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,6,18]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12544-015-0188-y"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.3109\/00207459008994241"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0064782"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/EMBC.2012.6347469"},{"key":"e_1_2_1_5_1","article-title":"Behavioural impacts of Advanced Driver Assistance Systems-an overview","volume":"1","author":"Brookhuis Karel","year":"2019","journal-title":"European Journal of Transport and Infrastructure Research"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.aap.2017.11.038"},{"key":"e_1_2_1_8_1","volume-title":"1--17","author":"Roadmap NCAP.","year":"2017"},{"key":"e_1_2_1_9_1","volume-title":"2010 18th European Signal Processing Conference. 209--213","author":"Friedrichs F."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2016.06.042"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.3390\/app9173555"},{"key":"e_1_2_1_12_1","volume-title":"CarMaker: Virtual testing of automobiles and light-duty vehicles. https:\/\/ipg-automotive.com\/products-services\/simulation-software\/carmaker\/ (accesssed on","author":"Automotive IPG","year":"2020"},{"key":"e_1_2_1_13_1","unstructured":"National Highway Traffic Safety Administration and Us Department of Transportation. 2015. TRAFFIC SAFETY FACTS Critical Reasons for Crashes Investigated in the National Motor Vehicle Crash Causation Survey. (2015). https:\/\/crashstats.nhtsa.dot.gov\/Api\/Public\/ViewPublication\/812115  National Highway Traffic Safety Administration and Us Department of Transportation. 2015. TRAFFIC SAFETY FACTS Critical Reasons for Crashes Investigated in the National Motor Vehicle Crash Causation Survey. (2015). https:\/\/crashstats.nhtsa.dot.gov\/Api\/Public\/ViewPublication\/812115"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1469-8986.1973.tb00801.x"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1087-0792(98)90050-8"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1093\/sleep\/14.6.540"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1049\/iet-its.2012.0032"},{"key":"e_1_2_1_19_1","volume-title":"The Potential of Wrist-Worn Wearables for Driver Drowsiness Detection: A Feasibility Analysis (UMAP '20)","author":"Kundinger Thomas","year":"2020"},{"key":"e_1_2_1_20_1","doi-asserted-by":"crossref","unstructured":"Thomas Kundinger Andreas Riener Nikoletta Sofra and Klemens Weigl. 2018. Drowsiness Detection and Warning in Manual and Automated Driving: Results from Subjective Evaluation (AutomotiveUI '18). 229--236. https:\/\/doi.org\/10.1145\/3239060.3239073  Thomas Kundinger Andreas Riener Nikoletta Sofra and Klemens Weigl. 2018. Drowsiness Detection and Warning in Manual and Automated Driving: Results from Subjective Evaluation (AutomotiveUI '18). 229--236. https:\/\/doi.org\/10.1145\/3239060.3239073","DOI":"10.1145\/3239060.3239073"},{"key":"e_1_2_1_21_1","volume-title":"Assessment of the Potential of Wrist-Worn Wearable Sensors for Driver Drowsiness Detection. Sensors","volume":"20","author":"Kundinger Thomas","year":"2020"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1108\/IJPCC-03-2019-0017"},{"key":"e_1_2_1_23_1","first-page":"134","article-title":"Wearable Driver Drowsiness Detection System Based on Smartwatch","volume":"15","author":"Li G.","year":"2014","journal-title":"Korea Institute of Signal Processing and Systems"},{"key":"e_1_2_1_24_1","volume-title":"Electronics","volume":"8","author":"Lee Hyeonjeong","year":"2019"},{"key":"e_1_2_1_25_1","doi-asserted-by":"crossref","unstructured":"L. B. Leng L. B. Giin and W. Chung. 2015. Wearable driver drowsiness detection system based on biomedical and motion sensors. In 2015 IEEE Sensors Journal. 1--4. https:\/\/doi.org\/10.1109\/ICSENS.2015.7370355  L. B. Leng L. B. Giin and W. Chung. 2015. Wearable driver drowsiness detection system based on biomedical and motion sensors. In 2015 IEEE Sensors Journal. 1--4. https:\/\/doi.org\/10.1109\/ICSENS.2015.7370355","DOI":"10.1109\/ICSENS.2015.7370355"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.3390\/s150820873"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSEN.2015.2473679"},{"key":"e_1_2_1_28_1","unstructured":"Quanzhe Li Juan Wu Shin-Dug Kim and Cheong-Ghil Kim. 2015b. Hybrid Driver Fatigue Detection System Based on Data Fusion with Wearable Sensor Devices.  Quanzhe Li Juan Wu Shin-Dug Kim and Cheong-Ghil Kim. 2015b. Hybrid Driver Fatigue Detection System Based on Data Fusion with Wearable Sensor Devices."},{"key":"e_1_2_1_29_1","volume-title":"Renjie Li, Bo Cheng, and Jinliang Shi.","author":"Li Zuojin","year":"2017"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.14311\/APP.2017.12.0064"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1037\/e572172013-456"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSEN.2019.2904222"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2869.2011.00984.x"},{"key":"e_1_2_1_35_1","volume-title":"Prevalence of Drowsy Driving Crashes: Estimates from a Large-Scale Naturalistic Driving Study. (Research Brief.)","author":"Owens J. M.","year":"2018"},{"key":"e_1_2_1_36_1","first-page":"93","article-title":"Determination the Levels of Subjective and Observer Rating of Drowsiness and Their Associations with Facial Dynamic Changes","volume":"46","author":"Poursadeghiyan Mohsen","year":"2017","journal-title":"Iranian Journal of Public Health"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.3390\/s121216937"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/IVS.2016.7535417"},{"key":"e_1_2_1_39_1","volume-title":"Shapiro","author":"Shahid Azmeh","year":"2012"},{"key":"e_1_2_1_40_1","first-page":"6","article-title":"Driver Fatigue Detection Systems","volume":"20","author":"Sikander G.","year":"2019","journal-title":"A Review. IEEE Transactions on Intelligent Transportation Systems"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.mehy.2008.05.035"},{"key":"e_1_2_1_42_1","unstructured":"Sleep Health Foundation. 2015. Sleep Needs Across The Lifespan. http:\/\/www.sleephealthfoundation.org.au\/files\/pdfs\/Sleep-Needs-Across-Lifespan.pdf (accesssed on 21 April 2020).  Sleep Health Foundation. 2015. Sleep Needs Across The Lifespan. http:\/\/www.sleephealthfoundation.org.au\/files\/pdfs\/Sleep-Needs-Across-Lifespan.pdf (accesssed on 21 April 2020)."},{"key":"e_1_2_1_43_1","unstructured":"Society of Automotive Engineers (SAE) International. 2018. Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles.  Society of Automotive Engineers (SAE) International. 2018. Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.17077\/drivingassessment.1394"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2014.6907445"},{"key":"e_1_2_1_46_1","unstructured":"Veronika Weinbeer T. Muhr Klaus Bengler C. Baur J. Radlmayr and J. Bill. 2017. Highly automated driving: How to get the driver drowsy and how does drowsiness influence various take-over-aspects?. In 8. Tagung Fahrerassistenz. Lehrstuhl f\u00fcr Fahrzeugtechnik mit T\u00dc V S\u00dc D Akademie M\u00fcnchen.  Veronika Weinbeer T. Muhr Klaus Bengler C. Baur J. Radlmayr and J. Bill. 2017. Highly automated driving: How to get the driver drowsy and how does drowsiness influence various take-over-aspects?. In 8. Tagung Fahrerassistenz. Lehrstuhl f\u00fcr Fahrzeugtechnik mit T\u00dc V S\u00dc D Akademie M\u00fcnchen."},{"key":"e_1_2_1_47_1","volume-title":"Development and Evaluation of a Naturalistic Observer Rating of Drowsiness Protocol Final Report","author":"Wiegand Douglas M","year":"2009"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1016\/0001-4575(94)90019-1"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3394980","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3394980","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:48:00Z","timestamp":1750193280000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3394980"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,18]]},"references-count":45,"journal-issue":{"issue":"EICS","published-print":{"date-parts":[[2020,6,18]]}},"alternative-id":["10.1145\/3394980"],"URL":"https:\/\/doi.org\/10.1145\/3394980","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,18]]},"assertion":[{"value":"2020-06-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}