{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,4]],"date-time":"2025-07-04T05:55:46Z","timestamp":1751608546820,"version":"3.41.0"},"reference-count":35,"publisher":"Association for Computing Machinery (ACM)","issue":"ETRA","license":[{"start":{"date-parts":[[2022,5,13]],"date-time":"2022-05-13T00:00:00Z","timestamp":1652400000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2022,5,13]]},"abstract":"<jats:p>After the success of smartphones and smartwatches, smart glasses are expected to be the next smart wearable. While novel display technology allows the seamlessly embedding of content into the FOV, interaction methods with glasses, requiring the user for active interaction, limiting the user experience. One way to improve this and drive immersive augmentation is to reduce user interactions to a necessary minimum by adding context awareness to smart glasses. For this, we propose an approach based on human activity recognition, which incorporates features, derived from the user's head- and eye-movement. Towards this goal, we combine an commercial eye-tracker and an IMU to capture eye- and head-movement features of 7 activities performed by 20 participants. From a methodological perspective, we introduce U-HAR, a convolutional network optimized for activity recognition. By applying a few-shot learning, our model reaches an macro-F1-score of 86.59%, allowing us to derive contextual information.<\/jats:p>","DOI":"10.1145\/3530884","type":"journal-article","created":{"date-parts":[[2022,5,13]],"date-time":"2022-05-13T22:17:43Z","timestamp":1652480263000},"page":"1-19","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["U-HAR"],"prefix":"10.1145","volume":"6","author":[{"given":"Johannes","family":"Meyer","sequence":"first","affiliation":[{"name":"Robert Bosch GmbH, Renningen, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Adrian","family":"Frank","sequence":"additional","affiliation":[{"name":"University of T\u00fcbingen, T\u00fcbingen, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Thomas","family":"Schlebusch","sequence":"additional","affiliation":[{"name":"Robert Bosch GmbH, Renningen, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Enkelejda","family":"Kasneci","sequence":"additional","affiliation":[{"name":"University of T\u00fcbingen, T\u00fcbingen, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,5,13]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"Amazon. 2019. Echo Frames - Eyeglasses with Alexa. online. https:\/\/www.amazon.com\/Staging-Product-Not-Retail-Sale\/dp\/B07W72XKPJ  Amazon. 2019. Echo Frames - Eyeglasses with Alexa. online. https:\/\/www.amazon.com\/Staging-Product-Not-Retail-Sale\/dp\/B07W72XKPJ"},{"key":"e_1_2_2_2_1","volume-title":"Driver-Activity Recognition in the Context of Conditionally Autonomous Driving. In 2015 IEEE 18th International Conference on Intelligent Transportation Systems . 1652--1657","author":"Braunagel C.","year":"2015","unstructured":"C. Braunagel , E. Kasneci , W. Stolzmann , and W. Rosenstiel . 2015 . Driver-Activity Recognition in the Context of Conditionally Autonomous Driving. In 2015 IEEE 18th International Conference on Intelligent Transportation Systems . 1652--1657 . https:\/\/doi.org\/10.1109\/ITSC. 2015 .268 10.1109\/ITSC.2015.268 C. Braunagel, E. Kasneci, W. Stolzmann, and W. Rosenstiel. 2015. Driver-Activity Recognition in the Context of Conditionally Autonomous Driving. In 2015 IEEE 18th International Conference on Intelligent Transportation Systems . 1652--1657. https:\/\/doi.org\/10.1109\/ITSC.2015.268"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2499621"},{"key":"e_1_2_2_4_1","article-title":"Eye Movement Analysis for Activity Recognition Using Electrooculography. Pattern Analysis and Machine Intelligence","volume":"33","author":"Bulling Andreas","year":"2011","unstructured":"Andreas Bulling , Jamie Ward , Hans Gellersen , and Gerhard Tr\u00f6ster . 2011 . Eye Movement Analysis for Activity Recognition Using Electrooculography. Pattern Analysis and Machine Intelligence , IEEE Transactions on , Vol. 33 (05 2011), 741 -- 753. https:\/\/doi.org\/10.1109\/TPAMI.2010.86 10.1109\/TPAMI.2010.86 Andreas Bulling, Jamie Ward, Hans Gellersen, and Gerhard Tr\u00f6ster. 2011. Eye Movement Analysis for Activity Recognition Using Electrooculography. Pattern Analysis and Machine Intelligence, IEEE Transactions on , Vol. 33 (05 2011), 741 -- 753. https:\/\/doi.org\/10.1109\/TPAMI.2010.86","journal-title":"IEEE Transactions on"},{"volume-title":"Robust Recognition of Reading Activity in Transit Using Wearable Electrooculography","author":"Bulling Andreas","key":"e_1_2_2_5_1","unstructured":"Andreas Bulling , Jamie A. Ward , Hans Gellersen , and Gerhard Tr\u00f6ster . 2008. Robust Recognition of Reading Activity in Transit Using Wearable Electrooculography . In Pervasive Computing, Jadwiga Indulska, Donald J. Patterson, Tom Rodden, and Max Ott (Eds.). Springer Berlin Heidelberg , Berlin, Heidelberg , 19--37. Andreas Bulling, Jamie A. Ward, Hans Gellersen, and Gerhard Tr\u00f6ster. 2008. Robust Recognition of Reading Activity in Transit Using Wearable Electrooculography. In Pervasive Computing, Jadwiga Indulska, Donald J. Patterson, Tom Rodden, and Max Ott (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 19--37."},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470697"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2019.2921977"},{"volume-title":"Machine Learning and Data Mining in Pattern Recognition","author":"Chikhaoui Belkacem","key":"e_1_2_2_8_1","unstructured":"Belkacem Chikhaoui , Frank Gouineau , and Martin Sotir . 2018. A CNN Based Transfer Learning Model for Automatic Activity Recognition from Accelerometer Sensors . In Machine Learning and Data Mining in Pattern Recognition , Petra Perner (Ed.). Springer International Publishing , Cham , 302--315. Belkacem Chikhaoui, Frank Gouineau, and Martin Sotir. 2018. A CNN Based Transfer Learning Model for Automatic Activity Recognition from Accelerometer Sensors. In Machine Learning and Data Mining in Pattern Recognition, Petra Perner (Ed.). Springer International Publishing, Cham, 302--315."},{"key":"e_1_2_2_9_1","doi-asserted-by":"crossref","unstructured":"Philip Elspas Yannick Klose Simon T Isele Johannes Bach and Eric Sax. 2021. Time Series Segmentation for Driving Scenario Detection with Fully Convolutional Networks.. In VEHITS. 56--64.  Philip Elspas Yannick Klose Simon T Isele Johannes Bach and Eric Sax. 2021. Time Series Segmentation for Driving Scenario Detection with Fully Convolutional Networks.. In VEHITS. 56--64.","DOI":"10.5220\/0010404700560064"},{"key":"e_1_2_2_10_1","article-title":"PULP-NN: accelerating quantized neural networks on parallel ultra-low-power RISC-V processors","volume":"378","author":"Garofalo Angelo","year":"2020","unstructured":"Angelo Garofalo , Manuele Rusci , Francesco Conti , Davide Rossi , and Luca Benini . 2020 . PULP-NN: accelerating quantized neural networks on parallel ultra-low-power RISC-V processors . Philosophical Transactions of the Royal Society A , Vol. 378 , 2164 (2020), 20190155. Angelo Garofalo, Manuele Rusci, Francesco Conti, Davide Rossi, and Luca Benini. 2020. PULP-NN: accelerating quantized neural networks on parallel ultra-low-power RISC-V processors. Philosophical Transactions of the Royal Society A , Vol. 378, 2164 (2020), 20190155.","journal-title":"Philosophical Transactions of the Royal Society A"},{"key":"e_1_2_2_11_1","volume-title":"https:\/\/www.google.com\/glass\/start\/. Online","author":"Glass Glass","year":"2021","unstructured":"Google. 2021. Glass - Glass . https:\/\/www.google.com\/glass\/start\/. Online ; accessed 15 April 2021 . Google. 2021. Glass - Glass . https:\/\/www.google.com\/glass\/start\/. Online; accessed 15 April 2021."},{"key":"#cr-split#-e_1_2_2_12_1.1","doi-asserted-by":"crossref","unstructured":"Tomoki Hayashi Masafumi Nishida Norihide Kitaoka Tomoki TODA and Kazuya Takeda. 2018. Daily Activity Recognition with Large-Scaled Real-Life Recording Datasets Based on Deep Neural Network Using Multi-Modal Signals. IEICE Transactions on Fundamentals of Electronics Communications and Computer Sciences Vol. E101.A (01 2018) 199--210. https:\/\/doi.org\/10.1587\/transfun.E101.A.199 10.1587\/transfun.E101.A.199","DOI":"10.1587\/transfun.E101.A.199"},{"key":"#cr-split#-e_1_2_2_12_1.2","doi-asserted-by":"crossref","unstructured":"Tomoki Hayashi Masafumi Nishida Norihide Kitaoka Tomoki TODA and Kazuya Takeda. 2018. Daily Activity Recognition with Large-Scaled Real-Life Recording Datasets Based on Deep Neural Network Using Multi-Modal Signals. IEICE Transactions on Fundamentals of Electronics Communications and Computer Sciences Vol. E101.A (01 2018) 199--210. https:\/\/doi.org\/10.1587\/transfun.E101.A.199","DOI":"10.1587\/transfun.E101.A.199"},{"key":"e_1_2_2_13_1","volume-title":"ACM International Conference Proceeding Series, 150--153","author":"Ishimaru Shoya","year":"2014","unstructured":"Shoya Ishimaru , Kai Kunze , Koichi Kise , Jens Weppner , Andreas Dengel , Paul Lukowicz , and Andreas Bulling . 2014 a. In the Blink of an Eye - Combining Head Motion and Eye Blink Frequency for Activity Recognition with Google Glass . ACM International Conference Proceeding Series, 150--153 . https:\/\/doi.org\/10.1145\/2582051.2582066 10.1145\/2582051.2582066 Shoya Ishimaru, Kai Kunze, Koichi Kise, Jens Weppner, Andreas Dengel, Paul Lukowicz, and Andreas Bulling. 2014a. In the Blink of an Eye - Combining Head Motion and Eye Blink Frequency for Activity Recognition with Google Glass. ACM International Conference Proceeding Series, 150--153. https:\/\/doi.org\/10.1145\/2582051.2582066"},{"key":"#cr-split#-e_1_2_2_14_1.1","doi-asserted-by":"crossref","unstructured":"Shoya Ishimaru Kai Kunze Katsuma Tanaka Yuji Uema Koichi Kise and Masahiko Inami. 2014b. Smarter Eyewear- Using Commercial EOG Glasses for Activity Recognition. UbiComp 2014 - Adjunct Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing 153--156. https:\/\/doi.org\/10.1145\/2638728.2638795 10.1145\/2638728.2638795","DOI":"10.1145\/2638728.2638795"},{"key":"#cr-split#-e_1_2_2_14_1.2","doi-asserted-by":"crossref","unstructured":"Shoya Ishimaru Kai Kunze Katsuma Tanaka Yuji Uema Koichi Kise and Masahiko Inami. 2014b. Smarter Eyewear- Using Commercial EOG Glasses for Activity Recognition. UbiComp 2014 - Adjunct Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing 153--156. https:\/\/doi.org\/10.1145\/2638728.2638795","DOI":"10.1145\/2638728.2638795"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2638728.2641695"},{"key":"e_1_2_2_16_1","doi-asserted-by":"crossref","unstructured":"B.C. Kress. 2020. Optical Architectures for Augmented- Virtual- and Mixed-Reality Headsets .SPIE. 5--12 pages. 2019050126  B.C. Kress. 2020. Optical Architectures for Augmented- Virtual- and Mixed-Reality Headsets .SPIE. 5--12 pages. 2019050126","DOI":"10.1117\/3.2559304"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1964897.1964918"},{"volume-title":"GazeGraph: Graph-Based Few-Shot Cognitive Context Sensing from Human Visual Behavior","author":"Lan Guohao","key":"e_1_2_2_18_1","unstructured":"Guohao Lan , Bailey Heit , Tim Scargill , and Maria Gorlatova . 2020. GazeGraph: Graph-Based Few-Shot Cognitive Context Sensing from Human Visual Behavior . Association for Computing Machinery , New York, NY, USA , 422--435. https:\/\/doi.org\/10.1145\/3384419.3430774 10.1145\/3384419.3430774 Guohao Lan, Bailey Heit, Tim Scargill, and Maria Gorlatova. 2020. GazeGraph: Graph-Based Few-Shot Cognitive Context Sensing from Human Visual Behavior .Association for Computing Machinery, New York, NY, USA, 422--435. https:\/\/doi.org\/10.1145\/3384419.3430774"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-18656-2_20"},{"key":"e_1_2_2_20_1","first-page":"24 pages. https","volume-title":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.","volume":"5","author":"Meyer Johannes","year":"2022","unstructured":"Johannes Meyer , Adrian Frank , Thomas Schlebusch , and Enkeljeda Kasneci . 2022 . A CNN-Based Human Activity Recognition System Combining a Laser Feedback Interferometry Eye Movement Sensor and an IMU for Context-Aware Smart Glasses . Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. , Vol. 5 , 4, Article 172 (dec 2022), bibinfonumpages 24 pages. https:\/\/doi.org\/10 .1145\/3494998 10.1145\/3494998 Johannes Meyer, Adrian Frank, Thomas Schlebusch, and Enkeljeda Kasneci. 2022. A CNN-Based Human Activity Recognition System Combining a Laser Feedback Interferometry Eye Movement Sensor and an IMU for Context-Aware Smart Glasses. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. , Vol. 5, 4, Article 172 (dec 2022), bibinfonumpages24 pages. https:\/\/doi.org\/10.1145\/3494998"},{"key":"e_1_2_2_21_1","volume-title":"2020 a. A Novel Camera-Free Eye Tracking Sensor for Augmented Reality based on Laser Scanning","author":"Meyer Johannes","year":"2020","unstructured":"Johannes Meyer , Thomas Schlebusch , Wolfgang Fuhl , and Enkelejda Kasneci . 2020 a. A Novel Camera-Free Eye Tracking Sensor for Augmented Reality based on Laser Scanning . IEEE Sensors Journal ( 2020 ), 1--9. Johannes Meyer, Thomas Schlebusch, Wolfgang Fuhl, and Enkelejda Kasneci. 2020 a. A Novel Camera-Free Eye Tracking Sensor for Augmented Reality based on Laser Scanning. IEEE Sensors Journal (2020), 1--9."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411763"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3379156.3391352"},{"key":"e_1_2_2_24_1","volume-title":"Poul J\u00f8rgen Jennum, and Christian Igel","author":"Perslev Mathias","year":"2021","unstructured":"Mathias Perslev , Sune Darkner , Lykke Kempfner , Miki Nikolic , Poul J\u00f8rgen Jennum, and Christian Igel . 2021 . U-Sleep: resilient high-frequency sleep staging. NPJ digital medicine , Vol. 4 , 1 (2021), 1--12. Mathias Perslev, Sune Darkner, Lykke Kempfner, Miki Nikolic, Poul J\u00f8rgen Jennum, and Christian Igel. 2021. U-Sleep: resilient high-frequency sleep staging. NPJ digital medicine , Vol. 4, 1 (2021), 1--12."},{"key":"e_1_2_2_25_1","first-page":"4415","article-title":"U-Time: A Fully Convolutional Network for Time Series Segmentation Applied to Sleep Staging","volume":"32","author":"Perslev Mathias","year":"2019","unstructured":"Mathias Perslev , Michael Jensen , Sune Darkner , Poul J\u00f8rgen Jennum , and Christian Igel . 2019 . U-Time: A Fully Convolutional Network for Time Series Segmentation Applied to Sleep Staging . Advances in Neural Information Processing Systems , Vol. 32 (2019), 4415 -- 4426 . Mathias Perslev, Michael Jensen, Sune Darkner, Poul J\u00f8rgen Jennum, and Christian Igel. 2019. U-Time: A Fully Convolutional Network for Time Series Segmentation Applied to Sleep Staging. Advances in Neural Information Processing Systems , Vol. 32 (2019), 4415--4426.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00474"},{"volume-title":"CalibMe: Fast and Unsupervised Eye Tracker Calibration for Gaze-Based Pervasive Human-Computer Interaction","author":"Santini Thiago","key":"e_1_2_2_28_1","unstructured":"Thiago Santini , Wolfgang Fuhl , and Enkelejda Kasneci . 2017. CalibMe: Fast and Unsupervised Eye Tracker Calibration for Gaze-Based Pervasive Human-Computer Interaction . Association for Computing Machinery , New York, NY, USA , 2594--2605. https:\/\/doi.org\/10.1145\/3025453.3025950 10.1145\/3025453.3025950 Thiago Santini, Wolfgang Fuhl, and Enkelejda Kasneci. 2017. CalibMe: Fast and Unsupervised Eye Tracker Calibration for Gaze-Based Pervasive Human-Computer Interaction .Association for Computing Machinery, New York, NY, USA, 2594--2605. https:\/\/doi.org\/10.1145\/3025453.3025950"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/MEMSYS.2017.7863402"},{"key":"e_1_2_2_30_1","doi-asserted-by":"crossref","unstructured":"N. Sarkar D. Strathearn G. Lee M. Olfat A. Rohani and R. R. Mansour. 2015. A large angle low voltage small footprint micromirror for eye tracking and near-eye display applications. In 2015 Transducers - 2015 18th International Conference on Solid-State Sensors Actuators and Microsystems (TRANSDUCERS). 855--858.  N. Sarkar D. Strathearn G. Lee M. Olfat A. Rohani and R. R. Mansour. 2015. A large angle low voltage small footprint micromirror for eye tracking and near-eye display applications. In 2015 Transducers - 2015 18th International Conference on Solid-State Sensors Actuators and Microsystems (TRANSDUCERS). 855--858.","DOI":"10.1109\/TRANSDUCERS.2015.7181058"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3229434.3229439"},{"key":"e_1_2_2_32_1","unstructured":"Lech Swirski and Neil Dodgson. 2013. A fully-automatic temporal approach to single camera glint-free 3D eye model fitting.  Lech Swirski and Neil Dodgson. 2013. A fully-automatic temporal approach to single camera glint-free 3D eye model fitting."},{"key":"e_1_2_2_33_1","volume-title":"Deep Face Recognition: A Survey. Neurocomputing","author":"Wang Mei","year":"2020","unstructured":"Mei Wang and Weihong Deng . 2020. Deep Face Recognition: A Survey. Neurocomputing ( 2020 ). Mei Wang and Weihong Deng. 2020. Deep Face Recognition: A Survey. Neurocomputing (2020)."}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3530884","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3530884","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:09:26Z","timestamp":1750183766000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3530884"}},"subtitle":["A Convolutional Approach to Human Activity Recognition Combining Head and Eye Movements for Context-Aware Smart Glasses"],"short-title":[],"issued":{"date-parts":[[2022,5,13]]},"references-count":35,"journal-issue":{"issue":"ETRA","published-print":{"date-parts":[[2022,5,13]]}},"alternative-id":["10.1145\/3530884"],"URL":"https:\/\/doi.org\/10.1145\/3530884","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2022,5,13]]},"assertion":[{"value":"2022-05-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}