{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T11:23:03Z","timestamp":1780053783472,"version":"3.54.0"},"reference-count":61,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2021,12,27]],"date-time":"2021-12-27T00:00:00Z","timestamp":1640563200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2021,12,27]]},"abstract":"<jats:p>Smart glasses are considered the next breakthrough in wearables. As the successor of smart watches and smart ear wear, they promise to extend reality by immersive embedding of content in the user's field of view. While advancements in display technology seems to fulfill this promises, interaction concepts are derived from established wearable concepts like touch interaction or voice interaction, preventing full immersion as they require the user to frequently interact with the glasses. To minimize interactions, we propose to add context-awareness to smart glasses through human activity recognition (HAR) by combining head- and eye movement features to recognize a wide range of activities. To measure eye movements in unobtrusive way, we propose laser feedback interferometry (LFI) sensors. These tiny low power sensors are highly robust to ambient light. We combine LFI sensors and an IMU to collect eye and head movement features from 15 participants performing 7 cognitive and physical activities, leading to a unique data set. To recognize activities we propose a 1D-CNN model and apply transfer learning to personalize the classification, leading to an outstanding macro-F1 score of 88.15 % which outperforms state of the art methods. Finally, we discuss the applicability of the proposed system in a smart glasses setup.<\/jats:p>","DOI":"10.1145\/3494998","type":"journal-article","created":{"date-parts":[[2021,12,30]],"date-time":"2021-12-30T17:40:33Z","timestamp":1640886033000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":27,"title":["A CNN-based Human Activity Recognition System Combining a Laser Feedback Interferometry Eye Movement Sensor and an IMU for Context-aware Smart Glasses"],"prefix":"10.1145","volume":"5","author":[{"given":"Johannes","family":"Meyer","sequence":"first","affiliation":[{"name":"Corporate Sector Research and Advance Engineering, Robert Bosch GmbH, Renningen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adrian","family":"Frank","sequence":"additional","affiliation":[{"name":"Corporate Sector Research and Advance Engineering, Robert Bosch GmbH, Renningen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"Schlebusch","sequence":"additional","affiliation":[{"name":"Corporate Sector Research and Advance Engineering, Robert Bosch GmbH, Renningen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Enkeljeda","family":"Kasneci","sequence":"additional","affiliation":[{"name":"Human-Computer Interaction, University of T\u00fcbingen, T\u00fcbingen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,12,30]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Amazon. 2019. Echo Frames - Eyeglasses with Alexa. online. https:\/\/www.amazon.com\/Staging-Product-Not-Retail-Sale\/dp\/B07W72XKPJ  Amazon. 2019. Echo Frames - Eyeglasses with Alexa. online. https:\/\/www.amazon.com\/Staging-Product-Not-Retail-Sale\/dp\/B07W72XKPJ"},{"key":"e_1_2_1_2_1","volume-title":"Ahmad Nickabadi, and Alexandre Alahi.","author":"Azar Sina Mokhtarzadeh","year":"2019","unstructured":"Sina Mokhtarzadeh Azar , Mina Ghadimi Atigh , Ahmad Nickabadi, and Alexandre Alahi. 2019 . Convolutional Relational Machine for Group Activity Recognition . arXiv:1904.03308 [cs.CV] Sina Mokhtarzadeh Azar, Mina Ghadimi Atigh, Ahmad Nickabadi, and Alexandre Alahi. 2019. Convolutional Relational Machine for Group Activity Recognition. arXiv:1904.03308 [cs.CV]"},{"key":"e_1_2_1_3_1","volume-title":"Intille","author":"Bao Ling","year":"2004","unstructured":"Ling Bao and Stephen S . Intille . 2004 . Activity Recognition from User-Annotated Acceleration Data. In Pervasive Computing, Alois Ferscha and Friedemann Mattern (Eds.). Springer Berlin Heidelberg , Berlin, Heidelberg, 1--17. Ling Bao and Stephen S. Intille. 2004. Activity Recognition from User-Annotated Acceleration Data. In Pervasive Computing, Alois Ferscha and Friedemann Mattern (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 1--17."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCOM.2017.1700030"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/MITS.2017.2743171"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC.2015.268"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2499621"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2010.86"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2134203.2134205"},{"key":"e_1_2_1_10_1","volume-title":"Robust Recognition of Reading Activity in Transit Using Wearable Electrooculography","author":"Bulling Andreas","unstructured":"Andreas Bulling , Jamie A. Ward , Hans Gellersen , and Gerhard Tr\u00f6ster . 2008. Robust Recognition of Reading Activity in Transit Using Wearable Electrooculography . In Pervasive Computing, Jadwiga Indulska, Donald J. Patterson, Tom Rodden, and Max Ott (Eds.). Springer Berlin Heidelberg , Berlin, Heidelberg , 19--37. Andreas Bulling, Jamie A. Ward, Hans Gellersen, and Gerhard Tr\u00f6ster. 2008. Robust Recognition of Reading Activity in Transit Using Wearable Electrooculography. In Pervasive Computing, Jadwiga Indulska, Donald J. Patterson, Tom Rodden, and Max Ott (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 19--37."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470697"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2019.2921977"},{"key":"e_1_2_1_13_1","unstructured":"Kaixuan Chen Dalin Zhang Lina Yao Bin Guo Zhiwen Yu and Yunhao Liu. 2020. Deep Learning for Sensor-based Human Activity Recognition: Overview Challenges and Opportunities. arXiv:2001.07416 [cs.HC]  Kaixuan Chen Dalin Zhang Lina Yao Bin Guo Zhiwen Yu and Yunhao Liu. 2020. Deep Learning for Sensor-based Human Activity Recognition: Overview Challenges and Opportunities. arXiv:2001.07416 [cs.HC]"},{"key":"e_1_2_1_14_1","volume-title":"Machine Learning and Data Mining in Pattern Recognition","author":"Chikhaoui Belkacem","unstructured":"Belkacem Chikhaoui , Frank Gouineau , and Martin Sotir . 2018. A CNN Based Transfer Learning Model for Automatic Activity Recognition from Accelerometer Sensors . In Machine Learning and Data Mining in Pattern Recognition , Petra Perner (Ed.). Springer International Publishing , Cham , 302--315. Belkacem Chikhaoui, Frank Gouineau, and Martin Sotir. 2018. A CNN Based Transfer Learning Model for Automatic Activity Recognition from Accelerometer Sensors. In Machine Learning and Data Mining in Pattern Recognition, Petra Perner (Ed.). Springer International Publishing, Cham, 302--315."},{"key":"e_1_2_1_15_1","unstructured":"Epson. 2020. Moverio BT-300 Smart Glasses. https:\/\/www.epson.com.hk\/For-Home\/Wearables\/Smart-Glasses\/Moverio-BT-300-Smart-Glasses\/p\/V11H756060  Epson. 2020. Moverio BT-300 Smart Glasses. https:\/\/www.epson.com.hk\/For-Home\/Wearables\/Smart-Glasses\/Moverio-BT-300-Smart-Glasses\/p\/V11H756060"},{"key":"e_1_2_1_16_1","volume-title":"W.","author":"Tonsen M.","year":"2016","unstructured":"Tonsen M. Bulling A. et al. Fuhl , W. 2016 . Pupil detection for head-mounted eye tracking in the wild: an evaluation of the state of the art. Springer Machine Vision and Applications 27 (2016). Tonsen M. Bulling A. et al. Fuhl, W. 2016. Pupil detection for head-mounted eye tracking in the wild: an evaluation of the state of the art. Springer Machine Vision and Applications 27 (2016)."},{"key":"e_1_2_1_17_1","article-title":"PULP-NN: accelerating quantized neural networks on parallel ultra-low-power RISC-V processors","volume":"378","author":"Garofalo Angelo","year":"2020","unstructured":"Angelo Garofalo , Manuele Rusci , Francesco Conti , Davide Rossi , and Luca Benini . 2020 . PULP-NN: accelerating quantized neural networks on parallel ultra-low-power RISC-V processors . Philosophical Transactions of the Royal Society A 378 , 2164 (2020), 20190155. Angelo Garofalo, Manuele Rusci, Francesco Conti, Davide Rossi, and Luca Benini. 2020. PULP-NN: accelerating quantized neural networks on parallel ultra-low-power RISC-V processors. Philosophical Transactions of the Royal Society A 378, 2164 (2020), 20190155.","journal-title":"Philosophical Transactions of the Royal Society A"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1088\/1464-4258"},{"key":"e_1_2_1_19_1","volume-title":"https:\/\/www.google.com\/glass\/start\/. Online","author":"Glass Google LLC.","year":"2021","unstructured":"Google LLC. 2021. Glass - Glass . https:\/\/www.google.com\/glass\/start\/. Online ; accessed 15 April 2021 . Google LLC. 2021. Glass - Glass. https:\/\/www.google.com\/glass\/start\/. Online; accessed 15 April 2021."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1587\/transfun.E101.A.199"},{"key":"e_1_2_1_21_1","volume-title":"Interacting with the Computer Using Gaze Gestures. In International Conference on Human Computer Interaction.","author":"Heiko Drewes Albrecht Schmidt","year":"2007","unstructured":"Albrecht Schmidt Heiko Drewes . 2007 . Interacting with the Computer Using Gaze Gestures. In International Conference on Human Computer Interaction. Albrecht Schmidt Heiko Drewes. 2007. Interacting with the Computer Using Gaze Gestures. In International Conference on Human Computer Interaction."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2017.09.027"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2582051.2582066"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2638728.2638795"},{"key":"e_1_2_1_25_1","unstructured":"Md Islam Shuji Sakamoto Yoshihiro Yamada Andrew Vargo Motoi Iwata Masakazu Iwamura Koichi Kise etal 2020. Self-supervised Deep Learning for Reading Activity Classification. arXiv preprint arXiv:2012.03598 (2020).  Md Islam Shuji Sakamoto Yoshihiro Yamada Andrew Vargo Motoi Iwata Masakazu Iwamura Koichi Kise et al. 2020. Self-supervised Deep Learning for Reading Activity Classification. arXiv preprint arXiv:2012.03598 (2020)."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411763.3451621"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2638728.2641695"},{"key":"e_1_2_1_28_1","volume-title":"Kingma and Jimmy Ba","author":"Diederik","year":"2017","unstructured":"Diederik P. Kingma and Jimmy Ba . 2017 . Adam : A Method for Stochastic Optimization . arXiv:1412.6980 [cs.LG] Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs.LG]"},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research","volume":"1944","author":"Kumar Ashish","year":"2017","unstructured":"Ashish Kumar , Saurabh Goyal , and Manik Varma . 2017 . Resource-efficient Machine Learning in 2 KB RAM for the Internet of Things . In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research , Vol. 70), Doina Precup and Yee Whye Teh (Eds.). PMLR, 1935-- 1944 . http:\/\/proceedings.mlr.press\/v70\/kumar17a.html Ashish Kumar, Saurabh Goyal, and Manik Varma. 2017. Resource-efficient Machine Learning in 2 KB RAM for the Internet of Things. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 70), Doina Precup and Yee Whye Teh (Eds.). PMLR, 1935--1944. http:\/\/proceedings.mlr.press\/v70\/kumar17a.html"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1964897.1964918"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3384419.3430774"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/JQE.1980.1070479"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2831081"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2994551.2994569"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-18656-2_20"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4471-6392-3_3"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSEN.2020.3011985"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3379156.3391352"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1117\/12.2593772"},{"key":"e_1_2_1_40_1","volume-title":"Azure Kinect DK - Develop AI models | Microsoft Azure. https:\/\/azure.microsoft.com\/en-gb\/services\/kinect-dk\/. Online","author":"Microsoft Corporation","year":"2021","unstructured":"Microsoft Corporation . 2021. Azure Kinect DK - Develop AI models | Microsoft Azure. https:\/\/azure.microsoft.com\/en-gb\/services\/kinect-dk\/. Online ; accessed 14 April 2021 . Microsoft Corporation. 2021. Azure Kinect DK - Develop AI models | Microsoft Azure. https:\/\/azure.microsoft.com\/en-gb\/services\/kinect-dk\/. Online; accessed 14 April 2021."},{"key":"e_1_2_1_41_1","volume-title":"Kinect - Windows app development. https:\/\/developer.microsoft.com\/en-gb\/windows\/kinect\/. Online","author":"Microsoft Corporation","year":"2021","unstructured":"Microsoft Corporation . 2021. Kinect - Windows app development. https:\/\/developer.microsoft.com\/en-gb\/windows\/kinect\/. Online ; accessed 14 April 2021 . Microsoft Corporation. 2021. Kinect - Windows app development. https:\/\/developer.microsoft.com\/en-gb\/windows\/kinect\/. Online; accessed 14 April 2021."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-019-01307-0"},{"key":"e_1_2_1_43_1","volume-title":"https:\/\/www.nintendo.co.uk\/Nintendo-Switch\/Nintendo-Switch-1148779.html. Online","author":"Nintendo Co.","year":"2021","unstructured":"Nintendo Co. , Ltd. 2021. Nintendo Switch | Nintendo Switch Family | Nintendo . https:\/\/www.nintendo.co.uk\/Nintendo-Switch\/Nintendo-Switch-1148779.html. Online ; accessed 14 April 2021 . Nintendo Co., Ltd. 2021. Nintendo Switch | Nintendo Switch Family | Nintendo. https:\/\/www.nintendo.co.uk\/Nintendo-Switch\/Nintendo-Switch-1148779.html. Online; accessed 14 April 2021."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.3390\/s16010115"},{"key":"e_1_2_1_45_1","volume-title":"Poul J\u00f8rgen Jennum, and Christian Igel","author":"Perslev Mathias","year":"2021","unstructured":"Mathias Perslev , Sune Darkner , Lykke Kempfner , Miki Nikolic , Poul J\u00f8rgen Jennum, and Christian Igel . 2021 . U-Sleep: resilient high-frequency sleep staging. NPJ digital medicine 4, 1 (2021), 1--12. Mathias Perslev, Sune Darkner, Lykke Kempfner, Miki Nikolic, Poul J\u00f8rgen Jennum, and Christian Igel. 2021. U-Sleep: resilient high-frequency sleep staging. NPJ digital medicine 4, 1 (2021), 1--12."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1117\/12.775131"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-2239-3"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2016.04.032"},{"key":"e_1_2_1_49_1","volume-title":"https:\/\/landing.runtopia.net\/. Online","author":"Ltd Runtopia Technology","year":"2021","unstructured":"Runtopia Technology Ltd . 2021. RUNTOPIA| REACH. Faster , Longer, Safer. https:\/\/landing.runtopia.net\/. Online ; accessed 14 April 2021 . Runtopia Technology Ltd. 2021. RUNTOPIA|REACH. Faster, Longer, Safer. https:\/\/landing.runtopia.net\/. Online; accessed 14 April 2021."},{"key":"e_1_2_1_50_1","volume-title":"https:\/\/www.salted.ltd\/en\/insole. Online","author":"Ltd SALTED","year":"2021","unstructured":"SALTED Ltd . 2021. SALTED. https:\/\/www.salted.ltd\/en\/insole. Online ; accessed 14 April 2021 . SALTED Ltd. 2021. SALTED. https:\/\/www.salted.ltd\/en\/insole. Online; accessed 14 April 2021."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/MEMSYS.2017.7863402"},{"key":"e_1_2_1_52_1","volume-title":"Sensoria Home Page. https:\/\/www.sensoriafitness.com\/. Online","author":"Sensoria Inc. 2021.","year":"2021","unstructured":"Sensoria Inc. 2021. Sensoria Home Page. https:\/\/www.sensoriafitness.com\/. Online ; accessed 14 April 2021 . Sensoria Inc. 2021. Sensoria Home Page. https:\/\/www.sensoriafitness.com\/. Online; accessed 14 April 2021."},{"key":"e_1_2_1_53_1","unstructured":"Bosch Sensortec. 2020. BMI270 6-axis smart low-power Inertial Measurement Unit for high-performance applications. https:\/\/www.bosch-sensortec.com\/products\/motion-sensors\/imus\/bmi270  Bosch Sensortec. 2020. BMI270 6-axis smart low-power Inertial Measurement Unit for high-performance applications. https:\/\/www.bosch-sensortec.com\/products\/motion-sensors\/imus\/bmi270"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2750858.2807520"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3229434.3229439"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1364\/AOP.7.000570"},{"key":"e_1_2_1_57_1","volume-title":"Chris Kay Baumann, and Kai Dierkes","author":"Tonsen Marc","year":"2020","unstructured":"Marc Tonsen , Chris Kay Baumann, and Kai Dierkes . 2020 . A High-Level Description and Performance Evaluation of Pupil Invisible . arXiv:2009.00508 [cs.CV] Marc Tonsen, Chris Kay Baumann, and Kai Dierkes. 2020. A High-Level Description and Performance Evaluation of Pupil Invisible. arXiv:2009.00508 [cs.CV]"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.4108\/eai.28-9-2015.2261470"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11036-019-01445-x"},{"key":"e_1_2_1_60_1","volume-title":"Deep Face Recognition: A Survey. Neurocomputing","author":"Wang Mei","year":"2020","unstructured":"Mei Wang and Weihong Deng . 2020. Deep Face Recognition: A Survey. Neurocomputing ( 2020 ). Mei Wang and Weihong Deng. 2020. Deep Face Recognition: A Survey. Neurocomputing (2020)."},{"key":"e_1_2_1_61_1","volume-title":"Xiaoli li, and Priyadarsini Shonali","author":"Yang Jian-Bo","year":"2015","unstructured":"Jian-Bo Yang , Nguyen Nhut , Phyo San , Xiaoli li, and Priyadarsini Shonali . 2015 . Deep Convolutional Neural Networks on Multichannel Time Series for Human Activity Recognition. IJCAI ( 07 2015). Jian-Bo Yang, Nguyen Nhut, Phyo San, Xiaoli li, and Priyadarsini Shonali. 2015. Deep Convolutional Neural Networks on Multichannel Time Series for Human Activity Recognition. IJCAI (07 2015)."}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3494998","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3494998","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:19:00Z","timestamp":1750191540000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3494998"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,27]]},"references-count":61,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,12,27]]}},"alternative-id":["10.1145\/3494998"],"URL":"https:\/\/doi.org\/10.1145\/3494998","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,27]]},"assertion":[{"value":"2021-12-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}