{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T14:45:10Z","timestamp":1781621110046,"version":"3.54.5"},"reference-count":109,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2023,9,27]],"date-time":"2023-09-27T00:00:00Z","timestamp":1695772800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2023,9,27]]},"abstract":"<jats:p>E-learning has emerged as an indispensable educational mode in the post-epidemic era. However, this mode makes it difficult for students to stay engaged in learning without appropriate activity monitoring. Our work explores a promising solution that combines gaze and mouse data to recognize students' activities, thereby facilitating activity monitoring and analysis during e-learning. We initially surveyed 200 students from a local university, finding more acceptance for eye trackers and mouse loggers compared to video surveillance. We then designed eight students' routine digital activities to collect a multimodal dataset and analyze the patterns and correlations between gaze and mouse across various activities. Our proposed Joint Cross-Attention Fusion Net, a multimodal activity recognition framework, leverages the gaze-mouse relationship to yield improved classification performance by integrating cross-modal representations through a cross-attention mechanism and integrating the joint features that characterize gaze-mouse coordination. Evaluation results show that our method can achieve up to 94.87% F1-score in predicting 8-classes activities, with an improvement of at least 7.44% over using gaze or mouse data independently. This research illuminates new possibilities for monitoring student engagement in intelligent education systems, also suggesting a promising strategy for melding perception and action modalities in behavioral analysis across a range of ubiquitous computing environments.<\/jats:p>","DOI":"10.1145\/3610876","type":"journal-article","created":{"date-parts":[[2023,9,27]],"date-time":"2023-09-27T15:45:03Z","timestamp":1695829503000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["Integrating Gaze and Mouse Via Joint Cross-Attention Fusion Net for Students' Activity Recognition in E-learning"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-3181-9445","authenticated-orcid":false,"given":"Rongrong","family":"Zhu","sequence":"first","affiliation":[{"name":"MOE KLINNS Lab, and Xi'an Jiaotong University, Xi'an, Shaanxi, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-6658-5777","authenticated-orcid":false,"given":"Liang","family":"Shi","sequence":"additional","affiliation":[{"name":"MOE KLINNS Lab, and Xi'an Jiaotong University, Xi'an, Shaanxi, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4186-0408","authenticated-orcid":false,"given":"Yunpeng","family":"Song","sequence":"additional","affiliation":[{"name":"MOE KLINNS Lab, and Xi'an Jiaotong University, Xi'an, Shaanxi, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4903-3992","authenticated-orcid":false,"given":"ZhongMin","family":"Cai","sequence":"additional","affiliation":[{"name":"MOE KLINNS Lab, and Xi'an Jiaotong University, Xi'an, Shaanxi, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,9,27]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351227"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445711"},{"key":"e_1_2_1_3_1","doi-asserted-by":"crossref","unstructured":"Dimah Al-Fraihat Mike Joy Jane Sinclair et al. 2020. Evaluating E-learning systems success: An empirical study. Computers in human behavior 102 (2020) 67--86.","DOI":"10.1016\/j.chb.2019.08.004"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1743666.1743688"},{"key":"e_1_2_1_5_1","volume-title":"The coordination of eye, head, and arm movements during reaching at a single visual target. Experimental brain research 46, 2","author":"Biguer B","year":"1982","unstructured":"B Biguer, Marc Jeannerod, and C Prablanc. 1982. The coordination of eye, head, and arm movements during reaching at a single visual target. Experimental brain research 46, 2 (1982), 301--304."},{"key":"e_1_2_1_6_1","volume-title":"Eye--hand coordination in goal-directed aiming. Human movement science 20, 4-5","author":"Binsted Gordon","year":"2001","unstructured":"Gordon Binsted, Romeo Chua, Werner Helsen, and Digby Elliott. 2001. Eye--hand coordination in goal-directed aiming. Human movement science 20, 4-5 (2001), 563--585."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2030112.2030172"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/MPRV.2010.49"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1620545.1620552"},{"key":"e_1_2_1_10_1","volume-title":"Eye movement analysis for activity recognition using electrooculography","author":"Bulling Andreas","year":"2010","unstructured":"Andreas Bulling, Jamie A Ward, Hans Gellersen, and Gerhard Tr\u00f6ster. 2010. Eye movement analysis for activity recognition using electrooculography. IEEE transactions on pattern analysis and machine intelligence 33, 4 (2010), 741--753."},{"key":"e_1_2_1_11_1","volume-title":"Dina Rindlisbacher, Marcus Cheetham, and Hugo Gamboa.","author":"Cepeda C\u00e1tia","year":"2020","unstructured":"C\u00e1tia Cepeda, Maria Camila Dias, Dina Rindlisbacher, Marcus Cheetham, and Hugo Gamboa. 2020. Eye-pointer Coordination in a Decision-making Task Under Uncertainty.. In BIOSIGNALS. 30--37."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3447744"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2011.51"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/634067.634234"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445428"},{"key":"e_1_2_1_16_1","volume-title":"Disengagement detection in online learning: Validation studies and perspectives","author":"Cocea Mihaela","year":"2010","unstructured":"Mihaela Cocea and Stephan Weibelzahl. 2010. Disengagement detection in online learning: Validation studies and perspectives. IEEE transactions on learning technologies 4, 2 (2010), 114--124."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.intcom.2011.02.008"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1142\/S0129065720500252"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1080\/0144929X.2016.1181209"},{"key":"e_1_2_1_20_1","volume-title":"Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1186\/s40561-018-0080-z"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","unstructured":"Elena Di Lascio Shkurta Gashi Maike E. Debus and Silvia Santini. 2021. Automatic Recognition of Flow During Work Activities Using Context and Physiological Signals. In 2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII). 1--8. https:\/\/doi.org\/10.1109\/ACII52823.2021.9597434","DOI":"10.1109\/ACII52823.2021.9597434"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411821"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3264913"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3123024.3123150"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173856"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1080\/20445911.2022.2080831"},{"key":"e_1_2_1_28_1","volume-title":"The gaze and mouse signal as additional source for user fingerprints in browser applications. arXiv preprint arXiv:2101.03793","author":"Fuhl Wolfgang","year":"2021","unstructured":"Wolfgang Fuhl, Nikolai Sanamrad, and Enkelejda Kasneci. 2021. The gaze and mouse signal as additional source for user fingerprints in browser applications. arXiv preprint arXiv:2101.03793 (2021)."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3580798"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411813"},{"key":"e_1_2_1_31_1","first-page":"613","article-title":"Using Eye Tracking to Trace a Cognitive Process: Gaze Behaviour During Decision Making in a Natural Environment","volume":"6","author":"Gidlf K.","year":"2013","unstructured":"K. Gidlf, A. Wallin, R. Dewhurst, and K. Holmqvist. 2013. Using Eye Tracking to Trace a Cognitive Process: Gaze Behaviour During Decision Making in a Natural Environment. Journal of Eye Movement Research 6, 1 (2013), 613--619.","journal-title":"Journal of Eye Movement Research"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-24600-5_57"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753846.1754025"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025669"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.5555\/3288251.3288290"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2020.2966319"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12528-017-9161-1"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11162-017-9474-y"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208591"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979125"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445269"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3079628.3079669"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478088"},{"key":"e_1_2_1_45_1","volume-title":"Deep learning for time series classification: a review. Data mining and knowledge discovery 33, 4","author":"Fawaz Hassan Ismail","year":"2019","unstructured":"Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. 2019. Deep learning for time series classification: a review. Data mining and knowledge discovery 33, 4 (2019), 917--963."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.21-17-06917.2001"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376817"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3405962.3405982"},{"key":"e_1_2_1_49_1","volume-title":"HCI International 2020 - Late Breaking Papers: Multimodality and Intelligence, Constantine Stephanidis, Masaaki Kurosu, Helmut Degen, and Lauren Reinerman-Jones (Eds.)","author":"Kirsh Ilan","unstructured":"Ilan Kirsh and Mike Joy. 2020. Exploring Pointer Assisted Reading (PAR): Using Mouse Movements to Analyze Web Users' Reading Behaviors and Patterns. In HCI International 2020 - Late Breaking Papers: Multimodality and Intelligence, Constantine Stephanidis, Masaaki Kurosu, Helmut Degen, and Lauren Reinerman-Jones (Eds.). Springer International Publishing, Cham, 156--173."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2448136.2448170"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174010"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/1344471.1344488"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2493988.2494354"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3172944.3173002"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3384419.3430774"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3015297.3015301"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3489849.3489880"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/2638728.2641692"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3380999"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/2807442.2807461"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3530886"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517482"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12528-021-09274-2"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557204"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2656886"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.2984086"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3494998"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.5220\/0006618800860093"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2016.2515084"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445235"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488471"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1152\/jn.2000.83.2.639"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377882"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2018.06.002"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1111\/exsy.12839"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445563"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00700"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW56347.2022.00278"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3161174"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN48605.2020.9207189"},{"key":"e_1_2_1_81_1","volume-title":"Parents' New Role and Needs During the COVID-19 Educational Emergency. Interchange","author":"Rousoulioti Thomais","year":"2022","unstructured":"Thomais Rousoulioti, Dina Tsagari, and Christina Nicole Giannikas. 2022. Parents' New Role and Needs During the COVID-19 Educational Emergency. Interchange (2022), 1--27."},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1186\/2193-1801-2-455"},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.1987.03390130104038"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2012.2223677"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1109\/THMS.2016.2558623"},{"key":"e_1_2_1_86_1","volume-title":"Hidden cognitive states revealed in choice reaching tasks. Trends in cognitive sciences 13, 8","author":"Song Joo-Hyun","year":"2009","unstructured":"Joo-Hyun Song and Ken Nakayama. 2009. Hidden cognitive states revealed in choice reaching tasks. Trends in cognitive sciences 13, 8 (2009), 360--366."},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1145\/3517332"},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287067"},{"key":"e_1_2_1_89_1","doi-asserted-by":"publisher","DOI":"10.1145\/2633043"},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.1145\/3432701"},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-019-00574-9"},{"key":"e_1_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359264"},{"key":"e_1_2_1_93_1","volume-title":"International journal of computer technology and applications 9, 27","author":"Sundar Praveen","year":"2016","unstructured":"Praveen Sundar and S Kumar. 2016. Disengagement detection in online learning using log file analysis. International journal of computer technology and applications 9, 27 (2016), 195--301."},{"key":"e_1_2_1_94_1","doi-asserted-by":"publisher","DOI":"10.1109\/TBME.2020.3026464"},{"key":"e_1_2_1_95_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207679"},{"key":"e_1_2_1_96_1","doi-asserted-by":"publisher","DOI":"10.1145\/3530811"},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.1145\/3551626.3564960"},{"key":"e_1_2_1_98_1","volume-title":"Attention is all you need. Advances in neural information processing systems 30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)."},{"key":"e_1_2_1_99_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACII.2017.8273604"},{"key":"e_1_2_1_100_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411763.3451651"},{"key":"e_1_2_1_101_1","doi-asserted-by":"publisher","DOI":"10.1145\/3375462.3375521"},{"key":"e_1_2_1_102_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2014.2316163"},{"key":"e_1_2_1_103_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2016.2589838"},{"key":"e_1_2_1_104_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858479"},{"key":"e_1_2_1_105_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10489-021-02916-z"},{"key":"e_1_2_1_106_1","doi-asserted-by":"publisher","DOI":"10.5555\/2832747.2832806"},{"key":"e_1_2_1_107_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445240"},{"key":"e_1_2_1_108_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2018.2834350"},{"key":"e_1_2_1_109_1","doi-asserted-by":"publisher","DOI":"10.4108\/icst.mobicase.2014.257786"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3610876","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3610876","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T16:27:57Z","timestamp":1753720077000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3610876"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,27]]},"references-count":109,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,9,27]]}},"alternative-id":["10.1145\/3610876"],"URL":"https:\/\/doi.org\/10.1145\/3610876","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,27]]},"assertion":[{"value":"2023-09-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}