{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,13]],"date-time":"2026-10-13T17:01:15Z","timestamp":1791910875208,"version":"4.3.4"},"reference-count":47,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,5,25]],"date-time":"2023-05-25T00:00:00Z","timestamp":1684972800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>Collaborative robots have gained popularity in industries, providing flexibility and increased productivity for complex tasks. However, their ability to interact with humans and adapt to their behavior is still limited. Prediction of human movement intentions is one way to improve the robots adaptation. This paper investigates the performance of using Transformers and MLP-Mixer based neural networks to predict the intended human arm movement direction, based on gaze data obtained in a virtual reality environment, and compares the results to using an LSTM network. The comparison will evaluate the networks based on accuracy on several metrics, time ahead of movement completion, and execution time. It is shown in the paper that there exists several network configurations and architectures that achieve comparable accuracy scores. The best performing Transformers encoder presented in this paper achieved an accuracy of 82.74%, for predictions with high certainty, on continuous data and correctly classifies 80.06% of the movements at least once. The movements are, in 99% of the cases, correctly predicted the first time, before the hand reaches the target and more than 19% ahead of movement completion in 75% of the cases. The results shows that there are multiple ways to utilize neural networks to perform gaze based arm movement intention prediction and it is a promising step toward enabling efficient human-robot collaboration.<\/jats:p>","DOI":"10.3389\/fnbot.2023.1157957","type":"journal-article","created":{"date-parts":[[2023,5,25]],"date-time":"2023-05-25T04:30:38Z","timestamp":1684989038000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["Comparison of LSTM, Transformers, and MLP-mixer neural networks for gaze based human intention prediction"],"prefix":"10.3389","volume":"17","author":[{"given":"Julius","family":"Pettersson","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Petter","family":"Falkman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,5,25]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"26","DOI":"10.3390\/computers5040026","article-title":"Development and evaluation of the virtual prototype of the first saudi Arabian-designed car","volume":"5","author":"Abidi","year":"2016","journal-title":"Computers"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.1177\/1687814016639824","article-title":"Development of a virtual manufacturing assembly simulation system","author":"Al-Ahmari","year":"2016","journal-title":"Adv. Mech. Eng."},{"key":"B3","doi-asserted-by":"publisher","first-page":"204","DOI":"10.1016\/j.ifacol.2016.11.168","article-title":"Artab-using virtual and augmented reality methods for an improved situation awareness for telemaintenance","volume":"49","author":"Aschenbrenner","year":"2016","journal-title":"IFAC Pap. Online"},{"key":"B4","first-page":"75","article-title":"\u201cHuman-robot collaboration by intention recognition using probabilistic state machines,\u201d","author":"Awais","year":"2010","journal-title":"19th International Workshop on Robotics in Alpe-Adria-Danube Region (RAAD 2010)"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1607.06450","article-title":"Layer normalization","author":"Ba","year":"2016","journal-title":"arXiv preprint arXiv:1607.06450"},{"key":"B6","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1016\/j.bspc.2019.02.011","article-title":"A review on EMG-based motor intention prediction of continuous human upper limb motion for human-robot collaboration","volume":"51","author":"Bi","year":"2019","journal-title":"Biomed. Signal Process. Control"},{"key":"B7","unstructured":"Blender. Blender Foundation2023"},{"key":"B8","first-page":"2296","article-title":"\u201cBayesian human intention inference through multiple model filtering with gaze-based priors,\u201d","author":"Chaandar Ravichandar","year":"2016","journal-title":"2016 19th International Conference on Information Fusion (FUSION)"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1016\/j.rcim.2019.101830","article-title":"Bi-directional navigation intent communication using spatial augmented reality and eye-tracking glasses for improved safety in human-robot interaction","author":"Chadalavada","year":"2020","journal-title":"Robot. Comput. Integr. Manufact."},{"key":"B10","first-page":"1","article-title":"\u201cVirtual reality commissioning in production systems preparation,\u201d","author":"Dahl","year":"2017","journal-title":"22nd IEEE International Conference on Emerging Technologies And Factory Automation (ETFA)"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2010.11929","article-title":"An image is worth 16x16 words: transformers for image recognition at scale","author":"Dosovitskiy","year":"2020","journal-title":"arXiv preprint arXiv:2010.11929"},{"key":"B12","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-57883-5","author":"Duchowski","year":"2017","journal-title":"Eye Tracking Methodology: Theory and Practice"},{"key":"B13","first-page":"1624","article-title":"\u201cDesign of a collaborative architecture for human-robot assembly tasks,\u201d","author":"El Makrini","year":"2017","journal-title":"2017 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)"},{"key":"B14","first-page":"1050","article-title":"\u201cDropout as a Bayesian approximation: representing model uncertainty in deep learning,\u201d","author":"Gal","year":"2016","journal-title":"International Conference on Machine Learning"},{"key":"B15","first-page":"428","article-title":"\u201cIntention recognition in human robot interaction based on eye tracking,\u201d","author":"Gomez Cubero","year":"2021","journal-title":"IFIP Conference on Human-Computer Interaction"},{"key":"B16","author":"Goodfellow","year":"2016","journal-title":"Deep Learning"},{"key":"B17","doi-asserted-by":"publisher","first-page":"633","DOI":"10.1016\/S0006-3223(01)01095-2","article-title":"Altered performance on an ocular fixation task in attention-deficit\/hyperactivity disorder","volume":"50","author":"Gould","year":"2001","journal-title":"Biol. Psychiatry"},{"key":"B18","doi-asserted-by":"publisher","first-page":"25","DOI":"10.3389\/frobt.2018.00025","article-title":"Exploiting three-dimensional gaze tracking for action recognition during bimanual manipulation to enhance human\u2013robot collaboration","volume":"5","author":"Haji Fathaliyan","year":"2018","journal-title":"Front. Robot. AI"},{"key":"B19","first-page":"770","article-title":"\u201cDeep residual learning for image recognition,\u201d","author":"He","year":"2016","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1606.08415","article-title":"Gaussian error linear units (GELUs)","author":"Hendrycks","year":"2016","journal-title":"arXiv preprint arXiv:1606.08415"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1207.0580","article-title":"Improving neural networks by preventing co-adaptation of feature detectors","author":"Hinton","year":"2012","journal-title":"arXiv preprint arXiv:1207.0580"},{"key":"B22","doi-asserted-by":"publisher","first-page":"585280","DOI":"10.3389\/fpsyg.2020.585280","article-title":"Pedestrians' understanding of a fully autonomous vehicle's intent to stop: a learning effect over time","volume":"11","author":"Hochman","year":"2020","journal-title":"Front. Psychol."},{"key":"B23","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"B24","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1109\/HRI.2016.7451737","article-title":"\u201cAnticipatory robot control for efficient human-robot collaboration,\u201d","author":"Huang","year":"2016","journal-title":"2016 11th ACM\/IEEE International Conference on Human-Robot Interaction (HRI)"},{"key":"B25","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1145\/3197768.3201565","article-title":"\u201cEyes are different than hands: an analysis of gaze as input modality for industrial man-machine interactions,\u201d","author":"Jungwirth","year":"2018","journal-title":"Proceedings of the 11th Pervasive Technologies Related to Assistive Environments Conference"},{"key":"B26","doi-asserted-by":"publisher","first-page":"283","DOI":"10.1016\/j.dr.2007.06.006","article-title":"Eye tracking studies of normative and atypical development","volume":"27","author":"Karatekin","year":"2007","journal-title":"Dev. Rev."},{"key":"B27","doi-asserted-by":"crossref","first-page":"1310","DOI":"10.1109\/EMBC.2019.8857054","article-title":"\u201cEye gaze-based early intent prediction utilizing CNN-LSTM,\u201d","author":"Koochaki","year":"2019","journal-title":"2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)"},{"key":"B28","doi-asserted-by":"publisher","first-page":"628","DOI":"10.1016\/j.cirp.2009.09.009","article-title":"Cooperation of human and machines in assembly lines","volume":"58","author":"Kr\u00fcger","year":"2009","journal-title":"CIRP Ann."},{"key":"B29","doi-asserted-by":"publisher","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc. IEEE"},{"key":"B30","first-page":"299","article-title":"\u201cHuman-robot collaborative manipulation planning using early prediction of human motion,\u201d","author":"Mainprice","year":"2013","journal-title":"2013 IEEE\/RSJ International Conference on Intelligent Robots and Systems"},{"key":"B31","first-page":"1","article-title":"\u201cDeep learning based eye gaze estimation and prediction,\u201d","author":"Mazzeo","year":"2021","journal-title":"2021 6th International Conference on Smart and Sustainable Technologies (SpliTech)"},{"key":"B32","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1007\/978-3-319-73751-5_19","article-title":"\u201cManufacturing systems at scale with big data streaming and online machine learning,\u201d","volume-title":"Service Orientation in Holonic and Multi-Agent Manufacturing","author":"Morariu","year":"2018"},{"key":"B33","doi-asserted-by":"publisher","first-page":"31","DOI":"10.4236\/ijcns.2017.103003","article-title":"Big data analysis in smart manufacturing: a review","volume":"10","author":"Nagorny","year":"2017","journal-title":"Int. J. Commun. Netw. Syst. Sci."},{"key":"B34","first-page":"1","article-title":"\u201cCognitive ability evaluation using virtual reality and eye tracking,\u201d","author":"Pettersson","year":"2018","journal-title":"2018 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA)"},{"key":"B35","article-title":"\u201cHuman movement direction classification using virtual reality and eye tracking,\u201d","author":"Pettersson","year":"2020","journal-title":"30th International Conference on Flexible Automation and Intelligent Manufacturing (FAIM)"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.1109\/ICIT46573.2021.9453581","article-title":"\u201cHuman movement direction prediction using virtual reality and eye tracking,\u201d","author":"Pettersson","year":"2021","journal-title":"2021 22nd IEEE International Conference on Industrial Technology (ICIT)"},{"key":"B37","article-title":"Intended human arm movement direction prediction using eye tracking","author":"Pettersson","year":"2022","journal-title":"Int. J. Comput. Integr. Manufact"},{"key":"B38","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1080\/09602010343000183","article-title":"Analysis of assets for virtual reality applications in neuropsychology","volume":"14","author":"Rizzo","year":"2004","journal-title":"Neuropsychol. Rehabil."},{"key":"B39","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1708.08296","article-title":"Explainable artificial intelligence: understanding, visualizing and interpreting deep learning models","author":"Samek","year":"2017","journal-title":"arXiv preprint arXiv:1708.08296"},{"key":"B40","doi-asserted-by":"publisher","first-page":"68","DOI":"10.3390\/robotics10020068","article-title":"Gazeemd: detecting visual intention in gaze-based human-robot interaction","volume":"10","author":"Shi","year":"2021","journal-title":"Robotics"},{"key":"B41","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1016\/j.rcim.2018.08.005","article-title":"The development and evaluation of robot light skin: a novel robot signalling system to improve communication in industrial human-robot collaboration","volume":"56","author":"Tang","year":"2019","journal-title":"Robot. Comput. Integr. Manufact."},{"key":"B42","unstructured":"Tobii Pro VR Integration-Based on HTC Vive Development Kit Description. Tobii AB2020"},{"key":"B43","first-page":"24261","article-title":"\u201cMLP-mixer: an all-MLP architecture for vision,\u201d","author":"Tolstikhin","year":"2021","journal-title":"Advances in Neural Information Processing Systems 34"},{"key":"B44","doi-asserted-by":"crossref","first-page":"2192","DOI":"10.1109\/CVPR52688.2022.00224","article-title":"\u201cEnd-to-end human-gaze-target detection with transformers,\u201d","author":"Tu","year":"2022","journal-title":"2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"B45","first-page":"5998","article-title":"\u201cAttention is all you need,\u201d","author":"Vaswani","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"key":"B46","doi-asserted-by":"publisher","first-page":"144","DOI":"10.1016\/j.jmsy.2018.01.003","article-title":"Deep learning for smart manufacturing: methods and applications","volume":"48","author":"Wang","year":"2018","journal-title":"J. Manufact. Syst."},{"key":"B47","doi-asserted-by":"publisher","first-page":"2339","DOI":"10.1109\/TASE.2021.3074873","article-title":"Predicting human intentions in human-robot hand-over tasks through multimodal learning","volume":"19","author":"Wang","year":"2021","journal-title":"IEEE Trans. Automat. Sci. Eng"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1157957\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,25]],"date-time":"2023-05-25T04:30:59Z","timestamp":1684989059000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1157957\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,25]]},"references-count":47,"alternative-id":["10.3389\/fnbot.2023.1157957"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2023.1157957","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,25]]},"article-number":"1157957"}}