{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,20]],"date-time":"2026-05-20T16:05:10Z","timestamp":1779293110325,"version":"3.51.4"},"reference-count":53,"publisher":"MDPI AG","issue":"15","license":[{"start":{"date-parts":[[2023,7,25]],"date-time":"2023-07-25T00:00:00Z","timestamp":1690243200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Experiences of virtual reality (VR) can easily break if the method of evaluating subjective user states is intrusive. Behavioral measures are increasingly used to avoid this problem. One such measure is eye tracking, which recently became more standard in VR and is often used for content-dependent analyses. This research is an endeavor to utilize content-independent eye metrics, such as pupil size and blinks, for identifying mental load in VR users. We generated mental load independently from visuals through auditory stimuli. We also defined and measured a new eye metric, focus offset, which seeks to measure the phenomenon of \u201cstaring into the distance\u201d without focusing on a specific surface. In the experiment, VR-experienced participants listened to two native and two foreign language stimuli inside a virtual phone booth. The results show that with increasing mental load, relative pupil size on average increased 0.512 SDs (0.118 mm), with 57% reduced variance. To a lesser extent, mental load led to fewer fixations, less voluntary gazing at distracting content, and a larger focus offset as if looking through surfaces (about 0.343 SDs, 5.10 cm). These results are in agreement with previous studies. Overall, we encourage further research on content-independent eye metrics, and we hope that hardware and algorithms will be developed in the future to further increase tracking stability.<\/jats:p>","DOI":"10.3390\/s23156667","type":"journal-article","created":{"date-parts":[[2023,7,26]],"date-time":"2023-07-26T01:09:01Z","timestamp":1690333741000},"page":"6667","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Identification of Language-Induced Mental Load from Eye Behaviors in Virtual Reality"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2133-067X","authenticated-orcid":false,"given":"Johannes","family":"Schirm","sequence":"first","affiliation":[{"name":"Graduate School of Science and Technology, Nara Institute of Science and Technology, Ikoma 630-0192, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-9901-7089","authenticated-orcid":false,"given":"Andr\u00e9s Roberto","family":"G\u00f3mez-Vargas","sequence":"additional","affiliation":[{"name":"Graduate School of Science and Technology, Nara Institute of Science and Technology, Ikoma 630-0192, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0486-1743","authenticated-orcid":false,"given":"Monica","family":"Perusqu\u00eda-Hern\u00e1ndez","sequence":"additional","affiliation":[{"name":"Graduate School of Science and Technology, Nara Institute of Science and Technology, Ikoma 630-0192, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2783-5257","authenticated-orcid":false,"given":"Richard T.","family":"Skarbez","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Technology, School of Computing, Engineering and Mathematical Sciences, La Trobe University, Melbourne Campus, Melbourne, VIC 3086, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6535-8439","authenticated-orcid":false,"given":"Naoya","family":"Isoyama","sequence":"additional","affiliation":[{"name":"Faculty of Social Information Studies, Otsuma Women\u2019s University, Tokyo 102-8357, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6119-1184","authenticated-orcid":false,"given":"Hideaki","family":"Uchiyama","sequence":"additional","affiliation":[{"name":"Graduate School of Science and Technology, Nara Institute of Science and Technology, Ikoma 630-0192, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2260-1707","authenticated-orcid":false,"given":"Kiyoshi","family":"Kiyokawa","sequence":"additional","affiliation":[{"name":"Graduate School of Science and Technology, Nara Institute of Science and Technology, Ikoma 630-0192, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,7,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"557","DOI":"10.1007\/s10055-019-00422-9","article-title":"Development and validation of a simulation workload measure: The simulation task load index (SIM-TLX)","volume":"24","author":"Harris","year":"2020","journal-title":"Virtual Real."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Augereau, O., Brocheton, G., and Do Prado Neto, P.P. (2022, January 12\u201316). An Open Platform for Research about Cognitive Load in Virtual Reality. Proceedings of the 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Christchurch, New Zealand.","DOI":"10.1109\/VRW55335.2022.00020"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/S0166-4115(08)62386-9","article-title":"Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research","volume":"Volume 52","author":"Hancock","year":"1988","journal-title":"Human Mental Workload"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1007\/s10648-017-9404-8","article-title":"Differentiating Different Types of Cognitive Load: A Comparison of Different Measures","volume":"30","author":"Korbach","year":"2018","journal-title":"Educ. Psychol. Rev."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2378","DOI":"10.1109\/TVCG.2017.2735098","article-title":"Cognitive Cost of Using Augmented Reality Displays","volume":"23","author":"Baumeister","year":"2017","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1016\/j.jesp.2014.01.009","article-title":"The relation of flow-experience and physiological arousal under stress\u2014Can u shape it?","volume":"53","author":"Peifer","year":"2014","journal-title":"J. Exp. Soc. Psychol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"864653","DOI":"10.3389\/frvir.2022.864653","article-title":"Eye-Tracking Beyond Peripersonal Space in Virtual Reality: Validation and Best Practices","volume":"3","author":"Lamb","year":"2022","journal-title":"Front. Virtual Real."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"650693","DOI":"10.3389\/fpsyg.2021.650693","article-title":"A Case for Studying Naturalistic Eye and Head Movements in Virtual Environments","volume":"12","author":"Barentine","year":"2021","journal-title":"Front. Psychol."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"3","DOI":"10.16910\/jemr.12.1.3","article-title":"Eye tracking in virtual reality","volume":"12","author":"Clay","year":"2019","journal-title":"J. Eye Mov. Res."},{"key":"ref_10","unstructured":"Orlosky, J., Reiners, D., and Weyers, B. (2021, January 8\u201310). Deep Learning on Eye Gaze Data to Classify Student Distraction Level in an Educational VR Environment. Proceedings of the ICAT-EGVE 2021\u2014International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments, Sankt Augustin, Germany."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Duchowski, A.T., Medlin, E., Cournia, N., Gramopadhye, A., Melloy, B., and Nair, S. (2002, January 25\u201327). 3D Eye Movement Analysis for VR Visual Inspection Training. Proceedings of the 2002 Symposium on Eye Tracking Research & Applications, ETRA\u201902, New Orleans, LA, USA.","DOI":"10.1145\/507072.507094"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1633","DOI":"10.1109\/TVCG.2018.2793599","article-title":"Saliency in VR: How Do People Explore Virtual Environments?","volume":"24","author":"Sitzmann","year":"2018","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"276","DOI":"10.1037\/0033-2909.91.2.276","article-title":"Task-evoked pupillary responses, processing load, and the structure of processing resources","volume":"91","author":"Beatty","year":"1982","journal-title":"Psychol. Bull."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"e12977","DOI":"10.1111\/cogs.12977","article-title":"How Reliably Do Eye Parameters Indicate Internal Versus External Attentional Focus?","volume":"45","author":"Walcher","year":"2021","journal-title":"Cogn. Sci."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1167\/8.3.33","article-title":"Vergence\u2013accommodation conflicts hinder visual performance and cause visual fatigue","volume":"8","author":"Hoffman","year":"2008","journal-title":"J. Vis."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"601","DOI":"10.3758\/s13415-016-0417-4","article-title":"Pupillary correlates of lapses of sustained attention","volume":"16","author":"Unsworth","year":"2016","journal-title":"Cogn. Affect. Behav. Neurosci."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1814","DOI":"10.3758\/s13423-015-0845-0","article-title":"Sudden insight is associated with shutting out visual inputs","volume":"22","author":"Salvi","year":"2015","journal-title":"Psychon. Bull. Rev."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"102883","DOI":"10.1016\/j.apergo.2019.102883","article-title":"Using biomechanics to investigate the effect of VR on eye vergence system","volume":"81","author":"Iskander","year":"2019","journal-title":"Appl. Ergon."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Weier, M., Roth, T., Hinkenjann, A., and Slusallek, P. (June, January 30). Predicting the Gaze Depth in Head-Mounted Displays Using Multiple Feature Regression. Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, ETRA\u201918, Tubingen, Germany.","DOI":"10.1145\/3204493.3204547"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Huang, M.X., Li, J., Ngai, G., Leong, H.V., and Bulling, A. (2019, January 21\u201325). Moment-to-Moment Detection of Internal Thought during Video Viewing from Eye Vergence Behavior. Proceedings of the 27th ACM International Conference on Multimedia, MM\u201919, Nice, France.","DOI":"10.1145\/3343031.3350573"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1097\/00006324-200306000-00014","article-title":"Accommodation and Vergence Require Effort-to-See","volume":"80","author":"Francis","year":"2003","journal-title":"Optom. Vis. Sci."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1016\/j.concog.2017.06.009","article-title":"Looking for ideas: Eye behavior during goal-directed internally focused cognition","volume":"53","author":"Walcher","year":"2017","journal-title":"Conscious. Cogn."},{"key":"ref_23","first-page":"265","article-title":"Analysis and measurement of mental load","volume":"Volume 2","author":"Bertelson","year":"1994","journal-title":"International Perspectives on Psychological Science, II: The State of the Art"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1075\/target.25.1.03see","article-title":"Cognitive load in simultaneous interpreting: Measures and methods","volume":"25","author":"Seeber","year":"2013","journal-title":"Target. Int. J. Transl. Stud."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1092","DOI":"10.3389\/fpsyg.2017.01092","article-title":"Eye Behavior Associated with Internally versus Externally Directed Cognition","volume":"8","author":"Benedek","year":"2017","journal-title":"Front. Psychol."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3432","DOI":"10.3758\/s13414-020-02068-1","article-title":"Eye behavior predicts susceptibility to visual distraction during internally directed cognition","volume":"82","author":"Walcher","year":"2020","journal-title":"Atten. Percept. Psychophys."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"4","DOI":"10.3991\/ijoe.v17i13.20167","article-title":"Data Visualization to Explore the Countries Dataset for Pattern Creation","volume":"17","author":"Khan","year":"2021","journal-title":"Int. J. Online Biomed. Eng. iJOE"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"104","DOI":"10.21833\/ijaas.2021.01.013","article-title":"Industrial internet of things: Investigation of the applications, issues, and challenges","volume":"8","author":"Khan","year":"2021","journal-title":"Int. J. Adv. Appl. Sci."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"256","DOI":"10.3389\/fpsyg.2022.815665","article-title":"A Scoping Review of Flow Research","volume":"13","author":"Peifer","year":"2022","journal-title":"Front. Psychol."},{"key":"ref_30","unstructured":"Csikszentmihalyi, M. (1975). Beyond Boredom and Anxiety: Experiencing Flow in Work and Play, Jossey-Bass."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"170","DOI":"10.1016\/j.chb.2015.07.045","article-title":"Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning","volume":"54","author":"Hamari","year":"2016","journal-title":"Comput. Hum. Behav."},{"key":"ref_32","unstructured":"Krashen, S.D. (1981). Second Language Acquisition and Second Language Learning, Pergamon Press Inc."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/14790710508668395","article-title":"Linguistic Distance: A Quantitative Measure of the Distance Between English and Other Languages","volume":"26","author":"Chiswick","year":"2005","journal-title":"J. Multiling. Multicult. Dev."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1016\/j.neuroimage.2015.11.068","article-title":"How does language distance between L1 and L2 affect the L2 brain network? An fMRI study of Korean\u2013Chinese\u2013English trilinguals","volume":"129","author":"Kim","year":"2016","journal-title":"NeuroImage"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"593108","DOI":"10.3389\/fnhum.2021.593108","article-title":"Language Familiarity and Proficiency Leads to Differential Cortical Processing during Translation between Distantly Related Languages","volume":"15","author":"Shinozuka","year":"2021","journal-title":"Front. Hum. Neurosci."},{"key":"ref_36","unstructured":"Audacity (2023, May 08). Audacity \u00ae|Free, Open Source, Cross-Platform Audio Software for Multi-Track Recording and Editing. Available online: https:\/\/www.audacityteam.org\/."},{"key":"ref_37","first-page":"71","article-title":"Japanese version of NASA Task Load Index: Sensitivity of its workload score to difficulty of three different laboratory tasks","volume":"32","author":"Haga","year":"1996","journal-title":"Jpn. J. Ergon."},{"key":"ref_38","unstructured":"Unity Technologies (2023, May 08). Unity Real-Time Development Platform|3D, 2D, VR & AR Engine. Available online: https:\/\/unity.com\/."},{"key":"ref_39","unstructured":"Saral, O. (2023, May 08). Phone Booth: 3D Exterior. Available online: https:\/\/assetstore.unity.com\/packages\/3d\/props\/exterior\/phone-booth-23178."},{"key":"ref_40","unstructured":"KrazyFX (2023, May 08). Phone Booth: 3D Environments. Available online: https:\/\/assetstore.unity.com\/packages\/3d\/environments\/phone-booth-21150."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"20","DOI":"10.3389\/frobt.2020.00020","article-title":"Audio in VR: Effects of a Soundscape and Movement-Triggered Step Sounds on Presence","volume":"7","author":"Kern","year":"2020","journal-title":"Front. Robot. AI"},{"key":"ref_42","unstructured":"Valve Corporation (2023, May 08). Steam Audio. Available online: https:\/\/valvesoftware.github.io\/steam-audio\/."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"4","DOI":"10.16910\/jemr.9.4.2.","article-title":"Pupil size affects measures of eye position in video eye tracking: Implications for recording vergence accuracy","volume":"9","author":"Jaschinski","year":"2016","journal-title":"J. Eye Mov. Res."},{"key":"ref_44","first-page":"5071","article-title":"Accuracy and precision of the HTC VIVE PRO eye tracking in head-restrained and head-free conditions","volume":"61","author":"Sipatchin","year":"2020","journal-title":"Investig. Ophthalmol. Vis. Sci."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"2232","DOI":"10.3758\/s13428-020-01374-8","article-title":"GazeR: A Package for Processing Gaze Position and Pupil Size Data","volume":"52","author":"Geller","year":"2020","journal-title":"Behav. Res. Methods"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"834","DOI":"10.3758\/s13428-017-0909-3","article-title":"Gazepath: An eye-tracking analysis tool that accounts for individual differences and data quality","volume":"50","author":"Raijmakers","year":"2018","journal-title":"Behav. Res. Methods"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Smallwood, J., Brown, K.S., Tipper, C., Giesbrecht, B., Franklin, M.S., Mrazek, M.D., Carlson, J.M., and Schooler, J.W. (2011). Pupillometric Evidence for the Decoupling of Attention from Perceptual Input during Offline Thought. PLoS ONE, 6.","DOI":"10.1371\/journal.pone.0018298"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Toyama, T., Orlosky, J., Sonntag, D., and Kiyokawa, K. (2014, January 27\u201329). A Natural Interface for Multi-Focal Plane Head Mounted Displays Using 3D Gaze. Proceedings of the 2014 International Working Conference on Advanced Visual Interfaces, AVI\u201914, Como, Italy.","DOI":"10.1145\/2598153.2598154"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Vortmann, L.M., Schult, M., Benedek, M., Walcher, S., and Putze, F. (2019, January 14\u201318). Real-Time Multimodal Classification of Internal and External Attention. Proceedings of the Adjunct of the 2019 International Conference on Multimodal Interaction, ICMI\u201919, Jiangsu, China.","DOI":"10.1145\/3351529.3360658"},{"key":"ref_50","unstructured":"R Foundation (2023, May 08). R: The R Project for Statistical Computing. Available online: https:\/\/www.r-project.org\/."},{"key":"ref_51","unstructured":"Wickham, H., Fran\u00e7ois, R., Henry, L., M\u00fcller, K., and Vaughan, D. (2023, May 08). A Grammar of Data Manipulation \u2022 Dplyr. Available online: https:\/\/dplyr.tidyverse.org\/."},{"key":"ref_52","unstructured":"Wickham, H., Chang, W., Henry, L., Pedersen, T.L., Takahashi, K., Wilke, C., Woo, K., Yutani, H., and Dunnington, D. (2023, May 08). Create Elegant Data Visualisations Using the Grammar of Graphics \u2022 Ggplot2. Available online: https:\/\/ggplot2.tidyverse.org\/."},{"key":"ref_53","unstructured":"Pedersen, T.L. (2023, May 08). The Composer of Plots \u2022 Patchwork. Available online: https:\/\/patchwork.data-imaginist.com\/."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/15\/6667\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:18:43Z","timestamp":1760127523000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/15\/6667"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,25]]},"references-count":53,"journal-issue":{"issue":"15","published-online":{"date-parts":[[2023,8]]}},"alternative-id":["s23156667"],"URL":"https:\/\/doi.org\/10.3390\/s23156667","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,25]]}}}