{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T18:18:53Z","timestamp":1776104333779,"version":"3.50.1"},"reference-count":74,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,8,22]],"date-time":"2024-08-22T00:00:00Z","timestamp":1724284800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100030807","name":"Commonwealth Cyber Initiative","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100030807","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2024,8,22]]},"abstract":"<jats:p>Shoulder surfing attacks (SSAs) are a type of observation attack designed to illicitly gather sensitive data from \"over the shoulder\" of victims. This attack can be directed at mobile devices, desktop screens, Personal Identification Number (PIN) pads at an Automated Teller Machine (ATM), or written text. Existing solutions are generally focused on authentication techniques (e.g., logins) and are limited to specific attack scenarios (e.g., mobile devices or PIN Pads). We present ShotjldAR, a mobile and usable system to detect SSAs using multimodal eye gaze information (i.e., from both the potential attacker and victim). ShouldAR uses an augmented reality headset as a platform to incorporate user eye gaze tracking, rear-facing image collection and eye gaze analysis, and user notification of potential attacks. In a 24-participant study, we show that the prototype is capable of detecting 87.28% of SSAs against both physical and digital targets, a two-fold improvement on the baseline solution using a rear-facing mirror, a widely used solution to the SSA problem. The ShouldAR approach provides an AR-based, active SSA defense that applies to both digital and physical information entry in sensitive environments.<\/jats:p>","DOI":"10.1145\/3678573","type":"journal-article","created":{"date-parts":[[2024,9,9]],"date-time":"2024-09-09T14:36:21Z","timestamp":1725892581000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["ShouldAR: Detecting Shoulder Surfing Attacks Using Multimodal Eye Tracking and Augmented Reality"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1315-5066","authenticated-orcid":false,"given":"Matthew","family":"Corbett","sequence":"first","affiliation":[{"name":"Virginia Tech, Blacksburg, Virginia, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3292-1130","authenticated-orcid":false,"given":"Brendan","family":"David-John","sequence":"additional","affiliation":[{"name":"Virginia Tech, Blacksburg, Virginia, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3695-0991","authenticated-orcid":false,"given":"Jiacheng","family":"Shang","sequence":"additional","affiliation":[{"name":"Montclair State University, Montclair, New Jersey, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0149-7509","authenticated-orcid":false,"given":"Bo","family":"Ji","sequence":"additional","affiliation":[{"name":"Virginia Tech, Blacksburg, Virginia, USA"}]}],"member":"320","published-online":{"date-parts":[[2024,9,9]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Retrieved","year":"2024","unstructured":"2024. Artificial Intelligence Act: MEPs adopt landmark law. Retrieved April 15, 2024 from https:\/\/www.europarl.europa.eu\/news\/en\/press-room\/20240308IPR19015\/artificial-intelligence-act-meps-adopt-landmark-law"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3530879"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3314111.3319837"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2638728.2638788"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2638728.2638788"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12652-018-0860-x"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.32604\/csse.2022.018563"},{"key":"e_1_2_1_8_1","volume-title":"Retrieved","year":"2023","unstructured":"Amazon. 2023. Treedix OV2640 Camera Module 140 Degree Wide Angle CMOS 2MP Camera Mini Camera Module. Retrieved Dec 16, 2023 from https:\/\/www.amazon.com\/Treedix-OV2640-Camera-Module-Degree\/dp\/B0894KKXHX\/ref=asc_df_B0894KKXHX\/?tag=hyprod-20&linkCode=df0&hvadid=658532215676&hvpos=&hvnetw=g&hvrand=13097930281404645000&hvpone=&hvptwo=&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9008695&hvtargid=pla-1489482268684&psc=1&mcid=6741e1d20ca03b7ba63f7457dfe91ada"},{"key":"e_1_2_1_9_1","volume-title":"Apple Vision Pro","year":"2023","unstructured":"Apple. 2023. Apple Vision Pro. Apple Corporation. Retrieved Dec 11, 2023 from https:\/\/www.apple.com\/apple-vision-pro\/"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.56553\/popets-2022-0090"},{"key":"e_1_2_1_11_1","volume-title":"India Today. Retrieved","author":"Bhati Divya","year":"2023","unstructured":"Divya Bhati. 2023. ATM card scams: What is Shoulder Surfing and how to protect your ATM PIN from scammers. India Today. Retrieved Dec 11, 2023 from https:\/\/www.indiatoday.in\/technology\/news\/story\/atm-card-scams-what-is-shoulder-surfing-and-how-to-protect-your-atm-pin-from-scammers-2402309-2023-07-05"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935701.1935740"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2333112.2333114"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/IDAP.2018.8620934"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/MySEC.2015.7475211"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3300061.3300119"},{"key":"e_1_2_1_17_1","volume-title":"Cyberspace Safety and Security","author":"Chen Shudi","unstructured":"Shudi Chen and Youwen Zhu. 2019. A Textual Password Entry Method Resistant to Human Shoulder-Surfing Attack. In Cyberspace Safety and Security, Jaideep Vaidya, Xiao Zhang, and Jin Li (Eds.). Springer International Publishing, Cham, 409--420."},{"key":"e_1_2_1_18_1","volume-title":"Facial recognition: for a debate living up to the challenges. Retrieved","author":"CNIL.","year":"2024","unstructured":"CNIL. 2019. Facial recognition: for a debate living up to the challenges. Retrieved April 9, 2024 from https:\/\/www.cnil.fr\/en\/facial-recognition-debate-living-challenges#:~:text=Facial%20recognition%20is%20raising%20new,and%20global%20public%20agendas%20alike."},{"key":"e_1_2_1_19_1","unstructured":"Asadullah Dal. 2023. Eyes-Position-Estimator-Mediapipe. Retrieved Feb 2 2024 from https:\/\/github.com\/Asadullah-Dal17\/Eyes-Position-Estimator-Mediapipe"},{"key":"e_1_2_1_20_1","volume-title":"Facial Detection and Smart Billboards: Analysing the 'Identified' Criterion of Personal Data in the GDPR. European Data Protection Law Review","author":"Earls Davis P. A.","year":"2020","unstructured":"P. A. Earls Davis. 2020. Facial Detection and Smart Billboards: Analysing the 'Identified' Criterion of Personal Data in the GDPR. European Data Protection Law Review (2020). https:\/\/api.semanticscholar.org\/CorpusID:226642930"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753490"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1324892.1324932"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3546155.3546663"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025636"},{"key":"e_1_2_1_25_1","volume-title":"Retrieved","year":"2023","unstructured":"Espressif. 2023. Espressif - ESP32. Retrieved Dec 16, 2023 from https:\/\/www.espressif.com\/en\/products\/socs\/esp32"},{"key":"e_1_2_1_26_1","volume-title":"Retrieved","author":"Gilbert Dylan","year":"2024","unstructured":"Dylan Gilbert and Michael Fagan. 2024. Journey into the Immersive Frontier: Preliminary NIST Research on Cybersecurity and Privacy Standards for Immersive Technologies. Retrieved Feb 13, 2024 from https:\/\/www.nist.gov\/blogs\/cybersecurity-insights\/journey-immersive-frontier-preliminary-nist-research-cybersecurity-and"},{"key":"e_1_2_1_27_1","volume-title":"Retrieved","year":"2023","unstructured":"Google. 2023. MediaPipe. Retrieved Dec 16, 2023 from https:\/\/developers.google.com\/mediapipe"},{"key":"e_1_2_1_28_1","volume-title":"Retrieved","year":"2024","unstructured":"Google. 2024. Google Scholar Search. Retrieved Feb 13, 2024 from https:\/\/scholar.google.com\/scholar?hl=en&as_sdt=0%2C47&as_vis=1&q=%22shoulder+surfing%22&btnG="},{"key":"e_1_2_1_29_1","volume-title":"Retrieved","author":"Gorthaus Michael","year":"2024","unstructured":"Michael Gorthaus. 2024. Vision Pro sales are really tanking, new supply chain data shows. Retrieved Apr 29, 2024 from https:\/\/www.fastcompany.com\/91112727\/vision-pro-sales-tanking-supply-chain-data-meta-quest"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","unstructured":"Ashtha Goyal Priya Matta and Yogesh Lohumi. 2024. Preventing Shoulder Surfing Attacks Matrix Based Graphical Technique. 53--58. https:\/\/doi.org\/10.1109\/ICWITE59797.2024.10502656","DOI":"10.1109\/ICWITE59797.2024.10502656"},{"key":"e_1_2_1_31_1","doi-asserted-by":"crossref","unstructured":"I.P. Howard and B.J. Rogers. 1995. Binocular Vision and Stereopsis. Oxford University Press. https:\/\/books.google.com\/books?id=I8vqITdETe0C","DOI":"10.1093\/acprof:oso\/9780195084764.001.0001"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2851581.2892314"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3136755.3136809"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/IST.2018.8577124"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jisa.2014.09.005"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3365610.3368412"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1280680.1280683"},{"key":"e_1_2_1_38_1","volume-title":"Analysis and improvement of a pin-entry method resilient to shoulder-surfing and recording attacks. Ieee transactions on information forensics and security 10, 2","author":"Kwon Taekyoung","year":"2014","unstructured":"Taekyoung Kwon and Jin Hong. 2014. Analysis and improvement of a pin-entry method resilient to shoulder-surfing and recording attacks. Ieee transactions on information forensics and security 10, 2 (2014), 278--292."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neubiorev.2016.06.006"},{"key":"e_1_2_1_40_1","volume-title":"Magic Leap. Retrieved","author":"Leap Magic","year":"2023","unstructured":"Magic Leap. 2023. magic Leap 2 - Device. Magic Leap. Retrieved Dec 11, 2023 from https:\/\/www.magicleap.com\/magic-leap-2"},{"key":"e_1_2_1_41_1","volume-title":"Retrieved","author":"Legislature The Florida","year":"2023","unstructured":"The Florida Legislature. 2023. The 2023 Florida Statutes (including Special Session C). OJ Title XXXVII (2023). Retrieved Dec 11, 2023 from http:\/\/www.leg.state.fl.us\/Statutes\/index.cfm?App_mode=Display_Statute&Search_String=&URL=0600-0699\/0655\/Sections\/0655.962.html"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","unstructured":"Oliver Lorenz and Ulrike Thomas. 2019. Real Time Eye Gaze Tracking System using CNN-based Facial Features for Human Attention Measurement. 598--606. https:\/\/doi.org\/10.5220\/0007565300002108","DOI":"10.5220\/0007565300002108"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3607822.3614515"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.3389\/fcomp.2021.733531"},{"key":"e_1_2_1_45_1","volume-title":"Likert Scale Definition, Examples and Analysis. Retrieved","author":"McLeod Saul","year":"2024","unstructured":"Saul McLeod. 2019. Likert Scale Definition, Examples and Analysis. Retrieved Feb 2, 2024 from https:\/\/www.simplypsychology.org\/likert-scale.html"},{"key":"e_1_2_1_46_1","volume-title":"Digital Trends. Retrieved","author":"Mehta Tushar","year":"2023","unstructured":"Tushar Mehta. 2023. How AR glasses are going from niche gadget to smartphone replacement. Digital Trends. Retrieved Dec 11, 2023 from https:\/\/www.digitaltrends.com\/mobile\/ar-glasses-replace-smartphones-future-how\/"},{"key":"e_1_2_1_47_1","volume-title":"Retrieved","year":"2024","unstructured":"Meta. 2024. Introducint Project Aria. Retrieved Apr 29, 2024 from https:\/\/www.projectaria.com\/"},{"key":"e_1_2_1_48_1","volume-title":"Retrieved","year":"2024","unstructured":"Meta. 2024. Shop Ray-Ban Meta Smart Glasses. Retrieved Apr 29, 2024 from https:\/\/www.meta.com\/smart-glasses\/shop-all"},{"key":"e_1_2_1_49_1","volume-title":"Retrieved","year":"2024","unstructured":"Microsoft. 2024. Create images from words with AI. Retrieved Feb 13, 2024 from https:\/\/www.bing.com\/images\/create\/a-person-working-at-a-desk-on-a-laptop-with-a-pers\/1-65cb93bd21814fdd927fa22c3e8d0aa7?id=EowoJwMdjsIQj%2fjndAM3uw%3d%3d&view=detailv2&idpp=genimg&idpbck=1&form=BICREC&idpview=singleimage&thid=OIG4.VXWiwzf4YiCerq6DxWjb"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.16910\/jemr.13.6.1"},{"key":"e_1_2_1_51_1","volume-title":"Retrieved","author":"Noor Syeda","year":"2024","unstructured":"Syeda Noor. 2024. Why Do ATMS Have Convex Mirror? Retrieved Feb 13, 2024 from https:\/\/www.boldsky.com\/insync\/pulse\/2016\/why-do-atms-have-a-convex-mirror\/articlecontent-pf137288-107799.html"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACII.2019.8925470"},{"key":"e_1_2_1_53_1","volume-title":"Google Scholar Search. Retrieved","author":"Research Last","year":"2024","unstructured":"Last 365 Days of SSA Research. 2019. Google Scholar Search. Retrieved Feb 2, 2024 from https:\/\/scholar.google.com\/scholar?start=0&q=%22shoulder+surfing%22&hl=en&scisbd=1&as_sdt=0,47&as_vis=1"},{"key":"e_1_2_1_54_1","volume-title":"Retrieved","year":"2023","unstructured":"OpenCV.org. 2023. MediaPipe. Retrieved Dec 16, 2023 from https:\/\/opencv.org\/"},{"key":"e_1_2_1_55_1","volume-title":"gaze-estimation-adas-0002. Retrieved","author":"VINO.","year":"2024","unstructured":"OpenVINO. 2023. gaze-estimation-adas-0002. Retrieved Feb 2, 2024 from https:\/\/docs.openvino.ai\/2022.3\/omz_models_model_gaze_estimation_adas_0002.html"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuropsychologia.2006.04.025"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2017.2725199"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.3390\/iot1020013"},{"key":"e_1_2_1_59_1","volume-title":"Retrieved","author":"PonemonInstitute","year":"2016","unstructured":"PonemonInstitute. 2016. Global Visual Hacking Experimental Study: Analysis. (2016). Retrieved Dec 11, 2023 from multimedia.3m.com\/mws\/media\/1254232O\/global-visual-hacking-experiment-study-summary.pdf"},{"key":"e_1_2_1_60_1","volume-title":"Retrieved","year":"2023","unstructured":"Reydar. 2023. The Future of Augmented Reality. Retrieved Dec 16, 2023 from https:\/\/www.reydar.com\/exploring-the-future-potential-of-augmented-reality-trends-technology-and-impact\/#:~:text=In%20the%20near%20future%2C%20we,and%20technology%20than%20today's%20smartphones"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/1030083.1030116"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3379157.3391422"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.2307\/1422228"},{"key":"e_1_2_1_64_1","volume-title":"Eye Tracking with Mediapipe. Retrieved","author":"Sediqi Monib","year":"2024","unstructured":"Monib Sediqi. 2023. Eye Tracking with Mediapipe. Retrieved Feb 2, 2024 from https:\/\/kh-monib.medium.com\/title-gaze-tracking-with-opencv-and-mediapipe-318ac0c9c2c3"},{"key":"e_1_2_1_65_1","volume-title":"Retrieved","author":"Shrestha Rashik","year":"2023","unstructured":"Rashik Shrestha. 2023. Perspective-n-Point(PnP). Retrieved Dec 16, 2023 from https:\/\/medium.com\/@rashik.shrestha\/perspective-n-point-pnp-f2c7dd4ef1ed"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3361218"},{"key":"e_1_2_1_67_1","volume-title":"Watching Me","author":"Stein Scott","year":"2022","unstructured":"Scott Stein. 2022. Watching Me, Watching You: How Eye Tracking Is Coming to VR and Beyond. Retrieved May 8 2022 from https:\/\/www.cnet.com\/tech\/computing\/watching-me-watching-you-how-eye-tracking-is-coming-to-vr-and-beyond\/"},{"key":"e_1_2_1_68_1","volume-title":"32nd USENIX Security Symposium (USENIX Security 23)","author":"Tang Brian Jay","unstructured":"Brian Jay Tang and Kang G. Shin. 2023. Eye-Shield: Real-Time Protection of Mobile Device Screen Information from Shoulder Surfing. In 32nd USENIX Security Symposium (USENIX Security 23). USENIX Association, Anaheim, CA, 5449--5466. https:\/\/www.usenix.org\/conference\/usenixsecurity23\/presentation\/tang"},{"key":"e_1_2_1_69_1","volume-title":"AC1900 Wireless MU-MIMO Wi-Fi 5 Router. Retrieved","year":"2023","unstructured":"TP-Link. 2023. AC1900 Wireless MU-MIMO Wi-Fi 5 Router. Retrieved Dec 16, 2023 from https:\/\/www.tp-link.com\/us\/home-networking\/wifi-router\/archer-c80\/"},{"key":"e_1_2_1_70_1","volume-title":"Retrieved","author":"Boston University","year":"2024","unstructured":"Boston University. 2024. Wilcoxon Signed Rank Test. Retrieved Apr 29, 2024 from https:\/\/sphweb.bumc.bu.edu\/otlt\/mph-modules\/bs\/bs704_nonparametric\/BS704_Nonparametric6.html"},{"key":"e_1_2_1_71_1","volume-title":"Retrieved","year":"2023","unstructured":"Varjo. 2023. Varjo XR-4 Series. Retrieved Dec 16, 2023 from https:\/\/varjo.com\/products\/xr-4\/"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03198755"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2003.1238328"},{"key":"e_1_2_1_74_1","volume-title":"Eye movements and Vision","author":"Yarbus A. L.","unstructured":"A. L. Yarbus. 1967. Eye movements and Vision. Springer."}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678573","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3678573","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T14:43:14Z","timestamp":1755787394000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678573"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,22]]},"references-count":74,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,8,22]]}},"alternative-id":["10.1145\/3678573"],"URL":"https:\/\/doi.org\/10.1145\/3678573","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,22]]},"assertion":[{"value":"2024-09-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}