{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T23:17:13Z","timestamp":1776122233490,"version":"3.50.1"},"reference-count":39,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2025,5,22]],"date-time":"2025-05-22T00:00:00Z","timestamp":1747872000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"crossref","award":["Project-ID 251654672 -- TRR 161"],"award-info":[{"award-number":["Project-ID 251654672 -- TRR 161"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"crossref","award":["project number 389792660 -- TRR 248"],"award-info":[{"award-number":["project number 389792660 -- TRR 248"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Daimler and Benz Foundation (project TITAN - Technologische Intelligenz zur Transformation, Automatisierung und Nutzerorientierung des Justizsystems","award":["grant no. 45-06\/24"],"award-info":[{"award-number":["grant no. 45-06\/24"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2025,5,22]]},"abstract":"<jats:p>\n            In time-critical detection tasks, such as drone monitoring, a key condition for users to effectively leverage AI assistance is to find an appropriate trade-off between making fast decisions and verifying AI suggestions, which we refer to as appropriate user reliance. However, assessing such reliance is often oversimplified by focusing solely on task outcomes, potentially overlooking whether users properly verify AI messages. We collected eye-tracking data from an AI-assisted monitoring task and developed a gaze-based reliance model:\n            <jats:italic toggle=\"yes\">RelEYEance,<\/jats:italic>\n            to assess the extent of user reliance on AI-suggested alarms. We found that gaze patterns related to verification behaviors distinguish between appropriate reliance, over-reliance, and under-reliance, influencing task performance. We validated our model in a second user study, showing it can reliably detect users' over- and under-reliance at run-time, which could be used e.g. for issuing intervention messages. The results demonstrate the potential for real-time human-AI reliance assessment, facilitating adaptive reliance calibration.\n          <\/jats:p>","DOI":"10.1145\/3725841","type":"journal-article","created":{"date-parts":[[2025,5,22]],"date-time":"2025-05-22T18:18:53Z","timestamp":1747937933000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["RelEYEance: Gaze-based Assessment of Users' AI-reliance at Run-time"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5233-2352","authenticated-orcid":false,"given":"Zekun","family":"Wu","sequence":"first","affiliation":[{"name":"Saarland University, Saarland Informatics Campus, Saarbr\u00fccken, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3633-8623","authenticated-orcid":false,"given":"Yao","family":"Wang","sequence":"additional","affiliation":[{"name":"University of Stuttgart, Stuttgart, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8165-1803","authenticated-orcid":false,"given":"Markus","family":"Langer","sequence":"additional","affiliation":[{"name":"Department of Psychology, University of Freiburg, Freiburg im Breisgau, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4168-6099","authenticated-orcid":false,"given":"Anna Maria","family":"Feit","sequence":"additional","affiliation":[{"name":"Saarland University, Saarland Informatics Campus, Saarbr\u00fccken, Germany"}]}],"member":"320","published-online":{"date-parts":[[2025,5,22]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449163"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445717"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449287"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3555572"},{"key":"e_1_2_1_5_1","unstructured":"SHIYE CAO SHICHANG KE YANYAN MO LIU ANQI and CHIEN-MING HUANG. 2023. Eyes are the Windows to AI Reliance: Toward Real-Time Human-AI Reliance Assessment. (2023)."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3447535.3462487"},{"key":"e_1_2_1_7_1","volume-title":"Operator scheduling strategies in supervisory control of multiple UAVs. Aerospace science and technology 11, 4","author":"Cummings ML","year":"2007","unstructured":"ML Cummings and PJ Mitchell. 2007. Operator scheduling strategies in supervisory control of multiple UAVs. Aerospace science and technology 11, 4 (2007), 339--348."},{"key":"e_1_2_1_8_1","volume-title":"PyGaze: An open-source, cross-platform toolbox for minimal-effort programming of eyetracking experiments. Behavior research methods 46","author":"Dalmaijer Edwin S","year":"2014","unstructured":"Edwin S Dalmaijer, Sebastiaan Math\u00f4t, and Stefan Van der Stigchel. 2014. PyGaze: An open-source, cross-platform toolbox for minimal-effort programming of eyetracking experiments. Behavior research methods 46 (2014), 913--921."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174220"},{"key":"e_1_2_1_10_1","volume-title":"https:\/\/www.dji.com\/de\/mavic-3\/downloads Accessed","author":"Downloads DJI.","year":"2023","unstructured":"DJI. 2023. Mavic 3 - Downloads. https:\/\/www.dji.com\/de\/mavic-3\/downloads Accessed: August 26, 2023."},{"key":"e_1_2_1_11_1","volume-title":"Drone Delivery System: Everything You Need to Know. https:\/\/www.flytbase.com\/blog\/drone-deliverysystem Accessed","year":"2023","unstructured":"FlytBase. 2023. Drone Delivery System: Everything You Need to Know. https:\/\/www.flytbase.com\/blog\/drone-deliverysystem Accessed: August 26, 2023."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1260\/1756-8293.6.4.211"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.3390\/fi13080188"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISMAR50242.2020.00082"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445923"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3557889"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1080\/19346182.2009.9648498"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2021.604977"},{"key":"e_1_2_1_19_1","first-page":"9","article-title":"Human control over automation: EU policy and AI ethics","volume":"12","author":"Koulu Riikka","year":"2020","unstructured":"Riikka Koulu. 2020. Human control over automation: EU policy and AI ethics. Eur. J. Legal Stud. 12 (2020), 9.","journal-title":"Eur. J. Legal Stud."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287590"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445570"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3332165.3347945"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.obhdp.2018.12.005"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/THMS.2019.2930980"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445562"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581058"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1080\/14639220500337708"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v8i1.7469"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397481.3450639"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1177\/0018720810376055"},{"key":"e_1_2_1_31_1","volume-title":"Overreliance on AI literature review. Microsoft Research","author":"Passi Samir","year":"2022","unstructured":"Samir Passi and Mihaela Vorvoreanu. 2022. Overreliance on AI literature review. Microsoft Research (2022)."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3641905"},{"key":"e_1_2_1_33_1","volume-title":"Should I follow AI-based advice? Measuring appropriate reliance in human-AI decision-making. arXiv preprint arXiv:2204.06916","author":"Schemmer Max","year":"2022","unstructured":"Max Schemmer, Patrick Hemmer, Niklas K\u00fchl, Carina Benz, and Gerhard Satzger. 2022. Should I follow AI-based advice? Measuring appropriate reliance in human-AI decision-making. arXiv preprint arXiv:2204.06916 (2022)."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","unstructured":"Sebastian Schirmer Christoph Torens Johann C. Dauer Jan Baumeister Bernd Finkbeiner and Kristin Y. Rozier. [n. d.]. A Hierarchy of Monitoring Properties for Autonomous Systems. doi:10.2514\/6.2023--2588 arXiv:https:\/\/arc.aiaa.org\/doi\/pdf\/10.2514\/6.2023--2588","DOI":"10.2514\/6.2023--2588"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642621"},{"key":"e_1_2_1_36_1","volume-title":"HHAI 2023: Augmenting Human Intellect","author":"Schoeffer Jakob","unstructured":"Jakob Schoeffer, Johannes Jakubik, Michael Voessing, Niklas Kuehl, and Gerhard Satzger. 2023. On the interdependence of reliance behavior and accuracy in AI-assisted decision-making. In HHAI 2023: Augmenting Human Intellect. IOS Press, 46--59."},{"key":"e_1_2_1_37_1","unstructured":"Sihan Sun. 2022. Exploring the Interface to Aid the Operator's Situation Awareness in Supervisory Control of Multiple Drones. Master's thesis. https:\/\/urn.kb.se\/resolve?urn=urn:nbn:se:kth:diva-323073"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.3389\/fdgth.2020.567502"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3610978.3640649"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3725841","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3725841","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,23]],"date-time":"2025-08-23T01:51:01Z","timestamp":1755913861000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3725841"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,22]]},"references-count":39,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,5,22]]}},"alternative-id":["10.1145\/3725841"],"URL":"https:\/\/doi.org\/10.1145\/3725841","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,22]]},"assertion":[{"value":"2025-05-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}