{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T16:08:05Z","timestamp":1787760485102,"version":"build-2784847793"},"reference-count":28,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,2,5]],"date-time":"2026-02-05T00:00:00Z","timestamp":1770249600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>The paper presents a visio-verbal teleimpedance interface for commanding 3D stiffness ellipsoids to the remote robot with a combination of the operator\u2019s gaze and verbal interaction. The gaze is detected by an eye-tracker, allowing the system to understand the context in terms of what the operator is currently looking at in the scene. Along with verbal interaction, a Vision-Language Model (VLM) processes this information, enabling the operator to communicate their intended action or provide corrections. Based on these inputs, the interface can then generate appropriate stiffness matrices for different physical interaction actions. To validate the proposed visio-verbal teleimpedance interface, we conducted a series of experiments on a setup including a Force Dimension Sigma.7 haptic device to control the motion of the remote Kuka LBR iiwa robotic arm. The human operator\u2019s gaze is tracked by Tobii Pro Glasses 2, while human verbal commands are processed by a VLM using GPT-4o. The first experiment explored the optimal prompt configuration for the interface. The second and third experiments demonstrated different functionalities of the interface on a slide-in-the-groove task.<\/jats:p>","DOI":"10.3389\/frobt.2026.1749105","type":"journal-article","created":{"date-parts":[[2026,2,5]],"date-time":"2026-02-05T05:22:55Z","timestamp":1770268975000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Visio-verbal teleimpedance interface: enabling semi-autonomous control of physical interaction via eye tracking and speech"],"prefix":"10.3389","volume":"13","author":[{"given":"Henk H. A.","family":"Jekel","sequence":"first","affiliation":[{"name":"Department of Cognitive Robotics, Delft University of Technology","place":["Delft, Netherlands"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alejandro","family":"D\u00edaz Rosales","sequence":"additional","affiliation":[{"name":"Department of Cognitive Robotics, Delft University of Technology","place":["Delft, Netherlands"]},{"name":"European Organization for Nuclear Research (CERN)","place":["Meyrin, Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luka","family":"Peternel","sequence":"additional","affiliation":[{"name":"Department of Cognitive Robotics, Delft University of Technology","place":["Delft, Netherlands"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,2,5]]},"reference":[{"key":"B1","first-page":"298","article-title":"Predicting user intent through eye gaze for shared autonomy","volume-title":"AAAI fall symposia","author":"Admoni","year":"2016"},{"key":"B2","doi-asserted-by":"publisher","first-page":"1642","DOI":"10.1177\/0278364912464668","article-title":"Tele-impedance: teleoperation with impedance regulation using a body\u2013machine interface","volume":"31","author":"Ajoudani","year":"2012","journal-title":"Int. J. Robotics Res."},{"key":"B3","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1177\/0278364917744035","article-title":"Reduced-complexity representation of the human arm active endpoint stiffness for supervisory control of remote manipulation","volume":"37","author":"Ajoudani","year":"2018","journal-title":"Int. J. Robotics Res."},{"key":"B4","first-page":"23716","article-title":"Flamingo: a visual language model for few-shot learning","volume":"35","author":"Alayrac","year":"2022","journal-title":"Adv. Neural Information Processing Systems"},{"key":"B5","doi-asserted-by":"crossref","first-page":"7307","DOI":"10.1109\/IROS51168.2021.9636551","article-title":"Inferring goals with gaze during teleoperated manipulation","volume-title":"2021 IEEE\/RSJ international conference on intelligent robots and systems (IROS)","author":"Aronson","year":"2021"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2005.14165","article-title":"Language models are few-shot learners","author":"Brown","year":"2020","journal-title":"arXiv Preprint arXiv:2005.14165"},{"key":"B7","doi-asserted-by":"crossref","first-page":"2870","DOI":"10.1109\/EMBC48229.2022.9871736","article-title":"Tele-impedance control approach using wearable sensors","volume-title":"2022 44th annual international conference of the IEEE engineering in medicine & biology Society (EMBC)","author":"Buscaglione","year":"2022"},{"key":"B8","article-title":"End-to-end dexterous arm-hand vla policies via shared autonomy: vr teleoperation augmented by autonomous hand vla policy for efficient data collection","author":"Cui","year":"2025"},{"key":"B9","first-page":"13479","volume-title":"Interactive multi-stiffness mixed reality interface: controlling and visualizing robot and environment stiffness. In 2024","author":"D\u00edaz Rosales","year":"2024"},{"key":"B10","doi-asserted-by":"publisher","first-page":"1282","DOI":"10.1109\/TRO.2020.3047064","article-title":"Analysis of coupling effect in human-commanded stiffness during bilateral tele-impedance","volume":"37","author":"Doornebosch","year":"2021","journal-title":"IEEE Trans. Robotics"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2509.10416","article-title":"Tasc: task-aware shared control for teleoperated manipulation","author":"Fu","year":"2025"},{"key":"B12","doi-asserted-by":"publisher","first-page":"2052","DOI":"10.1109\/tro.2021.3071530","article-title":"A scalable framework for multi-robot tele-impedance control","volume":"37","author":"Garate","year":"2021","journal-title":"IEEE Trans. Robotics"},{"key":"B13","first-page":"9276","volume-title":"Human guided trajectory and impedance adaptation for tele-operated physical assistance. In 2021","author":"Gourmelen","year":"2021"},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2001.08361","article-title":"Scaling laws for neural language models","author":"Kaplan","year":"2020","journal-title":"arXiv Preprint arXiv:2001.08361"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2401.17217","article-title":"GazeGPT: augmenting human capabilities using gaze-contingent contextual AI for smart eyewear","author":"Konrad","year":"2024","journal-title":"arXiv Preprint arXiv:2401.17217"},{"key":"B16","doi-asserted-by":"crossref","first-page":"690","DOI":"10.1109\/Humanoids58906.2024.10769954","article-title":"Design and evaluation of finger-operated teleimpedance interface enabling simultaneous control of 3d aspects of stiffness ellipsoid","volume-title":"2024 IEEE-RAS 23rd international conference on humanoid robots (Humanoids)","author":"Kraakman","year":"2024"},{"key":"B17","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2506.14727","article-title":"Casper: inferring diverse intents for assistive teleoperation with vision language models","author":"Liu","year":"2025","journal-title":"arXiv Preprint arXiv:2506"},{"key":"B18","doi-asserted-by":"publisher","first-page":"5429","DOI":"10.1109\/lra.2021.3066974","article-title":"Bilateral teleoperation with adaptive impedance control for contact tasks","volume":"6","author":"Michel","year":"2021","journal-title":"IEEE Robotics Automation Lett."},{"key":"B19","doi-asserted-by":"publisher","first-page":"1248","DOI":"10.1109\/lra.2021.3056371","article-title":"Learning optimal impedance control during complex 3d arm movements","volume":"6","author":"Naceri","year":"2021","journal-title":"IEEE Robotics Automation Lett."},{"key":"B20","doi-asserted-by":"publisher","first-page":"998","DOI":"10.1109\/tii.2018.2876676","article-title":"Programming by demonstration using the teleimpedance control scheme: verification by an semg-controlled ball-trapping robot","volume":"15","author":"Park","year":"2018","journal-title":"IEEE Trans. Industrial Inf."},{"key":"B21","doi-asserted-by":"publisher","first-page":"401","DOI":"10.1109\/thms.2022.3231703","article-title":"After a decade of teleimpedance: a survey","volume":"53","author":"Peternel","year":"2022","journal-title":"IEEE Trans. Human-Machine Syst."},{"key":"B22","first-page":"99","article-title":"Independently commanding size, shape and orientation of robot endpoint stiffness in tele-impedance by virtual ellipsoid interface","volume-title":"2021 20th international conference on advanced robotics","author":"Peternel","year":"2021"},{"key":"B23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1049\/ccs2.12005","article-title":"A review on manipulation skill acquisition through teleoperation-based learning from demonstration","volume":"3","author":"Si","year":"2021","journal-title":"Cognitive Comput. Syst."},{"key":"B24","doi-asserted-by":"crossref","first-page":"779","DOI":"10.1109\/Humanoids58906.2024.10769858","article-title":"Semi-autonomous teleimpedance based on visual detection of object geometry and material and its relation to environment","volume-title":"2024 IEEE-RAS 23rd international conference on humanoid robots (Humanoids)","author":"Siegemund","year":"2024"},{"key":"B25","doi-asserted-by":"publisher","first-page":"104224","DOI":"10.1016\/j.robot.2022.104224","article-title":"A survey of robot manipulation in contact","volume":"156","author":"Suomalainen","year":"2022","journal-title":"Robotics Aut. Syst."},{"key":"B26","first-page":"5998","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"B27","first-page":"1348","article-title":"Demonstrating the benefits of variable impedance to telerobotic task execution","volume-title":"2011 IEEE international conference on robotics and automation (ICRA)","author":"Walker","year":"2011"},{"key":"B28","doi-asserted-by":"publisher","first-page":"1193","DOI":"10.1109\/tmech.2018.2817589","article-title":"A dmps-based framework for robot learning and generalization of humanlike variable impedance skills","volume":"23","author":"Yang","year":"2018","journal-title":"IEEE\/ASME Trans. Mechatronics"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2026.1749105\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,9]],"date-time":"2026-02-09T07:49:55Z","timestamp":1770623395000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2026.1749105\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,5]]},"references-count":28,"alternative-id":["10.3389\/frobt.2026.1749105"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2026.1749105","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,5]]},"article-number":"1749105"}}