{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T03:25:47Z","timestamp":1781148347846,"version":"3.54.1"},"reference-count":76,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,4,3]],"date-time":"2024-04-03T00:00:00Z","timestamp":1712102400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>In a rapidly evolving digital landscape autonomous tools and robots are becoming commonplace. Recognizing the significance of this development, this paper explores the integration of Large Language Models (LLMs) like <jats:italic>Generative pre-trained transformer (GPT)<\/jats:italic> into human-robot teaming environments to facilitate variable autonomy through the means of verbal human-robot communication. In this paper, we introduce a novel simulation framework for such a GPT-powered multi-robot testbed environment, based on a Unity Virtual Reality (VR) setting. This system allows users to interact with simulated robot agents through natural language, each powered by individual GPT cores. By means of OpenAI\u2019s function calling, we bridge the gap between unstructured natural language input and structured robot actions. A user study with 12 participants explores the effectiveness of GPT-4 and, more importantly, user strategies when being given the opportunity to converse in natural language within a simulated multi-robot environment. Our findings suggest that users may have preconceived expectations on how to converse with robots and seldom try to explore the actual language and cognitive capabilities of their simulated robot collaborators. Still, those users who did explore were able to benefit from a much more natural flow of communication and human-like back-and-forth. We provide a set of lessons learned for future research and technical implementations of similar systems.<\/jats:p>","DOI":"10.3389\/frobt.2024.1347538","type":"journal-article","created":{"date-parts":[[2024,4,3]],"date-time":"2024-04-03T05:10:43Z","timestamp":1712121043000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":17,"title":["Exploring a GPT-based large language model for variable autonomy in a VR-based human-robot teaming simulation"],"prefix":"10.3389","volume":"11","author":[{"given":"Younes","family":"Lakhnati","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Max","family":"Pascher","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jens","family":"Gerken","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2024,4,3]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"509","DOI":"10.1109\/thms.2018.2791570","article-title":"A topology of shared control systems\u2014finding common ground in diversity","volume":"48","author":"Abbink","year":"2018","journal-title":"IEEE Trans. Human-Machine Syst."},{"key":"B3","doi-asserted-by":"publisher","first-page":"667","DOI":"10.1145\/3594806.3596572","article-title":"Towards designing a chatGPT conversational companion for elderly people","author":"Alessa","year":"2023","journal-title":"Proceedings of the 16th international conference on Pervasive technologies related to assistive environments"},{"key":"B4","doi-asserted-by":"publisher","first-page":"449","DOI":"10.1007\/s10514-018-9792-8","article-title":"Grounding natural language instructions to semantic goal representations for abstraction and generalization","volume":"43","author":"Arumugam","year":"2019","journal-title":"Aut. Robots"},{"key":"B5","doi-asserted-by":"publisher","first-page":"16100","DOI":"10.31004\/joe.v5i4.2745","article-title":"Can chat gpt replace the role of the teacher in the classroom: a fundamental analysis","volume":"5","author":"Ausat","year":"2023","journal-title":"J. Educ."},{"key":"B6","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1191\/1478088706qp063oa","article-title":"Using thematic analysis in psychology","volume":"3","author":"Braun","year":"2006","journal-title":"Qual. Res. Psychol."},{"key":"B7","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1016\/s1071-5819(03)00018-1","article-title":"Emotion and sociable humanoid robots","volume":"59","author":"Breazeal","year":"2003","journal-title":"Int. J. human-computer Stud."},{"key":"B8","doi-asserted-by":"publisher","first-page":"1877","DOI":"10.48550\/arXiv.2005.14165","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv. neural Inf. Process. Syst."},{"key":"B9","doi-asserted-by":"publisher","first-page":"3833","DOI":"10.1109\/LRA.2021.3064449","article-title":"Toward seamless transitions between shared control and supervised autonomy in robotic assistance","volume":"6","author":"Bustamante","year":"2021","journal-title":"IEEE Robotics Automation Lett."},{"key":"B10","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-47437-3_3","article-title":"Personalization framework for adaptive robotic feeding assistance","volume-title":"Social robotics","author":"Canal","year":"2016"},{"key":"B11","volume-title":"Evaluating large language models trained on code","author":"Chen","year":"2021"},{"key":"B12","doi-asserted-by":"crossref","first-page":"932","DOI":"10.1145\/3568294.3579957","article-title":"Variable autonomy for human-robot teaming (vat)","volume-title":"Companion of the 2023 ACM\/IEEE international conference on human-robot interaction","author":"Chiou","year":"2023"},{"key":"B13","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511620539","volume-title":"Using language","author":"Clark","year":"1996"},{"key":"B14","first-page":"401","article-title":"A human being wrote this law review article: gpt-3 and the practice of law","volume":"55","author":"Cyphert","year":"2021","journal-title":"UC Davis L. Rev."},{"key":"B15","doi-asserted-by":"publisher","first-page":"416","DOI":"10.4135\/9781446249215.n21","article-title":"Self-determination theory","volume":"1","author":"Deci","year":"2012","journal-title":"Handb. Theor. Soc. Psychol."},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1423","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume":"1","author":"Devlin","year":"2019","journal-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human language technologies, volume 1 (long and short papers)"},{"key":"B17","doi-asserted-by":"publisher","first-page":"282","DOI":"10.1016\/j.robot.2017.04.013","article-title":"The effect of robotic wheelchair control paradigm and interface on user performance, effort and preference: an experimental assessment","volume":"94","author":"Erdogan","year":"2017","journal-title":"Robotics Aut. Syst."},{"key":"B18","doi-asserted-by":"crossref","DOI":"10.31219\/osf.io\/9ge8m","volume-title":"How chat gpt can transform autodidactic experiences and open education","author":"Firat","year":"2023"},{"key":"B19","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1007\/s10111-019-00576-1","article-title":"Joining the blunt and the pointy end of the spear: towards a common framework of joint action, human\u2013machine cooperation, cooperative guidance and control, shared, traded and supervisory control","volume":"21","author":"Flemisch","year":"2019","journal-title":"Cognition, Technol. Work"},{"key":"B20","doi-asserted-by":"crossref","DOI":"10.1145\/3571884.3597137","article-title":"Roboclean: contextual language grounding for human-robot interactions in specialised low-resource environments","author":"Fuentes","year":"2023"},{"key":"B21","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1145\/1349822.1349842","article-title":"How people anthropomorphize robots","author":"Fussell","year":"2008","journal-title":"Proc. 3rd ACM\/IEEE Int. Conf. Hum. robot Interact."},{"key":"B22","doi-asserted-by":"crossref","DOI":"10.1109\/HRI.2019.8673309","article-title":"Transfer depends on acquisition: analyzing manipulation strategies for robotic feeding","author":"Gallenberger","year":"2019"},{"key":"B23","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1016\/j.tics.2003.10.016","article-title":"Why is conversation so easy?","volume":"8","author":"Garrod","year":"2004","journal-title":"Trends cognitive Sci."},{"key":"B24","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-031-02218-0","volume-title":"From tool to partner: the evolution of human-computer interaction","author":"Grudin","year":"2017"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2310.03659","article-title":"Balancing autonomy and alignment: a multi-dimensional taxonomy for autonomous LLM-powered multi-agent architectures","author":"H\u00e4ndler","year":"2023"},{"key":"B26","doi-asserted-by":"publisher","first-page":"3378","DOI":"10.3390\/ijerph20043378","article-title":"Diagnostic accuracy of differential-diagnosis lists generated by generative pretrained transformer 3 chatbot for clinical vignettes with common chief complaints: a pilot study","volume":"20","author":"Hirosawa","year":"2023","journal-title":"Int. J. Environ. Res. public health"},{"key":"B27","doi-asserted-by":"publisher","first-page":"287","DOI":"10.1017\/s1351324900002497","article-title":"Towards a tool for the subjective assessment of speech system interfaces (sassi)","volume":"6","author":"Hone","year":"2000","journal-title":"Nat. Lang. Eng."},{"key":"B28","volume-title":"Between reality and delusion: challenges of applying large language models to companion robots for open-domain dialogues with older adults","author":"Irfan","year":"2023"},{"key":"B77","first-page":"287","article-title":"Do as i can, not as i say: grounding language in robotic affordances","volume":"205","author":"Ichter","year":"2022","journal-title":"Proc of the 6th Con. on robot learning. Proc of Mach learning research"},{"key":"B29","doi-asserted-by":"publisher","first-page":"e590","DOI":"10.1093\/pubmed\/fdad028","article-title":"Chatgpt, public health communication and \u2018intelligent patient companionship","volume":"45","author":"Kahambing","year":"2023","journal-title":"J. public health"},{"key":"B30","doi-asserted-by":"publisher","first-page":"1007","DOI":"10.1002\/pra2.927","article-title":"Bing chat: the future of search engines?","volume":"60","author":"Kelly","year":"2023","journal-title":"Proc. Assoc. Inf. Sci. Technol."},{"key":"B31","first-page":"99","article-title":"Next-generation of virtual personal assistants (microsoft cortana, apple siri, amazon alexa and google home)","author":"Kepuska","year":"2018"},{"key":"B32","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1109\/TSMCA.2011.2159589","article-title":"How autonomy impacts performance and satisfaction: results from a study with spinal cord injured subjects using an assistive robot","volume":"42","author":"Kim","year":"2012","journal-title":"IEEE Trans. Syst. Man, Cybern. - Part A Syst. Humans"},{"key":"B33","first-page":"1","article-title":"Measuring user experience in conversational interfaces: a comparison of six questionnaires","author":"Kocabalil","year":"2018"},{"key":"B34","volume-title":"Structured and unstructured speech2action frameworks for human-robot collaboration: a user study","author":"Kodur","year":"2023"},{"key":"B35","volume-title":"Gpt-4 vs. gpt-3 5: A concise showdown","author":"Koubaa","year":""},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.20944\/preprints202304.0827.v2","article-title":"Rosgpt: next-generation human-robot interaction with chatgpt and ros","author":"Koubaa","year":"","journal-title":"Preprints"},{"key":"B37","doi-asserted-by":"publisher","first-page":"1747","DOI":"10.1007\/s12369-020-00743-9","article-title":"Attitudes toward robots as equipment and coworkers and the impact of robot autonomy level","volume":"13","author":"Latikka","year":"2021","journal-title":"Int. J. Soc. Robotics"},{"key":"B38","doi-asserted-by":"publisher","first-page":"1375","DOI":"10.1109\/lra.2017.2669369","article-title":"Learning by demonstration for planning activities of daily living in rehabilitation and assistive robotics","volume":"2","author":"Lauretti","year":"2017","journal-title":"IEEE Robotics Automation Lett."},{"key":"B39","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2308.16529","article-title":"Developing social robots with empathetic non-verbal cues using large language models","author":"Lee","year":"2023"},{"key":"B40","doi-asserted-by":"crossref","DOI":"10.1109\/InfoTech58664.2023.10266870","article-title":"System software architecture for enhancing human-robot interaction by conversational ai","author":"Lekova","year":"2023"},{"key":"B41","volume-title":"Robot ethics: the ethical and social implications of robotics","author":"Lin","year":"2014"},{"key":"B42","doi-asserted-by":"publisher","first-page":"172988141985140","DOI":"10.1177\/1729881419851402","article-title":"A review of methodologies for natural-language-facilitated human\u2013robot cooperation","volume":"16","author":"Liu","year":"2019","journal-title":"Int. J. Adv. Robotic Syst."},{"key":"B43","doi-asserted-by":"publisher","first-page":"281","DOI":"10.1177\/0278364915602060","article-title":"Tell me dave: context-sensitive grounding of natural language to manipulation instructions","volume":"35","author":"Misra","year":"2016","journal-title":"Int. J. Robotics Res."},{"key":"B44","article-title":"Webgpt: browser-assisted question-answering with human feedback","author":"Nakano","year":"2021"},{"key":"B45","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2201.10005","article-title":"Text and code embeddings by contrastive pre-training","author":"Neelakantan","year":"2022"},{"key":"B46","volume-title":"Shakey the robot","author":"Nilsson","year":"1984"},{"key":"B47","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.13375","article-title":"Capabilities of GPT-4 on medical challenge problems","author":"Nori","year":"2023"},{"key":"B48","doi-asserted-by":"publisher","first-page":"27730","DOI":"10.48550\/arXiv.2203.02155","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"B49","doi-asserted-by":"publisher","first-page":"103344","DOI":"10.1016\/j.robot.2019.103344","article-title":"Active robot-assisted feeding with a general-purpose mobile manipulator: design, evaluation, and lessons learned","volume":"124","author":"Park","year":"2020","journal-title":"Robotics Aut. Syst."},{"key":"B50","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2310.15887","article-title":"AdaptiX \u2013 a transitional xr framework for development and evaluation of shared control applications in assistive robotics","volume":"8","author":"Pascher","year":"2024","journal-title":"Proc. ACM Hum.-Comput. Interact."},{"key":"B51","doi-asserted-by":"crossref","DOI":"10.1145\/3544548.3580857","article-title":"How to communicate robot motion intent: a scoping review","author":"Pascher","year":""},{"key":"B52","first-page":"2300","article-title":"Time and space: towards usable adaptive control for assistive robotic arms","author":"Pascher","year":""},{"key":"B53","first-page":"4921","article-title":"Why that nao? how humans adapt to a conventional humanoid robot in taking turns-at-talk","author":"Pelikan","year":"2016"},{"key":"B54","doi-asserted-by":"crossref","DOI":"10.2139\/ssrn.4294197","article-title":"The implications of chatgpt for legal services and society","author":"Perlman","year":"2022"},{"key":"B55","unstructured":"Language models as knowledge bases?\n            PetroniF.\n            Rockt\u00e4schelT.\n            LewisP.\n            BakhtinA.\n            WuY.\n            MillerA. H.\n          2019"},{"key":"B56","doi-asserted-by":"publisher","first-page":"106469","DOI":"10.1016\/j.chb.2020.106469","article-title":"Stress in manual and autonomous modes of collaboration with a cobot","volume":"112","author":"Pollak","year":"2020","journal-title":"Comput. Hum. Behav."},{"key":"B57","first-page":"207","article-title":"Do animals have accents? talking with agents in multi-party conversation","author":"Porcheron","year":"2017"},{"key":"B58","article-title":"Improving language understanding by generative pre-training","author":"Radford","year":"2018"},{"key":"B59","first-page":"3962","article-title":"Autonomous object detection and grasping using deep learning for design of an intelligent assistive robot manipulation system","author":"Rakhimkul","year":"2019"},{"key":"B60","first-page":"2023","volume-title":"Evaluating chatgpt as an adjunct for radiologic decision-making. medRxiv","author":"Rao","year":"2023"},{"key":"B61","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1023\/a:1013298507114","article-title":"Theory of mind for a humanoid robot","volume":"12","author":"Scassellati","year":"2002","journal-title":"Aut. Robots"},{"key":"B62","doi-asserted-by":"publisher","first-page":"70","DOI":"10.1016\/j.tics.2005.12.009","article-title":"Joint action: bodies and minds moving together","volume":"10","author":"Sebanz","year":"2006","journal-title":"Trends cognitive Sci."},{"key":"B63","doi-asserted-by":"publisher","first-page":"e2325000","DOI":"10.1001\/jamanetworkopen.2023.25000","article-title":"Use of gpt-4 to analyze medical records of patients with extensive investigations and delayed diagnosis","volume":"6","author":"Shea","year":"2023","journal-title":"JAMA Netw. Open"},{"key":"B64","doi-asserted-by":"publisher","first-page":"3008","DOI":"10.48550\/arXiv.2009.01325","article-title":"Learning to summarize with human feedback","volume":"33","author":"Stiennon","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"B65","first-page":"244","article-title":"Story centaur: large language model few shot learning as a creative writing tool","volume-title":"Proceedings of the 16th conference of the European chapter of the association for computational linguistics: system demonstrations","author":"Swanson","year":"2021"},{"key":"B66","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.6853187","article-title":"The AI teacher test: measuring the pedagogical ability of blender and GPT-3 in educational dialogues","author":"Tack","year":"2022","journal-title":"Proceedings of the 15th international conference on educational data mining"},{"key":"B67","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1706.03762","article-title":"Natural language understanding and communication for multi-agent systems","volume":"2015","author":"Trott","year":"2015","journal-title":"AAAI Fall Symp. Ser."},{"key":"B68","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2308.06032","article-title":"Large language models in cryptocurrency securities cases: can a GPT model meaningfully assist lawyers?","author":"Trozze","year":"2023"},{"key":"B69","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1706.03762","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv. neural Inf. Process. Syst."},{"key":"B70","doi-asserted-by":"publisher","first-page":"1689","DOI":"10.1007\/s12369-020-00723-z","article-title":"Qualitative research in hri: a review and taxonomy","volume":"13","author":"Veling","year":"2021","journal-title":"Int. J. Soc. Robotics"},{"key":"B71","doi-asserted-by":"publisher","first-page":"998","DOI":"10.1016\/j.neunet.2010.06.002","article-title":"A minimal architecture for joint action","volume":"23","author":"Vesper","year":"2010","journal-title":"Neural Netw."},{"key":"B72","doi-asserted-by":"publisher","first-page":"3197","DOI":"10.1007\/s11845-023-03377-8","article-title":"Gpt-4: a new era of artificial intelligence in medicine","volume":"1971","author":"Waisberg","year":"2023","journal-title":"Ir. J. Med. Sci."},{"key":"B73","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/0010-0285(72)90002-3","article-title":"Understanding natural language","volume":"3","author":"Winograd","year":"1972","journal-title":"Cogn. Psychol."},{"key":"B74","doi-asserted-by":"crossref","DOI":"10.1145\/1499586.1499695","article-title":"Progress in natural language understanding","volume-title":"Proceedings of the June 4-8, 1973, national computer conference and exposition on - afips \u201973","author":"Woods","year":"1973"},{"key":"B75","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1906.08237","article-title":"Xlnet: generalized autoregressive pretraining for language understanding","volume":"32","author":"Yang","year":"2019","journal-title":"Adv. neural Inf. Process. Syst."},{"key":"B76","doi-asserted-by":"publisher","first-page":"48","DOI":"10.1016\/j.ijhcs.2016.12.008","article-title":"Can we control it? Autonomous robots threaten human identity, uniqueness, safety, and resources","volume":"100","author":"Z\u0142otowski","year":"2017","journal-title":"Int. J. Human-Computer Stud."}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2024.1347538\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,3]],"date-time":"2024-04-03T05:11:02Z","timestamp":1712121062000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2024.1347538\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,3]]},"references-count":76,"alternative-id":["10.3389\/frobt.2024.1347538"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2024.1347538","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,3]]},"article-number":"1347538"}}