{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,12]],"date-time":"2025-12-12T12:35:27Z","timestamp":1765542927639,"version":"3.48.0"},"reference-count":130,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,12,12]],"date-time":"2025-12-12T00:00:00Z","timestamp":1765497600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004063","name":"Knut och Alice Wallenbergs Stiftelse","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004063","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Comput. Sci."],"abstract":"<jats:p>As human speakers naturally adapt their linguistic styles to one another, voice user interfaces that prompt similar linguistic adaptations can augment human-like interaction. In this study, we leverage a corpus of human instructions to model the effectiveness of incremental instruction generation in artificial agents. Participants interacted with agents that guided them in selecting virtual puzzle pieces, varying the amount of information provided in each instruction. Through an empirical examination of the Gricean maxims in utterance construction, our initial perception study highlighted the significance of adaptive instruction generation. By employing mouse movements as a proxy for user understanding, we developed computational models that enabled agents to detect user uncertainty and refine instructions incrementally. Comparing speaker-based and listener-based models, we found that agents encouraging linguistic adaptations were preferred by users. Our findings offer new insights into the value of mouse movements as indicators of user comprehension and introduce a methodological framework for developing adaptive interactive systems that generate instructions dynamically.<\/jats:p>","DOI":"10.3389\/fcomp.2025.1634228","type":"journal-article","created":{"date-parts":[[2025,12,12]],"date-time":"2025-12-12T12:30:41Z","timestamp":1765542641000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Beyond speech: leveraging mouse movements for information adaptation in voice interfaces"],"prefix":"10.3389","volume":"7","author":[{"given":"Dimosthenis","family":"Kontogiorgos","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David","family":"Schlangen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2025,12,12]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"599","DOI":"10.1145\/2911451.2911505","article-title":"\u201cPredicting user engagement with direct displays using mouse cursor information,\u201d","author":"Arapakis","year":"2016","journal-title":"Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval"},{"key":"B2","doi-asserted-by":"publisher","first-page":"1309","DOI":"10.1145\/3397271.3401031","article-title":"\u201cLearning efficient representations of mouse movements to predict user attention,\u201d","author":"Arapakis","year":"2020","journal-title":"Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval"},{"key":"B3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3374210","article-title":"A price-per-attention auction scheme using mouse cursor information","volume":"38","author":"Arapakis","year":"2020","journal-title":"ACM Trans. Inf. Syst"},{"key":"B4","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1145\/1125451.1125557","article-title":"\u201cUsability tool for analysis of web designs using mouse tracks,\u201d","author":"Arroyo","year":"2006","journal-title":"CHI'06 Extended Abstracts on Human Factors in Computing Systems"},{"key":"B5","doi-asserted-by":"publisher","first-page":"1188","DOI":"10.1145\/1056808.1056873","article-title":"\u201cCombining head tracking and mouse input for a gui on multiple monitors,\u201d","author":"Ashdown","year":"2005","journal-title":"CHI'05 Extended Abstracts on Human Factors in Computing Systems"},{"key":"B6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3290605.3300484","article-title":"\u201cResilient chatbots: repair strategy preferences for conversational breakdowns,\u201d","author":"Ashktorab","year":"2019","journal-title":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems"},{"key":"B7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3383652.3423884","article-title":"\u201cUsing knowledge graphs and behaviour trees for feedback-aware presentation agents,\u201d","author":"Axelsson","year":"2020","journal-title":"Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents"},{"key":"B8","first-page":"37","article-title":"\u201cMeasuring the anthropomorphism, animacy, likeability, perceived intelligence and perceived safety of robots,\u201d","author":"Bartneck","year":"2008","journal-title":"Metrics for HRI Workshop, Technical Report"},{"key":"B9","article-title":"Fitting linear mixed-effects models using lme4","author":"Bates","year":"2014","journal-title":"arXiv preprint arXiv:1406.5823"},{"key":"B10","unstructured":"\u201cUsing Affordances to shape the interaction in a hybrid spoken dialogue system,\u201d\n          \n          1219\n          \n            \n              Baumann\n              T.\n            \n            \n              Paetzel\n              M.\n            \n            \n              Schlesinger\n              P.\n            \n            \n              Menzel\n              W.\n            \n          \n          Dresden\n          TUD Press\n          Studientexte zur Sprachkommunikation: Elektronische Sprachsignalverarbeitung 2013\n          \n          2013"},{"key":"B11","first-page":"103","article-title":"\u201cInpro_iss: a component for just-in-time incremental speech synthesis,\u201d","author":"Baumann","year":"2012","journal-title":"Proceedings of the ACL 2012 System Demonstrations"},{"key":"B12","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1609\/icaps.v30i1.6679","article-title":"\u201cNew developments for robert-assisting novice users even better in diy projects,\u201d","author":"Behnke","year":"2020","journal-title":"Proceedings of the International Conference on Automated Planning and Scheduling"},{"key":"B13","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1017\/S004740450001037X","article-title":"Language style as audience design","volume":"13","author":"Bell","year":"1984","journal-title":"Lang. Soc"},{"key":"B14","first-page":"34","article-title":"\u201cOpen-world dialog: challenges, directions, and prototype,\u201d","author":"Bohus","year":"2009","journal-title":"6th IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems"},{"key":"B15","doi-asserted-by":"publisher","first-page":"2355","DOI":"10.1016\/j.pragma.2009.12.012","article-title":"Linguistic alignment between people and computers","volume":"42","author":"Branigan","year":"2010","journal-title":"J. Pragmat"},{"key":"B16","doi-asserted-by":"publisher","first-page":"1482","DOI":"10.1037\/\/0278-7393.22.6.1482","article-title":"Conceptual pacts and lexical choice in conversation","volume":"22","author":"Brennan","year":"1996","journal-title":"J. Exper. Psychol"},{"key":"B17","doi-asserted-by":"publisher","first-page":"274","DOI":"10.1111\/j.1756-8765.2009.01019.x","article-title":"Partner-specific adaptation in dialog","volume":"1","author":"Brennan","year":"2009","journal-title":"Top. Cogn. Sci"},{"key":"B18","doi-asserted-by":"publisher","first-page":"274","DOI":"10.1006\/jmla.2000.2753","article-title":"How listeners compensate for disfluencies in spontaneous speech","volume":"44","author":"Brennan","year":"2001","journal-title":"J. Mem. Lang"},{"key":"B19","doi-asserted-by":"publisher","first-page":"2318","DOI":"10.1145\/3404835.3463055","article-title":"\u201cWhen choice happens: a systematic examination of mouse movement length for decision making in web search,\u201d","author":"Br\u00fcckner","year":"2021","journal-title":"Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval"},{"key":"B20","first-page":"295","article-title":"\u201cCombining incremental language generation and incremental speech synthesis for adaptive information presentation,\u201d","author":"Buschmeier","year":"2012","journal-title":"Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue"},{"key":"B21","doi-asserted-by":"publisher","first-page":"2012","DOI":"10.3758\/s13428-016-0839-5","article-title":"Analyzing spatial data from mouse tracker methodology: an entropic approach","volume":"49","author":"Calcagn\u00ed","year":"2017","journal-title":"Behav. Res. Methods"},{"key":"B22","first-page":"33","article-title":"\u201cCollaborative effort towards common ground in situated human-robot dialogue,\u201d","volume-title":"2014 9th ACM\/IEEE International Conference on Human-Robot Interaction (HRI)","author":"Chai","year":"2014"},{"key":"B23","doi-asserted-by":"publisher","first-page":"281","DOI":"10.1145\/634067.634234","article-title":"\u201cWhat can a mouse cursor tell us more? Correlation of eye\/mouse movements on web browsing,\u201d","author":"Chen","year":"2001","journal-title":"CHI'01 Extended Abstracts on Human Factors in Computing Systems"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511620539","author":"Clark","year":"1996","journal-title":"Using Language"},{"key":"B25","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1037\/10096-006","article-title":"\u201cGrounding in communication,\u201d","author":"Clark","year":"1991","journal-title":"Perspectives on socially shared cognition"},{"key":"B26","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1016\/j.jml.2003.08.004","article-title":"Speaking while monitoring addressees for understanding","volume":"50","author":"Clark","year":"2004","journal-title":"J. Mem. Lang"},{"key":"B27","first-page":"10","article-title":"\u201cDefinite knowledge and mutual knowledge,\u201d","author":"Clark","year":"1981","journal-title":"Elements of Discourse Understanding"},{"key":"B28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/0010-0277(86)90010-7","article-title":"Referring as a collaborative process","volume":"22","author":"Clark","year":"1986","journal-title":"Cognition"},{"key":"B29","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3290605.3300705","article-title":"\u201cWhat makes a good conversation? Challenges in designing truly conversational agents,\u201d","author":"Clark","year":"2019","journal-title":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems"},{"key":"B30","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1038\/d41586-021-01170-0","article-title":"Cooperative ai: machines must learn to find common ground","volume":"593","author":"Dafoe","year":"2021","journal-title":"Nature"},{"key":"B31","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1207\/s15516709cog1902_3","article-title":"Computational interpretations of the gricean maxims in the generation of referring expressions","volume":"19","author":"Dale","year":"1995","journal-title":"Cogn. Sci"},{"key":"B32","first-page":"78","article-title":"\u201cOptimising natural language generation decision making for situated dialogue,\u201d","author":"Dethlefs","year":"2011","journal-title":"Proceedings of the SIGDIAL 2011 Conference"},{"key":"B33","volume-title":"Contribution Tracking: Participating in Task-Oriented Dialogue Under Uncertainty","author":"DeVault","year":"2008"},{"key":"B34","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3115\/1225753.1225754","article-title":"\u201cAn information-state approach to collaborative reference,\u201d","author":"DeVault","year":"2005","journal-title":"Proceedings of the ACL Interactive Poster and Demonstration Sessions"},{"key":"B35","doi-asserted-by":"publisher","first-page":"1451","DOI":"10.1145\/2505515.2505717","article-title":"\u201cRobust models of mouse movement on dynamic web search results pages,\u201d","author":"Diaz","year":"2013","journal-title":"Proceedings of the 22nd ACM international conference on Information &Knowledge Management"},{"key":"B36","doi-asserted-by":"publisher","first-page":"103654","DOI":"10.1016\/j.robot.2020.103654","article-title":"The impact of adding perspective-taking to spatial referencing during human-robot interaction","volume":"134","author":"Do\u011fan","year":"2020","journal-title":"Rob. Auton. Syst"},{"key":"B37","article-title":"Open challenges on generating referring expressions for human-robot interaction","author":"Do\u011fan","year":"2021","journal-title":"arXiv preprint arXiv:2104.09193"},{"key":"B38","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1167\/11.5.14","article-title":"Visual search: a retrospective","volume":"11","author":"Eckstein","year":"2011","journal-title":"J. Vis"},{"key":"B39","doi-asserted-by":"publisher","first-page":"251","DOI":"10.1080\/25742442.2021.2007718","article-title":"Online data collection in auditory perception and cognition research: recruitment, testing, data quality and ethical considerations","volume":"4","author":"Eerola","year":"2021","journal-title":"Audit. Percept. Cogn"},{"key":"B40","doi-asserted-by":"publisher","first-page":"2981","DOI":"10.18653\/v1\/2020.findings-emnlp.268","article-title":"\u201cTurngpt: a transformer-based language model for predicting turn-taking in spoken dialog,\u201d","author":"Ekstedt","year":"2020","journal-title":"Findings of the Association for Computational Linguistics: EMNLP 2020"},{"key":"B41","doi-asserted-by":"publisher","first-page":"1354","DOI":"10.18653\/v1\/D13-1134","article-title":"\u201cPredicting the resolution of referring expressions from user behavior,\u201d","author":"Engonopoulos","year":"2013","journal-title":"Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing"},{"key":"B42","doi-asserted-by":"publisher","first-page":"271","DOI":"10.1145\/2696454.2696467","article-title":"\u201cEmbodied collaborative referring expression generation in situated human-robot interaction,\u201d","author":"Fang","year":"2015","journal-title":"Proceedings of the Tenth Annual ACM\/IEEE International Conference on Human-Robot Interaction"},{"key":"B43","first-page":"25","article-title":"\u201cPush-to-talk ain't always bad! Comparing different interactivity settings in task-oriented dialogue,\u201d","author":"Fern\u00e1ndez","year":"2007","journal-title":"Proceedings of the DECALOG 2007"},{"key":"B44","doi-asserted-by":"publisher","first-page":"381","DOI":"10.1037\/h0055392","article-title":"The information capacity of the human motor system in controlling the amplitude of movement","volume":"47","author":"Fitts","year":"1954","journal-title":"J. Exp. Psychol"},{"key":"B45","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1037\/\/0022-3514.62.3.378","article-title":"Coordination of knowledge in communication: effects of speakers' assumptions about what others know","volume":"62","author":"Fussell","year":"1992","journal-title":"J. Pers. Soc. Psychol"},{"key":"B46","doi-asserted-by":"publisher","first-page":"986","DOI":"10.1080\/01690965.2013.847190","article-title":"Generation of effective referring expressions in situated context","volume":"29","author":"Garoufi","year":"2014","journal-title":"Lang. Cogn. Neurosci"},{"key":"B47","doi-asserted-by":"publisher","first-page":"1671","DOI":"10.1111\/cogs.12298","article-title":"Exploiting listener gaze to improve situated communication in dynamic virtual environments","volume":"40","author":"Garoufi","year":"2016","journal-title":"Cogn. Sci"},{"key":"B48","article-title":"\u201cLearning task-oriented dialogues through various degrees of interactivity,\u201d","author":"Gigliobianco","year":"2024","journal-title":"Proceedings of the 28th Workshop on the Semantics and Pragmatics of Dialogue"},{"key":"B49","doi-asserted-by":"crossref","first-page":"264","DOI":"10.1109\/ROMAN.2010.5598618","article-title":"\u201cTowards a dialog strategy for handling miscommunication in human-robot dialog,\u201d","volume-title":"19th International Symposium in Robot and Human Interactive Communication","author":"Gonsior","year":"2010"},{"key":"B50","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1111\/tops.12007","article-title":"Knowledge and implicature: Modeling language understanding as social cognition","volume":"5","author":"Goodman","year":"2013","journal-title":"Top. Cogn. Sci"},{"key":"B51","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1163\/9789004368811_003","article-title":"\u201cLogic and conversation,\u201d","volume-title":"Speech Acts","author":"Grice","year":"1975"},{"key":"B52","volume-title":"Studies in the Way of Words","author":"Grice","year":"1989"},{"key":"B53","doi-asserted-by":"publisher","first-page":"707","DOI":"10.1145\/1390334.1390462","article-title":"\u201cExploring mouse movements for inferring query intent,\u201d","author":"Guo","year":"2008","journal-title":"Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval"},{"key":"B54","doi-asserted-by":"publisher","first-page":"3601","DOI":"10.1145\/1753846.1754025","article-title":"\u201cTowards predicting web searcher gaze position from mouse movements,\u201d","author":"Guo","year":"2010","journal-title":"CHI'10 Extended Abstracts on Human Factors in Computing Systems"},{"key":"B55","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2019-2820","article-title":"\u201cDo hesitations facilitate processing of partially defective system utterances? An exploratory eye tracking study,\u201d","author":"Haake","year":"2019","journal-title":"Proceedings of Interspeech"},{"key":"B56","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1017\/S0022226700016613","article-title":"Notes on transitivity and theme in English: Part 2","volume":"3","author":"Halliday","year":"1967","journal-title":"J. Linguist"},{"key":"B57","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1145\/302979.303030","article-title":"\u201cPrinciples of mixed-initiative user interfaces,\u201d","author":"Horvitz","year":"1999","journal-title":"Proceedings of the SIGCHI conference on Human Factors in Computing Systems"},{"key":"B58","doi-asserted-by":"publisher","first-page":"403","DOI":"10.1002\/9781119263685.ch16","article-title":"\u201cLearning from mouse movements: improving questionnaires and respondents' user experience through passive data collection,\u201d","author":"Horwitz","year":"2020","journal-title":"Advances in Questionnaire Design, Development, Evaluation and Testing"},{"key":"B59","doi-asserted-by":"publisher","first-page":"1341","DOI":"10.1145\/2207676.2208591","article-title":"\u201cUser see, user point: gaze and cursor alignment in web search,\u201d","author":"Huang","year":"","journal-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems"},{"key":"B60","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1145\/2348283.2348313","article-title":"\u201cImproving searcher models using mouse cursor activity,\u201d","author":"Huang","year":"","journal-title":"Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval"},{"key":"B61","doi-asserted-by":"publisher","first-page":"1225","DOI":"10.1145\/1978942.1979125","article-title":"\u201cNo clicks, no problem: using cursor movements to understand and improve search,\u201d","author":"Huang","year":"2011","journal-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems"},{"key":"B62","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1037\/\/0096-3445.116.1.26","article-title":"References in conversation between experts and novices","volume":"116","author":"Isaacs","year":"1987","journal-title":"J. Exper. Psychol"},{"key":"B63","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1109\/RO-MAN47096.2020.9223615","article-title":"\u201cUnderstanding the perception of incremental robot response in human-robot interaction,\u201d","volume-title":"2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)","author":"Jensen","year":"2020"},{"key":"B64","doi-asserted-by":"publisher","first-page":"802","DOI":"10.1002\/acp.2862","article-title":"Action as a window to perception: measuring attention with mouse movements","volume":"26","author":"Johnson","year":"2012","journal-title":"Appl. Cogn. Psychol"},{"key":"B65","doi-asserted-by":"publisher","first-page":"1041","DOI":"10.3115\/1220175.1220306","article-title":"\u201cIncremental generation of spatial referring expressions in situated dialog,\u201d","author":"Kelleher","year":"2006","journal-title":"Proceedings of the 21st international conference on computational linguistics and 44th annual meeting of the association for computational linguistics"},{"key":"B66","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1016\/j.csl.2016.04.002","article-title":"A simple generative model of incremental reference resolution for situated dialogue","volume":"41","author":"Kennington","year":"2017","journal-title":"Comput. Speech Lang"},{"key":"B67","doi-asserted-by":"crossref","first-page":"111","DOI":"10.4324\/9781315160559-9","article-title":"\u201cMouse-tracking: a practical guide to implementation and analysis 1,\u201d","volume-title":"A Handbook of Process Tracing Methods","author":"Kieslich","year":"2019"},{"key":"B68","doi-asserted-by":"publisher","first-page":"301","DOI":"10.3115\/1149293.1149332","article-title":"\u201cThe effects of remote gesturing on distance instruction,\u201d","author":"Kirk","year":"2017","journal-title":"Computer Supported Collaborative Learning 2005: The Next 10 Years"},{"key":"B69","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3419249.3421250","article-title":"\u201cUsing mouse movement heatmaps to visualize user attention to words,\u201d","author":"Kirsh","year":"2020","journal-title":"Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society"},{"key":"B70","first-page":"30","article-title":"\u201cEnhancing referential success by tracking hearer gaze,\u201d","author":"Koller","year":"2012","journal-title":"Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue"},{"key":"B71","doi-asserted-by":"publisher","DOI":"10.31237\/osf.io\/fpts4","author":"Kontogiorgos","year":"2022","journal-title":"Mutual understanding in situated interactions with conversational user interfaces: theory, studies, and computation"},{"key":"B72","doi-asserted-by":"publisher","first-page":"623657","DOI":"10.3389\/fpsyg.2021.623657","article-title":"Measuring collaboration load with pupillary responses-implications for the design of instructions in task-oriented HRI","volume":"12","author":"Kontogiorgos","year":"2021","journal-title":"Front. Psychol"},{"key":"B73","doi-asserted-by":"publisher","first-page":"311","DOI":"10.1145\/3371382.3378249","article-title":"\u201cTowards adaptive and least-collaborative-effort social robots,\u201d","author":"Kontogiorgos","year":"2020","journal-title":"Companion of the 2020 ACM\/IEEE International Conference on Human-Robot Interaction"},{"key":"B74","doi-asserted-by":"publisher","first-page":"414","DOI":"10.1145\/3340555.3353722","article-title":"\u201cEstimating uncertainty in task-oriented dialogue,\u201d","author":"Kontogiorgos","year":"2019","journal-title":"2019 International Conference on Multimodal Interaction"},{"key":"B75","doi-asserted-by":"publisher","first-page":"68","DOI":"10.3115\/v1\/W14-0212","article-title":"\u201cSituationally aware in-car information presentation using incremental speech generation: safer, and more effective,\u201d","author":"Kousidis","year":"2014","journal-title":"Proceedings of the EACL 2014 Workshop on Dialogue in Motion"},{"key":"B76","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1162\/COLI_a_00088","article-title":"Computational generation of referring expressions: a survey","volume":"38","author":"Krahmer","year":"2012","journal-title":"Comput. Linguist"},{"key":"B77","doi-asserted-by":"publisher","first-page":"83","DOI":"10.3390\/mti4040083","article-title":"Matmouse: a mouse movements tracking and analysis toolbox for visual search experiments","volume":"4","author":"Krassanakis","year":"2020","journal-title":"Multimodal Technol. Inter"},{"key":"B78","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1207\/S15327051HCI1812_2","article-title":"Visual information as a conversational resource in collaborative physical tasks","volume":"18","author":"Kraut","year":"2003","journal-title":"Hum. Comput. Inter"},{"key":"B79","first-page":"28","article-title":"\u201cReferring expression generation via visual dialogue,\u201d","volume-title":"CCF International Conference on Natural Language Processing and Chinese Computing","author":"Li","year":"2020"},{"key":"B80","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1007\/s10746-012-9213-5","article-title":"Instruction-in-interaction: the teaching and learning of a manual skill","volume":"35","author":"Lindwall","year":"2012","journal-title":"Hum. Stud"},{"key":"B81","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3544548.3580817","article-title":"\u201cwhat it wants me to say\u201d: Bridging the abstraction gap between end-user programmers and code-generating large language models,\u201d","author":"Liu","year":"2023","journal-title":"Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems"},{"key":"B82","doi-asserted-by":"publisher","first-page":"3113","DOI":"10.1109\/LRA.2018.2849607","article-title":"A multimodal classifier generative adversarial network for carry and place tasks from ambiguous language instructions","volume":"3","author":"Magassouba","year":"2018","journal-title":"IEEE Robot. Autom. Lett"},{"key":"B83","article-title":"\u201cUsing listener gaze to refer in installments benefits understanding,\u201d","author":"Mitev","year":"2018","journal-title":"CogSci"},{"key":"B84","doi-asserted-by":"publisher","first-page":"e0177851","DOI":"10.1371\/journal.pone.0177851","article-title":"The detection of faked identity using unexpected questions and mouse dynamics","volume":"12","author":"Monaro","year":"2017","journal-title":"PLoS ONE"},{"key":"B85","unstructured":"Morawiec\n              D.\n            \n          \n          sklearn-porter. Transpile trained scikit-learn estimators to C, Java, JavaScript and others\n          \n          2021"},{"key":"B86","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1145\/634067.634233","article-title":"\u201cCheese: tracking mouse movement activity on websites, a tool for user modeling,\u201d","author":"Mueller","year":"2001","journal-title":"CHI'01 Extended Abstracts on Human Factors in Computing Systems"},{"key":"B87","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1080\/13506280500527676","article-title":"Visual search and selective attention","volume":"14","author":"M\u00fcller","year":"2006","journal-title":"Vis. Cogn"},{"key":"B88","doi-asserted-by":"publisher","first-page":"2825","DOI":"10.5555\/1953048.2078195","article-title":"Scikit-learn: machine learning in python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res"},{"key":"B89","doi-asserted-by":"publisher","first-page":"4921","DOI":"10.1145\/2858036.2858478","article-title":"\u201cWhy that NAO? How humans adapt to a conventional humanoid robot in taking turns-at-talk,\u201d","author":"Pelikan","year":"2016","journal-title":"Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems"},{"key":"B90","doi-asserted-by":"publisher","first-page":"365","DOI":"10.1145\/3015783.3015794","article-title":"\u201cGaze-informed multimodal interaction,\u201d","author":"Qvarfordt","year":"2017","journal-title":"The Handbook of Multimodal-Multisensor Interfaces: Foundations, User Modeling, and Common Modality Combinations"},{"key":"B91","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1007\/BF00177327","article-title":"The elaboration theory of instruction: a model for sequencing and synthesizing instruction","volume":"9","author":"Reigeluth","year":"1980","journal-title":"Instruct. Sci"},{"key":"B92","article-title":"Sentence-bert: Sentence embeddings using siamese bert-networks","author":"Reimers","year":"2019","journal-title":"arXiv preprint arXiv:1908.10084"},{"key":"B93","first-page":"1982","article-title":"\u201cUse of mouse-tracking method to measure cognitive load,\u201d","volume-title":"Proceedings of the Human Factors and Ergonomics Society Annual Meeting","author":"Rheem","year":"2018"},{"key":"B94","doi-asserted-by":"publisher","first-page":"316","DOI":"10.18653\/v1\/2020.inlg-1.38","article-title":"\u201cFrom \u201cbefore\u201d to \u201cafter\u201d: Generating natural language instructions from image pairs in a simple visual domain,\u201d","author":"Rojowiec","year":"2020","journal-title":"Proceedings of the 13th International Conference on Natural Language Generation"},{"key":"B95","article-title":"\u201cTwo approaches to give: dynamic level adaptation versus playfulness,\u201d","author":"Rookhuiszen","year":"2009","journal-title":"Proceedings of the First NLG Challenge on Generating Instructions in Virtual Environments"},{"key":"B96","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1016\/B978-0-12-623550-0.50008-2","article-title":"\u201cA simplest systematics for the organization of turn taking for conversation,\u201d","volume-title":"Studies in the Organization of Conversational Interaction","author":"Sacks","year":"1978"},{"key":"B97","article-title":"Yes, this way! Learning to ground referring expressions into actions with intra-episodic feedback from supportive teachers","author":"Sadler","year":"2023","journal-title":"arXiv preprint arXiv:2305.12880"},{"key":"B98","doi-asserted-by":"publisher","first-page":"2098","DOI":"10.18653\/v1\/2023.eacl-main.154","article-title":"\u201cPento-diaref: a diagnostic dataset for learning the incremental algorithm for referring expression generation from examples,\u201d","author":"Sadler","year":"2023","journal-title":"Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics"},{"key":"B99","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2014.X.002","article-title":"\u201cEffective task training strategies for instructional robots,\u201d","author":"Saupp","year":"2014","journal-title":"Proceedings of the 10th Annual Robotics: Science and Systems Conference"},{"key":"B100","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1007\/s10514-015-9461-0","article-title":"Effective task training strategies for human and robot instructors","volume":"39","author":"Saupp\u00e9","year":"2015","journal-title":"Auton. Robots"},{"key":"B101","doi-asserted-by":"publisher","first-page":"51","DOI":"10.18653\/v1\/2007.sigdial-1.10","article-title":"\u201cBeyond repair-testing the limits of the conversational repair system,\u201d","author":"Schlangen","year":"2007","journal-title":"Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue"},{"key":"B102","author":"Schlangen","year":"2008","journal-title":"The Potsdam Dialogue Corpora: Transcription and Annotation Manual."},{"key":"B103","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1016\/0010-0285(89)90008-X","article-title":"Understanding by addressees and overhearers","volume":"21","author":"Schober","year":"1989","journal-title":"Cogn. Psychol"},{"key":"B104","doi-asserted-by":"publisher","first-page":"766","DOI":"10.3758\/s13423-020-01851-3","article-title":"Using mouse cursor tracking to investigate online cognition: preserving methodological ingenuity while moving toward reproducible science","volume":"28","author":"Schoemann","year":"2021","journal-title":"Psychon. Bull. Rev"},{"key":"B105","first-page":"1","article-title":"\u201cTowards incremental speech generation in dialogue systems,\u201d","author":"Skantze","year":"2010","journal-title":"Proceedings of the SIGDIAL 2010 Conference"},{"key":"B106","doi-asserted-by":"publisher","first-page":"979","DOI":"10.1145\/2600428.2609489","article-title":"\u201cMouse movement during relevance judging: implications for determining user attention,\u201d","author":"Smucker","year":"2014","journal-title":"Proceedings of the 37th International ACM SIGIR Conference on Research &Development in Information Retrieval"},{"key":"B107","article-title":"\u201cUsing listener gaze to augment speech generation in a virtual 3D environment,\u201d","author":"Staudte","year":"2012","journal-title":"Proceedings of the Annual Meeting of the Cognitive Science Society"},{"key":"B108","doi-asserted-by":"publisher","first-page":"81","DOI":"10.3115\/1706269.1706286","article-title":"\u201cNoun phrase generation for situated dialogs,\u201d","author":"Stoia","year":"2006","journal-title":"Proceedings of the Fourth International Natural Language Generation Conference"},{"key":"B109","first-page":"12","article-title":"\u201cReferring in installments: a corpus study of spoken object references in an interactive virtual environment,\u201d","author":"Striegnitz","year":"2012","journal-title":"INLG 2012 Proceedings of the Seventh International Natural Language Generation Conference"},{"key":"B110","doi-asserted-by":"publisher","DOI":"10.21437\/SSW.2019-44","article-title":"\u201cHow to train your fillers: uh and um in spontaneous speech synthesis,\u201d","author":"Sz\u00e9kely","year":"2019","journal-title":"The 10th ISCA Speech Synthesis Workshop"},{"key":"B111","unstructured":"Team\n              R Developement Core.\n            \n          \n          A language and environment for statistical computing\n          \n          2009"},{"key":"B112","doi-asserted-by":"publisher","first-page":"1507","DOI":"10.1609\/aaai.v25i1.7979","article-title":"\u201cUnderstanding natural language commands for robotic navigation and mobile manipulation,\u201d","author":"Tellex","year":"2011","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"B113","article-title":"\u201cThe dynamics of pragmatic enrichment during metaphor processing: activation vs. suppression,\u201d","author":"Tomlinson Jr","year":"2013","journal-title":"Proceedings of the Annual Meeting of the Cognitive Science Society"},{"key":"B114","article-title":"\u201cHow intonation contrains pragmatic inference,\u201d","author":"Tomlinson Jr","year":"2013","journal-title":"Proceedings of the Annual Meeting of the Cognitive Science Society"},{"key":"B115","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1109\/HRI.2013.6483599","article-title":"\u201cHow a robot should give advice,\u201d","volume-title":"2013 8th ACM\/IEEE International Conference on Human-Robot Interaction (HRI)","author":"Torrey","year":"2013"},{"key":"B116","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1145\/1228716.1228750","article-title":"\u201cExploring adaptive dialogue based on a robot's awareness of human gaze and task progress,\u201d","author":"Torrey","year":"2007","journal-title":"Proceedings of the ACM\/IEEE International Conference on Human-Robot Interaction"},{"key":"B117","doi-asserted-by":"publisher","first-page":"126","DOI":"10.1145\/1121241.1121264","article-title":"\u201cEffects of adaptive robot dialogue on information exchange and social relations,\u201d","author":"Torrey","year":"2006","journal-title":"Proceedings of the 1st ACM SIGCHI\/SIGART Conference on Human-Robot Interaction"},{"key":"B118","doi-asserted-by":"publisher","first-page":"575","DOI":"10.1111\/j.1467-8640.1992.tb00380.x","article-title":"Conversation acts in task-oriented spoken dialogue","volume":"8","author":"Traum","year":"1992","journal-title":"Comput. Intell"},{"key":"B119","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1093\/acprof:oso\/9780199231751.001.0001","article-title":"Embodied communication in humans and machines","volume":"26","author":"Wachsmuth","year":"2008","journal-title":"AI Magaz"},{"key":"B120","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.wocn.2014.11.001","article-title":"In defense of stylistic diversity in speech research","volume":"48","author":"Wagner","year":"2015","journal-title":"J. Phon"},{"key":"B121","doi-asserted-by":"publisher","first-page":"67","DOI":"10.3389\/frobt.2019.00067","article-title":"Generating spatial referring expressions in a social robot: dynamic vs. non-ambiguous","volume":"6","author":"Wallbridge","year":"2019","journal-title":"Front. Robot. AI"},{"key":"B122","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3481628","article-title":"The effectiveness of dynamically processed incremental descriptions in human robot interaction","volume":"11","author":"Wallbridge","year":"2021","journal-title":"ACM Trans. Hum. Robot Inter"},{"key":"B123","doi-asserted-by":"publisher","first-page":"251","DOI":"10.1145\/3382507.3418863","article-title":"\u201cGesture enhanced comprehension of ambiguous human-to-robot instructions,\u201d","author":"Weerakoon","year":"2020","journal-title":"Proceedings of the 2020 International Conference on Multimodal Interaction"},{"key":"B124","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1016\/S0747-5632(98)00036-3","article-title":"Analysis of cursor movements with a mouse","volume":"15","author":"Whisenand","year":"1999","journal-title":"Comput. Human Behav"},{"key":"B125","doi-asserted-by":"publisher","first-page":"75","DOI":"10.18653\/v1\/W17-3511","article-title":"\u201cReferring expression generation under uncertainty: algorithm and evaluation framework,\u201d","author":"Williams","year":"2017","journal-title":"Proceedings of the 10th International Conference on Natural Language Generation"},{"key":"B126","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1016\/j.concog.2014.04.004","article-title":"Semantic priming revealed by mouse movement trajectories","volume":"27","author":"Xiao","year":"2014","journal-title":"Conscious. Cogn"},{"key":"B127","first-page":"2771","article-title":"\u201cTalking with tact: polite language as a balance between kindness and informativity,\u201d","volume-title":"Proceedings of the 38th Annual Conference of the Cognitive Science Society","author":"Yoon","year":"2016"},{"key":"B128","first-page":"125","article-title":"\u201cPentoref: a corpus of spoken references in task-oriented dialogues,\u201d","author":"Zarrie\u00df","year":"2016","journal-title":"Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)"},{"key":"B129","doi-asserted-by":"publisher","first-page":"610","DOI":"10.18653\/v1\/P16-1058","article-title":"\u201cEasy things first: Installments improve referring expression generation for objects in photographs,\u201d","author":"Zarrie\u00df","year":"2016","journal-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"},{"key":"B130","first-page":"41","article-title":"\u201cPauses and the temporal structure of speech,\u201d","author":"Zellner","year":"1994"}],"container-title":["Frontiers in Computer Science"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fcomp.2025.1634228\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,12]],"date-time":"2025-12-12T12:30:48Z","timestamp":1765542648000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fcomp.2025.1634228\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,12]]},"references-count":130,"alternative-id":["10.3389\/fcomp.2025.1634228"],"URL":"https:\/\/doi.org\/10.3389\/fcomp.2025.1634228","relation":{},"ISSN":["2624-9898"],"issn-type":[{"value":"2624-9898","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,12]]},"article-number":"1634228"}}