{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,4]],"date-time":"2026-06-04T06:13:07Z","timestamp":1780553587061,"version":"3.54.1"},"reference-count":49,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2024,7,22]],"date-time":"2024-07-22T00:00:00Z","timestamp":1721606400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,7,22]],"date-time":"2024-07-22T00:00:00Z","timestamp":1721606400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100013000","name":"Politecnico di Torino","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100013000","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Pers Ubiquit Comput"],"published-print":{"date-parts":[[2024,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Smart speakers are entering our homes and enriching the connected ecosystem already present in them. Home inhabitants can use those to execute relatively simple commands, e.g., turning a lamp on. Their capabilities to interpret more complex and ambiguous commands (e.g., make this room warmer) are limited, if not absent. Large language models (LLMs) can offer creative and viable solutions to enable a practical and user-acceptable interpretation of such ambiguous commands. This paper introduces an interactive disambiguation approach that integrates visual and textual cues with natural language commands. After contextualizing the approach with a use case, we test it in an experiment where users are prompted to select the appropriate cue (an image or a textual description) to clarify ambiguous commands, thereby refining the accuracy of the system\u2019s interpretations. Outcomes from the study indicate that the disambiguation system produces responses well-aligned with user intentions, and that participants found the textual descriptions slightly more effective. Finally, interviews reveal heightened satisfaction with the smart-home system when engaging with the proposed disambiguation approach.<\/jats:p>","DOI":"10.1007\/s00779-024-01827-3","type":"journal-article","created":{"date-parts":[[2024,7,22]],"date-time":"2024-07-22T09:01:54Z","timestamp":1721638914000},"page":"985-1000","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Enhancing smart home interaction through multimodal command disambiguation"],"prefix":"10.1007","volume":"28","author":[{"given":"Tommaso","family":"Cal\u00f2","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7647-6652","authenticated-orcid":false,"given":"Luigi","family":"De Russis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,7,22]]},"reference":[{"issue":"4","key":"1827_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3447242","volume":"54","author":"R Dunne","year":"2021","unstructured":"Dunne R, Morris T, Harper S (2021) A survey of ambient intelligence. ACM Comput Surv (CSUR) 54(4):1\u201327","journal-title":"ACM Comput Surv (CSUR)"},{"issue":"3","key":"1827_CR2","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1145\/329124.329126","volume":"3","author":"M Weiser","year":"1999","unstructured":"Weiser M (1999) The computer for the 21st century. ACM SIGMOBILE Mobile Computing and Communications Review 3(3):3\u201311","journal-title":"ACM SIGMOBILE Mobile Computing and Communications Review"},{"issue":"12","key":"1827_CR3","doi-asserted-by":"publisher","first-page":"4278","DOI":"10.3390\/en15124278","volume":"15","author":"D Kim","year":"2022","unstructured":"Kim D, Yoon Y, Lee J, Mago PJ, Lee K, Cho H (2022) Design and implementation of smart buildings: a review of current research trend. Energies 15(12):4278","journal-title":"Energies"},{"key":"1827_CR4","doi-asserted-by":"publisher","first-page":"101168","DOI":"10.1016\/j.cosust.2022.101168","volume":"55","author":"A Branny","year":"2022","unstructured":"Branny A, M\u00f8ller MS, Korpilo S, McPhearson T, Gulsrud N, Olafsson AS, Raymond CM, Andersson E (2022) Smarter greener cities through a social-ecological-technological systems approach. Curr Opin Environ Sustain 55:101168","journal-title":"Curr Opin Environ Sustain"},{"issue":"12","key":"1827_CR5","doi-asserted-by":"publisher","first-page":"2470","DOI":"10.1109\/JPROC.2013.2262913","volume":"101","author":"G Acampora","year":"2013","unstructured":"Acampora G, Cook DJ, Rashidi P, Vasilakos AV (2013) A survey on ambient intelligence in healthcare. Proc IEEE 101(12):2470\u20132494","journal-title":"Proc IEEE"},{"key":"1827_CR6","doi-asserted-by":"publisher","first-page":"48","DOI":"10.1016\/j.jnca.2017.08.017","volume":"97","author":"M Alaa","year":"2017","unstructured":"Alaa M, Zaidan AA, Zaidan BB, Talal M, Mat Kiah ML (2017) A review of smart home applications based on internet of things. J Netw Comput Appl 97:48\u201365","journal-title":"J Netw Comput Appl"},{"key":"1827_CR7","unstructured":"Lutolf R (1992) Smart home concept and the integration of energy meters into a home based system. In: Seventh international conference on metering apparatus and tariffs for electricity supply, pp 277\u2013278"},{"key":"1827_CR8","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2020.106412","volume":"111","author":"C-WC Ki","year":"2020","unstructured":"Ki C-WC, Cho E, Lee J-E (2020) Can an intelligent personal assistant (IPA) be your friend? Para-friendship development mechanism between IPAs and their users. Comput Hum Behav 111:106412. https:\/\/doi.org\/10.1016\/j.chb.2020.106412","journal-title":"Comput Hum Behav"},{"key":"1827_CR9","doi-asserted-by":"publisher","first-page":"463","DOI":"10.1007\/s00779-014-0813-0","volume":"19","author":"C Wilson","year":"2015","unstructured":"Wilson C, Hargreaves T, Hauxwell-Baldwin R (2015) Smart homes and their users: a systematic analysis and key challenges. Pers Ubiquit Comput 19:463\u2013476","journal-title":"Pers Ubiquit Comput"},{"key":"1827_CR10","doi-asserted-by":"crossref","unstructured":"Reisinger MR, Prost S, Schrammel J, Fr\u00f6hlich P (2022) User requirements for the design of smart homes: dimensions and goals. J Ambient Intell Humaniz Comput, 1\u201320","DOI":"10.1007\/s12652-021-03651-6"},{"key":"1827_CR11","unstructured":"IFTTT (2023) IFTTT. https:\/\/ifttt.com\/. Retrieved May 1, 2023"},{"issue":"3","key":"1827_CR12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3132031","volume":"1","author":"M Clark","year":"2017","unstructured":"Clark M, Newman MW, Dutta P (2017) Devices and data and agents, oh my: how smart home abstractions prime end-user mental models. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1(3):1\u201326","journal-title":"Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"},{"key":"1827_CR13","doi-asserted-by":"crossref","unstructured":"Yu H, Hua J, Julien C (2021) Dataset: analysis of IFTTT recipes to study how humans use Internet-of-Things (IoT) devices. In: Proceedings of the 19th ACM conference on embedded networked sensor systems, pp 537\u2013541","DOI":"10.1145\/3485730.3494115"},{"key":"1827_CR14","doi-asserted-by":"publisher","unstructured":"Upadhyay P, Heung S, Azenkot S, Brewer RN (2023) Studying exploration & long-term use of voice assistants by older adults. In: Proceedings of the 2023 CHI conference on human factors in computing systems. CHI \u201923. Association for Computing Machinery, New York, USA. https:\/\/doi.org\/10.1145\/3544548.3580925","DOI":"10.1145\/3544548.3580925"},{"issue":"4","key":"1827_CR15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3373759","volume":"27","author":"A Pradhan","year":"2020","unstructured":"Pradhan A, Lazar A, Findlater L (2020) Use of intelligent voice assistants by older adults with low technology use. ACM Transactions on Computer-Human Interaction (TOCHI) 27(4):1\u201327","journal-title":"ACM Transactions on Computer-Human Interaction (TOCHI)"},{"key":"1827_CR16","doi-asserted-by":"crossref","unstructured":"Luger E, Sellen A (2016) \u201cLike Having a Really Bad PA\u201d: the gulf between user expectation and experience of conversational agents. In: Proceedings of the 2016 CHI conference on human factors in computing systems, pp 5286\u20135297","DOI":"10.1145\/2858036.2858288"},{"key":"1827_CR17","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2021.106914","volume":"124","author":"S Kim","year":"2021","unstructured":"Kim S, Choudhury A (2021) Exploring older adults\u2019 perception and use of smart speaker-based voice assistants: a longitudinal study. Comput Hum Behav 124:106914","journal-title":"Comput Hum Behav"},{"key":"1827_CR18","doi-asserted-by":"crossref","unstructured":"Cowan BR, Pantidi N, Coyle D, Morrissey K, Clarke P, Al-Shehri S, Earley D, Bandeira N (2017) \u201cWhat Can I Help You With?\u201d: infrequent users\u2019 experiences of intelligent personal assistants. In: Proceedings of the 19th international conference on human-computer interaction with mobile devices and services, pp 1\u201312","DOI":"10.1145\/3098279.3098539"},{"key":"1827_CR19","doi-asserted-by":"crossref","unstructured":"Upadhyay P, Heung S, Azenkot S, Brewer RN (2023) Studying exploration & long-term use of voice assistants by older adults. In: Proceedings of the 2023 CHI conference on human factors in computing systems, pp 1\u201311","DOI":"10.1145\/3544548.3580925"},{"key":"1827_CR20","doi-asserted-by":"publisher","unstructured":"King E, Yu H, Lee S, Julien C (2024) Sasha: creative goal-oriented reasoning in smart homes with large language models. Proc ACM Interact Mob Wearable Ubiquitous Technol 8(1). https:\/\/doi.org\/10.1145\/3643505","DOI":"10.1145\/3643505"},{"key":"1827_CR21","unstructured":"Rivkin D, Hogan F, Feriani A, Konar A, Sigal A, Liu S, Dudek G (2023) SAGE: smart home agent with grounded execution. arXiv preprint arXiv:2311.00772"},{"key":"1827_CR22","doi-asserted-by":"publisher","first-page":"171","DOI":"10.1007\/978-3-030-50578-3_13","volume-title":"Web engineering","author":"M Noura","year":"2020","unstructured":"Noura M, Heil S, Gaedke M (2020) VISH: does your smart home dialogue system also need training data? In: Bielikova M, Mikkonen T, Pautasso C (eds) Web engineering. Springer, Cham, pp 171\u2013187"},{"issue":"4","key":"1827_CR23","first-page":"1","volume":"6","author":"RD Manu","year":"2019","unstructured":"Manu RD, Kumar S, Snehashish S, Rekha K (2019) Smart home automation using IoT and deep learning. Int Res J Eng Technol 6(4):1\u20134","journal-title":"Int Res J Eng Technol"},{"key":"1827_CR24","doi-asserted-by":"crossref","unstructured":"Rani PJ, Bakthakumar J, Kumaar BP, Kumaar UP, Kumar S (2017) Voice controlled home automation system using Natural Language Processing (NLP) and Internet of Things (IoT). In: 2017 Third international conference on science technology engineering & management (ICONSTEM), pp 368\u2013373","DOI":"10.1109\/ICONSTEM.2017.8261311"},{"key":"1827_CR25","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1007\/s10796-016-9670-x","volume":"20","author":"J Palanca","year":"2018","unstructured":"Palanca J, Val E, Garcia-Fornes A, Billhardt H, Corchado JM, Juli\u00e1n V (2018) Designing a goal-oriented smart-home environment. Inf Syst Front 20:125\u2013142","journal-title":"Inf Syst Front"},{"key":"1827_CR26","doi-asserted-by":"crossref","unstructured":"Noura M, Heil S, Gaedke M (2020) Natural language goal understanding for smart home environments. In: Proceedings of the 10th international conference on the internet of things, pp 1\u20138","DOI":"10.1145\/3410992.3410996"},{"key":"1827_CR27","unstructured":"Gao L, Biderman S, Black S, Golding L, Hoppe T, Foster C, Phang J, He H, Thite A, Nabeshima N et al (2020) The Pile: an 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027"},{"issue":"8","key":"1827_CR28","first-page":"9","volume":"1","author":"A Radford","year":"2019","unstructured":"Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I et al (2019) Language models are unsupervised multitask learners. OpenAI Blog 1(8):9","journal-title":"OpenAI Blog"},{"key":"1827_CR29","first-page":"1877","volume":"33","author":"T Brown","year":"2020","unstructured":"Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A et al (2020) Language models are few-shot learners. Adv Neural Inf Process Syst 33:1877\u20131901","journal-title":"Adv Neural Inf Process Syst"},{"key":"1827_CR30","unstructured":"Wei J, Bosma M, Zhao V, Guu K, Yu AW, Lester B, Du N, Dai AM, Le QV (2022) Finetuned language models are zero-shot learners. In: International conference on learning representations. https:\/\/openreview.net\/forum?id=gEZrGCozdqR"},{"key":"1827_CR31","doi-asserted-by":"publisher","unstructured":"Liang J, Huang W, Xia F, Xu P, Hausman K, Ichter B, Florence P, Zeng A (2023) Code as policies: language model programs for embodied control. In: 2023 IEEE International conference on robotics and automation (ICRA), pp 9493\u20139500. https:\/\/doi.org\/10.1109\/ICRA48891.2023.10160591","DOI":"10.1109\/ICRA48891.2023.10160591"},{"key":"1827_CR32","doi-asserted-by":"publisher","unstructured":"Wu J, Antonova R, Kan A, Lepert M, Zeng A, Song S, Bohg J, Rusinkiewicz S, Funkhouser T (2023) TidyBot: personalized robot assistance with large language models. In: 2023 IEEE\/RSJ International conference on intelligent robots and systems (IROS), pp 3546\u20133553. https:\/\/doi.org\/10.1109\/IROS55552.2023.10341577","DOI":"10.1109\/IROS55552.2023.10341577"},{"key":"1827_CR33","doi-asserted-by":"publisher","unstructured":"Park JS, O\u2019Brien J, Cai CJ, Morris MR, Liang P, Bernstein MS (2023) Generative agents: interactive simulacra of human behavior. In: Proceedings of the 36th annual acm symposium on user interface software and technology. UIST \u201923. Association for Computing Machinery, New York, USA. https:\/\/doi.org\/10.1145\/3586183.3606763","DOI":"10.1145\/3586183.3606763"},{"key":"1827_CR34","unstructured":"Touvron H, Lavril T, Izacard G, Martinet X, Lachaux M-A, Lacroix T, Rozi\u00e8re B, Goyal N, Hambro E, Azhar F et al (2023) Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971"},{"key":"1827_CR35","doi-asserted-by":"crossref","unstructured":"K\u00ebpuska V, Bohouta G (2018) Next-generation of virtual personal assistants (Microsoft Cortana, Apple Siri, Amazon Alexa and Google Home). In: Proceedings of the 2018 IEEE 8th annual computing and communication workshop and conference (CCWC). IEEE, Las Vegas, NV, USA, pp 99\u2013103","DOI":"10.1109\/CCWC.2018.8301638"},{"key":"1827_CR36","doi-asserted-by":"crossref","unstructured":"Almeida N, Silva S, Teixeira A, Ketsmur M, Guimar\u00e3es D, Fonseca E (2018) Multimodal interaction for accessible smart homes. In: Proceedings of the 8th international conference on software development and technologies for enhancing accessibility and fighting info-exclusion. ACM, New York, USA, pp 63\u201370","DOI":"10.1145\/3218585.3218595"},{"key":"1827_CR37","doi-asserted-by":"crossref","unstructured":"Liu C, Xie W, Zhang P, Zhan J, Xiao Z (2018) Considerations on multimodal human-computer interaction. In: Proceedings of the 2018 5th IEEE international conference on cloud computing and intelligence systems (CCIS). IEEE, Nanjing, China, pp 331\u2013335","DOI":"10.1109\/CCIS.2018.8691254"},{"key":"1827_CR38","doi-asserted-by":"publisher","unstructured":"Almeida N, Teixeira A, Silva S, Ketsmur M (2019) The AM4I architecture and framework for multimodal interaction and its application to smart environments. Sensors 19:2587. https:\/\/doi.org\/10.3390\/s19112587","DOI":"10.3390\/s19112587"},{"key":"1827_CR39","doi-asserted-by":"crossref","unstructured":"Contreras-Casta\u00f1eda MA, Holgado-Terriza JA, Pomboza-Junez G, Paderewski-Rodr\u00edguez P, Guti\u00e9rrez-Vela FL (2019) Smart home: multimodal interaction for control of home devices. In: Proceedings of the XX international conference on human computer interaction. ACM, New York, NY, USA","DOI":"10.1145\/3335595.3335636"},{"key":"1827_CR40","doi-asserted-by":"publisher","unstructured":"Ne\u00dfelrath R, Lu C, Schulz CH, Frey J, Alexandersson J (2011) A gesture based system for context-sensitive interaction with smart homes. Springer, Berlin\/Heidelberg, Germany, pp 209\u2013219. https:\/\/doi.org\/10.1007\/978-3-642-18167-2_15","DOI":"10.1007\/978-3-642-18167-2_15"},{"key":"1827_CR41","doi-asserted-by":"publisher","unstructured":"Marques B, Dias P, Alves J, Santos BS (2020) In: Ahram T, Karwowski W, Pickl S, Taiar R (eds) Adaptive augmented reality user interfaces using face recognition for smart home control. Springer, Cham, Switzerland, pp 15\u201319. https:\/\/doi.org\/10.1007\/978-3-030-27928-8_3","DOI":"10.1007\/978-3-030-27928-8_3"},{"key":"1827_CR42","doi-asserted-by":"publisher","first-page":"4789803","DOI":"10.1155\/2016\/4789803","volume":"2016","author":"A Vlachostergiou","year":"2016","unstructured":"Vlachostergiou A, Stratogiannis G, Caridakis G, Siolas G, Mylonas P (2016) User adaptive and context-aware smart home using pervasive and semantic technologies. J Electr Comput Eng 2016:4789803. https:\/\/doi.org\/10.1155\/2016\/4789803","journal-title":"J Electr Comput Eng"},{"key":"1827_CR43","doi-asserted-by":"publisher","unstructured":"Gull\u00e1 F, Ceccacci S, Menghi R, Cavalieri L, Germani M (2017) In: Cavallo F, Marletta V, Monteri\u00f9 A, Siciliano P (eds) Adaptive interface for smart home: a new design approach. Springer, Cham, Switzerland, pp 107\u2013115. https:\/\/doi.org\/10.1007\/978-3-319-54283-6_8","DOI":"10.1007\/978-3-319-54283-6_8"},{"key":"1827_CR44","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1016\/j.eswa.2017.01.014","volume":"75","author":"P Chahuara","year":"2017","unstructured":"Chahuara P, Portet F, Vacher M (2017) Context-aware decision making under uncertainty for voice-based control of smart home. Expert Syst Appl 75:63\u201379. https:\/\/doi.org\/10.1016\/j.eswa.2017.01.014","journal-title":"Expert Syst Appl"},{"key":"1827_CR45","unstructured":"OpenAI (2024) OpenAI API. https:\/\/openai.com\/api\/. Accessed 27 Mar 2024"},{"key":"1827_CR46","unstructured":"LangChain (2024) A library for building applications with language models. https:\/\/github.com\/LangChain\/langchain. Accessed 27 Mar 2024"},{"key":"1827_CR47","unstructured":"OpenAI (2024) DALL$$\\cdot $$E: a neural network-based image generation model. https:\/\/openai.com\/dall-e. Accessed 27 Mar 2024"},{"key":"1827_CR48","unstructured":"React (2024) A JavaScript library for building user interfaces. https:\/\/reactjs.org\/. Accessed 27 Mar 2024"},{"issue":"2","key":"1827_CR49","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1191\/1478088706qp063oa","volume":"3","author":"V Braun","year":"2006","unstructured":"Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3(2):77\u2013101. https:\/\/doi.org\/10.1191\/1478088706qp063oa","journal-title":"Qual Res Psychol"}],"container-title":["Personal and Ubiquitous Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00779-024-01827-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00779-024-01827-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00779-024-01827-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,28]],"date-time":"2024-12-28T08:04:45Z","timestamp":1735373085000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00779-024-01827-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,22]]},"references-count":49,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,12]]}},"alternative-id":["1827"],"URL":"https:\/\/doi.org\/10.1007\/s00779-024-01827-3","relation":{},"ISSN":["1617-4909","1617-4917"],"issn-type":[{"value":"1617-4909","type":"print"},{"value":"1617-4917","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,22]]},"assertion":[{"value":"29 February 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 July 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 July 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no Conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}}]}}