{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,6]],"date-time":"2026-06-06T16:50:41Z","timestamp":1780764641051,"version":"3.54.1"},"reference-count":64,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,3,6]],"date-time":"2024-03-06T00:00:00Z","timestamp":1709683200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100006374","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1909221"],"award-info":[{"award-number":["1909221"]}],"id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2024,3,6]]},"abstract":"<jats:p>Smart home assistants function best when user commands are direct and well-specified---e.g., \"turn on the kitchen light\"---or when a hard-coded routine specifies the response. In more natural communication, however, human speech is unconstrained, often describing goals (e.g., \"make it cozy in here\" or \"help me save energy\") rather than indicating specific target devices and actions to take on those devices. Current systems fail to understand these under-specified commands since they cannot reason about devices and settings as they relate to human situations. We introduce large language models (LLMs) to this problem space, exploring their use for controlling devices and creating automation routines in response to under-specified user commands in smart homes. We empirically study the baseline quality and failure modes of LLM-created action plans with a survey of age-diverse users. We find that LLMs can reason creatively to achieve challenging goals, but they experience patterns of failure that diminish their usefulness. We address these gaps with Sasha, a smarter smart home assistant. Sasha responds to loosely-constrained commands like \"make it cozy\" or \"help me sleep better\" by executing plans to achieve user goals---e.g., setting a mood with available devices, or devising automation routines. We implement and evaluate Sasha in a hands-on user study, showing the capabilities and limitations of LLM-driven smart homes when faced with unconstrained user-generated scenarios.<\/jats:p>","DOI":"10.1145\/3643505","type":"journal-article","created":{"date-parts":[[2024,3,6]],"date-time":"2024-03-06T13:12:36Z","timestamp":1709730756000},"page":"1-38","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":69,"title":["Sasha"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4689-4591","authenticated-orcid":false,"given":"Evan","family":"King","sequence":"first","affiliation":[{"name":"University of Texas at Austin, Austin, TX, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3518-946X","authenticated-orcid":false,"given":"Haoxiang","family":"Yu","sequence":"additional","affiliation":[{"name":"University of Texas at Austin, Austin, TX, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3348-3163","authenticated-orcid":false,"given":"Sangsu","family":"Lee","sequence":"additional","affiliation":[{"name":"University of Texas at Austin, Austin, TX, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4131-4642","authenticated-orcid":false,"given":"Christine","family":"Julien","sequence":"additional","affiliation":[{"name":"University of Texas at Austin, Austin, TX, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,3,6]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2022. Google Nest API Documentation. https:\/\/developers.google.com\/nest\/device-access"},{"key":"e_1_2_1_2_1","unstructured":"2022. Insteon REST API Documentation. https:\/\/www.insteon.com\/developer"},{"key":"e_1_2_1_3_1","unstructured":"2022. Philips Hue API Documentation. https:\/\/developers.meethue.com\/"},{"key":"e_1_2_1_4_1","unstructured":"2023. IFTTT. Retrieved May 1 2023 from https:\/\/ifttt.com\/"},{"key":"e_1_2_1_5_1","unstructured":"2023. TP-Link Kasa Smart Plugs. https:\/\/www.kasasmart.com\/us\/products\/smart-plugs"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2013.2262913"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jnca.2017.08.017"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3264901"},{"key":"e_1_2_1_9_1","unstructured":"Robert Botsch. 2011. Chapter 12: Significance and measures of association. Scopes and Methods of Political Science (2011)."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cosust.2022.101168"},{"key":"e_1_2_1_11_1","unstructured":"Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020) 1877--1901."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1515\/9783110218329"},{"key":"e_1_2_1_13_1","unstructured":"Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra Adam Roberts Paul Barham Hyung Won Chung Charles Sutton Sebastian Gehrmann Parker Schuh Kensen Shi Sasha Tsvyashchenko Joshua Maynez Abhishek Rao Parker Barnes Yi Tay Noam Shazeer Vinodkumar Prabhakaran Emily Reif Nan Du Ben Hutchinson Reiner Pope James Bradbury Jacob Austin Michael Isard Guy Gur-Ari Pengcheng Yin Toju Duke Anselm Levskaya Sanjay Ghemawat Sunipa Dev Henryk Michalewski Xavier Garcia Vedant Misra Kevin Robinson Liam Fedus Denny Zhou Daphne Ippolito David Luan Hyeontaek Lim Barret Zoph Alexander Spiridonov Ryan Sepassi David Dohan Shivani Agrawal Mark Omernick Andrew M. Dai Thanumalayan Sankaranarayana Pillai Marie Pellat Aitor Lewkowycz Erica Moreira Rewon Child Oleksandr Polozov Katherine Lee Zongwei Zhou Xuezhi Wang Brennan Saeta Mark Diaz Orhan Firat Michele Catasta Jason Wei Kathy Meier-Hellstern Douglas Eck Jeff Dean Slav Petrov and Noah Fiedel. 2022. PaLM: Scaling Language Modeling with Pathways. arXiv:2204.02311 [cs.CL]"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132031"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2010.112"},{"key":"e_1_2_1_16_1","volume-title":"A definition of relevance for information retrieval. Information storage and retrieval 7, 1","author":"Cooper William S","year":"1971","unstructured":"William S Cooper. 1971. A definition of relevance for information retrieval. Information storage and retrieval 7, 1 (1971), 19--37."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3098279.3098539"},{"key":"e_1_2_1_18_1","volume-title":"Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)."},{"key":"e_1_2_1_19_1","volume-title":"Understanding and using context. Personal and ubiquitous computing 5","author":"Dey Anind K","year":"2001","unstructured":"Anind K Dey. 2001. Understanding and using context. Personal and ubiquitous computing 5 (2001), 4--7."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3447242"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858528"},{"key":"e_1_2_1_22_1","unstructured":"Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe Charles Foster Jason Phang Horace He Anish Thite Noa Nabeshima et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 (2020)."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3381002"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3004294"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1080\/00140130903581623"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.3390\/en15124278"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2021.106914"},{"key":"e_1_2_1_28_1","volume-title":"Exploring smarter smart spaces with help from large language models. arXiv preprint arXiv:2303.14143","author":"King Evan","year":"2023","unstructured":"Evan King, Haoxiang Yu, Sangsu Lee, and Christine Julien. 2023. \"Get ready for a party\": Exploring smarter smart spaces with help from large language models. arXiv preprint arXiv:2303.14143 (2023)."},{"key":"e_1_2_1_29_1","volume-title":"Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226","author":"Kudo Taku","year":"2018","unstructured":"Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226 (2018)."},{"key":"e_1_2_1_30_1","volume-title":"Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753","author":"Liang Jacky","year":"2022","unstructured":"Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2022. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753 (2022)."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3560815"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858288"},{"key":"e_1_2_1_33_1","volume-title":"Seventh International Conference on Metering Apparatus and Tariffs for Electricity Supply","author":"Lutolf R.","year":"1992","unstructured":"R. Lutolf. 1992. Smart Home concept and the integration of energy meters into a home based system. In Seventh International Conference on Metering Apparatus and Tariffs for Electricity Supply 1992. 277--278."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-05412-9_34"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290607.3312975"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3410992.3410996"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-50578-3_13"},{"key":"e_1_2_1_38_1","unstructured":"OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]"},{"key":"e_1_2_1_39_1","unstructured":"OpenAI. 2023. tiktoken: a fast BPE tokeniser for use with OpenAI's models. https:\/\/github.com\/openai\/tiktoken."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10796-016-9670-x"},{"key":"e_1_2_1_41_1","volume-title":"Percy Liang, and Michael S Bernstein.","author":"Park Joon Sung","year":"2023","unstructured":"Joon Sung Park, Joseph C O'Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442 (2023)."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3373759"},{"key":"e_1_2_1_43_1","volume-title":"Reasoning with Language Model Prompting: A Survey. arXiv preprint arXiv:2212.09597","author":"Qiao Shuofei","year":"2022","unstructured":"Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. 2022. Reasoning with Language Model Prompting: A Survey. arXiv preprint arXiv:2212.09597 (2022)."},{"key":"e_1_2_1_44_1","unstructured":"Alec Radford Karthik Narasimhan Tim Salimans Ilya Sutskever et al. 2018. Improving language understanding by generative pre-training. (2018)."},{"key":"e_1_2_1_45_1","unstructured":"Alec Radford Jeffrey Wu Rewon Child David Luan Dario Amodei Ilya Sutskever et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1 8 (2019) 9."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.5555\/3455716.3455856"},{"key":"e_1_2_1_47_1","volume-title":"User requirements for the design of smart homes: dimensions and goals. Journal of Ambient Intelligence and Humanized Computing","author":"Reisinger Michaela R","year":"2022","unstructured":"Michaela R Reisinger, Sebastian Prost, Johann Schrammel, and Peter Fr\u00f6hlich. 2022. User requirements for the design of smart homes: dimensions and goals. Journal of Ambient Intelligence and Humanized Computing (2022), 1--20."},{"key":"e_1_2_1_48_1","volume-title":"Chi, Nathanael Sch\u00e4rli, and Denny Zhou","author":"Shi Freda","year":"2023","unstructured":"Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Sch\u00e4rli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. arXiv preprint arXiv:2302.00093 (2023)."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.techfore.2018.06.029"},{"key":"e_1_2_1_50_1","volume-title":"Kourosh Darvish, Al\u00e1n Aspuru-Guzik, Florian Shkurti, and Animesh Garg.","author":"Skreta Marta","year":"2023","unstructured":"Marta Skreta, Naruki Yoshikawa, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bj\u00f8rn Kristensen, Kourosh Darvish, Al\u00e1n Aspuru-Guzik, Florian Shkurti, and Animesh Garg. 2023. Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Prompting. arXiv preprint arXiv:2303.14100 (2023)."},{"key":"e_1_2_1_51_1","volume-title":"Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al.","author":"Thoppilan Romal","year":"2022","unstructured":"Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239 (2022)."},{"key":"e_1_2_1_52_1","volume-title":"Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971","author":"Touvron Hugo","year":"2023","unstructured":"Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth\u00e9e Lacroix, Baptiste Rozi\u00e8re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)."},{"key":"e_1_2_1_53_1","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3580925"},{"key":"e_1_2_1_55_1","volume-title":"Attention is all you need. Advances in neural information processing systems 30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.3115\/992424.992434"},{"key":"e_1_2_1_57_1","volume-title":"Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.","author":"Wei Jason","year":"2021","unstructured":"Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021)."},{"key":"e_1_2_1_58_1","volume-title":"Chi, Quoc Le, and Denny Zhou","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022)."},{"key":"e_1_2_1_59_1","volume-title":"The computer for the 21st century. ACM SIGMOBILE mobile computing and communications review 3, 3","author":"Weiser Mark","year":"1999","unstructured":"Mark Weiser. 1999. The computer for the 21st century. ACM SIGMOBILE mobile computing and communications review 3, 3 (1999), 3--11."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00779-014-0813-0"},{"key":"e_1_2_1_61_1","unstructured":"Jimmy Wu Rika Antonova Adam Kan Marion Lepert Andy Zeng Shuran Song Jeannette Bohg Szymon Rusinkiewicz and Thomas Funkhouser. 2023. TidyBot: Personalized Robot Assistance with Large Language Models. arXiv:2305.05658 [cs.RO]"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517582"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3485730.3494115"},{"key":"e_1_2_1_64_1","volume-title":"Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593","author":"Ziegler Daniel M","year":"2019","unstructured":"Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 (2019)."}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643505","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3643505","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T13:01:52Z","timestamp":1755867712000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643505"}},"subtitle":["Creative Goal-Oriented Reasoning in Smart Homes with Large Language Models"],"short-title":[],"issued":{"date-parts":[[2024,3,6]]},"references-count":64,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,3,6]]}},"alternative-id":["10.1145\/3643505"],"URL":"https:\/\/doi.org\/10.1145\/3643505","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,6]]},"assertion":[{"value":"2024-03-06","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}