{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T05:06:56Z","timestamp":1755839216842,"version":"3.41.2"},"reference-count":91,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2021,10,13]],"date-time":"2021-10-13T00:00:00Z","timestamp":1634083200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2021,10,13]]},"abstract":"<jats:p>Voice assistants offer users access to an increasing variety of personalized functionalities. Researchers and engineers who build these experiences rely on various signals from users to create the machine learning models powering them. One type of signal is explicit feedback. While collecting explicit user feedback in situ via voice assistants would help improve and inspect the underlying models, from a user perspective it can be disruptive to the overall experience, and the user might not feel compelled to respond. However, careful design can help alleviate the friction in the experience. In this paper, we explore the opportunities and the design space for voice assistant explicit feedback elicitation. First, we present four usage categories of explicit feedback in situ for model evaluation and improvement, derived from interviews with machine learning practitioners. Then, using realistic scenarios generated for each category, we conducted an online study to evaluate multiple voice assistant designs. Our results reveal that when the voice assistant is introduced as a learner or a collaborator, users were more willing to respond to its request for feedback and felt less disruptive. In addition, giving users instructions on how to initiate feedback themselves can reduce the perceived disruptiveness compared to asking users for feedback directly. Based on our findings, we discuss the implications and potential future directions for designing voice assistants to elicit user feedback for personalized voice experiences.<\/jats:p>","DOI":"10.1145\/3479532","type":"journal-article","created":{"date-parts":[[2021,10,19]],"date-time":"2021-10-19T02:32:07Z","timestamp":1634610727000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Let Me Ask You This: How Can a Voice Assistant Elicit Explicit User Feedback?"],"prefix":"10.1145","volume":"5","author":[{"given":"Ziang","family":"Xiao","sequence":"first","affiliation":[{"name":"University of Illinois at Urbana-Champaign, Urbana, IL, USA"}]},{"given":"Sarah","family":"Mennicken","sequence":"additional","affiliation":[{"name":"Spotify, San Francisco, CA, USA"}]},{"given":"Bernd","family":"Huber","sequence":"additional","affiliation":[{"name":"Spotify, Boston, MA, USA"}]},{"given":"Adam","family":"Shonkoff","sequence":"additional","affiliation":[{"name":"Spotify, Boston, MA, USA"}]},{"given":"Jennifer","family":"Thom","sequence":"additional","affiliation":[{"name":"Spotify, Boston, MA, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,10,18]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Fifteenth Symposium on Usable Privacy and Security ($$SOUPS$$","author":"Abdi Noura","year":"2019","unstructured":"Noura Abdi, Kopo M Ramokapane, and Jose M Such. 2019. More than smart speakers: security and privacy perceptions of smart home personal assistants. In Fifteenth Symposium on Usable Privacy and Security ($$SOUPS$$ 2019)."},{"doi-asserted-by":"publisher","key":"e_1_2_2_2_1","DOI":"10.1145\/3411764.3445122"},{"doi-asserted-by":"publisher","key":"e_1_2_2_3_1","DOI":"10.5555\/647985.743843"},{"volume-title":"Recommender systems handbook","author":"Adomavicius Gediminas","unstructured":"Gediminas Adomavicius and Alexander Tuzhilin. 2011. Context-aware recommender systems. In Recommender systems handbook. Springer, 217--253.","key":"e_1_2_2_4_1"},{"key":"e_1_2_2_5_1","first-page":"15","article-title":"Ethnobotany in the third millennium: expectations and unresolved issues","volume":"45","author":"Alexiades Miguel N","year":"2003","unstructured":"Miguel N Alexiades. 2003. Ethnobotany in the third millennium: expectations and unresolved issues. Delpinoa, Vol. 45, 1 (2003), 15--28.","journal-title":"Delpinoa"},{"doi-asserted-by":"publisher","key":"e_1_2_2_6_1","DOI":"10.1007\/978-3-642-02247-0_24"},{"doi-asserted-by":"publisher","key":"e_1_2_2_7_1","DOI":"10.1145\/3311956"},{"doi-asserted-by":"publisher","key":"e_1_2_2_8_1","DOI":"10.4108\/eai.14-10-2015.2261679"},{"doi-asserted-by":"publisher","key":"e_1_2_2_9_1","DOI":"10.1007\/978-3-642-23014-1_8"},{"doi-asserted-by":"publisher","key":"e_1_2_2_10_1","DOI":"10.1145\/3264901"},{"doi-asserted-by":"publisher","key":"e_1_2_2_11_1","DOI":"10.1145\/3290605.3300270"},{"key":"e_1_2_2_12_1","volume-title":"5th Multidisciplinary Workshop on Advances in Preference Handling (MPREF'10)","author":"Broekens Joost","year":"2010","unstructured":"Joost Broekens, Alina Pommeranz, Pascal Wiggers, and Catholijn M Jonker. 2010. Factors influencing user motivation for giving online preference feedback. In 5th Multidisciplinary Workshop on Advances in Preference Handling (MPREF'10)."},{"doi-asserted-by":"publisher","key":"e_1_2_2_13_1","DOI":"10.1145\/3359325"},{"doi-asserted-by":"publisher","key":"e_1_2_2_14_1","DOI":"10.1145\/3322276.3322298"},{"doi-asserted-by":"publisher","key":"e_1_2_2_15_1","DOI":"10.1145\/3290605.3300488"},{"doi-asserted-by":"publisher","key":"e_1_2_2_16_1","DOI":"10.1145\/3313831.3376551"},{"key":"e_1_2_2_17_1","volume-title":"James P Giangola, and Jennifer Balogh.","author":"Cohen Michael H","year":"2004","unstructured":"Michael H Cohen, Michael Harris Cohen, James P Giangola, and Jennifer Balogh. 2004. Voice user interface design .Addison-Wesley Professional."},{"key":"e_1_2_2_18_1","volume-title":"Proc. Surv. Res. Meth. Sect. Am. Statist. Ass. 3835--3840","author":"Conrad Frederick G","year":"2005","unstructured":"Frederick G Conrad, Mick P Couper, Roger Tourangeau, Mirta Galesic, and Ting Yan. 2005. Interactive feedback can improve the quality of responses in web surveys. In Proc. Surv. Res. Meth. Sect. Am. Statist. Ass. 3835--3840."},{"doi-asserted-by":"publisher","key":"e_1_2_2_19_1","DOI":"10.1145\/3098279.3098539"},{"doi-asserted-by":"publisher","key":"e_1_2_2_20_1","DOI":"10.5555\/1771592.1771617"},{"doi-asserted-by":"publisher","key":"e_1_2_2_21_1","DOI":"10.1016\/j.promfg.2015.07.514"},{"volume-title":"Scale development: Theory and applications","author":"DeVellis Robert F","unstructured":"Robert F DeVellis. 2016. Scale development: Theory and applications. Vol. 26. Sage publications.","key":"e_1_2_2_22_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_23_1","DOI":"10.1080\/10447318.2014.986642"},{"doi-asserted-by":"publisher","key":"e_1_2_2_24_1","DOI":"10.1177\/1029864911399497"},{"key":"e_1_2_2_25_1","volume-title":"Scott C Roesch, Beth G Chung-Herrera, Kristy Nadler, and Kelsey Bradshaw.","author":"Ehrhart Mark G","year":"2009","unstructured":"Mark G Ehrhart, Karen Holcombe Ehrhart, Scott C Roesch, Beth G Chung-Herrera, Kristy Nadler, and Kelsey Bradshaw. 2009. Testing the latent factor structure and construct validity of the Ten-Item Personality Inventory. Personality and individual Differences, Vol. 47, 8 (2009), 900--905."},{"volume-title":"Extending the linear model with R: generalized linear, mixed effects and nonparametric regression models","author":"Faraway Julian J","unstructured":"Julian J Faraway. 2016. Extending the linear model with R: generalized linear, mixed effects and nonparametric regression models .CRC press.","key":"e_1_2_2_26_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_27_1","DOI":"10.1145\/1247660.1247670"},{"doi-asserted-by":"publisher","key":"e_1_2_2_28_1","DOI":"10.1145\/3313831.3376651"},{"unstructured":"Shane Griffith Kaushik Subramanian Jonathan Scholz Charles L Isbell and Andrea L Thomaz. 2013. Policy shaping: Integrating human feedback with reinforcement learning. In Advances in neural information processing systems. 2625--2633.","key":"e_1_2_2_29_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_30_1","DOI":"10.1007\/11527886_40"},{"doi-asserted-by":"publisher","key":"e_1_2_2_31_1","DOI":"10.1037\/0021-9010.64.4.349"},{"doi-asserted-by":"publisher","key":"e_1_2_2_32_1","DOI":"10.1145\/3287048"},{"doi-asserted-by":"publisher","key":"e_1_2_2_33_1","DOI":"10.1145\/1869446.1869453"},{"doi-asserted-by":"publisher","key":"e_1_2_2_34_1","DOI":"10.1145\/3130332.3130334"},{"doi-asserted-by":"publisher","key":"e_1_2_2_35_1","DOI":"10.1145\/3109859.3109873"},{"key":"e_1_2_2_36_1","volume-title":"Conceptual Metaphors Impact Perceptions of Human-AI Collaboration. arXiv preprint arXiv:2008.02311","author":"Khadpe Pranav","year":"2020","unstructured":"Pranav Khadpe, Ranjay Krishna, Li Fei-Fei, Jeffrey Hancock, and Michael Bernstein. 2020. Conceptual Metaphors Impact Perceptions of Human-AI Collaboration. arXiv preprint arXiv:2008.02311 (2020)."},{"key":"e_1_2_2_37_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3381009","article-title":"Interruptibility for In-vehicle Multitasking: Influence of Voice Task Demands and Adaptive Behaviors","volume":"4","author":"Kim Auk","year":"2020","unstructured":"Auk Kim, Jung-Mi Park, and Uichin Lee. 2020. Interruptibility for In-vehicle Multitasking: Influence of Voice Task Demands and Adaptive Behaviors. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Vol. 4, 1 (2020), 1--22.","journal-title":"Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"},{"doi-asserted-by":"publisher","key":"e_1_2_2_38_1","DOI":"10.1145\/2911451.2914754"},{"doi-asserted-by":"publisher","key":"e_1_2_2_39_1","DOI":"10.1007\/978-3-642-01516-8_13"},{"doi-asserted-by":"publisher","key":"e_1_2_2_40_1","DOI":"10.1177\/0146167206286627"},{"doi-asserted-by":"publisher","key":"e_1_2_2_41_1","DOI":"10.1145\/1557019.1557072"},{"doi-asserted-by":"publisher","key":"e_1_2_2_42_1","DOI":"10.1109\/ROMAN.2009.5326292"},{"doi-asserted-by":"publisher","key":"e_1_2_2_43_1","DOI":"10.1145\/3342775.3342789"},{"doi-asserted-by":"publisher","key":"e_1_2_2_44_1","DOI":"10.1145\/3274371"},{"doi-asserted-by":"publisher","key":"e_1_2_2_45_1","DOI":"10.1145\/3427228.3427250"},{"doi-asserted-by":"publisher","key":"e_1_2_2_46_1","DOI":"10.1016\/j.dss.2015.03.008"},{"doi-asserted-by":"publisher","key":"e_1_2_2_47_1","DOI":"10.1016\/j.chb.2014.04.043"},{"doi-asserted-by":"publisher","key":"e_1_2_2_48_1","DOI":"10.1145\/2858036.2858288"},{"doi-asserted-by":"publisher","key":"e_1_2_2_49_1","DOI":"10.1145\/1639950.1640068"},{"doi-asserted-by":"publisher","key":"e_1_2_2_50_1","DOI":"10.1007\/s12599-019-00600-8"},{"doi-asserted-by":"publisher","key":"e_1_2_2_51_1","DOI":"10.1145\/2702123.2702539"},{"unstructured":"Benjamin M Marlin. 2004. Modeling user rating profiles for collaborative filtering. In Advances in neural information processing systems. 627--634.","key":"e_1_2_2_52_1"},{"key":"e_1_2_2_53_1","volume-title":"1st International Workshop on Vocal Interactivity in-and-between Humans, Animals and Robots.","author":"Moore Roger K","year":"2017","unstructured":"Roger K Moore. 2017a. Appropriate voices for artefacts: some key insights. In 1st International Workshop on Vocal Interactivity in-and-between Humans, Animals and Robots."},{"volume-title":"Dialogues with Social Robots","author":"Moore Roger K","unstructured":"Roger K Moore. 2017b. Is spoken language all-or-nothing? Implications for future speech-based human-machine interaction. In Dialogues with Social Robots. Springer, 281--291.","key":"e_1_2_2_54_1"},{"key":"e_1_2_2_55_1","volume-title":"Age differences in technology adoption decisions: Implications for a changing work force. Personnel psychology","author":"Morris Michael G","year":"2000","unstructured":"Michael G Morris and Viswanath Venkatesh. 2000. Age differences in technology adoption decisions: Implications for a changing work force. Personnel psychology, Vol. 53, 2 (2000), 375--403."},{"doi-asserted-by":"publisher","key":"e_1_2_2_56_1","DOI":"10.1145\/3290605.3300277"},{"doi-asserted-by":"publisher","key":"e_1_2_2_57_1","DOI":"10.1006\/ijhc.1996.0073"},{"doi-asserted-by":"publisher","key":"e_1_2_2_58_1","DOI":"10.1145\/191666.191703"},{"unstructured":"David Nichols. 1998. Implicit rating and filtering. ERCIM.","key":"e_1_2_2_59_1"},{"unstructured":"Derry O'Sullivan Barry Smyth and David Wilson. 2003. Explicit vs implicit profiling-a case-study in electronic programme guides. In IJCAI. 1351.","key":"e_1_2_2_60_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_61_1","DOI":"10.1007\/978-3-642-40585-3_30"},{"doi-asserted-by":"publisher","key":"e_1_2_2_62_1","DOI":"10.1007\/11853565_8"},{"volume-title":"Designing voice user interfaces: principles of conversational experiences .\" O'Reilly Media","author":"Pearl Cathy","unstructured":"Cathy Pearl. 2016. Designing voice user interfaces: principles of conversational experiences .\" O'Reilly Media, Inc.\".","key":"e_1_2_2_63_1"},{"volume-title":"Emotions and personality in personalized services","author":"Pejovic Veljko","unstructured":"Veljko Pejovic, Neal Lathia, Cecilia Mascolo, and Mirco Musolesi. 2016. Mobile-based experience sampling for behaviour research. In Emotions and personality in personalized services. Springer, 141--161.","key":"e_1_2_2_64_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_65_1","DOI":"10.1007\/978-3-642-38844-6_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_66_1","DOI":"10.1007\/s11257-011-9116-6"},{"doi-asserted-by":"publisher","key":"e_1_2_2_67_1","DOI":"10.1145\/3173574.3174214"},{"doi-asserted-by":"crossref","unstructured":"Stephen R Porter. 2004. Overcoming survey research problems. (2004).","key":"e_1_2_2_68_1","DOI":"10.1002\/ir.101"},{"doi-asserted-by":"publisher","key":"e_1_2_2_69_1","DOI":"10.1145\/2043932.2043962"},{"doi-asserted-by":"publisher","key":"e_1_2_2_70_1","DOI":"10.1145\/1864349.1864393"},{"doi-asserted-by":"publisher","key":"e_1_2_2_71_1","DOI":"10.1145\/192844.192905"},{"key":"e_1_2_2_72_1","volume-title":"Diener","author":"Scollon Christie Napa","year":"2009","unstructured":"Christie Napa Scollon, Chu-Kim Prieto, and Ed Diener. 2009. Experience sampling: promises and pitfalls, strength and weaknesses. In Assessing well-being. Springer, 157--180."},{"doi-asserted-by":"publisher","key":"e_1_2_2_73_1","DOI":"10.1007\/s11251-011-9166-5"},{"doi-asserted-by":"publisher","key":"e_1_2_2_74_1","DOI":"10.1145\/3173574.3174178"},{"key":"e_1_2_2_75_1","volume-title":"AMIA Annual Symposium Proceedings","volume":"2009","author":"Tao Donghua","year":"2009","unstructured":"Donghua Tao. 2009. Intention to use and actual use of electronic information resources: further exploring Technology Acceptance Model (TAM). In AMIA Annual Symposium Proceedings, Vol. 2009. American Medical Informatics Association, 629."},{"unstructured":"Jenn Thom Angela Nazarian Ruth Brillman Henriette Cramer and Sarah Mennicken. 2020. \"Play Music\": User Motivations and Expectations for Non-Specific Voice Queries. (2020).","key":"e_1_2_2_76_1"},{"volume-title":"Recommender systems handbook","author":"Tintarev Nava","unstructured":"Nava Tintarev and Judith Masthoff. 2011. Designing and evaluating explanations for recommender systems. In Recommender systems handbook. Springer, 479--510.","key":"e_1_2_2_77_1"},{"key":"e_1_2_2_78_1","volume-title":"Computational approaches to dialogue","author":"Traum David","year":"2017","unstructured":"David Traum. 2017. Computational approaches to dialogue. The Routledge Handbook of Language and Dialogue. Taylor & Francis (2017), 143--161."},{"doi-asserted-by":"publisher","key":"e_1_2_2_79_1","DOI":"10.1145\/3301275.3302318"},{"doi-asserted-by":"publisher","key":"e_1_2_2_80_1","DOI":"10.1016\/S0968-090X(96)00025-3"},{"doi-asserted-by":"publisher","key":"e_1_2_2_81_1","DOI":"10.1145\/2213836.2213951"},{"key":"e_1_2_2_82_1","volume-title":"A longitudinal field investigation of gender differences in individual technology adoption decision-making processes. Organizational behavior and human decision processes","author":"Venkatesh Viswanath","year":"2000","unstructured":"Viswanath Venkatesh, Michael G Morris, and Phillip L Ackerman. 2000. A longitudinal field investigation of gender differences in individual technology adoption decision-making processes. Organizational behavior and human decision processes, Vol. 83, 1 (2000), 33--60."},{"key":"e_1_2_2_83_1","volume-title":"Refining the test phase of usability evaluation: How many subjects is enough? Human factors","author":"Virzi Robert A","year":"1992","unstructured":"Robert A Virzi. 1992. Refining the test phase of usability evaluation: How many subjects is enough? Human factors, Vol. 34, 4 (1992), 457--468."},{"doi-asserted-by":"publisher","key":"e_1_2_2_84_1","DOI":"10.1145\/3313831.3376493"},{"unstructured":"Jun Xiao Richard Catrambone and John T Stasko. 2003. Be quiet? evaluating proactive and reactive user interface assistants. Technical Report. Georgia Institute of Technology.","key":"e_1_2_2_85_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_86_1","DOI":"10.1145\/3301275.3302264"},{"doi-asserted-by":"publisher","key":"e_1_2_2_87_1","DOI":"10.1145\/3381804"},{"doi-asserted-by":"publisher","key":"e_1_2_2_88_1","DOI":"10.1109\/CEC.2009.84"},{"doi-asserted-by":"publisher","key":"e_1_2_2_89_1","DOI":"10.1145\/3172944.3172998"},{"volume-title":"Cognitive informatics for biomedicine","author":"Zheng Kai","unstructured":"Kai Zheng, David A Hanauer, Nadir Weibel, and Zia Agha. 2015. Computational ethnography: automated and unobtrusive means for collecting data in situ for human-computer interaction evaluation studies. In Cognitive informatics for biomedicine. Springer, 111--140.","key":"e_1_2_2_90_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_91_1","DOI":"10.1145\/3018661.3018665"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3479532","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3479532","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,14]],"date-time":"2025-07-14T05:01:50Z","timestamp":1752469310000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3479532"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,13]]},"references-count":91,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2021,10,13]]}},"alternative-id":["10.1145\/3479532"],"URL":"https:\/\/doi.org\/10.1145\/3479532","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2021,10,13]]},"assertion":[{"value":"2021-10-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}