{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,15]],"date-time":"2026-04-15T11:14:23Z","timestamp":1776251663962,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":59,"publisher":"ACM","funder":[{"name":"The Hong Kong Polytechnic University","award":["A0056446"],"award-info":[{"award-number":["A0056446"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,4,13]]},"DOI":"10.1145\/3772318.3791653","type":"proceedings-article","created":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T09:47:11Z","timestamp":1776073631000},"page":"1-19","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Agentic Audio Moderator vs Human Moderator in Think-Aloud Usability Testing: Results from a Randomized Controlled Trial"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9611-4800","authenticated-orcid":false,"given":"Wangda","family":"Zhu","sequence":"first","affiliation":[{"name":"School of Design, The Hong Kong Polytechnic University, Hong Kong, Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-2738-4129","authenticated-orcid":false,"given":"Guang","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Design, The Hong Kong Polytechnic University, Hong Kong, Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-6487-5950","authenticated-orcid":false,"given":"Yao","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7705-2031","authenticated-orcid":false,"given":"Pengcheng","family":"An","sequence":"additional","affiliation":[{"name":"School of Design, Southern University of Science and Technology, Shenzhen, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-4396-097X","authenticated-orcid":false,"given":"Jiachun","family":"Du","sequence":"additional","affiliation":[{"name":"Pettichat Technology, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3782-0737","authenticated-orcid":false,"given":"Chen","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Applied Social Sciences, The Hong Kong Polytechnic University, Hong Kong, Hong Kong and Department of Computing, The Hong Kong Polytechnic University, Hong Kong, Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2026,4,13]]},"reference":[{"key":"e_1_3_3_2_2_2","doi-asserted-by":"publisher","unstructured":"Tahir Abbas Ujwal Gadiraju Vassilis-Javed Khan and Panos Markopoulos. 2022. Understanding user perceptions of response delays in crowd-powered conversational systems. Proceedings of the ACM on Human-Computer Interaction 6 CSCW2 (2022) 1\u201342. 10.1145\/3555765","DOI":"10.1145\/3555765"},{"key":"e_1_3_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/SAI.2015.7237139"},{"key":"e_1_3_3_2_4_2","doi-asserted-by":"publisher","unstructured":"Terry Anderson and Julie Shattuck. 2012. Design-based research: A decade of progress in education research?Educational researcher 41 1 (2012) 16\u201325. 10.3102\/0013189X11428813","DOI":"10.3102\/0013189X11428813"},{"key":"e_1_3_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/1240624.1240838"},{"key":"e_1_3_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-89738-3_24-1"},{"key":"e_1_3_3_2_7_2","doi-asserted-by":"publisher","unstructured":"J.M.\u00a0Christian Bastien. 2010. Usability testing: a review of some methodological and technical aspects of the method. International Journal of Medical Informatics 79 4 (2010) e18\u2013e23. 10.1016\/j.ijmedinf.2008.12.004Human Factors Engineering for Healthcare Applications Special Issue.","DOI":"10.1016\/j.ijmedinf.2008.12.004"},{"key":"e_1_3_3_2_8_2","doi-asserted-by":"publisher","unstructured":"Pronaya Bhattacharya Vivek\u00a0Kumar Prasad Ashwin Verma Deepak Gupta Assadaporn Sapsomboon Wattana Viriyasitavat and Gaurav Dhiman. 2024. Demystifying chatgpt: An in-depth survey of openai\u2019s robust large language models. Archives of Computational Methods in Engineering 31 8 (2024) 4557\u20134600. 10.1007\/s11831-024-10115-5","DOI":"10.1007\/s11831-024-10115-5"},{"key":"e_1_3_3_2_9_2","doi-asserted-by":"publisher","unstructured":"Frank Biocca Chad Harms and Judee\u00a0K Burgoon. 2003. Toward a more robust theory and measure of social presence: Review and suggested criteria. Presence: Teleoperators & Virtual Environments 12 5 (2003) 456\u2013480. 10.1162\/105474603322761270","DOI":"10.1162\/105474603322761270"},{"key":"e_1_3_3_2_10_2","doi-asserted-by":"publisher","unstructured":"Ted Boren and Judith Ramey. 2000. Thinking aloud: Reconciling theory and practice. IEEE transactions on professional communication 43 3 (2000) 261\u2013278. 10.1109\/47.867942","DOI":"10.1109\/47.867942"},{"key":"e_1_3_3_2_11_2","doi-asserted-by":"publisher","unstructured":"Ann\u00a0L Brown. 1992. Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The journal of the learning sciences 2 2 (1992) 141\u2013178. 10.1207\/s15327809jls0202_2","DOI":"10.1207\/s15327809jls0202_2"},{"key":"e_1_3_3_2_12_2","unstructured":"Mert Cemri Melissa\u00a0Z Pan Shuyi Yang Lakshya\u00a0A Agrawal Bhavya Chopra Rishabh Tiwari Kurt Keutzer Aditya Parameswaran Dan Klein Kannan Ramchandran Matei Zaharia Joseph\u00a0E. Gonzalez and Ion Stoica. 2025. Why Do Multi-Agent LLM Systems Fail?ArXiv abs\/2503.13657 (2025)."},{"key":"e_1_3_3_2_13_2","doi-asserted-by":"publisher","unstructured":"Lara Christoforakos and Sarah Diefenbach. 2023. Technology as a social companion? An exploration of individual and product-related factors of anthropomorphism. Social Science Computer Review 41 3 (2023) 1039\u20131062. 10.1177\/08944393211065867","DOI":"10.1177\/08944393211065867"},{"key":"e_1_3_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-77750-9_2"},{"key":"e_1_3_3_2_15_2","doi-asserted-by":"publisher","unstructured":"Giuseppe Desolda Rosa Lanzilotti Danilo Caivano Maria\u00a0Francesca Costabile and Paolo Buono. 2023. Asynchronous remote usability tests using web-based tools versus laboratory usability tests: an experimental study. IEEE Transactions on Human-Machine Systems 53 4 (2023) 731\u2013742. 10.1109\/THMS.2023.3282225","DOI":"10.1109\/THMS.2023.3282225"},{"key":"e_1_3_3_2_16_2","doi-asserted-by":"publisher","unstructured":"Giuseppe Desolda Rosa Lanzilotti Danilo Caivano Maria\u00a0Francesca Costabile and Paolo Buono. 2023. Asynchronous Remote Usability Tests Using Web-Based Tools Versus Laboratory Usability Tests: An Experimental Study. IEEE Transactions on Human-Machine Systems 53 4 (2023) 731\u2013742. 10.1109\/THMS.2023.3282225","DOI":"10.1109\/THMS.2023.3282225"},{"key":"e_1_3_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1002\/9781405164535.ch33"},{"key":"e_1_3_3_2_18_2","doi-asserted-by":"publisher","unstructured":"Mingming Fan Jinglan Lin Christina Chung and Khai\u00a0N Truong. 2019. Concurrent think-aloud verbalizations and usability problems. ACM Transactions on Computer-Human Interaction (TOCHI) 26 5 (2019) 1\u201335. 10.1145\/3325281","DOI":"10.1145\/3325281"},{"key":"e_1_3_3_2_19_2","doi-asserted-by":"publisher","unstructured":"Mingming Fan Vinita Tibdewal Qiwen Zhao Lizhou Cao Chao Peng Runxuan Shu and Yujia Shan. 2022. Older adults\u2019 concurrent and retrospective think-aloud verbalizations for identifying user experience problems of VR games. Interacting with Computers 34 4 (2022) 99\u2013115. 10.1093\/iwc\/iwac039","DOI":"10.1093\/iwc\/iwac039"},{"key":"e_1_3_3_2_20_2","doi-asserted-by":"publisher","unstructured":"Jasper Feine Ulrich Gnewuch Stefan Morana and Alexander Maedche. 2019. A taxonomy of social cues for conversational agents. International Journal of human-computer studies 132 (2019) 138\u2013161. 10.1016\/j.ijhcs.2019.07.009","DOI":"10.1016\/j.ijhcs.2019.07.009"},{"key":"e_1_3_3_2_21_2","first-page":"113","volume-title":"European Conference on Information Systems","author":"Gnewuch Ulrich","year":"2018","unstructured":"Ulrich Gnewuch, Stefan Morana, Marc Thomas\u00a0Philipp Adam, and Alexander Maedche. 2018. Faster is Not Always Better: Understanding the Effect of Dynamic Response Delays in Human-Chatbot Interaction. In European Conference on Information Systems. 113. https:\/\/api.semanticscholar.org\/CorpusID:56147085"},{"key":"e_1_3_3_2_22_2","doi-asserted-by":"publisher","unstructured":"Alberto Greco Gaetano Valenza Antonio Lanata Enzo\u00a0Pasquale Scilingo and Luca Citi. 2015. cvxEDA: A convex optimization approach to electrodermal activity processing. IEEE transactions on biomedical engineering 63 4 (2015) 797\u2013804. 10.1109\/TBME.2015.2474131","DOI":"10.1109\/TBME.2015.2474131"},{"key":"e_1_3_3_2_23_2","doi-asserted-by":"publisher","unstructured":"H\u00a0Rex Hartson Terence\u00a0S Andre and Robert\u00a0C Williges. 2001. Criteria for evaluating usability evaluation methods. International journal of human-computer interaction 13 4 (2001) 373\u2013410. 10.1207\/S15327590IJHC1501_13","DOI":"10.1207\/S15327590IJHC1501_13"},{"key":"e_1_3_3_2_24_2","doi-asserted-by":"publisher","unstructured":"Marc Hassenzahl and Noam Tractinsky. 2006. User experience-a research agenda. Behaviour & information technology 25 2 (2006) 91\u201397. 10.1080\/01449290500330331","DOI":"10.1080\/01449290500330331"},{"key":"e_1_3_3_2_25_2","doi-asserted-by":"publisher","unstructured":"Morten Hertzum. 2024. Concurrent or Retrospective Thinking Aloud in Usability Tests: A Meta-Analytic Review. ACM Transactions on Computer-Human Interaction 31 3 (2024) 1\u201329. 10.1145\/3665327","DOI":"10.1145\/3665327"},{"key":"e_1_3_3_2_26_2","doi-asserted-by":"publisher","unstructured":"Morten Hertzum Pia Borlund and Kristina\u00a0B Kristoffersen. 2015. What do thinking-aloud participants say? A comparison of moderated and unmoderated usability sessions. International Journal of Human-Computer Interaction 31 9 (2015) 557\u2013570. 10.1080\/10447318.2015.1065691","DOI":"10.1080\/10447318.2015.1065691"},{"key":"e_1_3_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-23204-7_14"},{"key":"e_1_3_3_2_28_2","unstructured":"BSEN ISO. 2018. 9241-11: 2018 Ergonomics of human-system interaction. Usability: Definitions and concepts. British Standards Institution (2018)."},{"key":"e_1_3_3_2_29_2","doi-asserted-by":"publisher","unstructured":"Adam\u00a0N Joinson. 2001. Self-disclosure in computer-mediated communication: The role of self-awareness and visual anonymity. European journal of social psychology 31 2 (2001) 177\u2013192. 10.1002\/ejsp.36","DOI":"10.1002\/ejsp.36"},{"key":"e_1_3_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3706598.3713728"},{"key":"e_1_3_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-05311-5_16"},{"key":"e_1_3_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642168"},{"key":"e_1_3_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-61688-4_4"},{"key":"e_1_3_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-96-4016-4_10"},{"key":"e_1_3_3_2_35_2","doi-asserted-by":"publisher","unstructured":"Jia Liu. 2025. AI in Automated and Remote UX Evaluation: A Systematic Review (2014\u20132024). Advances in Human-Computer Interaction 2025 1 (2025) 7442179. 10.1155\/ahci\/7442179","DOI":"10.1155\/ahci\/7442179"},{"key":"e_1_3_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3711670.3764614"},{"key":"e_1_3_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1177\/2327857925141037"},{"key":"e_1_3_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858288"},{"key":"e_1_3_3_2_39_2","doi-asserted-by":"publisher","unstructured":"Kapil\u00a0Chalil Madathil and Joel\u00a0S Greenstein. 2017. An investigation of the efficacy of collaborative virtual reality systems for moderated remote usability testing. Applied ergonomics 65 (2017) 501\u2013514. 10.1016\/j.apergo.2017.02.011","DOI":"10.1016\/j.apergo.2017.02.011"},{"key":"e_1_3_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3605098.3636131"},{"key":"e_1_3_3_2_41_2","doi-asserted-by":"publisher","unstructured":"Graeme McLean and Kofi Osei-Frimpong. 2019. Hey Alexa\u2026 examine the variables influencing the use of artificial intelligent in-home voice assistants. Computers in human behavior 99 (2019) 28\u201337. 10.1016\/j.chb.2019.05.009","DOI":"10.1016\/j.chb.2019.05.009"},{"key":"e_1_3_3_2_42_2","doi-asserted-by":"publisher","unstructured":"Michael Meehan Sharif Razzaque Brent Insko Mary Whitton and Frederick\u00a0P. Brooks\u00a0Jr. 2002. Review of four studies on the use of physiological reaction as a measure of presence in stressful virtual environments. Applied Psychophysiology and Biofeedback 27 3 (2002) 243\u2013261. 10.1023\/A:1023413717485","DOI":"10.1023\/A:1023413717485"},{"key":"e_1_3_3_2_43_2","unstructured":"Matthew\u00a0R Miller. 2025. Timing matters: effects of response delay on perceived naturalness in robot conversations. Master\u2019s thesis. Oregon State University."},{"key":"e_1_3_3_2_44_2","doi-asserted-by":"publisher","unstructured":"Clifford Nass and Young Moon. 2000. Machines and mindlessness: Social responses to computers. Journal of Social Issues 56 1 (2000) 81\u2013103. 10.1111\/0022-4537.00153","DOI":"10.1111\/0022-4537.00153"},{"key":"e_1_3_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.5555\/2821575"},{"key":"e_1_3_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/97243.97281"},{"key":"e_1_3_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753685"},{"key":"e_1_3_3_2_48_2","doi-asserted-by":"publisher","unstructured":"Sara Pearsell and Daniel Pape. 2023. The effects of different voice qualities on the perceived personality of a speaker. Frontiers in communication 7 (2023) 909427. 10.3389\/fcomm.2022.909427","DOI":"10.3389\/fcomm.2022.909427"},{"key":"e_1_3_3_2_49_2","doi-asserted-by":"publisher","unstructured":"Sarah\u00a0A Powers and Mark\u00a0W Scerbo. 2023. Examining the effect of interruptions at different breakpoints and frequencies within a task. Human Factors 65 1 (2023) 22\u201336. 10.1177\/00187208211009010","DOI":"10.1177\/00187208211009010"},{"key":"e_1_3_3_2_50_2","volume-title":"Handbook of usability testing: How to plan, design, and conduct effective tests","author":"Rubin Jeffrey","year":"2008","unstructured":"Jeffrey Rubin and Dana Chisnell. 2008. Handbook of usability testing: How to plan, design, and conduct effective tests. John Wiley & Sons."},{"key":"e_1_3_3_2_51_2","doi-asserted-by":"publisher","unstructured":"Noah\u00a0L Schroeder Erin\u00a0K Chiou and Scotty\u00a0D Craig. 2021. Trust influences perceptions of virtual humans but not necessarily learning. Computers & Education 160 (2021) 104039. 10.1016\/j.compedu.2020.104039","DOI":"10.1016\/j.compedu.2020.104039"},{"key":"e_1_3_3_2_52_2","doi-asserted-by":"publisher","unstructured":"Ben Sheehan Hyun\u00a0Seung Jin Brett Martin and Hyoje\u00a0Jay Kim. 2024. Wow! Interjections improve chatbot performance: the mediating role of anthropomorphism and perceived listening. Communication Research 51 7 (2024) 891\u2013917. 10.1177\/00936502241273259","DOI":"10.1177\/00936502241273259"},{"key":"e_1_3_3_2_53_2","doi-asserted-by":"publisher","unstructured":"Donghee Shin and Ali Ahsan. 2025. How do people decide whether to trust information from generative AI? The effects of transparency on trust in and evaluation of a generated content. Journal of Media Business Studies 0 0 (2025) 1\u201324. 10.1080\/16522354.2025.2509060","DOI":"10.1080\/16522354.2025.2509060"},{"key":"e_1_3_3_2_54_2","doi-asserted-by":"publisher","unstructured":"John Short Ederyn Williams and Bruce Christie. 1976. The social psychology of telecommunications. Contemporary Sociology 7 1 (1976) 32. 10.2307\/2065899","DOI":"10.2307\/2065899"},{"key":"e_1_3_3_2_55_2","doi-asserted-by":"publisher","unstructured":"Richard Skarbez Frederick\u00a0P. Brooks\u00a0Jr and Mary\u00a0C. Whitton. 2017. A survey of presence and related concepts. Comput. Surveys 50 6 (2017) 96:1\u201396:39. 10.1145\/3134301","DOI":"10.1145\/3134301"},{"key":"e_1_3_3_2_56_2","doi-asserted-by":"publisher","unstructured":"Federica Spaccatini Giulia Corlito and Simona Sacchi. 2023. New dyads? The effect of social robots\u2019 anthropomorphization on empathy towards human beings. Computers in Human Behavior 146 (2023) 107821. 10.1016\/j.chb.2023.107821","DOI":"10.1016\/j.chb.2023.107821"},{"key":"e_1_3_3_2_57_2","doi-asserted-by":"publisher","unstructured":"Andr\u00e9\u00a0J Szameitat Jan Rummel Diana\u00a0P Szameitat and Annette Sterr. 2009. Behavioral and emotional consequences of brief delays in human\u2013computer interaction. International Journal of Human-Computer Studies 67 7 (2009) 561\u2013570. 10.1016\/j.ijhcs.2009.02.004","DOI":"10.1016\/j.ijhcs.2009.02.004"},{"key":"e_1_3_3_2_58_2","doi-asserted-by":"publisher","unstructured":"David\u00a0R. Thomas. 2006. A general inductive approach for analyzing qualitative evaluation data. AMERICAN JOURNAL OF EVALUATION 27 2 (JUN 2006) 237\u2013246. 10.1177\/1098214005283748","DOI":"10.1177\/1098214005283748"},{"key":"e_1_3_3_2_59_2","doi-asserted-by":"publisher","unstructured":"Hilde Voorveld Andreas Panteli Yoni Schirris Carolin Ischen Evangelos Kanoulas and Tom Lentz. 2025. Examining the persuasiveness of text and voice agents: prosody aligned with information structure increases human-likeness perceived personalisation and brand attitude. Behaviour & Information Technology 44 12 (2025) 2913\u20132928. 10.1080\/0144929x.2024.2420871","DOI":"10.1080\/0144929x.2024.2420871"},{"key":"e_1_3_3_2_60_2","doi-asserted-by":"publisher","unstructured":"Ya-Ling Wang and Chi-Wen Lo. 2025. The effects of response time on older and young adults\u2019 interaction experience with Chatbot. BMC psychology 13 1 (2025) 150. 10.1186\/s40359-025-02459-9","DOI":"10.1186\/s40359-025-02459-9"}],"event":{"name":"CHI 2026: CHI Conference on Human Factors in Computing Systems","location":"Barcelona Spain","acronym":"CHI '26","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3772318.3791653","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,15]],"date-time":"2026-04-15T10:36:24Z","timestamp":1776249384000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3772318.3791653"}},"subtitle":["Results from a Randomized Controlled Trial"],"short-title":[],"issued":{"date-parts":[[2026,4,13]]},"references-count":59,"alternative-id":["10.1145\/3772318.3791653","10.1145\/3772318"],"URL":"https:\/\/doi.org\/10.1145\/3772318.3791653","relation":{},"subject":[],"published":{"date-parts":[[2026,4,13]]},"assertion":[{"value":"2026-04-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}