{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T22:52:50Z","timestamp":1776120770952,"version":"3.50.1"},"reference-count":70,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2023,4,14]],"date-time":"2023-04-14T00:00:00Z","timestamp":1681430400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Hong Kong Research Grants Council - ECS","award":["CityU 21209419"],"award-info":[{"award-number":["CityU 21209419"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2023,4,14]]},"abstract":"<jats:p>Wizard of Oz (WoZ) as a prototyping method has been used to simulate intelligent user interfaces, particularly for speech-based systems. However, as our societies' expectations on artificial intelligence (AI) grows, the question remains whether a single Wizard is sufficient for it to simulate smarter systems and more complex interactions. Optimistic visions of 'what artificial intelligence (AI) can do' places demands on WoZ platforms to simulate smarter systems and more complex interactions. This raises the question of whether the typical approach of employing a single Wizard is sufficient. Moreover, while existing work has employed multiple Wizards in WoZ studies, a multi-Wizard approach has not been systematically studied in terms of feasibility, effectiveness, and challenges. We offer Wizundry, a real-time, web-based WoZ platform that allows multiple Wizards to collaboratively operate a speech-to-text based system remotely. We outline the design and technical specifications of our open-source platform, which we iterated over two design phases. We report on two studies in which participant-Wizards were tasked with negotiating how to cooperatively simulate an interface that can handle natural speech for dictation and text editing as well as other intelligent text processing tasks. We offer qualitative findings on the Multi-Wizard experience for Dyads and Triads of Wizards. Our findings reveal the promises and challenges of the multi-Wizard approach and open up new research questions.<\/jats:p>","DOI":"10.1145\/3579591","type":"journal-article","created":{"date-parts":[[2023,4,16]],"date-time":"2023-04-16T17:23:07Z","timestamp":1681665787000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Wizundry: A Cooperative Wizard of Oz Platform for Simulating Future Speech-based Interfaces with Multiple Wizards"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3824-2801","authenticated-orcid":false,"given":"Siying","family":"Hu","sequence":"first","affiliation":[{"name":"City University of Hong Kong, HongKong, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8212-4100","authenticated-orcid":false,"given":"Hen Chen","family":"Yen","sequence":"additional","affiliation":[{"name":"City University of Hong Kong &amp; University of Waterloo, Hong Kong, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9007-8906","authenticated-orcid":false,"given":"Ziwei","family":"Yu","sequence":"additional","affiliation":[{"name":"City University of Hong Kong &amp; Honor Device Co Ltd, Hong Kong, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1183-0857","authenticated-orcid":false,"given":"Mingjian","family":"Zhao","sequence":"additional","affiliation":[{"name":"City University of Hong Kong &amp; Washington University in St. Louis, Hong Kong, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7812-9096","authenticated-orcid":false,"given":"Katie","family":"Seaborn","sequence":"additional","affiliation":[{"name":"Tokyo Institute of Technology, Tokyo, Japan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3267-3317","authenticated-orcid":false,"given":"Can","family":"Liu","sequence":"additional","affiliation":[{"name":"City University of Hong Kong, Hong Kong, China"}]}],"member":"320","published-online":{"date-parts":[[2023,4,16]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/CMPEUR.1989.93377"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3343413.3377941"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_2_1_4_1","volume-title":"et al","author":"Bonial Claire","year":"2017","unstructured":"Claire Bonial, Matthew Marge, Ashley Foots, Felix Gervits, Cory J Hayes, Cassidy Henry, Susan G Hill, Anton Leuski, Stephanie M Lukin, Pooja Moolchandani, et al . 2017. Laying down the yellow brick road: Development of a wizard-of-oz interface for collecting human-robot dialogue. arXiv preprint arXiv:1710.06406 (2017)."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300270"},{"key":"e_1_2_1_6_1","volume-title":"MultiWOZ--A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling. arXiv preprint arXiv:1810.00278","author":"Budzianowski Pawel","year":"2018","unstructured":"Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Ga?ic. 2018. MultiWOZ--A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling. arXiv preprint arXiv:1810.00278 (2018)."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3405755.3406148"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.naacl-main.33"},{"key":"e_1_2_1_9_1","volume-title":"Natural language processing. Annual review of information science and technology 37, 1","author":"Chowdhury Gobinda G","year":"2003","unstructured":"Gobinda G Chowdhury. 2003. Natural language processing. Annual review of information science and technology 37, 1 (2003), 51--89."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1452392.1452419"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/257089.257904"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/0950--7051(93)90017-N"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/948496.948519"},{"key":"e_1_2_1_14_1","volume-title":"People and Computers V: Proceedings of the Fifth Conference of the British Computer Society","volume":"5","author":"Diaper Dan","year":"1989","unstructured":"Dan Diaper. 1989. The Wizard's Apprentice: A Program to Help Analyse Natural Language. In People and Computers V: Proceedings of the Fifth Conference of the British Computer Society, Vol. 5. Cambridge University Press, 231."},{"key":"e_1_2_1_15_1","volume-title":"An introduction to text-to-speech synthesis","author":"Dutoit Thierry","unstructured":"Thierry Dutoit. 1997. An introduction to text-to-speech synthesis. Vol. 3. Springer Science & Business Media."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/344949.345004"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1177\/160940690600500107"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3390889"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2163.358100"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2004.1374824"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1177\/154193128502900515"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.3115\/981175.981181"},{"key":"e_1_2_1_23_1","volume-title":"International Semantic Web Conference (Posters, Demos & Industry Tracks).","author":"Haase Peter","year":"2017","unstructured":"Peter Haase, Andriy Nikolov, Johannes Trame, Artem Kozlov, and Daniel M Herzig. 2017. Alexa, Ask Wikidata! Voice Interaction with Knowledge Graphs using Amazon Alexa.. In International Semantic Web Conference (Posters, Demos & Industry Tracks)."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1037\/e577632012-009"},{"key":"e_1_2_1_25_1","volume-title":"Advances in psychology.","author":"Hart Sandra G","unstructured":"Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology. Vol. 52. Elsevier, 139--183."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753478"},{"key":"e_1_2_1_27_1","volume-title":"Advances in natural language processing. Science 349, 6245","author":"Hirschberg Julia","year":"2015","unstructured":"Julia Hirschberg and Christopher D Manning. 2015. Advances in natural language processing. Science 349, 6245 (2015), 261--266."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v5i1.13313"},{"key":"e_1_2_1_29_1","unstructured":"Arne J\u00f6nsson and Nils Dahlb\u00e4ck. 1988. Talking to a computer is not like talking to your best friend. Universitetet i Link\u00f6ping\/Tekniska H\u00f6gskolan i Link\u00f6ping. Institutionen f\u00f6r . . . ."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2501988.2502034"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/354401.354406"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/99332.99366"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145354"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3003715.3005408"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702565"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047200"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2017.01.003"},{"key":"e_1_2_1_38_1","volume-title":"SYNC: A Crowdsourcing Platform for News Co-editing. In Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing. 126--129","author":"Lin Jin-An","year":"2021","unstructured":"Jin-An Lin, Feng-Yi Hsu, Hsin-Yu Yao, Shang-Hsun Lu, Tsai-Yu Kuo, Chieh-Kai Lin, and Yung-Ju Chang. 2021. SYNC: A Crowdsourcing Platform for News Co-editing. In Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing. 126--129."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASL.2006.878255"},{"key":"e_1_2_1_40_1","volume-title":"Applying the Wizard-of-Oz technique to multimodal human-robot dialogue. arXiv preprint arXiv:1703.03714","author":"Marge Matthew","year":"2017","unstructured":"Matthew Marge, Claire Bonial, Brendan Byrne, Taylor Cassidy, A William Evans, Susan G Hill, and Clare Voss. 2017. Applying the Wizard-of-Oz technique to multimodal human-robot dialogue. arXiv preprint arXiv:1703.03714 (2017)."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357236.3395440"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/169059.169215"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.3115\/981574.981581"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1080\/09537280310001647832"},{"key":"e_1_2_1_45_1","volume-title":"Knowledge graph refinement: A survey of approaches and evaluation methods. Semantic web 8, 3","author":"Paulheim Heiko","year":"2017","unstructured":"Heiko Paulheim. 2017. Knowledge graph refinement: A survey of approaches and evaluation methods. Semantic web 8, 3 (2017), 489--508."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3432942"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3432942"},{"key":"e_1_2_1_48_1","first-page":"33","article-title":"How should people and computers speak to each other","volume":"84","author":"Richards M","year":"1984","unstructured":"M Richards and K Underwood. 1984. How should people and computers speak to each other. In Proceedings of Interact, Vol. 84. 33--36.","journal-title":"Proceedings of Interact"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.5898\/JHRI.1.1.Riek"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-57433-6_51"},{"key":"e_1_2_1_51_1","unstructured":"Daniel Salber and Jo\u00eblle Coutaz. 1993. Requirements for Multimodal Wizard of Oz Platforms."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-018-0472-9"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/1822018.1822035"},{"key":"e_1_2_1_54_1","volume-title":"MobileHCI Workshop on Prototyping to Support the Interaction Designing in Mobile Application Development","author":"Schl\u00f6gl Stephan","year":"2013","unstructured":"Stephan Schl\u00f6gl, Gavin Doherty, and Saturnino Luz. 2013. Managing Consistency in Wizard of Oz Studies: A Challenge of Prototyping Natural Language Interactions. In MobileHCI Workshop on Prototyping to Support the Interaction Designing in Mobile Application Development (Munich, Germany: PID-MAD 2013."},{"key":"e_1_2_1_55_1","volume-title":"Proceedings of the SIGDIAL 2013 Conference. 160--162","author":"Schl\u00f6gl Stephan","year":"2013","unstructured":"Stephan Schl\u00f6gl, Saturnino Luz, and Gavin Doherty. 2013. WebWOZ: A Platform for Designing and Conducting Web-based Wizard of Oz Experiments. In Proceedings of the SIGDIAL 2013 Conference. 160--162."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386867"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12193-010-0042-4"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173965"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/HRI.2019.8673205"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/HRI.2019.8673205"},{"key":"e_1_2_1_61_1","doi-asserted-by":"crossref","unstructured":"Bobbie Spellman Elizabeth Gilbert and Katherine S Corker. 2017. Open science: What why and how. (2017).","DOI":"10.31234\/osf.io\/ak6jr"},{"key":"e_1_2_1_62_1","volume-title":"Wizard of Oz for Multimodal Interfaces Design: Deployment Considerations","author":"Taib Ronnie","unstructured":"Ronnie Taib and Natalie Ruiz. 2007. Wizard of Oz for Multimodal Interfaces Design: Deployment Considerations. In Human-Computer Interaction. Interaction Design and Usability, Julie A. Jacko (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 232--241."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445536"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173782"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027063.3053175"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/2909824.3020256"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.3115\/976815.976831"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2016.7745134"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3202185.3202207"},{"key":"e_1_2_1_70_1","article-title":"An intelligent chatbot utilizing BERT model and knowledge graph","volume":"24","author":"Yoo SoYeop","year":"2020","unstructured":"SoYeop Yoo and OkRan Jeong. 2020. An intelligent chatbot utilizing BERT model and knowledge graph. Journal of Society for e-Business Studies 24, 3 (2020).","journal-title":"Journal of Society for e-Business Studies"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3579591","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3579591","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:51:25Z","timestamp":1750182685000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3579591"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,14]]},"references-count":70,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2023,4,14]]}},"alternative-id":["10.1145\/3579591"],"URL":"https:\/\/doi.org\/10.1145\/3579591","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,14]]},"assertion":[{"value":"2023-04-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}