{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T03:16:32Z","timestamp":1760238992170,"version":"build-2065373602"},"reference-count":47,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2020,9,23]],"date-time":"2020-09-23T00:00:00Z","timestamp":1600819200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MTI"],"abstract":"<jats:p>This paper presents the design of a multimodal chatbot for use in an interactive theater performance. This chatbot has an architecture consisting of vision and natural language processing capabilities, as well as embodiment in a non-anthropomorphic movable LED array set in a stage. Designed for interaction with up to five users at a time, the system can perform tasks including face detection and emotion classification, tracking of crowd movement through mobile phones, and real-time conversation to guide users through a nonlinear story and interactive games. The final prototype, named ODO, is a tangible embodiment of a distributed multimedia system that solves several technical challenges to provide users with a unique experience through novel interaction.<\/jats:p>","DOI":"10.3390\/mti4040068","type":"journal-article","created":{"date-parts":[[2020,9,23]],"date-time":"2020-09-23T09:28:08Z","timestamp":1600853288000},"page":"68","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["ODO: Design of Multimodal Chatbot for an Experiential Media System"],"prefix":"10.3390","volume":"4","author":[{"given":"Ravi","family":"Bhushan","sequence":"first","affiliation":[{"name":"School of Electrical, Computer and Energy Engineering, Arizona State University, Phoenix, AZ 85281, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Karthik","family":"Kulkarni","sequence":"additional","affiliation":[{"name":"School of Electrical, Computer and Energy Engineering, Arizona State University, Phoenix, AZ 85281, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vishal Kumar","family":"Pandey","sequence":"additional","affiliation":[{"name":"School of Electrical, Computer and Energy Engineering, Arizona State University, Phoenix, AZ 85281, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Connor","family":"Rawls","sequence":"additional","affiliation":[{"name":"School of Arts, Media and Engineering, Arizona State University, Phoenix, AZ 85281, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Brandon","family":"Mechtley","sequence":"additional","affiliation":[{"name":"School of Arts, Media and Engineering, Arizona State University, Phoenix, AZ 85281, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Suren","family":"Jayasuriya","sequence":"additional","affiliation":[{"name":"School of Electrical, Computer and Energy Engineering, Arizona State University, Phoenix, AZ 85281, USA"},{"name":"School of Arts, Media and Engineering, Arizona State University, Phoenix, AZ 85281, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christian","family":"Ziegler","sequence":"additional","affiliation":[{"name":"School of Arts, Media and Engineering, Arizona State University, Phoenix, AZ 85281, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,9,23]]},"reference":[{"key":"ref_1","unstructured":"Catania, F., Spitale, M., Fisicaro, D., and Garzotto, F. (2019, January 20). CORK: A COnversational agent framewoRK exploiting both rational and emotional intelligence. Proceedings of the IUI Workshops, Los Angeles, CA, USA."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1162\/coli_a_00368","article-title":"The design and implementation of xiaoice, an empathetic social chatbot","volume":"46","author":"Zhou","year":"2020","journal-title":"Comput. Linguist."},{"key":"ref_3","unstructured":"Yin, J., Chen, Z., Zhou, K., and Yu, C. (2019). A Deep Learning Based Chatbot for Campus Psychological Therapy. arXiv."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Cer, D., Yang, Y., Kong, S.Y., Hua, N., Limtiaco, N., John, R.S., Constant, N., Guajardo-Cespedes, M., Yuan, S., and Tar, C. (2018). Universal sentence encoder. arXiv.","DOI":"10.18653\/v1\/D18-2029"},{"key":"ref_5","unstructured":"Arriaga, O., Valdenegro-Toro, M., and Pl\u00f6ger, P. (2017). Real-time convolutional neural networks for emotion and gender classification. arXiv."},{"key":"ref_6","unstructured":"Freed, A. (1997, January 25\u201330). Open sound control: A new protocol for communicating with sound synthesizers. Proceedings of the International Computer Music Conference (ICMC), Thessaloniki, Greece."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Lehmann, H.T. (2006). Postdramatic Theatre, Routledge.","DOI":"10.4324\/9780203088104"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1109\/MSP.2016.2617341","article-title":"The technology behind personal digital assistants: An overview of the system architecture and key components","volume":"34","author":"Sarikaya","year":"2017","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_9","unstructured":"(2020, May 01). Greenshoot Labs. Available online: https:\/\/www.opendialog.ai\/."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Sabharwal, N., and Agrawal, A. (2020). Introduction to Google Dialogflow. Cognitive Virtual Assistants Using Google Dialogflow, Springer.","DOI":"10.1007\/978-1-4842-5741-8"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Sabharwal, N., Barua, S., Anand, N., and Aggarwal, P. (2020). Bot Frameworks. Developing Cognitive Bots Using the IBM Watson Engine, Springer.","DOI":"10.1007\/978-1-4842-5555-1"},{"key":"ref_12","unstructured":"Mayo, J. (2017). Programming the Microsoft Bot Framework: A Multiplatform Approach to Building Chatbots, Microsoft Press."},{"key":"ref_13","unstructured":"Williams, S. (2018). Hands-On Chatbot Development with Alexa Skills and Amazon Lex: Create Custom Conversational and Voice Interfaces for Your Amazon Echo Devices and Web Platforms, Packt Publishing Ltd."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Gao, J., Galley, M., and Li, L. (2018, January 8\u201312). Neural approaches to conversational ai. Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, Ann Arbor, MI, USA.","DOI":"10.1145\/3209978.3210183"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Serban, I.V., Sordoni, A., Bengio, Y., Courville, A., and Pineau, J. (2016, January 12\u201317). Building end-to-end dialogue systems using generative hierarchical neural network models. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.","DOI":"10.1609\/aaai.v30i1.9883"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Sordoni, A., Galley, M., Auli, M., Brockett, C., Ji, Y., Mitchell, M., Nie, J.Y., Gao, J., and Dolan, B. (2015). A neural network approach to context-sensitive generation of conversational responses. arXiv.","DOI":"10.3115\/v1\/N15-1020"},{"key":"ref_17","unstructured":"Vinyals, O., and Le, Q. (2015). A neural conversational model. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"811","DOI":"10.1017\/S1351324916000243","article-title":"The return of the chatbots","volume":"22","author":"Dale","year":"2016","journal-title":"Nat. Lang. Eng."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Brandtzaeg, P.B., and F\u00f8lstad, A. (2017, January 22\u201324). Why people use chatbots. Proceedings of the International Conference on Internet Science, Thessaloniki, Greece.","DOI":"10.1007\/978-3-319-70284-1_30"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1145\/3085558","article-title":"Chatbots and the new world of HCI","volume":"24","year":"2017","journal-title":"Interactions"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Purington, A., Taft, J.G., Sannon, S., Bazarova, N.N., and Taylor, S.H. (2017, January 6\u201311). \u201cAlexa is my new BFF\u201d Social Roles, User Satisfaction, and Personification of the Amazon Echo. Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, Denver, CO, USA.","DOI":"10.1145\/3027063.3053246"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"539","DOI":"10.1016\/j.future.2018.01.055","article-title":"In the shades of the uncanny valley: An experimental study of human\u2013chatbot interaction","volume":"92","author":"Ciechanowski","year":"2019","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1016\/j.chb.2015.02.026","article-title":"Real conversations with artificial intelligence: A comparison between human\u2013human online conversations and human\u2013chatbot conversations","volume":"49","author":"Hill","year":"2015","journal-title":"Comput. Hum. Behav."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2502432","article-title":"Experiential media systems","volume":"9","author":"Sundaram","year":"2013","journal-title":"ACM Trans. Multimed. Comput. Commun. Appl."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1145\/792704.792729","article-title":"Experiential computing","volume":"46","author":"Jain","year":"2003","journal-title":"Commun. ACM"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Davis, M. (2003, January 7). Theoretical foundations for experiential systems design. Proceedings of the 2003 ACM SIGMM Workshop on Experiential Telepresence, Berkeley, CA, USA.","DOI":"10.1145\/982484.982491"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Chen, Y., Sundaram, H., Rikakis, T., Ingalls, T., Olson, L., and He, J. (2009). Experiential Media Systems\u2014The Biofeedback Project. Multimedia Content Analysis, Springer.","DOI":"10.1007\/978-0-387-76569-3_8"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Johnson, G.L., Peterson, B.J., Ingalls, T., and Wei, S.X. (2018, January 28\u201330). Lanterns: An Enacted and Material Approach to Ensemble Group Activity with Responsive Media. Proceedings of the 5th International Conference on Movement and Computing, MOCO\u2019 18, Genoa, Italy.","DOI":"10.1145\/3212721.3212848"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"536","DOI":"10.1162\/105474602320935865","article-title":"\u201cIt\/I\u201d: A theater play featuring an autonomous computer character","volume":"11","author":"Pinhanez","year":"2002","journal-title":"Presence Teleoperators Virtual Environ."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Knight, H. (2011, January 24\u201325). Eight lessons learned about non-verbal interactions through robot theater. Proceedings of the International Conference on Social Robotics, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-642-25504-5_5"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Hoffman, G., Kubat, R., and Breazeal, C. (2008, January 1\u20133). A hybrid control system for puppeteering a live robotic stage actor. Proceedings of the RO-MAN 2008\u2014The 17th IEEE International Symposium on Robot and Human Interactive Communication, Munich, Germany.","DOI":"10.1109\/ROMAN.2008.4600691"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Mazalek, A., Nitsche, M., R\u00e9bola, C., Wu, A., Clifton, P., Peer, F., and Drake, M. (2011, January 3\u20136). Pictures at an exhibition: A physical\/digital puppetry performance piece. Proceedings of the 8th ACM Conference on Creativity and Cognition, Atlanta, GA, USA.","DOI":"10.1145\/2069618.2069739"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Meyer, T., and Messom, C. (2004, January 1\u20133). Improvisation in theatre rehearsals for synthetic actors. Proceedings of the International Conference on Entertainment Computing, Eindhoven, The Netherlands.","DOI":"10.1007\/978-3-540-28643-1_23"},{"key":"ref_34","unstructured":"Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., and Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, Neural Information Processing Systems Foundation, Inc."},{"key":"ref_35","unstructured":"Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv."},{"key":"ref_36","unstructured":"Pandorabot (2020, May 01). Mitusku, Available online: https:\/\/www.pandorabots.com\/mitsuku\/."},{"key":"ref_37","unstructured":"Viola, P., and Jones, M. (2001, January 8\u201314). Rapid object detection using a boosted cascade of simple features. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2001, Kauai, HI, USA."},{"key":"ref_38","unstructured":"Cycling74 (2020, May 22). Max\/MSP. Available online: https:\/\/cycling74.com\/."},{"key":"ref_39","unstructured":"Ester, M., Kriegel, H.P., Sander, J., and Xu, X. (1996, January 2\u20134). A density-based algorithm for discovering clusters in large spatial databases with noise. Proceedings of the SIGKDD Conference on Knowledge Discovery and Data Mining, Portland, OR, USA."},{"key":"ref_40","unstructured":"Ziegler, C., Sayaka Kaiwa, T.S., and Paquete, H. (2020, May 01). COSMOS. Presented at ZKM Center for Art and Media, Karlsruhe in 2018. Available online: https:\/\/zkm.de\/en\/event\/2018\/03\/cosmos."},{"key":"ref_41","unstructured":"Art-Net (2020, May 01). 2020 Artistic Licence Holdings Ltd., United Kingdom. Available online: https:\/\/art-net.org.uk\/."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Iyyer, M., Manjunatha, V., Boyd-Graber, J., and Daum\u00e9, H. (2015, January 26\u201331). Deep unordered composition rivals syntactic methods for text classification. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, Beijing, China. Volume 1: Long Papers.","DOI":"10.3115\/v1\/P15-1162"},{"key":"ref_43","unstructured":"Ayman, O. (2020, May 01). Emotion-Recognition. Available online: https:\/\/github.com\/omar178\/Emotion-recognition."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Chollet, F. (2017, January 21\u201326). Xception: Deep learning with depthwise separable convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.195"},{"key":"ref_45","unstructured":"Ioffe, S., and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., and Lee, D.H. (2013, January 3\u20137). Challenges in representation learning: A report on three machine learning contests. Proceedings of the International Conference on Neural Information Processing, Daegu, Korea.","DOI":"10.1007\/978-3-642-42051-1_16"},{"key":"ref_47","unstructured":"Tirumala, A., Qin, F., Dugan, J.M., Ferguson, J.A., and Gibbs, K.A. (2020, May 01). iPerf: TCP\/UDP Bandwidth Measurement Tool. Available online: https:\/\/iperf.fr\/."}],"container-title":["Multimodal Technologies and Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2414-4088\/4\/4\/68\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:12:38Z","timestamp":1760177558000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2414-4088\/4\/4\/68"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,23]]},"references-count":47,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2020,12]]}},"alternative-id":["mti4040068"],"URL":"https:\/\/doi.org\/10.3390\/mti4040068","relation":{},"ISSN":["2414-4088"],"issn-type":[{"type":"electronic","value":"2414-4088"}],"subject":[],"published":{"date-parts":[[2020,9,23]]}}}