{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,3]],"date-time":"2025-12-03T17:49:12Z","timestamp":1764784152233,"version":"build-2065373602"},"reference-count":74,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2018,12,6]],"date-time":"2018-12-06T00:00:00Z","timestamp":1544054400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MTI"],"abstract":"<jats:p>Semantic fusion is a central requirement of many multimodal interfaces. Procedural methods like finite-state transducers and augmented transition networks have proven to be beneficial to implement semantic fusion. They are compliant with rapid development cycles that are common for the development of user interfaces, in contrast to machine-learning approaches that require time-costly training and optimization. We identify seven fundamental requirements for the implementation of semantic fusion: Action derivation, continuous feedback, context-sensitivity, temporal relation support, access to the interaction context, as well as the support of chronologically unsorted and probabilistic input. A subsequent analysis reveals, however, that there is currently no solution for fulfilling the latter two requirements. As the main contribution of this article, we thus present the Concurrent Cursor concept to compensate these shortcomings. In addition, we showcase a reference implementation, the Concurrent Augmented Transition Network (cATN), that validates the concept\u2019s feasibility in a series of proof of concept demonstrations as well as through a comparative benchmark. The cATN fulfills all identified requirements and fills the lack amongst previous solutions. It supports the rapid prototyping of multimodal interfaces by means of five concrete traits: Its declarative nature, the recursiveness of the underlying transition network, the network abstraction constructs of its description language, the utilized semantic queries, and an abstraction layer for lexical information. Our reference implementation was and is used in various student projects, theses, as well as master-level courses. It is openly available and showcases that non-experts can effectively implement multimodal interfaces, even for non-trivial applications in mixed and virtual reality.<\/jats:p>","DOI":"10.3390\/mti2040081","type":"journal-article","created":{"date-parts":[[2018,12,7]],"date-time":"2018-12-07T03:46:14Z","timestamp":1544154374000},"page":"81","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Semantic Fusion for Natural Multimodal Interfaces using Concurrent Augmented Transition Networks"],"prefix":"10.3390","volume":"2","author":[{"given":"Chris","family":"Zimmerer","sequence":"first","affiliation":[{"name":"Chair for Human\u2013Computer Interaction, University of W\u00fcrzburg, Am Hubland, 97074 W\u00fcrzburg, Germany"}]},{"given":"Martin","family":"Fischbach","sequence":"additional","affiliation":[{"name":"Chair for Human\u2013Computer Interaction, University of W\u00fcrzburg, Am Hubland, 97074 W\u00fcrzburg, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9340-9600","authenticated-orcid":false,"given":"Marc Erich","family":"Latoschik","sequence":"additional","affiliation":[{"name":"Chair for Human\u2013Computer Interaction, University of W\u00fcrzburg, Am Hubland, 97074 W\u00fcrzburg, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2018,12,6]]},"reference":[{"key":"ref_1","first-page":"45","article-title":"Perceptual User Interfaces: Multimodal Interfaces That Process What Comes Naturally","volume":"43","author":"Oviatt","year":"2000","journal-title":"Commun. ACM"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Nigay, L., and Coutaz, J. (1993, January 24\u201329). A design space for multimodal systems: Concurrent processing and data fusion. Proceedings of the INTERACT\u201993 and CHI\u201993 Conference on Human Factors in Computing Systems, Amsterdam, The Netherlands.","DOI":"10.1145\/169059.169143"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Kaiser, E., Olwal, A., McGee, D., Benko, H., Corradini, A., Li, X., Cohen, P., and Feiner, S. (2003, January 5\u20137). Mutual Disambiguation of 3D Multimodal Interaction in Augmented and Virtual Reality. Proceedings of the 5th International Conference on Multimodal Interfaces (ICMI \u201903), Vancouver, BC, Canada.","DOI":"10.1145\/958436.958438"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"853","DOI":"10.1109\/5.664275","article-title":"Toward multimodal human-computer interface","volume":"86","author":"Sharma","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Oviatt, S., Coulston, R., and Lunsford, R. (2004, January 13\u201315). When do we interact multimodally?: Cognitive load and multimodal communication patterns. Proceedings of the 6th International Conference on Multimodal Interfaces, State College, PA, USA.","DOI":"10.1145\/1027933.1027957"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Oviatt, S. (2012). Multimodal interfaces. The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, Lawrence Erlbaum Associates Inc.. [3rd ed.].","DOI":"10.1201\/b11963-ch-18"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Zimmerer, C., Fischbach, M., and Latoschik, M.E. (2018, January 18\u201322). Space Tentacles\u2014Integrating Multimodal Input into a VR Adventure Game. Proceedings of the 25th IEEE Virtual Reality (VR) Conference, Tuebingen\/Reutlingen, Germany.","DOI":"10.1109\/VR.2018.8446151"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Link, S., Barkschat, B., Zimmerer, C., Fischbach, M., Wiebusch, D., Lugrin, J.L., and Latoschik, M.E. (2016, January 19\u201323). An Intelligent Multimodal Mixed Reality Real-Time Strategy Game. Proceedings of the 23rd IEEE Virtual Reality (IEEE VR) Conference, Greenville, SC, USA.","DOI":"10.1109\/VR.2016.7504734"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Fischbach, M., Wiebusch, D., Giebler-Schubert, A., Latoschik, M.E., Rehfeld, S., and Tramberend, H. (2011, January 19\u201323). SiXton\u2019s curse\u2014Simulator X demonstration. Proceedings of the Virtual Reality Conference (VR), Singapore.","DOI":"10.1109\/VR.2011.5759495"},{"key":"ref_10","unstructured":"Fischbach, M.W. (2017). Enhancing Software Quality of Multimodal Interactive Systems. [Ph.D. Thesis, Universit\u00e4t W\u00fcrzburg]."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Peters, S., Johanssen, J.O., and Bruegge, B. (2016, January 12\u201316). An IDE for Multimodal Controls in Smart Buildings. Proceedings of the 18th ACM International Conference on Multimodal Interaction (ICMI), Tokyo, Japan.","DOI":"10.1145\/2993148.2993162"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Cacace, J., Finzi, A., and Lippiello, V. (September, January 28). A robust multimodal fusion framework for command interpretation in human-robot cooperation. Proceedings of the 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Lisbon, Portugal.","DOI":"10.1109\/ROMAN.2017.8172329"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Pfleger, N. (2004, January 13\u201315). Context Based Multimodal Fusion. Proceedings of the 6th International Conference on Multimodal Interfaces (ICMI \u201904), State College, PA, USA.","DOI":"10.1145\/1027933.1027977"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Lalanne, D., Nigay, L., Palanque, P., Robinson, P., Vanderdonckt, J., and Ladry, J.F. (2009, January 2\u20134). Fusion Engines for Multimodal Input: A Survey. Proceedings of the 2009 International Conference on Multimodal Interfaces, Cambridge, MA, USA.","DOI":"10.1145\/1647314.1647343"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Neal, J.G., Thielman, C.Y., Dobes, Z., Haller, S.M., and Shapiro, S.C. (1989, January 15\u201318). Natural Language with Integrated Deictic and Graphic Gestures. Proceedings of the Workshop on Speech and Natural Language (HLT \u201989), Cape Cod, MA, USA.","DOI":"10.3115\/1075434.1075499"},{"key":"ref_16","unstructured":"Latoschik, M.E. (2002, January 14\u201316). Designing Transition Networks for Multimodal VR-Interactions Using a Markup Language. Proceedings of the 4th IEEE International Conference on Multimodal Interfaces (ICMI \u201902), Pittsburgh, PA, USA."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Nigay, L., and Coutaz, J. (1995, January 7\u201311). A generic platform for addressing the multimodal challenge. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Denver, CO, USA.","DOI":"10.1145\/223904.223917"},{"key":"ref_18","unstructured":"Duarte, C., and Carri\u00e7o, L. (February, January 29). A Conceptual Framework for Developing Adaptive Multimodal Applications. Proceedings of the 11th International Conference on Intelligent User Interfaces (IUI \u201906), Sydney, Australia."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Holzapfel, H., Nickel, K., and Stiefelhagen, R. (2004, January 13\u201315). Implementation and Evaluation of a Constraint-based Multimodal Fusion System for Speech and 3D Pointing Gestures. Proceedings of the 6th International Conference on Multimodal Interfaces (ICMI \u201904), State College, PA, USA.","DOI":"10.1145\/1027933.1027964"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1342","DOI":"10.1109\/TVCG.2017.2657098","article-title":"Semantic Entity-Component State Management Techniques to Enhance Software Quality for Multimodal VR-Systems","volume":"23","author":"Fischbach","year":"2017","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Bolt, R.A. (1980, January 14\u201318). Put-that-there: Voice and Gesture at the Graphics Interface. Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH \u201980), Seattle, WA, USA.","DOI":"10.1145\/800250.807503"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Latoschik, M.E. (2005, January 4\u20136). A User Interface Framework for Multimodal VR Interactions. Proceedings of the 7th International Conference on Multimodal Interfaces, Torento, Italy.","DOI":"10.1145\/1088463.1088479"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Zhang, B., Essl, G., and Mower Provost, E. (2016, January 12\u201316). Automatic Recognition of Self-reported and Perceived Emotion: Does Joint Modeling Help?. Proceedings of the 18th ACM International Conference on Multimodal Interaction (ICMI 2016), Tokyo, Japan.","DOI":"10.1145\/2993148.2993173"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Kalimeri, K., and Saitis, C. (2016, January 12\u201316). Exploring Multimodal Biosignal Features for Stress Detection During Indoor Mobility. Proceedings of the 18th ACM International Conference on Multimodal Interaction (ICMI 2016), Tokyo, Japan.","DOI":"10.1145\/2993148.2993159"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Dibeklio\u011flu, H., Hammal, Z., Yang, Y., and Cohn, J.F. (2015, January 9\u201313). Multimodal Detection of Depression in Clinical Interviews. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction (ICMI \u201915), Seattle, WA, USA.","DOI":"10.1145\/2818346.2820776"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"P\u00e9rez-Rosas, V., Abouelenien, M., Mihalcea, R., and Burzo, M. (2015, January 9\u201313). Deception Detection Using Real-life Trial Data. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction (ICMI \u201915), Seattle, WA, USA.","DOI":"10.1145\/2818346.2820758"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Koons, D.B., and Sparrell, C.J. (1994, January 24\u201328). Iconic: Speech and Depictive Gestures at the Human-machine Interface. Proceedings of the Conference Companion on Human Factors in Computing Systems, Boston, MA, USA.","DOI":"10.1145\/259963.260487"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Dumas, B., Lalanne, D., and Ingold, R. (2009, January 2\u20134). HephaisTK: A Toolkit for Rapid Prototyping of Multimodal Interfaces. Proceedings of the 2009 International Conference on Multimodal Interfaces (ICMI-MLMI \u201909), Cambridge, MA, USA.","DOI":"10.1145\/1647314.1647360"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., and Clow, J. (1997, January 9\u201313). QuickSet: Multimodal Interaction for Distributed Applications. Proceedings of the Fifth International Conference on Multimedia, Seattle, WA, USA.","DOI":"10.1145\/266180.266328"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Johnston, M., Cohen, P.R., McGee, D., Oviatt, S.L., Pittman, J.A., and Smith, I. (1997, January 7\u201312). Unification-based Multimodal Integration. Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics (ACL \u201998), Madrid, Spain.","DOI":"10.3115\/976909.979653"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"972","DOI":"10.1109\/TNN.2002.1021897","article-title":"From members to teams to committee-a robust approach to gestural and multimodal recognition","volume":"13","author":"Wu","year":"2002","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Chai, J.Y., Hong, P., and Zhou, M.X. (2004, January 13\u201316). A Probabilistic Approach to Reference Resolution in Multimodal User Interfaces. Proceedings of the 9th International Conference on Intelligent User Interfaces (IUI \u201904), Funchal, Madeira, Portugal.","DOI":"10.1145\/964456.964457"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Dumas, B., Signer, B., and Lalanne, D. (2012, January 25\u201326). Fusion in Multimodal Interactive Systems: An HMM-based Algorithm for User-induced Adaptation. Proceedings of the 4th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS \u201912), Copenhagen, Denmark.","DOI":"10.1145\/2305484.2305490"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"369","DOI":"10.3115\/990820.990874","article-title":"Finite-state multimodal parsing and understanding","volume":"Volume 1","author":"Johnston","year":"2000","journal-title":"Proceedings of the 18th Conference on Computational Linguistics"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1017\/S1351324904003572","article-title":"Finite-state Multimodal Integration and Understanding","volume":"11","author":"Johnston","year":"2005","journal-title":"Nat. Lang. Eng."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"987184","DOI":"10.1155\/S1110865703211173","article-title":"Semantic Indexing of Multimedia Content Using Visual, Audio, and Text Cues","volume":"2003","author":"Adams","year":"2003","journal-title":"EURASIP J. Adv. Signal Process."},{"key":"ref_37","unstructured":"Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., and Ng, A.Y. (July, January 28). Multimodal deep learning. Proceedings of the 28th International Conference on Machine Learning (ICML-11), Bellevue, WA, USA."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Mart\u00ednez, H.P., and Yannakakis, G.N. (2014, January 12\u201316). Deep multimodal fusion: Combining discrete events and continuous signals. Proceedings of the 16th International Conference on Multimodal Interaction, Istanbul, Turkey.","DOI":"10.1145\/2663204.2663236"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1007\/s00530-010-0182-0","article-title":"Multimodal fusion for multimedia analysis: A survey","volume":"16","author":"Atrey","year":"2010","journal-title":"Multimed. Syst."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Hoste, L., Dumas, B., and Signer, B. (2011, January 14\u201318). Mudra: A Unified Multimodal Interaction Framework. Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI \u201911), Alicante, Spain.","DOI":"10.1145\/2070481.2070500"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Dumas, B., Lalanne, D., and Oviatt, S. (2009). Multimodal Interfaces: A Survey of Principles, Models and Frameworks. Human Machine Interaction, Springer.","DOI":"10.1007\/978-3-642-00437-7_1"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Potamianos, G., Marcheret, E., Mroueh, Y., Goel, V., Koumbaroulis, A., Vartholomaios, A., and Thermos, S. (2017). Audio and Visual Modality Combination in Speech Processing Applications. The Handbook of Multimodal-Multisensor Interfaces, Association for Computing Machinery and Morgan & Claypool.","DOI":"10.1145\/3015783.3015797"},{"key":"ref_43","first-page":"625","article-title":"Why Does Unsupervised Pre-training Help Deep Learning?","volume":"11","author":"Erhan","year":"2010","journal-title":"J. Mach. Learn. Res."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Oviatt, S., and Cohen, P.R. (2015). The Paradigm Shift to Multimodality in Contemporary Computer Interfaces, Morgan & Claypool Publishers.","DOI":"10.1007\/978-3-031-02213-5"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Putze, F., Popp, J., Hild, J., Beyerer, J., and Schultz, T. (2016, January 12\u201316). Intervention-free Selection Using EEG and Eye Tracking. Proceedings of the 18th ACM International Conference on Multimodal Interaction (ICMI 2016), Tokyo, Japan.","DOI":"10.1145\/2993148.2993199"},{"key":"ref_46","unstructured":"Snoek, J., Larochelle, H., and Adams, R.P. (2012, January 3\u20136). Practical bayesian optimization of machine learning algorithms. Proceedings of the 25th International Conference on Neural Information Processing Systems (NIPS\u201912), Lake Tahoe, Nevada."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1016\/S0004-3702(97)00063-5","article-title":"Selection of relevant features and examples in machine learning","volume":"97","author":"Blum","year":"1997","journal-title":"Artif. Intell."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Oviatt, S., Schuller, B., Cohen, P.R., Sonntag, D., Potamianos, G., and Kr\u00fcger, A. (2017). The Handbook of Multimodal-Multisensor Interfaces: Foundations, User Modeling, and Common Modality Combinations\u2014Volume 1, Association for Computing Machinery and Morgan & Claypool.","DOI":"10.1145\/3015783"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Mayhew, D.J. (1999, January 15\u201320). The Usability Engineering Lifecycle. Proceedings of the CHI \u201999 Extended Abstracts on Human Factors in Computing Systems (CHI EA \u201999), Pittsburgh, PA, USA.","DOI":"10.1145\/632716.632805"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"591","DOI":"10.1145\/355598.362773","article-title":"Transition Network Grammars for Natural Language Analysis","volume":"13","author":"Woods","year":"1970","journal-title":"Commun. ACM"},{"key":"ref_51","unstructured":"Zimmerer, C., Fischbach, M., and Latoschik, M.E. (2018, August 22). Concurrent Augmented Transition Network\u2014Project Page. Available online: https:\/\/www.hci.uni-wuerzburg.de\/projects\/mmi\/."},{"key":"ref_52","unstructured":"Poddar, I., Sethi, Y., Ozyildiz, E., and Sharma, R. (1998, January 5\u20136). Toward natural gesture\/speech HCI: A case study of weather narration. Proceedings of the Workshop on Perceptual User Interfaces (PUI98), San Francisco, CA, USA."},{"key":"ref_53","unstructured":"Krahnstoever, N., Kettebekov, S., Yeasin, M., and Sharma, R. (2002, January 14\u201316). A Real-Time Framework for Natural Multimodal Interaction with Large Screen Displays. Proceedings of the 4th IEEE International Conference on Multimodal Interfaces (ICMI \u201902), Pittsburgh, PA, USA."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Serrano, M., Nigay, L., Lawson, J.Y.L., Ramsay, A., Murray-Smith, R., and Denef, S. (2008, January 5\u201310). The Openinterface Framework: A Tool for Multimodal Interaction. Proceedings of the Extended Abstracts on Human Factors in Computing Systems, Florence, Italy.","DOI":"10.1145\/1358628.1358881"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Wagner, J., Lingenfelser, F., Baur, T., Damian, I., Kistler, F., and Andr\u00e9, E. (2013, January 21\u201325). The Social Signal Interpretation (SSI) Framework: Multimodal Signal Processing and Recognition in Real-time. Proceedings of the 21st ACM International Conference on Multimedia (MM \u201913), Barcelona, Spain.","DOI":"10.1145\/2502081.2502223"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Latoschik, M.E. (2001, January 14). A general framework for multimodal interaction in virtual reality systems: PrOSA. Proceedings of the Future of VR and AR Interfaces-Multimodal, Humanoid, Adaptive and Intelligent\u2014Workshop at IEEE Virtual Reality, Yokohama, Japan. No. 138.","DOI":"10.1145\/513867.513888"},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Bouchet, J., Nigay, L., and Ganille, T. (2004, January 13\u201315). ICARE Software Components for Rapidly Developing Multimodal Interfaces. Proceedings of the 6th International Conference on Multimodal Interfaces, State College, PA, USA.","DOI":"10.1145\/1027933.1027975"},{"key":"ref_58","unstructured":"Latoschik, M.E., and Tramberend, H. (October, January 27). Short Paper: Engineering Realtime Interactive Systems: Coupling & Cohesion of Architecture Mechanisms. Proceedings of the 16th Eurographics Conference on Virtual Environments & Second Joint Virtual Reality (EGVE\u2014JVRC\u201910), Stuttgart, Germany."},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Latoschik, M.E., and Fischbach, M. (2014, January 22\u201327). Engineering variance: Software techniques for scalable, customizable, and reusable multimodal processing. Proceedings of the International Conference on Human-Computer Interaction, Heraklion, Crete, Greece.","DOI":"10.1007\/978-3-319-07233-3_29"},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Fischbach, M. (2015, January 9\u201313). Software Techniques for Multimodal Input Processing in Realtime Interactive Systems. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction (ICMI \u201915), Seattle, WA, USA.","DOI":"10.1145\/2818346.2823308"},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"338","DOI":"10.1145\/77481.77486","article-title":"Improving a Human-computer Dialogue","volume":"33","author":"Molich","year":"1990","journal-title":"Commun. ACM"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1017\/S1351324997001605","article-title":"Transducer Parsing of Free and Frozen Sentences","volume":"2","author":"Roche","year":"1996","journal-title":"Nat. Lang. Eng."},{"key":"ref_63","first-page":"29","article-title":"A toolkit for creating and testing multimodal interface designs","volume":"2","author":"Bourguet","year":"2002","journal-title":"Companion Proc. UIST"},{"key":"ref_64","unstructured":"Hopcroft, J.E., and Ullman, J.D. (1990). Introduction To Automata Theory, Languages, And Computation, Addison-Wesley Longman Publishing Co., Inc.. [1st ed.]."},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"832","DOI":"10.1145\/182.358434","article-title":"Maintaining Knowledge About Temporal Intervals","volume":"26","author":"Allen","year":"1983","journal-title":"Commun. ACM"},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"Zimmerer, C., Fischbach, M., and Latoschik, M.E. (2016, January 2\u20134). Maintainable Management and Access of Lexical Knowledge for Multimodal Virtual Reality Interfaces. In Proceeding of the 22nd ACM Symposium on Virtual Reality Software and Technology (VRST), Munich, Germany.","DOI":"10.1145\/2993369.2996310"},{"key":"ref_67","first-page":"313","article-title":"Building a Large Annotated Corpus of English: The Penn Treebank","volume":"19","author":"Marcus","year":"1993","journal-title":"Comput. Linguist."},{"key":"ref_68","doi-asserted-by":"crossref","unstructured":"Wiebusch, D., Zimmerer, C., and Latoschik, M.E. (2017, January 19). Cherry-Picking RIS Functionality\u2014Integration of Game and VR Engine Sub-Systems based on Entities and Events. Proceedings of the 10th Workshop on Software Engineering and Architectures for Realtime Interactive Systems (SEARIS), Los Angeles, CA, USA.","DOI":"10.1109\/SEARIS41720.2017.9183669"},{"key":"ref_69","unstructured":"Heidrich, D., Zimmerer, C., Fischbach, M., and Latoschik, M.E. (2018, August 22). Robot Museum. Available online: https:\/\/www.hci.uni-wuerzburg.de\/2018\/06\/12\/robot-museum-demo\/."},{"key":"ref_70","unstructured":"Zimmerer, C., Fischbach, M., and Latoschik, M.E. (2018, August 22). Big Bang. Available online: https:\/\/www.hci.uni-wuerzburg.de\/2016\/10\/11\/planetarium\/."},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1109\/52.676963","article-title":"Building more usable APIs","volume":"15","author":"McLellan","year":"1998","journal-title":"IEEE Softw."},{"key":"ref_72","doi-asserted-by":"crossref","unstructured":"Piccioni, M., Furia, C.A., and Meyer, B. (2013, January 10\u201311). An Empirical Study of API Usability. In Proceeding of the 2013 ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement, Baltimore, MD, USA.","DOI":"10.1109\/ESEM.2013.14"},{"key":"ref_73","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1145\/2896587","article-title":"Improving API Usability","volume":"59","author":"Myers","year":"2016","journal-title":"Commun. ACM"},{"key":"ref_74","unstructured":"Thiran, J.P., Marqu\u00e9s, F., and Bourlard, H. (2010). Chapter 12\u2014Multimodal Input. Multimodal Signal Processing, Academic Press."}],"container-title":["Multimodal Technologies and Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2414-4088\/2\/4\/81\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T15:31:49Z","timestamp":1760196709000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2414-4088\/2\/4\/81"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,12,6]]},"references-count":74,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2018,12]]}},"alternative-id":["mti2040081"],"URL":"https:\/\/doi.org\/10.3390\/mti2040081","relation":{},"ISSN":["2414-4088"],"issn-type":[{"type":"electronic","value":"2414-4088"}],"subject":[],"published":{"date-parts":[[2018,12,6]]}}}