{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,12]],"date-time":"2026-06-12T23:22:59Z","timestamp":1781306579890,"version":"3.54.1"},"reference-count":45,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2025,5,26]],"date-time":"2025-05-26T00:00:00Z","timestamp":1748217600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002341","name":"Research Council Of Finland","doi-asserted-by":"publisher","award":["345448"],"award-info":[{"award-number":["345448"]}],"id":[{"id":"10.13039\/501100002341","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002341","name":"Research Council Of Finland","doi-asserted-by":"publisher","award":["80\/31\/2023"],"award-info":[{"award-number":["80\/31\/2023"]}],"id":[{"id":"10.13039\/501100002341","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100014438","name":"Business Finland","doi-asserted-by":"publisher","award":["345448"],"award-info":[{"award-number":["345448"]}],"id":[{"id":"10.13039\/501100014438","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100014438","name":"Business Finland","doi-asserted-by":"publisher","award":["80\/31\/2023"],"award-info":[{"award-number":["80\/31\/2023"]}],"id":[{"id":"10.13039\/501100014438","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MTI"],"abstract":"<jats:p>Applications of virtual reality (VR) have grown in significance in medicine, as they are able to recreate real-life scenarios in 3D while posing reduced risks to patients. However, there are several interaction challenges to overcome when moving from 2D screens to 3D VR environments, such as complex controls and slow user adaptation. More intuitive techniques are needed for enhanced user experience. Our research explored the potential of intelligent speech interfaces to enhance user interaction while conducting complex medical tasks. We developed a speech-based assistant within a VR application for maxillofacial implant planning, leveraging natural language processing (NLP) to interpret user intentions and to execute tasks such as obtaining surgical equipment or answering questions related to the VR environment. The objective of the study was to evaluate the usability and cognitive load of the speech-based assistant. We conducted a mixed-methods within-subjects user study with 20 participants and compared the voice-assisted approach to traditional interaction methods, such as button panels on the VR view, across various tasks. Our findings indicate that NLP-driven speech-based assistants can enhance interaction and accessibility in medical VR, especially in areas such as locating controls, easiness of control, user comfort, and intuitive interaction. These findings highlight the potential benefits of augmenting traditional controls with speech interfaces, particularly in complex VR scenarios where conventional methods may limit usability. We identified key areas for future research, including improving the intelligence, accuracy, and user experience of speech-based systems. Addressing these areas could facilitate the development of more robust, user-centric, voice-assisted applications in virtual reality environments.<\/jats:p>","DOI":"10.3390\/mti9060050","type":"journal-article","created":{"date-parts":[[2025,5,27]],"date-time":"2025-05-27T11:12:57Z","timestamp":1748344377000},"page":"50","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["A Study of NLP-Based Speech Interfaces in Medical Virtual Reality"],"prefix":"10.3390","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-5892-8960","authenticated-orcid":false,"given":"Mohit","family":"Nayak","sequence":"first","affiliation":[{"name":"TAUCHI Research Center, Faculty of Information Technology and Communication Sciences, Tampere University, 33100 Tampere, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1970-1862","authenticated-orcid":false,"given":"Jari","family":"Kangas","sequence":"additional","affiliation":[{"name":"TAUCHI Research Center, Faculty of Information Technology and Communication Sciences, Tampere University, 33100 Tampere, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3276-7866","authenticated-orcid":false,"given":"Roope","family":"Raisamo","sequence":"additional","affiliation":[{"name":"TAUCHI Research Center, Faculty of Information Technology and Communication Sciences, Tampere University, 33100 Tampere, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,5,26]]},"reference":[{"key":"ref_1","first-page":"187","article-title":"Using virtual reality for medical diagnosis, training and education","volume":"5","author":"McCrindle","year":"2006","journal-title":"Int. J. Disabil. Hum. Dev."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1109\/MCG.2006.131","article-title":"Liver surgery planning using virtual reality","volume":"26","author":"Reitinger","year":"2006","journal-title":"IEEE Comput. Graph. Appl."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"45442","DOI":"10.1109\/ACCESS.2024.3375770","article-title":"A Framework to Enhance the Experience of CBCT Data in Real-Time Using Immersive Virtual Reality: Impacting Dental Pre-Surgical Planning","volume":"12","author":"Bhat","year":"2024","journal-title":"IEEE Access"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Kangas, J., J\u00e4rnstedt, J., Ronkainen, K., M\u00e4kel\u00e4, J., Mehtonen, H., Huuskonen, P., and Raisamo, R. (2024). Towards the Emergence of the Medical Metaverse: A Pilot Study on Shared Virtual Reality for Orthognathic\u2013Surgical Planning. Appl. Sci., 14.","DOI":"10.3390\/app14031038"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Chen, W., Kuniewicz, M., Aminu, A.J., Karaesmen, I., Duong, N., Proniewska, K., van Dam, P., Iles, T.L., Ho\u0142da, M.K., and Walocha, J. (2025). High-resolution 3D visualization of human hearts with emphases on the cardiac conduction system components\u2014a new platform for medical education, mix\/virtual reality, computational simulation. Front. Med., 12.","DOI":"10.3389\/fmed.2025.1507005"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Kumar, S., Fred, A.L., Ajay Kumar, H., Miriam, L.J., Jane, I.C., Padmanabhan, P., and Guly\u00e1s, B. (2025). Role of Augmented Reality and Virtual Reality in Medical Imaging. Introduction to Extended Reality (XR) Technologies, Scrivener Publishing LLC.","DOI":"10.1002\/9781119857716.ch9"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.oooo.2023.07.001","article-title":"The use of virtual reality and augmented reality in oral and maxillofacial surgery: A narrative review","volume":"137","author":"Stucki","year":"2024","journal-title":"Oral Surg. Oral Med. Oral Pathol. Oral Radiol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1145\/260750.260768","article-title":"Medical applications of virtual reality","volume":"40","author":"Zajtchuk","year":"1997","journal-title":"Commun. ACM"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Tene, T., Vique L\u00f3pez, D.F., Valverde Aguirre, P.E., Orna Puente, L.M., and Vacacela Gomez, C. (2024). Virtual reality and augmented reality in medical education: An umbrella review. Front. Digit. Health, 6.","DOI":"10.3389\/fdgth.2024.1365345"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1640003","DOI":"10.1142\/S2424905X16400031","article-title":"An immersive virtual reality environment for diagnostic imaging","volume":"1","author":"King","year":"2016","journal-title":"J. Med. Robot. Res."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Bueckle, A., Buehling, K., Shih, P.C., and B\u00f6rner, K. (2021). 3D virtual reality vs. 2D desktop registration user interface comparison. PLoS ONE, 16.","DOI":"10.1371\/journal.pone.0258103"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"123565","DOI":"10.1109\/ACCESS.2023.3329371","article-title":"Effect of Auto-Erased Sketch Cue in Multiuser Surgical Planning Virtual Reality Collaboration System","volume":"11","author":"Park","year":"2023","journal-title":"IEEE Access"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Cockburn, A., and McKenzie, B. (2002, January 20\u201325). Evaluating the effectiveness of spatial memory in 2D and 3D physical and virtual environments. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Minneapolis, MN, USA.","DOI":"10.1145\/503411.503413"},{"key":"ref_14","unstructured":"Fernandez, J.A.V., Lee, J.J., Vacca, S.A.S., Magana, A., Pesam, R., Benes, B., and Popescu, V. (2024). Hands-Free VR. arXiv."},{"key":"ref_15","unstructured":"Ng, H.W., Koh, A., Foong, A., Ong, J., Tan, J.H., Khoo, E.T., and Liu, G. Real-time spoken language understanding for orthopedic clinical training in virtual reality. Proceedings of the International Conference on Artificial Intelligence in Education."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1080\/0142159X.2022.2130216","article-title":"Artificial intelligence in virtual standardized patients: Combining natural language understanding and rule based dialogue management to improve conversational fidelity","volume":"45","author":"Maicher","year":"2022","journal-title":"Med. Teach."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Prange, A., Barz, M., and Sonntag, D. (2017, January 19\u201325). Speech-based medical decision support in vr using a deep neural network. Proceedings of the IJCAI, Melbourne, Australia.","DOI":"10.24963\/ijcai.2017\/777"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1111\/acem.13308","article-title":"Using virtual reality simulation environments to assess competence for emergency medicine learners","volume":"25","author":"McGrath","year":"2018","journal-title":"Acad. Emerg. Med."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"62","DOI":"10.11648\/j.ajcst.20240703.11","article-title":"Conversational AI and Chatbots: Enhancing User Experience on Websites","volume":"11","author":"Dobbala","year":"2024","journal-title":"Am. J. Comput. Sci. Technol."},{"key":"ref_20","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4\u20139). Attention is All You Need. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA."},{"key":"ref_21","unstructured":"Kenton, J.D.M.W.C., and Toutanova, L.K. (2019, January 2\u20137). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the naacL-HLT, Minneapolis, MN, USA."},{"key":"ref_22","first-page":"36","article-title":"Speech to text and text to speech recognition systems-Areview","volume":"20","author":"Trivedi","year":"2018","journal-title":"IOSR J. Comput. Eng."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Abdul-Kader, S.A., and Woods, J. (2015). Survey on chatbot design techniques in speech conversation systems. Int. J. Adv. Comput. Sci. Appl., 6.","DOI":"10.14569\/IJACSA.2015.060712"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Kumari, S., Naikwadi, Z., Akole, A., and Darshankar, P. (2020, January 2\u20134). Enhancing college chat bot assistant with the help of richer human computer interaction and speech recognition. Proceedings of the 2020 International Conference on Electronics and Sustainable Communication Systems (ICESC), Coimbatore, India.","DOI":"10.1109\/ICESC48915.2020.9155951"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1","DOI":"10.2352\/ISSN.2470-1173.2021.3.MOBMU-035","article-title":"Integration of NLP and speech-to-text applications with chatbots","volume":"33","author":"Inupakutika","year":"2021","journal-title":"Electron. Imaging"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1080\/07853890.2018.1560068","article-title":"Extending medical interfaces towards virtual reality and augmented reality","volume":"51","author":"Lopes","year":"2019","journal-title":"Ann. Med."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"915","DOI":"10.21037\/atm-22-3115","article-title":"Application of VR and 3D printing in liver reconstruction","volume":"10","author":"Yang","year":"2022","journal-title":"Ann. Transl. Med."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1074","DOI":"10.21037\/atm-21-512","article-title":"Virtual reality and 3D printing improve preoperative visualization of 3D liver reconstructions\u2014Results from a preclinical comparison of presentation modalities and user\u2019s preference","volume":"9","author":"Huettl","year":"2021","journal-title":"Ann. Transl. Med."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1090","DOI":"10.1002\/lary.31874","article-title":"A Randomized Pilot Trial of Virtual Reality Surgical Planning for Head and Neck Oncologic Resection","volume":"135","author":"Nunes","year":"2024","journal-title":"Laryngoscope"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Isikay, I., Cekic, E., Baylarov, B., Tunc, O., and Hanalioglu, S. (2024). Narrative review of patient-specific 3D visualization and reality technologies in skull base neurosurgery: Enhancements in surgical training, planning, and navigation. Front. Surg., 11.","DOI":"10.3389\/fsurg.2024.1427844"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"102561","DOI":"10.1016\/j.ijhcs.2020.102561","article-title":"Evaluation of haptic virtual reality user interfaces for medical marking on 3D models","volume":"147","author":"Li","year":"2021","journal-title":"Int. J. -Hum.-Comput. Stud."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Rantamaa, H.R., Kangas, J., Kumar, S.K., Mehtonen, H., J\u00e4rnstedt, J., and Raisamo, R. (2023). Comparison of a vr stylus with a controller, hand tracking, and a mouse for object manipulation and medical marking tasks in virtual reality. Appl. Sci., 13.","DOI":"10.3390\/app13042251"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"634","DOI":"10.1109\/TMRB.2022.3175095","article-title":"A Virtual Reality and haptic simulator for ultrasound-guided needle insertion","volume":"4","author":"Alamilla","year":"2022","journal-title":"IEEE Trans. Med. Robot. Bionics"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1981","DOI":"10.1007\/s11548-022-02685-1","article-title":"Evaluation of voice commands for mode change in virtual reality implant planning procedure","volume":"17","author":"Rantamaa","year":"2022","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Yang, J., Chan, M., Uribe-Quevedo, A., Kapralos, B., Jaimes, N., and Dubrowski, A. (2020, January 7\u201310). Prototyping virtual reality interactions in medical simulation employing speech recognition. Proceedings of the 2020 22nd Symposium on Virtual and Augmented Reality (SVR), Porto de Galinhas, Brazil.","DOI":"10.1109\/SVR51698.2020.00059"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1145\/2541883.2541899","article-title":"Touchless interaction in surgery","volume":"57","author":"Gonzalez","year":"2014","journal-title":"Commun. ACM"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1093\/iwcomp\/iwaa002","article-title":"Gaze-based kinaesthetic interaction for virtual reality","volume":"32","author":"Li","year":"2020","journal-title":"Interact. Comput."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Rakkolainen, I., Farooq, A., Kangas, J., Hakulinen, J., Rantala, J., Turunen, M., and Raisamo, R. (2021). Technologies for multimodal interaction in extended reality\u2014A scoping review. Multimodal Technol. Interact., 5.","DOI":"10.3390\/mti5120081"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"104069","DOI":"10.1016\/j.cag.2024.104069","article-title":"Voice user interfaces for effortless navigation in medical virtual reality environments","volume":"124","author":"Hombeck","year":"2024","journal-title":"Comput. Graph."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Chen, L., Cai, Y., Wang, R., Ding, S., Tang, Y., Hansen, P., and Sun, L. (2024, January 16\u201321). Supporting text entry in virtual reality with large language models. Proceedings of the 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR), Orlando, FL, USA.","DOI":"10.1109\/VR58804.2024.00073"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1007\/s11280-024-01276-1","article-title":"When large language models meet personalization: Perspectives of challenges and opportunities","volume":"27","author":"Chen","year":"2024","journal-title":"World Wide Web"},{"key":"ref_42","first-page":"1","article-title":"Explainability for large language models: A survey","volume":"15","author":"Zhao","year":"2024","journal-title":"ACM Trans. Intell. Syst. Technol."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"2122","DOI":"10.1109\/TVCG.2014.2346452","article-title":"The effects of interactive latency on exploratory visual analysis","volume":"20","author":"Liu","year":"2014","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_44","unstructured":"Gallent-Iglesias, D., Serantes-Raposo, S., Botana, I.L.R., Gonz\u00e1lez-V\u00e1zquez, S., and Fernandez-Gra\u00f1a, P.M. (2023, January 27\u201329). IVAMED: Intelligent Virtual Assistant for Medical Diagnosis. Proceedings of the SEPLN (Projects and Demonstrations), Ja\u00e9n, Spain."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Trivedi, K.S. (2023). Fundamentals of Natural Language Processing. Microsoft Azure AI Fundamentals Certification Companion: Guide to Prepare for the AI-900 Exam, Springer.","DOI":"10.1007\/978-1-4842-9221-1"}],"container-title":["Multimodal Technologies and Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2414-4088\/9\/6\/50\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:40:47Z","timestamp":1760031647000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2414-4088\/9\/6\/50"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,26]]},"references-count":45,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2025,6]]}},"alternative-id":["mti9060050"],"URL":"https:\/\/doi.org\/10.3390\/mti9060050","relation":{},"ISSN":["2414-4088"],"issn-type":[{"value":"2414-4088","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,26]]}}}