{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T21:16:19Z","timestamp":1778274979692,"version":"3.51.4"},"reference-count":50,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2018,11,26]],"date-time":"2018-11-26T00:00:00Z","timestamp":1543190400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000155","name":"Social Sciences and Humanities Research Council of Canada","doi-asserted-by":"publisher","award":["Insight Grant"],"award-info":[{"award-number":["Insight Grant"]}],"id":[{"id":"10.13039\/501100000155","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Informatics"],"abstract":"<jats:p>Touch plays a fundamental role in our daily interactions, allowing us to interact with and perceive objects and their spatial properties. Despite its importance in the real-world, touch is often ignored in virtual environments. However, accurately simulating the sense of touch is difficult, requiring the use of high-fidelity haptic devices that are cost-prohibitive. Lower fidelity consumer-level haptic devices are becoming more widespread, yet are generally limited in perceived fidelity and the range of motion (degrees of freedom) required to realistically simulate many tasks. Studies into sound and vision suggest that the presence or absence of sound can influence task performance. Here, we explore whether the presence or absence of contextually relevant sound cues influences the performance of a simple haptic drilling task. Although the results of this study do not show any statistically significant difference in task performance with general (task-irrelevant) sound, we discuss how this is a necessary step in understanding the role of sound on haptic perception.<\/jats:p>","DOI":"10.3390\/informatics5040043","type":"journal-article","created":{"date-parts":[[2018,11,27]],"date-time":"2018-11-27T03:31:33Z","timestamp":1543289493000},"page":"43","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Multimodal Interaction of Contextual and Non-Contextual Sound and Haptics in Virtual Simulations"],"prefix":"10.3390","volume":"5","author":[{"given":"Mohammed","family":"Melaisi","sequence":"first","affiliation":[{"name":"Software Informatics Research Centre, University of Ontario Institute of Technology, Oshawa, ON L1H 7K4, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2351-3744","authenticated-orcid":false,"given":"David","family":"Rojas","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, University of Toronto, Toronto, ON M5S 1A1, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bill","family":"Kapralos","sequence":"additional","affiliation":[{"name":"Software Informatics Research Centre, University of Ontario Institute of Technology, Oshawa, ON L1H 7K4, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6728-3848","authenticated-orcid":false,"given":"Alvaro","family":"Uribe-Quevedo","sequence":"additional","affiliation":[{"name":"Software Informatics Research Centre, University of Ontario Institute of Technology, Oshawa, ON L1H 7K4, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Karen","family":"Collins","sequence":"additional","affiliation":[{"name":"Department of Communication Arts, University of Waterloo, Waterloo, ON N2L 3G1, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2018,11,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1108\/02602280410515770","article-title":"Haptic interfaces and devices","volume":"24","author":"Hayward","year":"2004","journal-title":"Sens. Rev."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Burdea, G., and Coiffet, P. (2003). Virtual Reality Technology, John Wiley & Sons.","DOI":"10.1162\/105474603322955950"},{"key":"ref_3","first-page":"1","article-title":"Haptics to improve task performance in people with disabilities: A review of previous studies and a guide to future research with children with disabilities","volume":"3","author":"Jafari","year":"2016","journal-title":"J. Rehabil. Assist. Technol. Eng."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1146\/annurev-control-060117-105043","article-title":"Haptics: The present and future of artificial touch sensation","volume":"1","author":"Culbertson","year":"2018","journal-title":"Annu. Rev. Control Robot. Auton. Syst."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"El Saddik, A., Orozco, M., Eid, M., and Cha, J. (2011). Springer Series on Touch and Haptic Systems. Haptic Technologies: Bringing Touch to Multimedia, Springer.","DOI":"10.1007\/978-3-642-22658-8"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1109\/TOH.2010.19","article-title":"The role of haptics in medical training simulators: A survey of the state of the art","volume":"4","author":"Coles","year":"2011","journal-title":"IEEE Trans. Haptics"},{"key":"ref_7","first-page":"32","article-title":"The future of selected fidelity in training devices","volume":"35","author":"Andrews","year":"1996","journal-title":"Educ. Technol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"867","DOI":"10.3109\/0142159X.2012.714886","article-title":"Comparative effectiveness of instructional design features in simulation-based education: Systematic review and meta-analysis","volume":"35","author":"Cook","year":"2013","journal-title":"Med. Teach."},{"key":"ref_9","unstructured":"Webster, R., Haluck, R.S., Zoppetti, G., Benson, A., Boyd, J., Charles, N., Reeser, J., and Sampson, S. (2003, January 25\u201327). A haptic surgical simulator for laparoscopic cholecystectomy using real-time deformable organs. Proceedings of the IASTED International Conference Biomedical Engineering, Salzburg, Austria."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1108\/WJSTSD-07-2016-0047","article-title":"Low-end haptic devices for knee bone drilling in a serious game","volume":"14","author":"Nguyen","year":"2017","journal-title":"World J. Sci. Technol. Sustain. Dev."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1445","DOI":"10.1068\/p5843","article-title":"Simultaneous and independent acquisition of multisensory and unisensory associations","volume":"36","author":"Seitz","year":"2007","journal-title":"Perception"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"55","DOI":"10.3758\/BF03199367","article-title":"The \u201cventriloquist effect\u201d: Visual dominance or response bias?","volume":"18","author":"Choe","year":"1975","journal-title":"Percept. Psychophys."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"746","DOI":"10.1038\/264746a0","article-title":"Hearing lips and seeing voices","volume":"264","author":"McGurk","year":"1976","journal-title":"Nature"},{"key":"ref_14","unstructured":"K\u00f6hler, W. (1947). Gestalt Psychology, Liveright Publishing. [2nd ed.]."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1016\/j.cognition.2014.03.015","article-title":"Touching words is not enough: How visual experience influences haptic\u2013auditory associations in the \u2018\u2018Bouba\u2013Kiki\u2019\u2019 effect","volume":"132","author":"Fryer","year":"2014","journal-title":"Cognition"},{"key":"ref_16","unstructured":"Collins, K. (2008). Left in the dark: Playing computer games with the sound turned off. From Pac-Man to Pop Music: Interactive Audio in Games and New Media, Ashgate Publishing Group."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"337","DOI":"10.2114\/jpa.23.337","article-title":"Immersion in the virtual environment: The effect of a musical score on the video gaming experience","volume":"23","author":"Lipscomb","year":"2004","journal-title":"J. Physiol. Anthropol. Appl. Hum. Sci."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"402","DOI":"10.1177\/1046878116688236","article-title":"The impact of health-related sounds on player experience","volume":"48","author":"Robb","year":"2017","journal-title":"Simul. Gaming"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"2371","DOI":"10.1016\/j.lfs.2004.11.011","article-title":"Physiological stress response to video-game playing: The contribution of built-in music","volume":"76","year":"2005","journal-title":"Life Sci."},{"key":"ref_20","unstructured":"Shilling, R., Zyda, M., and Wardynski, E.C. (2002). Introducing emotion into military simulation and videogame design: America\u2019s army: Operations and VIRTE. Game-On."},{"key":"ref_21","unstructured":"Sanders, R., and Scorgie, R. (2002). The Effect of Sound Delivery Methods on the User\u2019s Sense of Presence in a Virtual Environment. [Master\u2019s Thesis, Naval Postgraduate School]. (Unpublished)."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1016\/j.intcom.2010.04.005","article-title":"More than a feeling: Measurement of sonic user experience and psychophysiology in a first-person shooter game","volume":"22","author":"Nacke","year":"2010","journal-title":"Interact. Comput."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1347","DOI":"10.1007\/s00464-009-0772-0","article-title":"The effect of define auditory conditions versus mental loading on the laparoscopic motor skill performance of experts","volume":"24","author":"Conrad","year":"2010","journal-title":"Surg. Endosc."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"557","DOI":"10.1162\/105474600300040385","article-title":"Interactions in perceived quality of auditory-visual displays","volume":"9","author":"Storms","year":"2000","journal-title":"Presence Teleop. Virt."},{"key":"ref_25","unstructured":"Larsson, P., V\u00e4stj\u00e4ll, D., and Kleiner, M. (2003, January 23\u201325). On the quality of experience: A multi-modal approach to perceptual ego-motion and sensed presence in virtual environments. Proceedings of the First International Speech Communications Association Tutorial and Research Workshop on Auditory Quality of Systems, Herne, Germany."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1658349.1658350","article-title":"Bimodal perception of audio-visual material properties for virtual environments","volume":"7","author":"Bonneel","year":"2010","journal-title":"ACM Trans. Appl. Percept."},{"key":"ref_27","first-page":"57","article-title":"Maintaining frame rate perception in interactive environments by exploiting audio-visual cross-modal interaction","volume":"27","author":"Debattista","year":"2010","journal-title":"Vis. Comput."},{"key":"ref_28","first-page":"1572","article-title":"Developing effective serious games: The effect of ambient auditory conditions on visual fidelity perception in stereoscopic 3D","volume":"43","author":"Rojas","year":"2013","journal-title":"IEEE Trans. Syst. Man Cybern. B"},{"key":"ref_29","unstructured":"Rojas, D., Cowan, B., Kapralos, B., Collins, K., and Dubrowski, A. (2013, January 20\u201322). The effect of contextual sound cues on visual fidelity perception. Proceedings of the Medicine Meets Virtual Reality, Manhattan Beach, CA, USA."},{"key":"ref_30","unstructured":"Lam, J., Kapralos, B., Kanev, K., Collins, K., Hogue, A., and Jenkin, M. (2015, January 26\u201329). The effect of sound on visual realism perception and task completion time in a cel-shaded serious gaming virtual environment. Proceedings of the Qomex 2015 International Workshop on Quality of MultiMedia Experience, Costa Navarino, Greece."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"875","DOI":"10.1126\/science.1097011","article-title":"That\u2019s my hand! Activity in premotor cortex reflects feeling of ownership of a limb","volume":"305","author":"Ehrsson","year":"2004","journal-title":"Science"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Jousm\u00e4ki, V., and Hari, R. (1998). Parchment-skin illusion: Sound-biased touch. Curr. Biol., 8.","DOI":"10.1016\/S0960-9822(98)70120-4"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1007\/s00221-002-1164-z","article-title":"Audiotactile interactions in roughness perception","volume":"146","author":"Guest","year":"2002","journal-title":"Exp. Brain Res."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"DiFranco, D.E., Beauregard, G.L., and Srinivasan, M.A. (1997, January 17\u201322). The effect of auditory cues on the haptic perception of stiffness in virtual environments. Proceedings of the ASME International Mechanical Engineering Congress and Exposition, Dallas, TX, USA.","DOI":"10.1115\/IMECE1997-0372"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Jang, I., and Lee, D. (2014, January 23\u201326). On utilizing pseudo-haptics for cutaneous fingertip haptic device. Proceedings of the IEEE Haptics Symposium, Houston, TX, USA.","DOI":"10.1109\/HAPTICS.2014.6775529"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1162\/pres.18.1.39","article-title":"Simulating haptic feedback using vision: A survey of research and applications of pseudo-haptic feedback","volume":"18","year":"2009","journal-title":"Presence Teleop. Virt."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Ban, Y., Narumi, T., Tanikawa, T., and Hirose, M. (2014, January 11\u201313). Displaying shapes with various types of surfaces using visuo-haptic interaction. Proceedings of the ACM Symposium on Virtual Reality Software Technology, Edinburgh, UK.","DOI":"10.1145\/2671015.2671028"},{"key":"ref_38","unstructured":"Buurman, G.M. (2017). Hear me Interact. Total Interaction: Theory and Practice of a New Paradigm for the Design Disciplines, Birkh\u00e4user."},{"key":"ref_39","unstructured":"Ballas, J. (2007, January 29\u201330). Self-produced sound: Tightly binding haptics and audio. Proceedings of the International Workshop on Haptic and Audio Interaction Design, Seoul, Korea."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Adelstein, B.D., Begault, D.R., Anderson, M.R., and Wenzel, E.M. (2003, January 5\u20137). Sensitivity to haptic-audio asynchrony. Proceedings of the International Conference on Multimodal Interfaces, Vancouver, BC, Canada.","DOI":"10.1145\/958432.958448"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1007\/3-540-44589-7_13","article-title":"The effective combination of haptic and auditory textural information","volume":"20158","author":"McGee","year":"2001","journal-title":"Lect. Notes Comput. Sci."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1097","DOI":"10.1093\/cercor\/bhs076","article-title":"Multisensory interactions between auditory and haptic object recognition","volume":"23","author":"Kassuba","year":"2013","journal-title":"Cereb. Cortex"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Frid, E., Moll, J., Bresin, R., and Salln\u00e4s Pysander, E. (2018). Haptic Feedback combined with movement sonificiation using a friction sound improves task performance in a virtual throwing task. J. Multimodal User Interfaces, 1\u201312.","DOI":"10.1007\/s12193-018-0264-4"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"4002","DOI":"10.1121\/1.3699205","article-title":"Identification of walked-upon materials in auditory, kinesthetic, haptic, and audio-haptic conditions","volume":"131","author":"Giordano","year":"2012","journal-title":"J. Acoust. Soc. Am."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1007\/s10055-006-0028-4","article-title":"Influence of multisensory feedback on haptic accessibility tasks","volume":"10","author":"Hernantes","year":"2006","journal-title":"Virtual Real."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Collins, K. (2008). Game Sound, MIT Press.","DOI":"10.7551\/mitpress\/7909.001.0001"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Collins, K. (2013). Playing with Sound, MIT Press.","DOI":"10.7551\/mitpress\/9442.001.0001"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Melaisi, M., Nguyen, M., Uribe, A., and Kapralos, B. (2017, January 25\u201328). The effect of sound on haptic fidelity perception. Proceedings of the IEEE Global Engineering Education Conference, Athens, Greece.","DOI":"10.1109\/EDUCON.2017.7942926"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1002\/j.0022-0337.2016.80.1.tb06061.x","article-title":"Development of a drilling simulator for dental implant surgery","volume":"80","author":"Kinoshita","year":"2016","journal-title":"J. Dent. Educ."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Wang, Q., Qin, J., Wang, W., Shan, J., Zhang, J., Liu, X., and Heng, P.-A. (2015, January 21\u201324). Haptic rendering of drilling process in orthopedic surgical simulation based on the volumetric object. Proceedings of the IEEE International Conference on Digital Signal Processing, Singapore.","DOI":"10.1109\/ICDSP.2015.7252049"}],"container-title":["Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-9709\/5\/4\/43\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T15:32:04Z","timestamp":1760196724000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-9709\/5\/4\/43"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,11,26]]},"references-count":50,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2018,12]]}},"alternative-id":["informatics5040043"],"URL":"https:\/\/doi.org\/10.3390\/informatics5040043","relation":{},"ISSN":["2227-9709"],"issn-type":[{"value":"2227-9709","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,11,26]]}}}