{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,20]],"date-time":"2025-07-20T04:30:24Z","timestamp":1752985824404,"version":"3.41.0"},"reference-count":51,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2021,3,19]],"date-time":"2021-03-19T00:00:00Z","timestamp":1616112000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100014013","name":"UK Research and Innovation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100014013","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2021,3,19]]},"abstract":"<jats:p>A new generation of multimodal interfaces and interactions is emerging. Drawing on the principles of Sensory Substitution and Augmentation Devices (SSADs), these new interfaces offer the potential for rich, immersive human-computer interactions, but are difficult to design well, and take time to master, creating significant barriers towards wider adoption. Following a review of the literature surrounding existing SSADs, their metrics for success and their growing influence on interface design in Human Computer Interaction, we present a medium term (4-day) study comparing the effectiveness of various combinations of visual and haptic feedback (sensory concurrencies) in preparing users to perform a virtual maze navigation task using haptic feedback alone. Participants navigated 12 mazes in each of 3 separate sessions under a specific combination of visual and haptic feedback, before performing the same task using the haptic feedback alone. Visual sensory deprivation was shown to be inferior to visual &amp; haptic concurrency in enabling haptic signal comprehension, while a new hybridized condition combining reduced visual feedback with the haptic signal was shown to be superior. Potential explanations for the effectiveness of the hybrid mechanism are explored, and the scope and implications of its generalization to new sensory interfaces is presented.<\/jats:p>","DOI":"10.1145\/3448102","type":"journal-article","created":{"date-parts":[[2021,3,30]],"date-time":"2021-03-30T18:56:41Z","timestamp":1617130601000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Investigating the Effect of Sensory Concurrency on Learning Haptic Spatiotemporal Signals"],"prefix":"10.1145","volume":"5","author":[{"given":"Iain","family":"Carson","sequence":"first","affiliation":[{"name":"University of St Andrews"}]},{"given":"Aaron","family":"Quigley","sequence":"additional","affiliation":[{"name":"UNSW Sydney, New South Wales, AU"}]},{"given":"Loraine","family":"Clarke","sequence":"additional","affiliation":[{"name":"University of St Andrews"}]},{"given":"Uta","family":"Hinrichs","sequence":"additional","affiliation":[{"name":"University of St Andrews"}]}],"member":"320","published-online":{"date-parts":[[2021,3,30]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1068\/p5631"},{"volume-title":"Brain mechanisms in sensory substitution","author":"Rita Paul","key":"e_1_2_1_2_1","unstructured":"Paul Bach-y Rita . 1972. Brain mechanisms in sensory substitution . Academic Press , New York . 182 pages. Paul Bach-y Rita. 1972. Brain mechanisms in sensory substitution. Academic Press, New York. 182 pages."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2802083.2802088"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neubiorev.2016.05.026"},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the Fifth Conference on Australasian User Interface -","volume":"28","author":"Brewster Stephen","unstructured":"Stephen Brewster and Lorna M. Brown . 2004. Tactons: Structured Tactile Messages for Non-Visual Information Display . In Proceedings of the Fifth Conference on Australasian User Interface - Volume 28 (Dunedin, New Zealand) (AUIC '04). Australian Computer Society, Inc., AUS, 15--23. Stephen Brewster and Lorna M. Brown. 2004. Tactons: Structured Tactile Messages for Non-Visual Information Display. In Proceedings of the Fifth Conference on Australasian User Interface - Volume 28 (Dunedin, New Zealand) (AUIC '04). Australian Computer Society, Inc., AUS, 15--23."},{"volume-title":"Proceedings of the 4th Augmented Human International Conference","author":"Carton Anthony","key":"e_1_2_1_6_1","unstructured":"Anthony Carton and Lucy E. Dunne . 2013. Tactile Distance Feedback for Firefighters: Design and Preliminary Evaluation of a Sensory Augmentation Glove . In Proceedings of the 4th Augmented Human International Conference ( Stuttgart, Germany) (AH '13). Association for Computing Machinery, New York, NY, USA, 58--64. https:\/\/doi.org\/10.1145\/2459236.2459247 Anthony Carton and Lucy E. Dunne. 2013. Tactile Distance Feedback for Firefighters: Design and Preliminary Evaluation of a Sensory Augmentation Glove. In Proceedings of the 4th Augmented Human International Conference (Stuttgart, Germany) (AH '13). Association for Computing Machinery, New York, NY, USA, 58--64. https:\/\/doi.org\/10.1145\/2459236.2459247"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1179849.1179892"},{"key":"e_1_2_1_8_1","doi-asserted-by":"crossref","unstructured":"Jennifer Chen Pablo Castillo Robert Turcotte Ali Israr and Frances Lau. 2018. Feeling Speech on the Arm. https:\/\/doi.org\/10.1145\/3170427.3186537  Jennifer Chen Pablo Castillo Robert Turcotte Ali Israr and Frances Lau. 2018. Feeling Speech on the Arm. https:\/\/doi.org\/10.1145\/3170427.3186537","DOI":"10.1145\/3170427.3186537"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03210362"},{"volume-title":"Perception and its modalities","author":"Deroy Ophelia","key":"e_1_2_1_10_1","unstructured":"Ophelia Deroy and Malika Auvray . 2014. Perception and its modalities . Oxford University Press Oxford , UK , 198 Madison Avenue, New York, NY 10016, Chapter A crossmodal perspective on sensory substitution, 327--349. Ophelia Deroy and Malika Auvray. 2014. Perception and its modalities. Oxford University Press Oxford, UK, 198 Madison Avenue, New York, NY 10016, Chapter A crossmodal perspective on sensory substitution, 327--349."},{"key":"e_1_2_1_11_1","volume-title":"Paradiso","author":"Dublon Gershon","year":"2012","unstructured":"Gershon Dublon and Joseph A . Paradiso . 2012 . Tongueduino : Hackable, High-Bandwidth Sensory Augmentation. In CHI '12 Extended Abstracts on Human Factors in Computing Systems (Austin, Texas, USA) (CHI EA '12). Association for Computing Machinery , New York, NY, USA, 1453--1454. https:\/\/doi.org\/10.1145\/2212776.2212482 Gershon Dublon and Joseph A. Paradiso. 2012. Tongueduino: Hackable, High-Bandwidth Sensory Augmentation. In CHI '12 Extended Abstracts on Human Factors in Computing Systems (Austin, Texas, USA) (CHI EA '12). Association for Computing Machinery, New York, NY, USA, 1453--1454. https:\/\/doi.org\/10.1145\/2212776.2212482"},{"volume-title":"Elemente der psychophysik","author":"Fechner Gustav Theodor","key":"e_1_2_1_12_1","unstructured":"Gustav Theodor Fechner . 1860. Elemente der psychophysik . Vol. 2 . Breitkopf u. H\u00e4rtel, Leipzig . Gustav Theodor Fechner. 1860. Elemente der psychophysik. Vol. 2. Breitkopf u. H\u00e4rtel, Leipzig."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1080\/00140130903023709"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TOH.2011.57"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neubiorev.2016.06.043"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1163\/22134808-00002519"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.5555\/2597328.2597332"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1037\/e577632012-009"},{"key":"e_1_2_1_19_1","unstructured":"HTC. 2020. VIVE Cosmos Overview. HTC. https:\/\/www.vive.com\/us\/product\/vive-cosmos\/overview\/  HTC. 2020. VIVE Cosmos Overview. HTC. https:\/\/www.vive.com\/us\/product\/vive-cosmos\/overview\/"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.humov.2008.12.004"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/TOH.2013.2"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-444-53355-5.00008-7"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.3233\/RNN-160647"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.3390\/info11070353"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3384657.3384775"},{"key":"e_1_2_1_26_1","unstructured":"Ben Kuchera. 2017. How the Nintendo Switch's HD rumble makes Tumbleseed feel real. Polygon. https:\/\/www.polygon.com\/2017\/5\/1\/15499328\/tumbleseed-hd-rumble-nintendo-switch  Ben Kuchera. 2017. How the Nintendo Switch's HD rumble makes Tumbleseed feel real. Polygon. https:\/\/www.polygon.com\/2017\/5\/1\/15499328\/tumbleseed-hd-rumble-nintendo-switch"},{"key":"e_1_2_1_27_1","volume-title":"Learning New Sensorimotor Contingencies: Effects of Long-Term Use of Sensory Augmentation on the Brain and Conscious Perception. PLOS ONE 11, 12 (12","author":"K\u00f6nig Sabine U.","year":"2016","unstructured":"Sabine U. K\u00f6nig , Frank Schumann , Johannes Keyser , Caspar Goeke , Carina Krause , Susan Wache , Aleksey Lytochkin , Manuel Ebert , Vincent Brunsch , Basil Wahn , Kai Kaspar , Saskia K. Nagel , Tobias Meilinger , Heinrich B\u00fclthoff , Thomas Wolbers , Christian B\u00fcchel , and Peter K\u00f6nig . 2016. Learning New Sensorimotor Contingencies: Effects of Long-Term Use of Sensory Augmentation on the Brain and Conscious Perception. PLOS ONE 11, 12 (12 2016 ), 1--35. https:\/\/doi.org\/10.1371\/journal.pone.0166647 Sabine U. K\u00f6nig, Frank Schumann, Johannes Keyser, Caspar Goeke, Carina Krause, Susan Wache, Aleksey Lytochkin, Manuel Ebert, Vincent Brunsch, Basil Wahn, Kai Kaspar, Saskia K. Nagel, Tobias Meilinger, Heinrich B\u00fclthoff, Thomas Wolbers, Christian B\u00fcchel, and Peter K\u00f6nig. 2016. Learning New Sensorimotor Contingencies: Effects of Long-Term Use of Sensory Augmentation on the Brain and Conscious Perception. PLOS ONE 11, 12 (12 2016), 1--35. https:\/\/doi.org\/10.1371\/journal.pone.0166647"},{"key":"e_1_2_1_28_1","volume-title":"Sensory Substitution: Limits and Perspectives. Touching for Knowing Cognitive psychology of haptic manual perception 19","author":"Lenay Charles","year":"2003","unstructured":"Charles Lenay , Olivier Gapenne , Sylvain Hanneton , Catherine Marque , and Christelle Genou\u00eblle . 2003 . Sensory Substitution: Limits and Perspectives. Touching for Knowing Cognitive psychology of haptic manual perception 19 (2003), 275--292. http:\/\/books.google.com\/books?hl=fr{&}lr={&}id=GSOhMpdyobAC{&}pgis=1 Charles Lenay, Olivier Gapenne, Sylvain Hanneton, Catherine Marque, and Christelle Genou\u00eblle. 2003. Sensory Substitution: Limits and Perspectives. Touching for Knowing Cognitive psychology of haptic manual perception 19 (2003), 275--292. http:\/\/books.google.com\/books?hl=fr{&}lr={&}id=GSOhMpdyobAC{&}pgis=1"},{"key":"e_1_2_1_29_1","volume-title":"Cross-sensory transfer of sensory-motor information: visuomotor learning affects performance on an audiomotor task, using sensory-substitution. Scientific reports 2","author":"Levy-Tzedek Shelly","year":"2012","unstructured":"Shelly Levy-Tzedek , Itai Novick , Roni Arbel , Sami Abboud , Shachar Maidenbaum , Eilon Vaadia , and Amir Amedi . 2012. Cross-sensory transfer of sensory-motor information: visuomotor learning affects performance on an audiomotor task, using sensory-substitution. Scientific reports 2 ( 2012 ), 949. Shelly Levy-Tzedek, Itai Novick, Roni Arbel, Sami Abboud, Shachar Maidenbaum, Eilon Vaadia, and Amir Amedi. 2012. Cross-sensory transfer of sensory-motor information: visuomotor learning affects performance on an audiomotor task, using sensory-substitution. Scientific reports 2 (2012), 949."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1177\/1059712319875147"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2019.03.004"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0098801"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neubiorev.2013.11.007"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0072555"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/10.121642"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1167\/12.11.26"},{"key":"e_1_2_1_37_1","unstructured":"David Myers. 2004. Tactile Situational Awareness System ( TSAS ). 5 pages. https:\/\/www.sbir.gov\/sbirsearch\/detail\/124876  David Myers. 2004. Tactile Situational Awareness System ( TSAS ). 5 pages. https:\/\/www.sbir.gov\/sbirsearch\/detail\/124876"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2560\/2\/4\/R02"},{"key":"e_1_2_1_39_1","unstructured":"Neosensory. 2020. Neosensory - Experience sound as touch and gain new awareness. Neosensory. https:\/\/neosensory.com\/  Neosensory. 2020. Neosensory - Experience sound as touch and gain new awareness. Neosensory. https:\/\/neosensory.com\/"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00221-015-4346-1"},{"volume-title":"Oculus Rift: VR Headset for VR Ready PCs. Oculus. https:\/\/www.oculus.com\/rift\/","year":"2020","key":"e_1_2_1_41_1","unstructured":"Oculus. 2020 . Oculus Rift: VR Headset for VR Ready PCs. Oculus. https:\/\/www.oculus.com\/rift\/ Oculus. 2020. Oculus Rift: VR Headset for VR Ready PCs. Oculus. https:\/\/www.oculus.com\/rift\/"},{"key":"e_1_2_1_42_1","volume-title":"A sensorimotor account of vision and visual consciousness. Behavioral and brain sciences 24, 5","author":"O'Regan J Kevin","year":"2001","unstructured":"J Kevin O'Regan and Alva No\u00eb . 2001. A sensorimotor account of vision and visual consciousness. Behavioral and brain sciences 24, 5 ( 2001 ), 939. J Kevin O'Regan and Alva No\u00eb. 2001. A sensorimotor account of vision and visual consciousness. Behavioral and brain sciences 24, 5 (2001), 939."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neubiorev.2012.11.017"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0001840"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/TBME.2018.2836672"},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA '15 2","author":"Martinez Carlos Sainz","year":"2015","unstructured":"Carlos Sainz Martinez and Faustina Hwang . 2015 . TESSA -Toolkit for experimentation with multimodal sensory substitution and augmentation . Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA '15 2 (2015), 259--262. https:\/\/doi.org\/10.1145\/2702613.2725451 Carlos Sainz Martinez and Faustina Hwang. 2015. TESSA -Toolkit for experimentation with multimodal sensory substitution and augmentation. Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA '15 2 (2015), 259--262. https:\/\/doi.org\/10.1145\/2702613.2725451"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02919463"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.concog.2009.10.006"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neubiorev.2012.07.007"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341163.3346931"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173966"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3448102","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3448102","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:24:59Z","timestamp":1750195499000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3448102"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,19]]},"references-count":51,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,3,19]]}},"alternative-id":["10.1145\/3448102"],"URL":"https:\/\/doi.org\/10.1145\/3448102","relation":{},"ISSN":["2474-9567"],"issn-type":[{"type":"electronic","value":"2474-9567"}],"subject":[],"published":{"date-parts":[[2021,3,19]]},"assertion":[{"value":"2021-03-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}