{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T14:10:57Z","timestamp":1780668657785,"version":"3.54.1"},"reference-count":32,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2021,8,12]],"date-time":"2021-08-12T00:00:00Z","timestamp":1628726400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100014438","name":"Business Finland","doi-asserted-by":"publisher","award":["251\/31\/2018"],"award-info":[{"award-number":["251\/31\/2018"]}],"id":[{"id":"10.13039\/501100014438","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MTI"],"abstract":"<jats:p>Developments in sensor technology, artificial intelligence, and network technologies like 5G has made remote operation a valuable method of controlling various types of machinery. The benefits of remote operations come with an opportunity to access hazardous environments. The major limitation of remote operation is the lack of proper sensory feedback from the machine, which in turn negatively affects situational awareness and, consequently, may risk remote operations. This article explores how to improve situational awareness via multimodal feedback (visual, auditory, and haptic) and studies how it can be utilized to communicate warnings to remote operators. To reach our goals, we conducted a controlled, within-subjects experiment in eight conditions with twenty-four participants on a simulated remote driving system. Additionally, we gathered further insights with a UX questionnaire and semi-structured interviews. Gathered data showed that the use of multimodal feedback positively affected situational awareness when driving remotely. Our findings indicate that the combination of added haptic and visual feedback was considered the best feedback combination to communicate the slipperiness of the road. We also found that the feeling of presence is an important aspect of remote driving tasks, and a requested one, especially by those with more experience in operating real heavy machinery.<\/jats:p>","DOI":"10.3390\/mti5080044","type":"journal-article","created":{"date-parts":[[2021,8,12]],"date-time":"2021-08-12T10:54:41Z","timestamp":1628765681000},"page":"44","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Multimodal Warnings in Remote Operation: The Case Study on Remote Driving"],"prefix":"10.3390","volume":"5","author":[{"given":"Pekka","family":"Kallioniemi","sequence":"first","affiliation":[{"name":"The Unit of Computing Sciences, Tampere University, 33100 Tampere, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3799-3662","authenticated-orcid":false,"given":"Alisa","family":"Burova","sequence":"additional","affiliation":[{"name":"The Unit of Computing Sciences, Tampere University, 33100 Tampere, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John","family":"M\u00e4kel\u00e4","sequence":"additional","affiliation":[{"name":"The Unit of Computing Sciences, Tampere University, 33100 Tampere, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tuuli","family":"Keskinen","sequence":"additional","affiliation":[{"name":"The Unit of Computing Sciences, Tampere University, 33100 Tampere, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kimmo","family":"Ronkainen","sequence":"additional","affiliation":[{"name":"The Unit of Computing Sciences, Tampere University, 33100 Tampere, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6095-2570","authenticated-orcid":false,"given":"Ville","family":"M\u00e4kel\u00e4","sequence":"additional","affiliation":[{"name":"Stratford School of Interaction Design and Business, University Of Waterloo, Stratford, ON N5A 0C1, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8054-7265","authenticated-orcid":false,"given":"Jaakko","family":"Hakulinen","sequence":"additional","affiliation":[{"name":"The Unit of Computing Sciences, Tampere University, 33100 Tampere, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7395-0769","authenticated-orcid":false,"given":"Markku","family":"Turunen","sequence":"additional","affiliation":[{"name":"The Unit of Computing Sciences, Tampere University, 33100 Tampere, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,8,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1016\/j.tra.2020.09.024","article-title":"Non-technological challenges for the remote operation of automated vehicles","volume":"142","author":"Goodall","year":"2020","journal-title":"Transp. Res. Part A Policy Pract."},{"key":"ref_2","unstructured":"(2021, August 07). Radio-Driven Auto Runs Down Escort; Wireless Directs Test Car in Wobbly Course Through Heavy Broadway. The New York Times, Available online: https:\/\/www.nytimes.com\/1925\/07\/28\/archives\/radiodriven-auto-runs-down-escort-wireless-directs-test-car-in.html."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1023\/A:1011299910904","article-title":"Teleoperation User Interfaces for Mining Robotics","volume":"11","author":"Hainsworth","year":"2001","journal-title":"Auton. Robot."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Gusterson, H. (2016). Drone: Remote Control Warfare, MIT Press.","DOI":"10.1063\/1.5009234"},{"key":"ref_5","unstructured":"Hawkins, A.J. (2021, August 07). The First Remote-Controlled Electric Scooters Arrive at an Office Park in Georgia. Available online: https:\/\/mobility21.cmu.edu\/the-first-remote-controlled-electric-scooters-arrive-at-an-office-park-in-georgia\/."},{"key":"ref_6","first-page":"34","article-title":"A Natural Interface for Remote Operation of Underwater Robots","volume":"37","author":"Almeida","year":"2015","journal-title":"IEEE Comput. Graph. Appl."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1231","DOI":"10.1109\/TSMCC.2007.905819","article-title":"Human Performance Issues and User Interface Design for Teleoperated Robots","volume":"37","author":"Chen","year":"2007","journal-title":"IEEE Trans. Syst. Man Cybern. Part C Appl. Rev."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Neumeier, S., Wintersberger, P., Frison, A.-K., Becher, A., Facchi, C., and Riener, A. (2019, January 22\u201325). Teleoperation: The holy grail to solve problems of automated driving? Sure, but latency matters. Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Utrecht, The Netherlands.","DOI":"10.1145\/3342197.3344534"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"3551","DOI":"10.3182\/20140824-6-ZA-1003.02275","article-title":"Driver Modeling for Teleoperation with Time Delay","volume":"47","author":"Vozar","year":"2014","journal-title":"IFAC Proc. Vol."},{"key":"ref_10","unstructured":"Ardiyok, S., Canbeyli, A., and Skardiuze, J. (2021, August 07). European Union: Is Self-Driving Cars Still A Science Fiction? Autonomous Vehicles Legal Framework in the European Union. Available online: https:\/\/www.mondaq.com\/turkey\/rail-road-cycling\/892782\/is-self-driving-cars-still-a-science-fiction-autonomous-vehicles-legal-framework-in-the-european-union."},{"key":"ref_11","unstructured":"Akin, D.L., Howard, R.D., and Oliveria, J.S. (2021, August 07). Human Factors in Space Telepresence, Available online: https:\/\/ntrs.nasa.gov\/citations\/19840015051."},{"key":"ref_12","unstructured":"Will, S. (2017). Development of a Presence Model for Driving Simulators Based on Speed Perception in a Motorcycle Riding Simulator. [Ph.D. Thesis, Universit\u00e4t W\u00fcrzburg]."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Fong, T.W., Conti, F., Grange, S., and Baur, C. (2001). Novel interfaces for remote driving: Gesture, haptic, and PDA. Mobile Robots XV and Telemanipulator and Telepresence Technologies VII, Society of Photo Optical.","DOI":"10.1117\/12.417314"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Hosseini, A., and Lienkamp, M. (2016, January 19\u201322). Enhancing telepresence during the teleoperation of road vehicles using HMD-based mixed reality. Proceedings of the 2016 IEEE Intelligent Vehicles Symposium (IV), Gothenburg, Sweden.","DOI":"10.1109\/IVS.2016.7535568"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Hosseini, A., Richthammer, F., and Lienkamp, M. (2016, January 15\u201318). Predictive Haptic Feedback for Safe Lateral Control of Teleoperated Road Vehicles in Urban Areas. Proceedings of the 2016 IEEE 83rd Vehicular Technology Conference (VTC Spring), Nanjing, China.","DOI":"10.1109\/VTCSpring.2016.7504430"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Lindemann, P., Lee, T.-Y., and Rigoll, G. (2018). Catch My Drift: Elevating Situation Awareness for Highly Automated Driving with an Explanatory Windshield Display User Interface. Multimodal Technol. Interact., 2.","DOI":"10.3390\/mti2040071"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Park, B.-J., Lee, J.-W., Yoon, C., and Kim, K.-H. (2015, January 28\u201330). Augmented reality and representation in vehicle for safe driving at night. Proceedings of the 2015 International Conference on Information and Communication Technology Convergence (ICTC), Jeju, Korea.","DOI":"10.1109\/ICTC.2015.7354791"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"80","DOI":"10.21535\/just.v3i3.38","article-title":"Evaluation of Display Methods for Teleoperation of Road Vehicles","volume":"3","author":"Chucholowski","year":"2016","journal-title":"J. Unmanned Syst. Technol."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Lutz, R.R. (2018, January 15\u201318). Safe-AR: Reducing Risk while Augmenting Reality. Proceedings of the In2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE), Memphis, TN, USA.","DOI":"10.1109\/ISSRE.2018.00018"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1016\/j.trf.2018.04.001","article-title":"Take-over requests in highly automated driving: A crowdsourcing survey on auditory, vibrotactile, and visual displays","volume":"56","author":"Bazilinskyy","year":"2018","journal-title":"Transp. Res. Part F Traffic Psychol. Behav."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"e13","DOI":"10.7717\/peerj-cs.13","article-title":"Auditory interfaces in automated driving: An international survey","volume":"1","author":"Bazilinskyy","year":"2015","journal-title":"PeerJ Comput. Sci."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"53","DOI":"10.2174\/1874447800802010053","article-title":"Test of HMI Alternatives for Driver Support to Keep Safe Speed and Safe Distance\u2014A Simulator Study","volume":"2","author":"Adell","year":"2008","journal-title":"Open Transp. J."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Benz, T.M., and Nitsch, V. (June, January 29). Using multisensory cues for direction information in teleoperation: More is not always better. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.","DOI":"10.1109\/ICRA.2017.7989773"},{"key":"ref_24","unstructured":"National Land Survey of Finland (2021, April 28). Elevation Model 2 m 03\/2019. Available online: https:\/\/www.maanmittauslaitos.fi\/en\/maps-and-spatial-data\/expert-users\/product-descriptions\/elevation-model-2-m."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Hart, S.G., and Staveland, L.E. (1988). Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. Advances in Psychology, Elsevier.","DOI":"10.1016\/S0166-4115(08)62386-9"},{"key":"ref_26","unstructured":"Krippendorf, K. (2019). Content Analysis: An Introduction to Its Methodology, Sage Publications. [4th ed.]."},{"key":"ref_27","unstructured":"Zhang, Y., and Wildemuth, B. (2020, December 10). Qualitative Analysis of Content. Available online: https:\/\/www.ischool.utexas.edu\/~yanz\/Content_analysis.pdf."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1627","DOI":"10.1021\/ac60214a047","article-title":"Smoothing and Differentiation of Data by Simplified Least Squares Procedures","volume":"36","author":"Savitzky","year":"1964","journal-title":"Anal. Chem."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1038\/s41592-019-0686-2","article-title":"SciPy 1.0: Fundamental algorithms for scientific computing in Python","volume":"17","author":"Virtanen","year":"2020","journal-title":"Nat. Methods"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Kaptein, M. (2016). Using Generalized Linear (Mixed) Models in HCI. Modern Statistical Methods for HCI, Springer.","DOI":"10.1007\/978-3-319-26633-6_11"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"21","DOI":"10.3758\/s13423-012-0333-8","article-title":"Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review","volume":"20","author":"Sigrist","year":"2013","journal-title":"Psychon. Bull. Rev."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"705","DOI":"10.1177\/0954407010396006","article-title":"Development and validation of an advanced motorcycle riding simulator","volume":"225","author":"Cossalter","year":"2011","journal-title":"Proc. Inst. Mech. Eng. Part D J. Automob. Eng."}],"container-title":["Multimodal Technologies and Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2414-4088\/5\/8\/44\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:44:46Z","timestamp":1760165086000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2414-4088\/5\/8\/44"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,12]]},"references-count":32,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2021,8]]}},"alternative-id":["mti5080044"],"URL":"https:\/\/doi.org\/10.3390\/mti5080044","relation":{},"ISSN":["2414-4088"],"issn-type":[{"value":"2414-4088","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8,12]]}}}