{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T21:14:42Z","timestamp":1780521282351,"version":"3.54.1"},"reference-count":42,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2023,12,25]],"date-time":"2023-12-25T00:00:00Z","timestamp":1703462400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key R&amp;D Program of China","doi-asserted-by":"publisher","award":["2021YFB1600500"],"award-info":[{"award-number":["2021YFB1600500"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012166","name":"National Key R&amp;D Program of China","doi-asserted-by":"publisher","award":["JSZRHYKJ202308"],"award-info":[{"award-number":["JSZRHYKJ202308"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Marine Science and Technology Innovation Program of Jiangsu Province","award":["2021YFB1600500"],"award-info":[{"award-number":["2021YFB1600500"]}]},{"name":"Marine Science and Technology Innovation Program of Jiangsu Province","award":["JSZRHYKJ202308"],"award-info":[{"award-number":["JSZRHYKJ202308"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Systems"],"abstract":"<jats:p>Visualization systems play a crucial role in industry, education, and research domains by offering valuable insights and enhancing decision making. These systems enable the representation of complex workflows and data in a visually intuitive manner, facilitating better understanding, analysis, and communication of information. This paper explores the potential of augmented reality (AR) visualization systems that enhance multi-modal perception and interaction for complex decision making. The proposed system combines the physicality and intuitiveness of the real world with the immersive and interactive capabilities of AR systems. By integrating physical objects and virtual elements, users can engage in natural and intuitive interactions, leveraging multiple sensory modalities. Specifically, the system incorporates vision, touch, eye-tracking, and sound as multi-modal interaction methods to further improve the user experience. This multi-modal nature enables users to perceive and interact in a more holistic and immersive manner. The software and hardware engineering of the proposed system are elaborated in detail, and the system\u2019s architecture and preliminary function testing results are also included in the manuscript. The findings aim to aid visualization system designers, researchers, and practitioners in exploring and harnessing the capabilities of this integrated approach, ultimately leading to more engaging and immersive user experiences in various application domains.<\/jats:p>","DOI":"10.3390\/systems12010007","type":"journal-article","created":{"date-parts":[[2023,12,25]],"date-time":"2023-12-25T03:42:12Z","timestamp":1703475732000},"page":"7","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":["Enhancing Multi-Modal Perception and Interaction: An Augmented Reality Visualization System for Complex Decision Making"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-8661-7420","authenticated-orcid":false,"given":"Liru","family":"Chen","sequence":"first","affiliation":[{"name":"School of Cyber Science and Engineering, Southeast University, Nanjing 211189, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hantao","family":"Zhao","sequence":"additional","affiliation":[{"name":"School of Cyber Science and Engineering, Southeast University, Nanjing 211189, China"},{"name":"Purple Mountain Laboratories, Nanjing 211189, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chenhui","family":"Shi","sequence":"additional","affiliation":[{"name":"School of Cyber Science and Engineering, Southeast University, Nanjing 211189, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Youbo","family":"Wu","sequence":"additional","affiliation":[{"name":"School of Cyber Science and Engineering, Southeast University, Nanjing 211189, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xuewen","family":"Yu","sequence":"additional","affiliation":[{"name":"School of Cyber Science and Engineering, Southeast University, Nanjing 211189, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenze","family":"Ren","sequence":"additional","affiliation":[{"name":"School of Cyber Science and Engineering, Southeast University, Nanjing 211189, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ziyi","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Southeast University, Nanjing 211189, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8511-3295","authenticated-orcid":false,"given":"Xiaomeng","family":"Shi","sequence":"additional","affiliation":[{"name":"School of Transportation, Southeast University, Nanjing 211189, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,12,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"81555","DOI":"10.1109\/ACCESS.2019.2923736","article-title":"Visual Analytics: A Comprehensive Overview","volume":"7","author":"Cui","year":"2019","journal-title":"IEEE Access"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1002\/wics.89","article-title":"Information visualization","volume":"2","author":"Chen","year":"2010","journal-title":"Wiley Interdiscip. Rev. Comput. Stat."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Zhan, T., Yin, K., Xiong, J., He, Z., and Wu, S.T. (2020). Augmented Reality and Virtual Reality Displays: Perspectives and Challenges. iScience, 23.","DOI":"10.1016\/j.isci.2020.101397"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Satriadi, K.A., Smiley, J., Ens, B., Cordeil, M., Czauderna, T., Lee, B., Yang, Y., Dwyer, T., and Jenny, B. (May, January 29). Tangible Globes for Data Visualisation in Augmented Reality. Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, New York, NY, USA.","DOI":"10.1145\/3491102.3517715"},{"key":"ref_5","first-page":"2454","article-title":"Data Visualization","volume":"12","author":"Sadiku","year":"2016","journal-title":"Int. J. Eng. Res. Adv. Technol. (IJERAT)"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/2945.981847","article-title":"Information visualization and visual data mining","volume":"8","author":"Keim","year":"2002","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Xu, H., Berres, A., Liu, Y., Allen-Dumas, M.R., and Sanyal, J. (2022). An overview of visualization and visual analytics applications in water resources management. Environ. Model. Softw., 153.","DOI":"10.1016\/j.envsoft.2022.105396"},{"key":"ref_8","unstructured":"Zheng, J.G. (2017). Global Business Intelligence, Routledge."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"543","DOI":"10.1111\/cgf.13891","article-title":"A Survey of Visual Analytics for Public Health","volume":"39","author":"Preim","year":"2020","journal-title":"Comput. Graph. Forum"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"White, S., Kalkofen, D., and Sandor, C. (2011, January 26\u201329). Visualization in mixed reality environments. Proceedings of the 2011 10th IEEE International Symposium on Mixed and Augmented Reality, Basel, Switzerland.","DOI":"10.1109\/ISMAR.2011.6092361"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"14749","DOI":"10.1007\/s11042-021-10971-4","article-title":"Augmented Reality Situated Visualization in Decision-Making","volume":"81","author":"Martins","year":"2022","journal-title":"Multimed. Tools Appl."},{"key":"ref_12","first-page":"360","article-title":"A BIM-based location aware AR collaborative framework for facility maintenance management","volume":"24","author":"Chen","year":"2019","journal-title":"Electron. J. Inf. Technol. Constr."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Ma, N., Liu, Y., Qiao, A., and Du, J. (2008, January 16\u201318). Design of Three-Dimensional Interactive Visualization System Based on Force Feedback Device. Proceedings of the 2008 2nd International Conference on Bioinformatics and Biomedical Engineering, Shanghai, China.","DOI":"10.1109\/ICBBE.2008.774"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Han, W., and Schulz, H.J. (2020, January 8\u201310). Exploring Vibrotactile Cues for Interactive Guidance in Data Visualization. Proceedings of the 13th International Symposium on Visual Information Communication and Interaction, VINCI \u201920, New York, NY, USA.","DOI":"10.1145\/3430036.3430042"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Su, Y.P., Chen, X.Q., Zhou, C., Pearson, L.H., Pretty, C.G., and Chase, J.G. (2023). Integrating Virtual, Mixed, and Augmented Reality into Remote Robotic Applications: A Brief Review of Extended Reality-Enhanced Robotic Systems for Intuitive Telemanipulation and Telemanufacturing Tasks in Hazardous Conditions. Appl. Sci., 13.","DOI":"10.3390\/app132212129"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1162\/pres.1997.6.4.355","article-title":"A Survey of Augmented Reality","volume":"6","author":"Azuma","year":"1997","journal-title":"Presence Teleoper. Virtual Environ."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Tarng, W., Tseng, Y.C., and Ou, K.L. (2022). Application of Augmented Reality for Learning Material Structures and Chemical Equilibrium in High School Chemistry. Systems, 10.","DOI":"10.3390\/systems10050141"},{"key":"ref_18","unstructured":"Gavish, N. (2020). Systems Engineering in the Fourth Industrial Revolution: Big Data, Novel Technologies, and Modern Systems Engineering, Wiley Online Library."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1016\/j.compedu.2012.10.024","article-title":"Current status, opportunities and challenges of augmented reality in education","volume":"62","author":"Wu","year":"2013","journal-title":"Comput. Educ."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.edurev.2016.11.002","article-title":"Advantages and challenges associated with augmented reality for education: A systematic review of the literature","volume":"20","author":"Akcayr","year":"2017","journal-title":"Educ. Res. Rev."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Nishimoto, A., and Johnson, A.E. (2019, January 19\u201320). Extending Virtual Reality Display Wall Environments Using Augmented Reality. Proceedings of the Symposium on Spatial User Interaction, SUI \u201919, New York, NY, USA.","DOI":"10.1145\/3357251.3357579"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Liu, B., and Tanaka, J. (2021). Virtual Marker Technique to Enhance User Interactions in a Marker-Based AR System. Appl. Sci., 11.","DOI":"10.3390\/app11104379"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Gao, Q.H., Wan, T.R., Tang, W., and Chen, L. (2017, January 20\u201322). A Stable and Accurate Marker-Less Augmented Reality Registration Method. Proceedings of the 2017 International Conference on Cyberworlds (CW), Chester, UK.","DOI":"10.1109\/CW.2017.44"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Ye, H., Leng, J., Xiao, C., Wang, L., and Fu, H. (2023, January 23\u201328). ProObjAR: Prototyping Spatially-Aware Interactions of Smart Objects with AR-HMD. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI \u201923, New York, NY, USA.","DOI":"10.1145\/3544548.3580750"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Al-Ansi, A.M., Jaboob, M., Garad, A., and Al-Ansi, A. (2023). Analyzing augmented reality (AR) and virtual reality (VR) recent development in education. Soc. Sci. Humanit. Open, 8.","DOI":"10.1016\/j.ssaho.2023.100532"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Lee, N. (2019). Encyclopedia of Computer Graphics and Games, Springer International Publishing.","DOI":"10.1007\/978-3-319-08234-9"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Moro, M., Marchesi, G., Hesse, F., Odone, F., and Casadio, M. (2022). Markerless vs. Marker-Based Gait Analysis: A Proof of Concept Study. Sensors, 22.","DOI":"10.3390\/s22052011"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Wen, F., Sun, Z., Guo, X., He, T., and Lee, C. (2022). Artificial Intelligence-Enabled Sensing Technologies in the 5G\/Internet of Things Era: From Virtual Reality\/Augmented Reality to the Digital Twin. Adv. Intell. Syst., 4.","DOI":"10.1002\/aisy.202100228"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Syed, T.A., Siddiqui, M.S., Abdullah, H.B., Jan, S., Namoun, A., Alzahrani, A., Nadeem, A., and Alkhodre, A.B. (2023). In-Depth Review of Augmented Reality: Tracking Technologies, Development Tools, AR Displays, Collaborative AR, and Security Concerns. Sensors, 23.","DOI":"10.3390\/s23010146"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Chen, J.Y.C., and Fragomeni, G. (2022). Virtual, Augmented and Mixed Reality: Design and Development, Springer.","DOI":"10.1007\/978-3-031-05939-1"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Aouam, D., Benbelkacem, S., Zenati, N., Zakaria, S., and Meftah, Z. (2018, January 24\u201325). Voice-based Augmented Reality Interactive System for Car\u2019s Components Assembly. Proceedings of the 2018 3rd International Conference on Pattern Analysis and Intelligent Systems (PAIS), Tebessa, Algeria.","DOI":"10.1109\/PAIS.2018.8598516"},{"key":"ref_32","unstructured":"Kaimoto, H., Monteiro, K., Faridan, M., Li, J., Farajian, S., Kakehi, Y., Nakagaki, K., and Suzuki, R. (November, January 29). Sketched Reality: Sketching Bi-Directional Interactions Between Virtual and Physical Worlds with AR and Actuated Tangible UI. Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, UIST \u201922, New York, NY, USA."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Ishii, H., and Ullmer, B. (1997, January 22\u201327). Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms. Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, CHI \u201997, New York, NY, USA.","DOI":"10.1145\/258549.258715"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"L\u00f6ffler, D., Tscharn, R., and Hurtienne, J. (2018, January 18\u201321). Multimodal Effects of Color and Haptics on Intuitive Interaction with Tangible User Interfaces. Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction, TEI \u201918, New York, NY, USA.","DOI":"10.1145\/3173225.3173257"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1561\/1100000026","article-title":"Tangible User Interfaces: Past, Present, and Future Directions","volume":"3","author":"Shaer","year":"2010","journal-title":"Found. Trends Hum.-Comput. Interact."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"803","DOI":"10.1016\/j.ijhcs.2013.04.003","article-title":"To TUI or not to TUI: Evaluating performance and preference in tangible vs. graphical user interfaces","volume":"71","author":"Zuckerman","year":"2013","journal-title":"Int. J. Hum.-Comput. Stud."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1016\/j.ijcci.2018.01.003","article-title":"A review on complementary natures of tangible user interfaces (TUIs) and early spatial learning","volume":"16","author":"Baykal","year":"2018","journal-title":"Int. J. Child-Comput. Interact."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"He, F., Hu, X., Shi, J., Qian, X., Wang, T., and Ramani, K. (2023, January 23\u201328). Ubi Edge: Authoring Edge-Based Opportunistic Tangible User Interfaces in Augmented Reality. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI \u201923, New York, NY, USA.","DOI":"10.1145\/3544548.3580704"},{"key":"ref_39","unstructured":"Unity (2023, November 13). Vuforia SDK Overview. Available online: https:\/\/docs.unity3d.com\/2018.4\/Documentation\/Manual\/vuforia-sdk-overview.html."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Filonik, D., Buchan, A., Ogden-Doyle, L., and Bednarz, T. (2018, January 2\u20133). Interactive Scenario Visualisation in Immersive Virtual Environments for Decision Making Support. Proceedings of the 16th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry, VRCAI \u201918, New York, NY, USA.","DOI":"10.1145\/3284398.3284426"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Prange, S., Shams, A., Piening, R., Abdelrahman, Y., and Alt, F. (2021, January 8\u201313). PriView\u2014Exploring Visualisations to Support Users\u2019 Privacy Awareness. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI \u201921, New York, NY, USA.","DOI":"10.1145\/3411764.3445067"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Bermejo Fernandez, C., Lee, L.H., Nurmi, P., and Hui, P. (2021, January 18\u201322). PARA: Privacy Management and Control in Emerging IoT Ecosystems Using Augmented Reality. Proceedings of the 2021 International Conference on Multimodal Interaction, ICMI \u201921, New York, NY, USA.","DOI":"10.1145\/3462244.3479885"}],"container-title":["Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-8954\/12\/1\/7\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:41:36Z","timestamp":1760132496000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-8954\/12\/1\/7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,25]]},"references-count":42,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,1]]}},"alternative-id":["systems12010007"],"URL":"https:\/\/doi.org\/10.3390\/systems12010007","relation":{},"ISSN":["2079-8954"],"issn-type":[{"value":"2079-8954","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,25]]}}}