{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T07:28:15Z","timestamp":1772782095325,"version":"3.50.1"},"reference-count":60,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2021,11,29]],"date-time":"2021-11-29T00:00:00Z","timestamp":1638144000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MTI"],"abstract":"<jats:p>Usability is a principal aspect of the system development process to improve and augment system facilities and meet users\u2019 needs and necessities in all domains. It is no exception for cultural heritage. Usability problems of the interactive technology practice in cultural heritage museums should be recognized thoroughly from the viewpoints of experts and users. This paper reports on a two-phase empirical study to identify the usability problems in audio guides and websites of cultural heritage museums in Vietnam, as a developing country, and Australia, as a developed country. In phase one, five-user experience experts identified usability problems using the set of usability heuristics, and proposed suggestions to mitigate these issues. Ten usability heuristics identified a total of 176 problems for audio guides and websites. In phase two, we conducted field usability surveys to collect the real users\u2019 opinions to detect the usability issues and examine the negative-ranked usability. The outstanding issues for audio guides and websites were pointed out. Identification of relevant usability issues and users\u2019 and experts\u2019 suggestions for these technologies should be given immediate attention to helping organizations and interactive service providers improve technologies\u2019 adoptions. The paper\u2019s findings are reliable inputs for our future study about the preeminent UX framework for interactive technology in the CH domain.<\/jats:p>","DOI":"10.3390\/mti5120075","type":"journal-article","created":{"date-parts":[[2021,11,30]],"date-time":"2021-11-30T22:01:11Z","timestamp":1638309671000},"page":"75","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Identification of Usability Issues of Interactive Technologies in Cultural Heritage through Heuristic Evaluations and Usability Surveys"],"prefix":"10.3390","volume":"5","author":[{"given":"Duyen","family":"Lam","sequence":"first","affiliation":[{"name":"School of Information Technology, Deakin University, Geelong, VIC 3125, Australia"},{"name":"School of Social Science and Humanities, Can Tho University, Can Tho 900000, Vietnam"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7354-260X","authenticated-orcid":false,"given":"Thuong","family":"Hoang","sequence":"additional","affiliation":[{"name":"School of Information Technology, Deakin University, Geelong, VIC 3125, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0445-0573","authenticated-orcid":false,"given":"Atul","family":"Sajjanhar","sequence":"additional","affiliation":[{"name":"School of Information Technology, Deakin University, Geelong, VIC 3125, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,11,29]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1518682","DOI":"10.1155\/2018\/1518682","article-title":"Heuristic evaluation: Comparing generic and specific usability heuristics for identification of usability problems in a living museum mobile guide app","volume":"2018","author":"Othman","year":"2018","journal-title":"Adv. Hum.-Comput. Interact."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"465","DOI":"10.1016\/j.im.2016.10.004","article-title":"Impact of interactive technologies on stimulating learning experiences in a museum","volume":"54","author":"Pallud","year":"2017","journal-title":"Inf. Manag."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"424","DOI":"10.1080\/09647775.2019.1683880","article-title":"Framing collaborative processes of digital transformation in cultural organisations: From literary archives to augmented reality","volume":"35","author":"Arrigoni","year":"2020","journal-title":"Mus. Manag. Curatorship"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Nikolakopoulou, V., and Koutsabasis, P. (2020). Methods and Practices for Assessing the User Experience of Interactive Systems for Cultural Heritage. Applying Innovative Technologies in Heritage Science, IGI Global.","DOI":"10.4018\/978-1-7998-2871-6.ch009"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Subali, M., Andriansyah, M., Saptono, D., Purwanto, I., Antonius, I.S., Rahmadi, H., and Sudjono, L.A.L. (2018, January 17\u201318). Development of Banten E-Heritage using Virtual Reality Technology on Mobile Device. Proceedings of the 2018 Third International Conference on Informatics and Computing (ICIC), Palembang, Indonesia.","DOI":"10.1109\/IAC.2018.8780544"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Skovfoged, M.M., Viktor, M., Sokolov, M.K., Hansen, A., Nielsen, H.H., and Rodil, K. (2018, January 3). The tales of the Tokoloshe: Safeguarding intangible cultural heritage using virtual reality. Proceedings of the Second African Conference for Human Computer Interaction: Thriving Communities, New York, NY, USA.","DOI":"10.1145\/3283458.3283484"},{"key":"ref_7","unstructured":"Thomas, M. (2021, March 02). A Fading Taj and Shabby Museums: India Spends Less Than 1% of Its Annual Budget on Culture and It Shows. Available online: https:\/\/qz.com\/india\/897228\/a-fading-taj-and-shabby-museums-india-spends-less-than-1-of-its-annual-budget-on-culture-and-it-shows\/."},{"key":"ref_8","unstructured":"Saccone, D., and Bertacchini, E. (2011). Culture, Heritage and Economic Development-Empowering Developing Countries, University of Turin."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Altschul, J.H. (2018). Cultural heritage management in developing countries: Challenges and opportunities. Relevance and Application of Heritage in Contemporary Society, Routledge.","DOI":"10.4324\/9780203702277-13"},{"key":"ref_10","unstructured":"Vu, D. (2019, February 20). ICT and Museum Displays, Available online: http:\/\/dch.gov.vn\/pages\/news\/preview.aspx?n=380&c=64."},{"key":"ref_11","unstructured":"Tran, H. (2019, February 02). Museum Interactive and Experience in 4.0 Age. Available online: http:\/\/baovanhoa.vn\/van-hoa\/artmid\/428\/articleid\/13150\/tuong-tac-trai-nghiem-o-bao-tang-thoi-40."},{"key":"ref_12","unstructured":"Le, L., and Nguyen, H. (2020, May 07). Build a New Museum and the Best Practices. Available online: http:\/\/baotangtonducthang.vn\/xay-dung-bao-tang-moi-va-nhung-bat-cap-tu-thuc-tien-can-thay-doi.html."},{"key":"ref_13","unstructured":"Linh, G. (2019, April 01). Renovation and Improvement the Quality and Efficiency of Museum Activities Associated with Tourism Development, Available online: https:\/\/bvhttdl.gov.vn\/gan-hoat-dong-bao-tang-voi-phat-trien-du-lich-20190103113425089.htm."},{"key":"ref_14","unstructured":"Pham, T. (2019, September 07). Discussion about Interactive Experience from Museums (Part 1). Available online: http:\/\/baotanglichsu.vn\/vi\/Articles\/3099\/69198\/djoi-djieu-ve-tuong-tac-trai-nghiem-tu-trung-bay-bao-tang-phan-1.html."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"87","DOI":"10.3200\/JAML.39.2.87-100","article-title":"Australian museums and the modern public: A marketing context","volume":"39","author":"Lehman","year":"2009","journal-title":"J. Arts Manag. Law Soc."},{"key":"ref_16","unstructured":"Museums & Galleries of NSW (2019, July 07). Strategic Planning for Museums and Galleries. Available online: https:\/\/mgnsw.org.au\/sector\/resources\/online-resources\/new-facilities\/strategic-planning\/."},{"key":"ref_17","unstructured":"Museum, A. (2019, February 02). David Attenborough\u2019s Virtual Reality Experiences Come to Australian Museum from 8 April. Available online: https:\/\/australianmuseum.net.au\/about\/organisation\/media-centre\/david-attenboroughs-virtual-reality-experiences-come-to-australian-museum-from-8-april\/."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1016\/j.ijmedinf.2018.05.012","article-title":"Evaluating the agreement of users with usability problems identified by heuristic evaluation","volume":"117","author":"Khajouei","year":"2018","journal-title":"Int. J. Med. Inform."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3354002","article-title":"Adding Culture to UX: UX Research Methodologies and Applications in Cultural Heritage","volume":"13","author":"Konstantakis","year":"2020","journal-title":"J. Comput. Cult. Herit. (JOCCH)"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Konstantakis, M., Michalakis, K., Aliprantis, J., Kalatha, E., and Caridakis, G. (2017, January 9\u201310). Formalising and evaluating cultural user experience. Proceedings of the 2017 12th International Workshop on Semantic and Social Media Adaptation and Personalization (SMAP), Bratislava, Slovakia.","DOI":"10.1109\/SMAP.2017.8022675"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Nielsen, J., and Molich, R. (1990, January 1). Heuristic evaluation of user interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA.","DOI":"10.1145\/97243.97281"},{"key":"ref_22","unstructured":"Nielsen, J. (1995). 10 Usability Heuristics for User Interface Design, Nielsen Norman Group."},{"key":"ref_23","first-page":"434326","article-title":"Heuristic evaluation on mobile interfaces: A new checklist","volume":"2014","author":"Sevillano","year":"2014","journal-title":"Sci. World J."},{"key":"ref_24","unstructured":"Othman, M.K. (2012). Measuring Visitors\u2019 Experiences with Mobile Guide Technology in Cultural Spaces, University of York."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Battisti, F., and Di Stefano, C. (2018, January 26\u201328). Virtual Reality meets Degas: An immersive framework for art exploration and learning. In Proceedings of 2018 7th European Workshop on Visual Information Processing (EUVIP), Tampere, Finland.","DOI":"10.1109\/EUVIP.2018.8611753"},{"key":"ref_26","first-page":"45","article-title":"Evaluation of art museums\u2019 web sites worldwide","volume":"28","author":"Pallas","year":"2008","journal-title":"Inf. Serv. Use"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Nielsen, J. (1992, January 1). Finding usability problems through heuristic evaluation. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA.","DOI":"10.1145\/142750.142834"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1007\/s11219-006-7600-8","article-title":"Usability measurement and metrics: A consolidated model","volume":"14","author":"Seffah","year":"2006","journal-title":"Softw. Qual. J."},{"key":"ref_29","unstructured":"ISO (2019, March 02). Ergonomics of Human-System Interaction\u2014Part 11: Usability: Definitions and Concepts. Available online: https:\/\/www.iso.org\/obp\/ui\/#iso:std:iso:9241:-11:en."},{"key":"ref_30","unstructured":"Kim, G.J. (2015). Human-Computer Interaction: Fundamentals and Practice, CRC Press."},{"key":"ref_31","unstructured":"Saffer, D. (2010). Designing for Interaction: Creating Innovative Applications and Devices, New Riders."},{"key":"ref_32","unstructured":"Bevan, N. (2009, January 24\u201328). What is the difference between the purpose of usability and user experience evaluation methods. Proceedings of the Workshop UXEM, Uppsala, Sweden."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Nielsen, J. (1994). Usability Engineering, Morgan Kaufmann.","DOI":"10.1016\/B978-0-08-052029-2.50009-7"},{"key":"ref_34","unstructured":"Jeffries, R. (1994). Usability problem reports: Helping evaluators communicate effectively with developers. Usability Inspection Methods, John Wiley & Sons, Inc."},{"key":"ref_35","unstructured":"Proctor, R.W., and Vu, K.-P.L. (2005). State of the Art of Web Usability Guidelines. Handbook of Human Factors in Web Design, Lawrence Erlbaum Associates."},{"key":"ref_36","unstructured":"Mohamed, H., and Jaafar, A. (2011, January 12\u201314). AHP_HeGES: Tools to evaluate usability of educational computer game (UsaECG). In Proceedings of 2011 International Conference on User Science and Engineering (i-USEr), Kuala Lumpur, Malaysia."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1016\/j.jbi.2018.02.016","article-title":"Comparison of two heuristic evaluation methods for evaluating the usability of health information systems","volume":"80","author":"Khajouei","year":"2018","journal-title":"J. Biomed. Inform."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1177\/193758671000300305","article-title":"Developing a usability evaluation tool to assess the patient room bathroom","volume":"3","author":"Fink","year":"2010","journal-title":"HERD Health Environ. Res. Des. J."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.csi.2017.03.009","article-title":"How to develop usability heuristics: A systematic literature review","volume":"53","author":"Rusu","year":"2017","journal-title":"Comput. Stand. Interfaces"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Silva, P.A., Jordan, P., and Holden, K. (2014, January 11). Something old, something new, something borrowed: Gathering experts\u2019 feedback while performing heuristic evaluation with a list of heuristics targeted at older adults. Proceedings of the 2014 Workshops on Advances in Computer Entertainment Conference, New York, NY, USA.","DOI":"10.1145\/2693787.2693804"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Inostroza, R., Rusu, C., Roncagliolo, S., and Rusu, V. (2013, January 11). Usability heuristics for touchscreen-based mobile devices: Update. Proceedings of the 2013 Chilean Conference on Human-Computer Interaction, New York, NY, USA.","DOI":"10.1145\/2535597.2535602"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Joyce, G., and Lilley, M. (2014). Towards the development of usability heuristics for native smartphone mobile applications. International Conference of Design, User Experience, and Usability, Springer.","DOI":"10.1007\/978-3-319-07668-3_45"},{"key":"ref_43","unstructured":"Tehrani, S.E.M., Zainuddin, N.M.M., and Takavar, T. (2014, January 2\u20135). Heuristic evaluation for Virtual Museum on smartphone. Proceedings of the 2014 3rd International Conference on User Science and Engineering (i-USEr), Shah Alam, Malaysia."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Murtza, R., Monroe, S., and Youmans, R.J. (2017, January 9\u201313). Heuristic evaluation for virtual reality systems. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Austin, TX, USA.","DOI":"10.1177\/1541931213602000"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1504\/IJDCET.2008.020134","article-title":"Art, science\/technology and history museums on the web","volume":"1","author":"Fotakis","year":"2008","journal-title":"Int. J. Digit. Cult. Electron. Tour."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1080\/15332860802086318","article-title":"Measuring perceived website usability","volume":"6","author":"Wang","year":"2007","journal-title":"J. Internet Commer."},{"key":"ref_47","unstructured":"Lazarinis, F., Kanellopoulos, D., and Lalos, P. (2008). Heuristically evaluating Greek e-tourism and e-museum websites. Electron. J. Inf. Syst. Eval., 11."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"363","DOI":"10.15388\/Informatica.2006.143","article-title":"Usability evaluation of a virtual museum interface","volume":"17","author":"Karoulis","year":"2006","journal-title":"Informatica"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"D\u00edaz, J., Rusu, C., Pow-Sang, J.A., and Roncagliolo, S. (2013, January 11). A cultural-oriented usability heuristics proposal. Proceedings of the 2013 Chilean Conference on Human-Computer Interaction, New York, NY, USA.","DOI":"10.1145\/2535597.2535615"},{"key":"ref_50","unstructured":"Harms, I., and Schweibenz, W. (2020, March 20). Evaluating the Usability of a Museum Web Site, Available online: https:\/\/eric.ed.gov\/?id=ED482074."},{"key":"ref_51","unstructured":"Travis, D. (2020, September 01). 247 Web Usability Guidelines. Available online: https:\/\/www.userfocus.co.uk\/resources\/guidelines.html."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Othman, M.K., Petrie, H., and Power, C. (2011). Engaging visitors in museums with technology: Scales for the measurement of visitor and multimedia guide experience. IFIP Conference on Human-Computer Interaction, Springer.","DOI":"10.1007\/978-3-642-23768-3_8"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1191\/1478088706qp063oa","article-title":"Using thematic analysis in psychology","volume":"3","author":"Braun","year":"2006","journal-title":"Qual. Res. Psychol."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Mazlan, M.F.B., Sivaji, A., Tzuaan, S.S., and Lokman, A.M. (2012, January 3\u20134). Enhancing the heuristic evaluation (HE) by development and validation of a collaborative design measurement system (CDMS): Collaborative design and measurement system for designers and testers. In Proceedings of 2012 IEEE Colloquium on Humanities, Science and Engineering (CHUSER), Kota Kinabalu, Malaysia.","DOI":"10.1109\/CHUSER.2012.6504361"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"3189","DOI":"10.1007\/s10639-020-10112-8","article-title":"A framework for heuristic evaluation of mobile learning applications","volume":"25","author":"Kumar","year":"2020","journal-title":"Educ. Inf. Technol."},{"key":"ref_56","unstructured":"Bredin, H. (2020, December 07). Guiding Principles: What Are the Ingredients of a Good Audio Guide?. Available online: https:\/\/www.spectator.co.uk\/article\/guiding-principles."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"846","DOI":"10.1016\/j.dss.2012.05.020","article-title":"Understanding web enjoyment experiences and informal learning: A study in a museum context","volume":"53","author":"Lin","year":"2012","journal-title":"Decis. Support Syst."},{"key":"ref_58","unstructured":"Albert, W., and Tullis, T. (2013). Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics, Newnes."},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Karat, C.-M., Campbell, R., and Fiegel, T. (1992, January 1). Comparison of empirical testing and walkthrough methods in user interface evaluation. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA.","DOI":"10.1145\/142750.142873"},{"key":"ref_60","unstructured":"Budiu, R., and Nielsen, J. (2011). Usability of Ipad Apps and Websites, Nielsen Norman Group."}],"container-title":["Multimodal Technologies and Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2414-4088\/5\/12\/75\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:37:37Z","timestamp":1760168257000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2414-4088\/5\/12\/75"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,29]]},"references-count":60,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["mti5120075"],"URL":"https:\/\/doi.org\/10.3390\/mti5120075","relation":{},"ISSN":["2414-4088"],"issn-type":[{"value":"2414-4088","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,29]]}}}