{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T04:41:05Z","timestamp":1770957665103,"version":"3.50.1"},"reference-count":45,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2021,3,1]],"date-time":"2021-03-01T00:00:00Z","timestamp":1614556800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,3,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Usability testing is an essential element in human\u2013computer interaction studies, and its core purpose is not only to evaluate usability in general but also to identify specific problems. Synchronous usability testing methods are costly to conduct on a large scale, as well as subject to time constraints. Thus, in realistic practice, asynchronous testing is often performed and usually adopts an automatic logging approach to identify usability problems based on collected data. However, in prior research, an in-depth human evaluation review phase has been required to examine each user interaction for usability problems. This consumes considerable time, and the results are likely to be subjective. In this study, we propose a novel method to identify and prioritize usability problems quantitatively using an explainable neural network approach and conduct an experiment based on device usage logs collected in a usability test. Using an explainable neural network, we assess user interactions for their relative influence on an overall usability score, predicting each interaction\u2019s probability of causing a usability problem. Assigning usability scores to each task, we model the relative importance of each individual interaction by calculating a weight for each based on feature maps output by the neural network. Then, we identify usability problems by reviewing the interactions showing the highest importance weight values. The experimental results indicate that our proposed method effectively identifies usability problems and demonstrate that it performs quantitatively better compared to other benchmark methods.<\/jats:p>","DOI":"10.1093\/iwc\/iwab018","type":"journal-article","created":{"date-parts":[[2021,8,5]],"date-time":"2021-08-05T11:26:10Z","timestamp":1628162770000},"page":"155-166","source":"Crossref","is-referenced-by-count":2,"title":["Usability Problem Identification Based on Explainable Neural Network in Asynchronous Testing Environment"],"prefix":"10.1093","volume":"33","author":[{"given":"Hohyun","family":"Hwang","sequence":"first","affiliation":[{"name":"Department of Data Science, Seoul National University of Science and Technology, 232, Gongneung-ro, Nowon-gu, Seoul 01811, South Korea"}]},{"given":"Younghoon","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Industrial Engineering, Seoul National University of Science and Technology, 232, Gongneung-ro, Nowon-gu, Seoul 01811, South Korea"}]}],"member":"286","published-online":{"date-parts":[[2021,8,5]]},"reference":[{"key":"2021091412010641000_ref1","first-page":"61","article-title":"A comparative study of synchronous and asynchronous remote usability testing methods","volume":"1","author":"Alghamdi","year":"2013","journal-title":"Int. Rev. Basic Appl. Sci."},{"key":"2021091412010641000_ref2","doi-asserted-by":"crossref","first-page":"1405","DOI":"10.1145\/1240624.1240838","article-title":"What happened to remote usability testing? An empirical study of three methods","volume-title":"Proc. sigchi on human factors in computing systems","author":"Andreasen","year":"2007"},{"key":"2021091412010641000_ref3","doi-asserted-by":"crossref","first-page":"520","DOI":"10.1109\/ASE.2009.12","article-title":"A methodology and framework to simplify usability analysis of mobile applications","author":"Balagtas-Fernandez","year":"2009","journal-title":"2009 IEEE\/ACM international conference on automated software engineering"},{"key":"2021091412010641000_ref4","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1145\/2783258.2783296","article-title":"Heterogeneous network embedding via deep architectures","author":"Chang","year":"2015","journal-title":"Proc. 21th ACM SIGKDD international conference on knowledge discovery and data mining"},{"key":"2021091412010641000_ref5","doi-asserted-by":"crossref","first-page":"767","DOI":"10.1145\/2984511.2984581","article-title":"Erica: interaction mining mobile apps","volume-title":"Proc. 29th annual symposium on user interface software and technology","author":"Deka","year":"2016"},{"key":"2021091412010641000_ref6","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1145\/3097983.3098036","article-title":"metapath2vec: scalable representation learning for heterogeneous networks","volume-title":"Proc. 23rd ACM SIGKDD international conference on knowledge discovery and data mining","author":"Dong","year":"2017"},{"key":"2021091412010641000_ref7","first-page":"1","article-title":"Visualizing higher-layer features of a deep network","volume":"1341","author":"Erhan","year":"2009","journal-title":"Univ. Montreal"},{"key":"2021091412010641000_ref8","doi-asserted-by":"crossref","first-page":"202","DOI":"10.1007\/11497455_17","article-title":"Framework for integrating usability practices into the software process","volume-title":"Int. conf. product focused software process improvement","author":"Ferre","year":"2005"},{"key":"2021091412010641000_ref9","doi-asserted-by":"crossref","first-page":"636","DOI":"10.1145\/2786567.2792902","article-title":"Automated usability tests for mobile devices through live emotions logging","author":"Filho","year":"2015","journal-title":"In Proc. 17th int. conf. human\u2013computer interaction with mobile devices and services adjunct"},{"key":"2021091412010641000_ref10","doi-asserted-by":"crossref","first-page":"2127","DOI":"10.1145\/2207676.2208365","article-title":"Analysis in practical usability evaluation: a survey study","volume-title":"Proc. sigchi conf. human factors in computing systems","author":"F\u00f8lstad","year":"2012"},{"key":"2021091412010641000_ref11","doi-asserted-by":"crossref","first-page":"647","DOI":"10.1145\/1868914.1868995","article-title":"Analysis in usability evaluations: Anan exploratory study","author":"F\u00f8lstad","year":"2010","journal-title":"In Proc. 6th nordic conf. human\u2013computer interaction: extending boundaries"},{"key":"2021091412010641000_ref12","first-page":"1797","article-title":"Hin2vec: explore meta-paths in heterogeneous information networks for representation learning","author":"Fu","year":"2017","journal-title":"In Proc. 2017 ACM on conf. information and knowledge management"},{"key":"2021091412010641000_ref13","first-page":"383","article-title":"Paper2vec: combining graph and text information for scientific paper representation","author":"Ganguly","year":"2017","journal-title":"In European conf. information retrieval"},{"key":"2021091412010641000_ref14","doi-asserted-by":"crossref","first-page":"680","DOI":"10.1109\/ICDMW.2012.53","article-title":"Fine-grained product features extraction and categorization in reviews opinion mining","volume-title":"2012 IEEE 12th int. conf. data mining workshops (ICDMW)","author":"Huang","year":"2012"},{"key":"2021091412010641000_ref15","doi-asserted-by":"crossref","first-page":"465","DOI":"10.1145\/1553374.1553435","article-title":"A novel lexicalized HMM-based learning framework for web opinion mining","volume-title":"Proc. 26th annual int. conf. machine learning","author":"Jin","year":"2009"},{"key":"2021091412010641000_ref16","first-page":"149","article-title":"Automated usability testing for mobile applications","volume":"2","author":"Kluth","year":"2014","journal-title":"Webist"},{"key":"2021091412010641000_ref17","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1109\/CIDM.2013.6597214","article-title":"Interpreting individual classifications of hierarchical networks","author":"Landecker","year":"2013","journal-title":"2013 IEEE symposium on computational intelligence and data mining (CIDM)"},{"key":"2021091412010641000_ref18","first-page":"2912","article-title":"Analyzing classifiers: Fisher vectors and deep neural networks","author":"Lapuschkin","year":"2016","journal-title":"Proceedings of the IEEE conf. computer vision and pattern recognition"},{"key":"2021091412010641000_ref19","doi-asserted-by":"crossref","first-page":"631","DOI":"10.1080\/10447318.2019.1669320","article-title":"Abnormal usage sequence detection for identification of user needs via recurrent neural network semantic variational autoencoder","volume":"36","author":"Lee","year":"2020","journal-title":"Int. J. Hum.-Comput. Interact."},{"key":"2021091412010641000_ref20","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1007\/s10660-019-09347-6","article-title":"Determining user needs through abnormality detection and heterogeneous embedding of usage sequence","volume":"21","author":"Lee","year":"2021","journal-title":"Electron. Commer. Res."},{"key":"2021091412010641000_ref21","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1016\/j.tele.2017.12.007","article-title":"Smartphone user segmentation based on app usage sequence with neural networks","volume":"35","author":"Lee","year":"2018","journal-title":"Telemat. Inform."},{"key":"2021091412010641000_ref22","first-page":"118","article-title":"Automated and unsupervised user interaction logging as basis for usability evaluation of mobile applications","author":"Lettner","year":"2012","journal-title":"Proc. 10th int. conf. advances in mobile computing and multimedia"},{"key":"2021091412010641000_ref23","first-page":"1860","article-title":"Interactive paths embedding for semantic proximity search on heterogeneous graphs","volume-title":"Proc. 24th ACM SIGKDD int. conf. knowledge discovery and data mining","author":"Liu","year":"2018"},{"key":"2021091412010641000_ref24","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1007\/s11036-012-0421-z","article-title":"Design and implementation of a toolkit for usability testing of mobile apps","volume":"18","author":"Ma","year":"2013","journal-title":"Mobile Netw. Appl."},{"key":"2021091412010641000_ref25","first-page":"1969","article-title":"Changing perspectives on evaluation in HCI: past, present, and future","volume-title":"Chi\u201913 extended abstracts on human factors in computing systems","author":"MacDonald","year":"2013"},{"key":"2021091412010641000_ref26","first-page":"5188","article-title":"Understanding deep image representations by inverting them","author":"Mahendran","year":"2015","journal-title":"Proc. IEEE conf. computer vision and pattern recognition"},{"key":"2021091412010641000_ref27","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1016\/j.patcog.2016.11.008","article-title":"Explaining nonlinear classification decisions with deep taylor decomposition","volume":"65","author":"Montavon","year":"2017","journal-title":"Pattern Recognit."},{"key":"2021091412010641000_ref28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.dsp.2017.10.011","article-title":"Methods for interpreting and understanding deep neural networks","volume":"73","author":"Montavon","year":"2018","journal-title":"Digital Signal Process."},{"key":"2021091412010641000_ref29","first-page":"1","article-title":"The state of the art of mobile application usability evaluation","volume-title":"2012 25th IEEE Canadian conf. electrical and computer engineering (CCECE)","author":"Nayebi","year":"2012"},{"key":"2021091412010641000_ref30","author":"Nguyen","year":"2016"},{"key":"2021091412010641000_ref32","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1145\/2468356.2468404","article-title":"Mate: a mobile analysis tool for usability experts","author":"Porat","year":"2013","journal-title":"Chi\u201913 extended abstracts on human factors in computing systems"},{"key":"2021091412010641000_ref33","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1016\/j.knosys.2016.06.009","article-title":"Aspect extraction for opinion mining with a deep convolutional neural network","volume":"108","author":"Poria","year":"2016","journal-title":"Knowl.-Based Syst."},{"key":"2021091412010641000_ref34","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3299096","article-title":"How UX practitioners produce findings in usability testing","volume":"26","author":"Reeves","year":"2019","journal-title":"ACM Trans. Comput.-Hum. Interact. (TOCHI)"},{"key":"2021091412010641000_ref35","doi-asserted-by":"crossref","first-page":"1135","DOI":"10.1145\/2939672.2939778","article-title":"Why should I trust you? Explaining the predictions of any classifier","author":"Ribeiro","year":"2016","journal-title":"Proc. 22nd ACM SIGKDD int. conf. knowledge discovery and data mining"},{"key":"2021091412010641000_ref36","article-title":"How to plan, design, and conduct effective tests","volume":"348","author":"Rubin","year":"2008","journal-title":"Handbook Usability Test."},{"key":"2021091412010641000_ref37","article-title":"Explainable artificial intelligence: understanding, visualizing and interpreting deep learning models","author":"Samek","year":"2017"},{"key":"2021091412010641000_ref38","first-page":"618","article-title":"Grad-cam: visual explanations from deep networks via gradient-based localization","author":"Selvaraju","year":"2017","journal-title":"Iccv"},{"key":"2021091412010641000_ref39","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1109\/TKDE.2018.2833443","article-title":"Heterogeneous information network embedding for recommendation","volume":"31","author":"Shi","year":"2019","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"2021091412010641000_ref40","article-title":"Deep inside convolutional networks: Visualising image classification models and saliency maps","author":"Simonyan","year":"2013"},{"key":"2021091412010641000_ref41","doi-asserted-by":"crossref","first-page":"235","DOI":"10.3115\/v1\/S14-2038","article-title":"Dlirec: aspect term extraction and term polarity classification system","volume-title":"Proc. 8th int. workshop on semantic evaluation (semeval 2014)","author":"Toh","year":"2014"},{"key":"2021091412010641000_ref42","doi-asserted-by":"crossref","first-page":"1722","DOI":"10.18653\/v1\/P17-1158","article-title":"CANE: context-aware network embedding for relation modeling","volume-title":"Proc. 55th annual meeting of the association for computational linguistics (volume 1: long papers)","author":"Tu","year":"2017"},{"key":"2021091412010641000_ref43","doi-asserted-by":"crossref","first-page":"606","DOI":"10.18653\/v1\/D16-1058","article-title":"Attention-based LSTM for aspect-level sentiment classification","author":"Wang","year":"2016","journal-title":"In Proc. 2016 conf. empirical methods in processing"},{"key":"2021091412010641000_ref44","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1145\/1882362.1882443","article-title":"Software engineering issues for mobile application development","volume-title":"Proc. FSE\/SDP workshop on future of software engineering research","author":"Wasserman","year":"2010"},{"key":"2021091412010641000_ref45","first-page":"2048","article-title":"Show, attend and tell: neural image caption generation with visual attention","author":"Xu","year":"2015","journal-title":"Int. conf. machine learning"},{"key":"2021091412010641000_ref46","first-page":"3155","article-title":"ANRL: attributed network representation learning via deep neural networks","volume-title":"Ijcai","author":"Zhang","year":"2018"}],"container-title":["Interacting with Computers"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/iwc\/article-pdf\/33\/2\/155\/40358709\/iwab018.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/iwc\/article-pdf\/33\/2\/155\/40358709\/iwab018.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,6]],"date-time":"2023-11-06T23:13:02Z","timestamp":1699312382000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/iwc\/article\/33\/2\/155\/6342663"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3]]},"references-count":45,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2021,8,5]]},"published-print":{"date-parts":[[2021,3,12]]}},"URL":"https:\/\/doi.org\/10.1093\/iwc\/iwab018","relation":{},"ISSN":["0953-5438","1873-7951"],"issn-type":[{"value":"0953-5438","type":"print"},{"value":"1873-7951","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,3]]},"published":{"date-parts":[[2021,3]]}}}