{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T23:23:47Z","timestamp":1779924227214,"version":"3.53.1"},"reference-count":35,"publisher":"Oxford University Press (OUP)","issue":"e1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2017,4,1]]},"abstract":"<jats:p>Objective: There are several user-based and expert-based usability evaluation methods that may perform differently according to the context in which they are used. The objective of this study was to compare 2 expert-based methods, heuristic evaluation (HE) and cognitive walkthrough (CW), for evaluating usability of health care information systems.<\/jats:p><jats:p>Materials and methods: Five evaluators independently evaluated a medical office management system using HE and CW. We compared the 2 methods in terms of the number of identified usability problems, their severity, and the coverage of each method.<\/jats:p><jats:p>Results: In total, 156 problems were identified using the 2 methods. HE identified a significantly higher number of problems related to the \u201csatisfaction\u201d attribute (P\u2009=\u2009.002). The number of problems identified using CW concerning the \u201clearnability\u201d attribute was significantly higher than those identified using HE (P\u2009=\u2009.005). There was no significant difference between the number of problems identified by HE, based on different usability attributes (P\u2009=\u2009.232). Results of CW showed a significant difference between the number of problems related to usability attributes (P\u2009&amp;lt;\u2009.0001). The average severity of problems identified using CW was significantly higher than that of HE (P\u2009&amp;lt;\u2009.0001).<\/jats:p><jats:p>Conclusion: This study showed that HE and CW do not differ significantly in terms of the number of usability problems identified, but they differ based on the severity of problems and the coverage of some usability attributes. The results suggest that CW would be the preferred method for evaluating systems intended for novice users and HE for users who have experience with similar systems. However, more studies are needed to support this finding.<\/jats:p>","DOI":"10.1093\/jamia\/ocw100","type":"journal-article","created":{"date-parts":[[2016,8,8]],"date-time":"2016-08-08T00:08:11Z","timestamp":1470614891000},"page":"e55-e60","source":"Crossref","is-referenced-by-count":69,"title":["Comparison of heuristic and cognitive walkthrough usability evaluation methods for evaluating health information systems"],"prefix":"10.1093","volume":"24","author":[{"given":"Reza","family":"Khajouei","sequence":"first","affiliation":[{"name":"Medical Informatics Research Center, Institute for Futures Studies in Health, Kerman University of Medical Sciences, Kerman, Iran"},{"name":"Department of Health Information Management and Technology, Faculty of Management and Medical Information Sciences, Kerman University of Medical Sciences, Kerman, Iran"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Misagh","family":"Zahiri Esfahani","sequence":"additional","affiliation":[{"name":"Regional Knowledge Hub and World Health Organization Collaborating Centre for HIV Surveillance, Institute for Futures Studies in Health, Kerman University of Medical Sciences, Kerman, Iran,"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yunes","family":"Jahani","sequence":"additional","affiliation":[{"name":"Department of Biostatistics and Epidemiology, School of Public Health, Kerman University of Medical Sciences, Kerman, Iran"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2016,8,7]]},"reference":[{"issue":"4","key":"2020110612433415900_ocw100-B1","first-page":"51","article-title":"Analyzing computer-based patient records: a review of literature","volume":"17","author":"Erstad","year":"2003","journal-title":"J Healthc Inform Manag."},{"issue":"1","key":"2020110612433415900_ocw100-B2","doi-asserted-by":"crossref","first-page":"e7","DOI":"10.1542\/peds.2014-1115","article-title":"Use of electronic health record systems by office-based pediatricians","volume":"135","author":"Lehmann","year":"2015","journal-title":"Pediatrics."},{"issue":"1","key":"2020110612433415900_ocw100-B3","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1370\/afm.1461","article-title":"The rise of electronic health record adoption among family physicians","volume":"11","author":"Xierali","year":"2013","journal-title":"Ann Family Med."},{"key":"2020110612433415900_ocw100-B4","article-title":"Use and Characteristics of Electronic Health Record Systems Among Office-Based Physician Practices, United States, 2001-2012","author":"Hsiao","year":"2012"},{"key":"2020110612433415900_ocw100-B5","doi-asserted-by":"crossref","first-page":"373","DOI":"10.1145\/142750.142834","article-title":"Finding usability problems through heuristic evaluation","volume":"1992","author":"Nielsen","year":"1992","journal-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems"},{"issue":"1","key":"2020110612433415900_ocw100-B6","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1197\/jamia.M2187","article-title":"Correlates of electronic health record adoption in office practices: a statewide survey","volume":"14","author":"Simon","year":"2007","journal-title":"J Am Med Inform Assoc."},{"key":"2020110612433415900_ocw100-B7","first-page":"377","article-title":"Usability inspection methods: rationale and examples","volume":"1992","author":"Polson","year":"1992","journal-title":"The 8th Human Interface Symposium (HIS92)"},{"key":"2020110612433415900_ocw100-B8","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1145\/108844.108862","article-title":"User interface evaluation in the real world: a comparison of four techniques","volume":"1991","author":"Jeffries","year":"1991","journal-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems"},{"issue":"5","key":"2020110612433415900_ocw100-B9","doi-asserted-by":"crossref","first-page":"741","DOI":"10.1016\/0020-7373(92)90039-N","article-title":"Cognitive walkthroughs: a method for theory-based evaluation of user interfaces","volume":"36","author":"Polson","year":"1992","journal-title":"Int J Man-machine Stud."},{"key":"2020110612433415900_ocw100-B10","first-page":"221","article-title":"Integrating heuristic evaluation with cognitive walkthrough: development of a hybrid usability inspection method","volume":"208","author":"Kushniruk","year":"2014","journal-title":"Stud Health TechnolI Inform."},{"key":"2020110612433415900_ocw100-B11","first-page":"714","article-title":"A comparison of usability evaluation methods: heuristic evaluation versus end-user think-aloud protocol: an example from a web-based communication tool for nurse scheduling","volume":"2009","author":"Yen","year":"2009","journal-title":"AMIA Annual Symposium Proceedings\/AMIA Symposium AMIA Symposium."},{"key":"2020110612433415900_ocw100-B12","first-page":"322","article-title":"Comparative study of heuristic evaluation and usability testing methods","volume":"143","author":"Thyvalikakath","year":"2009","journal-title":"Stud Health Technol Inform."},{"issue":"5","key":"2020110612433415900_ocw100-B13","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1016\/j.ijmedinf.2011.02.005","article-title":"Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system","volume":"80","author":"Khajouei","year":"2011","journal-title":"Int J Med Inform."},{"key":"2020110612433415900_ocw100-B14","first-page":"23","article-title":"Consolidating the ISO usability models","volume":"2003","author":"Abran","year":"2003","journal-title":"Proceedings of 11th International Software Quality Management Conference"},{"key":"2020110612433415900_ocw100-B15","volume-title":"Usability Inspection Methods","author":"Mack","year":"1994"},{"key":"2020110612433415900_ocw100-B16","first-page":"249","article-title":"Heuristic evaluation of user interfaces","volume":"1990","author":"Nielsen","year":"1990","journal-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems"},{"key":"2020110612433415900_ocw100-B17","first-page":"105","article-title":"The cognitive walkthrough method: a practitioner's guide","volume":"1994","author":"Wharton","year":"1994","journal-title":"Usability Inspection Methods"},{"key":"2020110612433415900_ocw100-B18","first-page":"377","article-title":"Usability inspection methods","volume":"1995","author":"Nielsen","year":"1995","journal-title":"Conference Companion on Human Factors in Computing Systems"},{"issue":"10","key":"2020110612433415900_ocw100-B19","doi-asserted-by":"crossref","first-page":"690","DOI":"10.1016\/j.ijmedinf.2010.08.001","article-title":"Effect of predefined order sets and usability problems on efficiency of computerized medication ordering","volume":"79","author":"Khajouei","year":"2010","journal-title":"Int J Med Inform."},{"key":"2020110612433415900_ocw100-B20","article-title":"Severity ratings for usability problems","author":"Nielsen","year":"1995"},{"key":"2020110612433415900_ocw100-B21","first-page":"102","volume-title":"Usability Engineering","author":"Nielsen","year":"1993"},{"issue":"9","key":"2020110612433415900_ocw100-B22","doi-asserted-by":"crossref","first-page":"638","DOI":"10.1108\/00242530910997928","article-title":"Evaluating the usability and usefulness of a digital library","volume":"58","author":"Buchanan","year":"2009","journal-title":"Library Rev."},{"key":"2020110612433415900_ocw100-B23","first-page":"101","article-title":"A comparison of usability techniques for evaluating design","volume":"1997","author":"Doubleday","year":"1997","journal-title":"Proceedings of the 2nd Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques"},{"issue":"8","key":"2020110612433415900_ocw100-B24","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1016\/j.ijmedinf.2010.05.001","article-title":"Web-based education for low-literate parents in Neonatal Intensive Care Unit: development of a website and heuristic evaluation and usability testing","volume":"79","author":"Choi","year":"2010","journal-title":"Int J Med Inform."},{"issue":"1","key":"2020110612433415900_ocw100-B25","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1136\/jamia.2002.0090001","article-title":"Reference standards, judges, and comparison subjects","volume":"9","author":"Hripcsak","year":"2002","journal-title":"J Am Med Inform Assoc."},{"issue":"4","key":"2020110612433415900_ocw100-B26","doi-asserted-by":"crossref","first-page":"700","DOI":"10.1016\/j.jbi.2010.09.003","article-title":"Methodological concerns in usability evaluation of software prototypes","volume":"44","author":"Khajouei","year":"2011","journal-title":"J Biomed Inform."},{"issue":"1","key":"2020110612433415900_ocw100-B27","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1207\/S15327590IJHC1501_14","article-title":"The evaluator effect: A chilling fact about usability evaluation methods","volume":"15","author":"Hertzum","year":"2003","journal-title":"Int J Hum Comput Interact."},{"key":"2020110612433415900_ocw100-B28","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1145\/142750.142873","article-title":"Comparison of empirical testing and walkthrough methods in user interface evaluation","volume":"1992","author":"Karat","year":"1992","journal-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems"},{"issue":"5","key":"2020110612433415900_ocw100-B29","doi-asserted-by":"crossref","first-page":"340","DOI":"10.1016\/j.ijmedinf.2008.10.002","article-title":"A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence","volume":"78","author":"Jaspers","year":"2009","journal-title":"Int J Med Inform."},{"key":"2020110612433415900_ocw100-B30","first-page":"104","article-title":"Cognitive walkthrough","volume":"2","author":"Blackmon","year":"2004","journal-title":"Encyclopedia Hum Comput Interact."},{"issue":"2","key":"2020110612433415900_ocw100-B31","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1016\/j.jbi.2009.09.002","article-title":"Information system design for a hospital emergency department: a usability analysis of software prototypes","volume":"43","author":"Karahoca","year":"2010","journal-title":"J Biomed Inform."},{"key":"2020110612433415900_ocw100-B32","first-page":"532","article-title":"Usability evaluation of a computerized physician order entry for medication ordering","volume":"2009","author":"Khajouei","year":"2009","journal-title":"MIE."},{"key":"2020110612433415900_ocw100-B33","article-title":"Design approaches for developing user-interfaces accessible to illiterate users","author":"Huenerfauth","year":"2002","journal-title":"Proceedings of the 18th National Conference on Artificial Intelligence"},{"issue":"6","key":"2020110612433415900_ocw100-B34","doi-asserted-by":"crossref","first-page":"773","DOI":"10.1016\/j.intcom.2005.04.004","article-title":"Cross-user analysis: benefits of skill level comparison in usability testing","volume":"17","author":"Faulkner","year":"2005","journal-title":"Interact Comput."},{"issue":"3","key":"2020110612433415900_ocw100-B35","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1016\/S0953-5438(02)00063-2","article-title":"What is this evasive beast we call user satisfaction?","volume":"15","author":"Lindgaard","year":"2003","journal-title":"Interact Comput."}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/24\/e1\/e55\/34149263\/ocw100.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/24\/e1\/e55\/34149263\/ocw100.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,19]],"date-time":"2023-08-19T15:18:58Z","timestamp":1692458338000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/24\/e1\/e55\/2631488"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,8,7]]},"references-count":35,"journal-issue":{"issue":"e1","published-online":{"date-parts":[[2016,8,7]]},"published-print":{"date-parts":[[2017,4,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocw100","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2017,4]]},"published":{"date-parts":[[2016,8,7]]}}}