{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,6]],"date-time":"2026-01-06T02:22:11Z","timestamp":1767666131701},"reference-count":29,"publisher":"Georg Thieme Verlag KG","issue":"04","license":[{"start":{"date-parts":[[2020,8,12]],"date-time":"2020-08-12T00:00:00Z","timestamp":1597190400000},"content-version":"vor","delay-in-days":11,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Appl Clin Inform"],"published-print":{"date-parts":[[2020,8]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>\n          Background\u2003With the increased usage of dashboard reporting systems to monitor and track patient panels by clinical users, developers must ensure that the information displays they produce are accurate and intuitive. When evaluating usability of a clinical dashboard among potential end users, developers oftentimes rely on methods such as questionnaires as opposed to other, more time-intensive strategies that incorporate direct observation.<\/jats:p><jats:p>\n          Objectives\u2003Prior to release of the potentially inappropriate medication (PIM) clinical dashboard, designed to facilitate completion of a quality improvement project by clinician scholars enrolled in the Veterans Affairs (VA) workforce development Geriatric Scholars Program (GSP), we evaluated the usability of the system. This article describes the process of usability testing a dashboard reporting system with clinicians using direct observation and think-aloud moderating techniques.<\/jats:p><jats:p>\n          Methods\u2003We developed a structured interview protocol that combines virtual observation, think-aloud moderating techniques, and retrospective questioning of the overall user experience, including use of the System Usability Scale (SUS). Thematic analysis was used to analyze field notes from the interviews of three GSP alumni.<\/jats:p><jats:p>\n          Results\u2003Our structured approach to usability testing identified specific functional problems with the dashboard reporting system that were missed by results from the SUS. Usability testing lead to overall improvements in the intuitive use of the system, increased data transparency, and clarification of the dashboard's purpose.<\/jats:p><jats:p>\n          Conclusion\u2003Reliance solely on questionnaires and surveys at the end stages of dashboard development can mask potential functional problems that will impede proper usage and lead to misinterpretation of results. A structured approach to usability testing in the developmental phase is an important tool for developers of clinician friendly systems for displaying easily digested information and tracking outcomes for the purpose of quality improvement.<\/jats:p>","DOI":"10.1055\/s-0040-1714693","type":"journal-article","created":{"date-parts":[[2020,8,12]],"date-time":"2020-08-12T23:10:00Z","timestamp":1597273800000},"page":"528-534","source":"Crossref","is-referenced-by-count":26,"title":["Usability Testing a Potentially Inappropriate Medication Dashboard: A Core Component of the Dashboard Development Process"],"prefix":"10.1055","volume":"11","author":[{"given":"Regina","family":"Richter Lagha","sequence":"additional","affiliation":[{"name":"Veterans Affairs Greater Los Angeles Medical Center, Geriatric Research Education and Clinical Center, Los Angeles, California, United States"}]},{"given":"Zachary","family":"Burningham","sequence":"additional","affiliation":[{"name":"Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City, Utah, United States"},{"name":"Salt Lake City Veterans Affairs Medical Center, Health Services Research and Development Center, Salt Lake City, Utah, United States"}]},{"given":"Brian C.","family":"Sauer","sequence":"additional","affiliation":[{"name":"Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City, Utah, United States"},{"name":"Salt Lake City Veterans Affairs Medical Center, Health Services Research and Development Center, Salt Lake City, Utah, United States"}]},{"given":"Jianwei","family":"Leng","sequence":"additional","affiliation":[{"name":"Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City, Utah, United States"},{"name":"Salt Lake City Veterans Affairs Medical Center, Health Services Research and Development Center, Salt Lake City, Utah, United States"}]},{"given":"Celena","family":"Peters","sequence":"additional","affiliation":[{"name":"Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City, Utah, United States"},{"name":"Salt Lake City Veterans Affairs Medical Center, Health Services Research and Development Center, Salt Lake City, Utah, United States"}]},{"given":"Tina","family":"Huynh","sequence":"additional","affiliation":[{"name":"Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City, Utah, United States"},{"name":"Salt Lake City Veterans Affairs Medical Center, Health Services Research and Development Center, Salt Lake City, Utah, United States"}]},{"given":"Shardool","family":"Patel","sequence":"additional","affiliation":[{"name":"Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City, Utah, United States"},{"name":"Salt Lake City Veterans Affairs Medical Center, Health Services Research and Development Center, Salt Lake City, Utah, United States"}]},{"given":"Ahmad S.","family":"Halwani","sequence":"additional","affiliation":[{"name":"Salt Lake City Veterans Affairs Medical Center, Health Services Research and Development Center, Salt Lake City, Utah, United States"},{"name":"Division of Epidemiology, Department of Hematology, University of Utah, Salt Lake City, Utah, United States"}]},{"given":"B. Josea","family":"Kramer","sequence":"additional","affiliation":[{"name":"Veterans Affairs Greater Los Angeles Medical Center, Geriatric Research Education and Clinical Center, Los Angeles, California, United States"},{"name":"Division of Geriatric Medicine, David Geffen School of Medicine, University of California at Los Angeles, Los Angeles, California, United States"}]}],"member":"194","published-online":{"date-parts":[[2020,8,12]]},"reference":[{"issue":"06","key":"ref1","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1136\/bmjqs-2017-007563","article-title":"Using report cards and dashboards to drive quality improvement: lessons learnt and lessons still to learn","volume":"27","author":"N M Ivers","year":"2018","journal-title":"BMJ Qual Saf"},{"issue":"15","key":"ref2","doi-asserted-by":"crossref","first-page":"1339","DOI":"10.1001\/jama.280.15.1339","article-title":"Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review","volume":"280","author":"D L Hunt","year":"1998","journal-title":"JAMA"},{"issue":"04","key":"ref3","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1097\/00002727-200610000-00008","article-title":"Clinical dashboards: impact on workflow, care quality, and patient safety","volume":"29","author":"M Egan","year":"2006","journal-title":"Crit Care Nurs Q"},{"issue":"05","key":"ref4","doi-asserted-by":"crossref","first-page":"317","DOI":"10.5455\/aim.2015.23.317-321","article-title":"Development of performance dashboards in healthcare sector: key practical issues","volume":"23","author":"M Ghazisaeidi","year":"2015","journal-title":"Acta Inform Med"},{"issue":"04","key":"ref5","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1006\/jbin.2001.1024","article-title":"Heterogeneous database integration in biomedicine","volume":"34","author":"W Sujansky","year":"2001","journal-title":"J Biomed Inform"},{"issue":"04","key":"ref7","doi-asserted-by":"crossref","first-page":"e18","DOI":"10.1016\/j.ijmedinf.2008.12.004","article-title":"Usability testing: a review of some methodological and technical aspects of the method","volume":"79","author":"J MC Bastien","year":"2010","journal-title":"Int J Med Inform"},{"issue":"03","key":"ref8","doi-asserted-by":"crossref","first-page":"543","DOI":"10.1055\/s-0039-1693688","article-title":"Quality initiative using theory of change and visual analytics to improve controlled substance documentation discrepancies in the operating room","volume":"10","author":"J E Dolan","year":"2019","journal-title":"Appl Clin Inform"},{"key":"ref9","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1186\/s13012-016-0427-1","article-title":"A mixed methods evaluation of the maternal-newborn dashboard in Ontario: dashboard attributes, contextual factors, and facilitators and barriers to use: a study protocol","volume":"11","author":"S Dunn","year":"2016","journal-title":"Implement Sci"},{"key":"ref10","doi-asserted-by":"crossref","first-page":"98","DOI":"10.1016\/j.ijmedinf.2016.10.001","article-title":"A novel concept for integrating and delivering health information using a comprehensive digital dashboard: an analysis of healthcare professionals' intention to adopt a new system and the trend of its real usage","volume":"97","author":"K Lee","year":"2017","journal-title":"Int J Med Inform"},{"issue":"12","key":"ref11","first-page":"676","article-title":"User-centered collaborative design and development of an inpatient safety dashboard","volume":"43","author":"E Mlaver","year":"2017","journal-title":"Jt Comm J Qual Patient Saf"},{"key":"ref12","doi-asserted-by":"crossref","first-page":"45","DOI":"10.2345\/0899-8205-47.s2.45","article-title":"A review: healthcare usability evaluation methods","author":"G Bhutkar","year":"2013","journal-title":"Biomed Instrum Technol"},{"issue":"11","key":"ref14","doi-asserted-by":"crossref","first-page":"2343","DOI":"10.1111\/jgs.14382","article-title":"Veterans Affairs Geriatric Scholars Program: enhancing existing primary care clinician skills in caring for older veterans","volume":"64","author":"B J Kramer","year":"2016","journal-title":"J Am Geriatr Soc"},{"issue":"09","key":"ref15","first-page":"425","article-title":"VA Geriatric Scholars Program's impact on prescribing potentially inappropriate medications","volume":"25","author":"Z Burningham","year":"2019","journal-title":"Am J Manag Care"},{"issue":"04","key":"ref16","doi-asserted-by":"crossref","first-page":"290","DOI":"10.1136\/bmjqs-2013-001862","article-title":"Systematic review of the application of the plan-do-study-act method to improve quality in healthcare","volume":"23","author":"M J Taylor","year":"2014","journal-title":"BMJ Qual Saf"},{"key":"ref18","doi-asserted-by":"crossref","first-page":"674","DOI":"10.1111\/jgs.15767","article-title":"American Geriatrics Society 2019 Updated AGS Beers Criteria\u00ae for Potentially Inappropriate Medication Use in Older Adults","volume":"67","year":"2019","journal-title":"J Am Geriatr Soc"},{"key":"ref19","first-page":"189","volume-title":"SUS: A \u201cquick and dirty\u201d usability scale","author":"J Brooke","year":"1996"},{"key":"ref20","year":"2006"},{"key":"ref21","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/5657.001.0001","volume-title":"Protocol Analysis: Verbal Reports as Data, 2nd ed","author":"K Ericsson","year":"1993"},{"issue":"06","key":"ref23","doi-asserted-by":"crossref","first-page":"807","DOI":"10.1016\/j.cptl.2018.03.019","article-title":"Thematic analysis of qualitative research data: Is it as easy as it sounds?","volume":"10","author":"A Castleberry","year":"2018","journal-title":"Curr Pharm Teach Learn"},{"issue":"01","key":"ref24","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1177\/1609406917733847","article-title":"Thematic analysis: striving to meet the trustworthiness criteria","volume":"16","author":"L S Nowell","year":"2017","journal-title":"Int J Qual Methods"},{"issue":"08","key":"ref25","doi-asserted-by":"crossref","first-page":"743","DOI":"10.1001\/jama.2019.0161","article-title":"A decade of health information technology usability challenges and the path forward","volume":"321","author":"R M Ratwani","year":"2019","journal-title":"JAMA"},{"issue":"12","key":"ref26","doi-asserted-by":"crossref","first-page":"1276","DOI":"10.1001\/jama.2018.1171","article-title":"Electronic health record usability issues and potential contribution to patient harm","volume":"319","author":"J L Howe","year":"2018","journal-title":"JAMA"},{"key":"ref28","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1016\/j.ijmedinf.2019.03.018","article-title":"Methods of usability testing in the development of eHealth applications: a scoping review","volume":"126","author":"I Maramba","year":"2019","journal-title":"Int J Med Inform"},{"issue":"03","key":"ref29","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1055\/s-0038-1666842","article-title":"The development of heuristics for evaluation of dashboard visualizations","volume":"9","author":"D Dowding","year":"2018","journal-title":"Appl Clin Inform"},{"issue":"01","key":"ref30","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1097\/CIN.0000000000000484","article-title":"Usability evaluation of a dashboard for home care nurses","volume":"37","author":"D Dowding","year":"2019","journal-title":"Comput Inform Nurs"},{"issue":"09","key":"ref32","doi-asserted-by":"crossref","first-page":"727","DOI":"10.1136\/bmjqs-2013-002627","article-title":"The frequency of diagnostic errors in outpatient care: estimations from three large observational studies involving US adult populations","volume":"23","author":"H Singh","year":"2014","journal-title":"BMJ Qual Saf"},{"key":"ref33","doi-asserted-by":"crossref","first-page":"860","DOI":"10.1136\/bmj.326.7394.860","article-title":"Evaluating computerised health information systems: hard lessons still to be learnt","volume":"326","author":"P Littlejohns","year":"2003","journal-title":"BMJ"},{"issue":"03","key":"ref34","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1109\/47.867942","article-title":"Thinking aloud: reconciling theory and practice","volume":"43","author":"M T Boren","year":"2000","journal-title":"IEEE Trans Prof Commun"},{"issue":"03","key":"ref35","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1037\/0033-295X.84.3.231","article-title":"Telling more than we know: verbal reports on mental processes","volume":"84","author":"R E Nisbett","year":"1977","journal-title":"Psychol Rev"}],"container-title":["Applied Clinical Informatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.thieme-connect.de\/products\/ejournals\/pdf\/10.1055\/s-0040-1714693.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,8,12]],"date-time":"2020-08-12T23:10:35Z","timestamp":1597273835000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.thieme-connect.de\/DOI\/DOI?10.1055\/s-0040-1714693"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8]]},"references-count":29,"journal-issue":{"issue":"04","published-online":{"date-parts":[[2020,8,5]]},"published-print":{"date-parts":[[2020,8]]}},"URL":"https:\/\/doi.org\/10.1055\/s-0040-1714693","relation":{},"ISSN":["1869-0327"],"issn-type":[{"value":"1869-0327","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8]]}}}