{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T09:03:09Z","timestamp":1775120589202,"version":"3.50.1"},"reference-count":37,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"\n Background<\/jats:title>\n Usability evaluation both by experts and target users is an integral part of the process of developing and assessing digital solutions. Usability evaluation improves the probability of having digital solutions that are easier, safer, more efficient, and more pleasant to use. However, despite the widespread recognition of the importance of usability evaluation, there is a lack of research and consensus on related concepts and reporting standards.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n The aim of the study is to generate consensus on terms and procedures that should be considered when planning and reporting a study on a usability evaluation of health-related digital solutions both by users and experts and provide a checklist that can easily be used by researchers when conducting their usability studies.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n A Delphi study with 2 rounds was conducted with a panel of international participants experienced in usability evaluation. In the first round, they were asked to comment on definitions, rate the importance of preidentified methodological procedures using a 9-item Likert scale, and suggest additional procedures. In the second round, experienced participants were asked to reappraise the relevance of each procedure informed by round 1 results. Consensus on the relevance of each item was defined a priori when at least 70% or more experienced participants scored an item 7 to 9 and less than 15% of participants scored the same item 1 to 3.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n A total of 30 participants (n=20 females) from 11 different countries entered the Delphi study with a mean age of 37.2 (SD 7.7) years. Agreement was achieved on the definitions for all usability evaluation\u2013related terms proposed (usability assessment moderator, participant, usability evaluation method, usability evaluation technique, tasks, usability evaluation environment, usability evaluator, and domain evaluator). A total of 38 procedures related to usability evaluation planning and reporting were identified across rounds (28 were related to usability evaluation involving users and 10 related to usability evaluation involving experts). Consensus on the relevance was achieved for 23 (82%) of the procedures related to usability evaluation involving users and for 7 (70%) of the usability evaluation procedures involving experts. A checklist was proposed that can guide authors when designing and reporting usability studies.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n This study proposes a set of terms and respective definitions as well as a checklist to guide the planning and reporting of usability evaluation studies, constituting an important step toward a more standardized approach in the field of usability evaluation that may contribute to enhancing the quality of planning and reporting usability studies. Future studies can contribute to further validating this study work by refining the definitions, assessing the practical applicability of the checklist, or assessing whether using this checklist results in higher-quality digital solutions.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/44326","type":"journal-article","created":{"date-parts":[[2023,3,10]],"date-time":"2023-03-10T05:34:26Z","timestamp":1678426466000},"page":"e44326","source":"Crossref","is-referenced-by-count":10,"title":["Consensus on the Terms and Procedures for Planning and Reporting a Usability Evaluation of Health-Related Digital Solutions: Delphi Study and a Resulting Checklist"],"prefix":"10.2196","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5220-0528","authenticated-orcid":false,"given":"Ana Isabel","family":"Martins","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4732-5959","authenticated-orcid":false,"given":"Gon\u00e7alo","family":"Santinha","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7349-457X","authenticated-orcid":false,"given":"Ana Margarida","family":"Almeida","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4740-7951","authenticated-orcid":false,"given":"\u00d3scar","family":"Ribeiro","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9383-7659","authenticated-orcid":false,"given":"Telmo","family":"Silva","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3801-7249","authenticated-orcid":false,"given":"Nelson","family":"Rocha","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4386-5851","authenticated-orcid":false,"given":"Anabela G","family":"Silva","sequence":"additional","affiliation":[]}],"member":"1010","published-online":{"date-parts":[[2023,6,6]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.1007\/s10209-019-00659-w"},{"key":"ref2","author":"Nielsen, J","year":"1993","journal-title":"Usability Engineering"},{"key":"ref3","unstructured":"Ergonomics of human-system interaction \u2014 part 210: human-centred design for interactive systems (ISO 9241-210:2019)International Organization for Standardization20192023-05-02https:\/\/www.iso.org\/standard\/77520.html"},{"key":"ref4","author":"Dix, A","year":"2004","journal-title":"Human-Computer Interaction, 3rd ed"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2012-001458"},{"key":"ref6","doi-asserted-by":"crossref","first-page":"250","DOI":"10.4018\/978-1-4666-6485-2.ch013","author":"Martins, AI","year":"2015","journal-title":"Human Factors in Software Development and Design"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.1006\/ijhc.2001.0503"},{"key":"ref8","first-page":"1","author":"Petrie, H","year":"2009","journal-title":"The Universal Access Handbook"},{"key":"ref9","unstructured":"Dispositivos m\u00e9dicosInfarmed20132023-05-02http:\/\/www.webcitation.org\/6OqjwM9NN"},{"key":"ref10","first-page":"17","volume":"189","year":"1990","journal-title":"Off J Eur Comm"},{"key":"ref11","unstructured":"Human factors engineering for medical devicesEmergo UL20142023-05-02https:\/\/www.emergobyul.com\/resources\/human- factors-engineering-medical-devices"},{"key":"ref12","unstructured":"KayeRDNorthRAPetersonMKUPCARE: an analysis, description, and educational tool for medical device use problems2003Eighth Annual International Conference of Industrial Engineering Theory, Applications and Practice2003Las Vegas, NV27"},{"key":"ref13","unstructured":"WoodsDCookRIThe new look at error, safety, and failure: a primer for health careChicago National Patient Safety Foundation19992023-04-05http:\/\/csel.eng.ohio-state.edu\/productions\/pexis\/readings\/submod1\/NewLookPrimer.pdf"},{"key":"ref14","unstructured":"ShaverEHuman Factors and Medical Device Resources20132023-05-02https:\/\/www.ericshaver.com\/?s=Human+Factors+and +Medical+Device+Resources"},{"key":"ref15","first-page":"412","author":"Wiklund, ME","year":"2011","journal-title":"Usability Testing of Medical Devices. 1st ed"},{"key":"ref16","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2008.12.004"},{"key":"ref17","unstructured":"Association for the Advancement of Medical InstrumentationMedical devices\u2014application of usability engineering to medical devices ANSI\/AAMI\/IEC 62366:2007 (R2013)ANSI Webstore20132023-05-02https:\/\/webstore.ansi.org\/standards\/aami\/ansiaamiiec623662007r2013a1"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.3332\/ecancer.2014.399"},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.2196\/22774"},{"key":"ref20","doi-asserted-by":"publisher","DOI":"10.1177\/20552076211065271"},{"key":"ref21","doi-asserted-by":"publisher","DOI":"10.1017\/S0266462318000521"},{"key":"ref22","doi-asserted-by":"publisher","DOI":"10.3390\/ijerph17030785"},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2011.08.004"},{"key":"ref24","unstructured":"BevanNCommon industry format usability testsProceedings of Usability Professionals Association1999UPA'98June 29-July 2, 1999Scottsdale, AZ"},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocy050"},{"key":"ref26","unstructured":"CONSORT GroupCONSORT\u2014Consolidated Standards of Reporting Trials2021-03-30http:\/\/www.consort-statement.org\/"},{"issue":"4","key":"ref27","doi-asserted-by":"crossref","first-page":"1008","DOI":"10.1046\/j.1365-2648.2000.t01-1-01567.x","volume":"32","author":"Hasson, F","year":"2000","journal-title":"J Adv Nurs"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.7275\/pdz9-th90"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.1145\/3439231.3439238"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-020-01029-4"},{"key":"ref31","doi-asserted-by":"publisher","DOI":"10.1016\/j.jval.2017.06.011"},{"key":"ref32","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocw046"},{"key":"ref33","doi-asserted-by":"publisher","DOI":"10.1207\/s15327590ijhc1501_14"},{"key":"ref34","doi-asserted-by":"publisher","DOI":"10.1080\/0144929x.2013.783114"},{"key":"ref35","first-page":"258","author":"Riihiaho, S","year":"2018","journal-title":"The Wiley Handbook of Human Computer Interaction. Vol 1"},{"key":"ref36","doi-asserted-by":"publisher","DOI":"10.1093\/intqhc\/mzm042"},{"issue":"3","key":"ref37","doi-asserted-by":"crossref","first-page":"i-iv, 1","DOI":"10.3310\/hta2030","volume":"2","author":"Murphy, MK","year":"1998","journal-title":"Health Technol Assess"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2023,12,8]],"date-time":"2023-12-08T13:02:31Z","timestamp":1702040551000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2023\/1\/e44326"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,6]]},"references-count":37,"URL":"https:\/\/doi.org\/10.2196\/44326","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.44326","asserted-by":"object"}]},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,6]]}}}