{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,14]],"date-time":"2025-10-14T00:46:46Z","timestamp":1760402806054,"version":"build-2065373602"},"reference-count":65,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2020,1,27]],"date-time":"2020-01-27T00:00:00Z","timestamp":1580083200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IJERPH"],"abstract":"<jats:p>Background: There has been increasing use of mobile mHealth applications, including pain assessment and pain self-management apps. The usability of mHealth applications has vital importance as it affects the quality of apps. Thus, usability assessment with methodological rigor is essential to minimize errors and undesirable consequences, as well as to increase user acceptance. Objective: this study aimed to synthesize and evaluate existing studies on the assessment of the usability of pain-related apps using a newly developed scale. Methods: an electronic search was conducted in several databases, combining relevant keywords. Then titles and abstracts were screened against inclusion and exclusion criteria. The eligible studies were retrieved and independently screened for inclusion by two authors. Disagreements were resolved by discussion until consensus was reached. Results: a total of 31 articles were eligible for inclusion. Quality assessment revealed that most manuscripts did not assess usability using valid instruments or triangulation of methods of usability assessment. Most manuscripts also failed to assess the three domains of usability (effectiveness, efficiency and satisfaction). Conclusions: future studies should consider existing guidelines on usability assessment design, development and assessment of pain-related apps.<\/jats:p>","DOI":"10.3390\/ijerph17030785","type":"journal-article","created":{"date-parts":[[2020,1,27]],"date-time":"2020-01-27T11:41:57Z","timestamp":1580125317000},"page":"785","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Methodological Quality of Manuscripts Reporting on the Usability of Mobile Applications for Pain Assessment and Management: A Systematic Review"],"prefix":"10.3390","volume":"17","author":[{"given":"Ana F.","family":"Almeida","sequence":"first","affiliation":[{"name":"Institute of Electronics and Informatics Engineering of Aveiro (IEETA), Campus Universit\u00e1rio de Santiago, 3810-193 Aveiro, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3801-7249","authenticated-orcid":false,"given":"Nelson P.","family":"Rocha","sequence":"additional","affiliation":[{"name":"Institute of Electronics and Informatics Engineering of Aveiro (IEETA), Campus Universit\u00e1rio de Santiago, 3810-193 Aveiro, Portugal"},{"name":"Department of Medical Sciences, Universidade de Aveiro - Edif\u00edcio 30, Agras do Crasto - Campus Universit\u00e1rio de Santiago, 3810-193 Aveiro, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4386-5851","authenticated-orcid":false,"given":"Anabela G.","family":"Silva","sequence":"additional","affiliation":[{"name":"School of Health Sciences, Universidade de Aveiro - Edif\u00edcio 30, Agras do Crasto - Campus Universit\u00e1rio de Santiago, 3810-193 Aveiro, Portugal"},{"name":"Center for Health Technology and Services Research, Universidade de Aveiro (CINTESIS.UA), Campus Universit\u00e1rio de Santiago, 3810-193 Aveiro, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2020,1,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/j.mayocp.2014.09.010","article-title":"The epidemiology and economic consequences of pain","volume":"90","author":"Henschke","year":"2015","journal-title":"Mayo Clin. Proc."},{"key":"ref_2","unstructured":"Hughes, R.G. (2008). Improving the quality of care through pain assessment and management. Patient Safety and Quality: An Evidence-Based Handbook for Nurses, AHRQ Publication."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1093\/bja\/aen103","article-title":"Assessment of pain","volume":"101","author":"Breivik","year":"2008","journal-title":"Br. J. Anaesth."},{"key":"ref_4","unstructured":"Vital Wave Consulting (2009). \u201cmHealth for Development: The Opportunity of Mobile Technology for Healthcare in the Developing World\u201d, UN Foundation-Vodafone Foundation Partnership."},{"key":"ref_5","unstructured":"(2019, September 20). Research2Guidance, mHealth Economics 2016 \u2013 Current Status and Trends of the mHealth App Market. Available online: https:\/\/research2guidance.com\/product\/mhealth-app-developer-economics-2016\/."},{"key":"ref_6","unstructured":"(2019, September 20). Pain management-Apple. Available online: https:\/\/www.apple.com\/us\/search\/Pain-management?src=serp."},{"key":"ref_7","unstructured":"(2019, September 20). Pain management app - Android Apps on Google Play. Available online: https:\/\/play.google.com\/store\/search?q=pain%20management%20app&c=apps&hl=en_US."},{"key":"ref_8","unstructured":"(2019, September 20). Pain assessment app - Android Apps on Google Play. Available online: https:\/\/play.google.com\/store\/search?q=Pain%20assessment%20app&c=apps&hl=en_US."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"557","DOI":"10.1097\/AJP.0000000000000171","article-title":"There\u2019s a pain app for that\u2019: Review of patient-targeted smartphone applications for pain management","volume":"31","author":"Lalloo","year":"2015","journal-title":"Clin. J. Pain"},{"key":"ref_10","first-page":"63","article-title":"A systematic review of smartphone applications for chronic pain available for download in the United States","volume":"10","author":"Wallace","year":"2014","journal-title":"J. Opi. Manag."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"e12160","DOI":"10.2196\/12160","article-title":"Usability challenges for health and wellness mobile apps: mixed-methods study among mhealth experts and consumers","volume":"7","author":"Liew","year":"2019","journal-title":"JMIR mHealth uHealth"},{"key":"ref_12","unstructured":"(2019, September 20). ISO 9241-11. Ergonomics of Human-System Interaction \u2014 Part 11: Usability: Definitions and Concepts. Available online: https:\/\/www.google.com\/url?sa=t&rct=j&q=&esrc=s&source=web&cd=5&ved=2ahUKEwjCuYCa9qLnAhWNJaYKHQF9Cd0QFjAEegQIBhAB&url=https%3A%2F%2Finfostore.saiglobal.com%2Fpreview%2Fis%2Fen%2F2018%2Fi.s.eniso9241-11-2018.pdf%3Fsku%3D1980667&usg=AOvVaw1kP3G6g9MDGCVd_WazNF6c."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"663","DOI":"10.1080\/10447318.2014.930311","article-title":"Usability: Lessons learned \u2026 and yet to be learned","volume":"30","author":"Lewis","year":"2014","journal-title":"Int. J. Hum. Comput. Interact."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/j.ijmedinf.2016.06.004","article-title":"Usability in telemedicine systems-A literature survey","volume":"93","author":"Klaassen","year":"2016","journal-title":"Int. J. Med. Inform."},{"key":"ref_15","unstructured":"Daeed, S., Bajwa, I.S., and Mahmood, Z. (2015). Usability evaluation methods: A systematic review. Human Factors in Software Development and Design, IGI Global."},{"key":"ref_16","unstructured":"(2019, September 20). ISO 9241-210:2019. Ergonomics of Human-System Interaction \u2014 Part 210: Human-Centred Design for Interactive Systems. Available online: https:\/\/www.iso.org\/standard\/77520.html."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Middleton, B., Bloomrosen, M., Dente, M.A., Hashmat, B., Koppel, R., Overhage, J.M., Payne, T.H., Rosenbloom, S.T., Weaver, C., and Zhang, J. (2013). Enhancing patient safety and quality of care by improving the usability of electronic health record systems: Recommendations from AMIA. J. Am. Med. Inform. Assoc., 20.","DOI":"10.1136\/amiajnl-2012-001458"},{"key":"ref_18","first-page":"31","article-title":"Avalia\u00e7\u00e3o de usabilidade: Uma revis\u00e3o sistem\u00e1tica da literatura","volume":"11","author":"Martins","year":"2013","journal-title":"RISTI Rev. Iber. Sist. e Tecnol. Inf."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"898","DOI":"10.1111\/pme.12327","article-title":"Assessing the quality and usability of smartphone apps for pain self-management","volume":"15","author":"Reynoldson","year":"2014","journal-title":"PAIN Med."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"e13080","DOI":"10.2196\/13080","article-title":"Evaluation of self-management support functions in apps for people with persistent pain: Systematic review","volume":"7","author":"Devan","year":"2019","journal-title":"JMIR mHealth uHealth"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"e27","DOI":"10.2196\/mhealth.3422","article-title":"Mobile app rating scale: A new tool for assessing the quality of healh mobile apps","volume":"3","author":"Stoyanov","year":"2015","journal-title":"JMIR mHealth uHealth"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Shah, U.M., and Chiew, T.K. (2019). A systematic literature review of the design approach and usability evaluation of the pain management mobile applications. Symmetry, 11.","DOI":"10.3390\/sym11030400"},{"key":"ref_23","first-page":"e14829","article-title":"A scale to assess the methodological quality of studies assessing usability of eHealth products: A delphi study followed by validity and reliability testing","volume":"21","author":"Silva","year":"2019","journal-title":"J. Med. Inter. Res."},{"key":"ref_24","first-page":"104","article-title":"Mobile application and its global impact","volume":"10","author":"Islam","year":"2010","journal-title":"Int. J. Eng. Technol. IJET IJENS"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"276","DOI":"10.11613\/BM.2012.031","article-title":"Interrater reliability: The kappa statistic","volume":"22","author":"McHugh","year":"2012","journal-title":"Biochem. Medica"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Cruz-Cunha, M.M., Miranda, I.M., Martinho, R., and Rijo, R. (2016). Systematic review and evaluation of pain-related mobile applications. Encyclopedia of E-Health and Telemedicine, IGI Global.","DOI":"10.4018\/978-1-4666-9978-6"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"e35","DOI":"10.1093\/jamia\/ocw092","article-title":"A framework for evaluating electronic health record vendor user-centered design and usability testing processes","volume":"24","author":"Ratwani","year":"2017","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_28","first-page":"231","article-title":"Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM","volume":"19","author":"Weir","year":"2005","journal-title":"J. Strength Cond. Res."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1016\/j.procs.2016.09.171","article-title":"Mobile applications in the management of headache","volume":"100","author":"Dantas","year":"2016","journal-title":"Proc. Comput. Sci."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"356","DOI":"10.1016\/j.cmpb.2012.02.005","article-title":"An interactive 3-D application for pain management: Results from a pilot study in spinal cord injury rehabilitation","volume":"108","author":"Spyridonis","year":"2012","journal-title":"Comput. Meth. Prog. Biomed."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1007\/s11042-013-1358-3","article-title":"PainDroid: An android-based virtual reality application for pain assessment","volume":"72","author":"Spyridonis","year":"2014","journal-title":"Multimed. Tools Appl."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1177\/1043454217697022","article-title":"Development of a mhealth real-time pain self-management app for adolescents with cancer: An iterative usability testing study","volume":"34","author":"Jibb","year":"2017","journal-title":"J. Pediatr. Oncol. Nurs."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"e80","DOI":"10.2196\/mhealth.9319","article-title":"Perceptions of adolescents with cancer related to a pain management app and its evaluation: Qualitative study nested within a multicenter pilot feasibility study","volume":"6","author":"Jibb","year":"2018","journal-title":"JMIR mHealth uHealth"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3109\/07434618.2015.1100677","article-title":"Self-reporting tool on pain in people with intellectual disabilities (stop-id!): A usability study","volume":"32","author":"Lobbezoo","year":"2016","journal-title":"Altern. Commun."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Kaltenhauser, A., and Schacht, I. (2018). Quiri: Chronic pain assessment for patients and physicians. MobileHCI\u201918.","DOI":"10.1145\/3236112.3236170"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Jaatun, E.A.A., Haugen, D.F., Dahl, Y., and Kofod-Petersen, A. (2013). An improved digital pain body map. Healthcom.","DOI":"10.1109\/HealthCom.2013.6720765"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Diana, C., Cristina, B., Azucena, G.-P., Luis, F., and Ignacio, M. (2012). Experience-sampling methodology with a mobile device in fibromyalgia. Int. J. Telemed. Appl.","DOI":"10.1155\/2012\/162673"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1007","DOI":"10.4338\/ACI-2016-03-RA-0036","article-title":"Usability and workflow evaluation of \u2018RhEumAtic disease activitY\u2019 (READY). A mobile application for rheumatology patients and providers","volume":"7","author":"Yen","year":"2016","journal-title":"Appl. Clin. Inform."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1001","DOI":"10.1016\/j.jpain.2014.04.009","article-title":"Development and testing of painometer: A smartphone app to assess pain intensity","volume":"15","author":"Roset","year":"2014","journal-title":"J. Pain"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1177\/1359105316650509","article-title":"Fibroline: A mobile app for improving the quality of life of young people with fibromyalgia","volume":"23","author":"Roset","year":"2018","journal-title":"J. Heal. Psychol."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"900","DOI":"10.1097\/AJP.0000000000000618","article-title":"Validity, reliability, feasibility, and usefulness of pain monitor: A multidimensional smartphone app for daily monitoring of adults with heterogenous chronic pain","volume":"34","author":"Castilla","year":"2018","journal-title":"Clin. J. Pain"},{"key":"ref_42","first-page":"2460","article-title":"Evaluating the usability of a virtual reality-based android application in managing the pain experience of wheelchair users","volume":"2012","author":"Spyridonis","year":"2012","journal-title":"IEEE Eng. Med. Biol. Soc."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"557","DOI":"10.1016\/j.pmn.2012.12.001","article-title":"Feasibility of interactive technology for symptom monitoring in patients with fibromyalgia","volume":"15","author":"Vanderboom","year":"2014","journal-title":"Pain Manag. Nurs."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Bedson, J., Hill, J., White, D., Chen, Y., Wathall, S., Dent, S., Cooke, K., and van der Windt, D. (2019). Development and validation of a pain monitoring app for patients with musculoskeletal conditions (The Keele pain recorder feasibility study). BMC Med. Inform. Decis. Mak., 19.","DOI":"10.1186\/s12911-019-0741-z"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1515\/cdbme-2018-0151","article-title":"MigraineMonitor - Towards a system for the prediction of migraine attacks using electrostimulation","volume":"4","author":"Stefke","year":"2018","journal-title":"Curr. Dir. Biomed. Eng."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1089\/tmj.2010.0114","article-title":"Heuristic evaluation and usability testing of a computerized patient-reported outcomes survey for headache sufferers","volume":"17","author":"Smith","year":"2011","journal-title":"Telemed. J. E. Health."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"2","DOI":"10.2196\/mhealth.3879","article-title":"Testing the feasibility and psychometric properties of a mobile diary (myWHI) in adolescents and young adults with headaches","volume":"3","author":"Huguet","year":"2015","journal-title":"JMIR mHealth uHealth"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1093\/pm\/pny080","article-title":"User design and experience preferences in a novel smartphone application for migraine management: A think aloud study of the RELAXaHEAD application","volume":"20","author":"Minen","year":"2019","journal-title":"Pain Med."},{"key":"ref_49","first-page":"e51","article-title":"Development and testing of a multidimensional iPhone pain assessment application for adolescents with cancer","volume":"15","author":"Stinson","year":"2013","journal-title":"J. Med. Inter. Res."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1016\/j.ejon.2016.03.009","article-title":"Feasibility of a mobile and web-based intervention to support self-management in outpatients with cancer pain","volume":"23","author":"Hochstenbach","year":"2016","journal-title":"Eur. J. Oncol. Nurs."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"202","DOI":"10.1016\/j.compbiomed.2016.07.012","article-title":"Pain buddy: A novel use of m-health in the management of children\u2019s cancer pain","volume":"76","author":"Fortier","year":"2016","journal-title":"Comput. Biol. Med."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"e127","DOI":"10.2196\/mhealth.8409","article-title":"Designing a tablet-based software app for mapping bodily symptoms: Usability evaluation and reproducibility analysis","volume":"6","author":"Neubert","year":"2018","journal-title":"JMIR mHealth uHealth"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"e1","DOI":"10.2196\/humanfactors.3302","article-title":"Development and pilot evaluation of an online relapse-prevention program based on acceptance and commitment therapy for chronic pain patients","volume":"2","author":"Fledderus","year":"2015","journal-title":"JMIR Hum. Factor."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"280","DOI":"10.1080\/24740527.2018.1534543","article-title":"A parent-science partnership to improve postsurgical pain management in young children: Co-development and usability testing of the Achy Penguin smartphone-based app","volume":"2","author":"Birnie","year":"2018","journal-title":"Can. J. Pain"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"897","DOI":"10.1111\/pan.13471","article-title":"In-hospital usability and feasibility evaluation of Panda, an app for the management of pain in children at home","volume":"28","author":"Sun","year":"2018","journal-title":"Paediatr. Anaesth."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"1372","DOI":"10.1007\/978-3-319-96071-5_139","article-title":"My pain coach: A mobile system with tangible interface for pain assessment","volume":"824","author":"Caon","year":"2019","journal-title":"Adv. Intell. Syst. Comput."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"21","DOI":"10.24193\/jebp.2017.1.2","article-title":"A mobile intervention for core needle biopsy related pain and anxiety: A usability study","volume":"17","author":"Cardos","year":"2017","journal-title":"J. Evid. Based Psychother."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"e10187","DOI":"10.2196\/10187","article-title":"Usability, acceptability, and usefulness of an mhealth app for diagnosing and monitoring patients with breakthrough cancer pain","volume":"5","author":"Boceta","year":"2019","journal-title":"JMIR Can."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"1121","DOI":"10.1093\/pm\/pnx028","article-title":"Usability testing of the iphone app to improve pain assessment for older adults with cognitive impairment (prehospital setting): A qualitative study","volume":"19","author":"Docking","year":"2018","journal-title":"Pain Med."},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Streiner, D.L., Norman, G.R., and Cairney, J. (2015). Health Measurement Scales, Oxford University Press. [5th ed.].","DOI":"10.1093\/med\/9780199685219.001.0001"},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"349","DOI":"10.1093\/intqhc\/mzm042","article-title":"Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups","volume":"19","author":"Tong","year":"2007","journal-title":"Int. J. Qual. Heal. Care"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1136\/jnnp.2003.025833","article-title":"How to spot bias and other potential problems in randomised controlled trials","volume":"75","author":"Lewis","year":"2004","journal-title":"J. Neurol. Neurosurg. Psych."},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1093\/jamia\/ocw046","article-title":"An appraisal of published usability evaluations of electronic health records via systematic review","volume":"24","author":"Ellsworth","year":"2017","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_64","doi-asserted-by":"crossref","unstructured":"Frokjaer, E., Hertzum, M., and Hornb\u00e6k, K. (2000, January 1\u20136). Measuring Usability: Are Effectiveness, Efficiency, and Satisfaction Really Correlated?. Proceedings of the Conference on Human Factors in Computing Systems, Hague, The Netherlands.","DOI":"10.1145\/332040.332455"},{"key":"ref_65","first-page":"1","article-title":"There\u2019s an app for that: Mobile technology is a new advantage in managing chronic pain","volume":"21","author":"Vardeh","year":"2013","journal-title":"Pain Clin. Updat."}],"container-title":["International Journal of Environmental Research and Public Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1660-4601\/17\/3\/785\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T14:08:38Z","timestamp":1760364518000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1660-4601\/17\/3\/785"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,27]]},"references-count":65,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2020,2]]}},"alternative-id":["ijerph17030785"],"URL":"https:\/\/doi.org\/10.3390\/ijerph17030785","relation":{},"ISSN":["1660-4601"],"issn-type":[{"type":"electronic","value":"1660-4601"}],"subject":[],"published":{"date-parts":[[2020,1,27]]}}}