{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T12:51:06Z","timestamp":1787316666722,"version":"build-2736575974"},"reference-count":47,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Immune checkpoint inhibitors are widely used in oncology but can cause immune-related adverse events (irAEs), which may be severe or life-threatening if not detected early. Electronic patient-reported outcome (ePRO) symptom monitoring systems may facilitate timely recognition and management of irAEs. Usability testing is a critical stage in ePRO system development, yet no published examples of formal usability and acceptance testing exist.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to assess the usability and acceptance of a co-designed ePRO symptom monitoring prototype for irAEs embedded within the Epic electronic medical record.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Testing was conducted at an Australian quaternary cancer center. Eligible participants were patients who had received or were receiving immune checkpoint inhibitors, their caregivers, or clinicians (oncologists and nurse specialists). Participants completed baseline digital literacy assessments (16-item Mobile Device Proficiency Questionnaire [MDPQ-16] and 12-item Computer Proficiency Questionnaire [CPQ-12]) before a structured testing session. Each session involved role-specific tasks using the patient-facing Health Hub or the clinician-facing Epic electronic medical record. Usability was assessed using the System Usability Scale (SUS). Acceptance was assessed using a customized Unified Theory of Acceptance and Use of Technology (UTAUT) questionnaire. Semistructured interviews were used to capture qualitative feedback.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n A total of 30 participants (7 patients, 3 caregivers, 10 oncologists, and 10 nurse specialists) completed 10 testing sessions. Median MDPQ-16 and CPQ-12 scores were higher for clinicians compared to patients and caregivers. Median SUS scores indicated high usability\u2014patients and caregivers: 77.5% (IQR 70.0%\u201086.3%), oncologists: 82.5% (IQR 80.0%\u201090.0%), and nurse specialists: 80.0% (IQR 75.6%\u201094.4%). Median UTAUT scores demonstrated strong user acceptance\u2014patients or caregivers: 4.27 (IQR 4.09\u20104.58), oncologists: 4.33 (IQR 4\u20104.63), and nurse specialists: 4.23 (IQR 3.87\u20104.57). Health Hub usability themes highlighted overall ease of navigation and efficiency of reporting, but a need for clearer survey navigation, simplification of the actions page, and improved organization of trend graphs. For clinicians, themes included efficient side effect capture and intuitive system design, but a need to improve navigation to results, optimize data display, and facilitate team-based alert management. Health Hub acceptance themes highlighted patient empowerment to self-manage, enhanced patient-clinician communication, and reinforcement of existing care. However, concerns were raised about digital equity for vulnerable groups. Clinicians reported that the system streamlined side effect management between visits, aligned with existing Epic workflows, and could be tailored to personal preferences. Concerns remained regarding additional workload and medico-legal responsibilities associated with real-time alerts.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n The ePRO prototype demonstrated high levels of usability and acceptance across patients, caregivers, and clinicians. Limitations around navigation and data visualization, alongside equity and workload concerns, will guide refinements prior to implementation. These findings emphasize the value of rigorous formative usability and acceptance testing to optimize ePRO systems prior to deployment in routine cancer care.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/79694","type":"journal-article","created":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T18:24:23Z","timestamp":1773858263000},"page":"e79694-e79694","source":"Crossref","is-referenced-by-count":1,"title":["Usability and Acceptance Testing of an Electronic Patient-Reported Outcome Symptom Monitoring System for People Receiving Immune Checkpoint Inhibitors: Mixed Methods Study"],"prefix":"10.2196","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5388-1695","authenticated-orcid":false,"given":"Julia","family":"Lai-Kwon","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4637-4572","authenticated-orcid":false,"given":"Claudia","family":"Rutherford","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1107-8976","authenticated-orcid":false,"given":"Stephanie","family":"Best","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2439-2462","authenticated-orcid":false,"given":"Hasan Shahid","family":"Ferdous","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-3800-658X","authenticated-orcid":false,"given":"Iris","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-0407-229X","authenticated-orcid":false,"given":"Thai-Khoa","family":"Ly","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8616-9143","authenticated-orcid":false,"given":"Dishan","family":"Herath","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6710-681X","authenticated-orcid":false,"given":"Kate","family":"Burbury","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8792-7807","authenticated-orcid":false,"given":"Michael","family":"Jefford","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,3,18]]},"reference":[{"issue":"39","key":"R1","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1200\/EDBK_100018","article-title":"Managing immuno-oncology toxicity: top 10 innovative institutional solutions","volume":"39","author":"Cole","journal-title":"Am Soc Clin Oncol Educ Book"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","DOI":"10.1186\/s40425-017-0300-z","article-title":"Managing toxicities associated with immune checkpoint inhibitors: consensus recommendations from the Society for Immunotherapy of Cancer (SITC) Toxicity Management Working Group","volume":"5","author":"Puzanov","journal-title":"J Immunother Cancer"},{"issue":"2","key":"R3","doi-asserted-by":"publisher","first-page":"339","DOI":"10.4143\/crt.2020.790","article-title":"The pattern of time to onset and resolution of immune-related adverse events caused by immune checkpoint inhibitors in cancer: a pooled analysis of 23 clinical trials and 8,436 patients","volume":"53","author":"Tang","journal-title":"Cancer Res Treat"},{"issue":"12","key":"R4","doi-asserted-by":"publisher","first-page":"1721","DOI":"10.1001\/jamaoncol.2018.3923","article-title":"Fatal toxic effects associated with immune checkpoint inhibitors: a systematic review and meta-analysis","volume":"4","author":"Wang","journal-title":"JAMA Oncol"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejca.2020.02.025","article-title":"Emergency presentations in patients treated with immune checkpoint inhibitors","volume":"130","author":"Cooksley","journal-title":"Eur J Cancer"},{"issue":"5","key":"R6","doi-asserted-by":"publisher","first-page":"4019","DOI":"10.1007\/s00520-022-06826-9","article-title":"Immune checkpoint inhibitors: immune-related adverse events, healthcare utilization, and costs among commercial and Medicare Advantage patients","volume":"30","author":"Gunturu","journal-title":"Support Care Cancer"},{"issue":"15_suppl","key":"R7","doi-asserted-by":"publisher","first-page":"e20655","DOI":"10.1200\/JCO.2019.37.15_suppl.e20655","article-title":"Health care utilization associated with adverse events (AEs) among metastatic non-small cell lung cancer (mNSCLC) patients treated with immunotherapy or chemotherapy","volume":"37","author":"Engel-Nitz","journal-title":"J Clin Oncol"},{"issue":"7","key":"R8","doi-asserted-by":"publisher","first-page":"e1205","DOI":"10.1002\/onco.13812","article-title":"The impact of adverse events on health care resource utilization, costs, and mortality among patients treated with immune checkpoint inhibitors","volume":"26","author":"George","journal-title":"Oncologist"},{"issue":"2","key":"R9","doi-asserted-by":"publisher","first-page":"132","DOI":"10.1080\/13696998.2019.1696349","article-title":"Hospitalization and emergency department utilization in patients with advanced melanoma receiving pembrolizumab versus ipilimumab plus nivolumab in US academic centers","volume":"23","author":"Joseph","journal-title":"J Med Econ"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.1016\/j.eclinm.2024.102536","article-title":"Clinical spectrum and evolution of immune-checkpoint inhibitors toxicities over a decade-a worldwide perspective","volume":"70","author":"Gougis","journal-title":"EClinicalMedicine"},{"issue":"2","key":"R11","doi-asserted-by":"publisher","first-page":"E97","DOI":"10.1097\/NCC.0000000000000683","article-title":"Exploring the experiences of people treated with immunotherapies for advanced melanoma and those caring for them: \u201cReal-World\u201d data","volume":"43","author":"Milne","journal-title":"Cancer Nurs"},{"issue":"11","key":"R12","doi-asserted-by":"publisher","first-page":"e1190","DOI":"10.1634\/theoncologist.2018-0820","article-title":"Balancing the hype with reality: what do patients with advanced melanoma consider when making the decision to have immunotherapy?","volume":"24","author":"Wong","journal-title":"Oncologist"},{"issue":"9","key":"R13","doi-asserted-by":"publisher","first-page":"7249","DOI":"10.1007\/s00520-022-07133-z","article-title":"Exploring supportive care and information needs through a proposed eHealth application among melanoma patients undergoing systemic therapy: a qualitative study","volume":"30","author":"Fraterman","journal-title":"Support Care Cancer"},{"issue":"9","key":"R14","doi-asserted-by":"publisher","first-page":"878","DOI":"10.1016\/j.annonc.2022.04.007","article-title":"The role of patient-reported outcome measures in the continuum of cancer clinical care: ESMO Clinical Practice Guideline","volume":"33","author":"Di Maio","journal-title":"Ann Oncol"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.2196\/49089","article-title":"Examining the effectiveness of electronic patient-reported outcomes in people with cancer: systematic review and meta-analysis","volume":"26","author":"Perry","journal-title":"J Med Internet Res"},{"issue":"12","key":"R16","doi-asserted-by":"publisher","first-page":"843","DOI":"10.1007\/s00520-024-09034-9","article-title":"Co-design of an electronic patient-reported outcome symptom monitoring system for immunotherapy toxicities","volume":"32","author":"Lai-Kwon","journal-title":"Support Care Cancer"},{"issue":"15_suppl","key":"R17","doi-asserted-by":"publisher","first-page":"e14120","DOI":"10.1200\/JCO.2020.38.15_suppl.e14120","article-title":"Design of eSyM: an ePRO-based symptom management tool fully integrated in the electronic health record (Epic) to foster patient\/clinician engagement, sustainability, and clinical impact","volume":"38","author":"Hassett","journal-title":"J Clin Oncol"},{"issue":"6","key":"R18","doi-asserted-by":"publisher","DOI":"10.1200\/CCI.21.00137","article-title":"eSyM: an electronic health record-integrated patient-reported outcomes-based cancer symptom management program used by six diverse health systems","volume":"6","author":"Hassett","journal-title":"JCO Clin Cancer Inform"},{"issue":"2","key":"R19","doi-asserted-by":"publisher","first-page":"325","DOI":"10.1007\/s11136-019-02329-z","article-title":"Key methodological considerations for usability testing of electronic patient-reported outcome (ePRO) systems","volume":"29","author":"Aiyegbusi","journal-title":"Qual Life Res"},{"key":"R20","unstructured":"ISO 9241-11:2018(en) ergonomics of human-system interaction \u2014 part 11: usability: definitions and concepts. ISO. 2018. URL: https:\/\/www.iso.org\/obp\/ui\/#iso:std:iso:9241:-11:ed-2:v1:en [Accessed 18-11-2025]"},{"key":"R21","author":"Macefield","journal-title":"J Usability Stud"},{"key":"R22","doi-asserted-by":"crossref","unstructured":"Nielsen J Landauer TK . A mathematical model of the finding of usability problems. Presented at: CHI \u201993: Proceedings of the INTERACT \u201993 and CHI \u201993 Conference on Human Factors in Computing Systems; Apr 24-29, 1993:206-213; Amsterdam, The Netherlands. 1993.URL: http:\/\/portal.acm.org\/citation.cfm?doid=169059 [doi: 10.1145\/169059.169166]","DOI":"10.1145\/169059.169166"},{"issue":"1","key":"R23","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1007\/s10209-006-0054-8","article-title":"A systematic approach to the development of research-based web design guidelines for older people","volume":"6","author":"Zaphiris","journal-title":"Univ Access Inf Soc"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1177\/0894439303259876","article-title":"E-government visual accessibility for older adult users","volume":"22","author":"Becker","journal-title":"Soc Sci Comput Rev"},{"issue":"4","key":"R25","doi-asserted-by":"publisher","first-page":"419","DOI":"10.1111\/j.1524-4733.2008.00470.x","article-title":"Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO Good Research Practices Task Force report","volume":"12","author":"Coons","journal-title":"Value Health"},{"issue":"1","key":"R26","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1177\/1525822X05279903","article-title":"How many interviews are enough?: An experiment with data saturation and variability","volume":"18","author":"Guest","journal-title":"Field methods"},{"key":"R27","unstructured":"Marcotte E . Response web design. A Book Apart. 2010. URL: https:\/\/alistapart.com\/article\/responsive-web-design [Accessed 07-10-2024]"},{"key":"R28","unstructured":"Schade A . Responsive web design (RWD) and user experience. Nielson Norman Group. 2014. URL: https:\/\/www.nngroup.com\/articles\/responsive-web-design-definition [Accessed 07-10-2024]"},{"issue":"2","key":"R29","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1177\/0733464816642582","article-title":"A new tool for assessing mobile device proficiency in older adults: the mobile device proficiency questionnaire","volume":"37","author":"Roque","journal-title":"J Appl Gerontol"},{"issue":"3","key":"R30","doi-asserted-by":"publisher","first-page":"404","DOI":"10.1093\/geront\/gnt117","article-title":"Computer proficiency questionnaire: assessing low and high computer proficient seniors","volume":"55","author":"Boot","journal-title":"Gerontologist"},{"key":"R31","unstructured":"Brooke J . SUS: a \u201cquick and dirty\u201d usability scale. In: Jordan PW McClelland IL , editors. Usability Evaluation in Industry. Taylor & Francis; 1996. ISBN: 0748403140"},{"issue":"5","key":"R32","doi-asserted-by":"publisher","first-page":"328","DOI":"10.17705\/1jais.00428","article-title":"Unified theory of acceptance and use of technology: a synthesis and the road ahead","volume":"17","author":"Venkatesh","journal-title":"J Assoc Inf Syst"},{"issue":"3","key":"R33","doi-asserted-by":"publisher","first-page":"425","DOI":"10.2307\/30036540","article-title":"User acceptance of information technology: toward a unified view1","volume":"27","author":"Venkatesh","journal-title":"MIS Q"},{"issue":"9","key":"R34","doi-asserted-by":"publisher","first-page":"1277","DOI":"10.1177\/1049732305276687","article-title":"Three approaches to qualitative content analysis","volume":"15","author":"Hsieh","journal-title":"Qual Health Res"},{"key":"R35","unstructured":"Nichols E . IEC 62366 explained: what you need to know about usability engineering. Greenlight Guru. 2021. URL: https:\/\/www.greenlight.guru\/blog\/iec-62366-usability-engineering [Accessed 13-01-2025]"},{"key":"R36","unstructured":"Applying human factors and usability engineering to medical devices: guidance for industry and food and drug administration staff. Center for Devices and Radiological Health. Feb 3, 2016. URL: https:\/\/www.fda.gov\/media\/80481\/download [Accessed 25-02-2026]"},{"key":"R37","unstructured":"Nielsen J . Usability Engineering. Morgan Kaufmann Publishers Inc; 1994. ISBN: 0125184069"},{"key":"R38","unstructured":"Norman D . The Design of Everyday Things: Revised and Expanded Edition. Basic Books; 2013. ISBN: 9780465050659"},{"issue":"16","key":"R39","doi-asserted-by":"publisher","first-page":"9329","DOI":"10.3390\/su13169329","article-title":"Interpreting usability factors predicting sustainable adoption of cloud-based e-learning environment during COVID-19 pandemic","volume":"13","author":"Dutta","journal-title":"Sustainability"},{"key":"R40","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2021.104450","article-title":"Examining the acceptance of an integrated electronic health records system: insights from a repeated cross-sectional design","volume":"150","author":"Luyten","journal-title":"Int J Med Inform"},{"issue":"8","key":"R41","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.26956","article-title":"Electronic health record usability, satisfaction, and burnout for family physicians","volume":"7","author":"Holmgren","journal-title":"JAMA Netw Open"},{"issue":"2","key":"R42","doi-asserted-by":"publisher","DOI":"10.2196\/31758","article-title":"Assessing the usability of a clinical decision support system: heuristic evaluation","volume":"9","author":"Cho","journal-title":"JMIR Hum Factors"},{"key":"R43","doi-asserted-by":"publisher","DOI":"10.2196\/45974","article-title":"Patients\u2019 experiences of a national patient portal and its usability: cross-sectional survey study","volume":"25","author":"Simola","journal-title":"J Med Internet Res"},{"issue":"2","key":"R44","doi-asserted-by":"publisher","DOI":"10.3390\/ijerph20021347","article-title":"The use of a Technology Acceptance Model (TAM) to predict patients\u2019 usage of a personal health record system: the role of security, privacy, and usability","volume":"20","author":"Alsyouf","journal-title":"Int J Environ Res Public Health"},{"issue":"6","key":"R45","doi-asserted-by":"publisher","first-page":"1519","DOI":"10.1038\/s41591-024-02909-8","article-title":"A framework for implementing patient-reported outcomes in clinical care: the PROTEUS-practice guide","volume":"30","author":"Crossnohere","journal-title":"Nat Med"},{"issue":"43","key":"R46","doi-asserted-by":"publisher","DOI":"10.1200\/EDBK_390678","article-title":"Patient-reported outcomes, digital health, and the quest to improve health equity","volume":"43","author":"Pritchett","journal-title":"Am Soc Clin Oncol Educ Book"},{"issue":"3","key":"R47","doi-asserted-by":"publisher","first-page":"409","DOI":"10.1200\/OP.23.00493","article-title":"Implementing a patient-reported outcome dashboard in oncology telemedicine encounters: clinician and patient adoption and acceptability","volume":"20","author":"Mohindra","journal-title":"JCO Oncol Pract"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e79694","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:12:53Z","timestamp":1782843173000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e79694"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,18]]},"references-count":47,"URL":"https:\/\/doi.org\/10.2196\/79694","relation":{},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,18]]},"article-number":"v10i39e79694"}}