{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T02:17:41Z","timestamp":1790821061243,"version":"4.1.0"},"reference-count":39,"publisher":"MDPI AG","issue":"14","license":[{"start":{"date-parts":[[2022,7,8]],"date-time":"2022-07-08T00:00:00Z","timestamp":1657238400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Sigrid Jus\u00e9lius Foundation","award":["TYH2022219"],"award-info":[{"award-number":["TYH2022219"]}]},{"name":"The Finnish Cancer Society","award":["TYH2022219"],"award-info":[{"award-number":["TYH2022219"]}]},{"name":"Finska L\u00e4kares\u00e4llskapet","award":["TYH2022219"],"award-info":[{"award-number":["TYH2022219"]}]},{"name":"The Helsinki University Hospital Research Fund","award":["TYH2022219"],"award-info":[{"award-number":["TYH2022219"]}]},{"name":"The Turku University Hospital Fund","award":["TYH2022219"],"award-info":[{"award-number":["TYH2022219"]}]},{"name":"University of Helsinki","award":["TYH2022219"],"award-info":[{"award-number":["TYH2022219"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IJERPH"],"abstract":"<jats:p>Background: Machine learning models have been reported to assist in the proper management of cancer through accurate prognostication. Integrating such models as a web-based prognostic tool or calculator may help to improve cancer care and assist clinicians in making oral cancer management-related decisions. However, none of these models have been recommended in daily practices of oral cancer due to concerns related to machine learning methodologies and clinical implementation challenges. An instance of the concerns inherent to the science of machine learning is explainability. Objectives: This study measures the usability and explainability of a machine learning-based web prognostic tool that was designed for prediction of oral tongue cancer. We used the System Usability Scale (SUS) and System Causability Scale (SCS) to evaluate the explainability of the prognostic tool. In addition, we propose a framework for the evaluation of post hoc explainability of web-based prognostic tools. Methods: A SUS- and SCS-based questionnaire was administered amongst pathologists, radiologists, cancer and machine learning researchers and surgeons (n = 11) to evaluate the quality of explanations offered by the machine learning-based web prognostic tool to address the concern of explainability and usability of these models for cancer management. The examined web-based tool was developed by our group and is freely available online. Results: In terms of the usability of the web-based tool using the SUS, 81.9% (45.5% strongly agreed; 36.4% agreed) agreed that neither the support of a technical assistant nor a need to learn many things were required to use the web-based tool. Furthermore, 81.8% agreed that the evaluated web-based tool was not cumbersome to use (usability). The average score for the SCS (explainability) was 0.74. A total of 91.0% of the participants strongly agreed that the web-based tool can assist in clinical decision-making. These scores indicated that the examined web-based tool offers a significant level of usability and explanations about the outcome of interest. Conclusions: Integrating the trained and internally and externally validated model as a web-based tool or calculator is poised to offer an effective and easy approach towards the usage and acceptance of these models in the future daily practice. This approach has received significant attention in recent years. Thus, it is important that the usability and explainability of these models are measured to achieve such touted benefits. A usable and well-explained web-based tool further brings the use of these web-based tools closer to everyday clinical practices. Thus, the concept of more personalized and precision oncology can be achieved.<\/jats:p>","DOI":"10.3390\/ijerph19148366","type":"journal-article","created":{"date-parts":[[2022,7,8]],"date-time":"2022-07-08T11:37:08Z","timestamp":1657280228000},"page":"8366","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["Measuring the Usability and Quality of Explanations of a Machine Learning Web-Based Tool for Oral Tongue Cancer Prognostication"],"prefix":"10.3390","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7655-5924","authenticated-orcid":false,"given":"Rasheed Omobolaji","family":"Alabi","sequence":"first","affiliation":[{"name":"Research Program in Systems Oncology, Faculty of Medicine, University of Helsinki, 00100 Helsinki, Finland"},{"name":"Department of Industrial Digitalization, School of Technology and Innovations, University of Vaasa, 65200 Vaasa, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alhadi","family":"Almangush","sequence":"additional","affiliation":[{"name":"Research Program in Systems Oncology, Faculty of Medicine, University of Helsinki, 00100 Helsinki, Finland"},{"name":"Department of Pathology, University of Helsinki, Haartmaninkatu 3 (P.O. Box 21), FIN-00014 Helsinki, Finland"},{"name":"Institute of Biomedicine, University of Turku, Pathology, 20500 Turku, Finland"},{"name":"Faculty of Dentistry, Misurata University, Misurata 2478, Libya"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohammed","family":"Elmusrati","sequence":"additional","affiliation":[{"name":"Department of Industrial Digitalization, School of Technology and Innovations, University of Vaasa, 65200 Vaasa, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ilmo","family":"Leivo","sequence":"additional","affiliation":[{"name":"Institute of Biomedicine, University of Turku, Pathology, 20500 Turku, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0451-2404","authenticated-orcid":false,"given":"Antti","family":"M\u00e4kitie","sequence":"additional","affiliation":[{"name":"Research Program in Systems Oncology, Faculty of Medicine, University of Helsinki, 00100 Helsinki, Finland"},{"name":"Department of Otorhinolaryngology\u2014Head and Neck Surgery, University of Helsinki, Helsinki University Hospital, 00029 HUS Helsinki, Finland"},{"name":"Department of Clinical Sciences, Intervention and Technology, Division of Ear, Nose and Throat Diseases, Karolinska Institute, Karolinska University Hospital, 17177 Stockholm, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,7,8]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"489","DOI":"10.1007\/s00428-019-02642-5","article-title":"Machine Learning Application for Prediction of Locoregional Recurrences in Early Oral Tongue Cancer: A Web-Based Prognostic Tool","volume":"475","author":"Alabi","year":"2019","journal-title":"Virchows Arch."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"104068","DOI":"10.1016\/j.ijmedinf.2019.104068","article-title":"Comparison of Supervised Machine Learning Classification Techniques in Prediction of Locoregional Recurrences in Early Oral Tongue Cancer","volume":"136","author":"Alabi","year":"2019","journal-title":"Int. J. Med. Inform."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"100558","DOI":"10.1016\/j.eclinm.2020.100558","article-title":"A Deep Learning Algorithm for Detection of Oral Cavity Squamous Cell Carcinoma from Photographic Images: A Retrospective Study","volume":"27","author":"Fu","year":"2020","journal-title":"EClinicalMedicine"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1111\/odi.13825","article-title":"A Novel Lightweight Deep Convolutional Neural Network for Early Detection of Oral Cancer","volume":"28","author":"Jubair","year":"2021","journal-title":"Oral Diseases"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"849","DOI":"10.1111\/jop.13042","article-title":"The Use of Artificial Intelligence, Machine Learning and Deep Learning in Oncologic Histopathology","volume":"49","author":"Sultan","year":"2020","journal-title":"J. Oral Pathol. Med."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1038\/nature21056","article-title":"Dermatologist-Level Classification of Skin Cancer with Deep Neural Networks","volume":"542","author":"Esteva","year":"2017","journal-title":"Nature"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.media.2017.07.005","article-title":"A Survey on Deep Learning in Medical Image Analysis","volume":"42","author":"Litjens","year":"2017","journal-title":"Med. Image Anal."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"5110","DOI":"10.1038\/s41598-017-05300-5","article-title":"Location Sensitive Deep Convolutional Neural Networks for Segmentation of White Matter Hyperintensities","volume":"7","author":"Ghafoorian","year":"2017","journal-title":"Sci. Rep."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.media.2017.06.015","article-title":"Validation, Comparison, and Combination of Algorithms for Automatic Detection of Pulmonary Nodules in Computed Tomography Images: The LUNA16 Challenge","volume":"42","author":"Setio","year":"2017","journal-title":"Med. Image Anal."},{"key":"ref_10","first-page":"1","article-title":"Machine Learning for Prognosis of Oral Cancer: What Are the Ethical Challenges?","volume":"2737","author":"Alabi","year":"2020","journal-title":"CEUR-Workshop Proc."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"102060","DOI":"10.1016\/j.artmed.2021.102060","article-title":"Machine Learning in Oral Squamous Cell Carcinoma: Current Status, Clinical Concerns and Prospects for Future\u2014A Systematic Review","volume":"115","author":"Alabi","year":"2021","journal-title":"Artif. Intell. Med."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1007\/s13218-020-00636-z","article-title":"Measuring the Quality of Explanations: The System Causability Scale (SCS): Comparing Human and Machine Explanations","volume":"34","author":"Holzinger","year":"2020","journal-title":"KI\u2014K\u00fcnstliche Intell."},{"key":"ref_13","unstructured":"Carrington, A.M. (2018). Kernel Methods and Measures for Classifcation with Transparency, Interpretability and Accuracy in Health Care. [Ph.D. Theis, University of Waterloo]."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"103655","DOI":"10.1016\/j.jbi.2020.103655","article-title":"The Role of Explainability in Creating Trustworthy Artificial Intelligence for Health Care: A Comprehensive Survey of the Terminology, Design Choices, and Evaluation Strategies","volume":"113","author":"Markus","year":"2021","journal-title":"J. Biomed. Inform."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1007\/978-3-319-99740-7_23","article-title":"Evaluating Explanations by Cognitive Value","volume":"11015","author":"Holzinger","year":"2018","journal-title":"Machine Learning and Knowledge Extraction"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Lou, Y., Caruana, R., and Gehrke, J. (2012, January 12\u201316). Intelligible Models for Classification and Regression. Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining\u2014KDD \u201912, Beijing, China.","DOI":"10.1145\/2339530.2339556"},{"key":"ref_17","unstructured":"Lundberg, S., and Lee, S.-I. (2017). A Unified Approach to Interpreting Model Predictions. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1008","DOI":"10.3109\/00016489.2011.575796","article-title":"Increased Incidence of Oropharyngeal Cancer and P16 Expression","volume":"131","author":"Lundberg","year":"2011","journal-title":"Acta Oto-Laryngol."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016). \u201cWhy Should I Trust You?\u201d: Explaining the Predictions of Any Classifier. arXiv.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_20","unstructured":"Narayanan, M., Chen, E., He, J., Kim, B., Gershman, S., and Doshi-Velez, F. (2018). How Do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"105452","DOI":"10.1016\/j.oraloncology.2021.105452","article-title":"Machine Learning in Head and Neck Cancer: Importance of a Web-Based Prognostic Tool for Improved Decision Making","volume":"124","author":"Almangush","year":"2021","journal-title":"Oral Oncol."},{"key":"ref_22","unstructured":"(2022, May 18). Usability.Gov System Usability Scale (SUS), Available online: https:\/\/www.usability.gov\/how-to-and-tools\/methods\/system-usability-scale.html."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Jordan, P.W., Thomas, B., Weerdmeester, B.A., and McClelland, I.L. (1996). SUS: A Quick and Dirty Usability Scale. Usability Evaluation in Industry, Taylor and Francis.","DOI":"10.1201\/9781498710411"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.artint.2018.07.007","article-title":"Explanation in Artificial Intelligence: Insights from the Social Sciences","volume":"267","author":"Miller","year":"2019","journal-title":"Artif. Intell."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1007\/978-3-030-29726-8_3","article-title":"New Frontiers in Explainable AI: Understanding the GI to Interpret the GO","volume":"11713","author":"Holzinger","year":"2019","journal-title":"Machine Learning and Knowledge Extraction"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Kulesza, T., Stumpf, S., Burnett, M., Yang, S., Kwan, I., and Wong, W.-K. (2013, January 15\u201319). Too Much, Too Little, or Just Right? Ways Explanations Impact End Users\u2019 Mental Models. Proceedings of the 2013 IEEE Symposium on Visual Languages and Human Centric Computing; IEEE, San Jose, CA, USA.","DOI":"10.1109\/VLHCC.2013.6645235"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Escalante, H.J., Escalera, S., Guyon, I., Bar\u00f3, X., G\u00fc\u00e7l\u00fct\u00fcrk, Y., G\u00fc\u00e7l\u00fc, U., and van Gerven, M. (2018). Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges. Explainable and Interpretable Models in Computer Vision and Machine Learning, Springer International Publishing.","DOI":"10.1007\/978-3-319-98131-4"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1145\/3236386.3241340","article-title":"The Mythos of Model Interpretability: In Machine Learning, the Concept of Interpretability Is Both Important and Slippery","volume":"16","author":"Lipton","year":"2018","journal-title":"Queue"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., and Kagal, L. (2018, January 1\u20133). Explaining Explanations: An Overview of Interpretability of Machine Learning. Proceedings of the 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA); IEEE, Turin, Italy.","DOI":"10.1109\/DSAA.2018.00018"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"106685","DOI":"10.1016\/j.knosys.2020.106685","article-title":"Explainability in Deep Reinforcement Learning","volume":"214","author":"Heuillet","year":"2021","journal-title":"Knowl.-Based Syst."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"103473","DOI":"10.1016\/j.artint.2021.103473","article-title":"What Do We Want from Explainable Artificial Intelligence (XAI)?\u2014A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research","volume":"296","author":"Langer","year":"2021","journal-title":"Artif. Intell."},{"key":"ref_32","unstructured":"Doshi-Velez, F., and Kim, B. (2017). Towards A Rigorous Science of Interpretable Machine Learning. arXiv."},{"key":"ref_33","unstructured":"Hoffman, R.R., Mueller, S.T., Klein, G., and Litman, J. (2019). Metrics for Explainable AI: Challenges and Prospects. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3236009","article-title":"A Survey of Methods for Explaining Black Box Models","volume":"51","author":"Guidotti","year":"2019","journal-title":"ACM Comput. Surv."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1177\/0194599819889686","article-title":"Ethical Considerations in the Advent of Artificial Intelligence in Otolaryngology","volume":"162","author":"Arambula","year":"2020","journal-title":"Otolaryngol. Head Neck Surg."},{"key":"ref_36","unstructured":"European Commission (2019). High-Level Expert Group on Artificial Intelligence. Ethics Guidelines for Tustrworthy AI, European Commission."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"277","DOI":"10.1016\/j.jocn.2019.03.001","article-title":"Medical Ethics Considerations on Artificial Intelligence","volume":"64","author":"Keskinbora","year":"2019","journal-title":"J. Clin. Neurosci."},{"key":"ref_38","unstructured":"Select Committee on Artificial Intelligence (2019). The National Artificial Intelligence Research and Development Strategic Plan: 2019 Update."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1217","DOI":"10.1111\/j.1365-2929.2004.02012.x","article-title":"Likert Scales: How to (Ab)Use Them","volume":"38","author":"Jamieson","year":"2004","journal-title":"Med. Educ."}],"container-title":["International Journal of Environmental Research and Public Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1660-4601\/19\/14\/8366\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:46:48Z","timestamp":1760140008000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1660-4601\/19\/14\/8366"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,8]]},"references-count":39,"journal-issue":{"issue":"14","published-online":{"date-parts":[[2022,7]]}},"alternative-id":["ijerph19148366"],"URL":"https:\/\/doi.org\/10.3390\/ijerph19148366","relation":{},"ISSN":["1660-4601"],"issn-type":[{"value":"1660-4601","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,8]]}}}