{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,18]],"date-time":"2026-04-18T16:30:51Z","timestamp":1776529851715,"version":"3.51.2"},"reference-count":46,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2021,9,6]],"date-time":"2021-09-06T00:00:00Z","timestamp":1630886400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Cancers"],"abstract":"<jats:p>Melanoma is the most invasive skin cancer with the highest risk of death. While it is a serious skin cancer, it is highly curable if detected early. Melanoma diagnosis is difficult, even for experienced dermatologists, due to the wide range of morphologies in skin lesions. Given the rapid development of deep learning algorithms for melanoma diagnosis, it is crucial to validate and benchmark these models, which is the main challenge of this work. This research presents a new benchmarking and selection approach based on the multi-criteria analysis method (MCDM), which integrates entropy and the preference ranking organization method for enrichment of evaluations (PROMETHEE) methods. The experimental study is carried out in four phases. Firstly, 19 convolution neural networks (CNNs) are trained and evaluated on a public dataset of 991 dermoscopic images. Secondly, to obtain the decision matrix, 10 criteria, including accuracy, classification error, precision, sensitivity, specificity, F1-score, false-positive rate, false-negative rate, Matthews correlation coefficient (MCC), and the number of parameters are established. Third, entropy and PROMETHEE methods are integrated to determine the weights of criteria and rank the models. Fourth, the proposed benchmarking framework is validated using the VIKOR method. The obtained results reveal that the ResNet101 model is selected as the optimal diagnosis model for melanoma in our case study data. Thus, the presented benchmarking framework is proven to be useful at exposing the optimal melanoma diagnosis model targeting to ease the selection process of the proper convolutional neural network architecture.<\/jats:p>","DOI":"10.3390\/cancers13174494","type":"journal-article","created":{"date-parts":[[2021,9,6]],"date-time":"2021-09-06T10:59:12Z","timestamp":1630925952000},"page":"4494","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["A Comprehensive Evaluation and Benchmarking of Convolutional Neural Networks for Melanoma Diagnosis"],"prefix":"10.3390","volume":"13","author":[{"given":"Saeed","family":"Alzahrani","sequence":"first","affiliation":[{"name":"Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool L69 3GJ, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Baidaa","family":"Al-Bander","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, University of Diyala, Baqubah 32010, Iraq"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Waleed","family":"Al-Nuaimy","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool L69 3GJ, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,9,6]]},"reference":[{"key":"ref_1","unstructured":"Ries, L.A., Harkins, D., Krapcho, M., Mariotto, A., Miller, B., Feuer, E.J., Clegg, L.X., Eisner, M., Horner, M.J., and Howlader, N. (2021, September 05). SEER Cancer Statistics Review 1975\u20132003. National Cancer Institute\u00a0, Available online: https:\/\/seer.cancer.gov\/archive\/csr\/1975_2003\/."},{"key":"ref_2","unstructured":"(2021, May 01). Skin Cancer Statistics. Available online: https:\/\/www.nhs.uk\/conditions\/non-melanoma-skin-cancer\/."},{"key":"ref_3","unstructured":"(2021, May 01). Non-Melanoma Skin Cancer Statistics. Available online: https:\/\/www.cancerresearchuk.org\/health-professional\/cancer-statistics\/statistics-by-cancer-type\/non-melanoma-skin-cancer."},{"key":"ref_4","unstructured":"(2021, May 01). Melanoma Skin Cancer Statistics. Available online: https:\/\/www.cancerresearchuk.org\/health-professional\/cancer-statistics\/statistics-by-cancer-type\/melanoma-skin-cancer."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1068","DOI":"10.1016\/j.jaad.2010.03.039","article-title":"Diagnostic accuracy of dermatoscopy for melanocytic and nonmelanocytic pigmented lesions","volume":"64","author":"Rosendahl","year":"2011","journal-title":"J. Am. Acad. Dermatol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"27","DOI":"10.4103\/ijdpdd.ijdpdd_13_17","article-title":"Dermatoscopy: Physics and principles","volume":"4","author":"Nirmal","year":"2017","journal-title":"Indian J. Dermatopathol. Diagn. Dermatol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"828","DOI":"10.1001\/archderm.144.6.828","article-title":"Polarized and nonpolarized dermoscopy: The explanation for the observed differences","volume":"144","author":"Pan","year":"2008","journal-title":"Arch. Dermatol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"51","DOI":"10.7748\/ns.2017.e11030","article-title":"Using the ABCDE approach to assess the deteriorating patient","volume":"32","author":"Smith","year":"2017","journal-title":"Nurs. Stand."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"7","DOI":"10.3322\/caac.21654","article-title":"Cancer Statistics","volume":"71","author":"Siegel","year":"2021","journal-title":"CA Cancer J. Clin."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"472","DOI":"10.1001\/jamadermatol.2016.6032","article-title":"Analysis of trends in geographic distribution of US dermatology workforce density","volume":"153","author":"Glazer","year":"2017","journal-title":"JAMA Dermatol."},{"key":"ref_11","unstructured":"(2021, August 27). Skin Cancer Check-Ups a Long Time Coming as Australia Faces Huge Shortage of Dermatologists. Available online: https:\/\/www.abc.net.au\/news\/2021-06-14\/gps-to-help-ease-growing-skin-specialist-waiting-times\/100211834."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"509","DOI":"10.7861\/clinmedicine.15-6-509","article-title":"Dermatology: A specialty in crisis","volume":"15","author":"Eedy","year":"2015","journal-title":"Clin. Med."},{"key":"ref_13","unstructured":"Royal College of Physicians Dermatology (2021, August 27). London: RCP. Available online: www.rcplondon.ac.uk\/sites\/default\/files\/dermatology.pdf."},{"key":"ref_14","unstructured":"British Association of Dermatologists Clinical Services (2021, August 27). London: BAD. Available online: www.bad.org.uk\/healthcare-professionals\/clinical-services\/."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Alzahrani, S., Al-Nuaimy, W., and Al-Bander, B. (2019, January 28\u201331). Seven-point checklist with convolutional neural networks for melanoma diagnosis. Proceedings of the 2019 8th European Workshop on Visual Information Processing (EUVIP), Rome, Italy.","DOI":"10.1109\/EUVIP47703.2019.8946208"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1586\/edm.11.79","article-title":"Teledermatology: State-of-the-art and future perspectives","volume":"7","author":"Nami","year":"2012","journal-title":"Expert Rev. Dermatol."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Codella, N.C., Gutman, D., Celebi, M.E., Helba, B., Marchetti, M.A., Dusza, S.W., Kalloo, A., Liopyris, K., Mishra, N., and Kittler, H. (2017, January 18\u201321). Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). Proceedings of the 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2017), Melbourne, Australia.","DOI":"10.1109\/ISBI.2018.8363547"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"110575","DOI":"10.1109\/ACCESS.2020.3001507","article-title":"Malignant melanoma classification using deep learning: Datasets, performance measurements, challenges and opportunities","volume":"8","author":"Naeem","year":"2020","journal-title":"IEEE Access"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"101858","DOI":"10.1016\/j.media.2020.101858","article-title":"Convolutional neural networks for the automatic diagnosis of melanoma: An extensive experimental study","volume":"67","author":"Reyes","year":"2021","journal-title":"Med. Image Anal."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Keeney, R.L., Raiffa, H., and Meyer, R.F. (1993). Decisions with Multiple Objectives: Preferences and Value Trade-Offs, Cambridge University Press.","DOI":"10.1017\/CBO9781139174084"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1478-7547-4-14","article-title":"Priority setting of health interventions: The need for multi-criteria decision analysis","volume":"4","author":"Baltussen","year":"2006","journal-title":"Cost Eff. Resour. Alloc."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jval.2015.12.003","article-title":"Multiple criteria decision analysis for health care decision making\u2014An introduction: Report 1 of the ISPOR MCDA Emerging Good Practices Task Force","volume":"19","author":"Thokala","year":"2016","journal-title":"Value Health"},{"key":"ref_24","first-page":"1097","article-title":"Imagenet classification with deep convolutional neural networks","volume":"25","author":"Krizhevsky","year":"2012","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_26","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7\u201312). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27\u201330). Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.308"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. (2017, January 4\u20139). Inception-v4, inception-resnet and the impact of residual connections on learning. Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.","DOI":"10.1609\/aaai.v31i1.11231"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 21\u201326). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21\u201326). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.243"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Chollet, F. (2017, January 21\u201326). Xception: Deep learning with depthwise separable convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.195"},{"key":"ref_33","unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Zoph, B., Vasudevan, V., Shlens, J., and Le, Q.V. (2018, January 18\u201322). Learning transferable architectures for scalable image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00907"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Zhang, X., Zhou, X., Lin, M., and Sun, J. (2018, January 18\u201322). Shufflenet: An extremely efficient convolutional neural network for mobile devices. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00716"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Redmon, J., and Farhadi, A. (2017, January 21\u201326). YOLO9000: Better, faster, stronger. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.690"},{"key":"ref_37","unstructured":"Redmon, J., and Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv."},{"key":"ref_38","unstructured":"Tan, M., and Le, Q. (2019, January 9\u201315). Efficientnet: Rethinking model scaling for convolutional neural networks. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Hu, J., Shen, L., and Sun, G. (2018, January 18\u201322). Squeeze-and-excitation networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00745"},{"key":"ref_40","unstructured":"Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., and Keutzer, K. (2016). SqueezeNet: AlexNet-level accuracy with 50\u00d7 fewer parameters and <0.5 MB model size. arXiv."},{"key":"ref_41","unstructured":"Jahan, A., Edwards, K.L., and Bahraminasab, M. (2016). Multi-Criteria Decision Analysis for Supporting the Selection of Engineering Materials in Product Design, Butterworth-Heinemann."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"216","DOI":"10.1016\/j.ejor.2015.05.075","article-title":"Multi-criteria decision analysis for supporting the selection of medical devices under uncertainty","volume":"247","author":"Ivlev","year":"2015","journal-title":"Eur. J. Oper. Res."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1016\/j.ejor.2009.01.021","article-title":"PROMETHEE: A comprehensive literature review on methodologies and applications","volume":"200","author":"Behzadian","year":"2010","journal-title":"Eur. J. Oper. Res."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/j.jairtraman.2010.03.004","article-title":"A modified VIKOR multiple-criteria decision method for improving domestic airlines service quality","volume":"17","author":"Liou","year":"2011","journal-title":"J. Air Transp. Manag."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1093\/pan\/mpr025","article-title":"Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies","volume":"20","author":"Hainmueller","year":"2012","journal-title":"Political Anal."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Li, F.-F. (2009, January 20\u201325). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"}],"container-title":["Cancers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-6694\/13\/17\/4494\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:57:26Z","timestamp":1760165846000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-6694\/13\/17\/4494"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,6]]},"references-count":46,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2021,9]]}},"alternative-id":["cancers13174494"],"URL":"https:\/\/doi.org\/10.3390\/cancers13174494","relation":{},"ISSN":["2072-6694"],"issn-type":[{"value":"2072-6694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,6]]}}}