{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,13]],"date-time":"2026-03-13T06:27:02Z","timestamp":1773383222089,"version":"3.50.1"},"reference-count":49,"publisher":"MDPI AG","issue":"21","license":[{"start":{"date-parts":[[2022,11,2]],"date-time":"2022-11-02T00:00:00Z","timestamp":1667347200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Health Research Foundation of Central Denmark Region","award":["A3139"],"award-info":[{"award-number":["A3139"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IJERPH"],"abstract":"<jats:p>Deep learning for the analysis of H&amp;E stains requires a large annotated training set. This may form a labor-intensive task involving highly skilled pathologists. We aimed to optimize and evaluate computer-assisted annotation based on digital dual stains of the same tissue section. H&amp;E stains of primary and metastatic melanoma (N = 77) were digitized, re-stained with SOX10, and re-scanned. Because images were aligned, annotations of SOX10 image analysis were directly transferred to H&amp;E stains of the training set. Based on 1,221,367 annotated nuclei, a convolutional neural network for calculating tumor burden (CNNTB) was developed. For primary melanomas, precision of annotation was 100% (95%CI, 99% to 100%) for tumor cells and 99% (95%CI, 98% to 100%) for normal cells. Due to low or missing tumor-cell SOX10 positivity, precision for normal cells was markedly reduced in lymph-node and organ metastases compared with primary melanomas (p &lt; 0.001). Compared with stereological counts within skin lesions, mean difference in tumor burden was 6% (95%CI, \u22121% to 13%, p = 0.10) for CNNTB and 16% (95%CI, 4% to 28%, p = 0.02) for pathologists. Conclusively, the technique produced a large annotated H&amp;E training set with high quality within a reasonable timeframe for primary melanomas and subcutaneous metastases. For these lesion types, the training set generated a high-performing CNNTB, which was superior to the routine assessments of pathologists.<\/jats:p>","DOI":"10.3390\/ijerph192114327","type":"journal-article","created":{"date-parts":[[2022,11,3]],"date-time":"2022-11-03T03:11:21Z","timestamp":1667445081000},"page":"14327","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Computer-Assisted Annotation of Digital H&amp;E\/SOX10 Dual Stains Generates High-Performing Convolutional Neural Network for Calculating Tumor Burden in H&amp;E-Stained Cutaneous Melanoma"],"prefix":"10.3390","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7859-6321","authenticated-orcid":false,"given":"Patricia Switten","family":"Nielsen","sequence":"first","affiliation":[{"name":"Department of Pathology, Aarhus University Hospital, Palle Juul-Jensens Boulevard 35, DK-8200 Aarhus, Denmark"},{"name":"Department of Clinical Medicine, Aarhus University, Palle Juul-Jensens Boulevard 82, DK-8200 Aarhus, Denmark"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9452-2920","authenticated-orcid":false,"given":"Jeanette Baehr","family":"Georgsen","sequence":"additional","affiliation":[{"name":"Department of Pathology, Aarhus University Hospital, Palle Juul-Jensens Boulevard 35, DK-8200 Aarhus, Denmark"},{"name":"Department of Clinical Medicine, Aarhus University, Palle Juul-Jensens Boulevard 82, DK-8200 Aarhus, Denmark"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0149-583X","authenticated-orcid":false,"given":"Mads Sloth","family":"Vinding","sequence":"additional","affiliation":[{"name":"Department of Clinical Medicine, Aarhus University, Palle Juul-Jensens Boulevard 82, DK-8200 Aarhus, Denmark"},{"name":"Center of Functionally Integrative Neuroscience, Aarhus University Hospital, Palle Juul-Jensens Boulevard 99, DK-8200 Aarhus, Denmark"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lasse Riis","family":"\u00d8stergaard","sequence":"additional","affiliation":[{"name":"Department of Health Science and Technology, Aalborg University, Fredrik Bajers Vej 7E, DK-9220 Aalborg, Denmark"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Torben","family":"Steiniche","sequence":"additional","affiliation":[{"name":"Department of Pathology, Aarhus University Hospital, Palle Juul-Jensens Boulevard 35, DK-8200 Aarhus, Denmark"},{"name":"Department of Clinical Medicine, Aarhus University, Palle Juul-Jensens Boulevard 82, DK-8200 Aarhus, Denmark"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,11,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"412","DOI":"10.1038\/s41374-020-00514-0","article-title":"Artificial Intelligence and Computational Pathology","volume":"101","author":"Cui","year":"2021","journal-title":"Lab. Investig."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1111\/joim.13030","article-title":"Artificial Intelligence as the Next Step Towards Precision Pathology","volume":"288","author":"Acs","year":"2020","journal-title":"J. Intern. Med."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"627","DOI":"10.5858\/arpa.2011-0360-OA","article-title":"Automated Quantification of MART1-Verified Ki67 Indices by Digital Image Analysis in Melanocytic Lesions","volume":"136","author":"Nielsen","year":"2012","journal-title":"Arch. Pathol. Lab. Med."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"R89","DOI":"10.1186\/bcr2187","article-title":"Novel Image Analysis Approach for Quantifying Expression of Nuclear Proteins Assessed by Immunohistochemistry: Application to Measurement of Oestrogen and Progesterone Receptor Levels in Breast Cancer","volume":"10","author":"Rexhepaj","year":"2008","journal-title":"Breast Cancer Res."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"e195","DOI":"10.1016\/S2589-7500(20)30292-2","article-title":"Approval of Artificial Intelligence and Machine Learning-Based Medical Devices in the USA and Europe (2015\u201320): A Comparative Analysis","volume":"3","author":"Muehlematter","year":"2021","journal-title":"Lancet Digit. Health"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"349","DOI":"10.1111\/j.1600-0560.2007.00970.x","article-title":"Discordance in the Histopathologic Diagnosis of Difficult Melanocytic Neoplasms in the Clinical Setting","volume":"35","author":"Lodha","year":"2008","journal-title":"J. Cutan. Pathol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1186\/1746-1596-9-121","article-title":"Inter-Observer Variability between General Pathologists and a Specialist in Breast Pathology in the Diagnosis of Lobular Neoplasia, Columnar Cell Lesions, Atypical Ductal Hyperplasia and Ductal Carcinoma in Situ of the Breast","volume":"9","author":"Gomes","year":"2014","journal-title":"Diagn. Pathol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1093\/annonc\/mdp273","article-title":"The Impact of Inter-Observer Variation in Pathological Assessment of Node-Negative Breast Cancer on Clinical Risk Assessment and Patient Selection for Adjuvant Systemic Treatment","volume":"21","author":"Nuyten","year":"2010","journal-title":"Ann. Oncol."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"568","DOI":"10.1097\/PAI.0b013e3182a84b99","article-title":"Immunohistochemical Ki-67\/KL1 Double Stains Increase Accuracy of Ki-67 Indices in Breast Cancer and Simplify Automated Image Analysis","volume":"22","author":"Nielsen","year":"2014","journal-title":"Appl. Immunohistochem. Mol. Morphol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1038\/modpathol.2013.134","article-title":"The Estimation of Tumor Cell Percentage for Molecular Testing by Pathologists Is Not Accurate","volume":"27","author":"Smits","year":"2014","journal-title":"Mod. Pathol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1007\/s00428-016-2042-6","article-title":"Adequately Defining Tumor Cell Proportion in Tissue Samples for Molecular Testing Improves Interobserver Reproducibility of Its Assessment","volume":"470","author":"Lhermitte","year":"2017","journal-title":"Virchows Arch."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1038\/modpathol.2017.104","article-title":"Molecular Testing for BRAF Mutations to Inform Melanoma Treatment Decisions: A Move Toward Precision Medicine","volume":"31","author":"Cheng","year":"2018","journal-title":"Mod. Pathol."},{"key":"ref_13","unstructured":"Georgsen, J.B. (2021). Quantification of Tumor Burden by Artificial Intelligence in Tissue Sections of Melanoma. [Master\u2019s Thesis, Aalborg University]."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1090","DOI":"10.1038\/modpathol.2011.60","article-title":"Metastatic Colorectal Cancer KRAS Genotyping in Routine Practice: Results and Pitfalls","volume":"24","author":"Lamy","year":"2011","journal-title":"Mod. Pathol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1016\/j.ejca.2019.06.012","article-title":"Deep Learning Outperformed 11 Pathologists in the Classification of Histopathological Melanoma Images","volume":"118","author":"Hekler","year":"2019","journal-title":"Eur. J. Cancer"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"2199","DOI":"10.1001\/jama.2017.14585","article-title":"Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer","volume":"318","author":"Bejnordi","year":"2017","journal-title":"JAMA"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1301","DOI":"10.1038\/s41591-019-0508-1","article-title":"Clinical-Grade Computational Pathology Using Weakly Supervised Deep Learning on Whole Slide Images","volume":"25","author":"Campanella","year":"2019","journal-title":"Nat. Med."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"3655","DOI":"10.1109\/TMI.2020.3002244","article-title":"Weakly Supervised Deep Nuclei Segmentation Using Partial Points Annotation in Histopathology Images","volume":"39","author":"Qu","year":"2020","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1046\/j.1365-2559.2000.00894.x","article-title":"Morphological and Immunophenotypic Variations in Malignant Melanoma: Variations in Malignant Melanoma","volume":"36","author":"Banerjee","year":"2000","journal-title":"Histopathology"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"864","DOI":"10.1038\/s41598-018-37257-4","article-title":"Epithelium Segmentation Using Deep Learning in H&E-Stained Prostate Specimens with Immunohistochemistry as Reference Standard","volume":"9","author":"Bulten","year":"2019","journal-title":"Sci. Rep."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1638","DOI":"10.1038\/s41379-020-0526-z","article-title":"A Machine Learning Algorithm for Simulating Immunohistochemistry: Development of SOX10 Virtual IHC and Evaluation on Primarily Melanocytic Neoplasms","volume":"33","author":"Jackson","year":"2020","journal-title":"Mod. Pathol."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2126","DOI":"10.1109\/TMI.2018.2820199","article-title":"Whole-Slide Mitosis Detection in H&E Breast Histology Using PHH3 as a Reference to Train Distilled Stain-Invariant Convolutional Networks","volume":"37","author":"Tellez","year":"2018","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"422","DOI":"10.1109\/JBHI.2020.2994970","article-title":"Antibody Supervised Training of a Deep Learning Based Algorithm for Leukocyte Segmentation in Papillary Thyroid Carcinoma","volume":"25","author":"Stenman","year":"2021","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_24","unstructured":"Steiniche, T., Georgsen, J.B., Lauridsen, K.L., and Nielsen, P.S. (June, January 29). Multiplexing: Next-Generation Immunohistochemistry. Proceedings of the 14th European Congress on Digital Pathology, Helsinki, Finland."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"845","DOI":"10.1309\/AJCPW56PHGLFTKZC","article-title":"Prognostic Stratification of Ulcerated Melanoma","volume":"142","author":"Schmidt","year":"2014","journal-title":"Am. J. Clin. Pathol."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1097\/CMR.0000000000000515","article-title":"Innate Immune Cell Infiltration in Melanoma Metastases Affects Survival and Is Associated with BRAFV600E Mutation Status","volume":"29","author":"Dabrosin","year":"2019","journal-title":"Melanoma Res."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_28","unstructured":"Kingma, D.P., and Ba, J. (2017). Adam: A Method for Stochastic Optimization. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Sharma, D.K., Chatterjee, M., Kaur, G., and Vavilala, S. (2022). Deep Learning Applications for Disease Diagnosis. Deep Learning for Medical Applications with Unique Data, Elsevier.","DOI":"10.1016\/B978-0-12-824145-5.00005-8"},{"key":"ref_30","unstructured":"Manliguez, C. (2022, October 25). Generalized Confusion Matrix for Multiple Classes. Available online: https:\/\/www.researchgate.net\/publication\/310799885_Generalized_Confusion_Matrix_for_Multiple_Classes."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"857","DOI":"10.1002\/(SICI)1097-0258(19980430)17:8<857::AID-SIM777>3.0.CO;2-E","article-title":"Two-Sided Confidence Intervals for the Single Proportion: Comparison of Seven Methods","volume":"17","author":"Newcombe","year":"1998","journal-title":"Statist. Med."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"506","DOI":"10.1097\/PAI.0b013e318279bc0a","article-title":"SOX10 Expression in Malignant Melanoma, Carcinoma, and Normal Tissues","volume":"21","author":"Mohamed","year":"2013","journal-title":"Appl. Immunohistochem. Mol. Morphol."},{"key":"ref_33","first-page":"e89","article-title":"Nestin and SOX9 and SOX10 Transcription Factors Are Coexpressed in Melanoma: Nestin and SOX9\/10 Coexpression in Melanoma","volume":"19","author":"Bakos","year":"2009","journal-title":"Exp. Dermatol."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"882","DOI":"10.1038\/ncb2535","article-title":"Sox10 Promotes the Formation and Maintenance of Giant Congenital Naevi and Melanoma","volume":"14","author":"Shakhova","year":"2012","journal-title":"Nat. Cell Biol."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"468","DOI":"10.1097\/CMR.0b013e3283403ccd","article-title":"SOX10 Expression in Superficial Spreading and Nodular Malignant Melanomas","volume":"20","author":"Sooman","year":"2010","journal-title":"Melanoma Res."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1291","DOI":"10.1097\/PAS.0b013e3181658c14","article-title":"Sox10: A Pan-Schwannian and Melanocytic Marker","volume":"32","author":"Nonaka","year":"2008","journal-title":"Am. J. Surg. Pathol."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"544","DOI":"10.1111\/cup.12939","article-title":"SOX10 Immunohistochemistry in Sweat Ductal\/Glandular Neoplasms","volume":"44","author":"Cassarino","year":"2017","journal-title":"J. Cutan. Pathol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"666","DOI":"10.1016\/S0959-8049(02)00150-8","article-title":"Neovascularisation, Expression of Fibroblast Growth Factor-2, and Mast Cells with Tryptase Activity Increase Simultaneously with Pathological Progression in Human Malignant Melanoma","volume":"39","author":"Ribatti","year":"2003","journal-title":"Eur. J. Cancer"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Ladstein, R.G., Bachmann, I.M., Straume, O., and Akslen, L.A. (2010). Ki-67 Expression Is Superior to Mitotic Count and Novel Proliferation Markers PHH3, MCM4 and Mitosin as a Prognostic Factor in Thick Cutaneous Melanoma. BMC Cancer, 10.","DOI":"10.1186\/1471-2407-10-140"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1309\/R8N1CN04KKFV6M7T","article-title":"Primary Melanoma of the Skin and Cutaneous Melanomatous Metastases: Comparative Histologic Features and Immunophenotypes","volume":"122","author":"Hunt","year":"2004","journal-title":"Am. J. Clin. Pathol."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"233","DOI":"10.5858\/135.2.233","article-title":"Observer Variability in the Interpretation of HER2\/ Neu Immunohistochemical Expression With Unaided and Computer-Aided Digital Microscopy","volume":"135","author":"Gavrielides","year":"2011","journal-title":"Arch. Pathol. Lab. Med."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1153","DOI":"10.1016\/j.humpath.2014.01.009","article-title":"Automated Quantification of MART1-Verified Ki67 Indices: Useful Diagnostic Aid in Melanocytic Lesions","volume":"45","author":"Nielsen","year":"2014","journal-title":"Hum. Pathol."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1884","DOI":"10.1093\/annonc\/mdz411","article-title":"Cutaneous Melanoma: ESMO Clinical Practice Guidelines for Diagnosis, Treatment and Follow-Up","volume":"30","author":"Michielin","year":"2019","journal-title":"Ann. Oncol."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"The Italian Melanoma Intergroup (IMI), Bruno, W., Martinuzzi, C., Andreotti, V., Pastorino, L., Spagnolo, F., Dalmasso, B., Cabiddu, F., Gualco, M., and Ballestrero, A. (2017). Heterogeneity and Frequency of BRAF Mutations in Primary Melanoma: Comparison between Molecular Methods and Immunohistochemistry. Oncotarget, 8, 8069\u20138082.","DOI":"10.18632\/oncotarget.14094"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1111\/cup.13393","article-title":"High Concordance of BRAF Mutational Status in Matched Primary and Metastatic Melanoma","volume":"46","author":"Cormican","year":"2019","journal-title":"J. Cutan. Pathol."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"106","DOI":"10.1016\/j.ejca.2017.05.015","article-title":"Discrepancy in BRAF Status Among Patients With Metastatic Malignant Melanoma: A Meta-Analysis","volume":"81","author":"Valachis","year":"2017","journal-title":"Eur. J. Cancer"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"H\u00e9lias-Rodzewicz, Z., Funck-Brentano, E., Baudoux, L., Jung, C.K., Zimmermann, U., Marin, C., Clerici, T., Le Gall, C., Peschaud, F., and Taly, V. (2015). Variations of BRAF Mutant Allele Percentage in Melanomas. BMC Cancer, 15.","DOI":"10.1186\/s12885-015-1515-3"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"e25594","DOI":"10.4161\/onci.25594","article-title":"Biological Insights into BRAFV600 Mutations in Melanoma Patient: Not Mere Therapeutic Targets","volume":"2","author":"Improta","year":"2013","journal-title":"OncoImmunology"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2015, January 7\u201313). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.123"}],"container-title":["International Journal of Environmental Research and Public Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1660-4601\/19\/21\/14327\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:09:18Z","timestamp":1760144958000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1660-4601\/19\/21\/14327"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,2]]},"references-count":49,"journal-issue":{"issue":"21","published-online":{"date-parts":[[2022,11]]}},"alternative-id":["ijerph192114327"],"URL":"https:\/\/doi.org\/10.3390\/ijerph192114327","relation":{},"ISSN":["1660-4601"],"issn-type":[{"value":"1660-4601","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,2]]}}}