{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,9]],"date-time":"2026-03-09T14:41:34Z","timestamp":1773067294082,"version":"3.50.1"},"reference-count":43,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2022,9,20]],"date-time":"2022-09-20T00:00:00Z","timestamp":1663632000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01CA151306"],"award-info":[{"award-number":["R01CA151306"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01CA200690"],"award-info":[{"award-number":["R01CA200690"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["U01CA231782"],"award-info":[{"award-number":["U01CA231782"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100005190","name":"Melanoma Research Alliance","doi-asserted-by":"publisher","award":["622600"],"award-info":[{"award-number":["622600"]}],"id":[{"id":"10.13039\/100005190","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000005","name":"U.S. Department of Defense","doi-asserted-by":"publisher","award":["W81XWH-20-1-0797"],"award-info":[{"award-number":["W81XWH-20-1-0797"]}],"id":[{"id":"10.13039\/100000005","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000005","name":"U.S. Department of Defense","doi-asserted-by":"publisher","award":["W81XWH-20-1-0798"],"award-info":[{"award-number":["W81XWH-20-1-0798"]}],"id":[{"id":"10.13039\/100000005","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:p>A rapidly increasing rate of melanoma diagnosis has been noted over the past three decades, and nearly 1 in 4 skin biopsies are diagnosed as melanocytic lesions. The gold standard for diagnosis of melanoma is the histopathological examination by a pathologist to analyze biopsy material at both the cellular and structural levels. A pathologist's diagnosis is often subjective and prone to variability, while deep learning image analysis methods may improve and complement current diagnostic and prognostic capabilities. Mitoses are important entities when reviewing skin biopsy cases as their presence carries prognostic information; thus, their precise detection is an important factor for clinical care. In addition, semantic segmentation of clinically important structures in skin biopsies might help the diagnosis pipeline with an accurate classification. We aim to provide prognostic and diagnostic information on skin biopsy images, including the detection of cellular level entities, segmentation of clinically important tissue structures, and other important factors toward the accurate diagnosis of skin biopsy images. This paper is an overview of our work on analysis of digital whole slide skin biopsy images, including mitotic figure (mitosis) detection, semantic segmentation, diagnosis, and analysis of pathologists' viewing patterns, and with new work on melanocyte detection. Deep learning has been applied to our methods for all the detection, segmentation, and diagnosis work. In our studies, deep learning is proven superior to prior approaches to skin biopsy analysis. Our work on analysis of pathologists' viewing patterns is the only such work in the skin biopsy literature. Our work covers the whole spectrum from low-level entities through diagnosis and understanding what pathologists do in performing their diagnoses.<\/jats:p>","DOI":"10.3389\/frai.2022.1005086","type":"journal-article","created":{"date-parts":[[2022,9,20]],"date-time":"2022-09-20T14:27:37Z","timestamp":1663684057000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Automated analysis of whole slide digital skin biopsy images"],"prefix":"10.3389","volume":"5","author":[{"given":"Shima","family":"Nofallah","sequence":"first","affiliation":[]},{"given":"Wenjun","family":"Wu","sequence":"additional","affiliation":[]},{"given":"Kechun","family":"Liu","sequence":"additional","affiliation":[]},{"given":"Fatemeh","family":"Ghezloo","sequence":"additional","affiliation":[]},{"given":"Joann G.","family":"Elmore","sequence":"additional","affiliation":[]},{"given":"Linda G.","family":"Shapiro","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2022,9,20]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1007\/978-3-319-40618-3","article-title":"The eighth edition AJCC cancer staging manual: continuing to build a bridge from a population-based to a more \u201cpersonalized\u201d approach to cancer staging","volume":"67","author":"Amin","year":"2017","journal-title":"CA Cancer J. Clin"},{"key":"B2","doi-asserted-by":"publisher","first-page":"171","DOI":"10.1016\/j.jbi.2017.01.004","article-title":"Accuracy is in the eyes of the pathologist: the visual interpretive process and diagnostic accuracy with digital whole slide images","volume":"66","author":"Bruny\u00e9","year":"2017","journal-title":"J. Biomed. Inform"},{"key":"B3","first-page":"3640","article-title":"\u201cAttention to scale: scale-aware semantic image segmentation,\u201d","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition","author":"Chen","year":"2016"},{"key":"B4","first-page":"411","article-title":"\u201cMitosis detection in breast cancer histology images with deep neural networks,\u201d","volume-title":"International Conference on Medical Image Computing and Computer-Assisted Intervention","author":"Cire\u015fan","year":"2013"},{"key":"B5","article-title":"\u201cAn image is worth 16x16 words: transformers for image recognition at scale,\u201d","volume-title":"International Conference on Learning Representations","author":"Dosovitskiy","year":"2021"},{"key":"B6","doi-asserted-by":"publisher","first-page":"1471","DOI":"10.1245\/s10434-010-0985-4","article-title":"The American Joint Committee on cancer: the 7th edition of the ajcc cancer staging manual and the future of tnm","volume":"17","author":"Edge","year":"2010","journal-title":"Ann. Surg. Oncol"},{"key":"B7","doi-asserted-by":"publisher","first-page":"j2813","DOI":"10.1136\/bmj.j2813","article-title":"Pathologists' diagnosis of invasive melanoma and melanocytic proliferations: observer accuracy and reproducibility study","volume":"357","author":"Elmore","year":"2017","journal-title":"BMJ"},{"key":"B8","first-page":"132","article-title":"\u201cNuclei grading of clear cell renal cell carcinoma in histopathological image by composite high-resolution network,\u201d","volume-title":"International Conference on Medical Image Computing and Computer-Assisted Intervention","author":"Gao","year":"2021"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpi.2022.100104","article-title":"An analysis of pathologists' viewing processes as they diagnose whole slide digital images","author":"Ghezloo","year":"2022","journal-title":"J. Pathol. Inform"},{"key":"B10","first-page":"2672","article-title":"\u201cGenerative adversarial networks,\u201d","volume-title":"Advances in Neural Information Processing Systems","author":"Goodfellow","year":"2014"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2019.101563","article-title":"Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images","author":"Graham","year":"2019","journal-title":"Med. Image Anal"},{"key":"B12","first-page":"2961","article-title":"\u201cMask r-CNN,\u201d","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"He","year":"2017"},{"key":"B13","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2016.90","article-title":"\u201cDeep residual learning for image recognition,\u201d","volume-title":"2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","author":"He","year":"2016"},{"key":"B14","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2017.243","article-title":"\u201cDensely connected convolutional networks,\u201d","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Huang","year":"2017"},{"key":"B15","doi-asserted-by":"crossref","first-page":"6091","DOI":"10.1109\/EMBC.2013.6610942","article-title":"\u201cMulti-channels statistical and morphological features based mitosis detection in breast cancer histopathology,\u201d","volume-title":"2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)","author":"Irshad","year":"2013"},{"key":"B16","first-page":"1097","article-title":"\u201cImageNet classification with deep convolutional neural networks,\u201d","volume-title":"Advances in Neural Information Processing Systems 25","author":"Krizhevsky","year":"2012"},{"key":"B17","first-page":"1925","article-title":"\u201cRefineNet: multi-path refinement networks for high-resolution semantic segmentation,\u201d","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Lin","year":"2017"},{"key":"B18","doi-asserted-by":"publisher","first-page":"1977","DOI":"10.1109\/TMI.2021.3069874","article-title":"Unpaired stain transfer using pathology-consistent constrained generative adversarial networks","volume":"40","author":"Liu","year":"2021","journal-title":"IEEE Trans. Med. Imaging"},{"key":"B19","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1001\/jamadermatol.2017.4060","article-title":"Population-based analysis of histologically confirmed melanocytic proliferations using natural language processing","volume":"154","author":"Lott","year":"2018","journal-title":"JAMA Dermatol"},{"key":"B20","doi-asserted-by":"publisher","first-page":"284","DOI":"10.1109\/TITB.2012.2199595","article-title":"Automated segmentation of the melanocytes in skin histopathological images","volume":"17","author":"Lu","year":"","journal-title":"IEEE J. Biomed. Health Inform"},{"key":"B21","doi-asserted-by":"publisher","first-page":"509","DOI":"10.1016\/j.patcog.2012.07.020","article-title":"Detection of melanocytes in skin histopathological images using radial line scanning","volume":"46","author":"Lu","year":"","journal-title":"Pattern Recogn"},{"key":"B22","first-page":"5546","article-title":"\u201cEfficient epidermis segmentation for whole slide skin histopathological images,\u201d","volume-title":"2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society","author":"Lu","year":""},{"key":"B23","doi-asserted-by":"publisher","first-page":"594","DOI":"10.1109\/JBHI.2013.2277837","article-title":"Toward automatic mitotic cell detection and segmentation in multispectral histopathological images","volume":"18","author":"Lu","year":"","journal-title":"IEEE J. Biomed. Health Inform"},{"key":"B24","doi-asserted-by":"publisher","first-page":"2738","DOI":"10.1016\/j.patcog.2015.02.023","article-title":"Automated analysis and diagnosis of skin melanoma on whole slide histopathological images","volume":"48","author":"Lu","year":"2015","journal-title":"Pattern Recogn"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2022.102466","article-title":"End-to-end diagnosis of breast biopsy images with transformers","author":"Mehta","year":"2022","journal-title":"Med. Image Anal"},{"key":"B26","first-page":"663","article-title":"\u201cLearning to segment breast biopsy whole slide images,\u201d","volume-title":"2018 IEEE Winter Conference on Applications of Computer Vision (WACV)","author":"Mehta","year":""},{"key":"B27","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-01249-6_34","article-title":"\u201cESPNet: efficient spatial pyramid of dilated convolutions for semantic segmentation,\u201d","volume-title":"ECCV","author":"Mehta","year":""},{"key":"B28","first-page":"9190","article-title":"\u201cESPNetv2: a light-weight, power efficient, and general purpose convolutional neural network,\u201d","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Mehta","year":"2019"},{"key":"B29","doi-asserted-by":"crossref","first-page":"1179","DOI":"10.1109\/ICPR.2014.212","article-title":"\u201cLocalization of diagnostically relevant regions of interest in whole slide images,\u201d","volume-title":"2014 22nd international conference on pattern recognition","author":"Mercan","year":"2014"},{"key":"B30","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1007\/s10278-017-9990-5","article-title":"Characterizing diagnostic search patterns in digital breast pathology: scanners and drillers","volume":"31","author":"Mercan","year":"2018","journal-title":"J. Digit. Imaging"},{"key":"B31","doi-asserted-by":"publisher","first-page":"1713","DOI":"10.3390\/diagnostics12071713","article-title":"Improving the diagnosis of skin biopsies using tissue segmentation","volume":"12","author":"Nofallah","year":"","journal-title":"Diagnostics"},{"key":"B32","doi-asserted-by":"crossref","DOI":"10.1016\/j.compmedimag.2020.101832","article-title":"Machine learning techniques for mitoses classification","volume-title":"Comput. Med. Imaging Graph","author":"Nofallah","year":"2021"},{"key":"B33","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10278-022-00641-8","article-title":"Segmenting skin biopsy images with coarse and sparse annotations using U-Net","volume":"2","author":"Nofallah","year":"","journal-title":"J. Digit. Imaging"},{"key":"B34","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-24574-4_28","article-title":"U-Net: convolutional networks for biomedical image segmentation","volume-title":"arXiv preprint arXiv: 1505.04597","author":"Ronneberger","year":"2015"},{"key":"B35","doi-asserted-by":"crossref","DOI":"10.1200\/JCO.2010.31.5812","article-title":"Prognostic significance of mitotic rate in localized primary cutaneous melanoma: an analysis of patients in the multi-institutional American Joint Committee on cancer melanoma staging database","volume-title":"J. Clin. Oncol","author":"Thompson","year":"2011"},{"key":"B36","first-page":"6000","article-title":"\u201cAttention is all you need,\u201d","volume-title":"Advances in Neural Information Processing Systems, 30","author":"Vaswani","year":"2017"},{"key":"B37","doi-asserted-by":"crossref","first-page":"8798","DOI":"10.1109\/CVPR.2018.00917","article-title":"\u201cHigh-resolution image synthesis and semantic manipulation with conditional GANs,\u201d","volume-title":"2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Wang","year":"2018"},{"key":"B38","first-page":"3","article-title":"\u201cCBAM: convolutional block attention module,\u201d","volume-title":"Proceedings of the European Conference on Computer Vision (ECCV)","author":"Woo","year":"2018"},{"key":"B39","doi-asserted-by":"publisher","first-page":"163526","DOI":"10.1109\/ACCESS.2021.3132958","article-title":"Scale-aware transformers for diagnosing melanocytic lesions","volume":"9","author":"Wu","year":"2021","journal-title":"IEEE Access"},{"key":"B40","doi-asserted-by":"publisher","first-page":"124","DOI":"10.1016\/j.compmedimag.2018.01.008","article-title":"Automated analysis and classification of melanocytic tumor on skin whole slide images","volume":"66","author":"Xu","year":"2018","journal-title":"Comput. Med. Imaging Graph"},{"key":"B41","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13640-015-0076-3","article-title":"Epidermis segmentation in skin histopathological images based on thickness measurement and k-means algorithm","volume":"2015","author":"Xu","year":"2015","journal-title":"EURASIP J. Image Video Process"},{"key":"B42","article-title":"GAN-based virtual re-staining: a promising solution for whole slide image analysis","volume-title":"arXiv preprint arXiv:1901.04059","author":"Xu","year":"2019"},{"key":"B43","doi-asserted-by":"publisher","first-page":"236","DOI":"10.1038\/s42256-019-0052-1","article-title":"Pathologist-level interpretable whole-slide cancer diagnosis with deep learning","volume":"1","author":"Zhang","year":"2019","journal-title":"Nat. Mach. Intell"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2022.1005086\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,9,20]],"date-time":"2022-09-20T14:28:16Z","timestamp":1663684096000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2022.1005086\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,20]]},"references-count":43,"alternative-id":["10.3389\/frai.2022.1005086"],"URL":"https:\/\/doi.org\/10.3389\/frai.2022.1005086","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,20]]},"article-number":"1005086"}}