{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T21:21:51Z","timestamp":1792790511337,"version":"4.5.0"},"reference-count":22,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:00:00Z","timestamp":1750291200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Bioengineering"],"abstract":"<jats:p>Accurate segmentation of cellular structures in whole slide images (WSIs) is essential for quantitative analysis in computational pathology. However, the complexity and scale of WSIs present significant challenges for conventional segmentation methods. In this study, we propose a novel hybrid deep learning framework that integrates three complementary approaches, YOLOv11, StarDist, and Segment Anything Model v2 (SAM2), to achieve robust and precise cell segmentation. The proposed pipeline utilizes YOLOv11 as an object detector to localize regions of interest, generating bounding boxes or preliminary masks that are subsequently used either as prompts to guide SAM2 or to filter segmentation outputs. StarDist is employed to model cell and nuclear boundaries with high geometric precision using star-convex polygon representations, which are particularly effective in densely packed cellular regions. The framework was evaluated on a unique WSI dataset comprising 256 \u00d7 256 image tiles annotated with high-resolution cell-level masks. Quantitative evaluations using the Dice coefficient, intersection over union (IoU), F1-score, precision, and recall demonstrated that the proposed method significantly outperformed individual baseline models. The integration of object detection and prompt-based segmentation led to enhanced boundary accuracy, improved localization, and greater robustness across varied tissue types. This work contributes a scalable and modular solution for advancing automated histopathological image analysis.<\/jats:p>","DOI":"10.3390\/bioengineering12060674","type":"journal-article","created":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T08:43:58Z","timestamp":1750322638000},"page":"674","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["A Hybrid Deep Learning Framework for Accurate Cell Segmentation in Whole Slide Images Using YOLOv11, StarDist, and SAM2"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6549-940X","authenticated-orcid":false,"given":"Julius","family":"Bamwenda","sequence":"first","affiliation":[{"name":"Engineering Faculty, Electrical & Electronics Engineering Department, Dicle University, 21280 Diyarbak\u0131r, T\u00fcrkiye"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9368-8902","authenticated-orcid":false,"given":"Mehmet Sira\u00e7","family":"\u00d6zerdem","sequence":"additional","affiliation":[{"name":"Engineering Faculty, Electrical & Electronics Engineering Department, Dicle University, 21280 Diyarbak\u0131r, T\u00fcrkiye"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5673-8408","authenticated-orcid":false,"given":"Orhan","family":"Ayy\u0131ld\u0131z","sequence":"additional","affiliation":[{"name":"Medical Faculty, Department of Internal Medicine-Hematology, Dicle University, 21280 Diyarbak\u0131r, T\u00fcrkiye"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Veys\u0131","family":"Akpolat","sequence":"additional","affiliation":[{"name":"Medical Faculty, Department of Biophysics, Dicle University, 21280 Diyarbak\u0131r, T\u00fcrkiye"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,6,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.media.2017.07.005","article-title":"A survey on deep learning in medical image analysis","volume":"42","author":"Litjens","year":"2017","journal-title":"Med. Image Anal."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.csbj.2018.01.001","article-title":"Machine learning methods for histopathological image analysis","volume":"16","author":"Komura","year":"2018","journal-title":"Comput. Struct. Biotechnol. J."},{"key":"ref_3","first-page":"234","article-title":"U-Net: Convolutional networks for biomedical image segmentation","volume":"9351","author":"Ronneberger","year":"2015","journal-title":"Med. Image Comput. Comput. Assist. Interv."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep Residual Learning for Image Recognition. Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 770\u2013778.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"350","DOI":"10.1016\/S0140-6736(19)32998-8","article-title":"Deep learning for prediction of colorectal cancer outcome: A discovery and validation study","volume":"395","author":"Skrede","year":"2020","journal-title":"Lancet"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1301","DOI":"10.1038\/s41591-019-0508-1","article-title":"Clinical-grade computational pathology using weakly supervised deep learning on whole slide images","volume":"25","author":"Campanella","year":"2019","journal-title":"Nat. Med."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., and Lo, W.-Y. (2023). Segment Anything. arXiv.","DOI":"10.1109\/ICCV51070.2023.00371"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Li, X., Zhang, H., Guo, J., and Zhang, X. (2024). An improved YOLOv11 for precise detection in medical microscopy images. Biomed. Signal Process. Control, 89.","DOI":"10.1007\/978-981-97-9294-8"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Schmidt, U., Weigert, M., Broaddus, C., and Myers, G. (2018). Cell detection with star-convex polygons. Medical Image Computing and Computer-Assisted Intervention, Springer.","DOI":"10.1007\/978-3-030-00934-2_30"},{"key":"ref_10","unstructured":"Girdhar, R., Kirillov, A., Mintun, E., Doll\u00e1r, P., and Girshick, R. (2025, June 18). Segment Anything Model 2 (SAM 2): Efficient Promptable Segmentation. Meta AI Technical Report 2023. Available online: https:\/\/arxiv.org\/abs\/2408.00714."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1550","DOI":"10.1109\/TMI.2017.2677499","article-title":"A dataset and a technique for generalized nuclear segmentation for computational pathology","volume":"36","author":"Kumar","year":"2017","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"106","DOI":"10.1038\/s41586-021-03512-4","article-title":"AI-based pathology predicts origins for cancers of unknown primary","volume":"594","author":"Lu","year":"2021","journal-title":"Nature"},{"key":"ref_13","unstructured":"Bochkovskiy, A., Wang, C.Y., and Liao, H.Y.M. (2020). YOLOv4: Optimal speed and accuracy of object detection. arXiv."},{"key":"ref_14","unstructured":"Khanam, R., and Hussain, M. (2024). What Is YOLOv5: A Deep Look into the Internal Features of the Popular Object Detector. arXiv."},{"key":"ref_15","unstructured":"Wang, C.Y., Yeh, I.H., and Liao, H.Y.M. (2021). YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv."},{"key":"ref_16","unstructured":"K\u00f6hler, N., Buchholz, T.O., and Jug, F. (2020). StarDist 3D: Detection of cell nuclei in microscopy images. bioRxiv."},{"key":"ref_17","first-page":"109426","article-title":"Ensemble learning for cell detection using transformer-enhanced CNNs","volume":"139","author":"Ren","year":"2023","journal-title":"Pattern Recognit."},{"key":"ref_18","first-page":"101615","article-title":"Star-shaped object detection for histopathology image analysis","volume":"60","author":"Xing","year":"2020","journal-title":"Med. Image Anal."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1016\/j.celrep.2018.03.086","article-title":"Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images","volume":"23","author":"Saltz","year":"2018","journal-title":"Cell Rep."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Kosaraju, S., Park, J., Lee, H., Yang, J.W., and Kang, M. (2022). Deep learning-based framework for slide-based histopathological image analysis. Sci. Rep., 12.","DOI":"10.1038\/s41598-022-23166-0"},{"key":"ref_21","first-page":"102838","article-title":"Unified pre-training framework for medical image segmentation","volume":"87","author":"Xu","year":"2023","journal-title":"Med. Image Anal."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"100402","DOI":"10.1016\/j.pacs.2022.100402","article-title":"Photoacoustic image synthesis with generative adversarial networks","volume":"28","author":"Schellenberg","year":"2022","journal-title":"Photoacoustics"}],"container-title":["Bioengineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2306-5354\/12\/6\/674\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:55:00Z","timestamp":1760032500000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2306-5354\/12\/6\/674"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,19]]},"references-count":22,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2025,6]]}},"alternative-id":["bioengineering12060674"],"URL":"https:\/\/doi.org\/10.3390\/bioengineering12060674","relation":{},"ISSN":["2306-5354"],"issn-type":[{"value":"2306-5354","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,19]]}}}