{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,10]],"date-time":"2026-11-10T22:57:54Z","timestamp":1794351474599,"version":"4.9.0"},"reference-count":30,"publisher":"MDPI AG","issue":"14","license":[{"start":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T00:00:00Z","timestamp":1784246400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"KPJ Healthcare University","award":["KPJU\/KPJRC\/GRANT\/2024\/01(INT)"],"award-info":[{"award-number":["KPJU\/KPJRC\/GRANT\/2024\/01(INT)"]}]},{"award":["KPJU\/KPJRC\/GRANT\/2024\/01(INT)"],"award-info":[{"award-number":["KPJU\/KPJRC\/GRANT\/2024\/01(INT)"]}],"id":[{"id":"https:\/\/ror.org\/01he7e113","id-type":"ROR","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004035","name":"Jordan University of Science and Technology","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004035","id-type":"DOI","asserted-by":"crossref"}]},{"id":[{"id":"https:\/\/ror.org\/03y8mtb59","id-type":"ROR","asserted-by":"publisher"}]},{"name":"King Abdullah University Hospital"},{"id":[{"id":"https:\/\/ror.org\/02f6hdc06","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Diagnostics"],"abstract":"<jats:p>Background: Cervical cancer remains one of the leading causes of cancer-related morbidity among women worldwide. The Papanicolaou (Pap) smear is widely used for early detection; however, its manual interpretation is time-consuming, requires substantial expertise, and is often affected by inter-observer variability, particularly in cases with dense and overlapping cells. Methods: This study developed an optimized deep learning framework for cervical cell instance segmentation, specifically targeting the separation of overlapping cells in Pap smear images. The proposed framework was based on Mask R-CNN with a ResNet-50 backbone and Feature Pyramid Network. A public development dataset of 460 cervical smear images was used for model development and internal evaluation, while an independent 210-image dataset collected from King Abdullah University Hospital was reserved as an external held-out clinical assessment set. To improve small-cell detection and mask refinement, the Region Proposal Network was adapted using biologically informed anchor scales (16, 32, 64, 128, 256), followed by soft calibration-based post-processing. Results: The proposed framework achieved an AP50 of 89.90% and a Mask IoU of 90.44%. Small-cell performance, measured using APs under the COCO small-object convention, reached 22.80%, while APm and APl reached 68.45% and 81.42%, respectively. Clinical cell-count evaluation on the independent KAUH held-out set was performed using the mean count of five physicians as the reference standard and achieved an overall clinical detection accuracy of 93.00%. Conclusions: The optimized Mask R-CNN framework improved the detection and separation of overlapping cervical cells in Pap smear images and may serve as a supportive tool for cytopathology workflows. The results suggest that biologically informed anchor optimization and soft calibration can improve cell-level instance segmentation, particularly for small and overlapping cells. Further validation on larger multi-center datasets remains necessary before routine clinical deployment.<\/jats:p>","DOI":"10.3390\/diagnostics16142240","type":"journal-article","created":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T02:32:08Z","timestamp":1784514728000},"page":"2240","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Accurate Segmentation of Overlapping Cervical Cells Using an Optimized Deep Learning Framework for Cytology Screening"],"prefix":"10.3390","volume":"16","author":[{"given":"Amal A.","family":"Alzu\u2019bi","sequence":"first","affiliation":[{"name":"Department of Computer Information Systems, Jordan University of Science and Technology, P.O. Box 3030, Irbid 22110, Jordan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohammad","family":"Khatatbeh","sequence":"additional","affiliation":[{"name":"Department of Computer Information Systems, Jordan University of Science and Technology, P.O. Box 3030, Irbid 22110, Jordan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7260-6085","authenticated-orcid":false,"given":"Wan Azani","family":"Mustafa","sequence":"additional","affiliation":[{"name":"Advanced Computing (AdvCOMP), Centre of Excellence, Universiti Malaysia Perlis (UniMAP), Pauh Putra Campus, Arau 02600, Perlis, Malaysia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7216-087X","authenticated-orcid":false,"given":"Norhayati Mohd","family":"Zain","sequence":"additional","affiliation":[{"name":"Medical Imaging Department, School of Integrative Medicine & Life Sciences, KPJ Healthcare University, Nilai 71800, Negeri Sembilan, Malaysia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2966-6442","authenticated-orcid":false,"given":"Hiam","family":"Alquran","sequence":"additional","affiliation":[{"name":"Department of Biomedical Systems and Informatics Engineering, Yarmouk University, P.O. Box 566, Irbid 21163, Jordan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alia","family":"Al-Mohtaseb","sequence":"additional","affiliation":[{"name":"Department of Pathology, Jordan University of Science and Technology, P.O. Box 3030, Irbid 22110, Jordan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2615-1988","authenticated-orcid":false,"given":"Khaled Z.","family":"Alawneh","sequence":"additional","affiliation":[{"name":"Department of Radiology and Nuclear Medicine, Jordan University of Science and Technology, P.O. Box 3030, Irbid 22110, Jordan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-3128-9924","authenticated-orcid":false,"given":"Mohammad","family":"Fawaeer","sequence":"additional","affiliation":[{"name":"Public Health & Preventive Medicine Division, Directorate of the Royal Medical Services, P.O. Box 855005, Amman 11855, Jordan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-6634-7265","authenticated-orcid":false,"given":"Bara\u2019a","family":"Fawaeer","sequence":"additional","affiliation":[{"name":"Hematology and Blood Banking, Ministry of Health, Princess Basma Teaching Hospital, Irbid 21110, Jordan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shatha","family":"Salameh","sequence":"additional","affiliation":[{"name":"Department of Obstetrics and Gynecology, King Abdullah University Hospital, Jordan University of Science and Technology, P.O. Box 3030, Irbid 22110, Jordan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ahmad","family":"Alhussain","sequence":"additional","affiliation":[{"name":"Department of General Surgery, King Abdullah University Hospital, P.O. Box 3030, Irbid 22110, Jordan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,7,17]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/0304-3835(94)90098-1","article-title":"Computer-assisted cervical cancer screening using neural networks","volume":"77","author":"Mango","year":"1994","journal-title":"Cancer Lett."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"367","DOI":"10.32604\/biocell.2022.016814","article-title":"Cervical cancer situation in Malaysia: A systematic literature review","volume":"46","author":"Mustafa","year":"2022","journal-title":"Biocell"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1261","DOI":"10.1109\/TIP.2015.2389619","article-title":"An improved joint optimization of multiple level set functions for the segmentation of overlapping cervical cells","volume":"24","author":"Lu","year":"2015","journal-title":"IEEE Trans. Image Process."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"539","DOI":"10.1016\/j.cmpb.2013.12.012","article-title":"Automatic cervical cell segmentation and classification in Pap smears","volume":"113","author":"Chankong","year":"2014","journal-title":"Comput. Methods Programs Biomed."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Nosrati, M.S., and Hamarneh, G. (2015). Segmentation of overlapping cervical cells: A variational method with star-shape prior. Proceedings of the IEEE International Symposium on Biomedical Imaging, Brooklyn, NY, USA, 16\u201319 April 2015, IEEE.","DOI":"10.1109\/ISBI.2015.7163846"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Rodr\u00edguez, M., C\u00f3rdova, C., San Mart\u00edn, S., and Benjumeda, I. (2024). Automated cervical cancer screening using single-cell segmentation and deep learning: Enhanced performance with liquid-based cytology. Computation, 12.","DOI":"10.20944\/preprints202410.0386.v1"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1159\/000522212","article-title":"Endocervical cytology: Inter- and intra-observer variability in conventional Pap smears","volume":"66","author":"Pulkkinen","year":"2022","journal-title":"Acta Cytol."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Alias, N.A., Mustafa, W.A., Jamlos, M.A., Alquran, H., Hanafi, H.F., Ismail, S., and Rahman, K.S.A. (2022). Pap Smear Images Classification Using Machine Learning: A Literature Matrix. Diagnostics, 12.","DOI":"10.3390\/diagnostics12122900"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.media.2017.07.005","article-title":"A survey on deep learning in medical image analysis","volume":"42","author":"Litjens","year":"2017","journal-title":"Med. Image Anal."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Doll\u00e1r, P., and Girshick, R. (2015). Mask R-CNN. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22\u201329 October 2017, IEEE.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1016\/j.cmpb.2018.05.034","article-title":"A review of image analysis and machine learning techniques for automated cervical cancer screening","volume":"164","author":"William","year":"2018","journal-title":"Comput. Methods Programs Biomed."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"288","DOI":"10.1109\/TMI.2016.2606380","article-title":"Accurate cervical cell segmentation from overlapping clumps","volume":"36","author":"Song","year":"2017","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2687","DOI":"10.1007\/s10462-023-10588-z","article-title":"Deep learning-based cervical cytology screening: A systematic review","volume":"56","author":"Jiang","year":"2023","journal-title":"Artif. Intell. Rev."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"104483","DOI":"10.1016\/j.bspc.2022.104483","article-title":"Deep learning approach for cervical segmentation","volume":"81","year":"2023","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"128630","DOI":"10.1016\/j.neucom.2024.128630","article-title":"Systematic review on deep learning-based cervical cell image analysis","volume":"610","author":"Fang","year":"2024","journal-title":"Neurocomputing"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1016\/j.neucom.2019.06.086","article-title":"Accurate segmentation of overlapping cells in cervical cytology with deep convolutional neural networks","volume":"365","author":"Wan","year":"2019","journal-title":"Neurocomputing"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"116925","DOI":"10.1109\/ACCESS.2019.2936017","article-title":"Segmentation and classification of cervical cells using deep learning","volume":"7","author":"Kurnianingsih","year":"2019","journal-title":"IEEE Access"},{"key":"ref_19","first-page":"640","article-title":"IRNet: Instance relation network for overlapping cervical cell segmentation","volume":"Volume 11764","author":"Zhou","year":"2019","journal-title":"Medical Image Computing and Computer Assisted Intervention\u2014MICCAI 2019"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"14210","DOI":"10.1038\/s41598-024-64583-7","article-title":"An improved approach for automated cervical cell segmentation with PointRend","volume":"14","author":"Zhang","year":"2024","journal-title":"Sci. Rep."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"189387","DOI":"10.1109\/ACCESS.2024.3516935","article-title":"Enhanced MultiResUNet for cervical cytology segmentation","volume":"12","author":"Resmi","year":"2024","journal-title":"IEEE Access"},{"key":"ref_22","unstructured":"Jantzen, J., Norup, J., Dounias, G., and Bjerregaard, B. (2005). Pap-Smear Benchmark Data for Pattern Classification, Technical University of Denmark."},{"key":"ref_23","unstructured":"Plissiti, M.E., Dimitrakopoulos, P., Sfikas, G., Nikou, C., Krikoni, O., and Charchanti, A. (2018). SIPaKMeD dataset for cervical cell classification. 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece, 7\u201310 October 2018, IEEE."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"3890988","DOI":"10.1155\/2021\/3890988","article-title":"Segmentation of overlapping cervical cells with Mask R-CNN","volume":"2021","author":"Chen","year":"2021","journal-title":"Comput. Math. Methods Med."},{"key":"ref_25","first-page":"259","article-title":"Cervical cell image segmentation based on an improved Mask R-CNN model","volume":"Volume 220","author":"Wang","year":"2024","journal-title":"Proceedings of the 10th International Conference on Advanced Intelligent Systems and Informatics 2024"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"17715","DOI":"10.1038\/s41598-025-02379-z","article-title":"Optimized segmentation of overlapping cervical cells based on Mask2Former and denoising","volume":"15","author":"Zhang","year":"2025","journal-title":"Sci. Rep."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"3506","DOI":"10.1038\/s41467-025-58883-3","article-title":"Deep learning-enabled liquid-based cytology model for cervical precancer and cancer detection","volume":"16","author":"Xue","year":"2025","journal-title":"Nat. Commun."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Mosquera-Zamudio, A., Cancino, S., C\u00e1rdenas-Montoya, G., Garcia-Arteaga, J.D., Zambrano-Betancourt, C., and Parra-Medina, R. (2025). Deep-learning approaches for cervical cytology nuclei segmentation in whole slide images. J. Imaging, 11.","DOI":"10.3390\/jimaging11050137"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1038\/s41597-025-04374-5","article-title":"A large annotated cervical cytology image dataset for AI models","volume":"12","author":"Zhang","year":"2025","journal-title":"Sci. Data"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"4369","DOI":"10.1038\/s41467-024-48705-3","article-title":"Artificial intelligence enables precision diagnosis of cervical cytology grades and cervical cancer","volume":"15","author":"Wang","year":"2024","journal-title":"Nat. Commun."}],"container-title":["Diagnostics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-4418\/16\/14\/2240\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T04:15:20Z","timestamp":1784520920000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-4418\/16\/14\/2240"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,17]]},"references-count":30,"journal-issue":{"issue":"14","published-online":{"date-parts":[[2026,7]]}},"alternative-id":["diagnostics16142240"],"URL":"https:\/\/doi.org\/10.3390\/diagnostics16142240","relation":{},"ISSN":["2075-4418"],"issn-type":[{"value":"2075-4418","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,17]]}}}