{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,31]],"date-time":"2026-01-31T02:56:48Z","timestamp":1769828208430,"version":"3.49.0"},"reference-count":38,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2026,1,28]],"date-time":"2026-01-28T00:00:00Z","timestamp":1769558400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Research and Development Council of Hungary","award":["KDP 61835-69456"],"award-info":[{"award-number":["KDP 61835-69456"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Introduction: Accurate nuclear segmentation is essential for the reliable diagnostic interpretation of fluorescence in situ hybridization (FISH) results. However, automated 2D digital algorithms often fail in samples with dense or overlapping nuclei, such as lymphomas, due to the loss of spatial depth information. Here, we tested if AI-based 3D nuclear segmentation can improve the accuracy, reproducibility, and diagnostic reliability of FISH reading in critical situations. Materials and Methods: Formalin-fixed follicular lymphoma sections were FISH-labeled for BCL2 gene rearrangements and digitally scanned in multilayer Z-stacks. The analytic performance in nuclear segmentation of the adaptive thresholding-based FISHQuant, and the freely accessible AI-based NucleAIzer, StarDist, and Cellpose algorithms, were compared to the eye control-based traditional FISH testing, primarily focusing on nuclear segmentation. Results: We revealed that the Cellpose algorithm showed limited sensitivity to low-intensity signals and the adaptive thresholding 2D segmentation, and FISHQuant struggled to resolve densely packed nuclei, occasionally underestimating their counts. In contrast, 3D segmentation across focal planes significantly improved the nuclear separation and signal localization. AI-driven 3D models, especially NucleAIzer and StarDist, showed improved precision, lower variance (VP\/VS \u2248 0.96), and improved gene spot correlation (r &gt; 0.82) across multiple focal planes. The similar average number of gene spots per cell nuclei in the AI-based analyses as the eye control counting, despite the elevated number of cell nuclei found with AI, validated the AI nuclear segmentation results. Conclusions: Inaccurate segmentation limits automated diagnostic FISH signal evaluation. Deep learning 3D approaches, particularly NucleAIzer and StarDist, may overcome thresholding and 2D constraints and improve the consistency of nuclear detection, resulting in better classification of pathogenetic gene aberrations with automated workflows in digital pathology.<\/jats:p>","DOI":"10.3390\/s26030873","type":"journal-article","created":{"date-parts":[[2026,1,28]],"date-time":"2026-01-28T17:40:22Z","timestamp":1769622022000},"page":"873","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Enhanced Precision of Fluorescence In Situ Hybridization (FISH) Analysis Using Neural Network-Based Nuclear Segmentation for Digital Microscopy Samples"],"prefix":"10.3390","volume":"26","author":[{"given":"Annamaria","family":"Csizmadia","sequence":"first","affiliation":[{"name":"Doctoral School of Pathological Sciences, Semmelweis University, H-1085 Budapest, Hungary"},{"name":"3DHISTECH Ltd., H-1141 Budapest, Hungary"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bela","family":"Molnar","sequence":"additional","affiliation":[{"name":"3DHISTECH Ltd., H-1141 Budapest, Hungary"},{"name":"Department of Internal Medicine and Oncology, Semmelweis University, H-1083 Budapest, Hungary"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-6362-6763","authenticated-orcid":false,"given":"Marianna Dimitrova","family":"Kucarov","sequence":"additional","affiliation":[{"name":"Doctoral School of Applied Informatics and Applied Mathematics BioTech, Obuda University, H-1034 Budapest, Hungary"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3136-8978","authenticated-orcid":false,"given":"Krisztian","family":"Koos","sequence":"additional","affiliation":[{"name":"Synthetic and Systems Biology Unit, Biological Research Center, H-6726 Szeged, Hungary"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9738-8336","authenticated-orcid":false,"given":"Robert","family":"Paulik","sequence":"additional","affiliation":[{"name":"3DHISTECH Ltd., H-1141 Budapest, Hungary"},{"name":"Synthetic and Systems Biology Unit, Biological Research Center, H-6726 Szeged, Hungary"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dora","family":"Kapczar","sequence":"additional","affiliation":[{"name":"Department of Pathology and Experimental Cancer Research, Semmelweis University, \u00dcll\u0151i \u00fat 26, H-1085 Budapest, Hungary"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6541-3031","authenticated-orcid":false,"given":"Laszlo","family":"Krenacs","sequence":"additional","affiliation":[{"name":"T-Cell Ltd., Laboratory of Tumor Pathology and Molecular Diagnostics, H-6726 Szeged, Hungary"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Balazs","family":"Csernus","sequence":"additional","affiliation":[{"name":"Department of Pathology and Experimental Cancer Research, Semmelweis University, \u00dcll\u0151i \u00fat 26, H-1085 Budapest, Hungary"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-9462-3294","authenticated-orcid":false,"given":"Gergo","family":"Papp","sequence":"additional","affiliation":[{"name":"Department of Pathology and Experimental Cancer Research, Semmelweis University, \u00dcll\u0151i \u00fat 26, H-1085 Budapest, Hungary"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9164-065X","authenticated-orcid":false,"given":"Tibor","family":"Krenacs","sequence":"additional","affiliation":[{"name":"Department of Pathology and Experimental Cancer Research, Semmelweis University, \u00dcll\u0151i \u00fat 26, H-1085 Budapest, Hungary"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2026,1,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1002\/cjp2.64","article-title":"Fluorescence in situ hybridization in surgical pathology: Principles and applications","volume":"3","author":"Cheng","year":"2017","journal-title":"J. Pathol. Clin. Res."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"952","DOI":"10.1002\/cyto.a.23863","article-title":"Evaluation of Deep Learning Strategies for Nucleus Segmentation in Fluorescence Images","volume":"95","author":"Caicedo","year":"2019","journal-title":"Cytom. A."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Kleinberg, G., Wang, S., Comellas, E., Monaghan, J.R., and Shefelbine, S.J. (2022). Usability of deep learning pipelines for 3D nuclei identification with Stardist and Cellpose. Cells Dev., 172.","DOI":"10.1016\/j.cdev.2022.203806"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1016\/j.tcb.2021.12.004","article-title":"Nucleus segmentation: Towards automated solutions","volume":"32","author":"Hollandi","year":"2022","journal-title":"Trends Cell Biol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"234","DOI":"10.1109\/RBME.2016.2515127","article-title":"Robust Nucleus\/Cell Detection and Segmentation in Digital Pathology and Microscopy Images: A Comprehensive Review","volume":"9","author":"Xing","year":"2016","journal-title":"IEEE Rev. Biomed. Eng."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"595","DOI":"10.1002\/cyto.a.23124","article-title":"An optimized image analysis algorithm for detecting nuclear signals in digital whole slides for histopathology","volume":"91","author":"Paulik","year":"2017","journal-title":"Cytom. A"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Kiszler, G., Krecs\u00e1k, L., Csizmadia, A., Micsik, T., Szab\u00f3, D., J\u00f3n\u00e1s, V., Pr\u00e9musz, V., Kren\u00e1cs, T., and Moln\u00e1r, B. (2013). Semi-automatic FISH quantification on digital slides. Diagn. Pathol., 8.","DOI":"10.1186\/1746-1596-8-S1-S21"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Chea, V., Pleiner, V., Schweizer, V., Herzog, B., Bode, B., and Tinguely, M. (2021). Optimized workflow for digitalized FISH analysis in pathology. Diagn. Pathol., 16.","DOI":"10.1186\/s13000-021-01103-5"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1016\/j.cels.2020.04.003","article-title":"nucleAIzer: A Parameter-free Deep Learning Framework for Nucleus Segmentation Using Image Style Transfer","volume":"10","author":"Hollandi","year":"2020","journal-title":"Cell Syst."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1934","DOI":"10.1109\/TMI.2021.3069558","article-title":"Evaluation of Deep Learning Architectures for Complex Immunofluorescence Nuclear Image Segmentation","volume":"40","author":"Kromp","year":"2021","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"592","DOI":"10.1038\/s41592-025-02595-5","article-title":"Cellpose3: One-click image restoration for improved cellular segmentation","volume":"22","author":"Stringer","year":"2025","journal-title":"Nat. Methods"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1090","DOI":"10.1038\/s41592-018-0216-7","article-title":"Content-aware image restoration: Pushing the limits of fluorescence microscopy","volume":"15","author":"Weigert","year":"2018","journal-title":"Nat. Methods"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Riendeau, J.M., Gillette, A.A., Guzman, E.C., Cruz, M.C., Kralovec, A., Udgata, S., Schmitz, A., Deming, D.A., Cimini, B.A., and Skala, M.C. (2024). Cellpose as a reliable method for single-cell segmentation of autofluorescence microscopy images. bioRxiv, bioRxiv:2024.06.07.597994.","DOI":"10.1101\/2024.06.07.597994"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"134","DOI":"10.2353\/jmoldx.2007.060128","article-title":"Guidance for fluorescence in situ hybridization testing in hematologic disorders","volume":"9","author":"Wolff","year":"2007","journal-title":"J. Mol. Diagn."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"667","DOI":"10.1097\/GIM.0b013e3182227295","article-title":"Section E9 of the American College of Medical Genetics technical standards and guidelines: Fluorescence in situ hybridization","volume":"13","author":"Mascarello","year":"2011","journal-title":"Genet. Med."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"899","DOI":"10.1016\/S0092-8674(85)80070-2","article-title":"Cloning the chromosomal breakpoint of t(14;18) human lymphomas: Clustering around JH on chromosome 14 and near a transcriptional unit on 18","volume":"41","author":"Bakhshi","year":"1985","journal-title":"Cell"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Correia, C., Maurer, M.J., McDonough, S.J., Schneider, P.A., Ross, P.E., Novak, A.J., Feldman, A.L., Cerhan, J.R., Slager, S.L., and Witzig, T.E. (2023). Relationship between BCL2 mutations and follicular lymphoma outcome in the chemoimmunotherapy era. Blood Cancer J., 13.","DOI":"10.1038\/s41408-023-00847-1"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"340","DOI":"10.1038\/315340a0","article-title":"Clustering of breakpoints on chromosome 11 in human B-cell neoplasms with the t(11;14) chromosome translocation","volume":"315","author":"Tsujimoto","year":"1985","journal-title":"Nature"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1440","DOI":"10.1126\/science.3874430","article-title":"Involvement of the bcl-2 gene in human follicular lymphoma","volume":"228","author":"Tsujimoto","year":"1985","journal-title":"Science"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Stringer, C., and Pachitariu, M. (2024). Analysis methods for large-scale neuronal recordings. Science, 386.","DOI":"10.1126\/science.adp7429"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1038\/s41592-020-01018-x","article-title":"Cellpose: A generalist algorithm for cellular segmentation","volume":"18","author":"Stringer","year":"2021","journal-title":"Nat. Methods"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Taha, A.A., and Hanbury, A. (2015). Metrics for evaluating 3D medical image segmentation: Analysis, selection, and tool. BMC Med. Imaging, 15.","DOI":"10.1186\/s12880-015-0068-x"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Chalfoun, J., Majurski, M., Blattner, T., Bhadriraju, K., Keyrouz, W., Bajcsy, P., and Brady, M. (2017). MIST: Accurate and Scalable Microscopy Image Stitching Tool with Stage Modeling and Error Minimization. Sci. Rep., 7.","DOI":"10.1038\/s41598-017-04567-y"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Chrzanowska, N.M., Kowalewski, J., and Lewandowska, M.A. (2020). Use of Fluorescence In Situ Hybridization (FISH) in Diagnosis and Tailored Therapies in Solid Tumors. Molecules, 25.","DOI":"10.3390\/molecules25081864"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Kucarov, M.D., Csizmadia, A., Moln\u00e1r, B., and Kozlovszky, M. (2024, January 17\u201319). FITC & TRITC Spot Detection and Analysis at BCL2 Gene Locus as Follicular Lymphoma Marker on 2D Digital Tissue Slides and 3D Serial Sections. Proceedings of the 2024 IEEE 28th International Conference on Intelligent Engineering Systems (INES), Gammarth, Tunisia.","DOI":"10.1109\/INES63318.2024.10629118"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Bouilhol, E., Savulescu, A.F., Lefevre, E., Dartigues, B., Brackin, R., and Nikolski, M. (2022). DeepSpot: A deep neural network for RNA spot enhancement in single-molecule fluorescence in-situ hybridization microscopy images. Biol. Imaging, 2.","DOI":"10.1017\/S2633903X22000034"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Xue, T., Chang, H., Ren, M., Wang, H., Yang, Y., Wang, B., Lv, L., Tang, L., Fu, C., and Fang, Q. (2023). Deep learning to automatically evaluate HER2 gene amplification status from fluorescence in situ hybridization images. Sci. Rep., 13.","DOI":"10.1038\/s41598-023-36811-z"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Jian, Z., Song, T., Zhang, Z., Ai, Z., Zhao, H., Tang, M., and Liu, K. (2024). An Improved Nested U-Net Network for Fluorescence In Situ Hybridization Cell Image Segmentation. Sensors, 24.","DOI":"10.3390\/s24030928"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"481","DOI":"10.1007\/s00429-020-02194-4","article-title":"Optimization and evaluation of fluorescence in situ hybridization chain reaction in cleared fresh-frozen brain tissues","volume":"226","author":"Kumar","year":"2021","journal-title":"Brain Struct. Funct."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Zhang, J., Zhang, Y., Jin, Y., Xu, J., and Xu, X. (2023). MDU-Net: Multi-scale densely connected U-Net for biomedical image segmentation. Health Inf. Sci. Syst., 11.","DOI":"10.1007\/s13755-022-00204-9"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1233","DOI":"10.1038\/s41592-019-0403-1","article-title":"Deep learning for cellular image analysis","volume":"16","author":"Moen","year":"2019","journal-title":"Nat. Methods"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"3259","DOI":"10.1364\/BOE.477384","article-title":"Comparison of denoising tools for the reconstruction of nonlinear multimodal images","volume":"14","author":"Houhou","year":"2023","journal-title":"Biomed. Opt. Express"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"15254","DOI":"10.1364\/OE.546621","article-title":"Denoising 3D integral images by a single-shot unsupervised deep neural network","volume":"33","author":"Yaffe","year":"2025","journal-title":"Opt. Express"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1895","DOI":"10.1038\/s41592-024-02400-9","article-title":"Self-inspired learning for denoising live-cell super-resolution microscopy","volume":"21","author":"Qu","year":"2024","journal-title":"Nat. Methods"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"12925","DOI":"10.1021\/acs.analchem.5c00825","article-title":"Efficient Denoising of Shot-Noise in Mass Spectrometry Images by PCA-Assisted Self-Supervised Deep Learning","volume":"97","author":"Seydoux","year":"2025","journal-title":"Anal. Chem."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1558","DOI":"10.1038\/s41592-024-02244-3","article-title":"Pretraining a foundation model for generalizable fluorescence microscopy-based image restoration","volume":"21","author":"Ma","year":"2024","journal-title":"Nat. Methods"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1038\/nrclinonc.2017.166","article-title":"Tumour heterogeneity and resistance to cancer therapies","volume":"15","author":"Shaw","year":"2018","journal-title":"Nat. Rev. Clin. Oncol."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Zhu, L., Jiang, M., Wang, H., Sun, H., Zhu, J., Zhao, W., Fang, Q., Yu, J., Chen, P., and Wu, S. (2021). A narrative review of tumor heterogeneity and challenges to tumor drug therapy. Ann. Transl. Med., 9.","DOI":"10.21037\/atm-21-1948"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/3\/873\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T14:37:18Z","timestamp":1769783838000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/3\/873"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,28]]},"references-count":38,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2026,2]]}},"alternative-id":["s26030873"],"URL":"https:\/\/doi.org\/10.3390\/s26030873","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,28]]}}}