{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,17]],"date-time":"2026-01-17T07:56:49Z","timestamp":1768636609414,"version":"3.49.0"},"reference-count":31,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:00:00Z","timestamp":1750204800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Oncol."],"abstract":"<jats:sec><jats:title>Background<\/jats:title><jats:p>Cellular imaging analysis using the traditional retrospective approach is extremely time-consuming and labor-intensive. Although AI-based solutions are available, these approaches rely heavily on supervised learning techniques that require high quality, large labeled datasets from the same microscope to be reliable. In addition, primary patient samples are often heterogeneous cell populations and need to be stained to distinguish the cellular subsets. The resulting imaging data is analyzed and labeled manually by experts. Therefore, a method to distinguish cell populations across imaging devices without the need for staining and extensive manual labeling would help immensely to gain real-time insights into cell population dynamics. This especially holds true for recognizing specific cell types and states in response to treatments.<\/jats:p><\/jats:sec><jats:sec><jats:title>Objective<\/jats:title><jats:p>We aim to develop an unsupervised approach using general vision foundation models trained on diverse and extensive imaging datasets to extract rich visual features for cell-analysis across devices, including both stained and unstained live cells. Our method, Entropy-guided Weighted Combinational FAISS (EWC-FAISS), uses these models purely in an inference-only mode without task-specific retraining on the cellular data. Combining the generated embeddings in an efficient and adaptive k-nearest neighbor search allows for automated, cross device identification of cell types and states, providing a strong basis for AI-assisted cancer therapy.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>We utilized two publicly available datasets. The WBC dataset includes 14,424 images of stained white blood cell samples from patients with acute myeloid and lymphoid leukemia, as well as those without leukemic pathology. The LISC dataset comprises 257 images of white blood cell samples from healthy individuals. We generated four in-house datasets utilizing the JIMT-1 breast cancer cell line, as well as Jurkat and K562 (leukemic cell lines). These datasets were acquired using the Nanolive 3D Cell Explorer-fluo (CX-A) holotomographic microscope and the BioTek Lionheart FX automated brightfield microscope. The images from the in-house datasets were manually annotated using Roboflow software. To generate the embeddings, we used and optimized a concatenated combination of SAM, DINO, ConvNeXT, SWIN, CLIP and ViTMAE. The combined embeddings were used as input for the adaptive k-nearest neighbor search, building an approximate Hierarchical Navigable Small World FAISS index. We compared EWC-FAISS to fully fined-tuned ViT-Classifiers with DINO-, and SWIN-backbones, a ConvNeXT architecture, as well as to NMTune as a lightweight domain-adaptation method with frozen backbone.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>EWC-FAISS performed competitively with the baselines on the original datasets in terms of macro accuracy. Macro accuracy is the average of class-specific accuracies, treating all classes equally by averaging their individual accuracies. EWC-FAISS ranked second for the WBC dataset (macro accuracy: 97.6 \u00b1 0.2), first for cell state classification from Nanolive (macro accuracy: 90 \u00b1 0), and performed comparably for cell type classification from Lionheart (macro accuracy: 87 \u00b1 0). For the transfer to out-of-distribution (OOD) datasets, which the model had not seen during training, EWC-FAISS consistently outperformed the other baselines. For the LISC dataset, EWC-FAISS achieved a macro accuracy of 78.5 \u00b1 0.3, compared to DINO FT\u2019s 17 \u00b1 1, SWIN FT\u2019s 44 \u00b1 14, ConvNeXT FT\u2019s 45 \u00b1 9, and NMTune\u2019s 52 \u00b1 10. For the cell state classification from Lionheart, EWC-FAISS had a macro accuracy of 86 \u00b1 1, while DINO FT, SWIN FT, and ConvNeXT FT achieved 65 \u00b1 11, 68 \u00b1 16, and 81 \u00b1 1, respectively, and NMTune 81 \u00b1 7. For the transfer of cell type classification from Nanolive, EWC-FAISS attained a macro accuracy of 85 \u00b1 0, compared to DINO FT\u2019s 24.5 \u00b1 0.9, SWIN FT\u2019s 57 \u00b1 6, ConvNeXT FT\u2019s 54 \u00b1 4, and NMTune\u2019s 63 \u00b1 4. Additionally, building EWC-FAISS after embedding generation was significantly faster than training DINO FT (\u223c 6 minutes compared to <jats:italic>&amp;gt;<\/jats:italic> 10 hours). Lastly, EWC-FAISS performed comparably in distinguishing cancerous cell lines from Peripheral Blood Mononuclear Cells with a mean accuracy of 80 \u00b1 5, compared to CellMixer with a mean accuracy of 79.7.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>We present a novel approach to identify various cell lines and primary cells based on their identity and state using images acquired across various imaging platforms which vary in resolution, magnification and image quality. Despite these differences, we could show that our efficient, adaptive k-nearest neighbor search pipeline can be applied on a large image dataset containing different cell types and effectively differentiate between the cells and their states such as live, apoptotic or necrotic. There are several applications, particularly in distinguishing various cell populations in patient samples or monitoring therapy.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fonc.2025.1480384","type":"journal-article","created":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T05:48:40Z","timestamp":1750225720000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Leveraging a foundation model zoo for cell similarity search in oncological microscopy across devices"],"prefix":"10.3389","volume":"15","author":[{"given":"Gabriel","family":"Kalweit","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anusha","family":"Klett","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Paula","family":"Silvestrini","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jens","family":"Rahnfeld","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mehdi","family":"Naouar","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yannick","family":"Vogt","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Diana","family":"Infante","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rebecca","family":"Berger","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jes\u00fas","family":"Duque-Afonso","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tanja Nicole","family":"Hartmann","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marie","family":"Follo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Elitsa","family":"Bodurova-Spassova","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"L\u00fcbbert","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Roland","family":"Mertelsmann","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Joschka","family":"Boedecker","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Evelyn","family":"Ullrich","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Maria","family":"Kalweit","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2025,6,18]]},"reference":[{"key":"B1","article-title":"The biological data sustainability paradox","volume-title":"arXiv","author":"Johnson","year":"2023"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2108.07258","article-title":"On the opportunities and risks of foundation models","author":"Bommasani","year":"2021","journal-title":"ArXiv"},{"key":"B3","doi-asserted-by":"crossref","DOI":"10.1109\/ICCV48922.2021.00951","article-title":"Emerging properties in self-supervised vision transformers","author":"Caron","year":"2021"},{"key":"B4","article-title":"DINOv2: Learning robust visual features without supervision","author":"Oquab","year":"2024","journal-title":"Trans Mach Learn Res"},{"key":"B5","doi-asserted-by":"crossref","DOI":"10.1109\/ICCV51070.2023.00371","article-title":"Segment anything","author":"Kirillov","year":"2023"},{"key":"B6","doi-asserted-by":"publisher","first-page":"654","DOI":"10.1038\/s41467-024-44824-z","article-title":"Segment anything in medical images","volume":"15","author":"Ma","year":"2024","journal-title":"Nat Commun"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-02857-3","article-title":"Towards a general-purpose foundation model for computational pathology","volume":"30","author":"Chen","year":"2024","journal-title":"Nat Med"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.1101\/2023.06.16.545359","article-title":"Unbiased single-cell morphology with self-supervised vision transformers","author":"Doron","year":"2023"},{"key":"B9","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-031-72390-2_49","article-title":"Dinobloom: A foundation model for generalizable cell embeddings in hematology","volume-title":"Medical Image Computing and Computer Assisted Intervention \u2013 MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science","author":"Koch","year":"2024"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.1101\/2023.01.16.524226","article-title":"Self-supervised vision transformers accurately decode cellular state heterogeneity","author":"Pfaendler","year":"2023","journal-title":"bioRxiv"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1101\/2023.11.17.567630","article-title":"A foundation model for cell segmentation","author":"Israel","year":"2023","journal-title":"bioRxiv"},{"key":"B12","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR52733.2024.01117","article-title":"Masked autoencoders for microscopy are scalable learners of cellular biology","author":"Kraus","year":"2024"},{"key":"B13","first-page":"9992","article-title":"Swin transformer: Hierarchical vision transformer using shifted windows","author":"Liu","year":"2021"},{"key":"B14","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR52688.2022.01170","article-title":"Swin transformer v2: Scaling up capacity and resolution","author":"Liu","year":"2022"},{"key":"B15","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR52688.2022.01167","article-title":"A convNet for the 2020s","author":"Liu","year":"2022"},{"key":"B16","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR52729.2023.01548","article-title":"ConvNeXt v2: Co-designing and scaling ConvNets with masked autoencoders","author":"Woo","year":"2023"},{"key":"B17","article-title":"Learning transferable visual models from natural language supervision","author":"Radford","year":"2021"},{"key":"B18","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR52688.2022.01553","article-title":"Masked autoencoders are scalable vision learners","author":"He","year":"2022"},{"key":"B19","article-title":"The platonic representation hypothesis","author":"Huh","year":"2024"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.34","article-title":"Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs","volume":"42","author":"Malkov","year":"2020","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"B21","author":"Douze","year":"2024","journal-title":"The faiss library"},{"key":"B22","article-title":"Quick-tune: Quickly learning which pretrained model to finetune and how","author":"Arango","year":"2024"},{"key":"B23","article-title":"Understanding and mitigating the label noise in pre-training on downstream tasks","author":"Chen","year":"2024"},{"key":"B24","doi-asserted-by":"publisher","first-page":"466","DOI":"10.1038\/s41597-023-02378-7","article-title":"A high-resolution large-scale dataset of pathological and normal white blood cells","volume":"10","author":"Bodzas","year":"2023","journal-title":"Sci Data"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-13775-417","article-title":"Automatic recognition of five types of white blood cells in peripheral blood","volume":"6112","author":"Rezatofighi","year":"2010"},{"key":"B26","first-page":"38","article-title":"Transformers: Stateof-the-art natural language processing","author":"Wolf","year":"2020"},{"key":"B27","article-title":"Similarity of neural network representations revisited","volume-title":"Proceedings of the 36th International Conference on Machine Learning","author":"Kornblith","year":"2019"},{"key":"B28","author":"Naouar","year":"2023","journal-title":"Cellmixer: Annotation-free semantic cell segmentation of heterogeneous cell populations"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.1159\/000512172","article-title":"The emergence of artificial intelligence within radiation oncology treatment planning","volume":"99","author":"Netherton","year":"2020","journal-title":"Oncology"},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.1016\/j.apsb.2021.02.007","article-title":"Applying artificial intelligence for cancer immunotherapy","volume":"11","author":"Xu","year":"2021","journal-title":"Acta Pharm Sin B"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.3390\/cancers16040831","article-title":"Artificial intelligence-based treatment decisions: A new era for nsclc","volume":"16","author":"Fiste","year":"2024","journal-title":"Cancers"}],"container-title":["Frontiers in Oncology"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fonc.2025.1480384\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T05:48:42Z","timestamp":1750225722000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fonc.2025.1480384\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,18]]},"references-count":31,"alternative-id":["10.3389\/fonc.2025.1480384"],"URL":"https:\/\/doi.org\/10.3389\/fonc.2025.1480384","relation":{},"ISSN":["2234-943X"],"issn-type":[{"value":"2234-943X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,18]]},"article-number":"1480384"}}