{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,25]],"date-time":"2026-04-25T18:38:40Z","timestamp":1777142320894,"version":"3.51.4"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1012501","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2024,10,10]],"date-time":"2024-10-10T00:00:00Z","timestamp":1728518400000}}],"reference-count":28,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2024,9,30]],"date-time":"2024-09-30T00:00:00Z","timestamp":1727654400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Solid tumors are characterized by complex interactions between the tumor, the immune system and the microenvironment. These interactions and intra-tumor variations have both diagnostic and prognostic significance and implications. However, quantifying the underlying processes in patient samples requires expensive and complicated molecular experiments. In contrast, H&amp;E staining is typically performed as part of the routine standard process, and is very cheap. Here we present HIPI (H&amp;E Image Interpretation and Protein Expression Inference) for predicting cell marker expression from tumor H&amp;E images. We process paired H&amp;E and CyCIF images taken from serial sections of colorectal cancers to train our model. We show that our model accurately predicts the spatial distribution of several important cell markers, on both held-out tumor regions as well as new tumor samples taken from different patients. Moreover, using only the tissue image morphology, HIPI is able to colocalize the interactions between different cell types, further demonstrating its potential clinical significance.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1012501","type":"journal-article","created":{"date-parts":[[2024,9,30]],"date-time":"2024-09-30T13:39:35Z","timestamp":1727703575000},"page":"e1012501","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":3,"title":["HIPI: Spatially resolved multiplexed protein expression inferred from H&amp;E WSIs"],"prefix":"10.1371","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2477-3078","authenticated-orcid":true,"given":"Ron","family":"Zeira","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Leon","family":"Anavy","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zohar","family":"Yakhini","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ehud","family":"Rivlin","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daniel","family":"Freedman","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"340","published-online":{"date-parts":[[2024,9,30]]},"reference":[{"issue":"5","key":"pcbi.1012501.ref001","doi-asserted-by":"crossref","first-page":"e253","DOI":"10.1016\/S1470-2045(19)30154-8","article-title":"Digital pathology and artificial intelligence","volume":"20","author":"MKK Niazi","year":"2019","journal-title":"The lancet oncology"},{"issue":"7861","key":"pcbi.1012501.ref002","doi-asserted-by":"crossref","first-page":"106","DOI":"10.1038\/s41586-021-03512-4","article-title":"AI-based pathology predicts origins for cancers of unknown primary","volume":"594","author":"MY Lu","year":"2021","journal-title":"Nature"},{"issue":"2","key":"pcbi.1012501.ref003","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1080\/10520290500138372","article-title":"The long history of hematoxylin","volume":"80","author":"M Titford","year":"2005","journal-title":"Biotechnic & Histochemistry"},{"issue":"3","key":"pcbi.1012501.ref004","doi-asserted-by":"crossref","first-page":"382","DOI":"10.5306\/wjco.v5.i3.382","article-title":"Significance of immunohistochemistry in breast cancer","volume":"5","author":"DC Zaha","year":"2014","journal-title":"World journal of clinical oncology"},{"issue":"9","key":"pcbi.1012501.ref005","doi-asserted-by":"crossref","first-page":"997","DOI":"10.1038\/s41592-021-01203-6","article-title":"Spatial omics and multiplexed imaging to explore cancer biology","volume":"18","author":"SM Lewis","year":"2021","journal-title":"Nature methods"},{"issue":"2","key":"pcbi.1012501.ref006","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1038\/s41592-021-01358-2","article-title":"Squidpy: a scalable framework for spatial omics analysis","volume":"19","author":"G Palla","year":"2022","journal-title":"Nature methods"},{"issue":"1","key":"pcbi.1012501.ref007","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1038\/s41592-020-01038-7","article-title":"Spatially resolved transcriptomics adds a new dimension to genomics","volume":"18","author":"L Larsson","year":"2021","journal-title":"Nature methods"},{"key":"pcbi.1012501.ref008","doi-asserted-by":"crossref","first-page":"e31657","DOI":"10.7554\/eLife.31657","article-title":"Highly multiplexed immunofluorescence imaging of human tissues and tumors using t-CyCIF and conventional optical microscopes","volume":"7","author":"JR Lin","year":"2018","journal-title":"eLife"},{"issue":"12","key":"pcbi.1012501.ref009","doi-asserted-by":"crossref","first-page":"930","DOI":"10.1038\/s44222-023-00096-8","article-title":"Artificial intelligence for digital and computational pathology","volume":"1","author":"AH Song","year":"2023","journal-title":"Nature Reviews Bioengineering"},{"issue":"4","key":"pcbi.1012501.ref010","doi-asserted-by":"crossref","first-page":"686","DOI":"10.1038\/s41416-020-01122-x","article-title":"Deep learning in cancer pathology: a new generation of clinical biomarkers","volume":"124","author":"A Echle","year":"2021","journal-title":"British journal of cancer"},{"issue":"8","key":"pcbi.1012501.ref011","doi-asserted-by":"crossref","first-page":"865","DOI":"10.1016\/j.ccell.2022.07.004","article-title":"Pan-cancer integrative histology-genomic analysis via multimodal deep learning","volume":"40","author":"RJ Chen","year":"2022","journal-title":"Cancer Cell"},{"issue":"1","key":"pcbi.1012501.ref012","doi-asserted-by":"crossref","first-page":"6753","DOI":"10.1038\/s41467-022-34275-9","article-title":"Deep learning-based image analysis predicts PD-L1 status from H&E-stained histopathology images in breast cancer","volume":"13","author":"G Shamai","year":"2022","journal-title":"Nature Communications"},{"issue":"1","key":"pcbi.1012501.ref013","article-title":"Deep learning-enabled virtual histological staining of biological samples","volume":"12","author":"B Bai","year":"2023","journal-title":"Light: Science & Applications"},{"issue":"2","key":"pcbi.1012501.ref014","doi-asserted-by":"crossref","DOI":"10.1016\/j.modpat.2023.100377","article-title":"An End-to-End Platform for Digital Pathology Using Hyperspectral Autofluorescence Microscopy and Deep Learning-Based Virtual Histology","volume":"37","author":"C McNeil","year":"2024","journal-title":"Modern Pathology"},{"issue":"8","key":"pcbi.1012501.ref015","doi-asserted-by":"crossref","first-page":"827","DOI":"10.1038\/s41551-020-0578-x","article-title":"Integrating spatial gene expression and breast tumour morphology via deep learning","volume":"4","author":"B He","year":"2020","journal-title":"Nature biomedical engineering"},{"issue":"1","key":"pcbi.1012501.ref016","doi-asserted-by":"crossref","first-page":"4133","DOI":"10.1038\/s41598-022-07685-4","article-title":"Efficient prediction of a spatial transcriptomics profile better characterizes breast cancer tissue sections without costly experimentation","volume":"12","author":"T Monjo","year":"2022","journal-title":"Scientific Reports"},{"issue":"4","key":"pcbi.1012501.ref017","doi-asserted-by":"crossref","first-page":"476","DOI":"10.1038\/s41587-021-01075-3","article-title":"Super-resolved spatial transcriptomics by deep data fusion","volume":"40","author":"L Bergenstr\u00e5hle","year":"2022","journal-title":"Nature biotechnology"},{"issue":"1","key":"pcbi.1012501.ref018","doi-asserted-by":"crossref","first-page":"18802","DOI":"10.1038\/s41598-020-75708-z","article-title":"Spatial transcriptomics inferred from pathology whole-slide images links tumor heterogeneity to survival in breast and lung cancer","volume":"10","author":"A Levy-Jurgenson","year":"2020","journal-title":"Scientific Reports"},{"key":"pcbi.1012501.ref019","unstructured":"Xie R, Pang K, Chung SW, Perciani CT, MacParland SA, Wang B, et al. Spatially Resolved Gene Expression Prediction from H&E Histology Images via Bi-modal Contrastive Learning; 2023."},{"key":"pcbi.1012501.ref020","doi-asserted-by":"crossref","first-page":"1275402","DOI":"10.3389\/fbinf.2023.1275402","article-title":"3D multiplexed tissue imaging reconstruction and optimized region of interest (ROI) selection through deep learning model of channels embedding","volume":"3","author":"E Burlingame","year":"2023","journal-title":"Front Bioinform"},{"key":"pcbi.1012501.ref021","doi-asserted-by":"crossref","unstructured":"Caron M, Touvron H, Misra I, J\u00e9gou H, Mairal J, Bojanowski P, et al. Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE\/CVF international conference on computer vision; 2021. p. 9650\u20139660.","DOI":"10.1109\/ICCV48922.2021.00951"},{"key":"pcbi.1012501.ref022","doi-asserted-by":"crossref","unstructured":"Chen RJ, Chen C, Li Y, Chen TY, Trister AD, Krishnan RG, et al. Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2022. p. 16144\u201316155.","DOI":"10.1109\/CVPR52688.2022.01567"},{"key":"pcbi.1012501.ref023","doi-asserted-by":"crossref","unstructured":"Kang M, Song H, Park S, Yoo D, Pereira S. Benchmarking Self-Supervised Learning on Diverse Pathology Datasets; 2023.","DOI":"10.1109\/CVPR52729.2023.00326"},{"key":"pcbi.1012501.ref024","first-page":"1","article-title":"Immune Checkpoint Blockade Therapy in Patients With Colorectal Cancer Harboring Microsatellite Instability\/Mismatch Repair Deficiency in 2022","volume":"42","author":"T Andr\u00e9","year":"2022","journal-title":"Am Soc Clin Oncol Educ Book"},{"issue":"2","key":"pcbi.1012501.ref025","doi-asserted-by":"crossref","first-page":"363","DOI":"10.1016\/j.cell.2022.12.028","article-title":"Multiplexed 3D atlas of state transitions and immune interaction in colorectal cancer","volume":"186","author":"JR Lin","year":"2023","journal-title":"Cell"},{"key":"pcbi.1012501.ref026","doi-asserted-by":"crossref","unstructured":"Shen Y, Luo Y, Shen D, Ke J. RandStainNA: Learning Stain-Agnostic Features from Histology Slides by Bridging Stain Augmentation and Normalization. In: Lecture Notes in Computer Science. Springer Nature Switzerland; 2022. p. 212\u2013221. Available from: https:\/\/doi.org\/10.1007%2F978-3-031-16434-7_21.","DOI":"10.1007\/978-3-031-16434-7_21"},{"key":"pcbi.1012501.ref027","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition; 2015.","DOI":"10.1109\/CVPR.2016.90"},{"issue":"6657","key":"pcbi.1012501.ref028","doi-asserted-by":"crossref","first-page":"eabq4964","DOI":"10.1126\/science.abq4964","article-title":"The dawn of spatial omics","volume":"381","author":"D Bressan","year":"2023","journal-title":"Science"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1012501","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2024,10,10]],"date-time":"2024-10-10T00:00:00Z","timestamp":1728518400000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1012501","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,10]],"date-time":"2024-10-10T13:39:49Z","timestamp":1728567589000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1012501"}},"subtitle":[],"editor":[{"given":"Kamilla","family":"Naxerova","sequence":"first","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2024,9,30]]},"references-count":28,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,9,30]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1012501","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2024.03.26.586744","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,30]]}}}