{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,9]],"date-time":"2025-09-09T21:47:56Z","timestamp":1757454476196,"version":"3.40.3"},"publisher-location":"Cham","reference-count":34,"publisher":"Springer Nature Switzerland","isbn-type":[{"type":"print","value":"9783031730238"},{"type":"electronic","value":"9783031730245"}],"license":[{"start":{"date-parts":[[2024,11,24]],"date-time":"2024-11-24T00:00:00Z","timestamp":1732406400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,11,24]],"date-time":"2024-11-24T00:00:00Z","timestamp":1732406400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>A central problem in biology is to understand how organisms evolve and adapt to their environment by acquiring variations in the observable characteristics or traits of species across the tree of life. With the growing availability of large-scale image repositories in biology and recent advances in generative modeling, there is an opportunity to accelerate the discovery of evolutionary traits automatically from images. Toward this goal, we introduce Phylo-Diffusion, a novel framework for conditioning diffusion models with phylogenetic knowledge represented in the form of HIERarchical Embeddings (HIER-Embeds). We also propose two new experiments for perturbing the embedding space of Phylo-Diffusion: trait masking and trait swapping, inspired by counterpart experiments of gene knockout and gene editing\/swapping. Our work represents a novel methodological advance in generative modeling to structure the embedding space of diffusion models using tree-based knowledge. Our work also opens a new chapter of research in evolutionary biology by using generative models to visualize evolutionary changes directly from images. We empirically demonstrate the usefulness of Phylo-Diffusion in capturing meaningful trait variations for fishes and birds, revealing novel insights about the biological mechanisms of their evolution. (Model and code can be found at <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/imageomics.github.io\/phylo-diffusion\/\">imageomics.github.io\/phylo-diffusion<\/jats:ext-link>)\n<\/jats:p>","DOI":"10.1007\/978-3-031-73024-5_9","type":"book-chapter","created":{"date-parts":[[2024,11,25]],"date-time":"2024-11-25T16:39:30Z","timestamp":1732552770000},"page":"137-153","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Hierarchical Conditioning of\u00a0Diffusion Models Using Tree-of-Life for\u00a0Studying Species Evolution"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-9346-3206","authenticated-orcid":false,"given":"Mridul","family":"Khurana","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0009-0006-3319-1271","authenticated-orcid":false,"given":"Arka","family":"Daw","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0637-1753","authenticated-orcid":false,"given":"M.","family":"Maruf","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4624-9680","authenticated-orcid":false,"given":"Josef C.","family":"Uyeda","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3162-7490","authenticated-orcid":false,"given":"Wasila","family":"Dahdul","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9787-7081","authenticated-orcid":false,"given":"Caleb","family":"Charpentier","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6144-9440","authenticated-orcid":false,"given":"Yasin","family":"Bak\u0131\u015f","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5662-9444","authenticated-orcid":false,"suffix":"Jr.","given":"Henry L.","family":"Bart","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8455-3213","authenticated-orcid":false,"given":"Paula M.","family":"Mabee","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9107-0714","authenticated-orcid":false,"given":"Hilmar","family":"Lapp","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8688-6599","authenticated-orcid":false,"given":"James P.","family":"Balhoff","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1269-7231","authenticated-orcid":false,"given":"Wei-Lun","family":"Chao","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6532-6675","authenticated-orcid":false,"given":"Charles","family":"Stewart","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7610-1412","authenticated-orcid":false,"given":"Tanya","family":"Berger-Wolf","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1647-3534","authenticated-orcid":false,"given":"Anuj","family":"Karpatne","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,11,24]]},"reference":[{"key":"9_CR1","unstructured":"Great lakes invasives network project.https:\/\/greatlakesinvasives.org\/portal\/index.php"},{"key":"9_CR2","doi-asserted-by":"publisher","first-page":"e12581","DOI":"10.3897\/BDJ.5.e12581","volume":"5","author":"JA Rees","year":"2017","unstructured":"Rees, J.A., Cranston, K.: Automated assembly of a reference taxonomy for phylogenetic data synthesis. Biodivers. Data J. 5, e12581 (2017). https:\/\/doi.org\/10.3897\/BDJ.5.e12581","journal-title":"Biodivers. Data J."},{"key":"9_CR3","unstructured":"Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C., Su, J.K.: This looks like that: deep learning for interpretable image recognition. In: Advances in Neural Information Processing Systems, vol. 32 (2019)"},{"key":"9_CR4","unstructured":"Dhariwal, P., Nichol, A.: Diffusion models beat GANs on image synthesis. In: Advances in Neural Information Processing Systems, vol. 34, pp. 8780\u20138794 (2021)"},{"key":"9_CR5","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1093\/molbev\/msv223","volume":"33","author":"R Edmunds","year":"2015","unstructured":"Edmunds, R., et al.: Phenoscape: identifying candidate genes for evolutionary phenotypes. Mol. Biol. Evol. 33, 13\u201324 (2015). https:\/\/doi.org\/10.1093\/molbev\/msv223","journal-title":"Mol. Biol. Evol."},{"key":"9_CR6","doi-asserted-by":"crossref","unstructured":"Elhamod, M., et\u00a0al.: Discovering novel biological traits from images using phylogeny-guided neural networks. In: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3966\u20133978 (2023)","DOI":"10.1145\/3580305.3599808"},{"key":"9_CR7","unstructured":"Esser, P., et\u00a0al.: Scaling rectified flow transformers for high-resolution image synthesis. In: Forty-first International Conference on Machine Learning (2024)"},{"key":"9_CR8","doi-asserted-by":"crossref","unstructured":"Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 12873\u201312883 (2021)","DOI":"10.1109\/CVPR46437.2021.01268"},{"key":"9_CR9","unstructured":"Gharaee, Z., et\u00a0al.: A step towards worldwide biodiversity assessment: the BIOSCAN-1M insect dataset. In: Advances in Neural Information Processing Systems, vol. 36 (2024)"},{"key":"9_CR10","unstructured":"Griffiths, A.J.: An Introduction to Genetic Analysis. Macmillan (2005)"},{"key":"9_CR11","doi-asserted-by":"publisher","first-page":"2385","DOI":"10.1111\/j.1558-5646.2010.01025.x","volume":"64","author":"L Harmon","year":"2010","unstructured":"Harmon, L., et al.: Early bursts of body size and shape evolution are rare in comparative data. Evolution 64, 2385\u201396 (2010). https:\/\/doi.org\/10.1111\/j.1558-5646.2010.01025.x","journal-title":"Evolution"},{"key":"9_CR12","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770\u2013778 (2016)","DOI":"10.1109\/CVPR.2016.90"},{"key":"9_CR13","unstructured":"Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: Advances in Neural Information Processing Systems, vol. 33, pp. 6840\u20136851 (2020)"},{"issue":"7","key":"9_CR14","doi-asserted-by":"publisher","first-page":"428","DOI":"10.1038\/s41576-020-0233-0","volume":"21","author":"P Kapli","year":"2020","unstructured":"Kapli, P., Yang, Z., Telford, M.J.: Phylogenetic tree building in the genomic age. Nat. Rev. Genetics 21(7), 428\u2013444 (2020). https:\/\/doi.org\/10.1038\/s41576-020-0233-0","journal-title":"Nat. Rev. Genetics"},{"issue":"10","key":"9_CR15","doi-asserted-by":"publisher","first-page":"2318","DOI":"10.1109\/TKDE.2017.2720168","volume":"29","author":"A Karpatne","year":"2017","unstructured":"Karpatne, A., et al.: Theory-guided data science: a new paradigm for scientific discovery from data. IEEE Trans. Knowl. Data Eng. 29(10), 2318\u20132331 (2017)","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"9_CR16","unstructured":"Karpatne, A., Jia, X., Kumar, V.: Knowledge-guided machine learning: current trends and future prospects. arXiv preprint arXiv:2403.15989 (2024)"},{"key":"9_CR17","doi-asserted-by":"crossref","unstructured":"Karpatne, A., Kannan, R., Kumar, V.: Knowledge Guided Machine Learning: Accelerating Discovery using Scientific Knowledge and Data. CRC Press (2022)","DOI":"10.1201\/9781003143376"},{"key":"9_CR18","unstructured":"Kynk\u00e4\u00e4nniemi, T., Karras, T., Laine, S., Lehtinen, J., Aila, T.: Improved precision and recall metric for assessing generative models. In: Advances in Neural Information Processing Systems, vol. 32 (2019)"},{"key":"9_CR19","doi-asserted-by":"publisher","first-page":"642774","DOI":"10.3389\/fevo.2021.642774","volume":"9","author":"MD L\u00fcrig","year":"2021","unstructured":"L\u00fcrig, M.D., Donoughe, S., Svensson, E.I., Porto, A., Tsuboi, M.: Computer vision, machine learning, and the promise of phenomics in ecology and evolutionary biology. Front. Ecol. Evol. 9, 642774 (2021)","journal-title":"Front. Ecol. Evol."},{"key":"9_CR20","doi-asserted-by":"publisher","first-page":"561","DOI":"10.1002\/dvg.22878","volume":"53","author":"P Manda","year":"2015","unstructured":"Manda, P., Balhoff, J., Lapp, H., Mabee, P., Vision, T.: Using the phenoscape knowledgebase to relate genetic perturbations to phenotypic evolution. Genesis 53, 561\u2013571 (2015). https:\/\/doi.org\/10.1002\/dvg.22878","journal-title":"Genesis"},{"key":"9_CR21","doi-asserted-by":"crossref","unstructured":"Nauta, M., Schl\u00f6tterer, J., van Keulen, M., Seifert, C.: Pip-net: patch-based intuitive prototypes for interpretable image classification. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2744\u20132753 (2023)","DOI":"10.1109\/CVPR52729.2023.00269"},{"issue":"3","key":"9_CR22","doi-asserted-by":"publisher","first-page":"19","DOI":"10.32607\/20758251-2014-6-3-19-40","volume":"6","author":"AA Nemudryi","year":"2014","unstructured":"Nemudryi, A.A., Valetdinova, K.R., Medvedev, S.P., Zakian, S.M.: TALEN and CRISPR\/Cas genome editing systems: tools of discovery. Acta Naturae 6(3), 19\u201340 (2014)","journal-title":"Acta Naturae"},{"key":"9_CR23","unstructured":"Paul, D., et\u00a0al.: A simple interpretable transformer for fine-grained image classification and analysis. arXiv preprint arXiv:2311.04157 (2023)"},{"key":"9_CR24","doi-asserted-by":"publisher","first-page":"E33","DOI":"10.1086\/682022","volume":"186","author":"M Pennell","year":"2015","unstructured":"Pennell, M., FitzJohn, R., Cornwell, W., Harmon, L.: Model adequacy and the macroevolution of angiosperm functional traits. Am. Nat. 186, E33\u2013E50 (2015). https:\/\/doi.org\/10.1086\/682022","journal-title":"Am. Nat."},{"key":"9_CR25","unstructured":"Radford, A., et\u00a0al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, pp. 8748\u20138763. PMLR (2021)"},{"key":"9_CR26","doi-asserted-by":"crossref","unstructured":"Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684\u201310695 (2022)","DOI":"10.1109\/CVPR52688.2022.01042"},{"key":"9_CR27","doi-asserted-by":"crossref","unstructured":"Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K.: DreamBooth: fine tuning text-to-image diffusion models for subject-driven generation. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500\u201322510 (2023)","DOI":"10.1109\/CVPR52729.2023.02155"},{"issue":"2","key":"9_CR28","doi-asserted-by":"publisher","first-page":"198","DOI":"10.1111\/cla.12163","volume":"33","author":"TR Sim\u00f5es","year":"2017","unstructured":"Sim\u00f5es, T.R., Caldwell, M.W., Palci, A., Nydam, R.L.: Giant taxon-character matrices: quality of character constructions remains critical regardless of size. Cladistics 33(2), 198\u2013219 (2017)","journal-title":"Cladistics"},{"key":"9_CR29","unstructured":"Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)"},{"key":"9_CR30","doi-asserted-by":"crossref","unstructured":"Stevens, S., et\u00a0al.: Bioclip: a vision foundation model for the tree of life. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 19412\u201319424 (2024)","DOI":"10.1109\/CVPR52733.2024.01836"},{"key":"9_CR31","unstructured":"Van\u00a0Horn, G., Mac\u00a0Aodha, O.: iNat Challenge 2021 - FGVC8 (2021). https:\/\/kaggle.com\/competitions\/inaturalist-2021"},{"key":"9_CR32","unstructured":"Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: Cub-200-2011. Tech. Rep. CNS-TR-2011-001, California Institute of Technology (2011)"},{"issue":"2","key":"9_CR33","doi-asserted-by":"publisher","first-page":"179","DOI":"10.2307\/2413705","volume":"44","author":"ML Zelditch","year":"1995","unstructured":"Zelditch, M.L., Fink, W.L., Swiderski, D.L.: Morphometrics, homology, and phylogenetics: quantified characters as synapomorphies. Syst. Biol. 44(2), 179\u2013189 (1995)","journal-title":"Syst. Biol."},{"key":"9_CR34","doi-asserted-by":"crossref","unstructured":"Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: Proceedings of the IEEE\/CVF International Conference on Computer Vision, pp. 3836\u20133847 (2023)","DOI":"10.1109\/ICCV51070.2023.00355"}],"container-title":["Lecture Notes in Computer Science","Computer Vision \u2013 ECCV 2024"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-73024-5_9","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,25]],"date-time":"2024-11-25T17:09:46Z","timestamp":1732554586000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-73024-5_9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,24]]},"ISBN":["9783031730238","9783031730245"],"references-count":34,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-73024-5_9","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"type":"print","value":"0302-9743"},{"type":"electronic","value":"1611-3349"}],"subject":[],"published":{"date-parts":[[2024,11,24]]},"assertion":[{"value":"24 November 2024","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"ECCV","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"European Conference on Computer Vision","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Milan","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Italy","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2024","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"29 September 2024","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"4 October 2024","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"18","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"eccv2024","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/eccv2024.ecva.net\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}