{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T22:03:28Z","timestamp":1784153008697,"version":"3.55.0"},"reference-count":58,"publisher":"IOP Publishing","issue":"3","license":[{"start":{"date-parts":[[2024,9,5]],"date-time":"2024-09-05T00:00:00Z","timestamp":1725494400000},"content-version":"vor","delay-in-days":4,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2024,9,5]],"date-time":"2024-09-05T00:00:00Z","timestamp":1725494400000},"content-version":"tdm","delay-in-days":4,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62001289"],"award-info":[{"award-number":["62001289"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"NUPTSF","award":["NY220119"],"award-info":[{"award-number":["NY220119"]}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2024,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>The concept of photonic modes is the cornerstone in optics and photonics, which can describe the propagation of the light. The Maxwell\u2019s equations play the role in calculating the mode field based on the structure information, while this process needs a great deal of computations, especially in the handle with a three-dimensional model. To overcome this obstacle, we introduce the multi-modal diffusion model to predict the photonic modes in one certain structure. The Contrastive Language\u2013Image Pre-training (CLIP) model is used to build the connections between photonic structures and the corresponding modes. Then we exemplify Stable Diffusion (SD) model to realize the function of optical fields generation from structure information. Our work introduces multi-modal deep learning to construct complex mapping between structural information and optical field as high-dimensional vectors, and generates optical field images based on this mapping.<\/jats:p>","DOI":"10.1088\/2632-2153\/ad743f","type":"journal-article","created":{"date-parts":[[2024,8,27]],"date-time":"2024-08-27T23:00:57Z","timestamp":1724799657000},"page":"035069","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Photonic modes prediction via multi-modal diffusion model"],"prefix":"10.1088","volume":"5","author":[{"given":"Jinyang","family":"Sun","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xi","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiumei","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dandan","family":"Zhu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1763-7070","authenticated-orcid":true,"given":"Xingping","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2024,9,5]]},"reference":[{"key":"mlstad743fbib1","doi-asserted-by":"publisher","DOI":"10.1103\/RevModPhys.92.035005","article-title":"Modes and states in quantum optics","volume":"92","author":"Fabre","year":"2020","journal-title":"Rev. Mod. Phys."},{"key":"mlstad743fbib2","doi-asserted-by":"publisher","first-page":"679","DOI":"10.1364\/AOP.11.000679","article-title":"Waves, modes, communications, and optics: a tutorial","volume":"11","author":"Miller","year":"2019","journal-title":"Adv. Opt. Photon."},{"key":"mlstad743fbib3","doi-asserted-by":"publisher","author":"Joannopoulos","year":"2008","DOI":"10.1515\/9781400828241\/html)"},{"key":"mlstad743fbib4","doi-asserted-by":"publisher","DOI":"10.1364\/OE.18.013250","article-title":"Long distance transmission in few-mode fibers","volume":"18","author":"Yaman","year":"2010","journal-title":"Opt. Express"},{"key":"mlstad743fbib5","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-019-53530-6","article-title":"Space division multiplexing in standard multi-mode optical fibers based on speckle pattern classification","volume":"9","author":"Pauwels","year":"2019","journal-title":"Sci. Rep."},{"key":"mlstad743fbib6","doi-asserted-by":"publisher","first-page":"273","DOI":"10.1038\/s41566-019-0532-1","article-title":"Integrated photonic quantum technologies","volume":"14","author":"Wang","year":"2020","journal-title":"Nat. Photon."},{"key":"mlstad743fbib7","doi-asserted-by":"publisher","first-page":"285","DOI":"10.1038\/s41566-020-0609-x","article-title":"Hybrid integrated quantum photonic circuits","volume":"14","author":"Elshaari","year":"2020","journal-title":"Nat. Photon."},{"key":"mlstad743fbib8","doi-asserted-by":"publisher","first-page":"1783","DOI":"10.1038\/s41598-018-38183-1","article-title":"Ultra-short pulse generation in a three section tapered passively mode-locked quantum-dot semiconductor laser","volume":"9","author":"Meinecke","year":"2019","journal-title":"Sci. Rep."},{"key":"mlstad743fbib9","doi-asserted-by":"publisher","first-page":"341","DOI":"10.1103\/RevModPhys.51.341","article-title":"Propagation in glass optical waveguides","volume":"51","author":"Olshansky","year":"1979","journal-title":"Rev. Mod. Phys."},{"key":"mlstad743fbib10","doi-asserted-by":"publisher","DOI":"10.1016\/j.physletb.2023.137904","article-title":"Rapid search for massive black hole binary coalescences using deep learning","volume":"841","author":"Ruan","year":"2023","journal-title":"Phys. Rev. B"},{"key":"mlstad743fbib11","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1038\/nature23463","article-title":"Fast automated analysis of strong gravitational lenses with convolutional neural networks","volume":"548","author":"Hezaveh","year":"2017","journal-title":"Nature"},{"key":"mlstad743fbib12","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1038\/s41566-020-0685-y","article-title":"Deep learning for the design of photonic structures","volume":"15","author":"Ma","year":"2021","journal-title":"Nat. Photon."},{"key":"mlstad743fbib13","doi-asserted-by":"publisher","DOI":"10.1103\/RevModPhys.91.045002","article-title":"Machine learning and the physical sciences","volume":"91","author":"Carleo","year":"2019","journal-title":"Rev. Mod. Phys."},{"key":"mlstad743fbib14","doi-asserted-by":"publisher","first-page":"eaar4206","DOI":"10.1126\/sciadv.aar4206","article-title":"Nanophotonic particle simulation and inverse design using artificial neural networks","volume":"4","author":"Peurifoy","year":"2018","journal-title":"Sci. Adv."},{"key":"mlstad743fbib15","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1021\/acs.nanolett.9b03971","article-title":"Deep learning meets nanophotonics: a generalized accurate predictor for near fields and far fields of arbitrary 3D nanostructures","volume":"20","author":"Wiecha","year":"2019","journal-title":"Nano Lett."},{"key":"mlstad743fbib16","doi-asserted-by":"publisher","DOI":"10.1021\/acsami.9b05857","article-title":"Simultaneous inverse design of materials and structures via deep learning: demonstration of dipole resonance engineering using core\u2013shell nanoparticles","volume":"11","author":"So","year":"2019","journal-title":"ACS Appl. Mater. Interfaces"},{"key":"mlstad743fbib17","doi-asserted-by":"publisher","first-page":"6326","DOI":"10.1021\/acsnano.8b03569","article-title":"Deep-learning-enabled on-demand design of chiral metamaterials","volume":"12","author":"Ma","year":"2018","journal-title":"ACS Nano"},{"key":"mlstad743fbib18","doi-asserted-by":"publisher","DOI":"10.1364\/OE.27.027523","article-title":"Deep learning for accelerated all-dielectric metasurface design","volume":"27","author":"Nadell","year":"2019","journal-title":"Opt. Express"},{"key":"mlstad743fbib19","doi-asserted-by":"publisher","first-page":"481","DOI":"10.1021\/acsphotonics.0c01468","article-title":"Deep learning to accelerate scatterer-to-field mapping for inverse design of dielectric metasurfaces","volume":"8","author":"Zhelyeznyakov","year":"2021","journal-title":"ACS Photonics"},{"key":"mlstad743fbib20","doi-asserted-by":"publisher","DOI":"10.1002\/adma.202110022","article-title":"Pushing the limits of functionality\u2010multiplexing capability in metasurface design based on statistical machine learning","volume":"34","author":"Ma","year":"2022","journal-title":"Adv. Mater."},{"key":"mlstad743fbib21","doi-asserted-by":"publisher","DOI":"10.1002\/adma.202005864","article-title":"Realizing colorful holographic mimicry by metasurfaces","volume":"33","author":"Xiong","year":"2021","journal-title":"Adv. Mater."},{"key":"mlstad743fbib22","doi-asserted-by":"publisher","first-page":"1059","DOI":"10.1515\/nanoph-2019-0330","article-title":"Simulator-based training of generative neural networks for the inverse design of metasurfaces","volume":"9","author":"Jiang","year":"2019","journal-title":"Nanophotonics"},{"key":"mlstad743fbib23","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.123.213902","article-title":"Self-learning perfect optical chirality via a deep neural network","volume":"123","author":"Li","year":"2019","journal-title":"Phys. Rev. Lett."},{"key":"mlstad743fbib24","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1364\/OPTICA.449893","article-title":"Deep-learning-based colorimetric polarization-angle detection with metasurfaces","volume":"9","author":"Yang","year":"2022","journal-title":"Optica"},{"key":"mlstad743fbib25","doi-asserted-by":"publisher","DOI":"10.1364\/OE.27.036414","article-title":"Machine learning approach for computing optical properties of a photonic crystal fiber","volume":"27","author":"Chugh","year":"2019","journal-title":"Opt. Express"},{"key":"mlstad743fbib26","doi-asserted-by":"publisher","first-page":"7150","DOI":"10.1109\/JLT.2022.3199764","article-title":"Photonic crystal nanobeam cavity with a high experimental Q factor exceeding two million based on machine learning","volume":"40","author":"Liu","year":"2022","journal-title":"J. Light Technol."},{"key":"mlstad743fbib27","doi-asserted-by":"publisher","DOI":"10.1063\/5.0040793","article-title":"Fabrication and characterization of an L3 nanocavity designed by an iterative machine-learning method","volume":"6","author":"Shibata","year":"2021","journal-title":"APL Photonics"},{"key":"mlstad743fbib28","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1364\/OL.381616","article-title":"Optimization of an H0 photonic crystal nanocavity using machine learning","volume":"45","author":"Abe","year":"2020","journal-title":"Opt. Lett."},{"key":"mlstad743fbib29","doi-asserted-by":"publisher","DOI":"10.1364\/OE.26.032704","article-title":"Optimization of photonic crystal nanocavities based on deep learning","volume":"26","author":"Asano","year":"2018","journal-title":"Opt. Express"},{"key":"mlstad743fbib30","doi-asserted-by":"publisher","first-page":"4775","DOI":"10.1038\/s41467-019-12698-1","article-title":"Mapping the global design space of nanophotonic components using machine learning pattern recognition","volume":"10","author":"Melati","year":"2019","journal-title":"Nat. Commun."},{"key":"mlstad743fbib31","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1038\/s41566-020-00716-4","article-title":"Machine learning and applications in ultrafast photonics","volume":"15","author":"Genty","year":"2021","journal-title":"Nat. Photon."},{"key":"mlstad743fbib32","doi-asserted-by":"publisher","first-page":"4660","DOI":"10.1364\/OL.470826","article-title":"Tunable grating surfaces with high diffractive efficiency optimized by deep neural networks","volume":"47","author":"Qian","year":"2022","journal-title":"Opt. Lett."},{"key":"mlstad743fbib33","doi-asserted-by":"publisher","first-page":"1368","DOI":"10.1038\/s41598-018-37952-2","article-title":"Deep neural network inverse design of integrated photonic power splitters","volume":"9","author":"Tahersima","year":"2019","journal-title":"Sci. Rep."},{"key":"mlstad743fbib34","doi-asserted-by":"publisher","DOI":"10.1002\/lpor.202000287","article-title":"Generative deep learning model for inverse design of integrated nanophotonic devices","volume":"14","author":"Tang","year":"2020","journal-title":"Laser Photon. Rev."},{"key":"mlstad743fbib35","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.120.066401","article-title":"Machine learning topological invariants with neural networks","volume":"120","author":"Zhang","year":"2018","journal-title":"Phys. Rev. Lett."},{"key":"mlstad743fbib36","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.130.036601","article-title":"Unsupervised data-driven classification of topological gapped systems with symmetries","volume":"130","author":"Long","year":"2023","journal-title":"Phys. Rev. Lett."},{"key":"mlstad743fbib37","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.124.226401","article-title":"Unsupervised machine learning and band topology","volume":"124","author":"Scheurer","year":"2020","journal-title":"Phys. Rev. Lett."},{"key":"mlstad743fbib38","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.124.185501","article-title":"Unsupervised manifold clustering of topological phononics","volume":"124","author":"Long","year":"2020","journal-title":"Phys. Rev. Lett."},{"key":"mlstad743fbib39","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevB.102.054107","article-title":"Machine learning topological phases in real space","volume":"102","author":"Holanda","year":"2020","journal-title":"Phys. Rev. B"},{"key":"mlstad743fbib40","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevB.103.035413","article-title":"Machine learning non-Hermitian topological phases","volume":"103","author":"Narayan","year":"2021","journal-title":"Phys. Rev. B"},{"key":"mlstad743fbib41","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.126.240402","article-title":"Unsupervised learning of non-Hermitian topological phases","volume":"126","author":"Yu","year":"2021","journal-title":"Phys. Rev. Lett."},{"key":"mlstad743fbib42","doi-asserted-by":"publisher","DOI":"10.5555\/3104482.3104569","article-title":"Multimodal deep learning","author":"Ngiam","year":"2011"},{"key":"mlstad743fbib43","article-title":"Multimodal foundation models: from specialists to general-purpose assistants","author":"Li","year":"2023"},{"key":"mlstad743fbib44","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1109\/TPAMI.2018.2798607","article-title":"Multimodal machine learning: a survey and taxonomy","volume":"41","author":"Baltru\u0161aitis","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"mlstad743fbib45","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR52688.2022.01042","article-title":"High-resolution image synthesis with latent diffusion models","author":"Rombach","year":"2022"},{"key":"mlstad743fbib46","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1109\/JPROC.2020.3004555","article-title":"A comprehensive survey on transfer learning","volume":"109","author":"Zhuang","year":"2020","journal-title":"Proc. IEEE"},{"key":"mlstad743fbib47","article-title":"Learning transferable visual models from natural language supervision","volume":"vol 8748","author":"Radford","year":"2021"},{"key":"mlstad743fbib48","doi-asserted-by":"publisher","DOI":"10.5555\/3295222.3295349","article-title":"Attention is all you need","volume":"vol 30","author":"Vaswani","year":"2017"},{"key":"mlstad743fbib49","doi-asserted-by":"publisher","first-page":"2486","DOI":"10.1103\/PhysRevLett.58.2486","article-title":"Strong localization of photons in certain disordered dielectric superlattices","volume":"58","author":"John","year":"1987","journal-title":"Phys. Rev. Lett."},{"key":"mlstad743fbib50","doi-asserted-by":"publisher","first-page":"2059","DOI":"10.1103\/PhysRevLett.58.2059","article-title":"Inhibited spontaneous emission in solid-state physics and electronics","volume":"58","author":"Yablonovitch","year":"1987","journal-title":"Phys. Rev. Lett."},{"key":"mlstad743fbib51","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1364\/OE.8.000173","article-title":"Block-iterative frequency-domain methods for Maxwell\u2019s equations in a planewave basis","volume":"8","author":"Johnson","year":"2001","journal-title":"Opt. Express"},{"key":"mlstad743fbib52","first-page":"12","article-title":"Average hashing for perceptual image similarity in mobile phone application","volume":"4","author":"Haviana","year":"2016","journal-title":"J. Telemat. Inform."},{"key":"mlstad743fbib53","article-title":"Conditional frechet inception distance","author":"Soloveitchik","year":"2021"},{"key":"mlstad743fbib54","first-page":"p 1125","article-title":"Image-to-image translation with conditional adversarial networks","author":"Isola","year":"2017"},{"key":"mlstad743fbib55","article-title":"Any-to-any generation via composable diffusion","author":"Tang","year":"2023"},{"key":"mlstad743fbib56","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1038\/s42005-018-0060-1","article-title":"Machine learning classification for field distributions of photonic modes","volume":"1","author":"Barth","year":"2018","journal-title":"Commun. Phys."},{"key":"mlstad743fbib57","doi-asserted-by":"publisher","DOI":"10.1039\/C9NR03450A","article-title":"Plasmonic nanoparticle simulations and inverse design using machine learning","volume":"11","author":"He","year":"2019","journal-title":"Nanoscale"},{"key":"mlstad743fbib58","doi-asserted-by":"publisher","first-page":"1255","DOI":"10.1515\/nanoph-2019-0117","article-title":"Designing nanophotonic structures using conditional deep convolutional generative adversarial networks","volume":"8","author":"So","year":"2019","journal-title":"Nanophotonics"}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad743f","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad743f\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad743f","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad743f\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad743f\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad743f\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad743f\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad743f\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,5]],"date-time":"2024-09-05T08:42:30Z","timestamp":1725525750000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad743f"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,1]]},"references-count":58,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,9,5]]},"published-print":{"date-parts":[[2024,9,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/ad743f","relation":{},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,1]]},"assertion":[{"value":"Photonic modes prediction via multi-modal diffusion model","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2024 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2024-05-08","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-08-27","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-09-05","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}