{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T18:39:01Z","timestamp":1777919941522,"version":"3.51.4"},"reference-count":41,"publisher":"IOP Publishing","issue":"1","license":[{"start":{"date-parts":[[2024,2,21]],"date-time":"2024-02-21T00:00:00Z","timestamp":1708473600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,2,21]],"date-time":"2024-02-21T00:00:00Z","timestamp":1708473600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/100000084","name":"Directorate for Engineering","doi-asserted-by":"crossref","award":["1953222"],"award-info":[{"award-number":["1953222"]}],"id":[{"id":"10.13039\/100000084","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2024,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Solving partial differential equations (PDEs) is the core of many fields of science and engineering. While classical approaches are often prohibitively slow, machine learning models often fail to incorporate complete system information. Over the past few years, transformers have had a significant impact on the field of Artificial Intelligence and have seen increased usage in PDE applications. However, despite their success, transformers currently lack integration with physics and reasoning. This study aims to address this issue by introducing Physics Informed Token Transformer (PITT). The purpose of PITT is to incorporate the knowledge of physics by embedding PDEs into the learning process. PITT uses an equation tokenization method to learn an analytically-driven numerical update operator. By tokenizing PDEs and embedding partial derivatives, the transformer models become aware of the underlying knowledge behind physical processes. To demonstrate this, PITT is tested on challenging 1D and 2D PDE operator learning tasks. The results show that PITT outperforms popular neural operator models and has the ability to extract physically relevant information from governing equations.<\/jats:p>","DOI":"10.1088\/2632-2153\/ad27e3","type":"journal-article","created":{"date-parts":[[2024,2,9]],"date-time":"2024-02-09T17:21:50Z","timestamp":1707499310000},"page":"015032","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Physics informed token transformer for solving partial differential equations"],"prefix":"10.1088","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1410-9077","authenticated-orcid":true,"given":"Cooper","family":"Lorsung","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8566-7538","authenticated-orcid":false,"given":"Zijie","family":"Li","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2952-8576","authenticated-orcid":true,"given":"Amir","family":"Barati Farimani","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"266","published-online":{"date-parts":[[2024,2,21]]},"reference":[{"key":"mlstad27e3bib1","doi-asserted-by":"publisher","first-page":"521","DOI":"10.1007\/s11071-022-07207-x","article-title":"Bilinear residual network method for solving the exactly explicit solutions of nonlinear evolution equations","volume":"108","author":"Zhang","year":"2022","journal-title":"Nonlinear Dyn."},{"key":"mlstad27e3bib2","doi-asserted-by":"publisher","first-page":"3041","DOI":"10.1007\/s11071-018-04739-z","article-title":"Bilinear neural network method to obtain the exact analytical solutions of nonlinear partial differential equations and its application to p-gBKP equation","volume":"95","author":"Zhang","year":"2019","journal-title":"Nonlinear Dyn."},{"key":"mlstad27e3bib3","doi-asserted-by":"publisher","first-page":"1071","DOI":"10.1007\/s11071-020-06112-5","article-title":"Rogue wave solutions and the bright and dark solitons of the (3+1)-dimensional Jimbo\u2013Miwa equation","volume":"103","author":"Zhang","year":"2021","journal-title":"Nonlinear Dyn."},{"key":"mlstad27e3bib4","doi-asserted-by":"publisher","first-page":"122","DOI":"10.1007\/s11424-020-9392-5","article-title":"Fractal solitons, arbitrary function solutions, exact periodic wave and breathers for a nonlinear partial differential equation by using bilinear neural network method","volume":"34","author":"Zhang","year":"2021","journal-title":"J. Syst. Sci. Complex."},{"key":"mlstad27e3bib5","doi-asserted-by":"publisher","DOI":"10.1063\/5.0138039","article-title":"Mesh deep Q network: a deep reinforcement learning framework for improving meshes in computational fluid dynamics","volume":"13","author":"Lorsung","year":"2023","journal-title":"AIP Adv."},{"key":"mlstad27e3bib6","first-page":"pp 5997","article-title":"Reinforcement learning for adaptive mesh refinement","volume":"vol 206","author":"Yang","year":"2023"},{"key":"mlstad27e3bib7","article-title":"Deep reinforcement learning for adaptive mesh refinement","author":"Foucart","year":"2022"},{"key":"mlstad27e3bib8","article-title":"Learning controllable adaptive simulation for multi-resolution physics","author":"Wu","year":"2023"},{"key":"mlstad27e3bib9","doi-asserted-by":"publisher","DOI":"10.1063\/5.0054312","article-title":"Super-resolution and denoising of fluid flow using physics-informed convolutional neural networks without high-resolution labels","volume":"33","author":"Gao","year":"2021","journal-title":"Phys. Fluids"},{"key":"mlstad27e3bib10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3197517.3201304","article-title":"TempoGAN: a temporally coherent, volumetric GAN for super-resolution fluid flow","volume":"37","author":"Xie","year":"2018","journal-title":"ACM Trans. Graph."},{"key":"mlstad27e3bib11","doi-asserted-by":"publisher","DOI":"10.1063\/5.0062546","article-title":"Deep learning for reduced order modelling and efficient temporal evolution of fluid simulations","volume":"33","author":"Pant","year":"2021","journal-title":"Phys. Fluids"},{"key":"mlstad27e3bib12","doi-asserted-by":"publisher","DOI":"10.1016\/j.addlet.2023.100123","article-title":"Surrogate modeling of melt pool temperature field using deep learning","volume":"5","author":"Hemmasian","year":"2023","journal-title":"Addit. Manuf. Lett."},{"key":"mlstad27e3bib13","article-title":"Fourier neural operator for parametric partial differential equations","author":"Li","year":"2021"},{"key":"mlstad27e3bib14","doi-asserted-by":"publisher","first-page":"218","DOI":"10.1038\/s42256-021-00302-5","article-title":"Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators","volume":"3","author":"Lu","year":"2021","journal-title":"Nat. Mach. Intell."},{"key":"mlstad27e3bib15","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcp.2023.111972","article-title":"A physics-informed diffusion model for high-fidelity flow field reconstruction","volume":"478","author":"Shu","year":"2023","journal-title":"J. Comput. Phys."},{"key":"mlstad27e3bib16","first-page":"97","article-title":"Neural operator: learning maps between function spaces with applications to PDEs","volume":"24","author":"Kovachki","year":"2023","journal-title":"J. Mach. Learn. Res."},{"key":"mlstad27e3bib17","article-title":"Neural machine translation by jointly learning to align and translate","author":"Bahdanau","year":"2016"},{"key":"mlstad27e3bib18","article-title":"Attention is all you need","volume":"vol 30","author":"Vaswani","year":"2017"},{"key":"mlstad27e3bib19","article-title":"BERT: pre-training of deep bidirectional transformers for language understanding","author":"Devlin","year":"2019"},{"key":"mlstad27e3bib20","article-title":"Language models are few-shot learners","author":"Brown","year":"2020"},{"key":"mlstad27e3bib21","article-title":"An image is worth 16\u00d716 words: transformers for image recognition at scale","author":"Dosovitskiy","year":"2021"},{"key":"mlstad27e3bib22","doi-asserted-by":"publisher","first-page":"583","DOI":"10.1038\/s41586-021-03819-2","article-title":"Highly accurate protein structure prediction with AlphaFold","volume":"596","author":"Jumper","year":"2021","journal-title":"Nature"},{"key":"mlstad27e3bib23","first-page":"pp 24924","article-title":"Choose a transformer: Fourier or Galerkin","author":"Cao","year":"2021"},{"key":"mlstad27e3bib24","article-title":"Transformer for partial differential equations\u2019 operator learning","author":"Li","year":"2022"},{"key":"mlstad27e3bib25","author":"Hao","year":"2023"},{"key":"mlstad27e3bib26","article-title":"Predicting physics in mesh-reduced space with temporal attention","author":"Han","year":"2022"},{"key":"mlstad27e3bib27","article-title":"ViTO: vision transformer-operator","author":"Ovadia","year":"2023"},{"key":"mlstad27e3bib28","doi-asserted-by":"publisher","first-page":"272","DOI":"10.1016\/j.neunet.2021.11.022","article-title":"Transformers for modeling physical systems","volume":"146","author":"Geneva","year":"2022","journal-title":"Neural Netw."},{"key":"mlstad27e3bib29","article-title":"Transformer meets boundary value inverse problems","author":"Guo","year":"2023"},{"key":"mlstad27e3bib30","doi-asserted-by":"publisher","DOI":"10.1063\/5.0151515","article-title":"Reduced-order modeling of fluid flows with transformers","volume":"35","author":"Hemmasian","year":"2023","journal-title":"Phys. Fluids"},{"key":"mlstad27e3bib31","article-title":"RoFormer: enhanced transformer with rotary position embedding","author":"Su","year":"2022"},{"key":"mlstad27e3bib32","article-title":"Learning operators with coupled attention","author":"Kissas","year":"2022"},{"key":"mlstad27e3bib33","article-title":"Meta-learning dynamics forecasting using task inference","author":"Wang","year":"2022"},{"key":"mlstad27e3bib34","article-title":"CAPE: channel-attention-based PDE parameter embeddings for SciML","author":"Takamoto","year":"2023"},{"key":"mlstad27e3bib35","doi-asserted-by":"publisher","first-page":"686","DOI":"10.1016\/j.jcp.2018.10.045","article-title":"Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations","volume":"378","author":"Raissi","year":"2019","journal-title":"J. Comput. Phys."},{"key":"mlstad27e3bib36","article-title":"Characterizing possible failure modes in physics-informed neural networks","volume":"vol 34","author":"Krishnapriyan","year":"2021"},{"key":"mlstad27e3bib37","article-title":"Deep learning for symbolic mathematics","author":"Lample","year":"2020"},{"key":"mlstad27e3bib38","article-title":"PDEBench: an extensive benchmark for scientific machine learning","author":"Takamoto","year":"2022"},{"key":"mlstad27e3bib39","article-title":"Message passing neural PDE solvers","author":"Brandstetter","year":"2022"},{"key":"mlstad27e3bib40","doi-asserted-by":"publisher","first-page":"2365","DOI":"10.3390\/electronics11152365","article-title":"Numerical solution of the Poisson equation using finite difference matrix operators","volume":"11","author":"Zaman","year":"2022","journal-title":"Electronics"},{"key":"mlstad27e3bib41","author":"Lorsung","year":"2023"}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad27e3","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad27e3\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad27e3","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad27e3\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad27e3\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad27e3\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad27e3\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad27e3\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,2,21]],"date-time":"2024-02-21T06:38:00Z","timestamp":1708497480000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad27e3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,21]]},"references-count":41,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,2,21]]},"published-print":{"date-parts":[[2024,3,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/ad27e3","relation":{"has-review":[{"id-type":"doi","id":"10.1088\/2632-2153\/AD27E3\/v3\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/AD27E3\/v1\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/AD27E3\/v1\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/AD27E3\/v1\/review3","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/AD27E3\/v2\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/AD27E3\/v3\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/AD27E3\/v3\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/AD27E3\/v2\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/AD27E3\/v2\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/AD27E3\/v1\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/AD27E3\/v2\/response1","asserted-by":"object"}]},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,21]]},"assertion":[{"value":"Physics informed token transformer for solving partial differential equations","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2024 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2023-09-18","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-02-09","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-02-21","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}