{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,6]],"date-time":"2025-11-06T09:55:00Z","timestamp":1762422900304,"version":"build-2065373602"},"reference-count":47,"publisher":"IOP Publishing","issue":"4","license":[{"start":{"date-parts":[[2025,11,6]],"date-time":"2025-11-06T00:00:00Z","timestamp":1762387200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2025,11,6]],"date-time":"2025-11-06T00:00:00Z","timestamp":1762387200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/100030719","name":"National Institute of Health Sciences","doi-asserted-by":"crossref","award":["1R01GM157589-01"],"award-info":[{"award-number":["1R01GM157589-01"]}],"id":[{"id":"10.13039\/100030719","id-type":"DOI","asserted-by":"crossref"}]},{"name":"NSF MADE-PUBLIC Future Manufacturing Research Grant Program","award":["CMMI-2037026"],"award-info":[{"award-number":["CMMI-2037026"]}]},{"name":"Lehigh University\u2019s Research Computing infrastructure","award":["NSF Award 2019035"],"award-info":[{"award-number":["NSF Award 2019035"]}]},{"name":"NSF HAMMER-ERC","award":["EEC-2133630"],"award-info":[{"award-number":["EEC-2133630"]}]},{"DOI":"10.13039\/100000181","name":"Air Force Office of Scientific Research","doi-asserted-by":"crossref","award":["FA9550-22-1-0197"],"award-info":[{"award-number":["FA9550-22-1-0197"]}],"id":[{"id":"10.13039\/100000181","id-type":"DOI","asserted-by":"crossref"}]},{"name":"NSF Boosting Research Ideas for Transformative and Equitable Advances in Engineering (BRITE) Fellow Program","award":["CMMI 2227641"],"award-info":[{"award-number":["CMMI 2227641"]}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2025,12,30]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>In scientific machine learning (SciML), a key challenge is learning unknown, evolving physical processes and making predictions across spatio-temporal scales. For example, in real-world manufacturing problems like additive manufacturing, users adjust known machine settings while unknown environmental parameters simultaneously fluctuate. To make reliable predictions, it is desired for a model to not only capture long-range spatio-temporal interactions from data but also adapt to new and unknown environments; traditional machine learning models excel at the first task but often lack physical interpretability and struggle to generalize under varying environmental conditions. To tackle these challenges, we propose the attention-based spatio-temporal neural operator (ASNO), a novel architecture that combines separable attention mechanisms for spatial and temporal interactions and adapts to unseen physical parameters. Inspired by the backward differentiation formula, ASNO learns a transformer for temporal prediction and extrapolation and an attention-based neural operator for handling varying external loads, enhancing interpretability by isolating historical state contributions and external forces, enabling the discovery of underlying physical laws and generalizability to unseen physical environments. Empirical results on SciML benchmarks demonstrate that ASNO outperforms existing models, establishing its potential for engineering applications, physics discovery, and interpretable machine learning.<\/jats:p>","DOI":"10.1088\/2632-2153\/ae1277","type":"journal-article","created":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T22:52:17Z","timestamp":1760395937000},"page":"045036","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["An attention-based spatio-temporal neural operator for evolving physics"],"prefix":"10.1088","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3733-2415","authenticated-orcid":true,"given":"Vispi","family":"Karkaria","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4597-6643","authenticated-orcid":true,"given":"Doksoo","family":"Lee","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yi-Ping","family":"Chen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9150-3986","authenticated-orcid":true,"given":"Yue","family":"Yu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wei","family":"Chen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"266","published-online":{"date-parts":[[2025,11,6]]},"reference":[{"key":"mlstae1277bib1","doi-asserted-by":"publisher","first-page":"797","DOI":"10.1137\/0732037","type":"journal-article","article-title":"Implicit-explicit methods for time-dependent partial differential equations","volume":"32","author":"Ascher","year":"1995","journal-title":"SIAM J. Numer. Anal."},{"key":"mlstae1277bib2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s42254-024-00712-5","type":"journal-article","article-title":"Neural operators for accelerating scientific simulations and design","volume":"6","author":"Azizzadenesheli","year":"2024","journal-title":"Nat. Rev. Phys."},{"key":"mlstae1277bib3","doi-asserted-by":"publisher","first-page":"631","DOI":"10.1038\/s42256-024-00844-4","type":"journal-article","article-title":"Laplace neural operator for solving differential equations","volume":"6","author":"Cao","year":"2024","journal-title":"Nat. Mach. Intell."},{"key":"mlstae1277bib4","first-page":"pp 24924","type":"conference-proceedings","article-title":"Choose a transformer: Fourier or galerkin","volume":"vol 34","author":"Cao","year":"2021"},{"key":"mlstae1277bib5","doi-asserted-by":"crossref","DOI":"10.1016\/j.jmsy.2025.03.009","type":"preprint","article-title":"Real-time decision-making for digital twin in additive manufacturing with model predictive control using time-series deep neural networks","author":"Chen","year":"2025"},{"article-title":"Mamba neural operator: who wins? transformers vs. state-space models for PDES","year":"2024","author":"Cheng","key":"mlstae1277bib6","type":"preprint"},{"key":"mlstae1277bib7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3586074","type":"journal-article","article-title":"A practical survey on faster and lighter transformers","volume":"55","author":"Fournier","year":"2023","journal-title":"ACM Comput. Surv."},{"key":"mlstae1277bib8","doi-asserted-by":"publisher","first-page":"1917","DOI":"10.1137\/S0036142996306217","type":"journal-article","article-title":"A-bdf: a generalization of the backward differentiation formulae","volume":"35","author":"Fredebeul","year":"1998","journal-title":"SIAM J. Numer. Anal."},{"key":"mlstae1277bib9","first-page":"pp 37","type":"book","article-title":"Long short-term memory","author":"Graves","year":"2012"},{"key":"mlstae1277bib10","doi-asserted-by":"publisher","DOI":"10.1016\/j.addma.2024.104013","type":"journal-article","article-title":"Machine learning-assisted in-situ adaptive strategies for the control of defects and anomalies in metal additive manufacturing","volume":"81","author":"Gunasegaram","year":"2024","journal-title":"Add. Manuf."},{"key":"mlstae1277bib11","first-page":"pp 12556","type":"conference-proceedings","article-title":"GNOT: a general neural operator transformer for operator learning","author":"Hao","year":"2023"},{"key":"mlstae1277bib12","doi-asserted-by":"publisher","first-page":"641","DOI":"10.1090\/mcom\/3602","type":"journal-article","article-title":"On the uniform accuracy of implicit-explicit backward differentiation formulas (imex-bdf) for stiff hyperbolic relaxation systems and kinetic equations","volume":"90","author":"Hu","year":"2021","journal-title":"Math. Comput."},{"key":"mlstae1277bib13","doi-asserted-by":"publisher","first-page":"337","DOI":"10.1038\/s43588-021-00069-0","type":"journal-article","article-title":"A probabilistic graphical model foundation for enabling predictive digital twins at scale","volume":"1","author":"Kapteyn","year":"2021","journal-title":"Nat. Comput. Sci."},{"key":"mlstae1277bib14","doi-asserted-by":"publisher","first-page":"322","DOI":"10.1016\/j.jmsy.2024.04.023","type":"journal-article","article-title":"Towards a digital twin framework in additive manufacturing: machine learning and bayesian optimization for time series process optimization","volume":"75","author":"Karkaria","year":"2024a","journal-title":"J. Manuf. Syst."},{"key":"mlstae1277bib15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/0305215X.2024.2434201","type":"journal-article","article-title":"An optimization-centric review on integrating artificial intelligence and digital twin technologies in manufacturing","volume":"57","author":"Karkaria","year":"2024b","journal-title":"Eng. Optim."},{"key":"mlstae1277bib16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s42493-024-00106-w","type":"journal-article","article-title":"A review of physics informed neural networks for multiscale analysis and inverse problems","volume":"6","author":"Kim","year":"2024","journal-title":"Multiscale Sci. Eng."},{"article-title":"Reformer: the efficient transformer","year":"2020","author":"Kitaev","key":"mlstae1277bib17","type":"preprint"},{"key":"mlstae1277bib18","first-page":"p V002T02A019","type":"conference-proceedings","article-title":"Spatial-temporal modeling using deep learning for real-time monitoring of additive manufacturing","volume":"vol 86212","author":"Ko","year":"2022"},{"key":"mlstae1277bib19","first-page":"1","type":"journal-article","article-title":"Neural operator: learning maps between function spaces with applications to PDES","volume":"24","author":"Kovachki","year":"2023","journal-title":"J. Mach.: Learn. Res."},{"article-title":"Fourier neural operator for parametric partial differential equations","year":"2020a","author":"Li","key":"mlstae1277bib20","type":"preprint"},{"key":"mlstae1277bib21","first-page":"pp 6755","type":"conference-proceedings","article-title":"Multipole graph neural operator for parametric partial differential equations","volume":"vol 33","author":"Li","year":"2020b"},{"key":"mlstae1277bib22","doi-asserted-by":"publisher","first-page":"879","DOI":"10.1007\/s00466-023-02273-3","type":"journal-article","article-title":"Efficient gpu-accelerated thermomechanical solver for residual stress prediction in additive manufacturing","volume":"71","author":"Liao","year":"2023","journal-title":"Comput. Mech."},{"key":"mlstae1277bib23","doi-asserted-by":"publisher","first-page":"1748","DOI":"10.1016\/j.ijforecast.2021.03.012","type":"journal-article","article-title":"Temporal fusion transformers for interpretable multi-horizon time series forecasting","volume":"37","author":"Lim","year":"2021","journal-title":"Int. J. Forecast."},{"article-title":"HT-NET: hierarchical transformer based operator learning model for multiscale PDES","year":"2022","author":"Liu","key":"mlstae1277bib24","type":"other"},{"article-title":"Transformer learns the cross-task prior and regularization for in-context learning","year":"2025","author":"Lu","key":"mlstae1277bib25","type":"preprint"},{"article-title":"Deeponet: learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators","year":"2019","author":"Lu","key":"mlstae1277bib26","type":"preprint"},{"key":"mlstae1277bib27","doi-asserted-by":"publisher","first-page":"218","DOI":"10.1038\/s42256-021-00302-5","type":"journal-article","article-title":"Learning nonlinear operators via deeponet based on the universal approximation theorem of operators","volume":"3","author":"Lu","year":"2021","journal-title":"Nat. Mach. Intell."},{"article-title":"Separable deeponet: breaking the curse of dimensionality in physics-informed machine learning","year":"2024","author":"Mandl","key":"mlstae1277bib28","type":"preprint"},{"article-title":"Neural inverse operators for solving PDE inverse problems","year":"2023","author":"Molinaro","key":"mlstae1277bib29","type":"preprint"},{"year":"2020","author":"Molnar","key":"mlstae1277bib30","type":"book"},{"key":"mlstae1277bib31","doi-asserted-by":"publisher","first-page":"739","DOI":"10.1111\/j.1553-2712.1998.tb02493.x","type":"journal-article","article-title":"Time series analysis using autoregressive integrated moving average (arima) models","volume":"5","author":"Nelson","year":"1998","journal-title":"Acad. Emer. Med."},{"key":"mlstae1277bib32","doi-asserted-by":"publisher","first-page":"48","DOI":"10.1016\/j.neucom.2021.03.091","type":"journal-article","article-title":"A review on the attention mechanism of deep learning","volume":"452","author":"Niu","year":"2021","journal-title":"Neurocomputing"},{"key":"mlstae1277bib33","doi-asserted-by":"publisher","first-page":"15084","DOI":"10.1109\/ACCESS.2022.3148401","type":"journal-article","article-title":"SSNO: spatio-spectral neural operator for functional space learning of partial differential equations","volume":"10","author":"Rafiq","year":"2022","journal-title":"IEEE Access"},{"key":"mlstae1277bib34","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1214\/21-SS133","type":"journal-article","article-title":"Interpretable machine learning: fundamental principles and 10 grand challenges","volume":"16","author":"Rudin","year":"2022","journal-title":"Stat. Surv."},{"key":"mlstae1277bib35","first-page":"pp 23592","type":"conference-proceedings","article-title":"Probabilistic transformer for time series analysis","volume":"vol 34","author":"Tang","year":"2021"},{"key":"mlstae1277bib36","doi-asserted-by":"publisher","first-page":"354","DOI":"10.1007\/s00158-022-03425-4","type":"journal-article","article-title":"A comprehensive review of digital twin-part 1: modeling and twinning enabling technologies","volume":"65","author":"Thelen","year":"2022","journal-title":"Struct. Multidiscip. Optim."},{"key":"mlstae1277bib37","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1007\/s00158-023-03488-x","type":"journal-article","article-title":"Digital twins for the designs of systems: a perspective","volume":"66","author":"van Beek","year":"2023","journal-title":"Struct. Multidiscip. Optim."},{"key":"mlstae1277bib38","first-page":"p 30","type":"conference-proceedings","article-title":"Attention is all you need","author":"Vaswani","year":"2017"},{"article-title":"P2c2net: PDE-preserved coarse correction network for efficient prediction of spatiotemporal dynamics","year":"2024","author":"Wang","key":"mlstae1277bib39","type":"preprint"},{"key":"mlstae1277bib40","doi-asserted-by":"publisher","DOI":"10.1016\/j.advwatres.2022.104180","type":"journal-article","article-title":"U-fno-an enhanced fourier neural operator-based deep-learning model for multiphase flow","volume":"163","author":"Wen","year":"2022","journal-title":"Adv. Water Resour."},{"article-title":"Transolver: a fast transformer solver for PDES on general geometries","year":"2024","author":"Wu","key":"mlstae1277bib41","type":"preprint"},{"key":"mlstae1277bib42","doi-asserted-by":"publisher","DOI":"10.1016\/j.cma.2022.115296","type":"journal-article","article-title":"Learning deep implicit fourier neural operators (IFNOs) with applications to heterogeneous material modeling","volume":"398","author":"You","year":"2022","journal-title":"Comput. Methods Appl. Mech. Eng."},{"article-title":"Nonlocal attention operator: materializing hidden knowledge towards interpretable physics discovery","year":"2024","author":"Yu","key":"mlstae1277bib43","type":"preprint"},{"key":"mlstae1277bib44","first-page":"pp 2114","type":"conference-proceedings","article-title":"A transformer-based framework for multivariate time series representation learning","author":"Zerveas","year":"2021"},{"article-title":"Human activity recognition based on time series analysis using u-net","year":"2018","author":"Zhang","key":"mlstae1277bib45","type":"preprint"},{"key":"mlstae1277bib46","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2024.110994","type":"journal-article","article-title":"Tfformer: a time-frequency domain bidirectional sequence-level attention based transformer for interpretable long-term sequence forecasting","volume":"158","author":"Zhao","year":"2024","journal-title":"Pattern Recognit."},{"key":"mlstae1277bib47","first-page":"pp 11106","type":"conference-proceedings","article-title":"Informer: beyond efficient transformer for long sequence time-series forecasting","volume":"vol 35","author":"Zhou","year":"2021"}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1277","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1277\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1277","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1277\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1277\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1277\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1277\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1277\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,6]],"date-time":"2025-11-06T09:50:53Z","timestamp":1762422653000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1277"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,6]]},"references-count":47,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2025,11,6]]},"published-print":{"date-parts":[[2025,12,30]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/ae1277","relation":{},"ISSN":["2632-2153"],"issn-type":[{"type":"electronic","value":"2632-2153"}],"subject":[],"published":{"date-parts":[[2025,11,6]]},"assertion":[{"value":"An attention-based spatio-temporal neural operator for evolving physics","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2025 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2025-06-13","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2025-10-13","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2025-11-06","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}