{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,11]],"date-time":"2025-09-11T19:30:37Z","timestamp":1757619037059,"version":"3.44.0"},"reference-count":59,"publisher":"Springer Science and Business Media LLC","issue":"8","license":[{"start":{"date-parts":[[2025,7,15]],"date-time":"2025-07-15T00:00:00Z","timestamp":1752537600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,7,15]],"date-time":"2025-07-15T00:00:00Z","timestamp":1752537600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Department of Computer Science, University of Manchester"},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/Y030826\/1","EP\/R034303\/1"],"award-info":[{"award-number":["EP\/Y030826\/1","EP\/R034303\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2025,8]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Modelling the behaviour of highly nonlinear dynamical systems with robust uncertainty quantification is a challenging task which typically requires approaches specifically designed to address the problem at hand. We introduce a domain-agnostic model to address this issue termed the deep latent force model (DLFM), a deep Gaussian process with physics-informed kernels at each layer, derived from ordinary differential equations using the framework of process convolutions. Two distinct formulations of the DLFM are presented which utilise weight-space and variational inducing points-based Gaussian process approximations, both of which are amenable to doubly stochastic variational inference. We present empirical evidence of the capability of the DLFM to capture the dynamics present in highly nonlinear real-world multi-output time series data. Additionally, we find that the DLFM is capable of achieving comparable performance to a range of non-physics-informed probabilistic models on benchmark univariate regression tasks. We also empirically assess the negative impact of the inducing points framework on the extrapolation capabilities of LFM-based models.<\/jats:p>","DOI":"10.1007\/s10994-025-06824-y","type":"journal-article","created":{"date-parts":[[2025,7,15]],"date-time":"2025-07-15T19:45:46Z","timestamp":1752608746000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Deep latent force models: ODE-based process convolutions for Bayesian deep learning"],"prefix":"10.1007","volume":"114","author":[{"given":"Thomas","family":"Baldwin-McDonald","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xinxing","family":"Shi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mingxin","family":"Shen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mauricio A.","family":"\u00c1lvarez","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2025,7,15]]},"reference":[{"key":"6824_CR1","unstructured":"Alvarez, M., Luengo, D., & Lawrence, N.D. (2009). Latent force models. Artificial Intelligence and Statistics (pp. 9\u201316)."},{"key":"6824_CR2","first-page":"1459","volume":"12","author":"MA Alvarez","year":"2011","unstructured":"Alvarez, M. A., & Lawrence, N. D. (2011). Computationally efficient convolved multiple output gaussian processes. The Journal of Machine Learning Research, 12, 1459\u20131500.","journal-title":"The Journal of Machine Learning Research"},{"issue":"3","key":"6824_CR3","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1561\/2200000036","volume":"4","author":"MA Alvarez","year":"2012","unstructured":"Alvarez, M. A., Rosasco, L., & Lawrence, N. D. (2012). Kernels for vector-valued functions: A review. Foundations and Trends\u00ae in Machine Learning, 4(3), 195\u2013266.","journal-title":"Foundations and Trends\u00ae in Machine Learning"},{"key":"6824_CR4","unstructured":"\u00c1lvarez, M.A., Ward, W., & Guarnizo, C. (2019). Non-linear process convolutions for multi-output Gaussian processes. In The 22nd International conference on artificial intelligence and statistics (pp. 1969\u20131977)."},{"issue":"8","key":"6824_CR5","doi-asserted-by":"publisher","first-page":"1798","DOI":"10.1109\/TPAMI.2013.50","volume":"35","author":"Y Bengio","year":"2013","unstructured":"Bengio, Y., Courville, A., & Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8), 1798\u20131828.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"6824_CR6","doi-asserted-by":"crossref","unstructured":"Blomqvist, K., Kaski, S., & Heinonen, M. (2020). Deep convolutional Gaussian processes. In Machine learning and knowledge discovery in databases: European conference, ecml pkdd 2019, w\u00fcrzburg, germany, September 16\u201320, 2019, proceedings, part ii (pp. 582\u2013597).","DOI":"10.1007\/978-3-030-46147-8_35"},{"key":"6824_CR7","unstructured":"Bui, T.D., & Turner, R.E. (2014). Tree-structured Gaussian process approximations. Advances in Neural Information Processing Systems, 27"},{"key":"6824_CR8","unstructured":"Calandriello, D., Carratino, L., Lazaric, A., Valko, M., & Rosasco, L. (2019). Gaussian process optimization with adaptive sketching: Scalable and no regret. In Conference on learning theory (pp. 533\u2013557)."},{"key":"6824_CR9","unstructured":"Chen, R.T.Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D.K. (2018). Neural ordinary differential equations. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, & R. Garnett (Eds.), Advances in Neural Information Processing Systems (Vol. 31). Curran Associates, Inc."},{"key":"6824_CR10","unstructured":"Chung, J., Kastner, K., Dinh, L., Goel, K., Courville, A.C., & Bengio, Y. (2015). A recurrent latent variable model for sequential data. Advances in Neural Information Processing Systems, 28"},{"key":"6824_CR11","unstructured":"Cutajar, K., Bonilla, E.V., Michiardi, P., & Filippone, M. (2017). Random feature expansions for deep Gaussian processes. In International conference on machine learning (pp. 884\u2013893)."},{"key":"6824_CR12","unstructured":"Damianou, A. (2015). Deep Gaussian processes and variational propagation of uncertainty (Unpublished doctoral dissertation). University of Sheffield."},{"key":"6824_CR13","unstructured":"Damianou, A., & Lawrence, N.D. (2013). Deep Gaussian processes. Artificial Intelligence and Statistics (pp. 207\u2013215)."},{"key":"6824_CR14","unstructured":"Dua, D., & Graff, C. (2017). UCI machine learning repository. Retrieved from http:\/\/archive.ics.uci.edu\/ml"},{"issue":"54","key":"6824_CR15","first-page":"1","volume":"19","author":"MM Dunlop","year":"2018","unstructured":"Dunlop, M. M., Girolami, M. A., Stuart, A. M., & Teckentrup, A. L. (2018). How deep are deep gaussian processes? Journal of Machine Learning Research, 19(54), 1\u201346.","journal-title":"Journal of Machine Learning Research"},{"key":"6824_CR16","unstructured":"Duvenaud, D., Rippel, O., Adams, R., & Ghahramani, Z. (2014). Avoiding pathologies in very deep networks. Artificial Intelligence and Statistics (pp. 202\u2013210)."},{"key":"6824_CR17","first-page":"7576","volume":"31","author":"J Gardner","year":"2018","unstructured":"Gardner, J., Pleiss, G., Weinberger, K. Q., Bindel, D., & Wilson, A. G. (2018). GPyTorch: Blackbox matrix-matrix gaussian process inference with GPU acceleration. Advances in Neural Information Processing Systems, 31, 7576\u20137586.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"6824_CR18","unstructured":"Ghosh, S., Birrell, P., & De Angelis, D. (2021). Variational inference for nonlinear ordinary differential equations. In International conference on artificial intelligence and statistics (pp. 2719\u20132727)."},{"key":"6824_CR19","first-page":"22341","volume":"35","author":"M Giordano","year":"2022","unstructured":"Giordano, M., Ray, K., & Schmidt-Hieber, J. (2022). On the inability of gaussian process regression to optimally learn compositional functions. Advances in Neural Information Processing Systems, 35, 22341\u201322353.","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"23","key":"6824_CR20","doi-asserted-by":"publisher","first-page":"e215","DOI":"10.1161\/01.CIR.101.23.e215","volume":"101","author":"AL Goldberger","year":"2000","unstructured":"Goldberger, A. L., Amaral, L. A., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., & Stanley, H. E. (2000). PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals. Circulation, 101(23), e215\u2013e220.","journal-title":"Circulation"},{"key":"6824_CR21","unstructured":"Guarnizo, C., & \u00c1lvarez, M.A. (2018). Fast kernel approximations for latent force models and convolved multiple-output Gaussian processes. In Uncertainty in artificial intelligence: proceedings of the thirty-fourth conference (2018)."},{"key":"6824_CR22","first-page":"7506","volume":"2018","author":"M Havasi","year":"2018","unstructured":"Havasi, M., Hern\u00e1ndez-Lobato, J., & Murillo-Fuentes, J. (2018). Inference in deep gaussian processes using stochastic gradient hamiltonian monte carlo. Advances in Neural Information Processing Systems, 2018, 7506\u20137516.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"6824_CR23","unstructured":"Hensman, J., Fusi, N., & Lawrence, N.D. (2013). Gaussian processes for big data. In Proceedings of the twenty-ninth conference on uncertainty in artificial intelligence (pp. 282\u2013290)."},{"issue":"5","key":"6824_CR24","first-page":"1303","volume":"14","author":"MD Hoffman","year":"2013","unstructured":"Hoffman, M. D., Blei, D. M., Wang, C., & Paisley, J. (2013). Stochastic variational inference. Journal of Machine Learning Research, 14(5), 1303\u20131347.","journal-title":"Journal of Machine Learning Research"},{"issue":"6","key":"6824_CR25","doi-asserted-by":"publisher","first-page":"821","DOI":"10.1007\/s10877-015-9779-3","volume":"30","author":"N Kim","year":"2016","unstructured":"Kim, N., Krasner, A., Kosinski, C., Wininger, M., Qadri, M., Kappus, Z., & Craelius, W. (2016). Trending autoregulatory indices during treatment for traumatic brain injury. Journal of clinical monitoring and computing, 30(6), 821\u2013831.","journal-title":"Journal of clinical monitoring and computing"},{"key":"6824_CR26","unstructured":"Kingma, D.P., Salimans, T., & Welling, M. (2015). Variational dropout and the local reparameterization trick. Advances in Neural Information Processing Systems, 28"},{"key":"6824_CR27","unstructured":"Kingma, D.P., & Welling, M. (2014). Auto-encoding variational Bayes. In 2nd International conference on learning representations, ICLR 2014, banff, ab, canada, April 14\u201316, 2014, conference track proceedings. Retrieved from http:\/\/arxiv.org\/abs\/1312.6114"},{"key":"6824_CR28","doi-asserted-by":"crossref","unstructured":"Lai, G., Chang, W.-C., Yang, Y., & Liu, H. (2018). Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international acm sigir conference on research & development in information retrieval (pp. 95\u2013104).","DOI":"10.1145\/3209978.3210006"},{"issue":"7553","key":"6824_CR29","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436\u2013444.","journal-title":"Nature"},{"key":"6824_CR30","unstructured":"Lorenzi, M., & Filippone, M. (2018). Constraining the dynamics of deep probabilistic models. In International conference on machine learning (pp. 3227\u20133236)."},{"key":"6824_CR31","unstructured":"Loshchilov, I., & Hutter, F. (2018). Decoupled weight decay regularization. In International conference on learning representations."},{"key":"6824_CR32","first-page":"13809","volume":"34","author":"T McDonald","year":"2021","unstructured":"McDonald, T., & \u00c1lvarez, M. (2021). Compositional modeling of nonlinear dynamical systems with ODE-based random features. Advances in Neural Information Processing Systems, 34, 13809\u201313819.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"6824_CR33","unstructured":"McDonald, T.M., Ross, M., Smith, M.T., & \u00c1lvarez, M.A. (2022). Shallow and deep nonparametric convolutions for Gaussian processes. arXiv preprint arXiv:2206.08972"},{"key":"6824_CR34","unstructured":"McDonald, T.M., Ross, M., Smith, M.T., & \u00c1lvarez, M.A. (2023). Nonparametric Gaussian process covariances via multidimensional convolutions. In International conference on artificial intelligence and statistics (pp. 8279\u20138293)."},{"key":"6824_CR35","unstructured":"Meeds, T., Roeder, G., Grant, P., Phillips, A., & Dalchau, N. (2019). Efficient amortised Bayesian inference for hierarchical and nonlinear dynamical systems. In International conference on machine learning (pp. 4445\u20134455)."},{"key":"6824_CR36","unstructured":"Moss, J.D., Opolka, F.L., Dumitrascu, B., & Li\u00f3, P. (2021). Approximate latent force model inference. arXiv preprint arXiv:2109.11851"},{"key":"6824_CR37","unstructured":"Rahimi, A., & Recht, B. (2007). Random features for large-scale kernel machines. Advances in Neural Information Processing Systems, 20"},{"key":"6824_CR38","doi-asserted-by":"publisher","first-page":"686","DOI":"10.1016\/j.jcp.2018.10.045","volume":"378","author":"M Raissi","year":"2019","unstructured":"Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378, 686\u2013707.","journal-title":"Journal of Computational physics"},{"key":"6824_CR39","doi-asserted-by":"crossref","unstructured":"Rasmussen, C.E., & Williams, C. (2006). Gaussian Processes for Machine Learning. MIT Press. Retrieved from http:\/\/www.gaussianprocess.org\/gpml\/","DOI":"10.7551\/mitpress\/3206.001.0001"},{"key":"6824_CR40","first-page":"24099","volume":"34","author":"M Ross","year":"2021","unstructured":"Ross, M., Smith, M. T., & \u00c1lvarez, M. (2021). Learning nonparametric volterra kernels with gaussian processes. Advances in Neural Information Processing Systems, 34, 24099\u201324110.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"6824_CR41","unstructured":"Rubanova, Y., Chen, R.T., & Duvenaud, D.K. (2019). Latent ordinary differential equations for irregularly-sampled time series. Advances in Neural Information Processing Systems, 32"},{"key":"6824_CR42","unstructured":"Rudner, T.G., Sejdinovic, D., & Gal, Y. (2020). Inter-domain deep Gaussian processes. In International conference on machine learning (pp. 8286\u20138294)."},{"key":"6824_CR43","first-page":"4591","volume":"31","author":"H Salimbeni","year":"2017","unstructured":"Salimbeni, H., & Deisenroth, M. (2017). Doubly stochastic variational inference for deep gaussian processes. Nips, 31, 4591\u20134602.","journal-title":"Nips"},{"key":"6824_CR44","unstructured":"Salimbeni, H., Dutordoir, V., Hensman, J., & Deisenroth, M. (2019). Deep Gaussian processes with importance-weighted variational inference. InInternational conference on machine learning (pp. 5589\u20135598)."},{"key":"6824_CR45","unstructured":"Shukla, S.N., & Marlin, B. (2021). Multi-time attention networks for irregularly sampled time series. In International conference on learning representations."},{"key":"6824_CR46","unstructured":"Titsias, M. (2009). Variational learning of inducing variables in sparse Gaussian processes. Artificial Intelligence and Statistics (pp. 567\u2013574)."},{"key":"6824_CR47","unstructured":"Tobar, F., Bui, T.D., & Turner, R.E. (2015). Learning stationary time series using Gaussian processes with nonparametric kernels. Advances in Neural Information Processing Systems, 28"},{"key":"6824_CR48","unstructured":"Ustyuzhaninov, I., Kazlauskaite, I., Kaiser, M., Bodin, E., Campbell, N., & Ek, C.H. (2020). Compositional uncertainty in deep Gaussian processes. In Conference on uncertainty in artificial intelligence (pp. 480\u2013489)."},{"key":"6824_CR49","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30"},{"key":"6824_CR50","unstructured":"Wang, Z., Xing, W., Kirby, R., & Zhe, S. (2022). Physics informed deep kernel learning. In International conference on artificial intelligence and statistics (pp. 1206\u20131218)."},{"key":"6824_CR51","unstructured":"Ward, W., Ryder, T., Prangle, D., & Alvarez, M. (2020). Black-box inference for nonlinear latent force models. In International conference on artificial intelligence and statistics (pp. 3088\u20133098)."},{"key":"6824_CR52","unstructured":"Willard, J., Jia, X., Xu, S., Steinbach, M., & Kumar, V. (2020). Integrating physics-based modeling with machine learning: A survey. arXiv preprint arXiv:2003.04919 1 (1), 1\u201334,"},{"key":"6824_CR53","unstructured":"Wilson, J., Borovitskiy, V., Terenin, A., Mostowsky, P., & Deisenroth, M. (2020). Efficiently sampling functions from Gaussian process posteriors. In International conference on machine learning (pp. 10292\u201310302)."},{"key":"6824_CR54","first-page":"22419","volume":"34","author":"H Wu","year":"2021","unstructured":"Wu, H., Xu, J., Wang, J., & Long, M. (2021). Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34, 22419\u201322430.","journal-title":"Advances in neural information processing systems"},{"key":"6824_CR55","doi-asserted-by":"crossref","unstructured":"Yu, H., Chen, Y., Dai, Z., Low, K.H., & Jaillet, P. (2019). Implicit posterior variational inference for deep Gaussian processes. arXiv preprint arXiv:1910.11998","DOI":"10.1109\/IJCNN.2019.8852481"},{"key":"6824_CR56","doi-asserted-by":"publisher","unstructured":"Zerveas, G., Jayaraman, S., Patel, D., Bhamidipaty, A., & Eickhoff, C. (2021). A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th acm sigkdd conference on knowledge discovery & data mining (p. 2114\u20132124). New York, NY, USA: Association for Computing Machinery. Retrieved from https:\/\/doi.org\/10.1145\/3447548.3467401","DOI":"10.1145\/3447548.3467401"},{"key":"6824_CR57","doi-asserted-by":"publisher","first-page":"11106","DOI":"10.1609\/aaai.v35i12.17325","volume":"35","author":"H Zhou","year":"2021","unstructured":"Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., & Zhang, W. (2021). Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of the aaai conference on artificial intelligence, 35, 11106\u201311115.","journal-title":"Proceedings of the aaai conference on artificial intelligence"},{"key":"6824_CR58","unstructured":"Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., & Jin, R. (2022). Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning (pp. 27268\u201327286)."},{"key":"6824_CR59","unstructured":"Zhu, H., Balsells-Rodas, C., & Li, Y. (2023). Markovian gaussian process variational autoencoders. In International conference on machine learning (pp. 42938\u201342961)."}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-025-06824-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-025-06824-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-025-06824-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,7]],"date-time":"2025-09-07T11:23:15Z","timestamp":1757244195000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-025-06824-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,15]]},"references-count":59,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2025,8]]}},"alternative-id":["6824"],"URL":"https:\/\/doi.org\/10.1007\/s10994-025-06824-y","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"type":"print","value":"0885-6125"},{"type":"electronic","value":"1573-0565"}],"subject":[],"published":{"date-parts":[[2025,7,15]]},"assertion":[{"value":"3 May 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 June 2025","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 June 2025","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 July 2025","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no Conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Thomas Baldwin-McDonald thanks the Department of Computer Science at the University of Manchester for financial support. Mingxin Shen was supported by the Engineering and Physical Sciences Research Council (EPSRC) grant EP\/Y030826\/1. Mauricio A. \u00c1lvarez has been financed by the EPSRC Research Projects EP\/R034303\/1, EP\/T00343X\/2 and EP\/V029045\/1.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Statements and Declarations"}}],"article-number":"192"}}