{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,19]],"date-time":"2026-01-19T14:34:41Z","timestamp":1768833281967,"version":"3.49.0"},"reference-count":37,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2022,11,23]],"date-time":"2022-11-23T00:00:00Z","timestamp":1669161600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"BIODYNAMICS MCIN","award":["AEI\/10.13039\/501100011033"],"award-info":[{"award-number":["AEI\/10.13039\/501100011033"]}]},{"name":"BIODYNAMICS MCIN","award":["202070E062"],"award-info":[{"award-number":["202070E062"]}]},{"name":"BIODYNAMICS MCIN","award":["MCIN\/AEI\/10.13039\/501100011033"],"award-info":[{"award-number":["MCIN\/AEI\/10.13039\/501100011033"]}]},{"name":"BIODYNAMICS MCIN","award":["PID2020-113992RA-I00"],"award-info":[{"award-number":["PID2020-113992RA-I00"]}]},{"name":"Conseller\u00eda de Cultura, Educaci\u00f3n e Ordenaci\u00f3n Universitaria","award":["ED431F 2021\/003"],"award-info":[{"award-number":["ED431F 2021\/003"]}]},{"name":"Conseller\u00eda de Cultura, Educaci\u00f3n e Ordenaci\u00f3n Universitaria","award":["MCIN\/AEI\/10.13039\/501100011033"],"award-info":[{"award-number":["MCIN\/AEI\/10.13039\/501100011033"]}]},{"DOI":"10.13039\/501100006761","name":"Universidade de Vigo","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100006761","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,1,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Motivation<\/jats:title>\n                  <jats:p>Dynamic mechanistic modelling in systems biology has been hampered by the complexity and variability associated with the underlying interactions, and by uncertain and sparse experimental measurements. Ensemble modelling, a concept initially developed in statistical mechanics, has been introduced in biological applications with the aim of mitigating those issues. Ensemble modelling uses a collection of different models compatible with the observed data to describe the phenomena of interest. However, since systems biology models often suffer from a lack of identifiability and observability, ensembles of models are particularly unreliable when predicting non-observable states.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>We present a strategy to assess and improve the reliability of a class of model ensembles. In particular, we consider kinetic models described using ordinary differential equations with a fixed structure. Our approach builds an ensemble with a selection of the parameter vectors found when performing parameter estimation with a global optimization metaheuristic. This technique enforces diversity during the sampling of parameter space and it can quantify the uncertainty in the predictions of state trajectories. We couple this strategy with structural identifiability and observability analysis, and when these tests detect possible prediction issues we obtain model reparameterizations that surmount them. The end result is an ensemble of models with the ability to predict the internal dynamics of a biological process. We demonstrate our approach with models of glucose regulation, cell division, circadian oscillations and the JAK-STAT signalling pathway.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Availability and implementation<\/jats:title>\n                  <jats:p>The code that implements the methodology and reproduces the results is available at https:\/\/doi.org\/10.5281\/zenodo.6782638.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Supplementary information<\/jats:title>\n                  <jats:p>Supplementary data are available at Bioinformatics online.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btac755","type":"journal-article","created":{"date-parts":[[2022,11,23]],"date-time":"2022-11-23T09:59:55Z","timestamp":1669197595000},"source":"Crossref","is-referenced-by-count":6,"title":["Improving dynamic predictions with ensembles of observable models"],"prefix":"10.1093","volume":"39","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8740-825X","authenticated-orcid":false,"given":"Gemma","family":"Massonis","sequence":"first","affiliation":[{"name":"Computational Biology Lab, MBG-CSIC (Spanish National Research Council) , Pontevedra, Galicia 36143, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7401-7380","authenticated-orcid":false,"given":"Alejandro F","family":"Villaverde","sequence":"additional","affiliation":[{"name":"CITMAga , Santiago de Compostela, Galicia 15782, Spain"},{"name":"Department of Systems and Control Engineering, Universidade de Vigo , Vigo, Galicia 36310, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4245-0320","authenticated-orcid":false,"given":"Julio R","family":"Banga","sequence":"additional","affiliation":[{"name":"Computational Biology Lab, MBG-CSIC (Spanish National Research Council) , Pontevedra, Galicia 36143, Spain"}]}],"member":"286","published-online":{"date-parts":[[2022,11,23]]},"reference":[{"key":"2023010107521007400_btac755-B1","doi-asserted-by":"crossref","first-page":"3357","DOI":"10.1093\/bioinformatics\/btw411","article-title":"AMIGO2, a toolbox for dynamic modeling, optimization and control in systems biology","volume":"32","author":"Balsa-Canto","year":"2016","journal-title":"Bioinformatics"},{"key":"2023010107521007400_btac755-B2","doi-asserted-by":"crossref","DOI":"10.1103\/PhysRevE.68.021904","article-title":"Statistical mechanical approaches to models with many poorly known parameters","volume":"68","author":"Brown","year":"2003","journal-title":"Phys. Rev. E"},{"key":"2023010107521007400_btac755-B3","article-title":"STRIKE-GOLDD 4.0: user-friendly, efficient analysis of structural identifiability and observability","author":"D\u00edaz-Seoane","year":"2022","journal-title":"Bioinformatics"},{"key":"2023010107521007400_btac755-B4","volume-title":"Dynamic Systems Biology Modeling and Simulation","author":"Distefano","year":"2015"},{"key":"2023010107521007400_btac755-B5","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2105-15-136","article-title":"MEIGO: an open-source software suite based on metaheuristics for global optimization in systems biology and bioinformatics","volume":"15","author":"Egea","year":"2014","journal-title":"BMC Bioinformatics"},{"key":"2023010107521007400_btac755-B6","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1007\/978-3-319-12982-2_5","volume-title":"Computational Methods in Systems Biology","author":"Fr\u00f6hlich","year":"2014"},{"key":"2023010107521007400_btac755-B7","doi-asserted-by":"crossref","first-page":"3676","DOI":"10.1093\/bioinformatics\/btab227","article-title":"AMICI: high-performance sensitivity analysis for large ordinary differential equation models","volume":"37","author":"Fr\u00f6hlich","year":"2021","journal-title":"Bioinformatics"},{"key":"2023010107521007400_btac755-B8","first-page":"23","volume-title":"Analyzing and Constraining Signaling Networks: Parameter Estimation for the User","author":"Geier","year":"2012"},{"key":"2023010107521007400_btac755-B9","doi-asserted-by":"crossref","first-page":"746","DOI":"10.1198\/jasa.2011.r10138","article-title":"Making and evaluating point forecasts","volume":"106","author":"Gneiting","year":"2011","journal-title":"J. Am. Stat. Assoc"},{"key":"2023010107521007400_btac755-B10","doi-asserted-by":"crossref","first-page":"e1007536","DOI":"10.1371\/journal.pcbi.1007536","article-title":"Statistical inference in ensemble modeling of cellular metabolism","volume":"15","author":"Hameri","year":"2019","journal-title":"PLoS Comput. Biol"},{"key":"2023010107521007400_btac755-B11","doi-asserted-by":"crossref","first-page":"891","DOI":"10.3390\/metabo2040891","article-title":"Ensemble kinetic modeling of metabolic networks from dynamic metabolic profiles","volume":"2","author":"Jia","year":"2012","journal-title":"Metabolites"},{"key":"2023010107521007400_btac755-B12","doi-asserted-by":"crossref","first-page":"3923","DOI":"10.1016\/j.febslet.2009.10.074","article-title":"Systems analysis of cellular networks under uncertainty","volume":"583","author":"Kaltenbach","year":"2009","journal-title":"FEBS Lett"},{"key":"2023010107521007400_btac755-B13","doi-asserted-by":"crossref","first-page":"886","DOI":"10.15252\/msb.20167216","article-title":"Dynamical compensation in physiological circuits","volume":"12","author":"Karin","year":"2016","journal-title":"Mol. Syst. Biol"},{"key":"2023010107521007400_btac755-B14","doi-asserted-by":"crossref","first-page":"386","DOI":"10.1126\/science.aac9505","article-title":"Systems biology (un)certainties","volume":"350","author":"Kirk","year":"2015","journal-title":"Science"},{"key":"2023010107521007400_btac755-B15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12918-018-0604-8","article-title":"An ensemble of mathematical models showing diauxic growth behaviour","volume":"12","author":"Kremling","year":"2018","journal-title":"BMC Syst. Biol"},{"key":"2023010107521007400_btac755-B16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1752-0509-6-120","article-title":"Likelihood based observability analysis and confidence intervals for predictions of dynamic models","volume":"6","author":"Kreutz","year":"2012","journal-title":"BMC Syst. Biol"},{"key":"2023010107521007400_btac755-B17","doi-asserted-by":"crossref","first-page":"1001","DOI":"10.1038\/nbt1330","article-title":"Ensemble modeling for analysis of cell signaling dynamics","volume":"25","author":"Kuepfer","year":"2007","journal-title":"Nat. Biotechnol"},{"key":"2023010107521007400_btac755-B18","first-page":"1","article-title":"AutoRepar: a method to obtain identifiable and observable reparameterizations of dynamic models with mechanistic insights","author":"Massonis","year":"2021","journal-title":"Int. J. Robust Nonlin. Control"},{"key":"2023010107521007400_btac755-B19","doi-asserted-by":"crossref","first-page":"012920","DOI":"10.1103\/PhysRevE.92.012920","article-title":"Higher-order lie symmetries in identifiability and predictability analysis of dynamic models","volume":"92","author":"Merkt","year":"2015","journal-title":"Phys. Rev. E Stat. Nonlin. Soft Matter Phys"},{"key":"2023010107521007400_btac755-B20","doi-asserted-by":"crossref","first-page":"413","DOI":"10.1002\/bit.22932","article-title":"Modeling of uncertainties in biochemical reactions","volume":"108","author":"Mi\u0161kovi\u0107","year":"2011","journal-title":"Biotechnol. Bioeng"},{"key":"2023010107521007400_btac755-B22","first-page":"16","article-title":"Robust estimation and identifying outliers","volume":"16","author":"Rousseeuw","year":"1990"},{"key":"2023010107521007400_btac755-B23","doi-asserted-by":"crossref","first-page":"981","DOI":"10.1016\/j.biotechadv.2017.09.005","article-title":"Formulation, construction and analysis of kinetic models of metabolism: a review of modelling frameworks","volume":"35","author":"Saa","year":"2017","journal-title":"Biotechnol. Adv"},{"key":"2023010107521007400_btac755-B24","doi-asserted-by":"crossref","first-page":"396","DOI":"10.1016\/j.cels.2015.12.002","article-title":"A systematic ensemble approach to thermodynamic modeling of gene expression from sequence data","volume":"1","author":"Samee","year":"2015","journal-title":"Cell Syst"},{"key":"2023010107521007400_btac755-B25","doi-asserted-by":"crossref","first-page":"20200419","DOI":"10.1098\/rsif.2020.0419","article-title":"Multi-model and network inference based on ensemble estimates: avoiding the madness of crowds","volume":"17","author":"Stumpf","year":"2020","journal-title":"J. R. Soc. Interface"},{"key":"2023010107521007400_btac755-B26","first-page":"19","volume-title":"Ensemble Modeling of Biological Systems","author":"Swigon","year":"2012"},{"key":"2023010107521007400_btac755-B27","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1186\/1752-0509-5-177","article-title":"Inference of complex biological networks: distinguishability issues and optimization-based solutions","volume":"5","author":"Szederk\u00e9nyi","year":"2011","journal-title":"BMC Syst. Biol"},{"key":"2023010107521007400_btac755-B28","doi-asserted-by":"crossref","first-page":"5606","DOI":"10.1529\/biophysj.108.135442","article-title":"Ensemble modeling of metabolic networks","volume":"95","author":"Tran","year":"2008","journal-title":"Biophys. J"},{"key":"2023010107521007400_btac755-B29","doi-asserted-by":"crossref","first-page":"7328","DOI":"10.1073\/pnas.88.16.7328","article-title":"Modeling the cell division cycle: cdc2 and cyclin interactions","volume":"88","author":"Tyson","year":"1991","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023010107521007400_btac755-B30","doi-asserted-by":"crossref","first-page":"e103812","DOI":"10.1371\/journal.pone.0103812","article-title":"Ensemble inference and inferability of gene regulatory networks","volume":"9","author":"Ud-Dean","year":"2014","journal-title":"PLoS One"},{"key":"2023010107521007400_btac755-B31","doi-asserted-by":"crossref","first-page":"1130","DOI":"10.1093\/bioinformatics\/bts088","article-title":"An integrated strategy for prediction uncertainty analysis","volume":"28","author":"Vanlier","year":"2012","journal-title":"Bioinformatics"},{"key":"2023010107521007400_btac755-B32","doi-asserted-by":"crossref","first-page":"5988","DOI":"10.1073\/pnas.092133899","article-title":"Mechanisms of noise-resistance in genetic oscillators","volume":"99","author":"Vilar","year":"2002","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023010107521007400_btac755-B33","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1016\/j.cmpb.2015.02.001","article-title":"A consensus approach for estimating the predictive accuracy of dynamic models in biology","volume":"119","author":"Villaverde","year":"2015","journal-title":"Comput. Methods Programs Biomed"},{"key":"2023010107521007400_btac755-B34","doi-asserted-by":"crossref","first-page":"830","DOI":"10.1093\/bioinformatics\/bty736","article-title":"Benchmarking optimization methods for parameter estimation in large kinetic models","volume":"35","author":"Villaverde","year":"2019","journal-title":"Bioinformatics"},{"key":"2023010107521007400_btac755-B35","doi-asserted-by":"crossref","first-page":"20190043","DOI":"10.1098\/rsif.2019.0043","article-title":"Full observability and estimation of unknown inputs, states and parameters of nonlinear biological models","volume":"16","author":"Villaverde","year":"2019","journal-title":"J. R Soc. Interface"},{"key":"2023010107521007400_btac755-B36","first-page":"1","article-title":"A protocol for dynamic model calibration","volume":"23","author":"Villaverde","year":"2021","journal-title":"Brief. Bioinformatics"},{"key":"2023010107521007400_btac755-B37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TCBB.2022.3213914","article-title":"Assessment of prediction uncertainty quantification methods in systems biology","author":"Villaverde","year":"2022","journal-title":"IEEE\/ACM Trans. Comput. Biol. Bioinform"},{"key":"2023010107521007400_btac755-B38","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.coisb.2021.03.005","article-title":"On structural and practical identifiability","volume":"25","author":"Wieland","year":"2021","journal-title":"Curr. Opin. Syst. Biol"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btac755\/47758928\/btac755.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/39\/1\/btac755\/48448753\/btac755.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/39\/1\/btac755\/48448753\/btac755.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,1]],"date-time":"2023-01-01T10:09:35Z","timestamp":1672567775000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/doi\/10.1093\/bioinformatics\/btac755\/6842325"}},"subtitle":[],"editor":[{"given":"Alfonso","family":"Valencia","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2022,11,23]]},"references-count":37,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,11,23]]},"published-print":{"date-parts":[[2023,1,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btac755","relation":{},"ISSN":["1367-4811"],"issn-type":[{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,1,1]]},"published":{"date-parts":[[2022,11,23]]},"article-number":"btac755"}}