{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T02:01:49Z","timestamp":1776132109697,"version":"3.50.1"},"reference-count":40,"publisher":"Public Library of Science (PLoS)","issue":"12","license":[{"start":{"date-parts":[[2025,12,4]],"date-time":"2025-12-04T00:00:00Z","timestamp":1764806400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["P41EB023912"],"award-info":[{"award-number":["P41EB023912"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R24GM137787"],"award-info":[{"award-number":["R24GM137787"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100012116","name":"European Bioinformatics Institute","doi-asserted-by":"publisher","award":["core"],"award-info":[{"award-number":["core"]}],"id":[{"id":"10.13039\/100012116","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100013322","name":"European Federation of Pharmaceutical Industries and Associations","doi-asserted-by":"publisher","award":["116030"],"award-info":[{"award-number":["116030"]}],"id":[{"id":"10.13039\/100013322","id-type":"DOI","asserted-by":"publisher"}]},{"name":"European Union\u2019s Horizon 2020 Research and Innovation Programme and European Federation of Pharmaceutical Industries and Associations"}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>\n                    The BioModels Repository contains over 1000 manually curated mechanistic models from published literature, most often encoded in the Systems Biology Markup Language (SBML). This community-based standard formally specifies each model, but does not describe the computational experimental conditions to run a simulation and collect data. Therefore, it can be challenging to reproduce any figure or result from a publication with an SBML model alone. The Simulation Experiment Description Markup Language (SED-ML) provides a solution: a standard way to specify exactly how to run an experiment corresponding to a specific figure or result. BioModels was established years before SED-ML, and both systems evolved over time, both in content and acceptance. Hence, only about half of the entries in BioModels contained SED-ML files, and these files reflected the version of SED-ML that was available at the time. Additionally, almost all of these SED-ML files had at least one minor mistake that made them impossible to run. To make these models and their results more reproducible, we report here on our work updating, correcting and generating new SED-ML files for 1055 curated mechanistic models in BioModels. In addition, because SED-ML is implementation-independent, it can be used for\n                    <jats:italic>verification<\/jats:italic>\n                    , demonstrating that results hold across multiple simulation engines. We tested, corrected, and improved over 450 existing SED-ML files in the BioModels database, and created basic files for the rest of the entries. Then, we used a wrapper architecture for interpreting SED-ML, and report verification results across five different ODE-based biosimulation engines, after further improving the models, the wrappers, and the engines themselves. Our work with SED-ML and the BioModels collection aims to improve the utility of these models by making them more reproducible and credible. Improved reproducibility means these models are now even more fit for re-use, such as in new investigations and as components of multiscale models.\n                  <\/jats:p>","DOI":"10.1371\/journal.pcbi.1013239","type":"journal-article","created":{"date-parts":[[2025,12,4]],"date-time":"2025-12-04T19:02:51Z","timestamp":1764874971000},"page":"e1013239","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":1,"title":["Verification and reproducible curation of the BioModels repository"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7002-6386","authenticated-orcid":true,"given":"Lucian P.","family":"Smith","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rahuman S.","family":"Malik-Sheriff","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tung V. N.","family":"Nguyen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8479-0262","authenticated-orcid":true,"given":"Henning","family":"Hermjakob","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2605-5080","authenticated-orcid":true,"given":"Jonathan","family":"Karr","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5801-5510","authenticated-orcid":true,"given":"Bilal","family":"Shaikh","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Logan","family":"Drescher","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ion I.","family":"Moraru","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"James C.","family":"Schaff","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eran","family":"Agmon","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alexander A.","family":"Patrie","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael L.","family":"Blinov","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Joseph L.","family":"Hellerstein","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Elebeoba E.","family":"May","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4667-9779","authenticated-orcid":true,"given":"David P.","family":"Nickerson","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"John H.","family":"Gennari","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3659-6817","authenticated-orcid":true,"given":"Herbert M.","family":"Sauro","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"340","published-online":{"date-parts":[[2025,12,4]]},"reference":[{"issue":"7604","key":"pcbi.1013239.ref001","doi-asserted-by":"crossref","first-page":"452","DOI":"10.1038\/533452a","article-title":"1,500 scientists lift the lid on reproducibility","volume":"533","author":"M Baker","year":"2016","journal-title":"Nature."},{"issue":"11","key":"pcbi.1013239.ref002","doi-asserted-by":"crossref","first-page":"2628","DOI":"10.1073\/pnas.1708272114","article-title":"Opinion: is science really facing a reproducibility crisis, and do we need it to?","volume":"115","author":"D Fanelli","year":"2018","journal-title":"Proc Natl Acad Sci U S A."},{"key":"pcbi.1013239.ref003","unstructured":"of Sciences NA, Policy G, Affairs G, Research Data B, Information et al.. Reproducibility and replicability in science. National Academies Press. 2019."},{"issue":"2","key":"pcbi.1013239.ref004","doi-asserted-by":"crossref","DOI":"10.15252\/msb.20209982","article-title":"Reproducibility in systems biology modelling","volume":"17","author":"K Tiwari","year":"2021","journal-title":"Mol Syst Biol."},{"key":"pcbi.1013239.ref005","unstructured":"Thacker BH, Doebling SW, Hemez FM, Anderson MC, Pepin JE, Rodriguez EA. 2004. https:\/\/www.osti.gov\/servlets\/purl\/835920"},{"key":"pcbi.1013239.ref006","doi-asserted-by":"crossref","first-page":"1201673","DOI":"10.3389\/fcell.2023.1201673","article-title":"Reproducibility and FAIR principles: the case of a segment polarity network model","volume":"11","author":"P Mendes","year":"2023","journal-title":"Front Cell Dev Biol."},{"key":"pcbi.1013239.ref007","article-title":"BioModels-15 years of sharing computational models in life science","volume":"48","author":"RS Malik-Sheriff","year":"2020","journal-title":"Nucleic Acids Res."},{"issue":"1","key":"pcbi.1013239.ref008","doi-asserted-by":"crossref","first-page":"2695","DOI":"10.1038\/s41598-023-29340-2","article-title":"Bayesian estimation reveals that reproducible models in Systems Biology get more citations","volume":"13","author":"S H\u00f6pfl","year":"2023","journal-title":"Sci Rep."},{"issue":"8","key":"pcbi.1013239.ref009","doi-asserted-by":"crossref","DOI":"10.15252\/msb.20199110","article-title":"SBML Level 3: an extensible format for the exchange and reuse of biological models","volume":"16","author":"SM Keating","year":"2020","journal-title":"Mol Syst Biol."},{"key":"pcbi.1013239.ref010","article-title":"BiGG Models: a platform for integrating, standardizing and sharing genome-scale models","volume":"44","author":"ZA King","year":"2016","journal-title":"Nucleic Acids Res."},{"issue":"7","key":"pcbi.1013239.ref011","doi-asserted-by":"crossref","first-page":"566","DOI":"10.1038\/nbt.4163","article-title":"KBase: The United States Department of Energy Systems Biology Knowledgebase","volume":"36","author":"AP Arkin","year":"2018","journal-title":"Nat Biotechnol."},{"issue":"9","key":"pcbi.1013239.ref012","doi-asserted-by":"crossref","first-page":"977","DOI":"10.1038\/nbt.1672","article-title":"High-throughput generation, optimization and analysis of genome-scale metabolic models","volume":"28","author":"CS Henry","year":"2010","journal-title":"Nat Biotechnol."},{"key":"pcbi.1013239.ref013","unstructured":"List of systems biology modeling software. Wikipedia. 2025. https:\/\/en.wikipedia.org\/wiki\/List_of_systems_biology_modeling_software"},{"key":"pcbi.1013239.ref014","unstructured":"Hucka M, Keating S, Smith L, Bergmann F, Tabira A, Hoops S. sbmlteam\/sbml-test-suite: v3.5.0. 2025."},{"key":"pcbi.1013239.ref015","unstructured":"Bergmann F, Keating S, Smith L, Hucka M. The SBML test suite database. 2025. https:\/\/sbml.bioquant.uni-heidelberg.de\/Simulator"},{"issue":"17","key":"pcbi.1013239.ref016","doi-asserted-by":"crossref","first-page":"1963","DOI":"10.1093\/bioinformatics\/btn319","article-title":"Comparing simulation results of SBML capable simulators","volume":"24","author":"FT Bergmann","year":"2008","journal-title":"Bioinformatics."},{"key":"pcbi.1013239.ref017","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1186\/1752-0509-5-198","article-title":"Reproducible computational biology experiments with SED-ML\u2013the simulation experiment description markup language","volume":"5","author":"D Waltemath","year":"2011","journal-title":"BMC Syst Biol."},{"issue":"6","key":"pcbi.1013239.ref018","doi-asserted-by":"crossref","first-page":"1650035","DOI":"10.1142\/S0219720016500359","article-title":"phraSED-ML: a paraphrased, human-readable adaptation of SED-ML","volume":"14","author":"K Choi","year":"2016","journal-title":"J Bioinform Comput Biol."},{"issue":"3","key":"pcbi.1013239.ref019","doi-asserted-by":"crossref","first-page":"20210020","DOI":"10.1515\/jib-2021-0020","article-title":"OMEX metadata specification (version 1.2)","volume":"18","author":"JH Gennari","year":"2021","journal-title":"J Integr Bioinform."},{"issue":"12","key":"pcbi.1013239.ref020","doi-asserted-by":"crossref","first-page":"1509","DOI":"10.1038\/nbt1156","article-title":"Minimum information requested in the annotation of biochemical models (MIRIAM)","volume":"23","author":"N Le Nov\u00e8re","year":"2005","journal-title":"Nat Biotechnol."},{"issue":"1","key":"pcbi.1013239.ref021","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1186\/s12967-020-02540-4","article-title":"Credible practice of modeling and simulation in healthcare: ten rules from a multidisciplinary perspective","volume":"18","author":"A Erdemir","year":"2020","journal-title":"J Transl Med."},{"key":"pcbi.1013239.ref022","unstructured":"Food and Drug Administration, et al. 2021. https:\/\/www.fda.gov\/regulatory-information\/search-fda-guidance-documents\/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions"},{"issue":"8","key":"pcbi.1013239.ref023","doi-asserted-by":"crossref","first-page":"804","DOI":"10.1002\/psp4.12669","article-title":"Scientific and regulatory evaluation of mechanistic in silico drug and disease models in drug development: Building model credibility","volume":"10","author":"FT Musuamba","year":"2021","journal-title":"CPT Pharmacometrics Syst Pharmacol."},{"key":"pcbi.1013239.ref024","doi-asserted-by":"crossref","DOI":"10.1093\/nar\/gkac331","article-title":"BioSimulators: a central registry of simulation engines and services for recommending specific tools","volume":"50","author":"B Shaikh","year":"2022","journal-title":"Nucleic Acids Res."},{"issue":"24","key":"pcbi.1013239.ref025","doi-asserted-by":"crossref","first-page":"3067","DOI":"10.1093\/bioinformatics\/btl485","article-title":"COPASI\u2013a COmplex PAthway SImulator","volume":"22","author":"S Hoops","year":"2006","journal-title":"Bioinformatics."},{"key":"pcbi.1013239.ref026","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1016\/j.biosystems.2018.07.006","article-title":"Tellurium: An extensible python-based modeling environment for systems and synthetic biology","volume":"171","author":"K Choi","year":"2018","journal-title":"Biosystems."},{"key":"pcbi.1013239.ref027","doi-asserted-by":"crossref","unstructured":"Loew LM, Schaff JC, Slepchenko BM, Moraru II. The virtual cell project. Systems Biomedicine. Elsevier. 2010. p. 273\u201388. https:\/\/doi.org\/10.1016\/b978-0-12-372550-9.00011-0","DOI":"10.1016\/B978-0-12-372550-9.00011-0"},{"issue":"2","key":"pcbi.1013239.ref028","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1002\/wsbm.165","article-title":"Virtual cell: computational tools for modeling in cell biology","volume":"4","author":"DC Resasco","year":"2012","journal-title":"Wiley Interdiscip Rev Syst Biol Med."},{"issue":"4","key":"pcbi.1013239.ref029","doi-asserted-by":"crossref","first-page":"560","DOI":"10.1093\/bioinformatics\/bti046","article-title":"Modelling cellular systems with PySCeS","volume":"21","author":"BG Olivier","year":"2005","journal-title":"Bioinformatics."},{"issue":"20","key":"pcbi.1013239.ref030","doi-asserted-by":"crossref","first-page":"3676","DOI":"10.1093\/bioinformatics\/btab227","article-title":"AMICI: high-performance sensitivity analysis for large ordinary differential equation models","volume":"37","author":"F Fr\u00f6hlich","year":"2021","journal-title":"Bioinformatics."},{"key":"pcbi.1013239.ref031","doi-asserted-by":"crossref","unstructured":"Folk M, Heber G, Koziol Q, Pourmal E, Robinson D. An overview of the HDF5 technology suite and its applications. In: Proceedings of the EDBT\/ICDT 2011 Workshop on Array Databases. 2011. p. 36\u201347. https:\/\/doi.org\/10.1145\/1966895.1966900","DOI":"10.1145\/1966895.1966900"},{"issue":"12","key":"pcbi.1013239.ref032","article-title":"Curating models from BioModels: developing a workflow for creating OMEX files","volume":"19","author":"J Xu","year":"2024","journal-title":"PLoS One."},{"key":"pcbi.1013239.ref033","article-title":"Kinetic simulation algorithm ontology","author":"A Zhukova","year":"2011","journal-title":"Nature Precedings."},{"issue":"7825","key":"pcbi.1013239.ref034","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1038\/s41586-020-2649-2","article-title":"Array programming with NumPy","volume":"585","author":"CR Harris","year":"2020","journal-title":"Nature."},{"issue":"4","key":"pcbi.1013239.ref035","doi-asserted-by":"crossref","first-page":"370","DOI":"10.1038\/nbt0402-370","article-title":"Computational modeling of the dynamics of the MAP kinase cascade activated by surface and internalized EGF receptors","volume":"20","author":"B Schoeberl","year":"2002","journal-title":"Nat Biotechnol."},{"key":"pcbi.1013239.ref036","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1016\/0143-4160(91)90010-C","article-title":"Signal-induced Ca2+ oscillations: properties of a model based on Ca(2+)-induced Ca2+ release","volume":"12","author":"G Dupont","year":"1991","journal-title":"Cell Calcium."},{"key":"pcbi.1013239.ref037","unstructured":"Raman K, Kratochvil M, Olivier BG, K\u00f6nig M, Sengupta P, Kuppa Baskaran DK, et al. FROG analysis ensures the reproducibility of genome scale metabolic models. bioRxiv. 2024;:2024\u201309."},{"key":"pcbi.1013239.ref038","doi-asserted-by":"crossref","first-page":"4893","DOI":"10.1038\/ncomms5893","article-title":"An exact arithmetic toolbox for a consistent and reproducible structural analysis of metabolic network models","volume":"5","author":"L Chindelevitch","year":"2014","journal-title":"Nat Commun."},{"issue":"10","key":"pcbi.1013239.ref039","doi-asserted-by":"crossref","first-page":"831","DOI":"10.15252\/msb.20156157","article-title":"Do genome-scale models need exact solvers or clearer standards?","volume":"11","author":"A Ebrahim","year":"2015","journal-title":"Mol Syst Biol."},{"issue":"10","key":"pcbi.1013239.ref040","doi-asserted-by":"crossref","first-page":"830","DOI":"10.15252\/msb.20156548","article-title":"Reply to \u201cDo genome-scale models need exact solvers or clearer standards?\u201d","volume":"11","author":"L Chindelevitch","year":"2015","journal-title":"Mol Syst Biol."}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1013239","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,4]],"date-time":"2025-12-04T19:02:59Z","timestamp":1764874979000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1013239"}},"subtitle":[],"editor":[{"given":"Marc R.","family":"Birtwistle","sequence":"first","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2025,12,4]]},"references-count":40,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2025,12,4]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1013239","relation":{},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,4]]}}}