{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,22]],"date-time":"2026-08-22T05:25:22Z","timestamp":1787376322091,"version":"build-2736575974"},"reference-count":70,"publisher":"National Institute for Health and Care Research","license":[{"start":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T00:00:00Z","timestamp":1781568000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Medical Research Council","award":["MR\/Z503915\/1"],"award-info":[{"award-number":["MR\/Z503915\/1"]}]},{"DOI":"10.13039\/501100019219","name":"NIHR Applied Research Collaboration South West Peninsula","doi-asserted-by":"crossref","award":["NIHR200167"],"award-info":[{"award-number":["NIHR200167"]}],"id":[{"id":"10.13039\/501100019219","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["openresearch.nihr.ac.uk"],"crossmark-restriction":false},"short-container-title":["NIHR Open Res"],"abstract":"<ns5:p>Discrete-event simulation (DES) is a popular technique for exploring problems in healthcare. For these models to be reused and have lasting impact, they need to be reproducible, transparent and well structured. Reproducible analytical pipeline (RAP) approaches are structured robust workflows that ensure analyses can be reproduced. They have emerged as best practice, but modellers struggle to implement them due to gaps in accessible guidance, skills, and time. This paper presents an integrated set of resources designed to help modellers bridge this implementation gap: an open-access e-book and four worked example repositories demonstrating complete RAP workflows for DES in both Python and R. The online book provides step-by-step guidance through nine major sections covering introductory material, project set-up, model inputs, model building, output analysis, experimentation, verification and validation, style and documentation, and collaboration and sharing. The case studies demonstrate varying complexity: a classic M\/M\/s queueing model, and a replication of a stroke care pathway model. The worked examples serve dual purposes: they demonstrate that RAP principles are achievable for healthcare DES models (from canonical queueing systems to real-world clinical pathways), and they provide templates that modelling teams can adopt and adapt within routine decision-support projects.<\/ns5:p>","DOI":"10.3310\/nihropenres.14296.1","type":"journal-article","created":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T06:29:08Z","timestamp":1781591348000},"page":"68","update-policy":"https:\/\/doi.org\/10.3310\/nihropenres.crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Reproducible analytical pipelines for healthcare discrete\u2011event simulation: An open guide and worked examples"],"prefix":"10.3310","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6596-3479","authenticated-orcid":false,"given":"Amy","family":"Heather","sequence":"first","affiliation":[{"name":"Peninsula Collaboration for Health Operational Research and Data Science Exeter England University of Exeter Medical School"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2631-4481","authenticated-orcid":false,"given":"Thomas","family":"Monks","sequence":"additional","affiliation":[{"name":"Peninsula Collaboration for Health Operational Research and Data Science Exeter England University of Exeter Medical School"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5274-5037","authenticated-orcid":false,"given":"Alison","family":"Harper","sequence":"additional","affiliation":[{"name":"Centre for Simulation, Analytics, and Modelling Exeter England University of Exeter Business School"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-0252-560X","authenticated-orcid":false,"given":"Fatemeh","family":"Alidoost","sequence":"additional","affiliation":[{"name":"Centre for Simulation, Analytics, and Modelling Exeter England University of Exeter Business School"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert","family":"Challen","sequence":"additional","affiliation":[{"name":"School of Engineering Mathematics and Technology Bristol England University of Bristol Faculty of Science and Engineering"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-0838-7499","authenticated-orcid":false,"given":"Thomas","family":"Slater","sequence":"additional","affiliation":[{"name":"Exeter England University of Exeter Department of Mathematics and Statistics"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Navonil","family":"Mustafee","sequence":"additional","affiliation":[{"name":"Centre for Simulation, Analytics, and Modelling Exeter England University of Exeter Business School"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"6221","published-online":{"date-parts":[[2026,6,16]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.5960048","article-title":"Quarto.","author":"J Allaire","year":"2024","journal-title":"Zenodo."},{"key":"ref2","doi-asserted-by":"publisher","first-page":"e01801","DOI":"10.1002\/bes2.1801","article-title":"A Beginner\u2019s Guide to Conducting Reproducible Research.","volume":"102","author":"J Alston","year":"2021","journal-title":"Bull. Ecol. Soc. Am."},{"key":"ref3","article-title":"Reproducible Analytical Pipelines (RAP) \u2013 Government Analysis Function.","year":"2025"},{"key":"ref4","article-title":"Artifact Review and Badging - Current.","year":"Aug. 2020"},{"key":"ref5","doi-asserted-by":"crossref","DOI":"10.31219\/osf.io\/bnh7p","article-title":"Introducing a Framework for Open and Reproducible Research Training (FORRT).","author":"F Azevedo","year":"Dec. 2019"},{"key":"ref6","doi-asserted-by":"crossref","first-page":"452-454","DOI":"10.1038\/533452a","article-title":"1,500 scientists lift the lid on reproducibility.","volume":"533","author":"M Baker","year":"May 2016","journal-title":"Nature."},{"key":"ref7","doi-asserted-by":"publisher","first-page":"335-393","DOI":"10.1002\/9780470172445.ch10","author":"O Balci","year":"1998","journal-title":"Handbook of Simulation."},{"key":"ref8","doi-asserted-by":"publisher","first-page":"e66398-e66398","DOI":"10.2196\/66398","article-title":"Digital Capability, Open-Source Use, and Interoperability Standards Within the National Health Service in England: Survey of Health Care Trusts.","volume":"12","author":"M Bennion","year":"Aug. 2025","journal-title":"JMIR Hum. Factors."},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.3389\/fninf.2017.00069\/full","article-title":"Re-run, Repeat, Reproduce, Reuse, Replicate: Transforming Code into Scientific Contributions.","volume":"11","author":"F Benureau","year":"Jan. 2018","journal-title":"Front. Neuroinform."},{"key":"ref10","doi-asserted-by":"publisher","first-page":"e13933","DOI":"10.7717\/peerj.13933","article-title":"A survey of researchers\u2019 code sharing and code reuse practices, and assessment of interactive notebook prototypes.","volume":"10","author":"L Cadwallader","year":"Aug. 2022","journal-title":"PeerJ."},{"key":"ref11","doi-asserted-by":"publisher","first-page":"1-34","DOI":"10.18637\/jss.v064.i04","article-title":"fitdistrplus: An R Package for Fitting Distributions.","volume":"64","author":"M Delignette-Muller","year":"Mar. 2015","journal-title":"J. Stat. Softw."},{"key":"ref12","article-title":"About CHARTED.","year":"2026"},{"key":"ref13","doi-asserted-by":"publisher","first-page":"273-276","DOI":"10.1017\/S1049096516000196","article-title":"Lessons from a Decade of Replications at the Quarterly Journal of Political Science.","volume":"49","author":"N Eubank","year":"Apr. 2016","journal-title":"PS: Political Science & Politics."},{"key":"ref14","doi-asserted-by":"publisher","first-page":"1343-1356","DOI":"10.1287\/mnsc.2023.03556","article-title":"Reproducibility in Management Science.","volume":"70","author":"M Fi\u0161ar","year":"Mar. 2024","journal-title":"Manag. Sci."},{"key":"ref15","article-title":"flake8.","year":"2025"},{"key":"ref16","article-title":"Computational reproducibility.","year":"Feb. 2026"},{"key":"ref17","doi-asserted-by":"publisher","first-page":"e1007854","DOI":"10.1371\/journal.pcbi.1007854","article-title":"Ten simple rules for making training materials FAIR.","volume":"16","author":"L Garcia","year":"May 2020","journal-title":"PLoS Comput. Biol."},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.12688\/f1000research.172013.1","article-title":"Survey about Barriers and Solutions for Enhancing Computational Reproducibility in Scientific Research.","author":"Y Gelsleichter","year":"Nov. 2025"},{"key":"ref19","article-title":"Better, broader, safer: using health data for research and analysis.","author":"B Goldacre","year":"Apr. 2022"},{"key":"ref20","article-title":"The AQuA Book.","year":"July 2025"},{"key":"ref22","doi-asserted-by":"publisher","first-page":"129-139","DOI":"10.1016\/j.ecolmodel.2014.01.018","article-title":"Towards better modelling and decision support: Documenting model development, testing, and analysis using TRACE.","volume":"280","author":"V Grimm","year":"May 2014","journal-title":"Ecol. Model."},{"key":"ref23","doi-asserted-by":"publisher","first-page":"197-213","DOI":"10.1016\/j.ejor.2020.06.043","article-title":"Facets of trust in simulation studies.","volume":"289","author":"A Harper","year":"Feb. 2021","journal-title":"Eur. J. Oper. Res."},{"key":"ref24","doi-asserted-by":"crossref","first-page":"357-362","DOI":"10.1038\/s41586-020-2649-2","article-title":"Array programming with NumPy.","volume":"585","author":"C Harris","year":"Sept. 2020","journal-title":"Nature."},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.18888199","article-title":"Simple M\/M\/s queuing model: Python DES RAP.","author":"A Heather","year":"Mar. 2026a"},{"key":"ref26","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.18889030","article-title":"Simple M\/M\/s queuing model: R DES RAP.","author":"A Heather","year":"Mar. 2026b"},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.18608967","article-title":"Stroke capacity planning model: python DES RAP.","author":"A Heather","year":"Feb. 2026c"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.18609825","article-title":"Stroke capacity planning model: R DES RAP.","author":"A Heather","year":"Feb. 2026d"},{"key":"ref29","doi-asserted-by":"publisher","first-page":"1-25","DOI":"10.1080\/17477778.2025.2552177","article-title":"On the reproducibility of discrete-event simulation studies in health research: an empirical study using open models.","author":"A Heather","year":"Sept. 2025","journal-title":"Journal of Simulation."},{"key":"ref30","article-title":"Supplementary Materials for: \u201cReproducible Analytical Pipelines for Healthcare Discrete-Event Simulation: An Open Guide and Worked Examples\u201d.","author":"A Heather","year":"2026a","journal-title":"OSF."},{"key":"ref31","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.18714084","article-title":"DES RAP Book: Reproducible Discrete-Event Simulation in Python and R.","author":"A Heather","year":"Feb. 2026b"},{"key":"ref32","doi-asserted-by":"publisher","first-page":"100743","DOI":"10.1016\/j.epidem.2024.100743","article-title":"Reproducibility of COVID-era infectious disease models.","volume":"46","author":"A Henderson","year":"Mar. 2024","journal-title":"Epidemics."},{"key":"ref33","doi-asserted-by":"publisher","first-page":"813-820","DOI":"10.1016\/j.jval.2025.01.019","article-title":"Mapping the Landscape of Open Source Health Economic Models: A Systematic Database Review and Analysis: An ISPOR Special Interest Group Report.","volume":"28","author":"R Henderson","year":"June 2025","journal-title":"Value Health."},{"key":"ref34","doi-asserted-by":"publisher","first-page":"7240","DOI":"10.21105\/joss.07240","article-title":"Static Code Analysis for R.","volume":"10","author":"J Hester","year":"2025","journal-title":"J. Open Source Softw."},{"key":"ref35","doi-asserted-by":"publisher","first-page":"1632-1644","DOI":"10.1057\/jors.2009.121","article-title":"Automated selection of the number of replications for a discrete-event simulation.","volume":"61","author":"K Hoad","year":"Nov. 2010","journal-title":"J. Oper. Res. Soc."},{"key":"ref36","doi-asserted-by":"crossref","DOI":"10.31219\/osf.io\/f73a6","article-title":"A survey of code sharing practice and policy in computational biology.","author":"I Hrynaszkiewicz","year":"Apr. 2021"},{"key":"ref37","doi-asserted-by":"publisher","first-page":"20200069","DOI":"10.1098\/rsta.2020.0069","article-title":"Learning from reproducing computational results: introducing three principles and the Reproduction Package.","volume":"379","author":"M Krafczyk","year":"Mar. 2021","journal-title":"Philos. Trans. R. Soc. A Math. Phys. Eng. Sci."},{"key":"ref38","article-title":"pytest.","author":"H Krekel","year":"2004"},{"key":"ref39","article-title":"simEd: Simulation Education.","author":"B Lawson","year":"Sept. 2025"},{"key":"ref40","doi-asserted-by":"publisher","first-page":"231003","DOI":"10.1098\/rsos.231003","article-title":"Replicability of simulation studies for the investigation of statistical methods: the RepliSims project.","volume":"11","author":"K Luijken","year":"Jan. 2024","journal-title":"R. Soc. Open Sci."},{"key":"ref41","doi-asserted-by":"publisher","first-page":"1-16","DOI":"10.1080\/17477778.2023.2260772","article-title":"Computer model and code sharing practices in healthcare discrete-event simulation: a systematic scoping review.","author":"T Monks","year":"2023","journal-title":"Journal of Simulation."},{"key":"ref42","doi-asserted-by":"publisher","first-page":"530","DOI":"10.1186\/s12913-016-1789-4","article-title":"A modelling tool for capacity planning in acute and community stroke services.","volume":"16","author":"T Monks","year":"Sept. 2016","journal-title":"BMC Health Serv. Res."},{"key":"ref43","doi-asserted-by":"publisher","first-page":"55-67","DOI":"10.1080\/17477778.2018.1442155","article-title":"Strengthening the reporting of empirical simulation studies: Introducing the STRESS guidelines.","volume":"13","author":"T Monks","year":"Jan. 2019","journal-title":"Journal of Simulation."},{"key":"ref44","doi-asserted-by":"publisher","first-page":"1-20","DOI":"10.1080\/17477778.2024.2347882","article-title":"Towards sharing tools and artefacts for reusable simulations in healthcare.","author":"T Monks","year":"2024","journal-title":"Journal of Simulation."},{"key":"ref45","doi-asserted-by":"publisher","first-page":"1-24","DOI":"10.1080\/01605682.2025.2554751","article-title":"Unlocking the potential of past research: using generative AI to reconstruct healthcare simulation models.","author":"T Monks","year":"2025","journal-title":"J. Oper. Res. Soc."},{"key":"ref46","article-title":"sim-tools: fundamental tools to support the simulation process in python.","author":"T Monks","year":"Jan. 2026"},{"key":"ref47","doi-asserted-by":"publisher","first-page":"2323","DOI":"10.23889\/ijpds.v8i2.2323","article-title":"Introducing best practice for reproducibility in government.","volume":"8","author":"M Munro","year":"Sept. 2023","journal-title":"International Journal of Population Data Science."},{"key":"ref48","article-title":"nbQA.","year":"2026"},{"key":"ref49","article-title":"NHS service standard.","year":"Jan. 2026"},{"key":"ref50","article-title":"Levels of RAP.","year":"July 2025a"},{"key":"ref51","article-title":"RAP Community of Practice.","year":"2025b"},{"key":"ref52","doi-asserted-by":"publisher","first-page":"e143","DOI":"10.1017\/S0140525X18000791","article-title":"Verify original results through reanalysis before replicating.","volume":"41","author":"M Nuijten","year":"Jan. 2018","journal-title":"Behav. Brain Sci."},{"key":"ref53","article-title":"Style guide.","year":"2026"},{"key":"ref55","doi-asserted-by":"publisher","first-page":"473-479","DOI":"10.1016\/j.jval.2021.10.001","article-title":"Opportunities and Barriers to the Development and Use of Open Source Health Economic Models: A Survey.","volume":"25","author":"X Pouwels","year":"Apr. 2022","journal-title":"Value Health."},{"key":"ref56","article-title":"pylint.","year":"2026"},{"key":"ref58","doi-asserted-by":"publisher","first-page":"1-22","DOI":"10.1080\/17477778.2026.2663849","article-title":"Vidigi: a Python library for interactive-animated visualisations of discrete event simulation models.","author":"S Rosser","year":"2026","journal-title":"J. Simul."},{"key":"ref59","article-title":"HSMA - little book of DES.","author":"S Rosser","year":"Sept. 2025"},{"key":"ref60","doi-asserted-by":"publisher","first-page":"e1003285","DOI":"10.1371\/journal.pcbi.1003285","article-title":"Ten Simple Rules for Reproducible Computational Research.","volume":"9","author":"G Sandve","year":"Oct. 2013","journal-title":"PLoS Comput. Biol."},{"key":"ref61","article-title":"Online sustainability evaluation.","year":"2026"},{"key":"ref62","doi-asserted-by":"publisher","first-page":"2584-2589","DOI":"10.1073\/pnas.1708290115","article-title":"An empirical analysis of journal policy effectiveness for computational reproducibility.","volume":"115","author":"V Stodden","year":"Mar. 2018","journal-title":"Proc. Natl. Acad. Sci."},{"key":"ref63","article-title":"distfit: a python library for probability density fitting.","author":"E Taskesen","year":"June 2025"},{"key":"ref66","article-title":"Standards.","year":"Feb. 2026"},{"key":"ref67","article-title":"Tidyverse style guide.","year":"2025"},{"key":"ref68","article-title":"Guide for Reproducible Research.","year":"2025"},{"key":"ref69","doi-asserted-by":"publisher","first-page":"1-30","DOI":"10.18637\/jss.v090.i02","article-title":"Discrete-Event Simulation for R.","volume":"90","author":"I Ucar","year":"July 2019","journal-title":"J. Stat. Softw."},{"key":"ref70","first-page":"1843500 Bytes","article-title":"Survey of Wellcome researchers and their attitudes to open research.","author":"V Van den Eynden","year":"2016","journal-title":"Figshare."},{"key":"ref71","article-title":"PEP 8 \u2013 Style Guide for Python Code.","author":"G Rossum","year":"July 2001"},{"key":"ref72","doi-asserted-by":"publisher","first-page":"12262","DOI":"10.3390\/ijerph182212262","article-title":"Discrete-Event Simulation Modeling in Healthcare: A Comprehensive Review.","volume":"18","author":"J V\u00e1zquez-Serrano","year":"Nov. 2021","journal-title":"Int. J. Environ. Res. Public Health."},{"key":"ref73","doi-asserted-by":"publisher","first-page":"5","DOI":"10.32614\/RJ-2011-002","article-title":"testthat: Get Started with Testing.","volume":"3","author":"H Wickham","year":"2011","journal-title":"The R Journal."},{"key":"ref74","article-title":"and posit. roxygen2.","author":"H Wickham","year":"2025"},{"key":"ref75","doi-asserted-by":"publisher","first-page":"506-514","DOI":"10.1016\/j.jval.2020.01.005","article-title":"Reporting Quality of Discrete Event Simulations in Healthcare\u2014Results From a Generic Reporting Checklist.","volume":"23","author":"X Zhang","year":"Apr. 2020","journal-title":"Value Health."}],"container-title":["NIHR Open Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/openresearch.nihr.ac.uk\/articles\/6-68\/v1\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/openresearch.nihr.ac.uk\/articles\/6-68\/v1\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/openresearch.nihr.ac.uk\/articles\/6-68\/v1\/iparadigms","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,22]],"date-time":"2026-08-22T04:39:45Z","timestamp":1787373585000},"score":1,"resource":{"primary":{"URL":"https:\/\/openresearch.nihr.ac.uk\/articles\/6-68\/v1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,16]]},"references-count":70,"URL":"https:\/\/doi.org\/10.3310\/nihropenres.14296.1","relation":{"has-review":[{"id-type":"doi","id":"10.3310\/nihropenres.15573.r41350","asserted-by":"subject"},{"id-type":"doi","id":"10.3310\/nihropenres.15573.r41350","asserted-by":"object"}]},"ISSN":["2633-4402"],"issn-type":[{"value":"2633-4402","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,16]]},"assertion":[{"value":"Approved","URL":"https:\/\/openresearch.nihr.ac.uk\/articles\/6-68\/v1#article-reports","order":0,"name":"referee-status","label":"Referee status","group":{"name":"current-referee-status","label":"Current Referee Status"}},{"value":"10.3310\/nihropenres.15573.r41350, S M Atikur Rahman, University of Texas at El Paso, El Paso, TX, 22 Aug 2026, version 1, 1 approved","URL":"https:\/\/openresearch.nihr.ac.uk\/articles\/6-68\/v1#referee-response-41350","order":0,"name":"referee-response-41350","label":"Referee Report","group":{"name":"article-reports","label":"Article Reports"}},{"value":"This project is funded by the National Institute for Health and Care Research (NIHR) under the NIHR Applied Research Collaboration South West Peninsula (Grant Reference Number NIHR200167). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. \r\n\r\nThis work was also supported by the Medical Research Council under grant number\r\n(MR\/Z503915\/1). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.","order":1,"name":"grant-information","label":"Grant Information"},{"value":"This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.","order":0,"name":"copyright-info","label":"Copyright"}]}}