{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,3]],"date-time":"2026-08-03T06:34:39Z","timestamp":1785738879093,"version":"3.56.0"},"update-to":[{"DOI":"10.3310\/nihropenres.13467.1","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2023,12,15]],"date-time":"2023-12-15T00:00:00Z","timestamp":1702598400000}}],"reference-count":48,"publisher":"National Institute for Health and Care Research","license":[{"start":{"date-parts":[[2023,12,15]],"date-time":"2023-12-15T00:00:00Z","timestamp":1702598400000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100019219","name":"National Institute for Health Research Applied Research Collaboration South West Peninsula","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100019219","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["openresearch.nihr.ac.uk"],"crossmark-restriction":false},"short-container-title":["NIHR Open Res"],"abstract":"<ns3:p>One aim of Open Science is to increase the accessibility of research. Within health services research that uses discrete-event simulation, Free and Open Source Software (FOSS), such as Python, offers a way for research teams to share their models with other researchers and NHS decision makers. Although the code for healthcare discrete-event simulation models can be shared alongside publications, it may require specialist skills to use and run. This is a disincentive to researchers adopting Free and Open Source Software and open science practices. Building on work from other health data science disciplines, we propose that web apps offer a user-friendly interface for healthcare models that increase the accessibility of research to the NHS, and researchers from other disciplines. We focus on models coded in Python deployed as streamlit web apps. To increase uptake of these methods, we provide an approach to structuring discrete-event simulation model code in Python so that models are web app ready. The method is general across discrete-event simulation Python packages, and we include code for both simpy and ciw implementations of a simple urgent care call centre model. We then provide a step-by-step tutorial for linking the model to a streamlit web app interface, to enable other health data science researchers to reproduce and implement our method.<\/ns3:p>","DOI":"10.3310\/nihropenres.13467.2","type":"journal-article","created":{"date-parts":[[2023,12,15]],"date-time":"2023-12-15T05:40:09Z","timestamp":1702618809000},"page":"48","update-policy":"https:\/\/doi.org\/10.3310\/nihropenres.crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Improving the usability of open health service delivery simulation models using Python and web apps"],"prefix":"10.3310","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2631-4481","authenticated-orcid":false,"given":"Thomas","family":"Monks","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5274-5037","authenticated-orcid":false,"given":"Alison","family":"Harper","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"6221","published-online":{"date-parts":[[2023,12,15]]},"reference":[{"key":"ref-1","doi-asserted-by":"publisher","first-page":"479-94","DOI":"10.1016\/j.simpat.2003.11.006","article-title":"Simulation model reuse: definitions, benefits and obstacles.","volume":"12","author":"S Robinson","year":"2004","journal-title":"Simul Model Pract Theory."},{"key":"ref-2","doi-asserted-by":"publisher","first-page":"38-45","DOI":"10.1136\/bmjqs-2015-004430","article-title":"Systems modelling and simulation in health service design, delivery and decision making.","volume":"25","author":"M Pitt","year":"2016","journal-title":"BMJ Qual Saf."},{"key":"ref-3","doi-asserted-by":"publisher","first-page":"937-949","DOI":"10.1007\/s40273-017-0523-3","article-title":"Simulation Modelling in Healthcare: An Umbrella Review of Systematic Literature Reviews.","volume":"35","author":"S Salleh","year":"2017","journal-title":"Pharmacoeconomics."},{"key":"ref-4","doi-asserted-by":"publisher","DOI":"10.3390\/ijerph182212262","article-title":"Discrete-Event Simulation Modeling in Healthcare: A Comprehensive Review.","volume":"18","author":"J V\u00e1zquez-Serrano","year":"2021","journal-title":"Int J Environ Res Public Health."},{"key":"ref-5","doi-asserted-by":"publisher","first-page":"417-33","DOI":"10.3390\/modelling3040027","article-title":"Discrete-Event Simulation in Healthcare Settings: A Review.","volume":"3","author":"J Forbus","year":"2022","journal-title":"Modelling."},{"key":"ref-6","doi-asserted-by":"publisher","first-page":"2342-64","DOI":"10.1080\/01605682.2020.1790306","article-title":"Healthcare services: A systematic review of patient-centric logistics issues using simulation.","volume":"72","author":"S Roy","year":"2021","journal-title":"J Oper Res Soc."},{"key":"ref-7","doi-asserted-by":"publisher","first-page":"55-67","DOI":"10.1080\/17477778.2018.1442155","article-title":"Strengthening the reporting of empirical simulation studies: Introducing the STRESS guidelines.","volume":"13","author":"T Monks","year":"2019","journal-title":"J Simulation."},{"key":"ref-8","doi-asserted-by":"publisher","DOI":"10.1080\/17477778.2023.2260772","article-title":"Computer model and code sharing practices in health-care discrete-event simulation: a systematic scoping review.","author":"T Monks","year":"2023","journal-title":"J Simulation."},{"key":"ref-9","doi-asserted-by":"publisher","first-page":"473-479","DOI":"10.1016\/j.jval.2021.10.001","article-title":"Opportunities and Barriers to the Development and Use of Open Source Health Economic Models: A Survey.","volume":"25","author":"X Pouwels","year":"2022","journal-title":"Value Health."},{"key":"ref-10","doi-asserted-by":"publisher","first-page":"253-76","DOI":"10.1016\/j.simpat.2009.09.013","article-title":"A  review  of  Web-based  simulation and supporting tools.","volume":"18","author":"J Byrne","year":"2010","journal-title":"Simul Model Pract Theory."},{"key":"ref-11","doi-asserted-by":"publisher","first-page":"193-206","DOI":"10.1057\/jos.2015.9","article-title":"A review of open source discrete event simulation software for operations research.","volume":"10","author":"G Dagkakis","year":"2016","journal-title":"J Simulation."},{"key":"ref-12","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0237628","article-title":"A simulation modelling toolkit for organising outpatient dialysis services during the COVID-19 pandemic.","volume":"15","author":"M Allen","year":"2020","journal-title":"PLoS One."},{"key":"ref-13","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2020-043795","article-title":"Modelling palliative and end-of-life resource requirements during COVID-19: implications for quality care.","volume":"11","author":"D Chalk","year":"2021","journal-title":"BMJ Open."},{"key":"ref-14","doi-asserted-by":"publisher","first-page":"1223-34","DOI":"10.1109\/WSC57314.2022.10015462","article-title":"FACS-CHARM: A Hybrid Agent-Based and Discrete-Event Simulation Approach for Covid-19 Management at Regional Level","author":"A Anagnostou","year":"2022"},{"key":"ref-15","doi-asserted-by":"publisher","DOI":"10.1186\/s12913-022-08433-0","article-title":"Optimising acute stroke pathways through flexible use of bed capacity: a computer modelling study.","volume":"22","author":"R Wood","year":"2022","journal-title":"BMC Health Serv Res."},{"key":"ref-16","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.3794811","article-title":"Leveraging Multi-tier Healthcare Facility Network Simulations for Capacity Planning in a Pandemic","author":"S Mohd","year":"2021"},{"key":"ref-17","doi-asserted-by":"publisher","first-page":"1-16","DOI":"10.1080\/01605682.2023.2199094","article-title":"Here\u2019s something we prepared earlier: Development, use and reuse of a configurable, inter-disciplinary approach for tackling overcrowding in NHS hospitals.","author":"S Crowe","year":"2023","journal-title":"J Oper Res Soc."},{"key":"ref-18","doi-asserted-by":"publisher","first-page":"1005-1009","DOI":"10.1007\/s40273-022-01179-x","article-title":"Developing an Online Infrastructure to Enhance Model Ac- cessibility and Validation: The Peer Models Network.","volume":"40","author":"S Harvard","year":"2022","journal-title":"Pharmacoeconomics."},{"key":"ref-19","first-page":"6","article-title":"Towards a Community Framework for Agent-Based Modelling.","volume":"11","author":"M Janssen","year":"2008","journal-title":"J Artif Soc Soc Simul."},{"key":"ref-20","doi-asserted-by":"publisher","DOI":"10.1201\/9780429328534","article-title":"Applied Mathematics with Open-Source Software","author":"V Knight","year":"2022"},{"key":"ref-21","first-page":"250-60","article-title":"A Framework to Share Healthcare Simulations on the Web Using Free and Open Source Tools and Python","author":"A Harper","year":"2023"},{"key":"ref-22","doi-asserted-by":"publisher","first-page":"69","DOI":"10.12688\/wellcomeopenres.15807.2","article-title":"Making health economic models Shiny: A tutorial [version 2; peer review: 2 approved].","volume":"5","author":"R Smith","year":"2020","journal-title":"Wellcome Open Res."},{"key":"ref-23","doi-asserted-by":"publisher","DOI":"10.1002\/psp4.21","article-title":"Interactive pharmacometric applications using R and the shiny package.","volume":"4","author":"J Wojciechowski","year":"2015","journal-title":"CPT Pharmacometrics Syst Pharmacol."},{"key":"ref-24","doi-asserted-by":"publisher","first-page":"2023-05","DOI":"10.1101\/2023.05.31.23290774","article-title":"Post-Covid Orthopaedic Elective Resource Planning using Simulation Modelling.","author":"A Harper","year":"2023","journal-title":"medRxiv."},{"key":"ref-25","doi-asserted-by":"publisher","first-page":"765-782","DOI":"10.1080\/17477778.2022.2081521","article-title":"Improving uptake of simulation in healthcare: User-driven development of an open-source tool for modelling patient flow.","volume":"17","author":"J Tyler","year":"2023","journal-title":"J Simul."},{"key":"ref-26","article-title":"SimPy 3.0.11","year":"2020"},{"key":"ref-27","doi-asserted-by":"publisher","DOI":"10.21105\/joss.00767","article-title":"salabim:  discrete event simulation and animation in Python.","volume":"3","author":"R van der Ham","year":"2018","journal-title":"J Open Source Softw."},{"key":"ref-28","doi-asserted-by":"publisher","first-page":"68-82","DOI":"10.1080\/17477778.2018.1473909","article-title":"An open-source discrete event simulation library.","volume":"13","author":"G Palmer","year":"2019","journal-title":"J Simul."},{"key":"ref-29","article-title":"NHS England Open Source Programme","year":"2022"},{"key":"ref-30","article-title":"Better, broader, safer: using health data for research and analysis","author":"B Goldacre","year":"2022"},{"key":"ref-32","doi-asserted-by":"publisher","first-page":"407-21","DOI":"10.1080\/17477778.2021.2015259","article-title":"Simulating emergency patient flow during the COVID-19 pandemic.","volume":"17","author":"T Bovim","year":"2023","journal-title":"J Simul."},{"key":"ref-33","doi-asserted-by":"publisher","first-page":"850-7","DOI":"10.1111\/1754-9485.13255","article-title":"Application of a computational model in simulating an endovascular clot retrieval service system within regional Australia.","volume":"65","author":"Y Ren","year":"2021","journal-title":"J Med Imaging Radiat Oncol."},{"key":"ref-34","doi-asserted-by":"publisher","first-page":"e400-13","DOI":"10.1016\/j.wneu.2020.05.168","article-title":"Geographic Service Delivery for Endovascular Clot Retrieval: Using Discrete Event Simulation to Optimize Resources.","volume":"141","author":"Y Ren","year":"2020","journal-title":"World Neurosurg."},{"key":"ref-35","doi-asserted-by":"publisher","first-page":"1-11","DOI":"10.1109\/WSC52266.2021.9715505","article-title":"Modeling Of Waiting Lists For Chronic Heart Failure In The Wake Of The COVID-19 Pandemic.","author":"A Wise","year":"2021"},{"key":"ref-36","doi-asserted-by":"publisher","first-page":"2690-700","DOI":"10.1109\/WSC.2018.8632236","article-title":"Modeling the Impact of Make-ahead Chemotherapy Drug Policies through Discrete-Event Simulation.","author":"D Richardson","year":"2018"},{"key":"ref-37","doi-asserted-by":"publisher","first-page":"360-79","DOI":"10.1002\/hpm.3589","article-title":"A discrete event simulation for improving operating theatre efficiency.","volume":"38","author":"H Hassanzadeh","year":"2023","journal-title":"Int J Health Plann Manage."},{"key":"ref-38","article-title":"pandas: a foundational Python library for data analysis and statistics.","volume":"14","author":"W McKinney","year":"2011","journal-title":"Python for High Performance and Scientific Computing."},{"key":"ref-39","doi-asserted-by":"publisher","first-page":"22-30","DOI":"10.1109\/MCSE.2011.37","article-title":"The NumPy Array: A Structure for Efficient Numerical Computation.","volume":"13","author":"S van der Walt","year":"2011","journal-title":"Comput Sci Eng."},{"key":"ref-40","doi-asserted-by":"publisher","first-page":"90-5","DOI":"10.1109\/MCSE.2007.55","article-title":"Matplotlib: A 2D graphics environment.","volume":"9","author":"J Hunter","year":"2007","journal-title":"Comput Sci Eng."},{"key":"ref-41","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.7470333","article-title":"The Turing Way: A handbook for reproducible, ethical and collaborative research.","year":"2022","journal-title":"Zenodo."},{"key":"ref-42","doi-asserted-by":"publisher","first-page":"517-31","DOI":"10.1016\/j.ejor.2016.03.050","article-title":"Test-driven simulation modelling: A case study using agent-based maritime search-operation simulation.","volume":"254","author":"B Onggo","year":"2016","journal-title":"Eur J Oper Res."},{"key":"ref-43","doi-asserted-by":"publisher","first-page":"430-6","DOI":"10.1038\/s41586-020-2521-4","article-title":"Factors associated with COVID-19-related death using Open-SAFELY.","volume":"584","author":"E Williamson","year":"2020","journal-title":"Nature."},{"key":"ref-44","doi-asserted-by":"publisher","first-page":"194","DOI":"10.12688\/wellcomeopenres.17933.2","article-title":"Living HTA: Automating Health Economic Evaluation with R [version 2; peer review: 2 approved].","volume":"7","author":"R Smith","year":"2022","journal-title":"Wellcome Open Res."},{"key":"ref-45","doi-asserted-by":"publisher","first-page":"1329-39","DOI":"10.1007\/s40273-019-00837-x","article-title":"A Need for Change! A Coding Framework for Improving Transparency in Decision Modeling.","volume":"37","author":"F Alarid-Escudero","year":"2019","journal-title":"Pharmacoeconomics."},{"key":"ref-46","doi-asserted-by":"publisher","first-page":"419","DOI":"10.12688\/wellcomeopenres.19656.1","article-title":"Packaging cost-effectiveness models in R: A tutorial [version 1; peer review: 2 approved with reservations].","volume":"8","author":"R Smith","year":"2023","journal-title":"Wellcome Open Res."},{"key":"ref-47","article-title":"Survey of Wellcome researchers and their attitudes to open research","author":"V den Eynden","year":"2016"},{"key":"ref-48","doi-asserted-by":"publisher","first-page":"321-42","DOI":"10.1007\/BF02136833","article-title":"Visual interactive simulation: A methodological perspective.","volume":"53","author":"P Bell","year":"1994","journal-title":"Ann Oper Res."},{"key":"ref-49","first-page":"11-15","article-title":"Exploring Network Structure, Dynamics, and Function using NetworkX","author":"A Hagberg","year":"2008"}],"container-title":["NIHR Open Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/openresearch.nihr.ac.uk\/articles\/3-48\/v2\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/openresearch.nihr.ac.uk\/articles\/3-48\/v2\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/openresearch.nihr.ac.uk\/articles\/3-48\/v2\/iparadigms","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,16]],"date-time":"2024-05-16T09:55:16Z","timestamp":1715853316000},"score":1,"resource":{"primary":{"URL":"https:\/\/openresearch.nihr.ac.uk\/articles\/3-48\/v2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,15]]},"references-count":48,"URL":"https:\/\/doi.org\/10.3310\/nihropenres.13467.2","relation":{"has-review":[{"id-type":"doi","id":"10.3310\/nihropenres.14669.r30915","asserted-by":"subject"},{"id-type":"doi","id":"10.3310\/nihropenres.14669.r30913","asserted-by":"subject"},{"id-type":"doi","id":"10.3310\/nihropenres.14669.r30915","asserted-by":"object"},{"id-type":"doi","id":"10.3310\/nihropenres.14669.r30913","asserted-by":"object"}]},"ISSN":["2633-4402"],"issn-type":[{"value":"2633-4402","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,15]]},"assertion":[{"value":"Indexed","URL":"https:\/\/openresearch.nihr.ac.uk\/articles\/3-48\/v2#article-reports","order":0,"name":"referee-status","label":"Referee status","group":{"name":"current-referee-status","label":"Current Referee Status"}},{"value":"10.3310\/nihropenres.14611.r30522, Geraint Palmer, Cardiff University, Cardiff, Wales, 11 Oct 2023, version 1, indexed","URL":"https:\/\/openresearch.nihr.ac.uk\/articles\/3-48\/v1#referee-response-30522","order":0,"name":"referee-response-30522","label":"Referee Report","group":{"name":"article-reports","label":"Article Reports"}},{"value":"<b>Thomas Monks<\/b>; \n<i>Posted: 15 Dec 2023<\/i>; \n<b>Response to Reviewer 1 (R1)  R1 This is a very good paper on an important topic, and highlights one tool that can be useful in facilitating open and accessible software. I learnt a lot here, and have only some minor comments on the manuscript.<\/b>  \n<i>Thank you for your review and feedback. We are very grateful to you for giving experience and time to improve the manuscript. We are also glad that you learnt something new in our material.<\/i>  \n<b>R1: For the code listings I recommend following following the PEP8 style guide (https:\/\/peps.python.org\/pep-0008\/). An easy way to do this is to run the script through Black, a Python linter (https:\/\/black.readthedocs.io\/en\/stable\/). This will make the listings much more readable and readily accessible to those familiar with Python.<\/b>  \n<i>Thank you for your advise to use Black. This is an excellent piece of software. We confirm that we have done as requested.&nbsp; Please note that there are instances where we have deviated slightly from Black's reformatting in order to better fit in with the journal's formatting requirements. For example, in Listing 5 the Black recommended that Line 26 was not split (it is only 70 characters in length). This is because the journal article cannot accommodate the full length of the line and wraps to a unlabelled line below.&nbsp; Instead we introduced a manual split in the article.&nbsp; We checked all changes from Black auto-formatting with Flake8 to make sure we did not introduce new PEP8 violations.&nbsp; We also checked and updated references to listing line numbers in the main text where needed.<\/i>  \n<b>R1: The &quot;Methods&quot; section on page 6 has a nice description of 'swap-able' code segments. Can you also add, that in the language of software development, this would be called 'modular' code<\/b>  \n<i>As requested we have described the code as modular.<\/i>  \n<b>R1: In Step 5 on page 16, a description of 'set_page_config' is given, but this does not appear in the listings. Similarly, line numbers of not given for 'st.columns(2)', only placeholders.<\/b>  \n<i>Thank you for identifying this mistake.&nbsp; We have removed the reference to `set_page_config` as we only intended to show an extract of the full listing here. We have also inserted the correct line number for `st.columns(2)`<\/i>  \n<b>R1: On page 11 under &quot;Code to test the model&quot;, it would be good to also&nbsp;mention that automated testing of code, such as unittesting, is standard practice in software development (https:\/\/software.ac.uk\/resources\/guides\/testing-your-software) and can be useful here too for simulation verification.<\/b>  \n<i>This is a very good point. To get readers started we have direct them to the Turing Way's section on code testing (that covers different testing strategies) and Onggo and Karatas (2016) that introduces test driven development and unit testing in the context of (agent based) simulation. We have added the following test to the manuscript:<\/i>  &quot;In a real simulation project it is recommended that more extensive code testing and verification of model logic is conducted. The full breadth of testing strategies available is out of scope for this tutorial. Interested readers are directed to the code testing section of The Turing Way and in an introduction to Test Driven Development and Unit Testing in the context of computer simulation&quot;.  \n<u>References: <\/u>  The Turing Way Community. (2022). The Turing Way: A handbook for reproducible, ethical and collaborative research. Zenodo. doi: 10.5281\/zenodo.3233853. Bhakti Stephan Onggo, Mumtaz Karatas (2016). Test-driven simulation modelling: A case study using agent-based maritime search-operation simulation, European Journal of Operational Research, Volume 254, Issue 2, https:\/\/doi.org\/10.1016\/j.ejor.2016.03.050.   \n<b>R1: Listing 1 is pseudocode, but written and formatted as Python code, which may be confusing to readers following the tutorial. Can this be formatted differently?<\/b>  \n<i>This is a very good point. We agree that Listing 1 can be improved. We have modified Listing 1 to use more standard psuedocode notation such as SUBROUTINE and left arrows for variable assignment.<\/i> &nbsp; \n<b>R1: I am not sure the section on &quot;Simulation Model Logic&quot; on page 8 contributes to the understanding of the overall methodology and is needed. Those unfamiliar with Simpy might still not understand after reading this, and this is not a paper on Simpy<\/b>  \n<i>Thank you for raising this point. We have considered this point and decided that we would like to keep the section in the manuscript.&nbsp; The reason is that we would like to be able to clearly sign-post the reader to our Extended Material (and external website) where we have additional tutorial material that can be used to recreate and understand the simulation model implementation. We agree that the papers focus is not simpy, but we would like to keep a clear audit trail for reproducibility.&nbsp;&nbsp; <\/i>","URL":"https:\/\/openresearch.nihr.ac.uk\/articles\/3-48\/v2#referee-comment-3301","order":1,"name":"referee-comment-3301","label":"Referee Comment","group":{"name":"article-reports","label":"Article Reports"}},{"value":"10.3310\/nihropenres.14669.r30915, Robert Smith, The University of Sheffield, Sheffield, England, 28 Dec 2023, version 2, indexed","URL":"https:\/\/openresearch.nihr.ac.uk\/articles\/3-48\/v2#referee-response-30915","order":2,"name":"referee-response-30915","label":"Referee Report","group":{"name":"article-reports","label":"Article Reports"}},{"value":"10.3310\/nihropenres.14669.r30913, Apostolos Kritikos, Aristotle University of Thessaloniki, Thessaloniki, 28 Dec 2023, version 2, indexed","URL":"https:\/\/openresearch.nihr.ac.uk\/articles\/3-48\/v2#referee-response-30913","order":3,"name":"referee-response-30913","label":"Referee Report","group":{"name":"article-reports","label":"Article Reports"}},{"value":"This report is independent research supported by the National Institute for Health and Care Research Applied Research Collaboration South West Peninsula. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.","order":4,"name":"grant-information","label":"Grant Information"},{"value":"This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.","order":0,"name":"copyright-info","label":"Copyright"}]}}