{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,13]],"date-time":"2026-07-13T23:50:22Z","timestamp":1783986622519,"version":"3.55.0"},"reference-count":55,"publisher":"BMJ","issue":"11","license":[{"start":{"date-parts":[[2020,11,3]],"date-time":"2020-11-03T00:00:00Z","timestamp":1604361600000},"content-version":"unspecified","delay-in-days":2,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100004417","name":"Global Fund to Fight AIDS, Tuberculosis and Malaria","doi-asserted-by":"publisher","award":["Prospective Country Evaluation (PCE)"],"award-info":[{"award-number":["Prospective Country Evaluation (PCE)"]}],"id":[{"id":"10.13039\/100004417","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["BMJ Glob Health"],"accepted":{"date-parts":[[2020,9,18]]},"published-print":{"date-parts":[[2020,11]]},"abstract":"<jats:sec>\n                  <jats:title>Introduction<\/jats:title>\n                  <jats:p>Understanding how to deliver interventions more effectively is a growing emphasis in Global Health. Simultaneously, health system strengthening is a key component to improving delivery. As a result, it is challenging to evaluate programme implementation while reflecting real-world complexity. We present our experience in using a health systems modelling approach as part of a mixed-methods evaluation and describe applications of these models.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>We developed a framework for how health systems translate financial inputs into health outcomes, with in-country and international experts. We collated available data to measure framework indicators and developed models for malaria in Democratic Republic of the Congo (DRC), and tuberculosis in Guatemala and Senegal using Bayesian structural equation modelling. We conducted several postmodelling analyses: measuring efficiency, assessing bottlenecks, understanding mediation, analysing the cascade of care and measuring subnational effectiveness.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The DRC model indicated a strong relationship between shipment of commodities and utilisation thereof. In Guatemala, the strongest model coefficients were more evenly distributed. Results in Senegal varied most, but pathways related to community care had the strongest relationships. In DRC, we used model results to estimate the end-to-end cost of delivering commodities. In Guatemala, we used model results to identify potential bottlenecks and understand mediation. In Senegal, we used model results to identify potential weak links in the cascade of care, and explore subnationally.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>This study demonstrates a complementary modelling approach to traditional evaluation methods. Although these models have limitations, they can be applied in a variety of ways to gain greater insight into implementation and functioning of health service delivery.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1136\/bmjgh-2020-002441","type":"journal-article","created":{"date-parts":[[2020,11,4]],"date-time":"2020-11-04T03:20:58Z","timestamp":1604460058000},"page":"e002441","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":4,"title":["Bringing a health systems modelling approach to complex evaluations: multicountry applications in HIV, TB and malaria"],"prefix":"10.1136","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2569-1015","authenticated-orcid":false,"given":"David E","family":"Phillips","sequence":"first","affiliation":[{"name":"Institute for Health Metrics and Evaluation, University of Washington, Seattle, Washington, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guillermo","family":"Ambrosio","sequence":"additional","affiliation":[{"name":"Centro de Investigaci\u00f3n Epidemiol\u00f3gica en Salud Sexual y Reproductiva (CIESAR), Guatemala City, Guatemala"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Audrey","family":"Batzel","sequence":"additional","affiliation":[{"name":"Institute for Health Metrics and Evaluation, University of Washington, Seattle, Washington, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carmen","family":"Cerezo","sequence":"additional","affiliation":[{"name":"Centro de Investigaci\u00f3n Epidemiol\u00f3gica en Salud Sexual y Reproductiva (CIESAR), Guatemala City, Guatemala"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Herbert","family":"Duber","sequence":"additional","affiliation":[{"name":"Institute for Health Metrics and Evaluation, University of Washington, Seattle, Washington, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adama","family":"Faye","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, Universite Cheikh Anta Diop, Dakar, Senegal"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ibrahima","family":"Gaye","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, Universite Cheikh Anta Diop, Dakar, Senegal"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bernardo","family":"Hern\u00e1ndez Prado","sequence":"additional","affiliation":[{"name":"Institute for Health Metrics and Evaluation, University of Washington, Seattle, Washington, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bethany","family":"Huntley","sequence":"additional","affiliation":[{"name":"Institute for Health Metrics and Evaluation, University of Washington, Seattle, Washington, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Edgar","family":"Kestler","sequence":"additional","affiliation":[{"name":"Centro de Investigaci\u00f3n Epidemiol\u00f3gica en Salud Sexual y Reproductiva (CIESAR), Guatemala City, Guatemala"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Constant","family":"Kingongo","sequence":"additional","affiliation":[{"name":"PATH, Kinshasa, Democratic Republic of Congo"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephen S","family":"Lim","sequence":"additional","affiliation":[{"name":"Institute for Health Metrics and Evaluation, University of Washington, Seattle, Washington, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emily","family":"Linebarger","sequence":"additional","affiliation":[{"name":"Institute for Health Metrics and Evaluation, University of Washington, Seattle, Washington, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jorge","family":"Matute","sequence":"additional","affiliation":[{"name":"Centro de Investigaci\u00f3n Epidemiol\u00f3gica en Salud Sexual y Reproductiva (CIESAR), Guatemala City, Guatemala"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Godefroid","family":"Mpanya","sequence":"additional","affiliation":[{"name":"PATH, Kinshasa, Democratic Republic of Congo"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Salva","family":"Mulongo","sequence":"additional","affiliation":[{"name":"PATH, Kinshasa, Democratic Republic of Congo"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Caitlin","family":"O'Brien-Carelli","sequence":"additional","affiliation":[{"name":"Institute for Health Metrics and Evaluation, University of Washington, Seattle, Washington, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Erin","family":"Palmisano","sequence":"additional","affiliation":[{"name":"Institute for Health Metrics and Evaluation, University of Washington, Seattle, Washington, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Francisco","family":"Rios Casas","sequence":"additional","affiliation":[{"name":"Institute for Health Metrics and Evaluation, University of Washington, Seattle, Washington, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6550-5446","authenticated-orcid":false,"given":"Katharine","family":"Shelley","sequence":"additional","affiliation":[{"name":"PATH, Seattle, Washington, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Roger","family":"Tine","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, Universite Cheikh Anta Diop, Dakar, Senegal"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Whitaker","sequence":"additional","affiliation":[{"name":"Technical Evaluation Reference Group, The Global Fund to Fight AIDS Tuberculosis and Malaria, Gen\u00e8ve, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer M","family":"Ross","sequence":"additional","affiliation":[{"name":"Institute for Health Metrics and Evaluation, University of Washington, Seattle, Washington, USA"},{"name":"Global Health and Medicine, University of Washington, Seattle, Washington, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2020,11,3]]},"reference":[{"key":"2025072914460116000_5.11.e002441.1","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(13)61047-8"},{"key":"2025072914460116000_5.11.e002441.2","doi-asserted-by":"publisher","DOI":"10.1136\/bmjgh-2016-000115"},{"key":"2025072914460116000_5.11.e002441.3","unstructured":"Global Delivery Initiative . Available: http:\/\/www.globaldeliveryinitiative.org\/"},{"key":"2025072914460116000_5.11.e002441.4","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(16)30168-4"},{"key":"2025072914460116000_5.11.e002441.5","doi-asserted-by":"publisher","DOI":"10.1186\/1744-8603-9-30"},{"key":"2025072914460116000_5.11.e002441.6","unstructured":"Institute for Health Metrics and Evaluation (IHME) . Financing global health visualization. Seattle, WA: IHME, University of Washington, 2019. http:\/\/vizhub.healthdata.org\/fgh\/"},{"key":"2025072914460116000_5.11.e002441.7","unstructured":"The Global Fund to Fight AIDS, TB and Malaria . The global fund strategy 2017-2022: investing to end epidemics. Geneva, Switzerland: The Global Fund, 2016."},{"key":"2025072914460116000_5.11.e002441.8","unstructured":"The Global Fund to Fight AIDS, TB and Malaria . Building resilient and sustainable systems for health (RSSH): information note. Geneva, Switzerland: The Global Fund, 2019."},{"key":"2025072914460116000_5.11.e002441.9","doi-asserted-by":"crossref","DOI":"10.3402\/gha.v9.31899","article-title":"The role of implementation science training in global health: from the perspective of graduates of the field's first dedicated doctoral program","volume":"9","author":"Means","year":"2016","journal-title":"Glob Health Action"},{"key":"2025072914460116000_5.11.e002441.10","unstructured":"Kim JY , Porter M , Rhatigan J , et al . Scaling up effective delivery models worldwide. Reimagining Glob Health Introd 2013:184\u2013211."},{"key":"2025072914460116000_5.11.e002441.11","unstructured":"Global Delivery Initiative . The fly on the bird on the elephant: how can you measure whether your adaptive approach is actually fixing problems that matter? 2019. Available: http:\/\/www.globaldeliveryinitiative.org\/blogs\/fly-bird-elephant-how-can-you-measure-whether-your-adaptive-approach-actually-fixing-problems"},{"key":"2025072914460116000_5.11.e002441.12","unstructured":"Carnahan E , Gurley N , Asiimwe G , et al . Lessons learned from implementing prospective, multi-country mixed methods evaluations for Gavi and the global fund. Glob Health Sci Pract."},{"key":"2025072914460116000_5.11.e002441.13","doi-asserted-by":"crossref","first-page":"2256","DOI":"10.1016\/S0140-6736(14)60791-1","article-title":"Improving the assessment and Attribution of effects of development assistance for health","volume":"384","author":"Ataya","year":"2014","journal-title":"Lancet"},{"key":"2025072914460116000_5.11.e002441.14","unstructured":"Britt H , Hummelbrunner R , Greene J . Causal link monitoring. BetterEvaluation Collaboration, 2017. Available: http:\/\/www.betterevaluation.org\/resources\/overview\/Causal_Link_Monitoring"},{"key":"2025072914460116000_5.11.e002441.15","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1016\/j.socscimed.2017.09.005","article-title":"Dynamic modeling approaches to characterize the functioning of health systems: a systematic review of the literature","volume":"194","author":"Chang","year":"2017","journal-title":"Soc Sci Med"},{"key":"2025072914460116000_5.11.e002441.16","doi-asserted-by":"publisher","DOI":"10.1136\/bmjgh-2018-001311"},{"key":"2025072914460116000_5.11.e002441.17","doi-asserted-by":"crossref","first-page":"334","DOI":"10.1007\/s10729-014-9312-0","article-title":"Optimizing an HIV testing program using a system dynamics model of the continuum of care","volume":"18","author":"Kok","year":"2015","journal-title":"Health Care Manag Sci"},{"key":"2025072914460116000_5.11.e002441.18","doi-asserted-by":"publisher","DOI":"10.5588\/ijtld.11.0062"},{"key":"2025072914460116000_5.11.e002441.19","doi-asserted-by":"publisher","DOI":"10.1057\/jors.2014.136"},{"key":"2025072914460116000_5.11.e002441.20","doi-asserted-by":"crossref","DOI":"10.1186\/1478-4491-8-11","article-title":"A model linking clinical workforce skill mix planning to health and health care dynamics","volume":"8","author":"Masnick","year":"2010","journal-title":"Hum Resour Health"},{"key":"2025072914460116000_5.11.e002441.21","doi-asserted-by":"publisher","DOI":"10.1186\/1478-4505-12-28"},{"key":"2025072914460116000_5.11.e002441.22","doi-asserted-by":"publisher","DOI":"10.1258\/0956462054679124"},{"key":"2025072914460116000_5.11.e002441.23","doi-asserted-by":"crossref","first-page":"1238","DOI":"10.1057\/jors.2009.90","article-title":"System dynamic simulation of treatment policies to address colliding epidemics of tuberculosis, drug resistant tuberculosis and injecting drug users driven HIV in Russia","volume":"61","author":"Lebcir","year":"2010","journal-title":"J Oper Res Soc"},{"key":"2025072914460116000_5.11.e002441.24","doi-asserted-by":"publisher","DOI":"10.1093\/cid\/cit687"},{"key":"2025072914460116000_5.11.e002441.25","doi-asserted-by":"crossref","first-page":"e581","DOI":"10.1016\/S2214-109X(14)70291-8","article-title":"Assessment of the patient, health system, and population effects of Xpert MTB\/RIF and alternative diagnostics for tuberculosis in Tanzania: an integrated modelling approach","volume":"2","author":"Langley","year":"2014","journal-title":"Lancet Glob Health"},{"key":"2025072914460116000_5.11.e002441.26","unstructured":"Patton MQ . Essentials of utilization-focused evaluation. Sage, 2011."},{"key":"2025072914460116000_5.11.e002441.27","unstructured":"Prospective country evaluations [Internet], 2020. Available: https:\/\/www.theglobalfund.org\/en\/technical-evaluation-reference-group\/prospective-country-evaluations\/"},{"key":"2025072914460116000_5.11.e002441.28","unstructured":"Evaluation Planning Tool [Internet] , 2019. Available: https:\/\/www.radar-project.org\/evaluation-planning-tool"},{"key":"2025072914460116000_5.11.e002441.29","unstructured":"R Core Team . R: A language and environment for statistical computing [Internet]. Vienna, Austria: R Foundation for Statistical Computing, 2019."},{"key":"2025072914460116000_5.11.e002441.30","doi-asserted-by":"crossref","unstructured":"Honaker J , King G , Blackwell M . Amelia II: A Program for Missing Data. J Stat Softw 2011;45.doi:10.18637\/jss.v045.i07","DOI":"10.18637\/jss.v045.i07"},{"key":"2025072914460116000_5.11.e002441.31","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(18)32225-6"},{"key":"2025072914460116000_5.11.e002441.32","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v085.i04","article-title":"blavaan : Bayesian structural equation models via parameter expansion","volume":"85","author":"Merkle","year":"2018","journal-title":"J Stat Softw"},{"key":"2025072914460116000_5.11.e002441.33","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-1545-0"},{"key":"2025072914460116000_5.11.e002441.34","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0213605","article-title":"Using mathematical modeling to inform health policy: a case study from voluntary medical male circumcision scale-up in eastern and southern Africa and proposed framework for success","volume":"14","author":"Njeuhmeli","year":"2019","journal-title":"PLoS One"},{"key":"2025072914460116000_5.11.e002441.35","doi-asserted-by":"crossref","first-page":"e190","DOI":"10.1016\/S2352-3018(18)30024-9","article-title":"The global optima HIV allocative efficiency model: targeting resources in efforts to end AIDS","volume":"5","author":"Kelly","year":"2018","journal-title":"Lancet HIV"},{"key":"2025072914460116000_5.11.e002441.36","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(18)32335-3"},{"key":"2025072914460116000_5.11.e002441.37","doi-asserted-by":"crossref","first-page":"e22","DOI":"10.1016\/S2214-109X(16)30331-X","article-title":"Making health systems research work: time to shift funding to locally-led research in the South","volume":"5","author":"Hasnida","year":"2017","journal-title":"Lancet Glob Health"},{"key":"2025072914460116000_5.11.e002441.38","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1332\/174426416X14779418584665","article-title":"Causal thinking for embedded, integrated implementation research","volume":"15","author":"Kainz","year":"2019","journal-title":"Evid Policy"},{"key":"2025072914460116000_5.11.e002441.39","first-page":"578","article-title":"Health information systems: the foundations of public health","volume":"83","author":"AbouZahr","year":"2005","journal-title":"Bull World Health Organ"},{"key":"2025072914460116000_5.11.e002441.40","doi-asserted-by":"crossref","first-page":"69","DOI":"10.2471\/BLT.19.239970","article-title":"Using district health information to monitor sustainable development","volume":"98","author":"Farnham","year":"2020","journal-title":"Bull World Health Organ"},{"key":"2025072914460116000_5.11.e002441.41","unstructured":"Saunders-Hastings P . DHIS2 as a tool for health impact assessment in low-resource settings: examining opportunities for expanding use of routine health data."},{"key":"2025072914460116000_5.11.e002441.42","unstructured":"Reach V . Starting at the last mile. Rethinking medical supply chains in low-income countries, 2012. Available: https:\/\/www.villagereach.org\/wp-content\/uploads\/2012\/03\/02272012.Starting-at-the-Last-Mile-Whitepaper.pdf"},{"key":"2025072914460116000_5.11.e002441.43","unstructured":"Ecumenical Pharmaceutical Network . Effective pharmaceutical supply chains: on the road in low income countries, 2011. Available: http:\/\/apps.who.int\/medicinedocs\/documents\/s19965en\/s19965en.pdf"},{"key":"2025072914460116000_5.11.e002441.44","unstructured":"The Global Fund to Fight AIDS, TB and Malaria . Databases from grant operating system and grant management system (Internal communication). Geneva, Switzerland."},{"key":"2025072914460116000_5.11.e002441.45","unstructured":"The Global Fund to Fight AIDS, TB and Malaria. . Progress update and Sibursement Request tool. Geneva, Switzerland: The Global Fund, 2019."},{"key":"2025072914460116000_5.11.e002441.46","unstructured":"Institute for Health Metrics and Evaluation (IHME) . Development Assistance for Health Database 1990-2018 [Internet], 2019. Available: http:\/\/internal-ghdx.healthdata.org\/record\/ihme-data\/development-assistance-health-database-1990-2018"},{"key":"2025072914460116000_5.11.e002441.47","unstructured":"Institute for Health Metrics and Evaluation (IHME) , 2019. Available: http:\/\/ghdx.healthdata.org\/record\/ihme-data\/global-malaria-spending-2000-2016"},{"key":"2025072914460116000_5.11.e002441.48","unstructured":"Programme National de Lutte contre le Paludisme (PNLP) . Routine monitoring and evaluation data (internal communication. Minist\u00e8re de la Sant\u00e9 RDC, 2018."},{"key":"2025072914460116000_5.11.e002441.49","unstructured":"Minist\u00e8re de la Sant\u00e9 Publique, R\u00e9publique D\u00e9mocratique du Congo . Syst\u00e8me National d\u2019Information Sanitaire [Internet]. DHIS2, 2019. Available: https:\/\/snisrdc.com\/"},{"key":"2025072914460116000_5.11.e002441.50","doi-asserted-by":"publisher","DOI":"10.1038\/nature15535"},{"key":"2025072914460116000_5.11.e002441.51","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(19)31097-9"},{"key":"2025072914460116000_5.11.e002441.52","unstructured":"Sistema de Contabilidad Integrada (SICOIN) . Routine monitoring and evaluation data. Guatemala Ministerio de Finanzas P\u00fablicas, 2019."},{"key":"2025072914460116000_5.11.e002441.53","unstructured":"National Tuberculosis Program . Routine monitoring and evaluation data 2016-2018. Guatemala Ministry of Health and Welfare, 2019."},{"key":"2025072914460116000_5.11.e002441.54","unstructured":"Programme National de Lutte Contre la Tuberculose Senegal (PNT) . Routine monitoring and evaluation data, 2014-2018, 2018."},{"key":"2025072914460116000_5.11.e002441.55","unstructured":"PLAN International, Inc . Routine reporting data, 2014-2018, 2019."}],"container-title":["BMJ Global Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bmjgh-2020-002441","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,29]],"date-time":"2025-07-29T21:46:22Z","timestamp":1753825582000},"score":1,"resource":{"primary":{"URL":"https:\/\/gh.bmj.com\/lookup\/doi\/10.1136\/bmjgh-2020-002441"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11]]},"references-count":55,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2020,11,3]]},"published-print":{"date-parts":[[2020,11]]}},"alternative-id":["10.1136\/bmjgh-2020-002441"],"URL":"https:\/\/doi.org\/10.1136\/bmjgh-2020-002441","relation":{},"ISSN":["2059-7908"],"issn-type":[{"value":"2059-7908","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,11]]}}}