{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T23:47:37Z","timestamp":1790725657803,"version":"4.1.0"},"reference-count":41,"publisher":"SAGE Publications","issue":"5","license":[{"start":{"date-parts":[[2025,1,11]],"date-time":"2025-01-11T00:00:00Z","timestamp":1736553600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2025,1,11]],"date-time":"2025-01-11T00:00:00Z","timestamp":1736553600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"name":"Policy and Operations Evaluation Department, Ministry of Foreign Affairs, Government of the Netherlands"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Eval Rev"],"published-print":{"date-parts":[[2025,10]]},"abstract":"<jats:p>Official development agencies are increasingly supporting civil society lobby and advocacy (L&amp;A) to address poverty and human rights. However, there are challenges in evaluating L&amp;A. As programme objectives are often to change policies or practices in a single institution like a Government Ministry, L&amp;A programmes are often not amenable to large-n impact evaluation methods. They often work in strategic partnerships to foster change; hence, contribution may be a more relevant evaluation question than attribution. Small-n qualitative approaches are available to measure the effectiveness of L&amp;A which use the theory of change as their analytical framework. We conducted a meta-evaluation of 36 evaluations of multi-component international programmes to support civil society L&amp;A across Asia, Africa and Latin America, comprising the majority of programmatic support from one international donor. We assessed the confidence in causal claims in the evaluations using a new tool that we developed. Assessments of the contribution of the programmes to the changes in outcomes were not provided in many of the evaluations, nor were predictable sources of bias addressed. Given that L&amp;A programmes are likely to adopt an influencing approach where many different inside-track and outside-track engagement objectives, opportunities and strategies are attempted, many of which might be expected to fail, there appeared to be a clear bias in the evaluations towards reporting outcomes that were achieved, ignoring those that were not. We provide guidance on how to improve the design, conduct and reporting of small-n qualitative evaluations of aid effectiveness.<\/jats:p>","DOI":"10.1177\/0193841x251314731","type":"journal-article","created":{"date-parts":[[2025,1,11]],"date-time":"2025-01-11T11:32:52Z","timestamp":1736595172000},"page":"851-879","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":4,"title":["What Can We Learn From Qualitative Impact Evaluations About the Effectiveness of Lobby and Advocacy? A Meta-Evaluation of Dutch aid Programmes and Assessment Tool"],"prefix":"10.1177","volume":"49","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3859-3342","authenticated-orcid":false,"given":"Hugh","family":"Sharma Waddington","sequence":"first","affiliation":[{"name":"London School of Hygiene and Tropical Medicine and London International Development Centre, London, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-6454-7767","authenticated-orcid":false,"given":"Hikari","family":"Umezawa","sequence":"additional","affiliation":[{"name":"The Campbell Collaboration, Philadelphia, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Howard","family":"White","sequence":"additional","affiliation":[{"name":"Global Development Network, Lanzhou University and Director of Evaluation, New Delhi, India"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2025,1,11]]},"reference":[{"key":"e_1_3_6_2_1","volume-title":"Advocacy for development: Effectiveness, monitoring and evaluation","author":"Barrett J. B.","year":"2016","unstructured":"Barrett J. B., Van Wessel M., Hilhorst D. (2016). Advocacy for development: Effectiveness, monitoring and evaluation. Wageningen University."},{"key":"e_1_3_6_3_1","volume-title":"Who counts? The quiet revolution of participation and numbers (IDS working paper 296)","author":"Chambers R.","year":"2007","unstructured":"Chambers R. (2007). Who counts? The quiet revolution of participation and numbers (IDS working paper 296). Institute of Development Studies."},{"key":"e_1_3_6_4_1","doi-asserted-by":"publisher","DOI":"10.1002\/cl2.1194"},{"key":"e_1_3_6_5_1","doi-asserted-by":"publisher","DOI":"10.3362\/9781780447469"},{"key":"e_1_3_6_6_1","unstructured":"Critical Appraisal Skills Programme (CASP). (2018). CASP checklist: 10 questionsto help you make sense of a qualitative research. Retrieved November 17 2022 from. https:\/\/casp-uk.net\/images\/checklist\/documents\/CASP-Qualitative-Studies-Checklist\/CASP-Qualitative-Checklist-2018_fillable_form.pdf"},{"key":"e_1_3_6_7_1","doi-asserted-by":"publisher","DOI":"10.51744\/CIP15"},{"key":"e_1_3_6_8_1","unstructured":"Davies R. Dart J. (2005). The \u2018most significant change\u2019 (MSC) technique. A guide to its use. Available at: https:\/\/www.mande.co.uk\/wp-content\/uploads\/2005\/MSCGuide.pdf (accessed 1 January 2022 Version 1.00)."},{"key":"e_1_3_6_9_1","doi-asserted-by":"publisher","DOI":"10.4073\/csr.2019.2"},{"key":"e_1_3_6_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/0749-5978(89)90059-9"},{"key":"e_1_3_6_11_1","doi-asserted-by":"publisher","DOI":"10.1515\/9781503602335"},{"key":"e_1_3_6_12_1","volume-title":"Magenta book march 2020 central government guidance on evaluation","author":"HM Treasury","year":"2020","unstructured":"HM Treasury. (2020). Magenta book march 2020 central government guidance on evaluation. Crown Copyright."},{"key":"e_1_3_6_13_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.282.11.1054"},{"key":"e_1_3_6_14_1","article-title":"Dialogue and Dissent Theory of Change 2.0 : Supporting civil society\u2019s political role","author":"Kamstra J.","year":"2017","unstructured":"Kamstra J. (2017). Dialogue and Dissent Theory of Change 2.0 : Supporting civil society\u2019s political role. Ministry of Foreign Affairs of the Netherlands Social Development Department Civil society unit (DSO\/MO). Retrieved September 20, 2022 from. https:\/\/includeplatform.net\/wp-content\/uploads\/2020\/07\/Annex1_DialogueandDissentTheoryofChange-June2017.pdf","journal-title":"Ministry of Foreign Affairs of the Netherlands Social Development Department Civil society unit (DSO\/MO)"},{"key":"e_1_3_6_15_1","doi-asserted-by":"publisher","DOI":"10.1177\/1356389017750195"},{"key":"e_1_3_6_16_1","volume-title":"The cement of the universe: A study of causation","author":"Mackie J. L.","year":"1974","unstructured":"Mackie J. L. (1974). The cement of the universe: A study of causation. Clarendon Press."},{"key":"e_1_3_6_17_1","unstructured":"Mayne J. (2020). A brief on contribution analysis: Principles and concepts. Retrieved October 14 2022 from. https:\/\/evaluatingadvocacy.org\/doc\/A-brief-on-contribution-analysis-Principles-and-concepts.pdf"},{"key":"e_1_3_6_18_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.c869"},{"key":"e_1_3_6_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2013.03.006"},{"key":"e_1_3_6_20_1","volume-title":"Mapp: A new method for impact assessment of poverty alleviation projects","author":"Neubert S.","year":"1998","unstructured":"Neubert S. (1998). Mapp: A new method for impact assessment of poverty alleviation projects. German Development Institute (GDI)."},{"key":"e_1_3_6_21_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.n71"},{"key":"e_1_3_6_22_1","volume-title":"A map of evidence maps relating to sustainable development in low- and middle- income countries: Evidence gap map report","author":"Phillips D.","year":"2017","unstructured":"Phillips D., Coffey C., Tsoli S., Stevenson J., Waddington H., Eyers J., White H., Snilstveit B. (2017). A map of evidence maps relating to sustainable development in low- and middle- income countries: Evidence gap map report. CEDIL Pre-Inception Paper."},{"key":"e_1_3_6_23_1","doi-asserted-by":"publisher","DOI":"10.1177\/1356389018781361"},{"key":"e_1_3_6_24_1","unstructured":"Policy and Operations Evaluation Department (IOB). (2022). IOB evaluation quality criteria. Retrieved August 31 2023 from. https:\/\/english.iob-evaluatie.nl\/publications\/guidelines\/2022\/04\/22\/evaluation-quality-criteria"},{"key":"e_1_3_6_25_1","doi-asserted-by":"publisher","DOI":"10.1177\/10982140211062815"},{"key":"e_1_3_6_26_1","doi-asserted-by":"publisher","DOI":"10.1177\/109821400402500103"},{"key":"e_1_3_6_27_1","doi-asserted-by":"publisher","DOI":"10.1080\/19439342.2018.1483414"},{"key":"e_1_3_6_28_1","doi-asserted-by":"publisher","DOI":"10.56645\/jmde.v5i9.160"},{"key":"e_1_3_6_29_1","unstructured":"Sharma Waddington H. Umezawa H. White H. (2023). The effectiveness of support to lobby and advocacy: Synthesis of evidence from Dutch MFA programme support. Ministry of Foreign Affairs the Hague. Available at: https:\/\/english.iob-evaluatie.nl\/publications\/reports\/2023\/04\/15\/effectiveness-support-to-lobby-and-advocacy (accessed 25 June 2024)."},{"key":"e_1_3_6_30_1","volume-title":"End-term evaluation of the citizen agency Consortium green and inclusive energy program","author":"Sloot H.","year":"2020","unstructured":"Sloot H., Becker S., Wolfkamp M. (2020). End-term evaluation of the citizen agency Consortium green and inclusive energy program. Citizen Agency Consortium. https:\/\/www.hivos.org\/assets\/2020\/09\/ETE-Green-and-Inclusive-Energy.pdf"},{"key":"e_1_3_6_31_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.l4898"},{"issue":"3","key":"e_1_3_6_32_1","first-page":"38","article-title":"The elusive craft of evaluating advocacy","volume":"9","author":"Teles S.","year":"2011","unstructured":"Teles S., Schmitt M. (2011). The elusive craft of evaluating advocacy. Stanford Social Innovation Review, 9(3), 38\u201343. https:\/\/doi.org\/10.48558\/Y90Q-VE61","journal-title":"Stanford Social Innovation Review"},{"key":"e_1_3_6_33_1","volume-title":"Evaluation of international development interventions: An overview of approaches and methods. Independent evaluation group","author":"Vaessen J.","year":"2020","unstructured":"Vaessen J., Lemire S., Befani B. (2020). Evaluation of international development interventions: An overview of approaches and methods. Independent evaluation group. World Bank. https:\/\/hdl.handle.net\/10986\/34962"},{"key":"e_1_3_6_34_1","doi-asserted-by":"publisher","DOI":"10.1177\/1356389018796021"},{"key":"e_1_3_6_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(07)61602-X"},{"key":"e_1_3_6_36_1","unstructured":"Wells G. A. Shea B. O\u2019Connell D. Peterson J. Welch V. Losos M. Tugwell P. (nd). The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Retrieved September 15 2023 from. https:\/\/www.ohri.ca\/programs\/clinical_epidemiology\/oxford.asp"},{"key":"e_1_3_6_37_1","doi-asserted-by":"publisher","DOI":"10.1002\/cl2.1125"},{"key":"e_1_3_6_38_1","volume-title":"Addressing attribution of cause and effect in small n impact evaluations: Towards an integrated framework. (3ie working paper series 15)","author":"White H.","year":"2012","unstructured":"White H., Phillips D. (2012). Addressing attribution of cause and effect in small n impact evaluations: Towards an integrated framework. (3ie working paper series 15). International Initiative for Impact Evaluation (3ie). https:\/\/doi.org\/10.23846\/wp0015"},{"key":"e_1_3_6_39_1","volume-title":"Evidence and gap map of interventions to prevent children getting involved in violence: Technical report on the first edition","author":"White H.","year":"2021","unstructured":"White H., Saran A., Verma A., Oprea E., Babudu P. (2021). Evidence and gap map of interventions to prevent children getting involved in violence: Technical report on the first edition. Youth Endowment Fund and Campbell Collaboration."},{"key":"e_1_3_6_40_1","volume-title":"Outcome harvesting: Principles, steps and evaluation approaches","author":"Wilson-Grau R.","year":"2019","unstructured":"Wilson-Grau R. (2019). Outcome harvesting: Principles, steps and evaluation approaches. Information Age Publishing."},{"key":"e_1_3_6_41_1","doi-asserted-by":"publisher","DOI":"10.1596\/0-8213-5468-X"},{"key":"e_1_3_6_42_1","doi-asserted-by":"publisher","DOI":"10.1596\/978-1-4648-0950-7"}],"container-title":["Evaluation Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0193841X251314731","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/0193841X251314731","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0193841X251314731","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T20:47:23Z","timestamp":1777409243000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/0193841X251314731"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,11]]},"references-count":41,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,10]]}},"alternative-id":["10.1177\/0193841X251314731"],"URL":"https:\/\/doi.org\/10.1177\/0193841x251314731","relation":{},"ISSN":["0193-841X","1552-3926"],"issn-type":[{"value":"0193-841X","type":"print"},{"value":"1552-3926","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,11]]}}}