{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T22:02:30Z","timestamp":1767650550917,"version":"3.41.2"},"reference-count":47,"publisher":"Emerald","issue":"1","license":[{"start":{"date-parts":[[2016,3,14]],"date-time":"2016-03-14T00:00:00Z","timestamp":1457913600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2016,3,14]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-heading\">Purpose<\/jats:title><jats:p>\u2013 There are five factors acting as a barrier to the effective evaluation of educational technology (edtech), which are as follows: premature timing, inappropriate techniques, rapid change, complexity of context and inconsistent terminology. The purpose of this paper is to identify new evaluation approaches that will address these and reflect on the evaluation imperative for complex technology initiatives.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Approach<\/jats:title><jats:p>\u2013 An initial investigation of traditional evaluative approaches used within the technology domain was broadened to investigate the evaluation practices within social and public policy domains. Realist evaluation, a branch of theory-based evaluation, was identified and reviewed in detail. The realist approach was then refined, proposing two additional necessary steps to support mapping the technical complexity of initiatives.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Findings<\/jats:title><jats:p>\u2013 A refined illustrative example of a realist evaluation framework is presented, including two novel architectural edtech domain reference models to support mapping.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Practical implications<\/jats:title><jats:p>\u2013 Recommendations include building individual evaluator capacity; adopting the realist framework; the use of architectural edtech domain reference models; phased evaluation to first build theories in technology \u201ccontext\u201d and then iteratively during complex implementation chains; and community contribution to a shared map of technical and organisational complexity.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Originality<\/jats:title><jats:p>\u2013 This paper makes a novel contribution by arguing the imperative for a theory-based realist approach to help redefine evaluative thinking within the IT and complex system domain. It becomes an innovative proposal with the addition of two domain reference models that tailor the approach for edtech. Its widespread adoption will help build a shared evidence base that synthesizes and surfaces \u201cwhat works, for whom, in which contexts and why\u201d, benefiting educators, IT managers, funders, policymakers and future learners.<\/jats:p><\/jats:sec>","DOI":"10.1108\/jsit-06-2015-0059","type":"journal-article","created":{"date-parts":[[2016,3,7]],"date-time":"2016-03-07T04:00:08Z","timestamp":1457323208000},"page":"18-40","source":"Crossref","is-referenced-by-count":9,"title":["Bridging the edtech evidence gap"],"prefix":"10.1108","volume":"18","member":"140","reference":[{"key":"key2020121704315659200_b1","unstructured":"Allen, T.J. (1977), Managing the Flow of Technology , MIT Press, Cambridge."},{"key":"key2020121704315659200_b2","doi-asserted-by":"crossref","unstructured":"Astbury, B. and Leeuw, F.L. (2010), \u201cUnpacking black boxes: mechanisms and theory building in evaluation\u201d, American Journal of Evaluation , Vol. 31 No. 3, pp. 363-381.","DOI":"10.1177\/1098214010371972"},{"key":"key2020121704315659200_b3","unstructured":"Bhaskar, R. (1978), A Realist Theory of Science , Routledge, London."},{"key":"key2020121704315659200_b4","doi-asserted-by":"crossref","unstructured":"Bulfin, S. , Johnson, F.N. and Bigum, C. (Eds) (2015), Critical Perspectives on Technology and Education , Palgrave Macmillan, Basingstoke.","DOI":"10.1057\/9781137385451"},{"key":"key2020121704315659200_b5","doi-asserted-by":"crossref","unstructured":"Chen, H.T. and Rossi, P.H. (1980), \u201cThe multi-goal, theory-driven approach to evaluation: a model linking basic and applied social science\u201d, Social Forces , Vol. 59 No. 1, pp. 106-122.","DOI":"10.2307\/2577835"},{"key":"key2020121704315659200_b6","unstructured":"Clarke, J.A. , Nelson, K.J. and Stoodley, I.D. (2013), \u201cThe place of higher education institutions in assessing student engagement, success and retention: a maturity model to guide practice\u201d, Higher Education Research and Development Society of Australasia , AUT University, Aukland."},{"key":"key2020121704315659200_b7","doi-asserted-by":"crossref","unstructured":"Coryn, C.L. , Noakes, L.A. , Westine, C.D. and Schr\u00f6ter, D.C. (2011), \u201cA systematic review of theory-driven evaluation practice from 1990 to 2009\u201d, American Journal of Evaluation , Vol. 32 No. 2, pp. 199-226.","DOI":"10.1177\/1098214010389321"},{"key":"key2020121704315659200_b8","unstructured":"Dalkin, S.M. , Greenhalgh, J. , Jones, D. , Cunningham, B. and Lhussier, M. (2015), \u201cWhat\u2019s in a mechanism? Development of a key concept in realist evaluation\u201d, Implementation Science , Vol. 10 No. 49."},{"key":"key2020121704315659200_b9","doi-asserted-by":"crossref","unstructured":"Dobson, J.E. and Fisher, P.F. (2007), \u201cThe panopticon\u2019s changing geography\u201d, The Geographical Review , Vol. 97 No. 3, pp. 307-323.","DOI":"10.1111\/j.1931-0846.2007.tb00508.x"},{"key":"key2020121704315659200_b10","unstructured":"European Commission . (2013), Evalsed Sourcebook: Methods and Techniques , European Commission, Regional Policy, Brussels."},{"key":"key2020121704315659200_b11","unstructured":"EvalPartners (2015), \u201cGlobal Evaluation Agenda 2016-2020 Preliminary outcomes of the global networked online consultation, from EvalPartners\u201d, available at: http:\/\/mymande.org\/sites\/default\/files\/files\/Global_Evaluation_Agenda2016-2020_v3.pdf (accessed 6 January 2015)."},{"key":"key2020121704315659200_b12","doi-asserted-by":"crossref","unstructured":"Groenewegen, P. and Moser, C. (2014), \u201cOnline communities: challenges and opportunities for social network research\u201d, in Brass, D.J. , Labianca, G. , Mehra, A. , Halgin, D.S. and Borgatti, S.P. (Eds), Research in the Sociology of Organizations (Contemporary Perspectives on Organizational Social Networks ), Emerald Publishing, Bingley, Vol. 40, pp. 464-477.","DOI":"10.1108\/S0733-558X(2014)0000040023"},{"key":"key2020121704315659200_b13","unstructured":"HEFCE (2011), \u201cSummative evaluation of the CETL programme \u2013 Final report by SQW to HEFCE and DEL\u201d, HEFCE."},{"key":"key2020121704315659200_b14","unstructured":"Hil, R. (2012), Whackademia: An Insider\u2019s Account of the Troubled University , UNSW Press, New South Wales."},{"key":"key2020121704315659200_b15","unstructured":"Hsu, C.C. and Sandford, B.A. (2007), \u201cThe Delphi technique: making sense of consensus\u201d, Practical Assessment, Research & Evaluation , Vol. 12 No. 10."},{"key":"key2020121704315659200_b16","unstructured":"IPPR Commission on the Future of Higher Education . (2013), A Critical Path: Securing the Future of Higher Education , The Institute for Public Policy Research, IPPR, London."},{"key":"key2020121704315659200_b17","unstructured":"King, M. , Dawson, R. , Batmaz, F. and Rothberg, S. (2014), \u201cThe need for evidence innovation in educational technology evaluation\u201d, in Uhomoibi, J. (Ed.), Proceedings of INSPIRE XIX: Global Issues in IT Education , British Computer Society, Southampton, pp. 9-23."},{"key":"key2020121704315659200_b18","unstructured":"Kolb, D.A. (2015), Experiential Learning: Experience as the Source of Learning and Development , in Neidlinger, A. (Ed.), 2nd ed., Pearson Education Inc, New Jersey."},{"key":"key2020121704315659200_b19","unstructured":"Lucas-Conwell, F. (2006), \u201cTechnology evangelists: a leadership survey, from Growth Resources\u201d, available at: www.gri.co\/pub\/res\/pdf\/TechEvangelist.pdf (accessed 3 August 2015)."},{"key":"key2020121704315659200_b20","unstructured":"Marshall, S. (2010), \u201cA quality framework for continuous improvement of e-learning: the e-learning maturity model\u201d, Journal of Distance Education , Vol. 24 No. 1, pp. 143-166."},{"key":"key2020121704315659200_b21","unstructured":"Michie, S. , Stralen, M.M. and West, R. (2011), \u201cThe behaviour change wheel: a new method for characterising and designing behaviour change interventions\u201d, Implementation Science , Vol. 6 No. 42."},{"key":"key2020121704315659200_b26","unstructured":"Nuffield Department of Primary Care Health Sciences (2015), \u201cMailing list\u201d, The RAMESES Projects, available at: www.ramesesproject.org\/index.php?pr=Mailing_list (accessed 8 February 2016)."},{"key":"key2020121704315659200_b22","doi-asserted-by":"crossref","unstructured":"Oroviogoicoechea, C. and Watson, R. (2009), \u201cA quantitative analysis of the impact of a computerized information system on nurses\u2019 clinical practice using a realistic evaluation framework\u201d, International Journal of Medical Informatics , Vol. 78 No. 12, pp. 839-849.","DOI":"10.1016\/j.ijmedinf.2009.08.008"},{"key":"key2020121704315659200_b23","doi-asserted-by":"crossref","unstructured":"Pawson, R. (2013), The Science of Evaluation: A Realist Manifesto , Sage, London.","DOI":"10.4135\/9781473913820"},{"key":"key2020121704315659200_b24","doi-asserted-by":"crossref","unstructured":"Pawson, R. and Sridharan, S. (2010), \u201cTheory-driven evaluation of public health programmes\u201d, in Killoran, A. and Kelly, A. (Eds), Evidence Based Public Health , Oxford University Press, Oxford.","DOI":"10.1093\/acprof:oso\/9780199563623.003.04"},{"key":"key2020121704315659200_b25","unstructured":"Pawson, R. and Tilley, N. (1997), Realist Evaluation , Sage, London."},{"key":"key2020121704315659200_b27","unstructured":"Rogers, E.M. (2003), Diffusion of Innovations , 5th ed., Free Press, New York, NY."},{"key":"key2020121704315659200_b28","doi-asserted-by":"crossref","unstructured":"Rogers, P.J. (2008), \u201cUsing programme theory to evaluate complicated and complex aspects of interventions\u201d, Evaluation , Vol. 14 No. 1, pp. 29-48.","DOI":"10.1177\/1356389007084674"},{"key":"key2020121704315659200_b29","unstructured":"Rugg, D. (2013), May. \u201cUNEG joins EvalPartners in declaring 2015 as international year of evaluation, from United Nations Evaluation Group (UNEG)\u201d, available at: www.unevaluation.org\/document\/detail\/1378 (accessed 6 January 2015)."},{"key":"key2020121704315659200_b30","doi-asserted-by":"crossref","unstructured":"Scheiner, C.W. , Baccarella, C.V. , Bessant, J. and Voigt, K.I. (2014), \u201cThinking patterns and gut feeling in technology identification and evaluation\u201d, Technological Forecasting & Social Change , Vol. 12 No. 1.","DOI":"10.1016\/j.techfore.2013.12.003"},{"key":"key2020121704315659200_b31","doi-asserted-by":"crossref","unstructured":"Selwyn, N. (2014a), Digital Technology and the Contemporary University , Routledge, London.","DOI":"10.4324\/9781315768656"},{"key":"key2020121704315659200_b32","doi-asserted-by":"crossref","unstructured":"Selwyn, N. (2014b), Distrusting Educational Technology: Critical Questions for Changing Times , Routledge, London.","DOI":"10.4324\/9781315886350"},{"key":"key2020121704315659200_b33","unstructured":"Stufflebeam, D.L. and Shinkfield, A.J. (2007), Evaluation Theory, Models, & Application , Jossey-Bass, San Francisco."},{"key":"key2020121704315659200_b34","unstructured":"Suchman, E. (1967), Evaluative Research , Russell Sage Foundation, New York, NY."},{"key":"key2020121704315659200_b35","unstructured":"The Agile Admin (2010), \u201cWhat is DevOps?\u201d, The Agile Admin, available at: http:\/\/theagileadmin.com\/what-is-devops\/ (accessed 25 February 2015)."},{"key":"key2020121704315659200_b37","unstructured":"Trowler, P. , Ashwin, P. and Saunders, M. (2014), The Role of HEFCE in Teaching and Learning Enhancement: A Review of Evaluative Evidence , Lancaster University, Centre for Higher Education Research and Evaluation, Higher Education Academy, Lancaster."},{"key":"key2020121704315659200_b38","unstructured":"UCISA (2014), 2014 Survey of Technology Enhanced Learning for Higher Education in the UK , Universities and Colleges Information Systems Association, UCISA, Oxford."},{"key":"key2020121704315659200_b39","unstructured":"Udas, K. and Feldstein, M. (2006), Apples to Apples: Guidelines for Comparative Evaluation of Proprietary and Open Educational Technology Systems , State University of New York, SUNY Learning Network, The Observatory on Borderless Higher Education, London."},{"key":"key2020121704315659200_b40","unstructured":"Universities UK (2011), Efficiency and Effectiveness in Higher Education , Universities UK, London."},{"key":"key2020121704315659200_b41","doi-asserted-by":"crossref","unstructured":"Venkatesh, V. , Morris, M.G. , Davis, G.B. and Davis, F.D. (2003), \u201cUser acceptance of information technology: toward a unified view\u201d, MIS Quarterly , Vol. 27 No. 3, pp. 425-478.","DOI":"10.2307\/30036540"},{"key":"key2020121704315659200_b42","unstructured":"Walker, G. (2001), IT Problem Management , Prentice Hall, New Jersey."},{"key":"key2020121704315659200_b43","unstructured":"Weiss, C.H. (1995), \u201cNothing as practical as good theory: exploring theory-based evaluation for comprehensive community initiatives for children and families\u201d, in Connell, J.I. , Kubisch, A.C. , Schorr, L.B. and Weiss, C.H. (Eds), New Approaches to Evaluating Community Initiatives , The Aspen Institute, Washington DC, pp. 65-92."},{"key":"key2020121704315659200_b44","unstructured":"Westhorp, G. (2014), Realist Impact Evaluation: An Introduction , Overseas Development Institute (ODI), Methods Lab, ODI, London."},{"key":"key2020121704315659200_b45","unstructured":"Wilson, A. (2011), Review of the Joint Information Systems Committee , HEFCE, London."},{"key":"key2020121704315659200_b46","unstructured":"Wong, G. , Greenhalgh, T. and Pawson, R. (2010), \u201cInternet-based medical education: a realist review of what works, for whom and in what circumstances\u201d, BMC Medical Education , Vol. 10 No. 12."},{"key":"key2020121704315659200_b47","unstructured":"Wong, G. , Greenhalgh, T. , Westhorp, G. , Buckingham, J. and Pawson, R. (2013), \u201cRAMESES Publication standards: realist syntheses\u201d, BMC Medicine , Vol. 11 No. 21."},{"key":"key2020121704315659200_frd1","unstructured":"The Open University (n.d.), \u201cOpen Education \u2013 5.5 Rhizomatic Learning\u201d, OpenLearn, available at: www.open.edu\/openlearn\/education\/open-education\/content-section-7.5 (accessed 8 March 2015)."}],"container-title":["Journal of Systems and Information Technology"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/JSIT-06-2015-0059","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/JSIT-06-2015-0059\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/JSIT-06-2015-0059\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T22:23:49Z","timestamp":1753395829000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/jsit\/article\/18\/1\/18-40\/253459"}},"subtitle":["A realist evaluation framework refined for complex technology initiatives"],"short-title":[],"issued":{"date-parts":[[2016,3,14]]},"references-count":47,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2016,3,14]]}},"alternative-id":["10.1108\/JSIT-06-2015-0059"],"URL":"https:\/\/doi.org\/10.1108\/jsit-06-2015-0059","relation":{},"ISSN":["1328-7265"],"issn-type":[{"type":"print","value":"1328-7265"}],"subject":[],"published":{"date-parts":[[2016,3,14]]}}}