{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T16:01:07Z","timestamp":1784390467102,"version":"3.55.0"},"reference-count":93,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2024,2,5]],"date-time":"2024-02-05T00:00:00Z","timestamp":1707091200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"EPSRC, including the Centre for Digital Civics","award":["EP\/L016176\/1"],"award-info":[{"award-number":["EP\/L016176\/1"]}]},{"name":"EPSRC and BBC iCASE scholarship","award":["1623937"],"award-info":[{"award-number":["1623937"]}]},{"DOI":"10.13039\/501100002713","name":"Al Imam Mohammad Ibn Saud Islamic University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100002713","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput.-Hum. Interact."],"published-print":{"date-parts":[[2024,4,30]]},"abstract":"<jats:p>\n            HCI researchers are increasingly interested in the evaluation of educational technologies in context, yet acknowledge that challenges remain regarding the logistical, material and methodological constraints of this approach to research [\n            <jats:xref ref-type=\"bibr\">18<\/jats:xref>\n            ,\n            <jats:xref ref-type=\"bibr\">53<\/jats:xref>\n            ].\n          <\/jats:p>\n          <jats:p>Through the analysis of the authors\u2019 contributed thematic research vignettes, the following article exposes the practical realities of evaluating educational technologies in school settings. This includes insights into the planning stages of evaluation, the relationship between the researcher and the school environment, and the impact of the school context on the data collection process.<\/jats:p>\n          <jats:p>We conclude by providing an orientation for the design of HCI educational technology research undertaken in school contexts, providing guidance such as considering the role of modular research design, clarifying goals and expectations with school partners, and reporting researcher positionality.<\/jats:p>","DOI":"10.1145\/3635146","type":"journal-article","created":{"date-parts":[[2023,12,1]],"date-time":"2023-12-01T12:03:21Z","timestamp":1701432201000},"page":"1-33","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["The Realities of Evaluating Educational Technology in School Settings"],"prefix":"10.1145","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8668-2019","authenticated-orcid":false,"given":"Megan","family":"Venn-Wycherley","sequence":"first","affiliation":[{"name":"Open Lab, Newcastle University, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3461-4161","authenticated-orcid":false,"given":"Ahmed","family":"Kharrufa","sequence":"additional","affiliation":[{"name":"Open Lab, Newcastle University, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8644-5646","authenticated-orcid":false,"given":"Susan","family":"Lechelt","sequence":"additional","affiliation":[{"name":"University of Edinburgh, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3541-0048","authenticated-orcid":false,"given":"Rebecca","family":"Nicholson","sequence":"additional","affiliation":[{"name":"Digital Learning Lab, Northumbria University, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7394-4991","authenticated-orcid":false,"given":"Kate","family":"Howland","sequence":"additional","affiliation":[{"name":"University of Sussex, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5709-2140","authenticated-orcid":false,"given":"Abrar","family":"Almjally","sequence":"additional","affiliation":[{"name":"Al Imam Mohammad Ibn Saud Islamic University\/University of Sussex, Saudi Arabia\/UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5090-0136","authenticated-orcid":false,"given":"Anthony","family":"Trory","sequence":"additional","affiliation":[{"name":"University of Sussex, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8041-0215","authenticated-orcid":false,"given":"Vidya","family":"Sarangapani","sequence":"additional","affiliation":[{"name":"Open Lab, Newcastle University, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,2,5]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1177\/2332858416667726"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3580916"},{"key":"e_1_3_1_4_2","first-page":"673","volume-title":"Proceedings of the Handbook of Qualitative Research (2nd. ed.)","author":"Angrosino M. V.","year":"2000","unstructured":"M. V. Angrosino and K. A. Mays de Perez. 2000. Rethinking observation: From method to context. In Proceedings of the Handbook of Qualitative Research (2nd. ed.). Sage, Thousand Oaks, 673\u2013702."},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1080\/1359866X.2019.1644611"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1177\/1356389014564719"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1080\/23265507.2017.1300068"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300832"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1080\/0268093022000043065"},{"key":"e_1_3_1_10_2","volume-title":"Conducting Research in Educational Contexts","author":"Basit Tehmina N.","year":"2010","unstructured":"Tehmina N. Basit. 2010. Conducting Research in Educational Contexts. Bloomsbury Publishing."},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1741-5446.2006.00241.x"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1002\/berj.3509"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/1182475.1182476"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/2804405"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1111\/bjet.12730"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1191\/1478088706qp063oa"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1080\/14780887.2020.1769238"},{"key":"e_1_3_1_18_2","volume-title":"Making Sense of Teaching","author":"Brown Sally A.","year":"1993","unstructured":"Sally A. Brown and Donald McIntyre. 1993. Making Sense of Teaching. Open University Press, Buckingham [England] ; Philadelphia."},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290607.3311757"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/2317956.2318078"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1080\/03634523.2020.1713386"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/s40692-018-0102-4"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11251-014-9339-0"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1057\/9781137025968"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1080\/09650790902914241"},{"key":"e_1_3_1_26_2","first-page":"15","article-title":"The case for randomized experiments","author":"Cook Thomas D.","year":"2003","unstructured":"Thomas D. Cook, Barbara Means, Geneva D. Haertel, and Vera Michalchik. 2003. The case for randomized experiments. Evaluating Educational Technology: Effective Research Designs for Improving Learning (2003), 15\u201337. Publisher: Teachers College Press New York.","journal-title":"Evaluating Educational Technology: Effective Research Designs for Improving Learning"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1188\/14.ONF.89-91"},{"key":"e_1_3_1_28_2","volume-title":"Teachers and Machines: The Classroom Use of Technology Since 1920","author":"Cuban Larry","year":"1986","unstructured":"Larry Cuban. 1986. Teachers and Machines: The Classroom Use of Technology Since 1920. Teachers college press."},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-99295-8_9"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/KELVAR.2016.7563677"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11528-011-0527-3"},{"issue":"3","key":"e_1_3_1_32_2","first-page":"269\u2013280.","article-title":"Technology-capable teachers transitioning to technology-challenged schools","volume":"15","author":"Derbel Faiza","year":"2017","unstructured":"Faiza Derbel. 2017. Technology-capable teachers transitioning to technology-challenged schools. Electronic Journal of e-learning 15, 3 (2017), 269\u2013280.","journal-title":"Electronic Journal of e-learning"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-05900-2_11"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4419-5716-0_26"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1177\/2096531120913039"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2012.02.001"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1177\/146879410200200205"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1177\/104973202129120052"},{"key":"e_1_3_1_39_2","volume-title":"Strategies for Evaluating Technology in Education and Training","author":"Foshay Wellesley","year":"2010","unstructured":"Wellesley Foshay and D. Quinn. 2010. Strategies for Evaluating Technology in Education and Training."},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1080\/15391523.2019.1679055"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3359292"},{"key":"e_1_3_1_42_2","volume-title":"Evaluating Educational Technology: Effective Research Designs for Improving Learning","author":"Haertel G. D.","year":"2003","unstructured":"G. D. Haertel and B. Means. 2003. Evaluating Educational Technology: Effective Research Designs for Improving Learning. Teachers College Press. 2003044046 Retrieved from https:\/\/books.google.co.uk\/books?id=U85YwnEGbxcC"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1111\/1471-0528.15199"},{"key":"e_1_3_1_44_2","first-page":"1","volume-title":"Proceedings of the Alt. Chi. Session at the SIGCHI Conference on Human Factors in Computing Systems San Jose, California","author":"Harrison Steve","year":"2007","unstructured":"Steve Harrison, Deborah Tatar, and Phoebe Sengers. 2007. The three paradigms of HCI. In Proceedings of the Alt. Chi. Session at the SIGCHI Conference on Human Factors in Computing Systems San Jose, California. 1\u201318."},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/2212877.2212890"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1300\/J025v18n02_07"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2015.09.005"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1111\/bjet.12770"},{"key":"e_1_3_1_49_2","article-title":"Implementation and process evaluation (IPE) for interventions in education settings: An introductory handbook","volume":"1","author":"Humphrey Neil","year":"2016","unstructured":"Neil Humphrey, Ann Lendrum, Emma Ashworth, Kirsty Frearson, Robert Buck, and Kirstin Kerr. 2016. Implementation and process evaluation (IPE) for interventions in education settings: An introductory handbook. Education Endowment Foundation 1 (2016).","journal-title":"Education Endowment Foundation"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11423-015-9409-2"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.4324\/9780203054000"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.edurev.2019.100284"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11423-016-9484-z"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397617.3398065"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466130"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025679"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1080\/0142159X.2020.1755030"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2019.01.010"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445263"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445749"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025535"},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.1080\/09523980802284317"},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","DOI":"10.1080\/09500790108666984"},{"key":"e_1_3_1_64_2","volume-title":"The Outward Looking School and Its Ecosystem","author":"Mueller Sabine","year":"2015","unstructured":"Sabine Mueller and Olivier Toutain. 2015. The Outward Looking School and Its Ecosystem. Technical Report. OECD. Retrieved fromhttps:\/\/www.oecd.org\/cfe\/leed\/Outward-Looking-School-and-Ecosystem.pdf"},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517667"},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2013.04.012"},{"issue":"4","key":"e_1_3_1_67_2","first-page":"20","article-title":"An introduction to the evaluation of learning technology","volume":"3","author":"Oliver Martin","year":"2000","unstructured":"Martin Oliver. 2000. An introduction to the evaluation of learning technology. Educational Technology and Society 3, 4 (2000), 20\u201330. Retrieved from https:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-0005026996&partnerID=40&md5=b68659cdead23bf12c46414ece58dc26Cited by: 88.","journal-title":"Educational Technology and Society"},{"key":"e_1_3_1_68_2","doi-asserted-by":"publisher","DOI":"10.4324\/9780429446764-14"},{"key":"e_1_3_1_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411290"},{"key":"e_1_3_1_70_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12528-018-9195-z"},{"key":"e_1_3_1_71_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-05900-2_2"},{"key":"e_1_3_1_72_2","doi-asserted-by":"publisher","DOI":"10.3102\/0013189X11421826"},{"key":"e_1_3_1_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/2940299.2940302"},{"key":"e_1_3_1_74_2","doi-asserted-by":"publisher","DOI":"10.1080\/0951839032000060635"},{"key":"e_1_3_1_75_2","doi-asserted-by":"publisher","DOI":"10.14742\/ajet.4314"},{"key":"e_1_3_1_76_2","doi-asserted-by":"publisher","DOI":"10.1093\/swr\/svu028"},{"key":"e_1_3_1_77_2","doi-asserted-by":"publisher","DOI":"10.1145\/1978822.1978834"},{"key":"e_1_3_1_78_2","volume-title":"HCI Theory: Classical, Modern, and Contemporary","author":"Rogers Yvonne","year":"2022","unstructured":"Yvonne Rogers. 2022. HCI Theory: Classical, Modern, and Contemporary. Springer Nature."},{"key":"e_1_3_1_79_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-74853-3_20"},{"key":"e_1_3_1_80_2","doi-asserted-by":"publisher","DOI":"10.2466\/pr0.1966.19.1.115"},{"issue":"1","key":"e_1_3_1_81_2","first-page":"17","article-title":"Educational technology research past and present: Balancing rigor and relevance to impact school learning","volume":"1","author":"R. Lowther D. L. Ross S. M., Morrison G.","year":"2010","unstructured":"Lowther D. L. Ross S. M., Morrison G. R.2010. Educational technology research past and present: Balancing rigor and relevance to impact school learning. Contemporary Educational Technology 1, 1 (2010), 17\u201335.","journal-title":"Contemporary Educational Technology"},{"key":"e_1_3_1_82_2","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1016\/B978-0-08-097086-8.92080-4","volume-title":"International Encyclopedia of Social and Behavioral Sciences, 2nd Edition","author":"Scheerens Jaap","year":"2015","unstructured":"Jaap Scheerens. 2015. School effectiveness research. In International Encyclopedia of Social and Behavioral Sciences, 2nd Edition. Elsevier, 80\u201385."},{"key":"e_1_3_1_83_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2729.2009.00338.x"},{"key":"e_1_3_1_84_2","doi-asserted-by":"publisher","DOI":"10.4324\/9781315886350"},{"key":"e_1_3_1_85_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.iheduc.2020.100730"},{"key":"e_1_3_1_86_2","volume-title":"Participatory Design of Science Curricula: The Case for Research for Practice.","author":"Shrader G.","year":"2001","unstructured":"G. Shrader, K. Williams, J. Lachance-Whitcomb, L.-E. Finn, and L. Gomez. 2001. Participatory Design of Science Curricula: The Case for Research for Practice.Technical Report. Seattle, WA."},{"key":"e_1_3_1_87_2","volume-title":"Doing Qualitative Research: A Practical Handbook","author":"Silverman David","year":"2013","unstructured":"David Silverman. 2013. Doing Qualitative Research: A Practical Handbook. SAGE Publications."},{"key":"e_1_3_1_88_2","doi-asserted-by":"publisher","DOI":"10.5555\/38407"},{"key":"e_1_3_1_89_2","doi-asserted-by":"publisher","DOI":"10.25304\/rlt.v25.1998"},{"key":"e_1_3_1_90_2","first-page":"221","volume-title":"Proceedings of the Education Matters","author":"Torgerson Carole J.","year":"2012","unstructured":"Carole J. Torgerson and David J. Torgerson. 2012. The need for randomised controlled trials in educational research. In Proceedings of the Education Matters. Routledge, 221\u2013232."},{"key":"e_1_3_1_91_2","doi-asserted-by":"publisher","DOI":"10.1080\/17439884.2021.1867572"},{"key":"e_1_3_1_92_2","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3366923"},{"key":"e_1_3_1_93_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300548"},{"key":"e_1_3_1_94_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290607.3311761"}],"container-title":["ACM Transactions on Computer-Human Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3635146","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3635146","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:49:06Z","timestamp":1750286946000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3635146"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,5]]},"references-count":93,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,4,30]]}},"alternative-id":["10.1145\/3635146"],"URL":"https:\/\/doi.org\/10.1145\/3635146","relation":{},"ISSN":["1073-0516","1557-7325"],"issn-type":[{"value":"1073-0516","type":"print"},{"value":"1557-7325","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,5]]},"assertion":[{"value":"2022-06-13","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-09-10","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-05","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}