{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T13:48:39Z","timestamp":1776088119877,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":39,"publisher":"ACM","funder":[{"DOI":"10.13039\/100007880","name":"Johns Hopkins University","doi-asserted-by":"publisher","award":["2023 Catalyst Award to Amanda Neitzel"],"award-info":[{"award-number":["2023 Catalyst Award to Amanda Neitzel"]}],"id":[{"id":"10.13039\/100007880","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,4,13]]},"DOI":"10.1145\/3772363.3798755","type":"proceedings-article","created":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T01:55:28Z","timestamp":1776045328000},"page":"1-8","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["MetaMate: Understanding How Educational Researchers Experience AI-Assisted Data Extraction for Systematic Reviews"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0854-6393","authenticated-orcid":false,"given":"Xue","family":"Wang","sequence":"first","affiliation":[{"name":"School of Education, Johns Hopkins University, Baltimore, Maryland, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6172-9784","authenticated-orcid":false,"given":"Gaoxiang","family":"Luo","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, University of Minnesota Twin Cities, Minneapolis, Minnesota, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2026,4,13]]},"reference":[{"key":"e_1_3_3_2_2_2","doi-asserted-by":"publisher","unstructured":"Muhammad M.\u00a0M. Abdel\u00a0Latif. 2019. Using think-aloud protocols and interviews in investigating writers\u2019 composing processes: combining concurrent and retrospective data. International Journal of Research & Method in Education 42 2 (2019) 111\u2013123. 10.1080\/1743727X.2018.1439003","DOI":"10.1080\/1743727X.2018.1439003"},{"key":"e_1_3_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613905.3636301"},{"key":"e_1_3_3_2_4_2","doi-asserted-by":"publisher","unstructured":"Rohit Borah Andrew\u00a0W Brown Patrice\u00a0L Capers and Kathryn\u00a0A Kaiser. 2017. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open 7 2 (2017) e012545. 10.1136\/bmjopen-2016-012545","DOI":"10.1136\/bmjopen-2016-012545"},{"key":"e_1_3_3_2_5_2","doi-asserted-by":"publisher","unstructured":"Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3 2 (2006) 77\u2013101. 10.1191\/1478088706qp063oa","DOI":"10.1191\/1478088706qp063oa"},{"key":"e_1_3_3_2_6_2","doi-asserted-by":"publisher","unstructured":"Olga Chernikova Matthias Stadler Ivan Melev and Frank Fischer. 2024. Using machine learning for continuous updating of meta-analysis in educational context. Computers in Human Behavior 156 (2024) 108215. 10.1016\/j.chb.2024.108215","DOI":"10.1016\/j.chb.2024.108215"},{"key":"e_1_3_3_2_7_2","doi-asserted-by":"publisher","unstructured":"Ramon Cierco\u00a0Jimenez Teresa Lee Nicol\u00e1s Rosillo Reynalda Cordova Ian\u00a0A Cree Angel Gonzalez and Blanca\u00a0Iciar Indave\u00a0Ruiz. 2022. Machine learning computational tools to assist the performance of systematic reviews: A mapping review. BMC Medical Research Methodology 22 1 (2022) 322. 10.1186\/s12874-022-01805-4","DOI":"10.1186\/s12874-022-01805-4"},{"key":"e_1_3_3_2_8_2","doi-asserted-by":"publisher","unstructured":"David\u00a0A Cook and Colin\u00a0P West. 2012. Conducting systematic reviews in medical education: a stepwise approach. Medical Education 46 10 (2012) 943\u2013952. 10.1111\/j.1365-2923.2012.04328.x","DOI":"10.1111\/j.1365-2923.2012.04328.x"},{"key":"e_1_3_3_2_9_2","volume-title":"Designing and conducting mixed methods research.","author":"Creswell John\u00a0W.","year":"2017","unstructured":"John\u00a0W. Creswell and Vicki L.\u00a0Plano Clark. 2017. Designing and conducting mixed methods research.Sage Publications, Inc, Thousand Oaks, CA, US. xviii, 275\u2013xviii, 275 pages."},{"key":"e_1_3_3_2_10_2","doi-asserted-by":"publisher","unstructured":"Jennifer Evans and Pauline Benefield. 2001. Systematic Reviews of Educational Research: Does the medical model fit? British Educational Research Journal 27 5 (2001) 527\u2013541. 10.1080\/01411920120095717","DOI":"10.1080\/01411920120095717"},{"key":"e_1_3_3_2_11_2","doi-asserted-by":"publisher","unstructured":"Gerald Gartlehner Shannon Kugley Karen Crotty Meera Viswanathan Andreea Dobrescu Barbara Nussbaumer-Streit Graham Booth Jonathan\u00a0R Treadwell Jung\u00a0Min Han Jesse Wagner Eric\u00a0A Apaydin Erin\u00a0L Coppola Margaret Maglione Rainer Hilscher Robert Chew Meagan Pilar Bryan Swanton and Leila\u00a0C Kahwati. 2025. Artificial Intelligence\u2013Assisted Data Extraction With a Large Language Model: A Study Within Reviews. Annals of Internal Medicine 178 12 (2025) 1763\u20131771. 10.7326\/ANNALS-25-00739","DOI":"10.7326\/ANNALS-25-00739"},{"key":"e_1_3_3_2_12_2","doi-asserted-by":"publisher","unstructured":"Ella Glikson and Anita\u00a0Williams Woolley. 2020. Human trust in artificial intelligence: Review of empirical research. Academy of Management Annals 14 2 (2020) 627\u2013660. 10.5465\/annals.2018.0057","DOI":"10.5465\/annals.2018.0057"},{"key":"e_1_3_3_2_13_2","doi-asserted-by":"publisher","unstructured":"Peter\u00a0C G\u00f8tzsche Asbj\u00f8rn Hr\u00f3bjartsson Katja Mari\u0107 and Britta Tendal. 2007. Data Extraction Errors in Meta-analyses That Use Standardized Mean Differences. JAMA 298 4 (2007) 430\u2013437. 10.1001\/jama.298.4.430","DOI":"10.1001\/jama.298.4.430"},{"key":"e_1_3_3_2_14_2","doi-asserted-by":"publisher","unstructured":"Eliza\u00a0M Grames Andrew\u00a0N Stillman Morgan\u00a0W Tingley and Chris\u00a0S Elphick. 2019. An automated approach to identifying search terms for systematic reviews using keyword co-occurrence networks. Methods in Ecology and Evolution 10 10 (2019) 1645\u20131654. 10.1111\/2041-210X.13268","DOI":"10.1111\/2041-210X.13268"},{"key":"e_1_3_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1002\/9781119536604"},{"key":"e_1_3_3_2_16_2","doi-asserted-by":"publisher","unstructured":"Jackie Horton Ben Vandermeer Lisa Hartling Lisa Tjosvold Terry\u00a0P Klassen and Nina Buscemi. 2010. Systematic review data extraction: cross-sectional study showed that experience did not increase accuracy. Journal of Clinical Epidemiology 63 3 (2010) 289\u2013298. 10.1016\/j.jclinepi.2009.04.007","DOI":"10.1016\/j.jclinepi.2009.04.007"},{"key":"e_1_3_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/302979.303030"},{"key":"e_1_3_3_2_18_2","doi-asserted-by":"publisher","unstructured":"Qusai Khraisha Sophie Put Johanna Kappenberg Azza Warraitch and Kristin Hadfield. 2024. Can large language models replace humans in systematic reviews? Evaluating GPT-4\u2019s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. Research Synthesis Methods 15 4 (2024) 616\u2013626. 10.1002\/jrsm.1715","DOI":"10.1002\/jrsm.1715"},{"key":"e_1_3_3_2_19_2","doi-asserted-by":"publisher","unstructured":"Marc\u00a0J Lajeunesse. 2016. Facilitating systematic reviews data extraction and meta-analysis with the metagear package for R. Methods in Ecology and Evolution 7 3 (2016) 323\u2013330. 10.1111\/2041-210X.12472","DOI":"10.1111\/2041-210X.12472"},{"key":"e_1_3_3_2_20_2","doi-asserted-by":"publisher","unstructured":"John\u00a0D Lee and Katrina\u00a0A See. 2004. Trust in Automation: Designing for Appropriate Reliance. Human Factors: The Journal of the Human Factors and Ergonomics Society 46 1 (2004) 50\u201380. 10.1518\/hfes.46.1.50.30392","DOI":"10.1518\/hfes.46.1.50.30392"},{"key":"e_1_3_3_2_21_2","doi-asserted-by":"publisher","unstructured":"Lingbo Li Anuradha Mathrani and Teo Susnjak. 2025. What Level of Automation is \"Good Enough\"? A Benchmark of Large Language Models for Meta-Analysis Data Extraction. arxiv:https:\/\/arXiv.org\/abs\/2507.15152\u00a0[cs.CL] 10.48550\/arXiv.2507.15152","DOI":"10.48550\/arXiv.2507.15152"},{"key":"e_1_3_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-4002"},{"key":"e_1_3_3_2_23_2","doi-asserted-by":"publisher","unstructured":"Iain\u00a0J Marshall and Byron\u00a0C Wallace. 2019. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Systematic Reviews 8 1 (2019) 163. 10.1186\/s13643-019-1074-9","DOI":"10.1186\/s13643-019-1074-9"},{"key":"e_1_3_3_2_24_2","doi-asserted-by":"publisher","unstructured":"Tim Mathes Pauline Kla\u00dfen and Dawid Pieper. 2017. Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. BMC Medical Research Methodology 17 1 (2017) 152. 10.1186\/s12874-017-0431-4","DOI":"10.1186\/s12874-017-0431-4"},{"key":"e_1_3_3_2_25_2","doi-asserted-by":"publisher","unstructured":"Luke\u00a0A McGuinness and Julian P\u00a0T Higgins. 2021. Risk-of-bias VISualization (robvis): An R package and Shiny web app for visualizing risk-of-bias assessments. Research Synthesis Methods 12 1 (2021) 55\u201361. 10.1002\/jrsm.1411","DOI":"10.1002\/jrsm.1411"},{"key":"e_1_3_3_2_26_2","doi-asserted-by":"publisher","unstructured":"Matthew Michelson and Katja Reuter. 2019. The significant cost of systematic reviews and meta-analyses: A call for greater involvement of machine learning to assess the promise of clinical trials. Contemporary Clinical Trials Communications 16 (2019) 100443. 10.1016\/j.conctc.2019.100443","DOI":"10.1016\/j.conctc.2019.100443"},{"key":"e_1_3_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-658-27602-7_1"},{"key":"e_1_3_3_2_28_2","doi-asserted-by":"publisher","unstructured":"Jakob Nielsen. 1994. Estimating the Number of Subjects Needed for a Thinking Aloud Test. International Journal of Human-Computer Studies 41 3 (1994) 385\u2013397. 10.1006\/ijhc.1994.1065","DOI":"10.1006\/ijhc.1994.1065"},{"key":"e_1_3_3_2_29_2","doi-asserted-by":"publisher","unstructured":"Mourad Ouzzani Hossam Hammady Zbys Fedorowicz and Ahmed Elmagarmid. 2016. Rayyan\u2014a web and mobile app for systematic reviews. Systematic Reviews 5 1 (2016) 210. 10.1186\/s13643-016-0384-4","DOI":"10.1186\/s13643-016-0384-4"},{"key":"e_1_3_3_2_30_2","doi-asserted-by":"publisher","unstructured":"Richeek Pradhan David\u00a0C Hoaglin Matthew Cornell Weisong Liu Victoria Wang and Hong Yu. 2019. Automatic extraction of quantitative data from ClinicalTrials.gov to conduct meta-analyses. Journal of Clinical Epidemiology 105 (2019) 92\u2013100. 10.1016\/j.jclinepi.2018.08.023","DOI":"10.1016\/j.jclinepi.2018.08.023"},{"key":"e_1_3_3_2_31_2","doi-asserted-by":"publisher","unstructured":"Lena Schmidt Kaitlyn Hair Sergio Graziosi Fiona Campbell Claudia Kapp Alireza Khanteymoori Dawn Craig Mark Engelbert and James Thomas. 2025. Exploring the use of a Large Language Model for data extraction in systematic reviews: a rapid feasibility study. arxiv:https:\/\/arXiv.org\/abs\/2405.14445\u00a0[cs.CL] 10.48550\/arXiv.2405.14445","DOI":"10.48550\/arXiv.2405.14445"},{"key":"e_1_3_3_2_32_2","doi-asserted-by":"publisher","unstructured":"Lena Schmidt Babatunde\u00a0K Olorisade Luke\u00a0A McGuinness James Thomas and Julian P\u00a0T Higgins. 2021. Data extraction methods for systematic review (semi)automation: A living systematic review. F1000Research 10 (2021) 401. 10.12688\/f1000research.51117.1","DOI":"10.12688\/f1000research.51117.1"},{"key":"e_1_3_3_2_33_2","doi-asserted-by":"publisher","unstructured":"Esra Sever and Hayati Akyol. 2022. The impact of collaborative learning techniques on written expression self-regulation and writing motivation. International Electronic Journal of Elementary Education 14 5 (2022) 587\u2013603. 10.26822\/iejee.2022.265","DOI":"10.26822\/iejee.2022.265"},{"key":"e_1_3_3_2_34_2","doi-asserted-by":"publisher","unstructured":"Sabrina Vandevelde Hilde Van\u00a0Keer and Emmelien Merchie. 2017. The challenge of promoting self-regulated learning among primary school children with a low socioeconomic and immigrant background. The Journal of Educational Research 110 2 (2017) 113\u2013139. 10.1080\/00220671.2014.999363","DOI":"10.1080\/00220671.2014.999363"},{"key":"e_1_3_3_2_35_2","doi-asserted-by":"publisher","unstructured":"Byron\u00a0C. Wallace Jo\u00ebl Kuiper Aakash Sharma Mingxi Zhu and Iain\u00a0J. Marshall. 2016. Extracting PICO sentences from clinical trial reports using supervised distant supervision. J. Mach. Learn. Res. 17 1 (Jan. 2016) 4572\u20134596. 10.5555\/2946645.3007085","DOI":"10.5555\/2946645.3007085"},{"key":"e_1_3_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3641917"},{"key":"e_1_3_3_2_37_2","doi-asserted-by":"publisher","unstructured":"Martin\u00a0J Westgate. 2019. revtools: An R package to support article screening for evidence synthesis. Research Synthesis Methods 10 4 (2019) 606\u2013614. 10.1002\/jrsm.1374","DOI":"10.1002\/jrsm.1374"},{"key":"e_1_3_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2306.13063"},{"key":"e_1_3_3_2_39_2","doi-asserted-by":"publisher","unstructured":"Chang Xu Suhail\u00a0A.R. Doi Xiaoqin Zhou Lifeng Lin Luis Furuya-Kanamori and Fangbiao Tao. 2022. Data reproducibility issues and their potential impact on conclusions from evidence syntheses of randomized controlled trials in sleep medicine. Sleep Medicine Reviews 66 (Dec. 2022) 101708. 10.1016\/j.smrv.2022.101708","DOI":"10.1016\/j.smrv.2022.101708"},{"key":"e_1_3_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-658-27602-7"}],"event":{"name":"CHI EA '26: Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems","location":"Barcelona , Spain","acronym":"CHI EA '26","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Proceedings of the Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3772363.3798755","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T12:53:03Z","timestamp":1776084783000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3772363.3798755"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,13]]},"references-count":39,"alternative-id":["10.1145\/3772363.3798755","10.1145\/3772363"],"URL":"https:\/\/doi.org\/10.1145\/3772363.3798755","relation":{},"subject":[],"published":{"date-parts":[[2026,4,13]]},"assertion":[{"value":"2026-04-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}