{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T22:53:10Z","timestamp":1781650390503,"version":"3.54.5"},"reference-count":73,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T00:00:00Z","timestamp":1776124800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T00:00:00Z","timestamp":1780617600000},"content-version":"vor","delay-in-days":52,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Generative AI\u2013Empowered Enhancement of University Teachers\u2019 Digital Literacy and Optimization Pathways of Teaching Practices","award":["CX20250740"],"award-info":[{"award-number":["CX20250740"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Despite a burgeoning body of research on using AI scoring systems in English assessments, concerns regarding their reliability persist. To fill this gap, this meta-analysis examined the AI-human scoring differences and the variables moderating these differences by synthesizing the results of 21 empirical studies with a total of 401,698 participants. Results indicate no statistically significant differences between AI and human scoring; the small effect size implies that the average systematic difference between the two was relatively modest. However, extremely high heterogeneity suggests that this overall finding masks considerable variability across study conditions. Moderator analyses reveal that AI-human scoring differences are significantly influenced by factors such as AI system type, number of human raters, agreement index employed, learner proficiency level (CEFR), and publication year. These findings suggest that while AI cannot fully replace human judgment, it can serve as a diagnostic reference tool within broader quality assurance frameworks. When significant discrepancies arise, they warrant investigation of both scoring sources. Based on these findings, this study offers evidence-based recommendations for educators on the effective use of AI scoring systems in language assessments.<\/jats:p>","DOI":"10.1038\/s41598-026-48053-w","type":"journal-article","created":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T10:35:17Z","timestamp":1776162917000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Differences between human and AI scoring: A meta-analysis of english language assessments"],"prefix":"10.1038","volume":"16","author":[{"given":"Bingwei","family":"Li","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiao","family":"Qunhan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chunhua","family":"Mao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,4,14]]},"reference":[{"key":"48053_CR1","first-page":"386","volume":"4","author":"CM Myford","year":"2003","unstructured":"Myford, C. M. & Wolfe, E. W. Detecting and measuring rater effects using many-facet Rasch measurement: Part I. J. Appl. Meas. 4, 386\u2013422 (2003).","journal-title":"J. Appl. Meas."},{"key":"48053_CR2","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1080\/15434303.2020.1731752","volume":"17","author":"MS Park","year":"2020","unstructured":"Park, M. S. Rater effects on L2 oral assessment: Focusing on accent familiarity of L2 teachers. Lang. Assess. Q. 17, 231\u2013243. https:\/\/doi.org\/10.1080\/15434303.2020.1731752 (2020).","journal-title":"Lang. Assess. Q."},{"key":"48053_CR3","doi-asserted-by":"publisher","first-page":"602","DOI":"10.1177\/0265532221994052","volume":"38","author":"U Knoch","year":"2021","unstructured":"Knoch, U., Deygers, B. & Khamboonruang, A. Revisiting rating scale development for rater-mediated language performance assessments: Modelling construct and contextual choices made by scale developers. Lang Test. 38, 602\u2013626. https:\/\/doi.org\/10.1177\/0265532221994052 (2021).","journal-title":"Lang Test."},{"key":"48053_CR4","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1080\/15434300903464418","volume":"7","author":"K Barkaoui","year":"2010","unstructured":"Barkaoui, K. Variability in ESL essay rating processes: The role of the rating scale and rater experience. Lang. Assess. Q. 7, 54\u201374. https:\/\/doi.org\/10.1080\/15434300903464418 (2010).","journal-title":"Lang. Assess. Q."},{"key":"48053_CR5","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1177\/0265532211411078","volume":"29","author":"B Bridgeman","year":"2012","unstructured":"Bridgeman, B., Powers, D., Stone, E. & Mollaun, P. TOEFL iBT speaking test scores as indicators of oral communicative language proficiency. Lang. Test. 29, 91\u2013108. https:\/\/doi.org\/10.1177\/0265532211411078 (2012).","journal-title":"Lang. Test."},{"key":"48053_CR6","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1177\/0265532211430367","volume":"29","author":"T McNamara","year":"2012","unstructured":"McNamara, T. & Knoch, U. The Rasch wars: The emergence of Rasch measurement in language testing. Lang. Test. 29, 555\u2013576. https:\/\/doi.org\/10.1177\/0265532211430367 (2012).","journal-title":"Lang. Test."},{"key":"48053_CR7","first-page":"3","volume":"10","author":"A Phakiti","year":"2021","unstructured":"Phakiti, A. & Isaacs, T. Classroom assessment and validity: Psychometric and edumetric approaches. Eur. J. Appl. Linguist. TEFL 10, 3\u201324 (2021).","journal-title":"Eur. J. Appl. Linguist. TEFL"},{"key":"48053_CR8","doi-asserted-by":"crossref","unstructured":"Shermis, M. D. & Burstein, J. (eds) Handbook of automated essay evaluation (Routledge, 2013).","DOI":"10.4324\/9780203122761"},{"key":"48053_CR9","unstructured":"Lu, J., Batra, D., Parikh, D., Lee, S. & Vilbert Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances Neural Inform. Process. Systems 32 (2019)."},{"key":"48053_CR10","first-page":"26","volume":"12","author":"K Barkaoui","year":"2023","unstructured":"Barkaoui, K. & Woodworth, J. An exploratory study of the construct measured by automated writing scores across task types and test occasions. Stud. Lang. 12, 26 (2023).","journal-title":"Stud. Lang."},{"key":"48053_CR11","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1080\/0969594X.2020.1735995","volume":"28","author":"L Gu","year":"2021","unstructured":"Gu, L., Davis, L., Tao, J. & Zechner, K. Using spoken language technology for generating feedback to prepare for the TOEFL iBT\u00ae test: A user perception study. Assess. Educ. Princ. Policy Pract. 28, 58\u201376. https:\/\/doi.org\/10.1080\/0969594X.2020.1735995 (2021).","journal-title":"Assess. Educ. Princ. Policy Pract."},{"key":"48053_CR12","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1177\/1461445605050365","volume":"7","author":"K Hyland","year":"2005","unstructured":"Hyland, K. Stance and engagement: A model of interaction in academic discourse. Discourse Stud. 7, 173\u2013192. https:\/\/doi.org\/10.1177\/1461445605050365 (2005).","journal-title":"Discourse Stud."},{"key":"48053_CR13","doi-asserted-by":"publisher","first-page":"219","DOI":"10.1080\/15434303.2018.1453816","volume":"15","author":"E Galaczi","year":"2018","unstructured":"Galaczi, E. & Taylor, L. Interactional competence: Conceptualisations, operationalisations, and outstanding questions. Lang. Assess. Q. 15, 219\u2013236. https:\/\/doi.org\/10.1080\/15434303.2018.1453816 (2018).","journal-title":"Lang. Assess. Q."},{"key":"48053_CR14","doi-asserted-by":"publisher","first-page":"2041","DOI":"10.1007\/s10639-024-12891-w","volume":"30","author":"NM Bui","year":"2025","unstructured":"Bui, N. M. & Barrot, J. S. ChatGPT as an automated essay scoring tool in the writing classrooms: How it compares with human scoring. Educ. Inf. Technol. 30, 2041\u20132058. https:\/\/doi.org\/10.1007\/s10639-024-12891-w (2025).","journal-title":"Educ. Inf. Technol."},{"key":"48053_CR15","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1080\/15434303.2023.2291488","volume":"20","author":"X Xi","year":"2023","unstructured":"Xi, X. Advancing language assessment with AI and ML\u2013leaning into AI is inevitable, but can theory keep up? Lang. Assess. Q. 20, 357\u2013376. https:\/\/doi.org\/10.1080\/15434303.2023.2291488 (2023).","journal-title":"Lang. Assess. Q."},{"key":"48053_CR16","doi-asserted-by":"publisher","first-page":"44","DOI":"10.26970\/j.cnki.fltt.1004-5112.2018.03.006","volume":"3","author":"LF Bai","year":"2018","unstructured":"Bai, L. F. & Wang, J. \u4eba\u5de5\u548c\u673a\u5668\u8bc4\u5206\u5dee\u5f02\u6bd4\u8f83\u53ca\u6210\u56e0\u5206\u6790 [A comparison of differences between human and machine scoring and an analysis of their causes]. Foreign Lang. Test. Teach. 3, 44\u201354. https:\/\/doi.org\/10.26970\/j.cnki.fltt.1004-5112.2018.03.006 (2018).","journal-title":"Foreign Lang. Test. Teach."},{"key":"48053_CR17","doi-asserted-by":"publisher","DOI":"10.1186\/s40862-022-00171-4","volume":"7","author":"H Chen","year":"2022","unstructured":"Chen, H. & Pan, J. Computer or human: A comparative study of automated evaluation scoring and instructors\u2019 feedback on Chinese college students\u2019 English writing. Asian-Pac. J. Second Foreign Lang. Educ. 7, 34. https:\/\/doi.org\/10.1186\/s40862-022-00171-4 (2022).","journal-title":"Asian-Pac. J. Second Foreign Lang. Educ."},{"key":"48053_CR18","doi-asserted-by":"crossref","unstructured":"Eckes, T. Introduction to many-facet Rasch measurement (Peter Lang, 2023).","DOI":"10.3726\/b20875"},{"key":"48053_CR19","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/001316446002000104","volume":"20","author":"J Cohen","year":"1960","unstructured":"Cohen, J. A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20, 37\u201346 (1960).","journal-title":"Educ. Psychol. Meas."},{"key":"48053_CR20","doi-asserted-by":"publisher","first-page":"276","DOI":"10.11613\/BM.2012.031","volume":"22","author":"ML McHugh","year":"2012","unstructured":"McHugh, M. L. Interrater reliability: The kappa statistic. Biochem. Med. 22, 276\u2013282 (2012).","journal-title":"Biochem. Med."},{"key":"48053_CR21","doi-asserted-by":"publisher","first-page":"291","DOI":"10.1177\/0265532210364643","volume":"27","author":"X Xi","year":"2010","unstructured":"Xi, X. Automated scoring and feedback systems: Where are we and where are we heading? Lang. Test. 27, 291\u2013300. https:\/\/doi.org\/10.1177\/0265532210364643 (2010).","journal-title":"Lang. Test."},{"key":"48053_CR22","doi-asserted-by":"crossref","unstructured":"Weigle, S. C. Assessing writing (Cambridge University Press, 2002).","DOI":"10.1017\/CBO9780511732997"},{"key":"48053_CR23","first-page":"189","volume":"5","author":"CM Myford","year":"2004","unstructured":"Myford, C. M. & Wolfe, E. W. Detecting and measuring rater effects using many-facet Rasch measurement: Part II. J. Appl. Meas. 5, 189\u2013227 (2004).","journal-title":"J. Appl. Meas."},{"key":"48053_CR24","doi-asserted-by":"crossref","unstructured":"Barkaoui, K. & Hadidi, A. Assessing change in English second language writing performance (Routledge, 2020).","DOI":"10.4324\/9781003092346"},{"key":"48053_CR25","doi-asserted-by":"publisher","first-page":"1547","DOI":"10.1002\/sim.2294","volume":"25","author":"Y Saito","year":"2006","unstructured":"Saito, Y., Sozu, T., Hamada, C. & Yoshimura, I. Effective number of subjects and number of raters for inter-rater reliability studies. Stat. Med. 25, 1547\u20131560. https:\/\/doi.org\/10.1002\/sim.2294 (2006).","journal-title":"Stat. Med."},{"key":"48053_CR26","doi-asserted-by":"publisher","unstructured":"Guo, S. et al. Artificial intelligence bias on English language learners in automatic scoring. in International Conference on Artificial Intelligence in Education 268\u2013275 (2025). https:\/\/doi.org\/10.1007\/978-3-031-98462-4_34","DOI":"10.1007\/978-3-031-98462-4_34"},{"key":"48053_CR27","doi-asserted-by":"publisher","DOI":"10.1016\/j.asw.2024.100815","volume":"60","author":"J Wilson","year":"2024","unstructured":"Wilson, J. & Huang, Y. Validity of automated essay scores for elementary-age English language learners: Evidence of bias?. Assess. Writing 60, 100815. https:\/\/doi.org\/10.1016\/j.asw.2024.100815 (2024).","journal-title":"Assess. Writing"},{"key":"48053_CR28","doi-asserted-by":"publisher","first-page":"1155","DOI":"10.1080\/02635143.2021.1993179","volume":"41","author":"KKC Cheung","year":"2023","unstructured":"Cheung, K. K. C. & Tai, K. W. The use of intercoder reliability in qualitative interview data analysis in science education. Res. Sci. Technol. Educ. 41, 1155\u20131175. https:\/\/doi.org\/10.1080\/02635143.2021.1993179 (2023).","journal-title":"Res. Sci. Technol. Educ."},{"key":"48053_CR29","doi-asserted-by":"publisher","first-page":"493","DOI":"10.1002\/rrq.281","volume":"55","author":"R Correnti","year":"2020","unstructured":"Correnti, R. et al. Automated scoring of students\u2019 use of text evidence in writing. Read. Res. Q. 55, 493\u2013520. https:\/\/doi.org\/10.1002\/rrq.281 (2020).","journal-title":"Read. Res. Q."},{"key":"48053_CR30","doi-asserted-by":"publisher","first-page":"1227","DOI":"10.3758\/s13428-015-0651-7","volume":"48","author":"SA Crossley","year":"2016","unstructured":"Crossley, S. A., Kyle, K. & McNamara, D. S. The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion. Behav. Res. Methods 48, 1227\u20131237. https:\/\/doi.org\/10.3758\/s13428-015-0651-7 (2016).","journal-title":"Behav. Res. Methods"},{"key":"48053_CR31","doi-asserted-by":"publisher","first-page":"1385","DOI":"10.1001\/jama.1990.03440100097014","volume":"263","author":"K Dickersin","year":"1990","unstructured":"Dickersin, K. The existence of publication bias and risk factors for its occurrence. JAMA 263, 1385\u20131389. https:\/\/doi.org\/10.1001\/jama.1990.03440100097014 (1990).","journal-title":"JAMA"},{"key":"48053_CR32","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1111\/j.1745-3992.2011.00223.x","volume":"31","author":"DM Williamson","year":"2012","unstructured":"Williamson, D. M., Xi, X. & Breyer, F. J. A framework for evaluation and use of automated scoring. Educ. Meas. Issues Pract. 31, 2\u201313. https:\/\/doi.org\/10.1111\/j.1745-3992.2011.00223.x (2012).","journal-title":"Educ. Meas. Issues Pract."},{"key":"48053_CR33","doi-asserted-by":"publisher","first-page":"322","DOI":"10.3991\/ijet.v16i11.19657","volume":"16","author":"J Gao","year":"2021","unstructured":"Gao, J. Exploring the feedback quality of an automated writing evaluation system Pigai. Int. J. Emerg. Technol. Learn. (iJET). 16, 322\u2013330. https:\/\/doi.org\/10.3991\/ijet.v16i11.19657 (2021).","journal-title":"Int. J. Emerg. Technol. Learn. (iJET)"},{"key":"48053_CR34","doi-asserted-by":"publisher","first-page":"104","DOI":"10.1016\/j.asw.2014.02.001","volume":"21","author":"L Perelman","year":"2014","unstructured":"Perelman, L. When the state of the art is counting words. Assess. Writ. 21, 104\u2013111. https:\/\/doi.org\/10.1016\/j.asw.2014.02.001 (2014).","journal-title":"Assess. Writ."},{"key":"48053_CR35","doi-asserted-by":"publisher","first-page":"20","DOI":"10.21449\/ijate.1517994","volume":"12","author":"AC Uyar","year":"2025","unstructured":"Uyar, A. C. & B\u00fcy\u00fckah\u0131ska, D. Artificial intelligence as an automated essay scoring tool: A focus on ChatGPT. Int. J. Assess. Tools Educ. 12, 20\u201332. https:\/\/doi.org\/10.21449\/ijate.1517994 (2025).","journal-title":"Int. J. Assess. Tools Educ."},{"key":"48053_CR36","doi-asserted-by":"publisher","DOI":"10.4324\/9780203771587","author":"J Cohen","year":"2013","unstructured":"Cohen, J. Statistical power analysis for the behavioral sciences (2nd ed). Routledge https:\/\/doi.org\/10.4324\/9780203771587 (2013).","journal-title":"Routledge"},{"key":"48053_CR37","doi-asserted-by":"publisher","first-page":"957","DOI":"10.1007\/s40299-024-00865-y","volume":"33","author":"X Huang","year":"2024","unstructured":"Huang, X., Xu, W., Li, F. & Yu, Z. A meta-analysis of effects of automated writing evaluation on anxiety, motivation, and second language writing skills. Asia Pac. Educ. Res. 33, 957\u2013976. https:\/\/doi.org\/10.1007\/s40299-024-00865-y (2024).","journal-title":"Asia Pac. Educ. Res."},{"key":"48053_CR38","doi-asserted-by":"publisher","first-page":"25","DOI":"10.19360\/j.cnki.11-3303\/g4.2020.07.004","volume":"7","author":"Y Jin","year":"2020","unstructured":"Jin, Y., Wang, W., Zhang, X. Y. & Zhao, Y. H. \u5927\u5b66\u82f1\u8bed\u56db\u7ea7\u53e3\u8bed\u8003\u8bd5\u81ea\u52a8\u8bc4\u5206\u6548\u5ea6\u521d\u63a2 [A preliminary study on the validity of automated scoring for the CET-4 spoken English test]. China Exam. 7, 25\u201333. https:\/\/doi.org\/10.19360\/j.cnki.11-3303\/g4.2020.07.004 (2020).","journal-title":"China Exam."},{"key":"48053_CR39","doi-asserted-by":"publisher","first-page":"130","DOI":"10.1016\/j.edurev.2007.05.002","volume":"2","author":"A Jonsson","year":"2007","unstructured":"Jonsson, A. & Svingby, G. The use of scoring rubrics: Reliability, validity and educational consequences. Educ. Res. Rev. 2, 130\u2013144. https:\/\/doi.org\/10.1016\/j.edurev.2007.05.002 (2007).","journal-title":"Educ. Res. Rev."},{"key":"48053_CR40","first-page":"1","volume":"6","author":"J Wang","year":"2007","unstructured":"Wang, J. & Brown, M. S. Automated essay scoring versus human scoring: A comparative study. J. Technol. Learn. Assess. 6, 1\u201328 (2007).","journal-title":"J. Technol. Learn. Assess."},{"key":"48053_CR41","first-page":"310","volume":"8","author":"J Wang","year":"2008","unstructured":"Wang, J. & Brown, M. S. Automated essay scoring versus human scoring: A correlational study. Contemp. Issues Technol. Teacher Educ. 8, 310\u2013325 (2008).","journal-title":"Contemp. Issues Technol. Teacher Educ."},{"key":"48053_CR42","doi-asserted-by":"publisher","unstructured":"Wang, Z. & von Davier, A. A. Monitoring of scoring using the e-rater\u00ae automated scoring system and human raters on a writing test. ETS Research Report Series 1\u201321, (2014). https:\/\/doi.org\/10.1002\/ets2.12005 (2014).","DOI":"10.1002\/ets2.12005"},{"key":"48053_CR43","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1080\/15434303.2016.1230121","volume":"13","author":"GTL Hoang","year":"2016","unstructured":"Hoang, G. T. L. & Kunnan, A. J. Automated essay evaluation for English language learners: A case study of MY access. Lang. Assess. Q. 13, 359\u2013376. https:\/\/doi.org\/10.1080\/15434303.2016.1230121 (2016).","journal-title":"Lang. Assess. Q."},{"key":"48053_CR44","unstructured":"Yang, Z. et al. Xlnet: Generalized autoregressive pretraining for language understanding. Advances Neural Inform. Process. Systems 32 (2019)."},{"key":"48053_CR45","doi-asserted-by":"publisher","DOI":"10.1177\/21582440231193818","volume":"13","author":"B Zou","year":"2023","unstructured":"Zou, B., Du, Y., Wang, Z., Chen, J. & Zhang, W. An investigation into artificial intelligence speech evaluation programs with automatic feedback for developing EFL learners\u2019 speaking skills. Sage Open 13, 21582440231193818. https:\/\/doi.org\/10.1177\/21582440231193818 (2023).","journal-title":"Sage Open"},{"key":"48053_CR46","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1017\/S027226311900010X","volume":"41","author":"K Barkaoui","year":"2019","unstructured":"Barkaoui, K. What can L2 writers\u2019 pausing behavior tell us about their L2 writing processes?. Stud. Second Lang. Acquis. 41, 529\u2013554. https:\/\/doi.org\/10.1017\/S027226311900010X (2019).","journal-title":"Stud. Second Lang. Acquis."},{"key":"48053_CR47","doi-asserted-by":"publisher","first-page":"411","DOI":"10.1080\/0969594X.2021.1979467","volume":"28","author":"J Xu","year":"2021","unstructured":"Xu, J., Jones, E., Laxton, V. & Galaczi, E. Assessing L2 English speaking using automated scoring technology: Examining automarker reliability. Assess. Educ. Princ. Policy Pract. 28, 411\u2013436. https:\/\/doi.org\/10.1080\/0969594X.2021.1979467 (2021).","journal-title":"Assess. Educ. Princ. Policy Pract."},{"key":"48053_CR48","first-page":"84","volume":"38","author":"JH Lu","year":"2022","unstructured":"Lu, J. H. \u4eba\u5de5\u667a\u80fd\u80cc\u666f\u4e0b\u673a\u5668\u8bc4\u5206\u4e0e\u4eba\u5de5\u8bc4\u5206\u7684\u6548\u5ea6\u6bd4\u8f83\u2014\u2014\u4ee5\u82f1\u8bed\u5b66\u4e60\u8005\u6545\u4e8b\u590d\u8ff0\u8bc4\u5206\u4e3a\u4f8b [A comparative study on the validity of machine scoring versus human scoring in artificial intelligence: A case study of English learners\u2019 story retelling assessment]. Journal of Chengdu Normal University 38, 84\u201392 (2022).","journal-title":"Journal of Chengdu Normal University"},{"key":"48053_CR49","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.b2535","author":"D Moher","year":"2009","unstructured":"Moher, D., Liberati, A., Tetzlaff, J. & Altman, D. G. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement.. BMJ https:\/\/doi.org\/10.1136\/bmj.b2535 (2009).","journal-title":"BMJ"},{"key":"48053_CR50","doi-asserted-by":"crossref","unstructured":"Chong, S. W. Synthesis methods and reporting tool (SMART) for research syntheses in applied linguistics.. Res. Synth. Appl. Linguist. 22. (2025).","DOI":"10.31219\/osf.io\/b5a87_v1"},{"key":"48053_CR51","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2024.100234","volume":"6","author":"A Pack","year":"2024","unstructured":"Pack, A., Barrett, A. & Escalante, J. Large language models and automated essay scoring of English language learner writing: Insights into validity and reliability. Computers and Education: Artificial Intelligence 6, 100234. https:\/\/doi.org\/10.1016\/j.caeai.2024.100234 (2024).","journal-title":"Computers and Education: Artificial Intelligence"},{"key":"48053_CR52","doi-asserted-by":"publisher","unstructured":"Schmidgall, J. E. The consistency of TOEIC\u00ae Speaking scores across ratings and tasks. ETS Research Report Series 1\u20138, (2017). https:\/\/doi.org\/10.1002\/ets2.12178 (2017).","DOI":"10.1002\/ets2.12178"},{"key":"48053_CR53","doi-asserted-by":"publisher","first-page":"57","DOI":"10.13978\/j.cnki.wyyj.2020.04.009","volume":"37","author":"HY Sun","year":"2020","unstructured":"Sun, H. Y. & Zhang, M. \u82f1\u8bed\u53e3\u8bed\u673a\u5668\u8bc4\u5206\u548c\u4eba\u5de5\u8bc4\u5206\u7684\u5bf9\u6bd4\u7814\u7a76 [A comparative study of machine scoring and human scoring in English speaking assessment]. Foreign Lang. Res. 37, 57\u201362. https:\/\/doi.org\/10.13978\/j.cnki.wyyj.2020.04.009 (2020).","journal-title":"Foreign Lang. Res."},{"key":"48053_CR54","doi-asserted-by":"publisher","DOI":"10.1016\/j.heliyon.2024.e34262","author":"X Tang","year":"2024","unstructured":"Tang, X., Chen, H., Lin, D. & Li, K. Harnessing LLMs for multi-dimensional writing assessment: Reliability and alignment with human judgments. Heliyon https:\/\/doi.org\/10.1016\/j.heliyon.2024.e34262 (2024).","journal-title":"Heliyon"},{"key":"48053_CR55","doi-asserted-by":"publisher","first-page":"36","DOI":"10.30191\/ETS.202507_28(3).SP04","volume":"28","author":"AC Topuz","year":"2025","unstructured":"Topuz, A. C., Y\u0131ld\u0131z, M., Ta\u015fl\u0131beyaz, E., Polat, H. & Kur\u015fun, E. Is generative AI ready to replace human raters in scoring EFL writing? Comparison of human and automated essay evaluation. Educ. Technol. Soc. 28, 36\u201350. https:\/\/doi.org\/10.30191\/ETS.202507_28(3).SP04 (2025).","journal-title":"Educ. Technol. Soc."},{"key":"48053_CR56","doi-asserted-by":"crossref","unstructured":"Toranj, S. & Ansari, D. N. Automated versus human essay scoring: A comparative study. Theory & Practice in Language Studies (TPLS) 2 (2012).","DOI":"10.4304\/tpls.2.4.719-725"},{"key":"48053_CR57","doi-asserted-by":"publisher","DOI":"10.1016\/j.asw.2023.100719","volume":"56","author":"Y Vo","year":"2023","unstructured":"Vo, Y., Rickels, H., Welch, C. & Dunbar, S. Human scoring versus automated scoring for English learners in a statewide evidence-based writing assessment. Assess. Writing 56, 100719. https:\/\/doi.org\/10.1016\/j.asw.2023.100719 (2023).","journal-title":"Assess. Writing"},{"key":"48053_CR58","doi-asserted-by":"publisher","first-page":"68","DOI":"10.5539\/ijel.v11n2p68","volume":"11","author":"J Wang","year":"2021","unstructured":"Wang, J. & Bai, L. Unveiling the scoring validity of two Chinese automated writing evaluation systems: A quantitative study. Int. J. Engl. Linguist. 11, 68\u201384. https:\/\/doi.org\/10.5539\/ijel.v11n2p68 (2021).","journal-title":"Int. J. Engl. Linguist."},{"key":"48053_CR59","doi-asserted-by":"publisher","DOI":"10.1016\/j.rmal.2024.100133","volume":"3","author":"T Yamashita","year":"2024","unstructured":"Yamashita, T. An application of many-facet Rasch measurement to evaluate automated essay scoring: A case of ChatGPT-4.0. Res. Methods Appl. Linguist. 3, 100133. https:\/\/doi.org\/10.1016\/j.rmal.2024.100133 (2024).","journal-title":"Res. Methods Appl. Linguist."},{"key":"48053_CR60","first-page":"464","volume":"9","author":"X Yuan","year":"2023","unstructured":"Yuan, X. The impact of automated evaluation feedback on students\u2019 writing revision. Int. J. Lang. Literature Linguistics. 9, 464 (2023).","journal-title":"Int. J. Lang. Literature Linguistics"},{"key":"48053_CR61","first-page":"41","volume":"2","author":"LD Zhang","year":"2022","unstructured":"Zhang, L. D. & Zhu, Y. Q. \u57fa\u4e8e\u6df1\u5ea6\u5b66\u4e60\u7684\u4e3b\u89c2\u9898\u81ea\u52a8\u8bc4\u5206\u6548\u5ea6\u7814\u7a76\u2014\u2014\u4ee5\u5927\u5b66\u6821\u672c\u82f1\u8bed\u6c34\u5e73\u8003\u8bd5\u6c49\u8bd1\u82f1\u8bd5\u9898\u4e3a\u4f8b [A study on the validity of automated scoring for subjective questions based on deep learning: A case study of Chinese-to-English translation questions in a university-based English proficiency test]. Foreign Lang. World 2, 41\u201348 (2022).","journal-title":"Foreign Lang. World"},{"key":"48053_CR62","doi-asserted-by":"publisher","first-page":"7031","DOI":"10.1007\/s10639-022-11473-y","volume":"28","author":"R Zhao","year":"2023","unstructured":"Zhao, R., Zhuang, Y., Zou, D., Xie, Q. & Yu, P. L. AI-assisted automated scoring of picture-cued writing tasks for language assessment. Educ. Inf. Technol. 28, 7031\u20137063. https:\/\/doi.org\/10.1007\/s10639-022-11473-y (2023).","journal-title":"Educ. Inf. Technol."},{"key":"48053_CR63","doi-asserted-by":"publisher","first-page":"105","DOI":"10.15858\/engtea.78.3.202309.105","volume":"78","author":"J Yun","year":"2023","unstructured":"Yun, J. Meta-analysis of inter-rater agreement and discrepancy between human and automated English essay scoring. Engl. Teach. 78, 105\u2013124. https:\/\/doi.org\/10.15858\/engtea.78.3.202309.105 (2023).","journal-title":"Engl. Teach."},{"key":"48053_CR64","first-page":"62","volume":"5","author":"R Zribi","year":"2021","unstructured":"Zribi, R. & Smaoui, C. Automated versus human essay scoring: A comparative study. Int. J. Inform. Technol. Lang. Stud. 5, 62\u201371 (2021).","journal-title":"Int. J. Inform. Technol. Lang. Stud."},{"key":"48053_CR65","doi-asserted-by":"publisher","first-page":"878","DOI":"10.1111\/lang.12079","volume":"64","author":"L Plonsky","year":"2014","unstructured":"Plonsky, L. & Oswald, F. L. How big is big? Interpreting effect sizes in L2 research. Lang. Learn. 64, 878\u2013912. https:\/\/doi.org\/10.1111\/lang.12079 (2014).","journal-title":"Lang. Learn."},{"key":"48053_CR66","unstructured":"Gwet, K. L. Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (Advanced Analytics, 2014)."},{"key":"48053_CR67","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1037\/1082-989X.7.1.105","volume":"7","author":"SB Morris","year":"2002","unstructured":"Morris, S. B. & DeShon, R. P. Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychol. Methods 7, 105 (2002).","journal-title":"Psychol. Methods"},{"key":"48053_CR68","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v036.i03","volume":"36","author":"W Viechtbauer","year":"2010","unstructured":"Viechtbauer, W. Conducting meta-analyses in R with the metafor package. J. Stat. Softw. 36, 1\u201348. https:\/\/doi.org\/10.18637\/jss.v036.i03 (2010).","journal-title":"J. Stat. Softw."},{"key":"48053_CR69","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1177\/0265532216686999","volume":"35","author":"SA Wind","year":"2018","unstructured":"Wind, S. A. & Peterson, M. E. A systematic review of methods for evaluating rating quality in language assessment. Lang. Test. 35, 161\u2013192. https:\/\/doi.org\/10.1177\/0265532216686999 (2018).","journal-title":"Lang. Test."},{"key":"48053_CR70","doi-asserted-by":"publisher","first-page":"629","DOI":"10.1136\/bmj.315.7109.629","volume":"315","author":"M Egger","year":"1997","unstructured":"Egger, M., Smith, G. D., Schneider, M. & Minder, C. Bias in meta-analysis detected by a simple, graphical test. BMJ 315, 629\u2013634. https:\/\/doi.org\/10.1136\/bmj.315.7109.629 (1997).","journal-title":"BMJ"},{"key":"48053_CR71","doi-asserted-by":"publisher","first-page":"290","DOI":"10.1017\/S0958344022000179","volume":"35","author":"S Koltovskaia","year":"2023","unstructured":"Koltovskaia, S. Postsecondary L2 writing teachers\u2019 use and perceptions of Grammarly as a complement to their feedback. ReCALL 35, 290\u2013304. https:\/\/doi.org\/10.1017\/S0958344022000179 (2023).","journal-title":"ReCALL"},{"key":"48053_CR72","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2025.105244","volume":"228","author":"Y Li","year":"2025","unstructured":"Li, Y. et al. Can AI support human grading? Examining machine attention and confidence in short answer scoring.. Comput. Educ. 105244. https:\/\/doi.org\/10.1016\/j.compedu.2025.105244 (2025).","journal-title":"Comput. Educ."},{"key":"48053_CR73","doi-asserted-by":"publisher","DOI":"10.1080\/15434303.2025.2507012","author":"J Liu","year":"2025","unstructured":"Liu, J. & Shi, Z. Dimensionality of written accuracy: Exploring the factor structure of linguistic accuracy measures.. Lang. Assess. Q. https:\/\/doi.org\/10.1080\/15434303.2025.2507012 (2025).","journal-title":"Lang. Assess. Q."}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-48053-w","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-48053-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-48053-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T22:01:23Z","timestamp":1781647283000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-48053-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,14]]},"references-count":73,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["48053"],"URL":"https:\/\/doi.org\/10.1038\/s41598-026-48053-w","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,14]]},"assertion":[{"value":"15 January 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 April 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 April 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"17412"}}