{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T06:07:30Z","timestamp":1789106850303,"version":"build-2803163510"},"reference-count":63,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T00:00:00Z","timestamp":1789084800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100023673","name":"Deanship of Scientific Research, Imam Mohammed Ibn Saud Islamic University","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100023673","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Res. Metr. Anal."],"abstract":"<jats:sec>\n                    <jats:title>Introduction<\/jats:title>\n                    <jats:p>Large language models (LLMs) are increasingly used for scoring and feedback in second-language (L2) assessment, yet the validity of the resulting interpretations remains contested. This review evaluated when LLM-mediated assessment is psychometrically and educationally defensible using an argument-based validity framework focused on construct representation, reliability and reproducibility, fairness, and washback.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Following PRISMA 2020, we systematically searched nine databases and repositories for empirical studies published from January 2022 to December 2025. Fifty-two studies met the inclusion criteria. Quantitative synthesis used REML random-effects models where effects were sufficiently comparable; Pearson correlations, rank correlations, ICC\/\u03ba\/QWK, and other psychometric indices were otherwise retained on metric-appropriate scales. Washback outcomes were synthesized using Hedges' g.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      Source-level auditing showed that directly comparable Pearson evidence was sparse. Three source-verified L2 studies yielded moderate human-LLM score correspondence (\n                      <jats:italic>r<\/jats:italic>\n                      = 0.66, 95% CI [0.53, 0.75]) with substantial heterogeneity (\n                      <jats:italic>I<\/jats:italic>\n                      <jats:sup>2<\/jats:sup>\n                      = 70.8%). Evidence was strongest in rubric-guided, structurally constrained, and human-supervised contexts, whereas construct representation, fairness, operational reproducibility, and generalizability remained underdeveloped. Across 14 studies, LLM-mediated formative feedback produced a small-to-moderate positive effect on revision quality and short-term writing outcomes (\n                      <jats:italic>g<\/jats:italic>\n                      = 0.42, 95% CI [0.27, 0.57]), although heterogeneity was substantial (\n                      <jats:italic>I<\/jats:italic>\n                      <jats:sup>2<\/jats:sup>\n                      = 69.8%) and concerns about cognitive offloading, learner agency, and reproducibility persisted.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>The findings support a context-dependent interpretation of validity rather than a universal claim about LLM assessment capability. Current evidence supports carefully bounded formative and human-supervised applications but does not justify autonomous high-stakes scoring or broad claims of construct validity, fairness, and generalizability.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/frma.2026.1908592","type":"journal-article","created":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T05:30:18Z","timestamp":1789104618000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Conditional validity in LLM-mediated L2 assessment: an argument-based systematic review and meta-analysis"],"prefix":"10.3389","volume":"11","author":[{"given":"Latifah Hamdan","family":"Alghamdi","sequence":"first","affiliation":[{"name":"ELC, King Khalid University","place":["Abha, Saudi Arabia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Talal Musaed","family":"Alghizzi","sequence":"additional","affiliation":[{"name":"College of Languages and translation, Imam Mohammad ibn Saud Islamic University (IMSIU)","place":["Riyadh, Saudi Arabia"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,9,11]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"205","DOI":"10.21833\/ijaas.2025.02.022","article-title":"Assessing the efficacy of ChatGPT's automated corrective feedback in enhancing students' writing proficiency","volume":"12","author":"Alanazi","year":"2025","journal-title":"Int. J. Adv. Appl. Sci."},{"key":"B2","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1093\/applin\/14.2.115","article-title":"Does washback exist?","volume":"14","author":"Alderson","year":"1993","journal-title":"Appl. Ling."},{"key":"B3","volume-title":"Standards for Educational and Psychological Testing","year":"2014"},{"key":"B4","doi-asserted-by":"publisher","first-page":"101766","DOI":"10.1016\/j.tsc.2025.101766","article-title":"The impact of integrating ChatGPT with teachers' feedback on EFL writing skills","volume":"56","author":"Asadi","year":"2025","journal-title":"Thinking Skills Creativity"},{"key":"B5","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","article-title":"Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI","volume":"58","author":"Barredo Arrieta","year":"2020","journal-title":"Inf. Fusion"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2410.11005","article-title":"One language, many gaps: evaluating dialect fairness in large language models","author":"Blodgett","year":"2024","journal-title":"arXiv"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2108.07258","article-title":"On the opportunities and risks of foundation models","author":"Bommasani","year":"2021","journal-title":"arXiv"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.1145\/3641289","article-title":"A survey on evaluation of large language models","author":"Chang","year":"2024","journal-title":"arXiv"},{"key":"B9","doi-asserted-by":"publisher","first-page":"26777","DOI":"10.1007\/s10639-025-13775-3","article-title":"ChatGPT as an automated writing evaluation tool: effects on EFL learners' writing development","volume":"30","author":"Ding","year":"2025","journal-title":"Educ. Inf. Technol"},{"key":"B10","doi-asserted-by":"publisher","first-page":"5683","DOI":"10.3390\/app15105683","article-title":"Large language model-powered automated assessment: a systematic review","volume":"15","author":"Emirtekin","year":"2025","journal-title":"Appl. Sci."},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2508.02442","article-title":"Assessing the reliability and validity of large language models for automated assessment of student essays in higher education","author":"Gaggioli","year":"2025","journal-title":"arXiv"},{"key":"B12","doi-asserted-by":"publisher","first-page":"1097","DOI":"10.1162\/coli_a_00524","article-title":"Bias and fairness in large language models: a survey","volume":"50","author":"Gallegos","year":"2024","journal-title":"Comput. Ling"},{"key":"B13","doi-asserted-by":"publisher","first-page":"S251","DOI":"10.1002\/tesq.70011","article-title":"Exploring the potential of ChatGPT for evaluating English essays in a criterion-based assessment","volume":"59","author":"Gjorevski","year":"2025","journal-title":"TESOL Q."},{"key":"B14","doi-asserted-by":"publisher","first-page":"8435","DOI":"10.1007\/s10639-023-12146-0","article-title":"To resist it or to embrace it? Examining ChatGPT's potential to support teacher feedback in EFL writing","volume":"29","author":"Guo","year":"2024","journal-title":"Educ. Inf. Technol"},{"key":"B15","doi-asserted-by":"publisher","first-page":"1272229","DOI":"10.3389\/feduc.2023.1272229","article-title":"Is GPT-4 a reliable rater?","volume":"8","author":"Hackl","year":"2023","journal-title":"Front. Educ."},{"key":"B16","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1016\/j.system.2024.103502","article-title":"Exploring ChatGPT-supported teacher feedback in the EFL context","volume":"126","author":"Han","year":"2024","journal-title":"System"},{"key":"B17","doi-asserted-by":"publisher","first-page":"285","DOI":"10.3233\/EFI-180221","article-title":"The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers","volume":"34","author":"Hong","year":"2018","journal-title":"Educ. Inf."},{"key":"B18","first-page":"18","article-title":"\u201cEvaluating LLM-based automated essay scoring: accuracy, fairness, and validity,\u201d","volume-title":"Proceedings of the 2025 ACL Workshop on AI in Education","author":"Huang","year":"2025"},{"key":"B19","doi-asserted-by":"publisher","first-page":"151538","DOI":"10.1109\/ACCESS.2025.3603191","article-title":"Large language model-driven dynamic assessment of grammatical accuracy in English learner writing","volume":"13","author":"Jaganov","year":"2025","journal-title":"arXiv"},{"key":"B20","doi-asserted-by":"publisher","first-page":"103737","DOI":"10.1016\/j.system.2025.103737","article-title":"When AI meets source use: exploring ChatGPT's potential in L2 summary writing assessment","volume":"133","author":"Jin","year":"2025","journal-title":"System"},{"key":"B21","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1186\/s40468-025-00389-2","article-title":"From red ink to algorithms: investigating the use of large language models in academic writing feedback","volume":"15","author":"Jovic","year":"2025","journal-title":"Lang. Test. Asia"},{"key":"B22","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1111\/jedm.12000","article-title":"Validating the interpretations and uses of test scores","volume":"50","author":"Kane","year":"2013","journal-title":"J. Educ. Meas."},{"key":"B23","doi-asserted-by":"publisher","first-page":"101254","DOI":"10.1016\/j.jslw.2025.101254","article-title":"From algorithms to annotations: rethinking feedback practices in academic writing through AI-human comparison","volume":"70","author":"Kashiha","year":"2025","journal-title":"J. Second Lang. Writ."},{"key":"B24","doi-asserted-by":"publisher","first-page":"S318","DOI":"10.1002\/tesq.3405","article-title":"Automated essay scoring with GPT-4 for a local placement test: investigating prompting strategies, intra-rater reliability, and alignment with human scores","volume":"59","author":"Kim","year":"2025","journal-title":"TESOL Q."},{"key":"B25","doi-asserted-by":"publisher","first-page":"22","DOI":"10.32038\/ltrq.2024.43.02","article-title":"The intersection of AI and language assessment: a study on the reliability of ChatGPT in grading IELTS writing task 2","volume":"43","author":"Koraishi","year":"2024","journal-title":"Lang. Teach. Res. Q."},{"key":"B26","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1007\/s44163-024-00147-y","article-title":"Performance of the pre-trained large language model GPT-4 on automated short answer grading","volume":"4","author":"Kortemeyer","year":"2024","journal-title":"Discover Artif. Intell."},{"key":"B27","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1186\/s13012-017-0688-3","article-title":"Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series","volume":"13","author":"Lewin","year":"2018","journal-title":"Implementation Sci."},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2512.14561","article-title":"Agreement between large language models and human raters in essay scoring: a research synthesis","author":"Li","year":"2025","journal-title":"arXiv"},{"key":"B29","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1177\/21582440251369938","article-title":"ChatGPT4o as an AI peer assessor in EFL speaking classrooms: examining scoring reliability and feedback effectiveness","volume":"15","author":"Li","year":"2025","journal-title":"Sage Open"},{"key":"B30","doi-asserted-by":"publisher","first-page":"204","DOI":"10.1145\/3709026.3709030","article-title":"\u201cOn automated essay grading using large language models,\u201d","author":"Liew","year":"2025","journal-title":"Proceedings of the 2024 8th International Conference on Computer Science and Artificial Intelligence (CSAI '24)"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.1080\/09588221.2025.2518430","article-title":"Enhancing GPT-based automated essay scoring: the impact of fine-tuning and linguistic complexity measures","author":"Liu","year":"2025","journal-title":"Comput. Assisted Lang. Learn"},{"key":"B32","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/09588221.2024.2389923","article-title":"Integrating large language models into EFL writing instruction: effects on performance, self-regulated learning strategies, and motivation","volume":"39","author":"Liu","year":"2024","journal-title":"Comput. Assisted Lang. Learn."},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2025-1793","article-title":"Assessment of L2 oral proficiency using speech large language models","author":"Ma","year":"2025","journal-title":"Int. Speech Commun. Assoc"},{"key":"B34","doi-asserted-by":"publisher","DOI":"10.1075\/lllt.31","author":"Manch\u00f3n","year":"2011","journal-title":"Learning-to-Write and Writing-to-Learn in an Additional Language"},{"key":"B35","first-page":"13","article-title":"\u201cValidity,\u201d","volume-title":"Educational Measurement, 3rd edn","author":"Messick","year":"1989"},{"key":"B36","doi-asserted-by":"publisher","first-page":"100199","DOI":"10.1016\/j.caeai.2023.100199","article-title":"Using LLMs to bring evidence-based feedback into the classroom: AI-generated feedback increases secondary students' text revision, motivation, and positive emotions","volume":"6","author":"Meyer","year":"2023","journal-title":"Comput. Educ. Artif. Intell."},{"key":"B37","doi-asserted-by":"publisher","first-page":"100116","DOI":"10.1016\/j.rmal.2024.100116","article-title":"Testing the viability of ChatGPT as a companion in L2 writing accuracy assessment","volume":"3","author":"Mizumoto","year":"2024","journal-title":"Res. Methods Appl. Ling."},{"key":"B38","doi-asserted-by":"publisher","first-page":"e52935","DOI":"10.2196\/52935","article-title":"Evaluation of large language model performance and reliability for citations and references in scholarly writing: cross-disciplinary study","volume":"26","author":"Mugaanyi","year":"2024","journal-title":"J. Med. Internet Res."},{"key":"B39","doi-asserted-by":"publisher","first-page":"100234","DOI":"10.1016\/j.caeai.2024.100234","article-title":"Large language models and automated essay scoring of English language learner writing: insights into validity and reliability","volume":"6","author":"Pack","year":"2024","journal-title":"Comput. Educ. Artif. Intell."},{"key":"B40","doi-asserted-by":"publisher","first-page":"n71","DOI":"10.1136\/bmj.n71","article-title":"The PRISMA 2020 statement: an updated guideline for reporting systematic reviews","volume":"372","author":"Page","year":"2021","journal-title":"BMJ"},{"key":"B41","doi-asserted-by":"publisher","first-page":"2410101","DOI":"10.1080\/2331186X.2024.2410101","article-title":"The impact of ChatGPT feedback on the development of EFL students' writing skills","volume":"11","author":"Polakova","year":"2024","journal-title":"Cogent Educ."},{"key":"B42","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1145\/3351095.3372873","article-title":"\u201cClosing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing,\u201d","author":"Raji","year":"2020","journal-title":"Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency"},{"key":"B43","doi-asserted-by":"publisher","first-page":"100428","DOI":"10.1016\/j.caeai.2025.100428","article-title":"Is GPT-4 fair? An empirical analysis in automatic short answer grading","volume":"8","author":"Rodrigues","year":"2025","journal-title":"Comput. Educ. Artif. Intell"},{"key":"B44","first-page":"210","article-title":"\u201cFairness in automated essay scoring: a comparative analysis of algorithms on German learner essays from secondary education,\u201d","volume-title":"Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024)","author":"Schaller","year":"2024"},{"key":"B45","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1145\/3287560.3287598","article-title":"\u201cFairness and abstraction in sociotechnical systems,\u201d","author":"Selbst","year":"2019"},{"key":"B46","doi-asserted-by":"publisher","first-page":"540","DOI":"10.3390\/bs15040540","article-title":"Complementing but not replacing: comparing the impacts of GPT-4 and native-speaker interaction on Chinese L2 writing outcomes","volume":"15","author":"Shan","year":"2025","journal-title":"Behav. Sci."},{"key":"B47","doi-asserted-by":"publisher","DOI":"10.1016\/j.asw.2025.100988","article-title":"Using ChatGPT to score essays and short-form constructed responses","author":"Shermis","year":"2025","journal-title":"arXiv"},{"key":"B48","doi-asserted-by":"publisher","first-page":"1148","DOI":"10.1080\/09588221.2025.2454541","article-title":"Comparing the effects of ChatGPT and automated writing evaluation on students' writing and ideal L2 writing self","volume":"39","author":"Shi","year":"2026","journal-title":"Comput. Assisted Lang. Learn"},{"key":"B49","doi-asserted-by":"publisher","first-page":"24735","DOI":"10.1007\/s10639-024-12817-6","article-title":"Exploratory study on the potential of ChatGPT as a rater of second language writing","volume":"29","author":"Shin","year":"2024","journal-title":"Educ. Inf. Technol."},{"key":"B50","doi-asserted-by":"publisher","DOI":"10.1080\/15305058.2026.2684584","article-title":"Exploring LLM autoscoring reliability in large-scale writing assessments using generalizability theory","author":"Song","year":"2025","journal-title":"arXiv"},{"key":"B51","volume-title":"Evaluating LLM performance in essay assessment: a comparative analysis of AI grading and feedback systems for university English courses","author":"Stasuik","year":"2025"},{"key":"B52","first-page":"95","article-title":"\u201cLanguaging, agency and collaboration in advanced second language proficiency,\u201d","volume-title":"Advanced Language Learning: The Contribution of Halliday and Vygotsky","author":"Swain","year":"2006"},{"key":"B53","doi-asserted-by":"publisher","first-page":"e34262","DOI":"10.1016\/j.heliyon.2024.e34262","article-title":"Harnessing LLMs for multi-dimensional writing assessment: reliability and alignment with human judgments","volume":"10","author":"Tang","year":"2024","journal-title":"Heliyon"},{"key":"B54","doi-asserted-by":"publisher","first-page":"100170","DOI":"10.1016\/j.caeai.2024.100270","article-title":"\u201cChatGPT is the companion, not enemies\u201d: EFL learners' perceptions and experiences of using ChatGPT in writing","volume":"5","author":"Teng","year":"2024","journal-title":"Comput. Educ. Artif. Intell."},{"key":"B55","doi-asserted-by":"publisher","first-page":"8450","DOI":"10.3758\/s13428-024-02485-2","article-title":"Linking essay-writing tests using many-facet models and neural automated essay scoring","volume":"56","author":"Uto","year":"2024","journal-title":"Behav. Res. Methods"},{"key":"B56","doi-asserted-by":"publisher","first-page":"20","DOI":"10.21449\/ijate.1517994","article-title":"Artificial intelligence as an automated essay scoring tool: a focus on ChatGPT","volume":"12","author":"Uyar","year":"2025","journal-title":"Int. J. Assess. Tools Educ."},{"key":"B57","first-page":"5998","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"B58","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v036.i03","article-title":"Conducting meta-analyses in R with the metafor package","volume":"36","author":"Viechtbauer","year":"2010","journal-title":"J. Stat. Software"},{"key":"B59","doi-asserted-by":"publisher","first-page":"100481","DOI":"10.1016\/j.caeai.2025.100481","article-title":"Evaluating large language models as raters in large-scale writing assessments: a psychometric framework for reliability and validity","volume":"9","author":"Wang","year":"2025","journal-title":"Comput. Educ. Artif. Intell."},{"key":"B60","doi-asserted-by":"publisher","first-page":"293","DOI":"10.1145\/3706468.3706507","article-title":"\u201cHuman-AI collaborative essay scoring: a dual-process framework with LLMs,\u201d","author":"Xiao","year":"2025","journal-title":"Proceedings of the 15th International Learning Analytics and Knowledge Conference (LAK '25)"},{"key":"B61","doi-asserted-by":"publisher","first-page":"344","DOI":"10.1177\/02655322251329435","article-title":"Exploring potential biases in GPT-4o's ratings of English language learners' essays","volume":"42","author":"Yamashita","year":"2025","journal-title":"Lang. Testing"},{"key":"B62","doi-asserted-by":"publisher","first-page":"576","DOI":"10.18653\/v1\/2023.bea-1.49","article-title":"\u201cRating short L2 essays on the CEFR scale with GPT-4,\u201d","author":"Yancey","year":"2023","journal-title":"Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)"},{"key":"B63","doi-asserted-by":"publisher","first-page":"49","DOI":"10.34293\/education.v12i4.7855","article-title":"The reliability of using ChatGPT in rating EFL writings","volume":"12","author":"Yang","year":"2024","journal-title":"Shanlax Int. J. Edu."}],"container-title":["Frontiers in Research Metrics and Analytics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frma.2026.1908592\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T05:30:21Z","timestamp":1789104621000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frma.2026.1908592\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,11]]},"references-count":63,"alternative-id":["10.3389\/frma.2026.1908592"],"URL":"https:\/\/doi.org\/10.3389\/frma.2026.1908592","relation":{},"ISSN":["2504-0537"],"issn-type":[{"value":"2504-0537","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,11]]},"article-number":"1908592"}}