{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T17:39:12Z","timestamp":1775842752197,"version":"3.50.1"},"reference-count":68,"publisher":"Elsevier BV","issue":"3","license":[{"start":{"date-parts":[[2024,12,16]],"date-time":"2024-12-16T00:00:00Z","timestamp":1734307200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,12,16]],"date-time":"2024-12-16T00:00:00Z","timestamp":1734307200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J Artif Intell Educ"],"published-print":{"date-parts":[[2025,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Automated writing evaluation (AWE) is an artificial intelligence (AI)-empowered educational technology designed to assist writing instruction and improve students\u2019 writing proficiency. The present study adopted a quasi-experimental design using the inverse probability of treatment weighting method to explore the long-term effects of an AWE system known as Utah Compose on students\u2019 state test English Language Arts (ELA) performance. The participants included 134,425 students in Grades 4\u201310 in Utah from school year 2015 to 2018. Findings showed AWE\u2019s cumulative benefit to students\u2019 ELA performance, but those cumulative effects decreased each year and peaked after three years of implementation. This study is the largest evaluation of AWE effects to date in terms of both its sample size and the duration of investigation. The findings regarding AWE\u2019s cumulative effects on students\u2019 state test ELA performance, which is a distal outcome at the state level, have significant implications for policy and practice regarding large-scale AWE implementation.<\/jats:p>","DOI":"10.1007\/s40593-024-00443-9","type":"journal-article","created":{"date-parts":[[2024,12,16]],"date-time":"2024-12-16T16:48:14Z","timestamp":1734367694000},"page":"1528-1559","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Exploring the Long-Term Effects of the Statewide Implementation of an Automated Writing Evaluation System on Students\u2019 State Test ELA Performance"],"prefix":"10.1016","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2175-9852","authenticated-orcid":false,"given":"Yue","family":"Huang","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7192-3510","authenticated-orcid":false,"given":"Joshua","family":"Wilson","sequence":"additional","affiliation":[]},{"given":"Henry","family":"May","sequence":"additional","affiliation":[]}],"member":"78","published-online":{"date-parts":[[2024,12,16]]},"reference":[{"key":"443_CR1","unstructured":"American Institutes for Research. (2017). Utah state assessments 2016\u20132017 technical report \u2013 Volume 1: Annual technical report. Utah State Board of Education. Archived."},{"key":"443_CR2","unstructured":"American Institutes for Research. (2018). Utah state assessments 2017\u20132018 technical report \u2013 Volume 1: Annual technical report. Utah State Board of Education. Archived at https:\/\/schools.utah.gov\/file\/97391cfd-9251-4ad1-9266-47b2ebe88e84"},{"issue":"5","key":"443_CR3","doi-asserted-by":"publisher","first-page":"18","DOI":"10.58680\/ej20097117","volume":"98","author":"AN Applebee","year":"2009","unstructured":"Applebee, A. N., & Langer, J. A. (2009). What is happening in the teaching of writing? English Journal, 98(5), 18\u201328.","journal-title":"English Journal"},{"key":"443_CR4","doi-asserted-by":"publisher","first-page":"399","DOI":"10.1080\/00273171.2011.568786","volume":"46","author":"PC Austin","year":"2011","unstructured":"Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46, 399\u2013424. https:\/\/doi.org\/10.1080\/00273171.2011.568786","journal-title":"Multivariate Behavioral Research"},{"key":"443_CR5","doi-asserted-by":"publisher","first-page":"337","DOI":"10.1007\/s40593-020-00201-7","volume":"30","author":"R Balyan","year":"2020","unstructured":"Balyan, R., McCarthy, K. S., & McNamara, D. S. (2020). Applying natural language processing and hierarchical machine learning approaches to text difficulty classification. International Journal of Artificial Intelligence in Education, 30, 337\u2013370. https:\/\/doi.org\/10.1007\/s40593-020-00201-7","journal-title":"International Journal of Artificial Intelligence in Education"},{"key":"443_CR6","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1016\/j.asw.2016.05.003","volume":"29","author":"N Behizadeh","year":"2016","unstructured":"Behizadeh, N., & Pang, M. E. (2016). Awaiting a new wave: The status of state writing assessment in the United States. Assessing Writing, 29, 25\u201341. https:\/\/doi.org\/10.1016\/j.asw.2016.05.003","journal-title":"Assessing Writing"},{"key":"443_CR7","unstructured":"Caccamise, D., Franzke, M., Eckhoff, A., Kintsch, E., & Kintsch, W. (2007). Guided practice in technology-based summary writing. In D. S. McNamara (Ed.), Reading comprehension strategies: Theories, interventions, and technologies (pp. 375\u2013396). Lawrence Erlbaum Associates Publishers."},{"key":"443_CR8","unstructured":"Campuzano, L., Dynarski, M., Agodini, R., & Rall, K. (2009). Effectiveness of reading and mathematics software products: Findings from two student cohorts\u2014Executive summary (NCEE 2009\u20134042). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from https:\/\/ies.ed.gov\/ncee\/pubs\/20094041\/pdf\/20094042.pdf"},{"issue":"6","key":"443_CR9","doi-asserted-by":"publisher","first-page":"1122","DOI":"10.3102\/00028312221106773","volume":"59","author":"D Chen","year":"2022","unstructured":"Chen, D., Hebert, M., & Wilson, J. (2022). Examining human and automated ratings of elementary students\u2019 writing quality: A multivariate generalizability theory application. American Educational Research Journal, 59(6), 1122\u20131156. https:\/\/doi.org\/10.3102\/00028312221106773","journal-title":"American Educational Research Journal"},{"key":"443_CR10","unstructured":"Coe, M., Hanita, M., Nishioka, V., & Smiley, R. (2011). An investigation of the impact of the 6 + 1 trait writing model on grade 5 student writing achievement (Final Report NCEE 2012\u20134010). National Center for Education Evaluation and Regional Assistance. Retrieved from https:\/\/files.eric.ed.gov\/fulltext\/ED527445.pdf"},{"key":"443_CR11","unstructured":"Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates, Publishers."},{"key":"443_CR12","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1007\/s40593-020-00227-x","volume":"31","author":"P Deane","year":"2021","unstructured":"Deane, P., Wilson, J., Zhang, M., Li, C., van Rijn, P., Guo, H., Roth, A., Winchester, E., & Richter, T. (2021). The sensitivity of a scenario-based assessment of written argumentation to school differences in curriculum and instruction. International Journal of Artificial Intelligence in Education, 31, 57\u201398. https:\/\/doi.org\/10.1007\/s40593-020-00227-x","journal-title":"International Journal of Artificial Intelligence in Education"},{"key":"443_CR13","doi-asserted-by":"publisher","unstructured":"Escalante, J., Pack, A., & Barrett, A. (2023). AI-generated feedback on writing: Insights into efficacy and ENL student preference. International Journal of Educational Technology in Higher Education, 20(57). https:\/\/doi.org\/10.1186\/s41239-023-00425-2","DOI":"10.1186\/s41239-023-00425-2"},{"key":"443_CR14","doi-asserted-by":"publisher","first-page":"1162454","DOI":"10.3389\/frai.2023.1162454","volume":"6","author":"J Fleckenstein","year":"2023","unstructured":"Fleckenstein, J., Liebenow, L. W., & Meyer, J. (2023). Automated feedback and writing: A multi-level meta-analysis of effects on students\u2019 performance. Frontiers in Artificial Intelligence, 6, 1162454. https:\/\/doi.org\/10.3389\/frai.2023.1162454","journal-title":"Frontiers in Artificial Intelligence"},{"key":"443_CR15","doi-asserted-by":"publisher","first-page":"53","DOI":"10.2190\/DH8F-QJWM-J457-FQVB","volume":"33","author":"M Franzke","year":"2005","unstructured":"Franzke, M., Kintsch, E., Caccamise, D., Johnson, N., & Dooley, S. (2005). Summary Street: Computer support for comprehension and writing. Journal of Educational Computing Research, 33, 53\u201380. https:\/\/doi.org\/10.2190\/DH8F-QJWM-J457-FQVB","journal-title":"Journal of Educational Computing Research"},{"key":"443_CR16","doi-asserted-by":"publisher","unstructured":"Fu, Q., Zou, D., Xie, H., & Cheng, G. (2022). A review of AWE feedback: Types, learning outcomes, and implications. Computer Assisted Language Learning, 1\u201343. https:\/\/doi.org\/10.1080\/09588221.2022.2033787","DOI":"10.1080\/09588221.2022.2033787"},{"key":"443_CR17","unstructured":"Goldberg, A., Russell, M., & Cook, A. (2003). The effect of computers on student writing: A meta-analysis of studies from 1992 to 2002. The Journal of Technology, Learning and Assessment, 2(1), 1. https:\/\/ejournals.bc.edu\/index.php\/jtla\/article\/view\/1661"},{"issue":"1","key":"443_CR18","doi-asserted-by":"publisher","first-page":"277","DOI":"10.3102\/0091732X18821125","volume":"43","author":"S Graham","year":"2019","unstructured":"Graham, S. (2019). Changing how writing is taught. Review of Research in Education, 43(1), 277\u2013303. https:\/\/doi.org\/10.3102\/0091732X18821125","journal-title":"Review of Research in Education"},{"key":"443_CR19","unstructured":"Graham, S., & Hebert, M. (2010). Writing to read: Evidence for how writing can improve reading \u2013 A report from Carnegie Corporation of New York [Report]. Alliance for Excellent Education. Retrieved December 2, 2024, from https:\/\/acuresearchbank.acu.edu.au\/item\/8v6x5\/writing-to-read-evidence-for-how-writing-can-improve-reading-a-report-from-carnegie-corporation-of-new-york"},{"issue":"4","key":"443_CR20","doi-asserted-by":"publisher","first-page":"710","DOI":"10.17763\/haer.81.4.t2k0m13756113566","volume":"81","author":"S Graham","year":"2011","unstructured":"Graham, S., & Hebert, M. (2011). Writing to read: A meta-analysis of the impact of writing and writing instruction on reading. Harvard Educational Review, 81(4), 710\u2013744. https:\/\/doi.org\/10.17763\/haer.81.4.t2k0m13756113566","journal-title":"Harvard Educational Review"},{"key":"443_CR21","unstructured":"Graham, S., & Perin, D. (2007). Writing next: Effective strategies to improve writing of adolescents in middle and high schools \u2013 A report to Carnegie Corporation of New York. Washington, DC: Alliance for Excellent Education. https:\/\/www.carnegie.org\/publications\/writing-next-effective-strategies-to-improve-writing-of-adolescents-in-middle-and-high-schools\/"},{"issue":"6","key":"443_CR22","doi-asserted-by":"publisher","first-page":"1015","DOI":"10.1007\/s11145-013-9495-7","volume":"27","author":"S Graham","year":"2014","unstructured":"Graham, S., Capizzi, A., Harris, K. R., Hebert, M., & Morphy, P. (2014). Teaching writing to middle school students: A national survey. Reading and Writing, 27(6), 1015\u20131042. https:\/\/doi.org\/10.1007\/s11145-013-9495-7","journal-title":"Reading and Writing"},{"issue":"4","key":"443_CR23","doi-asserted-by":"publisher","first-page":"523","DOI":"10.1086\/681947","volume":"115","author":"S Graham","year":"2015","unstructured":"Graham, S., Hebert, M., & Harris, K. R. (2015). Formative assessment and writing: A meta-analysis. The Elementary School Journal, 115(4), 523\u2013547. https:\/\/doi.org\/10.1086\/681947","journal-title":"The Elementary School Journal"},{"key":"443_CR24","doi-asserted-by":"publisher","first-page":"879","DOI":"10.1037\/a0029185","volume":"104","author":"S Graham","year":"2012","unstructured":"Graham, S., McKeown, D., Kiuhara, S., & Harris, K. R. (2012). A meta-analysis of writing instruction for students in the elementary grades. Journal of Educational Psychology, 104, 879\u2013896.","journal-title":"Journal of Educational Psychology"},{"issue":"6","key":"443_CR25","first-page":"4","volume":"8","author":"D Grimes","year":"2010","unstructured":"Grimes, D., & Warschauer, M. (2010). Utility in a fallible tool: A multi-site case study of automated writing evaluation. The Journal of Technology, Learning, and Assessment, 8(6), 4\u201343.","journal-title":"The Journal of Technology, Learning, and Assessment"},{"key":"443_CR26","unstructured":"Guo, S., & Fraser, M. W. (2014). Propensity score analysis: Statistical methods and applications (2nd ed.). SAGE Publications."},{"issue":"4","key":"443_CR27","doi-asserted-by":"publisher","first-page":"1161","DOI":"10.1111\/1468-0262.00442","volume":"71","author":"K Hirano","year":"2003","unstructured":"Hirano, K., Imbens, G. W., & Ridder, G. (2003). Efficient estimation of average treatment effects using the estimated propensity score. Econometrica, 71(4), 1161\u20131189. https:\/\/doi.org\/10.1111\/1468-0262.00442","journal-title":"Econometrica"},{"issue":"4","key":"443_CR28","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1080\/15434303.2016.1230121","volume":"13","author":"GTL Hoang","year":"2016","unstructured":"Hoang, G. T. L., & Kunnan, A. J. (2016). Automated essay evaluation for English language learners: A case study of MY Access. Language Assessment Quarterly, 13(4), 359\u2013376. https:\/\/doi.org\/10.1080\/15434303.2016.1230121","journal-title":"Language Assessment Quarterly"},{"key":"443_CR29","unstructured":"Huang, Y. (2023). Exploring the effects of the statewide implementation of an automated writing evaluation system among K\u201312 students [Doctoral dissertation, University of Delaware]. Retrieved December 2, 2024, from https:\/\/udspace.udel.edu\/handle\/19716\/33521"},{"key":"443_CR30","unstructured":"Huang, Y., & Wilson, J. (Under review). Exploring the effectiveness of large-scale automated writing evaluation implementation on state test performance using generalized boosted modeling. Manuscript submitted for publication."},{"key":"443_CR31","doi-asserted-by":"publisher","first-page":"102675","DOI":"10.1016\/j.compcom.2021.102675","volume":"62","author":"Y Huang","year":"2021","unstructured":"Huang, Y., & Wilson, J. (2021). Using automated feedback to develop writing proficiency. Computers and Composition, 62, 102675. https:\/\/doi.org\/10.1016\/j.compcom.2021.102675","journal-title":"Computers and Composition"},{"key":"443_CR32","doi-asserted-by":"publisher","unstructured":"Huang, Y., Lee, H., Lentz, A., & Shores, K. A. (2023). Misclassification of career and technical education concentrators: Analysis and policy recommendations (EdWorkingPaper No. 23-889). Annenberg Institute at Brown University. Retrieved from https:\/\/doi.org\/10.26300\/esab-m159","DOI":"10.26300\/esab-m159"},{"issue":"1","key":"443_CR33","doi-asserted-by":"publisher","first-page":"136","DOI":"10.1037\/a0013097","volume":"101","author":"SA Kiuhara","year":"2009","unstructured":"Kiuhara, S. A., Graham, S., & Hawken, L. S. (2009). Teaching writing to high school students: A national survey. Journal of Educational Psychology, 101(1), 136\u2013160. https:\/\/doi.org\/10.1037\/a0013097","journal-title":"Journal of Educational Psychology"},{"key":"443_CR34","doi-asserted-by":"publisher","first-page":"538","DOI":"10.1007\/s40593-020-00211-5","volume":"31","author":"VS Kumar","year":"2021","unstructured":"Kumar, V. S., & Boulanger, D. (2021). Automated essay scoring and the deep learning black box: How are rubric scores determined? International Journal of Artificial Intelligence in Education, 31, 538\u2013584. https:\/\/doi.org\/10.1007\/s40593-020-00211-5","journal-title":"International Journal of Artificial Intelligence in Education"},{"key":"443_CR35","doi-asserted-by":"publisher","DOI":"10.4135\/9781071802854","volume-title":"Practical propensity score methods using R","author":"W Leite","year":"2017","unstructured":"Leite, W. (2017). Practical propensity score methods using R. SAGE Publications."},{"issue":"3","key":"443_CR36","doi-asserted-by":"publisher","first-page":"773","DOI":"10.1111\/bjet.13294","volume":"54","author":"R Li","year":"2022","unstructured":"Li, R. (2022). Still a fallible tool? revisiting effects of automated writing evaluation from activity theory perspective. British Journal of Educational Technology, 54(3), 773\u2013789. https:\/\/doi.org\/10.1111\/bjet.13294","journal-title":"British Journal of Educational Technology"},{"key":"443_CR37","doi-asserted-by":"publisher","first-page":"638","DOI":"10.1007\/s40593-020-00204-4","volume":"31","author":"A Liaqat","year":"2021","unstructured":"Liaqat, A., Munteanu, C., & Epp, C. D. (2021). Collaborating with mature English language learners to combine peer and automated feedback: A user-centered approach to designing writing support. International Journal of Artificial Intelligence in Education, 31, 638\u2013679. https:\/\/doi.org\/10.1007\/s40593-020-00204-4","journal-title":"International Journal of Artificial Intelligence in Education"},{"issue":"2","key":"443_CR38","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1002\/rev3.3114","volume":"6","author":"CW Little","year":"2018","unstructured":"Little, C. W., Clark, J. C., Tani, N. E., & Connor, C. M. (2018). Improving writing skills through technology-based instruction: A meta-analysis. Review of Education, 6(2), 183\u2013201. https:\/\/doi.org\/10.1002\/rev3.3114","journal-title":"Review of Education"},{"issue":"4","key":"443_CR39","doi-asserted-by":"publisher","first-page":"403","DOI":"10.1037\/1082-989X.9.4.403","volume":"9","author":"DF McCaffrey","year":"2004","unstructured":"McCaffrey, D. F., Ridgeway, G., & Morral, A. R. (2004). Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychological Methods, 9(4), 403\u2013425. https:\/\/doi.org\/10.1037\/1082-989X.9.4.403","journal-title":"Psychological Methods"},{"issue":"1","key":"443_CR40","doi-asserted-by":"publisher","first-page":"149","DOI":"10.17239\/jowr-2016.08.01.05","volume":"8","author":"NS Moore","year":"2016","unstructured":"Moore, N. S., & MacArthur, C. A. (2016). Student use of automated essay evaluation technology during revision. Journal of Writing Research, 8(1), 149\u2013175. https:\/\/doi.org\/10.17239\/jowr-2016.08.01.05","journal-title":"Journal of Writing Research"},{"issue":"2","key":"443_CR41","doi-asserted-by":"publisher","first-page":"599","DOI":"10.1111\/jcal.12635","volume":"38","author":"A Nunes","year":"2022","unstructured":"Nunes, A., Cordeiro, C., Limpo, T., & Castro, S. L. (2022). Effectiveness of automated writing evaluation systems in school settings: A systematic review of studies from 2000 to 2020. Journal of Computer Assisted Learning, 38(2), 599\u2013620. https:\/\/doi.org\/10.1111\/jcal.12635","journal-title":"Journal of Computer Assisted Learning"},{"key":"443_CR42","doi-asserted-by":"publisher","first-page":"255","DOI":"10.1016\/j.cedpsych.2018.07.002","volume":"54","author":"C Palermo","year":"2018","unstructured":"Palermo, C., & Thomson, M. M. (2018). Teacher implementation of Self-Regulated Strategy Development with an automated writing evaluation system: Effects on the argumentative writing performance of middle school students. Contemporary Educational Psychology, 54, 255\u2013270. https:\/\/doi.org\/10.1016\/j.cedpsych.2018.07.002","journal-title":"Contemporary Educational Psychology"},{"issue":"1","key":"443_CR43","doi-asserted-by":"publisher","first-page":"63","DOI":"10.17239\/jowr-2020.12.01.04","volume":"12","author":"C Palermo","year":"2020","unstructured":"Palermo, C., & Wilson, J. (2020). Implementing automated writing evaluation in different instructional contexts: A mixed-methods study. Journal of Writing Research, 12(1), 63\u2013108. https:\/\/doi.org\/10.17239\/jowr-2020.12.01.04","journal-title":"Journal of Writing Research"},{"key":"443_CR44","doi-asserted-by":"publisher","first-page":"665","DOI":"10.1007\/s40593-017-0158-8","volume":"27","author":"RJ Passonneau","year":"2017","unstructured":"Passonneau, R. J., McNamara, D., Muresan, S., & Perin, D. (2017). Preface: Special issue on multidisciplinary approaches to AI and education for reading and writing. International Journal of Artificial Intelligence in Education, 27, 665\u2013670. https:\/\/doi.org\/10.1007\/s40593-017-0158-8","journal-title":"International Journal of Artificial Intelligence in Education"},{"issue":"3","key":"443_CR45","doi-asserted-by":"publisher","first-page":"1557","DOI":"10.1007\/s11423-021-10004-9","volume":"69","author":"A Potter","year":"2021","unstructured":"Potter, A., & Wilson, J. (2021). Statewide implementation of automated writing evaluation: Analyzing usage and associations with state test performance in Grades 4\u201311. Educational Technology Research and Development, 69(3), 1557\u20131578. https:\/\/doi.org\/10.1007\/s11423-021-10004-9","journal-title":"Educational Technology Research and Development"},{"key":"443_CR46","doi-asserted-by":"publisher","unstructured":"Rad, H. S., Alipour, R., & Jafarpour, A. (2023). Using artificial intelligence to foster students\u2019 writing feedback literacy, engagement, and outcome: A case of Wordtune application. Interactive Learning Environments. https:\/\/doi.org\/10.1080\/10494820.2023.2208170","DOI":"10.1080\/10494820.2023.2208170"},{"key":"443_CR47","doi-asserted-by":"publisher","first-page":"95","DOI":"10.1007\/978-1-4612-1284-3_2","volume-title":"Statistical models in epidemiology, the environment, and clinical trials","author":"J Robins","year":"2000","unstructured":"Robins, J. (2000). Marginal structural models versus structural nested models as tools for causal inference. In M. E. Halloran & D. Berry (Eds.), Statistical models in epidemiology, the environment, and clinical trials (pp. 95\u2013134). Springer."},{"issue":"1","key":"443_CR48","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1093\/biomet\/70.1.41","volume":"70","author":"PR Rosenbaum","year":"1983","unstructured":"Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41\u201355.","journal-title":"Biometrika"},{"issue":"447","key":"443_CR49","doi-asserted-by":"publisher","first-page":"702","DOI":"10.2307\/2669980","volume":"97","author":"BD Rubin","year":"1999","unstructured":"Rubin, B. D., & Frangakis, C. E. (1999). Estimation of the causal effect of a time-varying exposure on the marginal mean of a repeated binary outcome: Comment. Journal of the American Statistical Sssosication, 97(447), 702\u2013704. https:\/\/doi.org\/10.2307\/2669980","journal-title":"Journal of the American Statistical Sssosication"},{"issue":"3","key":"443_CR50","doi-asserted-by":"publisher","first-page":"432","DOI":"10.1080\/10511250802476269","volume":"19","author":"J Schroeder","year":"2008","unstructured":"Schroeder, J., Grohe, B., & Pogue, R. (2008). The impact of criterion writing evaluation technology on criminal justice student writing skills. Journal of Criminal Justice Education, 19(3), 432\u2013445. https:\/\/doi.org\/10.1080\/10511250802476269","journal-title":"Journal of Criminal Justice Education"},{"key":"443_CR51","unstructured":"Shermis, M. D., Burstein, J. C., & Bliss, L. (2004, April). The impact of automated essay scoring on high stakes writing assessments. Paper presented at the annual meeting of the National Council on Measurement in Education, April 2004, San Diego, CA."},{"key":"443_CR52","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1016\/j.asw.2013.04.001","volume":"20","author":"MD Shermis","year":"2014","unstructured":"Shermis, M. D. (2014). State-of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstration. Assessing Writing, 20, 53\u201376. https:\/\/doi.org\/10.1016\/j.asw.2013.04.001","journal-title":"Assessing Writing"},{"key":"443_CR53","first-page":"395","volume-title":"Handbook of writing research","author":"MD Shermis","year":"2016","unstructured":"Shermis, M. D., Burstein, J. C., Elliot, N., Miel, S., & Foltz, P. W. (2016). Automated writing evaluation: An expanding body of knowledge. In C. A. McArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of writing research (pp. 395\u2013409). New York, NY: Guilford."},{"issue":"3","key":"443_CR54","doi-asserted-by":"publisher","first-page":"250","DOI":"10.1037\/a0018719","volume":"15","author":"PM Steiner","year":"2010","unstructured":"Steiner, P. M., Cook, T. D., Shadish, W. R., & Clark, M. H. (2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods, 15(3), 250\u2013267. https:\/\/doi.org\/10.1037\/a0018719","journal-title":"Psychological Methods"},{"key":"443_CR55","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1016\/j.asw.2013.11.007","volume":"19","author":"M Stevenson","year":"2014","unstructured":"Stevenson, M., & Phakiti, A. (2014). The effects of computer-generated feedback on the quality of writing. Assessing Writing, 19, 51\u201365. https:\/\/doi.org\/10.1016\/j.asw.2013.11.007","journal-title":"Assessing Writing"},{"key":"443_CR56","unstructured":"Troia, G. (2014). Evidence-based practices for writing instruction (Document No. IC-5). Retrieved December 2, 2024, from University of Florida, Collaboration for Effective Educator, Development, Accountability, and Reform Center website: http:\/\/ceedar.education.ufl.edu\/tools\/innovation-configuration\/"},{"key":"443_CR57","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1007\/s40593-019-00174-2","volume":"29","author":"TD Ullmann","year":"2019","unstructured":"Ullmann, T. D. (2019). Automated analysis of reflection in writing: Validating machine learning approaches. International Journal of Artificial Intelligence in Education, 29, 217\u2013257. https:\/\/doi.org\/10.1007\/s40593-019-00174-2","journal-title":"International Journal of Artificial Intelligence in Education"},{"key":"443_CR58","unstructured":"U.S. Department of Education, Office of Educational Technology. (2023). Artificial intelligence and the future of teaching and learning: Insights and recommendations. Retrieved from https:\/\/www.ed.gov\/sites\/ed\/files\/documents\/ai-report\/ai-report.pdf"},{"issue":"1","key":"443_CR59","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1080\/15544800701771580","volume":"3","author":"M Warschauer","year":"2008","unstructured":"Warschauer, M., & Grimes, D. (2008). Automated writing assessment in the classroom. Pedagogies: An International Journal, 3(1), 22\u201336. https:\/\/doi.org\/10.1080\/15544800701771580","journal-title":"Pedagogies: An International Journal"},{"issue":"2","key":"443_CR60","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1191\/1362168806lr190oa","volume":"10","author":"M Warschauer","year":"2006","unstructured":"Warschauer, M., & Ware, P. (2006). Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research, 10(2), 157\u2013180. https:\/\/doi.org\/10.1191\/1362168806lr190oa","journal-title":"Language Teaching Research"},{"key":"443_CR61","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1111\/j.1745-3992.2011.00223.x","volume":"31","author":"DM Williamson","year":"2012","unstructured":"Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31, 2\u201313. https:\/\/doi.org\/10.1111\/j.1745-3992.2011.00223.x","journal-title":"Educational Measurement: Issues and Practice"},{"key":"443_CR62","doi-asserted-by":"publisher","unstructured":"Wilson, J., & Andrada, G. N. (2016). Using automated feedback to improve writing quality: Opportunities and challenges. In Y. Rosen, S. Ferrara, & M. Mosharraf (Eds.), Handbook of research on technology tools for real-world skill development (pp. 678\u2013703). Hershey, PA: IGI Global. https:\/\/doi.org\/10.4018\/978-1-4666-9441-5.ch026","DOI":"10.4018\/978-1-4666-9441-5.ch026"},{"key":"443_CR63","doi-asserted-by":"publisher","first-page":"104208","DOI":"10.1016\/j.compedu.2021.104208","volume":"168","author":"J Wilson","year":"2021","unstructured":"Wilson, J., Ahrendt, C., Fudge, E. A., Raiche, A., Beard, G., & MacArthur, C. (2021a). Elementary teachers\u2019 perceptions of automated feedback and automated scoring: Transforming the teaching and learning of writing using automated writing evaluation. Computers & Education, 168, 104208. https:\/\/doi.org\/10.1016\/j.compedu.2021.104208","journal-title":"Computers & Education"},{"key":"443_CR64","doi-asserted-by":"publisher","first-page":"619","DOI":"10.1037\/edu0000311","volume":"111","author":"J Wilson","year":"2019","unstructured":"Wilson, J., Chen, D., Sandbank, M. P., & Hebert, M. (2019). Generalizability of automated scores of writing quality in grades 3\u20135. Journal of Educational Psychology, 111, 619\u2013640. https:\/\/doi.org\/10.1037\/edu0000311","journal-title":"Journal of Educational Psychology"},{"key":"443_CR65","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1016\/j.compedu.2016.05.004","volume":"100","author":"J Wilson","year":"2016","unstructured":"Wilson, J., & Czik, A. (2016). Automated essay evaluation software in English language arts classrooms: Effects on teacher feedback, student motivation, and writing quality. Computers & Education, 100, 94\u2013109. https:\/\/doi.org\/10.1016\/j.compedu.2016.05.004","journal-title":"Computers & Education"},{"issue":"2","key":"443_CR66","doi-asserted-by":"publisher","first-page":"234","DOI":"10.1007\/s40593-020-00236-w","volume":"31","author":"J Wilson","year":"2021","unstructured":"Wilson, J., Huang, Y., Palermo, C., Beard, G., & MacArthur, C. A. (2021b). Automated feedback and automated scoring in the elementary grades: Usage, attitudes, and associations with writing outcomes in a districtwide implementation of MI Write. International Journal of Artificial Intelligence in Education, 31(2), 234\u2013276. https:\/\/doi.org\/10.1007\/s40593-020-00236-w","journal-title":"International Journal of Artificial Intelligence in Education"},{"issue":"1","key":"443_CR67","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1177\/0735633119830764","volume":"58","author":"J Wilson","year":"2020","unstructured":"Wilson, J., & Roscoe, R. D. (2020). Automated writing evaluation and feedback: Multiple metrics of efficacy. Journal of Educational Computing Research, 58(1), 87\u2013125. https:\/\/doi.org\/10.1177\/0735633119830764","journal-title":"Journal of Educational Computing Research"},{"issue":"4","key":"443_CR68","doi-asserted-by":"publisher","first-page":"875","DOI":"10.1177\/07356331221127300","volume":"61","author":"N Zhai","year":"2022","unstructured":"Zhai, N., & Ma, X. (2022). The effectiveness of automated writing evaluation on writing quality: A meta-analysis. Journal of Educational Computing Research, 61(4), 875\u2013900. https:\/\/doi.org\/10.1177\/07356331221127300","journal-title":"Journal of Educational Computing Research"}],"container-title":["International Journal of Artificial Intelligence in Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40593-024-00443-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40593-024-00443-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40593-024-00443-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,4]],"date-time":"2026-03-04T18:12:36Z","timestamp":1772647956000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40593-024-00443-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,16]]},"references-count":68,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,9]]}},"alternative-id":["443"],"URL":"https:\/\/doi.org\/10.1007\/s40593-024-00443-9","relation":{},"ISSN":["1560-4292","1560-4306"],"issn-type":[{"value":"1560-4292","type":"print"},{"value":"1560-4306","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,16]]},"assertion":[{"value":"13 November 2024","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 December 2024","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Dr. Wilson and Dr. May have no conflicts of interest to declare that are relevant to the content of this manuscript. Dr. Huang is employed by Measurement Incorporated.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}}]}}