{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,26]],"date-time":"2026-07-26T17:01:00Z","timestamp":1785085260848,"version":"3.55.0"},"reference-count":103,"publisher":"SAGE Publications","license":[{"start":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T00:00:00Z","timestamp":1784764800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Applied Psychological Measurement"],"abstract":"<jats:p>The rapid adoption of Large Language Models (LLMs) in educational assessment has reshaped scoring practices, yet evaluation remains tethered to aggregate reliability metrics like Quadratic Weighted Kappa, which obscure discrimination and rater effects. This study applies Signal Detection Theory to evaluate eight state-of-the-art LLMs (including Claude 3.5 Haiku, DeepSeek-V3, Gemini 3 Flash, GPT-4o, and Grok 4.1) against expert human raters across 1,726 essays. By decoupling discrimination from response criteria, I provide a diagnostic analysis of AI scoring behavior. Results indicate that human raters exhibit significantly superior evaluative precision, with average discrimination estimates approximately double those of the AI models. Furthermore, LLMs are prone to pronounced centrality effects and score compression, systematically failing to award the highest rubric tiers. These findings demonstrate that low human-machine agreement stems from both a deficit in discriminative accuracy and systematic shifts in response criteria. Ultimately, this research provides a robust framework for calibrating and selecting AI scoring systems based on specific pedagogical goals and fairness requirements.<\/jats:p>","DOI":"10.1177\/01466216261471171","type":"journal-article","created":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T04:16:22Z","timestamp":1784866582000},"update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":0,"title":["Evaluating AI-Based Automated Essay Scoring Through Signal Detection Theory: Beyond Aggregate Agreement Metrics"],"prefix":"10.1177","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0467-6123","authenticated-orcid":false,"given":"Xiaoliang","family":"Zhou","sequence":"first","affiliation":[{"name":"Methodology and Measurement, Australian Council for Educational Research, Melbourne, VIC, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2026,7,23]]},"reference":[{"key":"e_1_3_5_2_1","doi-asserted-by":"crossref","unstructured":"Alansari A. Luqman H. (2025). Large language models hallucination: A comprehensive survey. arXiv preprint arXiv:2510.06265. https:\/\/doi.org\/10.48550\/arXiv.2510.06265","DOI":"10.1016\/j.cosrev.2026.100970"},{"key":"e_1_3_5_3_1","unstructured":"Anthropic. (2024). Claude 3.5 Haiku (Version claude-3-5-haiku-20241022) [large language model]. https:\/\/www.anthropic.com\/claude"},{"key":"e_1_3_5_4_1","first-page":"181","volume-title":"Handbook of automated essay evaluation","author":"Attali Y.","year":"2013","unstructured":"Attali Y. (2013). Validity and reliability of automated essay scoring. In Handbook of automated essay evaluation (pp. 181\u2013198). Routledge."},{"issue":"5","key":"e_1_3_5_5_1","first-page":"1189","article-title":"Decision processes in discrimination: Fundamental misrepresentations of signal detection theory","volume":"25","author":"Balakrishnan J. D.","year":"1999","unstructured":"Balakrishnan J. D. (1999). Decision processes in discrimination: Fundamental misrepresentations of signal detection theory. Journal of Experimental Psychology: Human Perception and Performance, 25(5), 1189\u20131206. https:\/\/doi.org\/10.1037\/0096-1523.25.5.1189","journal-title":"Journal of Experimental Psychology: Human Perception and Performance"},{"issue":"2","key":"e_1_3_5_6_1","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1037\/h0029531","article-title":"Signal detection theory and human memory","volume":"74","author":"Banks W. P.","year":"1970","unstructured":"Banks W. P. (1970). Signal detection theory and human memory. Psychological Bulletin, 74(2), 81\u201399. https:\/\/doi.org\/10.1037\/h0029531","journal-title":"Psychological Bulletin"},{"key":"e_1_3_5_7_1","first-page":"49","volume-title":"Automated scoring of complex tasks in computer-based testing","author":"Bejar I. I.","year":"2006","unstructured":"Bejar I. I., Williamson D. M., Mislevy R. J. (2006). Human scoring. In Automated scoring of complex tasks in computer-based testing (pp. 49\u201382). Lawrence Erlbaum Associates."},{"key":"e_1_3_5_8_1","doi-asserted-by":"publisher","DOI":"10.1080\/10824669.2018.1523734"},{"key":"e_1_3_5_9_1","first-page":"221","volume-title":"Handbook of automated essay evaluation","author":"Bridgeman B.","year":"2013","unstructured":"Bridgeman B. (2013). Human ratings and automated essay evaluation. In Handbook of automated essay evaluation (pp. 221\u2013232). Routledge."},{"key":"e_1_3_5_10_1","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown T.","year":"2020","unstructured":"Brown T., Mann B., Ryder N., Subbiah M., Kaplan J., Dhariwal P., Neelakantan A., Shyam P., Sastry G., Askell A., Agarwal S., Herbert-Voss A., Krueger G., Henighan T., Child R., Ramesh A., Ziegler D., Wu J., Winter C., Hesse C., Chen M., Sigler E., Litwin M., Gray S., Chess B., Clark J., Berner C., McCandlish S., Radford A., Sutskever I., Amodei D. (2020). Language models are few-shot learners. Advances in Neural Information processing Systems, 33, 1877\u20131901. https:\/\/doi.org\/10.48550\/arXiv.2005.14165","journal-title":"Advances in Neural Information processing Systems"},{"key":"e_1_3_5_11_1","unstructured":"Bulut O. Beiting-Parrish M. Casabianca J. M. Slater S. C. Jiao H. Song D. Ormerod C. Fabiyi D. G. Ivan R. Walsh C. Rios O. Wilson J. Yildirim-Erbasli S. N. Wongvorachan T. Liu J. X. Tan B. Morilova P. (2024). The rise of artificial intelligence in educational measurement: Opportunities and ethical challenges. arXiv. https:\/\/doi.org\/10.48550\/arXiv.2406.18900"},{"key":"e_1_3_5_12_1","first-page":"419","volume-title":"Seminars in nuclear medicine","author":"Burgess A. E.","year":"2011","unstructured":"Burgess A. E. (2011, November). Visual perception studies and observer models in medical imaging. In Seminars in nuclear medicine (Vol. 41(6), pp. 419\u2013436). WB Saunders. https:\/\/doi.org\/10.1053\/j.semnuclmed.2011.06.005"},{"key":"e_1_3_5_13_1","first-page":"61","volume-title":"Proceedings of the artificial intelligence in measurement and education conference (AIME-Con): Coordinated session papers","author":"Burleigh T.","year":"2025","unstructured":"Burleigh T., Chen J., DiCerbo K. (2025, October). Pre-pilot optimization of conversation-based assessment items using synthetic response data. In Proceedings of the artificial intelligence in measurement and education conference (AIME-Con): Coordinated session papers (pp. 61\u201368). https:\/\/aclanthology.org\/2025.aimecon-sessions.7.pdf"},{"key":"e_1_3_5_14_1","doi-asserted-by":"crossref","first-page":"107003","DOI":"10.52202\/079017-3397","article-title":"Risk-averse fine-tuning of large language models","volume":"37","author":"Chaudhary S.","year":"2024","unstructured":"Chaudhary S., Dinesha U., Kalathil D., Shakkottai S. (2024). Risk-averse fine-tuning of large language models. Advances in Neural Information Processing Systems, 37, 107003\u2013107038. https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2024\/hash\/c182ec594f38926b7fcb827635b9a8f4-Abstract-Conference.html.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_5_15_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0026256"},{"key":"e_1_3_5_16_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-3984.2000.tb01081.x"},{"key":"e_1_3_5_17_1","unstructured":"Cossio M. (2025). A comprehensive taxonomy of hallucinations in large language models. arXiv preprint arXiv:2508.01781. https:\/\/doi.org\/10.48550\/arXiv.2508.01781"},{"key":"e_1_3_5_18_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327906MBR3704_01"},{"key":"e_1_3_5_19_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.0022-0655.2005.00004.x"},{"key":"e_1_3_5_20_1","volume-title":"Studies of a latent class signal detection model for constructed response scoring","author":"DeCarlo L. T.","year":"2008","unstructured":"DeCarlo L. T. (2008). Studies of a latent class signal detection model for constructed response scoring. Educational Testing Service. ETS Technical Report, RR-08-63. https:\/\/doi.org\/10.1002\/j.2333-8504.2008.tb02149.x"},{"issue":"3","key":"e_1_3_5_21_1","doi-asserted-by":"crossref","first-page":"304","DOI":"10.1016\/j.jmp.2010.01.001","article-title":"On the statistical and theoretical basis of signal detection theory and extensions: Unequal variance, random coefficient, and mixture models","volume":"54","author":"DeCarlo L. T.","year":"2010","unstructured":"DeCarlo L. T. (2010). On the statistical and theoretical basis of signal detection theory and extensions: Unequal variance, random coefficient, and mixture models. Journal of Mathematical Psychology, 54(3), 304\u2013313. https:\/\/doi.org\/10.1016\/j.jmp.2010.01.001","journal-title":"Journal of Mathematical Psychology"},{"key":"e_1_3_5_22_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-3984.2011.00143.x"},{"key":"e_1_3_5_23_1","doi-asserted-by":"publisher","DOI":"10.1111\/jedm.12265"},{"key":"e_1_3_5_24_1","doi-asserted-by":"crossref","first-page":"347","DOI":"10.1002\/9781119373780.ch24","article-title":"Standard setting methods in medical education: High\u2010stakes assessment","author":"De Champlain A. F.","year":"2018","unstructured":"De Champlain A. F. (2018). Standard setting methods in medical education: High\u2010stakes assessment. Understanding Medical Education: Evidence, Theory, and practice, 347\u2013359. https:\/\/doi.org\/10.1002\/9781119373780.ch24","journal-title":"Understanding Medical Education: Evidence, Theory, and practice"},{"key":"e_1_3_5_25_1","unstructured":"DeepSeek. (2024). DeepSeek-V3 (model: Deepseek-chat) [large language model]. https:\/\/www.deepseek.com"},{"key":"e_1_3_5_26_1","unstructured":"Deng H. Farber C. Lee J. Tang D. (2025). Rubric-conditioned LLM grading: Alignment uncertainty and robustness. arXiv preprint arXiv:2601.08843. https:\/\/doi.org\/10.48550\/arXiv.2601.08843"},{"key":"e_1_3_5_27_1","doi-asserted-by":"publisher","DOI":"10.17705\/1jais.00867"},{"key":"e_1_3_5_28_1","doi-asserted-by":"publisher","DOI":"10.1027\/1015-5759\/a000904"},{"key":"e_1_3_5_29_1","doi-asserted-by":"publisher","DOI":"10.3390\/app15105683"},{"key":"e_1_3_5_30_1","first-page":"224","volume-title":"Large-scale assessment programs for all students","author":"Engelhard G.","year":"2012","unstructured":"Engelhard G. (2012). Monitoring raters in performance assessments. In Large-scale assessment programs for all students (pp. 224\u2013249). Routledge."},{"key":"e_1_3_5_31_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-3984.1994.tb00436.x"},{"issue":"3","key":"e_1_3_5_32_1","doi-asserted-by":"crossref","first-page":"288","DOI":"10.1111\/jedm.12333","article-title":"Validity arguments for AI\u2010based automated scores: Essay scoring as an illustration","volume":"59","author":"Ferrara S.","year":"2022","unstructured":"Ferrara S., Qunbar S. (2022). Validity arguments for AI\u2010based automated scores: Essay scoring as an illustration. Journal of Educational Measurement, 59(3), 288\u2013313. https:\/\/doi.org\/10.1111\/jedm.12333","journal-title":"Journal of Educational Measurement"},{"key":"e_1_3_5_33_1","doi-asserted-by":"publisher","DOI":"10.1027\/2151-2604\/a000567"},{"key":"e_1_3_5_34_1","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1007\/978-3-032-10738-1_9","volume-title":"Elements of deep learning","author":"Ghojogh B.","year":"2026","unstructured":"Ghojogh B., Ghodsi A. (2026). Attention mechanism and transformers. In Elements of deep learning (pp. 231\u2013257): Springer Nature Switzerland."},{"key":"e_1_3_5_35_1","first-page":"001316442614176","article-title":"From agreement to epistemic alignment: A signal detection\u2013theoretic model of inter-rater reliability","author":"Gianeselli I.","year":"2026","unstructured":"Gianeselli I. (2026). From agreement to epistemic alignment: A signal detection\u2013theoretic model of inter-rater reliability. Educational and Psychological Measurement, 00131644261417643. https:\/\/doi.org\/10.1177\/00131644261417643","journal-title":"Educational and Psychological Measurement"},{"key":"e_1_3_5_36_1","volume-title":"Gemini 2.5 (Flash\/Flash Lite\/Pro)","author":"Google","year":"2025","unstructured":"Google. (2025a). Gemini 2.5 (Flash\/Flash Lite\/Pro) [large language model]. https:\/\/gemini.google.com"},{"key":"e_1_3_5_37_1","volume-title":"Gemini 3 Flash Preview","author":"Google","year":"2025","unstructured":"Google. (2025b). Gemini 3 Flash Preview [Large language model]. https:\/\/aistudio.google.com"},{"key":"e_1_3_5_38_1","first-page":"1969","volume-title":"Signal detection theory and psychophysics","author":"Green D. M.","year":"1966","unstructured":"Green D. M., Swets J. A. (1966). Signal detection theory and psychophysics (Vol. 1, pp. 1969\u20132012). Wiley."},{"key":"e_1_3_5_39_1","volume-title":"Psychometric methods","author":"Guilford J. P.","year":"1954","unstructured":"Guilford J. P. (1954). Psychometric methods (2nd ed.). McGraw-Hill.","edition":"2"},{"key":"e_1_3_5_40_1","volume-title":"The Hewlett foundation: Automated essay scoring","author":"Hamner B.","year":"2012","unstructured":"Hamner B., Morgan J., lynnvandev, Shermis M., Vander Ark T. (2012). The Hewlett foundation: Automated essay scoring. Kaggle. https:\/\/www.kaggle.com\/competitions\/asap-aes"},{"key":"e_1_3_5_41_1","doi-asserted-by":"crossref","first-page":"2250","DOI":"10.18653\/v1\/2025.findings-naacl.120","volume-title":"Findings of the association for computational linguistics: NAACL 2025","author":"Hardy M.","year":"2025","unstructured":"Hardy M. (2025, April). \u201cAll that glitters\u201d: Techniques for evaluations with unreliable model and human annotations. In Findings of the association for computational linguistics: NAACL 2025 (pp. 2250\u20132278). https:\/\/doi.org\/10.18653\/v1\/2025.findings-naacl.120"},{"issue":"1","key":"e_1_3_5_42_1","first-page":"5215722","article-title":"Artificial intelligence for assessment and feedback to enhance student success in higher education","volume":"2022","author":"Hooda M.","year":"2022","unstructured":"Hooda M., Rana C., Dahiya O., Rizwan A., Hossain M. S. (2022). Artificial intelligence for assessment and feedback to enhance student success in higher education. Mathematical Problems in Engineering, 2022(1), 5215722. https:\/\/doi.org\/10.1155\/2022\/5215722","journal-title":"Mathematical Problems in Engineering"},{"key":"e_1_3_5_43_1","unstructured":"Hu X. Qian Y. Yu J. Liu J. Ji X. Xu C. Tang P. Xu C. Tang P. Liu J. Yu X. Zhang G. Yu X. Liao Y. Pan J. Xu Z. Jian B. Wu K. Zhang J. Gao S. Jin Y. Wu Z. Fu Y. Li H. B. Menze B. Jiang Y.-G. Rueckert D. Yan S. (2025). The landscape of medical agents: A survey. Authorea Preprints. https:\/\/d197for5662m48.cloudfront.net\/documents\/publicationstatus\/296148\/preprint_pdf\/2ee39730675f77c8b41e0826952b6154.pdf"},{"key":"e_1_3_5_44_1","doi-asserted-by":"crossref","first-page":"19900","DOI":"10.18653\/v1\/2025.emnlp-main.1006","volume-title":"Proceedings of the 2025 conference on empirical methods in natural language processing","author":"Hua A.","year":"2025","unstructured":"Hua A., Tang K., Gu C., Gu J., Wong E., Qin Y. (2025, November). Flaw or artifact? Rethinking prompt sensitivity in evaluating LLMs. In Proceedings of the 2025 conference on empirical methods in natural language processing (pp. 19900\u201319910). https:\/\/doi.org\/10.18653\/v1\/2025.emnlp-main.1006"},{"key":"e_1_3_5_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-024-10824-0"},{"key":"e_1_3_5_46_1","doi-asserted-by":"publisher","DOI":"10.1007\/s40299-024-00865-y"},{"key":"e_1_3_5_47_1","first-page":"71","volume-title":"Proceedings of the artificial intelligence in measurement and education conference (AIME-Con): Works in progress","author":"Huang Y.","year":"2025","unstructured":"Huang Y., Wilson J. (2025, October). Evaluating LLM-based automated essay scoring: Accuracy, fairness, and validity. In Proceedings of the artificial intelligence in measurement and education conference (AIME-Con): Works in progress (pp. 71\u201383). https:\/\/aclanthology.org\/2025.aimecon-wip.9.pdf"},{"key":"e_1_3_5_48_1","unstructured":"Jiao H. Song D. Lee W. C. (2025). Comparing human and AI rater effects using the many-facet rasch model. arXiv preprint arXiv:2505.18486. https:\/\/doi.org\/10.48550\/arXiv.2505.184"},{"key":"e_1_3_5_49_1","unstructured":"Joshi S. (2025a). A comprehensive review of deepseek: Performance architecture and capabilities. https:\/\/www.preprints.org\/frontend\/manuscript\/e9d16deacca317bc355d4bffa25692b4\/download_pub"},{"key":"e_1_3_5_50_1","doi-asserted-by":"crossref","unstructured":"Joshi S. (2025b). Evaluation of large language models: Review of metrics applications and methodologies. https:\/\/doi.org\/10.20944\/preprints202504.0369.v1","DOI":"10.20944\/preprints202504.0369.v2"},{"key":"e_1_3_5_51_1","doi-asserted-by":"crossref","first-page":"100364","DOI":"10.1016\/j.caeo.2026.100364","article-title":"A systematic comparison of large language models for automated assignment assessment in programming education: Exploring the importance of architecture and vendor","volume":"10","author":"Jukiewicz M.","year":"2026","unstructured":"Jukiewicz M. (2026). A systematic comparison of large language models for automated assignment assessment in programming education: Exploring the importance of architecture and vendor. Computers and Education Open, 10, 100364. https:\/\/doi.org\/10.1016\/j.caeo.2026.100364","journal-title":"Computers and Education Open"},{"key":"e_1_3_5_52_1","first-page":"968289","volume-title":"Frontiers in education","author":"Kaldaras L.","year":"2022","unstructured":"Kaldaras L., Haudek K. C. (2022, December). Validation of automated scoring for learning progression-aligned next generation science standards performance assessments. In Frontiers in education (Vol. 7, p. 968289). Frontiers Media SA. https:\/\/doi.org\/10.3389\/feduc.2022.968289"},{"key":"e_1_3_5_53_1","doi-asserted-by":"crossref","first-page":"164","DOI":"10.18653\/v1\/2023.eval4nlp-1.14","volume-title":"Proceedings of the 4th workshop on evaluation and comparison of NLP systems","author":"Kim J.","year":"2023","unstructured":"Kim J., Lee S., Han S. H., Park S., Lee J., Jeong K., Kang P. (2023, November). Which is better? Exploring prompting strategy for llm-based metrics. In Proceedings of the 4th workshop on evaluation and comparison of NLP systems (pp. 164\u2013183). https:\/\/doi.org\/10.18653\/v1\/2023.eval4nlp-1.14"},{"key":"e_1_3_5_54_1","volume-title":"Combining constructed response items and multiple choice items using a hierarchical rater model","author":"Kim Y. K.","year":"2009","unstructured":"Kim Y. K. (2009). Combining constructed response items and multiple choice items using a hierarchical rater model. Columbia University."},{"key":"e_1_3_5_55_1","doi-asserted-by":"publisher","DOI":"10.1007\/s40593-020-00211-5"},{"issue":"1","key":"e_1_3_5_56_1","first-page":"100213","article-title":"Applying large language models and chain-of-thought for automatic scoring","volume":"6","author":"Lee G.-G.","year":"2024","unstructured":"Lee G.-G., Latif E., Wu X., Liu N., Zhai X. (2024). Applying large language models and chain-of-thought for automatic scoring. Computers and Education: Artificial Intelligence, 6(1), 100213. https:\/\/doi.org\/10.1016\/j.caeai.2024.100213","journal-title":"Computers and Education: Artificial Intelligence"},{"issue":"1","key":"e_1_3_5_57_1","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1002\/sapm1946251261","article-title":"The wiener (root mean square) error criterion in filter design and prediction","volume":"25","author":"Levinson N.","year":"1946","unstructured":"Levinson N. (1946). The wiener (root mean square) error criterion in filter design and prediction. Journal of Mathematics and Physics, 25(1-4), 261\u2013278. https:\/\/doi.org\/10.7551\/mitpress\/2946.001.0001","journal-title":"Journal of Mathematics and Physics"},{"key":"e_1_3_5_58_1","unstructured":"Li Y. Fu X. Verma G. Buitelaar P. Liu M. (2025). Mitigating hallucination in large language models (LLMs): An application-oriented survey on RAG Reasoning and Agentic Systems. arXiv preprint arXiv:2510.24476. https:\/\/doi.org\/10.48550\/arXiv.2510.24476"},{"issue":"2","key":"e_1_3_5_59_1","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1037\/h0029536","article-title":"Memory and the theory of signal detection","volume":"74","author":"Lockhart R. S.","year":"1970","unstructured":"Lockhart R. S., Murdock B. B. (1970). Memory and the theory of signal detection. Psychological Bulletin, 74(2), 100\u2013109. https:\/\/doi.org\/10.1037\/h0029536","journal-title":"Psychological Bulletin"},{"key":"e_1_3_5_60_1","volume-title":"Comparing the robustness of deep learning and classical automated scoring approaches to gaming strategies","author":"Lottridge S.","year":"2021","unstructured":"Lottridge S., Godek B., Jafari A., Patel M. (2021). Comparing the robustness of deep learning and classical automated scoring approaches to gaming strategies. Cambium Assessment Inc. Technical report. https:\/\/files.portal.cambiumast.com\/corporate-site\/documents\/CAI-Cambium-Comparing-Robustness-Automated-Scoring-Approaches.pdf"},{"key":"e_1_3_5_61_1","volume-title":"Signal detection theory. Stevens\u2019 handbook of experimental psychology","author":"Macmillan N. A.","year":"2002","unstructured":"Macmillan N. A. (2002). Signal detection theory. Stevens\u2019 handbook of experimental psychology. John Wiley & Sons, Inc."},{"key":"e_1_3_5_62_1","unstructured":"Mars F. (2026). Transitive expert error and routing problems in complex AI systems. arXiv preprint arXiv:2601.04416. https:\/\/doi.org\/10.48550\/arXiv.2601.04416"},{"key":"e_1_3_5_63_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.rmal.2023.100050"},{"key":"e_1_3_5_64_1","doi-asserted-by":"publisher","DOI":"10.1177\/014662169201600206"},{"key":"e_1_3_5_65_1","unstructured":"Murff M. J. (2025). AI-Enabled innovations in automated essay scoring and Feedback Systems. https:\/\/scholarsarchive.byu.edu\/etd\/10933"},{"issue":"4","key":"e_1_3_5_66_1","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1037\/dec0000244","article-title":"Dimensions of disagreement: Divergence and misalignment in cognitive science and artificial intelligence","volume":"11","author":"Oktar K.","year":"2024","unstructured":"Oktar K., Sucholutsky I., Lombrozo T., Griffiths T. L. (2024). Dimensions of disagreement: Divergence and misalignment in cognitive science and artificial intelligence. Decision, 11(4), 511\u2013522. https:\/\/doi.org\/10.1037\/dec0000244","journal-title":"Decision"},{"key":"e_1_3_5_67_1","volume-title":"GPT-4o","author":"OpenAI","year":"2024","unstructured":"OpenAI. (2024). GPT-4o [large language model]. https:\/\/openai.com\/"},{"key":"e_1_3_5_68_1","doi-asserted-by":"publisher","DOI":"10.52202\/068431-2011"},{"issue":"12983","key":"e_1_3_5_69_1","first-page":"100234","article-title":"Large language models and automated essay scoring of English language learner writing: Insights into validity and reliability","volume":"6","author":"Pack A.","year":"2024","unstructured":"Pack A., Barrett A., Escalante J. (2024). Large language models and automated essay scoring of English language learner writing: Insights into validity and reliability. Computers and Education: Artificial Intelligence, 6(12983), 100234. https:\/\/doi.org\/10.1016\/j.caeai.2024.100234","journal-title":"Computers and Education: Artificial Intelligence"},{"issue":"5","key":"e_1_3_5_70_1","first-page":"238","article-title":"The imminence of grading essays by computer","volume":"47","author":"Page E. B.","year":"1966","unstructured":"Page E. B. (1966). The imminence of grading essays by computer. The Phi Delta Kappan, 47(5), 238\u2013243. https:\/\/www.jstor.org\/stable\/20371545","journal-title":"The Phi Delta Kappan"},{"key":"e_1_3_5_71_1","unstructured":"Park C. Liu M. Kong D. Zhang K. Ozdaglar A. (2024). Rlhf from heterogeneous feedback via personalization and preference aggregation. arXiv preprint arXiv:2405.00254. https:\/\/doi.org\/10.48550\/arXiv.2405.00254"},{"issue":"757","key":"e_1_3_5_72_1","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1126\/science.30.757.23","article-title":"Determination of the coefficient of correlation","volume":"30","author":"Pearson K.","year":"1909","unstructured":"Pearson K. (1909). Determination of the coefficient of correlation. Science, 30(757), 23\u201325. https:\/\/doi.org\/10.1126\/science.30.757.23","journal-title":"Science"},{"issue":"7","key":"e_1_3_5_73_1","doi-asserted-by":"crossref","first-page":"443","DOI":"10.3390\/a18070443","article-title":"Explainability and interpretability in concept and data drift: A systematic literature review","volume":"18","author":"Pelosi D.","year":"2025","unstructured":"Pelosi D., Cacciagrano D., Piangerelli M. (2025). Explainability and interpretability in concept and data drift: A systematic literature review. Algorithms, 18(7), 443. https:\/\/doi.org\/10.3390\/a18070443","journal-title":"Algorithms"},{"issue":"1","key":"e_1_3_5_74_1","doi-asserted-by":"crossref","first-page":"962","DOI":"10.1186\/s12909-024-05881-6","article-title":"Reliability of ChatGPT in automated essay scoring for dental undergraduate examinations","volume":"24","author":"Quah B.","year":"2024","unstructured":"Quah B., Zheng L., Sng T. J. H., Yong C. W., Islam I. (2024). Reliability of ChatGPT in automated essay scoring for dental undergraduate examinations. BMC Medical Education, 24(1), 962. https:\/\/doi.org\/10.1186\/s12909-024-05881-6","journal-title":"BMC Medical Education"},{"key":"e_1_3_5_75_1","first-page":"303","volume-title":"European conference on information retrieval","author":"Razavi A.","year":"2025","unstructured":"Razavi A., Soltangheis M., Arabzadeh N., Salamat S., Zihayat M., Bagheri E. (2025, April). Benchmarking prompt sensitivity in large language models. In European conference on information retrieval (pp. 303\u2013313). Springer Nature Switzerland. https:\/\/doi.org\/10.1007\/978-3-031-88714-7_29"},{"key":"e_1_3_5_76_1","first-page":"384","volume-title":"When algorithms rate performance: Do large language models replicate human evaluation biases?","author":"Rilke R. M.","year":"2026","unstructured":"Rilke R. M., Sliwka D. (2026). When algorithms rate performance: Do large language models replicate human evaluation biases? (p. 384). University of Bonn and University of Cologne. https:\/\/www.econtribute.de\/RePEc\/ajk\/ajkdps\/ECONtribute_384_2026.pdf"},{"key":"e_1_3_5_77_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.88.2.413"},{"key":"e_1_3_5_78_1","doi-asserted-by":"crossref","first-page":"413","DOI":"10.56294\/ai2025413","article-title":"Human-in-the-Loop models for ethical AI grading: Combining AI speed with human ethical oversight","volume":"4","author":"Selvam M.","year":"2025","unstructured":"Selvam M., Vallejo R. G. (2025). Human-in-the-Loop models for ethical AI grading: Combining AI speed with human ethical oversight. EthAIca: Journal of Ethics, AI and Critical Analysis, 4, 413. https:\/\/doi.org\/10.56294\/ai2025413","journal-title":"EthAIca: Journal of Ethics, AI and Critical Analysis"},{"key":"e_1_3_5_79_1","doi-asserted-by":"crossref","first-page":"462","DOI":"10.1145\/3706468.3706527","volume-title":"Proceedings of the 15th international learning analytics and knowledge conference","author":"Se\u00dfler K.","year":"2025","unstructured":"Se\u00dfler K., F\u00fcrstenberg M., B\u00fchler B., Kasneci E. (2025, March). Can AI grade your essays? A comparative analysis of large language models and teacher ratings in multidimensional essay scoring. In Proceedings of the 15th international learning analytics and knowledge conference (pp. 462\u2013472). https:\/\/doi.org\/10.1145\/3706468.3706527"},{"key":"e_1_3_5_80_1","doi-asserted-by":"publisher","DOI":"10.4324\/9780203122761"},{"key":"e_1_3_5_81_1","doi-asserted-by":"crossref","first-page":"313","DOI":"10.4324\/9780203122761","volume-title":"Handbook of automated essay evaluation","author":"Shermis M. D.","year":"2013","unstructured":"Shermis M. D., Hamner B. (2013). Contrasting state-of-the-art automated scoring of essays. In Handbook of automated essay evaluation (pp. 313\u2013346). Routledge."},{"key":"e_1_3_5_82_1","doi-asserted-by":"publisher","DOI":"10.1109\/TLT.2024.3396873"},{"key":"e_1_3_5_83_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03207704"},{"key":"e_1_3_5_84_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/09588221.2025.2481397","article-title":"Can we reliably score meaning recall vocabulary tests using AI? A comparison of human vs. AI scoring","author":"Stewart J.","year":"2025","unstructured":"Stewart J., Anthony L., Batty A. O., Nakamura K., Nicklin C., McLean S., Tomaru K. (2025). Can we reliably score meaning recall vocabulary tests using AI? A comparison of human vs. AI scoring. Computer Assisted Language Learning, 1\u201323. https:\/\/doi.org\/10.1080\/09588221.2025.2481397","journal-title":"Computer Assisted Language Learning"},{"key":"e_1_3_5_85_1","doi-asserted-by":"publisher","DOI":"10.4324\/9781315806167"},{"key":"e_1_3_5_86_1","doi-asserted-by":"publisher","DOI":"10.1152\/advan.00137.2024"},{"issue":"5","key":"e_1_3_5_87_1","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1111\/j.1743-498X.2012.00567.x","article-title":"Diagnostic testing and educational assessment","volume":"9","author":"Tweed M.","year":"2012","unstructured":"Tweed M., Wilkinson T. (2012). Diagnostic testing and educational assessment. The Clinical Teacher, 9(5), 299\u2013303. https:\/\/doi.org\/10.1111\/j.1743-498X.2012.00567.x","journal-title":"The Clinical Teacher"},{"issue":"1","key":"e_1_3_5_88_1","doi-asserted-by":"crossref","first-page":"e000046","DOI":"10.1136\/bmjdhai-2025-000046","article-title":"Importance of model governance in clinical AI models: Case study on the relevance of data drift detection","volume":"1","author":"van der Vorst J. P.","year":"2025","unstructured":"van der Vorst J. P., Smit J. M., van de Sande D., van der Ster B., Daams F., Schasfoort R., Gommers D., Verhoef C., Gr\u00fcnhagen D. J., van Genderen M. E., Hilling D. E. (2025). Importance of model governance in clinical AI models: Case study on the relevance of data drift detection. BMJ Digital Health & AI, 1(1), Article e000046. https:\/\/doi.org\/10.1136\/bmjdhai-2025-000046","journal-title":"BMJ Digital Health & AI"},{"key":"e_1_3_5_89_1","unstructured":"Vermunt J. K. Magidson J. (2021). Latent GOLD (version 6.1). Statistical Innovations Inc. [Computer software]. https:\/\/www.statisticalinnovations.com\/latent-gold-6-1"},{"key":"e_1_3_5_90_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2025.100481"},{"key":"e_1_3_5_91_1","doi-asserted-by":"publisher","DOI":"10.52202\/068431-1800"},{"key":"e_1_3_5_92_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-3992.2011.00223.x"},{"issue":"4","key":"e_1_3_5_93_1","doi-asserted-by":"crossref","first-page":"619","DOI":"10.1037\/edu0000311","article-title":"Generalizability of automated scores of writing quality in grades 3\u20135","volume":"111","author":"Wilson J.","year":"2019","unstructured":"Wilson J., Chen D., Sandbank M. P., Hebert M. (2019). Generalizability of automated scores of writing quality in grades 3\u20135. Journal of Educational Psychology, 111(4), 619. 640. https:\/\/doi.org\/10.1037\/edu0000311. https:\/\/psycnet.apa.org\/buy\/2018-53139-001","journal-title":"Journal of Educational Psychology"},{"issue":"1","key":"e_1_3_5_94_1","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1080\/15305058.2017.1361426","article-title":"The influence of rater effects in training sets on the psychometric quality of automated scoring for writing assessments","volume":"18","author":"Wind S. A.","year":"2018","unstructured":"Wind S. A., Wolfe E. W., Engelhard Jr G., Foltz P., Rosenstein M. (2018). The influence of rater effects in training sets on the psychometric quality of automated scoring for writing assessments. International Journal of Testing, 18(1), 27\u201349. https:\/\/doi.org\/10.1080\/15305058.2017.1361426","journal-title":"International Journal of Testing"},{"key":"e_1_3_5_95_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.114.1.152"},{"issue":"2","key":"e_1_3_5_96_1","first-page":"1","article-title":"Unveiling scoring processes: Dissecting the differences between llms and human graders in automatic scoring","volume":"31","author":"Wu X.","year":"2025","unstructured":"Wu X., Saraf P. P., Lee G., Latif E., Liu N., Zhai X. (2025). Unveiling scoring processes: Dissecting the differences between llms and human graders in automatic scoring. Technology, Knowledge and Learning, 31(2), 1\u201316. https:\/\/doi.org\/10.1007\/s10758-025-09836-8","journal-title":"Technology, Knowledge and Learning"},{"key":"e_1_3_5_97_1","unstructured":"xAI. (2025). Grok 4.1 Fast Reasoning (Version grok-4-1-fast-reasoning) [Large language model]. https:\/\/x.ai"},{"key":"e_1_3_5_98_1","doi-asserted-by":"publisher","DOI":"10.1111\/jedm.70002"},{"key":"e_1_3_5_99_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.bea-1.49"},{"issue":"4","key":"e_1_3_5_100_1","doi-asserted-by":"crossref","first-page":"3","DOI":"10.15858\/engtea.80.4.202512.3","article-title":"AI-based L2 pragmatic speaking assessment: Evidence from many-facet rasch measurement","volume":"80","author":"Youn S. J.","year":"2025","unstructured":"Youn S. J. (2025). AI-based L2 pragmatic speaking assessment: Evidence from many-facet rasch measurement. English Teaching, 80(4), 3\u201328. https:\/\/doi.org\/10.15858\/engtea.80.4.202512.3","journal-title":"English Teaching"},{"issue":"2","key":"e_1_3_5_101_1","first-page":"1","article-title":"Contrasting automated and human scoring of essays","volume":"21","author":"Zhang M.","year":"2013","unstructured":"Zhang M. (2013). Contrasting automated and human scoring of essays. R & D Connections, 21(2), 1\u201311. https:\/\/www.ets.org\/Media\/Research\/pdf\/RD_Connections_21.pdf","journal-title":"R & D Connections"},{"key":"e_1_3_5_102_1","first-page":"1","volume-title":"2023 IEEE international conference on teaching, assessment and learning for engineering (TALE)","author":"Zheng W.","year":"2023","unstructured":"Zheng W., Tse A. W. C. (2023, November). The impact of generative artificial intelligence-based formative feedback on the mathematical motivation of Chinese grade 4 students: A case study. In 2023 IEEE international conference on teaching, assessment and learning for engineering (TALE) (pp. 1\u20138). IEEE. https:\/\/doi.org\/10.1109\/TALE56641.2023.10398319"},{"key":"e_1_3_5_103_1","volume-title":"Studies of extensions of HRM-SDT for constructed responses","author":"Zhou X.","year":"2019","unstructured":"Zhou X. (2019). Studies of extensions of HRM-SDT for constructed responses. Columbia University."},{"key":"e_1_3_5_104_1","first-page":"1","article-title":"On a hierarchical rater model with ordered perceptions","author":"Zhou X.","year":"2026","unstructured":"Zhou X., Tan L. (2026). On a hierarchical rater model with ordered perceptions. Applied Measurement in Education, 1\u201326. https:\/\/doi.org\/10.1080\/08957347.2026.2644921","journal-title":"Applied Measurement in Education"}],"container-title":["Applied Psychological Measurement"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/01466216261471171","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/01466216261471171","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/01466216261471171","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,26]],"date-time":"2026-07-26T16:52:08Z","timestamp":1785084728000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/01466216261471171"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,23]]},"references-count":103,"alternative-id":["10.1177\/01466216261471171"],"URL":"https:\/\/doi.org\/10.1177\/01466216261471171","relation":{},"ISSN":["0146-6216","1552-3497"],"issn-type":[{"value":"0146-6216","type":"print"},{"value":"1552-3497","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,23]]},"article-number":"01466216261471171"}}