{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,26]],"date-time":"2026-06-26T03:29:10Z","timestamp":1782444550565,"version":"3.54.5"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2023,7,10]],"date-time":"2023-07-10T00:00:00Z","timestamp":1688947200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,7,10]],"date-time":"2023-07-10T00:00:00Z","timestamp":1688947200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004245","name":"Uninorte","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004245","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Educ Inf Technol"],"published-print":{"date-parts":[[2024,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Today reading comprehension is considered an essential skill in modern life, therefore, higher education students require more specific skills to understand, interpret and evaluate texts effectively. Short answer questions (SAQs) are one of the relevant and proper tools for assessing reading comprehension skills. Unlike multiple-choice questions, SAQs allow for the assessment of cognitive abilities such as attention, language, perception, and problem solving. However, the task of SAQs scoring is time-consuming and susceptible to ambiguity. Automatic Short Answer Grading (ASAG) is a new paradigm that could help solve these problems. This experimental analysis aims to implement ASAG using several approaches to sentence embedding based on deep learning with a multilayer perceptron regression layer on the top, trained with a reading comprehension dataset based on aphorisms. For experimental testing, the available dataset is composed of answers given by 199 undergraduate students in Spanish. BERT and Skip-Thought models are tested with different hyperparameters to find the best performance in terms of Pearson correlation coefficient and RMSE against human experts grades. The result of the current study showed that BERT model performed better than other approaches.<\/jats:p>","DOI":"10.1007\/s10639-023-11890-7","type":"journal-article","created":{"date-parts":[[2023,7,10]],"date-time":"2023-07-10T08:02:22Z","timestamp":1688976142000},"page":"4565-4590","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["A deep-learning-based grading system (ASAG) for reading comprehension assessment by using aphorisms as open-answer-questions"],"prefix":"10.1007","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5068-4653","authenticated-orcid":false,"given":"Ivan D.","family":"Mardini G.","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christian G.","family":"Quintero M.","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"C\u00e9sar A.","family":"Viloria N.","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Winston S.","family":"Percybrooks B.","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Heydy S.","family":"Robles N.","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Karen","family":"Villalba R.","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,7,10]]},"reference":[{"issue":"1","key":"11890_CR1","doi-asserted-by":"publisher","first-page":"19","DOI":"10.21742\/IJHIT.2020.13.1.03","volume":"13","author":"AC Adamuthe","year":"2020","unstructured":"Adamuthe, A. C. (2020). Improved text classification using long short-term memory and word embedding technique. International Journal of Hybrid Information Technology, 13(1), 19\u201332.","journal-title":"International Journal of Hybrid Information Technology"},{"key":"11890_CR2","unstructured":"Adams, O., Roy, S., & Krishnapuram, R. (2016). Distributed vector Rrepresentations for unsupervised automatic short answer grading. In Proceedings of the 3rd Workshop on Natural Language Processing Techniques for Educational Applications (NLPTEA2016) (pp. 20\u201329)."},{"key":"11890_CR3","unstructured":"Almeida, F., & Xex\u00e9o, G. (2019). Word embeddings: A survey. Preprint retrieved from http:\/\/arxiv.org\/abs\/1901.09069"},{"key":"11890_CR4","doi-asserted-by":"publisher","first-page":"391","DOI":"10.1162\/tacl_a_00236","volume":"1","author":"S Basu","year":"2013","unstructured":"Basu, S., Jacobs, C., & Vanderwende, L. (2013). Powergrading: A clustering approach to amplify human effort for short answer grading. Transactions of the Association for Computational Linguistics, 1, 391\u2013402. https:\/\/doi.org\/10.1162\/tacl_a_00236","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"11890_CR5","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1162\/tacl_a_00051","volume":"5","author":"P Bojanowski","year":"2017","unstructured":"Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5, 135\u2013146.","journal-title":"Transactions of the Association for Computational Linguistics"},{"issue":"1","key":"11890_CR6","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1007\/s40593-014-0026-8","volume":"25","author":"S Burrows","year":"2015","unstructured":"Burrows, S., Gurevych, I., & Stein, B. (2015). The eras and trends of automatic short answer grading. International Journal of Artificial Intelligence in Education, 25(1), 60\u2013117.","journal-title":"International Journal of Artificial Intelligence in Education"},{"key":"11890_CR7","unstructured":"Canete, J., Chaperon, G., Fuentes, R., & P\u00e9rez, J. (2020). Spanish pre-trained Bert model and evaluation data. PML4DC at ICLR, 2020."},{"key":"11890_CR8","doi-asserted-by":"crossref","unstructured":"Camacho-Collados, J., & Pilehvar, M. T. (2020). Embeddings in natural language processing. In Proceedings of the 28th International Conference on Computational Linguistics: Tutorial Abstracts (pp. 10\u201315).","DOI":"10.18653\/v1\/2020.coling-tutorials.2"},{"key":"11890_CR9","doi-asserted-by":"crossref","unstructured":"Cho, K., Van Merri\u00ebnboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. Preprint retrieved from http:\/\/arxiv.org\/abs\/1406.1078","DOI":"10.3115\/v1\/D14-1179"},{"key":"11890_CR10","unstructured":"Devlin, J., Chang, M., Lee, K., & Toutanova, K. (2018). BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs\/1810.04805. Preprint retrieved from http:\/\/arxiv.org\/abs\/1810.04805"},{"issue":"10","key":"11890_CR11","doi-asserted-by":"publisher","first-page":"44","DOI":"10.5120\/ijca2017913766","volume":"163","author":"S Drolia","year":"2017","unstructured":"Drolia, S., Rupani, S., Agarwal, P., & Singh, A. (2017). Automated essay rater using natural language processing. International Journal of Computer Applications, 163(10), 44\u201346.","journal-title":"International Journal of Computer Applications"},{"key":"11890_CR12","unstructured":"Eisenstein, J. (2019). Introduction to natural language processing. Adaptive Computation and Machine Learning series. MIT Press, London. https:\/\/books.google.com.co\/books?id=72yuDwAAQBAJ"},{"key":"11890_CR13","unstructured":"Gomaa, W. H., & Fahmy, A. A. (2011). Tapping into the power of automatic scoring. In The Eleventh International Conference on Language Engineering, Egyptian Society of Language Engineering (ESOLEC)."},{"key":"11890_CR14","doi-asserted-by":"crossref","unstructured":"Gomaa, W. H., & Fahmy, A. A. (2019). Ans2vec: A scoring system for short answers. In International Conference on Advanced Machine Learning Technologies and Applications (pp. 586\u2013595). Springer.","DOI":"10.1007\/978-3-030-14118-9_59"},{"issue":"6","key":"11890_CR15","first-page":"127","volume":"8","author":"T Gong","year":"2019","unstructured":"Gong, T., & Yao, X. (2019). An attention-based deep model for automatic short answer score. International Journal of Computer Science and Software Engineering, 8(6), 127\u2013132.","journal-title":"International Journal of Computer Science and Software Engineering"},{"key":"11890_CR16","doi-asserted-by":"crossref","unstructured":"Guti\u00e9rrez, L., & Keith, B. (2018). A systematic literature review on word embeddings. In International Conference on Software Process Improvement (pp. 132\u2013141). Springer.","DOI":"10.1007\/978-3-030-01171-0_12"},{"key":"11890_CR17","doi-asserted-by":"crossref","unstructured":"Ghavidel, H. A., Zouaq, A., & Desmarais, M. C. (2020). Using BERT and XLNET for the Automatic Short Answer Grading Task. In CSEDU (1) (pp. 58\u201367).","DOI":"10.5220\/0009422400580067"},{"key":"11890_CR18","unstructured":"Haley, D., Thomas, P., De\u00a0Roeck, A., & Petre, M. (2007). Measuring improvement in latent semantic analysis-based marking systems: Using a computer to mark questions about html. Conferences in Research and Practice in Information Technology Series, 66"},{"key":"11890_CR19","doi-asserted-by":"crossref","unstructured":"Heilman, M., & Madnani, N. (2015). The impact of training data on automated short answer scoring performance. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications (pp. 81\u201385).","DOI":"10.3115\/v1\/W15-0610"},{"key":"11890_CR20","doi-asserted-by":"crossref","unstructured":"Huang, Y., Yang, X., Zhuang, F., Zhang, L., & Yu, S. (2018). Automatic Chinese reading comprehension grading by LSTM with knowledge adaptation. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (pp. 118\u2013129). Springer.","DOI":"10.1007\/978-3-319-93034-3_10"},{"key":"11890_CR21","unstructured":"IBM Cloud Education. (2020). Natural Language Processing (NLP). https:\/\/www.ibm.com\/cloud\/learn\/natural-language-processing"},{"key":"11890_CR22","unstructured":"Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., & Fidler, S. (2015). Skip-thought vectors. In Advances in Neural Information Processing Systems (pp. 3294\u20133302)."},{"issue":"3","key":"11890_CR23","doi-asserted-by":"publisher","first-page":"538","DOI":"10.1007\/s40593-020-00211-5","volume":"31","author":"VS Kumar","year":"2021","unstructured":"Kumar, V. S., & Boulanger, D. (2021). Automated essay scoring and the deep learning black box: How are rubric scores determined? International Journal of Artificial Intelligence in Education, 31(3), 538\u2013584.","journal-title":"International Journal of Artificial Intelligence in Education"},{"key":"11890_CR24","doi-asserted-by":"crossref","unstructured":"Lun, J., Zhu, J., Tang, Y., & Yang, M. (2020). Multiple data augmentation strategies for improving performance on automatic short answer scoring. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, pp. 13389\u201313396).","DOI":"10.1609\/aaai.v34i09.7062"},{"issue":"3","key":"11890_CR25","first-page":"18","volume":"4","author":"CH Maduabuchi","year":"2016","unstructured":"Maduabuchi, C. H., & Emechebe, V. I. (2016). ICT and the teaching of reading comprehension in English as a second language in secondary schools: Problems and prospects. International Journal of Education and Literacy Studies, 4(3), 18\u201323.","journal-title":"International Journal of Education and Literacy Studies"},{"key":"11890_CR26","unstructured":"Magooda, A. E., Zahran, M., Rashwan, M., Raafat, H., & Fayek, M. (2016). Vector based techniques for short answer grading. In The Twenty-Ninth International Flairs Conference."},{"key":"11890_CR27","unstructured":"Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. Preprint retrieved from http:\/\/arxiv.org\/abs\/1301.3781"},{"issue":"3","key":"11890_CR28","doi-asserted-by":"publisher","first-page":"10","DOI":"10.18869\/acadpub.ijree.2.3.10","volume":"2","author":"N Mohseni Takaloo","year":"2017","unstructured":"Mohseni Takaloo, N., & Ahmadi, M. R. (2017). The effect of learners\u2019 motivation on their reading comprehension skill: A literature review. International journal of research in English education, 2(3), 10\u201321.","journal-title":"International journal of research in English education"},{"key":"11890_CR29","unstructured":"Mohler, M., Bunescu, R., & Mihalcea, R. (2011). Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (pp. 752\u2013762)."},{"key":"11890_CR30","doi-asserted-by":"crossref","unstructured":"Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. Preprint retrieved from http:\/\/arxiv.org\/abs\/1802.05365","DOI":"10.18653\/v1\/N18-1202"},{"key":"11890_CR31","doi-asserted-by":"crossref","unstructured":"Pennington, J., Socher, R., & Manning, C. D. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 1532\u20131543).","DOI":"10.3115\/v1\/D14-1162"},{"key":"11890_CR32","unstructured":"Pollo\u00a0Cattaneo, M. F., Pytel, P., Vegega, C., Ram\u00f3n, H. D., Deroche, A., Straccia, L., Bernal, L., & Acosta, M. P. (2016) Implementaci\u00f3n de sistemas inteligentes para la asistencia a alumnos y docentes de la carrera de ingenier\u00eda en sistemas de informaci\u00f3n. In XVIII Workshop de Investigadores en Ciencias de la Computaci\u00f3n (WICC 2016, Entre R\u00edos, Argentina)"},{"key":"11890_CR33","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1016\/j.asw.2013.04.001","volume":"20","author":"MD Shermis","year":"2014","unstructured":"Shermis, M. D. (2014). State-of-the-art automated essay scoring: Competition, results, and future directions from a united states demonstration. Assessing Writing, 20, 53\u201376. https:\/\/doi.org\/10.1016\/j.asw.2013.04.001","journal-title":"Assessing Writing"},{"issue":"1","key":"11890_CR34","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1080\/10627197.2015.997617","volume":"20","author":"MD Shermis","year":"2015","unstructured":"Shermis, M. D. (2015). Contrasting state-of-the-art in the machine scoring of short-form constructed responses. Educational Assessment, 20(1), 46\u201365. https:\/\/doi.org\/10.1080\/10627197.2015.997617","journal-title":"Educational Assessment"},{"issue":"8","key":"11890_CR35","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","volume":"9","author":"J Schmidhuber","year":"1997","unstructured":"Schmidhuber, J., & Hochreiter, S. (1997). Long short-term memory. Neural Computation, 9(8), 1735\u20131780.","journal-title":"Neural Computation"},{"key":"11890_CR36","doi-asserted-by":"crossref","unstructured":"Sung, C., Dhamecha, T., Saha, S., Ma, T., Reddy, V., & Arora, R. (2019). Pre-training Bert on domain resources for short answer grading. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 6073\u20136077)","DOI":"10.18653\/v1\/D19-1628"},{"key":"11890_CR37","doi-asserted-by":"publisher","first-page":"264","DOI":"10.1016\/j.cogsys.2019.09.025","volume":"59","author":"O Sychev","year":"2020","unstructured":"Sychev, O., Anikin, A., & Prokudin, A. (2020). Automatic grading and hinting in open-ended text questions. Cognitive Systems Research, 59, 264\u2013272.","journal-title":"Cognitive Systems Research"},{"key":"11890_CR38","doi-asserted-by":"publisher","unstructured":"TF\u2013IDF BT - Encyclopedia of Machine Learning (pp. 986\u2013987). (2010). Springer, Boston, MA. https:\/\/doi.org\/10.1007\/978-0-387-30164-8_832","DOI":"10.1007\/978-0-387-30164-8_832"},{"issue":"2","key":"11890_CR39","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1016\/j.eij.2018.11.001","volume":"20","author":"TS Walia","year":"2019","unstructured":"Walia, T. S., Josan, G. S., & Singh, A. (2019). An efficient automated answer scoring system for Punjabi language. Egyptian Informatics Journal, 20(2), 89\u201396.","journal-title":"Egyptian Informatics Journal"},{"issue":"3","key":"11890_CR40","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1109\/MCI.2018.2840738","volume":"13","author":"T Young","year":"2018","unstructured":"Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018). Recent trends in deep learning based natural language processing. IEEE Computational Intelligence Magazine, 13(3), 55\u201375.","journal-title":"IEEE Computational Intelligence Magazine"},{"key":"11890_CR41","unstructured":"Ziai, R., Ott, N., & Meurers, D. (2012). Short answer assessment: Establishing links between research strands. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP (pp. 190\u2013200)."},{"key":"11890_CR42","doi-asserted-by":"crossref","unstructured":"Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015). Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. Preprint retrieved from http:\/\/arxiv.org\/abs\/1506.06724","DOI":"10.1109\/ICCV.2015.11"}],"container-title":["Education and Information Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10639-023-11890-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10639-023-11890-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10639-023-11890-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,8]],"date-time":"2024-03-08T14:11:41Z","timestamp":1709907101000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10639-023-11890-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,10]]},"references-count":42,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,3]]}},"alternative-id":["11890"],"URL":"https:\/\/doi.org\/10.1007\/s10639-023-11890-7","relation":{},"ISSN":["1360-2357","1573-7608"],"issn-type":[{"value":"1360-2357","type":"print"},{"value":"1573-7608","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,10]]},"assertion":[{"value":"10 February 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 May 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 July 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no relevant financial or non-financial interests to disclose.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}