{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T16:00:11Z","timestamp":1782316811528,"version":"3.54.5"},"reference-count":80,"publisher":"Springer Science and Business Media LLC","issue":"10","license":[{"start":{"date-parts":[[2023,5,19]],"date-time":"2023-05-19T00:00:00Z","timestamp":1684454400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,5,19]],"date-time":"2023-05-19T00:00:00Z","timestamp":1684454400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100016112","name":"Universit\u00e0 di Foggia","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100016112","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Knowl Inf Syst"],"published-print":{"date-parts":[[2023,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Automatic short answer grading (ASAG), a hot field of natural language understanding, is a research area within learning analytics. ASAG solutions are conceived to offload teachers and instructors, especially those in higher education, where classes with hundreds of students are the norm and the task of grading (short)answers to open-ended questionnaires becomes tougher. Their outcomes are precious both for the very grading and for providing students with \u201c<jats:italic>ad hoc<\/jats:italic>\u201d feedback. ASAG proposals have also enabled different intelligent tutoring systems. Over the years, a variety of ASAG solutions have been proposed, still there are a series of gaps in the literature that we fill in this paper. The present work proposes <jats:italic>GradeAid<\/jats:italic>, a framework for ASAG. It is based on the joint analysis of lexical and semantic features of the students\u2019 answers through state-of-the-art regressors; differently from any other previous work, <jats:italic>(i)<\/jats:italic> it copes with non-English datasets, <jats:italic>(ii)<\/jats:italic> it has undergone a robust validation and benchmarking phase, and <jats:italic>(iii)<\/jats:italic> it has been tested on every dataset publicly available and on a new dataset (now available for researchers). <jats:italic>GradeAid<\/jats:italic> obtains performance comparable to the systems presented in the literature (root-mean-squared errors down to 0.25 based on the specific tuple <jats:inline-formula><jats:alternatives><jats:tex-math>$$\\langle $$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mo>\u27e8<\/mml:mo>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula>dataset-question<jats:inline-formula><jats:alternatives><jats:tex-math>$$\\rangle $$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mo>\u27e9<\/mml:mo>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula>). We argue it represents a strong baseline for further developments in the field.<\/jats:p>","DOI":"10.1007\/s10115-023-01892-9","type":"journal-article","created":{"date-parts":[[2023,5,19]],"date-time":"2023-05-19T12:02:36Z","timestamp":1684497756000},"page":"4295-4334","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":29,"title":["GradeAid: a framework for automatic short answers grading in educational contexts\u2014design, implementation and evaluation"],"prefix":"10.1007","volume":"65","author":[{"given":"Emiliano","family":"del\u00a0Gobbo","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alfonso","family":"Guarino","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Barbara","family":"Cafarelli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luca","family":"Grilli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,5,19]]},"reference":[{"key":"1892_CR1","unstructured":"Rodriguez CO (2012) Moocs and the AI-stanford like courses: two successful and distinct course formats for massive open online courses. Eur J Open Distance E-Learning"},{"issue":"3","key":"1892_CR2","doi-asserted-by":"publisher","first-page":"16","DOI":"10.5539\/hes.v10n3p16","volume":"10","author":"W Ali","year":"2020","unstructured":"Ali W (2020) Online and remote learning in higher education institutes: a necessity in light of covid-19 pandemic. High Educ Stud 10(3):16\u201325","journal-title":"High Educ Stud"},{"issue":"1","key":"1892_CR3","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1007\/s11125-020-09464-3","volume":"49","author":"J Daniel","year":"2020","unstructured":"Daniel J (2020) Education and the covid-19 pandemic. Prospects 49(1):91\u201396","journal-title":"Prospects"},{"key":"1892_CR4","unstructured":"Hodges CB, Moore S, Lockee BB, Trust T, Bond MA (2020) The difference between emergency remote teaching and online learning"},{"issue":"3","key":"1892_CR5","doi-asserted-by":"publisher","first-page":"492","DOI":"10.1080\/13523260.2020.1761749","volume":"41","author":"MP Murphy","year":"2020","unstructured":"Murphy MP (2020) Covid-19 and emergency elearning: consequences of the securitization of higher education for post-pandemic pedagogy. Contemp Secur Policy 41(3):492\u2013505","journal-title":"Contemp Secur Policy"},{"issue":"2\u20133","key":"1892_CR6","doi-asserted-by":"publisher","first-page":"355","DOI":"10.1016\/0014-2921(93)90024-5","volume":"37","author":"C Prendergast","year":"1993","unstructured":"Prendergast C, Topel R (1993) Discretion and bias in performance evaluation. Eur Econ Rev 37(2\u20133):355\u2013365","journal-title":"Eur Econ Rev"},{"issue":"1","key":"1892_CR7","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1007\/s40593-014-0026-8","volume":"25","author":"S Burrows","year":"2015","unstructured":"Burrows S, Gurevych I, Stein B (2015) The eras and trends of automatic short answer grading. Int J Artif Intell Educ 25(1):60\u2013117. https:\/\/doi.org\/10.1007\/s40593-014-0026-8","journal-title":"Int J Artif Intell Educ"},{"key":"1892_CR8","doi-asserted-by":"crossref","unstructured":"Kumar S, Chakrabarti S, Roy S (2017) Earth mover\u2019s distance pooling over siamese lstms for automatic short answer grading. In: IJCAI, pp 2046\u20132052","DOI":"10.24963\/ijcai.2017\/284"},{"issue":"1","key":"1892_CR9","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1109\/TLT.2019.2897997","volume":"13","author":"A Sahu","year":"2019","unstructured":"Sahu A, Bhowmick PK (2019) Feature engineering and ensemble-based approach for improving automatic short-answer grading performance. IEEE Trans Learn Technol 13(1):77\u201390","journal-title":"IEEE Trans Learn Technol"},{"key":"1892_CR10","first-page":"25","volume":"4","author":"T Pedersen","year":"2004","unstructured":"Pedersen T, Patwardhan S, Michelizzi J et al (2004) Wordnet: similarity-measuring the relatedness of concepts. AAAI 4:25\u201329","journal-title":"AAAI"},{"key":"1892_CR11","doi-asserted-by":"crossref","unstructured":"Mohler M, Mihalcea R (2009) Text-to-text semantic similarity for automatic short answer grading. In: Proceedings of the 12th conference of the European chapter of the ACL (EACL 2009), pp 567\u2013575","DOI":"10.3115\/1609067.1609130"},{"key":"1892_CR12","unstructured":"Mohler M, Bunescu R, Mihalcea R (2011) Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. In: Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pp 752\u2013762"},{"issue":"1","key":"1892_CR13","doi-asserted-by":"publisher","first-page":"177","DOI":"10.1080\/10494820.2019.1648300","volume":"30","author":"L Zhang","year":"2022","unstructured":"Zhang L, Huang Y, Yang X, Yu S, Zhuang F (2022) An automatic short-answer grading model for semi-open-ended questions. Interact Learn Environ 30(1):177\u2013190. https:\/\/doi.org\/10.1080\/10494820.2019.1648300","journal-title":"Interact Learn Environ"},{"key":"1892_CR14","doi-asserted-by":"publisher","unstructured":"Prabhudesai A, Duong TNB (2019) Automatic short answer grading using siamese bidirectional LSTM based regression. In: TALE 2019\u20132019 IEEE international conference on engineering, technology and education. https:\/\/doi.org\/10.1109\/TALE48000.2019.9226026","DOI":"10.1109\/TALE48000.2019.9226026"},{"key":"1892_CR15","doi-asserted-by":"crossref","unstructured":"Galhardi LB, Brancher JD (2018) Machine learning approach for automatic short answer grading: a systematic review. In: Ibero-American conference on artificial intelligence, pp 380\u2013391. Springer","DOI":"10.1007\/978-3-030-03928-8_31"},{"key":"1892_CR16","doi-asserted-by":"publisher","first-page":"208","DOI":"10.7717\/peerj-cs.208","volume":"5","author":"MA Hussein","year":"2019","unstructured":"Hussein MA, Hassan H, Nassef M (2019) Automated language essay scoring systems: a literature review. PeerJ Comput Sci 5:208","journal-title":"PeerJ Comput Sci"},{"key":"1892_CR17","doi-asserted-by":"publisher","unstructured":"Prasain B, Bajaj SK (2020) Analysis of algorithms in automated marking in education: a proposed hybrid algorithm. In: CITISIA 2020\u2014IEEE conference on innovative technologies in intelligent systems and industrial applications, proceedings. https:\/\/doi.org\/10.1109\/CITISIA50690.2020.9371783","DOI":"10.1109\/CITISIA50690.2020.9371783"},{"key":"1892_CR18","doi-asserted-by":"publisher","first-page":"264","DOI":"10.1016\/j.cogsys.2019.09.025","volume":"59","author":"O Sychev","year":"2020","unstructured":"Sychev O, Anikin A, Prokudin A (2020) Automatic grading and hinting in open-ended text questions. Cognit Syst Res 59:264\u2013272. https:\/\/doi.org\/10.1016\/j.cogsys.2019.09.025","journal-title":"Cognit Syst Res"},{"key":"1892_CR19","doi-asserted-by":"publisher","unstructured":"Ge S, Chen X (2020) The application of deep learning in automated essay evaluation vol 11984 LNCS, pp 310\u2013318. https:\/\/doi.org\/10.1007\/978-3-030-38778-5_34","DOI":"10.1007\/978-3-030-38778-5_34"},{"key":"1892_CR20","unstructured":"Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering"},{"issue":"1","key":"1892_CR21","doi-asserted-by":"publisher","first-page":"13","DOI":"10.2196\/jmir.5876","volume":"19","author":"A Roehrs","year":"2017","unstructured":"Roehrs A, Da Costa CA, da Rosa Righi R, De Oliveira KSF (2017) Personal health records: a systematic literature review. J Med Internet Res 19(1):13","journal-title":"J Med Internet Res"},{"key":"1892_CR22","unstructured":"de Clercq O, van Hoecke S (2020) An exploratory study into automated pr\u00e9cis grading. In: LREC 2020\u201412th international conference on language resources and evaluation, conference proceedings, pp 397\u2013404"},{"key":"1892_CR23","doi-asserted-by":"crossref","unstructured":"Tay Y, Phan MC, Tuan LA, Hui SC (2018) SKIPFLOW: incorporating neural coherence features for end-to-end automatic text scoring. In: 32nd AAAI conference on artificial intelligence, AAAI 2018, pp 5948\u20135955","DOI":"10.1609\/aaai.v32i1.12045"},{"key":"1892_CR24","doi-asserted-by":"publisher","unstructured":"Cai C (2019) Automatic essay scoring with recurrent neural network. In: ACM international conference proceeding series, pp 1\u20137. https:\/\/doi.org\/10.1145\/3318265.3318296","DOI":"10.1145\/3318265.3318296"},{"key":"1892_CR25","doi-asserted-by":"publisher","unstructured":"Chen Z, Zhou Y (2019) Research on automatic essay scoring of composition based on CNN and OR. In: 2019 2nd international conference on artificial intelligence and big data, ICAIBD 2019, pp 13\u201318. https:\/\/doi.org\/10.1109\/ICAIBD.2019.8837007","DOI":"10.1109\/ICAIBD.2019.8837007"},{"key":"1892_CR26","doi-asserted-by":"publisher","unstructured":"Chimingyang H (2020) An automatic system for essay questions scoring based on LSTM and word embedding. In: Proceedings\u20142020 5th international conference on information science, computer technology and transportation, ISCTT 2020, pp 355\u2013364. https:\/\/doi.org\/10.1109\/ISCTT51595.2020.00068","DOI":"10.1109\/ISCTT51595.2020.00068"},{"key":"1892_CR27","doi-asserted-by":"publisher","unstructured":"Wiratmo A, Fatichah C (2020) Assessment of Indonesian short essay using transfer learning siamese dependency tree-LSTM. In: ICICoS 2020\u2014proceeding: 4th international conference on informatics and computational sciences. https:\/\/doi.org\/10.1109\/ICICoS51170.2020.9299044","DOI":"10.1109\/ICICoS51170.2020.9299044"},{"key":"1892_CR28","doi-asserted-by":"publisher","unstructured":"Hussein MA, Hassan HA, Nassef M (2020) A trait-based deep learning automated essay scoring system with adaptive feedback. Int J Adv Comput Sci Appl 11(5):287\u2013293. https:\/\/doi.org\/10.14569\/IJACSA.2020.0110538","DOI":"10.14569\/IJACSA.2020.0110538"},{"issue":"7","key":"1892_CR29","doi-asserted-by":"publisher","first-page":"1227","DOI":"10.1177\/0735633120927486","volume":"58","author":"GG Smith","year":"2020","unstructured":"Smith GG, Haworth R, \u017ditnik S (2020) Computer science meets education: natural language processing for automatic grading of open-ended questions in eBooks. J Educ Comput Res 58(7):1227\u20131255. https:\/\/doi.org\/10.1177\/0735633120927486","journal-title":"J Educ Comput Res"},{"key":"1892_CR30","doi-asserted-by":"publisher","unstructured":"Wang Z, Liu J, Dong R (2018) Intelligent auto-grading system. In: Proceedings of 2018 5th IEEE international conference on cloud computing and intelligence systems (CCIS), pp 430\u2013435. IEEE, 345 E 47th st, New York, NY 10017 USA. https:\/\/doi.org\/10.1109\/CCIS.2018.8691244. IEEE; IEEE Beijing Sect; Chinese Assoc Artificial Intelligence; Nanjing Univ Posts & Telecommunicat; Shanghai Univ; Jiangsu Engineering Lab Big Data Anal and Control Active Distribut Network; Nanjing Univ Sci and Technol; Swinburne Univ Technol; Shanghai K","DOI":"10.1109\/CCIS.2018.8691244"},{"key":"1892_CR31","doi-asserted-by":"crossref","unstructured":"Yang Z, Yang D, Dyer C, He X, Smola A, Hovy E (2016) Hierarchical attention networks for document classification. In: Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pp 1480\u20131489","DOI":"10.18653\/v1\/N16-1174"},{"key":"1892_CR32","doi-asserted-by":"crossref","unstructured":"Lun J, Zhu J, Tang Y, Yang M (2020) Multiple data augmentation strategies for improving performance on automatic short answer scoring. In: AAAI 2020\u201434th AAAI conference on artificial intelligence, pp 13446\u201313453","DOI":"10.1609\/aaai.v34i09.7062"},{"key":"1892_CR33","doi-asserted-by":"crossref","unstructured":"Ghavidel HA, Zouaq A, Desmarais MC (2020) Using BERT and XLNET for the automatic short answer grading task. In: CSEDU 2020\u2014proceedings of the 12th international conference on computer supported education, vol 1, pp 58\u201367","DOI":"10.5220\/0009422400580067"},{"key":"1892_CR34","doi-asserted-by":"crossref","unstructured":"Sung C, Ma T, Dhamecha TI, Reddy V, Saha S, Arora R (2020) Pre-training BERT on domain resources for short answer grading. In: EMNLP-IJCNLP 2019\u20142019 conference on empirical methods in natural language processing and 9th international joint conference on natural language processing, proceedings of the conference, pp 6071\u20136075","DOI":"10.18653\/v1\/D19-1628"},{"key":"1892_CR35","doi-asserted-by":"publisher","DOI":"10.1080\/10494820.2020.1855207","author":"H Tan","year":"2020","unstructured":"Tan H, Wang C, Duan Q, Lu Y, Zhang H, Li R (2020) Automatic short answer grading by encoding student responses via a graph convolutional network. Interact Learn Environ. https:\/\/doi.org\/10.1080\/10494820.2020.1855207","journal-title":"Interact Learn Environ"},{"key":"1892_CR36","doi-asserted-by":"publisher","unstructured":"Hassan S, Fahmy AA, El-Ramly M (2018) Automatic short answer scoring based on paragraph embeddings. Int J Adv Comput Sci Appl 9(10):397\u2013402. https:\/\/doi.org\/10.14569\/IJACSA.2018.091048","DOI":"10.14569\/IJACSA.2018.091048"},{"issue":"1","key":"1892_CR37","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1109\/TLT.2019.2897997","volume":"13","author":"A Sahu","year":"2020","unstructured":"Sahu A, Bhowmick PK (2020) Feature engineering and ensemble-based approach for improving automatic short-answer grading performance. IEEE Trans Learn Technol 13(1):77\u201390. https:\/\/doi.org\/10.1109\/TLT.2019.2897997","journal-title":"IEEE Trans Learn Technol"},{"key":"1892_CR38","doi-asserted-by":"publisher","unstructured":"Gomaa WH, Fahmy AA (2020) Ans2vec: a scoring system for short answers, vol. 921, pp 586\u2013595. https:\/\/doi.org\/10.1007\/978-3-030-14118-9_59","DOI":"10.1007\/978-3-030-14118-9_59"},{"key":"1892_CR39","doi-asserted-by":"publisher","unstructured":"Beseiso M, Alzahrani S (2020) An empirical analysis of BERT embedding for automated essay scoring. Int J Adv Comput Sci Appl 11(10):204\u2013210. https:\/\/doi.org\/10.14569\/IJACSA.2020.0111027","DOI":"10.14569\/IJACSA.2020.0111027"},{"key":"1892_CR40","doi-asserted-by":"publisher","unstructured":"Chen S, Li L (2021) Incorporating question information to enhance the performance of automatic short answer grading. In: Qiu H, Zhang C, Fei Z, Qiu M, Kung S (eds) Knowledge science, engineering and management, PT III. Lecture notes in artificial intelligence, vol 12817, pp 124\u2013136. Springer AG, Gewerbestrasse 11, CHAM, CH-6330, SWITZERLAND. https:\/\/doi.org\/10.1007\/978-3-030-82153-1_11. Springer LNCS; Waseda Univ; N Amer Chinese Talents Assoc; Longxiang High Tech Grp Inc","DOI":"10.1007\/978-3-030-82153-1_11"},{"key":"1892_CR41","doi-asserted-by":"publisher","first-page":"19270","DOI":"10.1109\/ACCESS.2021.3054346","volume":"9","author":"CN Tulu","year":"2021","unstructured":"Tulu CN, Ozkaya O, Orhan U (2021) Automatic Short Answer Grading with SemSpace Sense Vectors and MaLSTM. IEEE Access 9:19270\u201319280. https:\/\/doi.org\/10.1109\/ACCESS.2021.3054346","journal-title":"IEEE Access"},{"key":"1892_CR42","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2021.115146","volume":"180","author":"U Orhan","year":"2021","unstructured":"Orhan U, Tulu CN (2021) A novel embedding approach to learn word vectors by weighting semantic relations: Semspace. Expert Syst Appl 180:115146","journal-title":"Expert Syst Appl"},{"key":"1892_CR43","doi-asserted-by":"crossref","unstructured":"Mueller J, Thyagarajan A (2016) Siamese recurrent architectures for learning sentence similarity. In: Proceedings of the AAAI conference on artificial intelligence, vol 30","DOI":"10.1609\/aaai.v30i1.10350"},{"key":"1892_CR44","doi-asserted-by":"publisher","unstructured":"Tashu TM, Horvath T (2018) Pair-wise: Automatic essay evaluation using Word Mover\u2019s distance. In: CSEDU 2018\u2014proceedings of the 10th international conference on computer supported education, vol 1, pp 59\u201366. https:\/\/doi.org\/10.5220\/0006679200590066","DOI":"10.5220\/0006679200590066"},{"key":"1892_CR45","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1007\/978-3-030-21151-6_5","volume":"1022","author":"TM Tashu","year":"2019","unstructured":"Tashu TM, Horv\u00e1th T (2019) A layered approach to automatic essay evaluation using word-embedding 1022:77\u201394. https:\/\/doi.org\/10.1007\/978-3-030-21151-6_5","journal-title":"A layered approach to automatic essay evaluation using word-embedding"},{"key":"1892_CR46","doi-asserted-by":"publisher","unstructured":"Tashu TM, Esclamado JP, Horvath T (2019) Intelligent on-line exam management and evaluation system vol 11528 LNCS, pp 105\u2013111. https:\/\/doi.org\/10.1007\/978-3-030-22244-4_14","DOI":"10.1007\/978-3-030-22244-4_14"},{"key":"1892_CR47","doi-asserted-by":"crossref","unstructured":"Gomaa WH, Fahmy AA (2019) Ans2vec: a scoring system for short answers. In: International conference on advanced machine learning technologies and applications. Springer, pp 586\u2013595","DOI":"10.1007\/978-3-030-14118-9_59"},{"key":"1892_CR48","doi-asserted-by":"publisher","unstructured":"S\u00fczen N, Gorban AN, Levesley J, Mirkes EM (2020) Automatic short answer grading and feedback using text mining methods. Procedia Computer Science 169(2019):726\u2013743. https:\/\/doi.org\/10.1016\/j.procs.2020.02.171. arXiv:1807.10543","DOI":"10.1016\/j.procs.2020.02.171"},{"key":"1892_CR49","doi-asserted-by":"crossref","unstructured":"Mathias S, Bhattacharyya P (2020) Can neural networks automatically score essay traits? In: Innovative use of NLP for building educational applications, pp 85\u201391. Assoc computational linguistics-ACL, 209 N eighth street, Stroudsburg, PA 18360 USA. Natl Board Med Examiners; ETS","DOI":"10.18653\/v1\/2020.bea-1.8"},{"key":"1892_CR50","doi-asserted-by":"publisher","unstructured":"Tashu TM (2020) Off-topic essay detection using C-BGRU siamese. In: Proceedings\u201414th IEEE international conference on semantic computing, ICSC 2020, pp 221\u2013225. https:\/\/doi.org\/10.1109\/ICSC.2020.00046","DOI":"10.1109\/ICSC.2020.00046"},{"issue":"3","key":"1892_CR51","doi-asserted-by":"publisher","first-page":"276","DOI":"10.11613\/BM.2012.031","volume":"22","author":"ML McHugh","year":"2012","unstructured":"McHugh ML (2012) Interrater reliability: the kappa statistic. Biochemia Medica 22(3):276\u2013282","journal-title":"Biochemia Medica"},{"key":"1892_CR52","doi-asserted-by":"crossref","unstructured":"Salloum S, Gaber T, Vadera S, Sharan K (2022) A systematic literature review on phishing email detection using natural language processing techniques. IEEE Access","DOI":"10.1109\/ACCESS.2022.3183083"},{"issue":"2","key":"1892_CR53","first-page":"139","volume":"7","author":"Y Kang","year":"2020","unstructured":"Kang Y, Cai Z, Tan C-W, Huang Q, Liu H (2020) Natural language processing (nlp) in management research: a literature review. J Manag Anal 7(2):139\u2013172","journal-title":"J Manag Anal"},{"issue":"1","key":"1892_CR54","doi-asserted-by":"publisher","first-page":"168","DOI":"10.1007\/s13278-022-00995-5","volume":"12","author":"M Nirav Shah","year":"2022","unstructured":"Nirav Shah M, Ganatra A (2022) A systematic literature review and existing challenges toward fake news detection models. Soc Netw Anal Min 12(1):168","journal-title":"Soc Netw Anal Min"},{"key":"1892_CR55","doi-asserted-by":"publisher","unstructured":"del Gobbo E, Guarino A, Cafarelli B, Grilli L, Limone P Automatic evaluation of open-ended questions for online learning. A systematic mapping. Studies in Educational Evaluation. Article currently in Press. Temporary https:\/\/doi.org\/10.1016\/j.stueduc.2023.101258","DOI":"10.1016\/j.stueduc.2023.101258"},{"key":"1892_CR56","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2020.106339","volume":"207","author":"Y Liu","year":"2020","unstructured":"Liu Y, Zhang X, Huang F, Zhou Z, Zhao Z, Li Z (2020) Visual question answering via combining inferential attention and semantic space mapping. Knowl-Based Syst 207:106339","journal-title":"Knowl-Based Syst"},{"issue":"2","key":"1892_CR57","doi-asserted-by":"publisher","first-page":"669","DOI":"10.1007\/s11205-020-02442-4","volume":"156","author":"E del Gobbo","year":"2021","unstructured":"del Gobbo E, Fontanella S, Sarra A, Fontanella L (2021) Emerging topics in Brexit debate on twitter around the deadlines. Soc Indic Res 156(2):669\u2013688. https:\/\/doi.org\/10.1007\/s11205-020-02442-4","journal-title":"Soc Indic Res"},{"key":"1892_CR58","doi-asserted-by":"crossref","unstructured":"Guarino A, Malandrino D, Zaccagnino R (2021) An automatic mechanism to provide privacy awareness and control over unwittingly dissemination of online private information. Comput Netw, 108614","DOI":"10.1016\/j.comnet.2021.108614"},{"issue":"24","key":"1892_CR59","doi-asserted-by":"publisher","first-page":"17569","DOI":"10.1007\/s00521-021-06343-6","volume":"33","author":"A Guarino","year":"2021","unstructured":"Guarino A, Lettieri N, Malandrino D, Zaccagnino R (2021) A machine learning-based approach to identify unlawful practices in online terms of service: analysis, implementation and evaluation. Neural Comput Appl 33(24):17569\u201317587","journal-title":"Neural Comput Appl"},{"key":"1892_CR60","unstructured":"Turian J, Ratinov L, Bengio Y (2010) Word representations: a simple and general method for semi-supervised learning. In: Proceedings of the 48th annual meeting of the association for computational linguistics. Association for Computational Linguistics, pp 384\u2013394"},{"key":"1892_CR61","volume-title":"1st International conference on learning representations, ICLR 2013, Scottsdale, Arizona, USA, May 2\u20134, 2013","author":"T Mikolov","year":"2013","unstructured":"Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. In: Bengio Y, LeCun Y (eds) 1st International conference on learning representations, ICLR 2013, Scottsdale, Arizona, USA, May 2\u20134, 2013. Workshop Track Proceedings"},{"key":"1892_CR62","unstructured":"Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems, pp 3111\u20133119"},{"key":"1892_CR63","doi-asserted-by":"crossref","unstructured":"Wang C, Miao Z, Lin Y, Gao J (2019) User and topic hybrid context embedding for finance-related text data mining. In: 2019 International conference on data mining workshops (ICDMW), pp 751\u2013760","DOI":"10.1109\/ICDMW.2019.00112"},{"key":"1892_CR64","doi-asserted-by":"crossref","unstructured":"Yang Z, Li L (2019) An online retrieval question answering system for featured snippets triggering. In: 2019 International conference on data mining workshops (ICDMW), pp 49\u201355. IEEE","DOI":"10.1109\/ICDMW.2019.00016"},{"key":"1892_CR65","unstructured":"Arora S, Liang Y, Ma T (2016) A simple but tough-to-beat baseline for sentence embeddings"},{"key":"1892_CR66","unstructured":"Devlin J, Chang M-W, Lee K, Toutanova K (2018) BERT: pre-training of deep bidirectional transformers for language understanding. arXiv. arXiv:1810.04805"},{"key":"1892_CR67","doi-asserted-by":"crossref","unstructured":"Reimers N, Gurevych I (2019) Sentence-BERT: sentence embeddings using siamese BERT-networks. arXiv arXiv:1908.10084","DOI":"10.18653\/v1\/D19-1410"},{"issue":"3\/4","key":"1892_CR68","doi-asserted-by":"publisher","first-page":"591","DOI":"10.2307\/2333709","volume":"52","author":"SS Shapiro","year":"1965","unstructured":"Shapiro SS, Wilk MB (1965) An analysis of variance test for normality (complete samples). Biometrika 52(3\/4):591\u2013611","journal-title":"Biometrika"},{"key":"1892_CR69","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511921803","volume-title":"Evaluating learning algorithms: a classification perspective","author":"N Japkowicz","year":"2011","unstructured":"Japkowicz N, Shah M (2011) Evaluating learning algorithms: a classification perspective. Cambridge University Press, Cambridge"},{"key":"1892_CR70","doi-asserted-by":"crossref","unstructured":"McKnight PE, Najab J (2010): Mann-whitney u test. The Corsini encyclopedia of psychology, 1","DOI":"10.1002\/9780470479216.corpsy0524"},{"key":"1892_CR71","doi-asserted-by":"crossref","unstructured":"Chi Z, Dong L, Wei F, Yang N, Singhal S, Wang W, Song X, Mao X-L, Huang H, Zhou M (2020) Infoxlm: an information-theoretic framework for cross-lingual language model pre-training. arXiv preprint arXiv:2007.07834","DOI":"10.18653\/v1\/2021.naacl-main.280"},{"key":"1892_CR72","doi-asserted-by":"crossref","unstructured":"Chi Z, Dong L, Zheng B, Huang S, Mao X-L, Huang H, Wei F (2021) Improving pretrained cross-lingual language models via self-labeled word alignment. arXiv preprint arXiv:2106.06381","DOI":"10.18653\/v1\/2021.acl-long.265"},{"key":"1892_CR73","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1109\/THFE2.1960.4503259","volume":"1","author":"JC Licklider","year":"1960","unstructured":"Licklider JC (1960) Man-computer symbiosis. IRE Trans Hum Factors Electron 1:4\u201311","journal-title":"IRE Trans Hum Factors Electron"},{"issue":"4","key":"1892_CR74","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1145\/3328485","volume":"26","author":"W Xu","year":"2019","unstructured":"Xu W (2019) Toward human-centered AI: a perspective from human-computer interaction. Interactions 26(4):42\u201346","journal-title":"Interactions"},{"key":"1892_CR75","doi-asserted-by":"crossref","unstructured":"Hohman F, Head A, Caruana R, DeLine R, Drucker SM (2019) Gamut: a design probe to understand how data scientists understand machine learning models. In: Proceedings of the 2019 CHI conference on human factors in computing systems, pp 1\u201313","DOI":"10.1145\/3290605.3300809"},{"issue":"4","key":"1892_CR76","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3419764","volume":"10","author":"B Shneiderman","year":"2020","unstructured":"Shneiderman B (2020) Bridging the gap between ethics and practice: guidelines for reliable, safe, and trustworthy human-centered AI systems. ACM Trans Interact Intell Syst 10(4):1\u201331","journal-title":"ACM Trans Interact Intell Syst"},{"issue":"6","key":"1892_CR77","doi-asserted-by":"publisher","first-page":"70","DOI":"10.1145\/3282486","volume":"62","author":"DS Weld","year":"2019","unstructured":"Weld DS, Bansal G (2019) The challenge of crafting intelligible intelligence. Commun ACM 62(6):70\u201379","journal-title":"Commun ACM"},{"key":"1892_CR78","doi-asserted-by":"crossref","unstructured":"Wenskovitch J, Zhou M, Collins C, Chang R, Dowling M, Endert A, Xu K (2020) Putting the \u201ci\u201d in interaction: interactive interfaces personalized to individuals. IEEE Comput Graphics Appl 40(3):73\u201382","DOI":"10.1109\/MCG.2020.2982465"},{"key":"1892_CR79","doi-asserted-by":"publisher","first-page":"177","DOI":"10.1007\/978-3-030-52237-7_15","volume-title":"Artificial intelligence in education","author":"A Filighera","year":"2020","unstructured":"Filighera A, Steuer T, Rensing C (2020) Fooling automatic short answer grading systems. In: Bittencourt II, Cukurova M, Muldner K, Luckin R, Mill\u00e1n E (eds) Artificial intelligence in education. Springer, Cham, pp 177\u2013190"},{"key":"1892_CR80","doi-asserted-by":"publisher","first-page":"102","DOI":"10.1007\/978-3-031-16290-9_8","volume-title":"Educating for a new future: making sense of technology-enhanced learning adoption","author":"RF Mello","year":"2022","unstructured":"Mello RF, Neto R, Fiorentino G, Alves G, Ar\u00eades V, Silva JVGF, Falc\u00e3o TP, Ga\u0161evi\u0107 D (2022) Enhancing instructors\u2019 capability to assess open-response using natural language processing and learning analytics. In: Hilliger I, Mu\u00f1oz-Merino PJ, De Laet T, Ortega-Arranz A, Farrell T (eds) Educating for a new future: making sense of technology-enhanced learning adoption. Springer, Cham, pp 102\u2013115"}],"container-title":["Knowledge and Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10115-023-01892-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10115-023-01892-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10115-023-01892-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,22]],"date-time":"2023-08-22T17:09:49Z","timestamp":1692724189000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10115-023-01892-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,19]]},"references-count":80,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2023,10]]}},"alternative-id":["1892"],"URL":"https:\/\/doi.org\/10.1007\/s10115-023-01892-9","relation":{},"ISSN":["0219-1377","0219-3116"],"issn-type":[{"value":"0219-1377","type":"print"},{"value":"0219-3116","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,19]]},"assertion":[{"value":"21 July 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 March 2023","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 April 2023","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 May 2023","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declaration"}},{"value":"The authors have no competing interests to declare that are relevant to the content of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}