{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T16:01:38Z","timestamp":1787241698055,"version":"3.56.0"},"reference-count":78,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,3,18]],"date-time":"2021-03-18T00:00:00Z","timestamp":1616025600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,3,18]],"date-time":"2021-03-18T00:00:00Z","timestamp":1616025600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["RPTEL"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Learning through the internet becomes popular that facilitates learners to learn anything, anytime, anywhere from the web resources. Assessment is most important in any learning system. An assessment system can find the self-learning gaps of learners and improve the progress of learning. The manual question generation takes much time and labor. Therefore, automatic question generation from learning resources is the primary task of an automated assessment system. This paper presents a survey of automatic question generation and assessment strategies from textual and pictorial learning resources. The purpose of this survey is to summarize the state-of-the-art techniques for generating questions and evaluating their answers automatically.<\/jats:p>","DOI":"10.1186\/s41039-021-00151-1","type":"journal-article","created":{"date-parts":[[2021,3,18]],"date-time":"2021-03-18T12:48:10Z","timestamp":1616071690000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":104,"title":["Automatic question generation and answer assessment: a survey"],"prefix":"10.1186","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8588-1913","authenticated-orcid":false,"given":"Bidyut","family":"Das","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mukta","family":"Majumder","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Santanu","family":"Phadikar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arif Ahmed","family":"Sekh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,3,18]]},"reference":[{"issue":"4","key":"151_CR1","doi-asserted-by":"publisher","first-page":"1245","DOI":"10.1016\/j.ipm.2019.02.018","volume":"56","author":"A. Abdi","year":"2019","unstructured":"Abdi, A., Shamsuddin, S.M., Hasan, S., Piran, J. (2019). Deep learning-based sentiment classification of evaluative text based on multi-feature fusion. Information Processing & Management, 56(4), 1245\u20131259.","journal-title":"Information Processing & Management"},{"key":"151_CR2","unstructured":"Agarwal, M. (2012). Cloze and open cloze question generation systems and their evaluation guidelines. Master\u2019s thesis. International Institute of Information Technology, (IIIT), Hyderabad, India."},{"key":"151_CR3","first-page":"56","volume-title":"Proceedings of the 6th Workshop on Innovative Use of NLP for Building Educational Applications","author":"M. Agarwal","year":"2011","unstructured":"Agarwal, M., & Mannem, P. (2011). Automatic gap-fill question generation from text books. In Proceedings of the 6th Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics, Portland, (pp. 56\u201364)."},{"key":"151_CR4","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1007\/978-3-642-14770-8_5","volume-title":"Proceedings of the 7th International Conference on Advances in Natural Language Processing","author":"I. Aldabe","year":"2010","unstructured":"Aldabe, I., & Maritxalar, M. (2010). Automatic distractor generation for domain specific texts. In Proceedings of the 7th International Conference on Advances in Natural Language Processing. Springer-Verlag, Berlin, (pp. 27\u201338)."},{"issue":"1","key":"151_CR5","doi-asserted-by":"publisher","first-page":"34","DOI":"10.18178\/ijiet.2018.8.1.1008","volume":"8","author":"N. Alruwais","year":"2018","unstructured":"Alruwais, N., Wills, G., Wald, M. (2018). Advantages and challenges of using e-assessment. International Journal of Information and Education Technology, 8(1), 34\u201337.","journal-title":"International Journal of Information and Education Technology"},{"key":"151_CR6","doi-asserted-by":"publisher","first-page":"307","DOI":"10.18653\/v1\/W18-6537","volume-title":"Proceedings of The 11th International Natural Language Generation Conference","author":"J. Amidei","year":"2018","unstructured":"Amidei, J., Piwek, P., Willis, A. (2018). Evaluation methodologies in automatic question generation 2013-2018. In Proceedings of The 11th International Natural Language Generation Conference. Association for Computational Linguistics, Tilburg University, (pp. 307\u2013317)."},{"key":"151_CR7","doi-asserted-by":"crossref","unstructured":"Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Lawrence Zitnick, C., Parikh, D. (2015). Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, (pp. 2425\u20132433).","DOI":"10.1109\/ICCV.2015.279"},{"key":"151_CR8","doi-asserted-by":"publisher","first-page":"733","DOI":"10.1007\/978-3-642-45062-4_104","volume-title":"Proceedings of the Pattern Recognition and Machine Intelligence","author":"A.S. Bhatia","year":"2013","unstructured":"Bhatia, A.S., Kirti, M., Saha, S.K. (2013). Automatic generation of multiple choice questions using wikipedia. In Proceedings of the Pattern Recognition and Machine Intelligence. Springer-Verlag, Berlin, (pp. 733\u2013738)."},{"issue":"3","key":"151_CR9","doi-asserted-by":"publisher","first-page":"354","DOI":"10.1177\/1073191112446655","volume":"19","author":"W.B. Bilker","year":"2012","unstructured":"Bilker, W.B., Hansen, J.A., Brensinger, C.M., Richard, J., Gur, R.E., Gur, R.C. (2012). Development of abbreviated nine-item forms of the Raven\u2019s standard progressive matrices test. Assessment, 19(3), 354\u2013369.","journal-title":"Assessment"},{"key":"151_CR10","doi-asserted-by":"publisher","first-page":"735","DOI":"10.1109\/CSSE.2008.623","volume-title":"2008 International Conference on Computer Science and Software Engineering","author":"L. Bin","year":"2008","unstructured":"Bin, L., Jun, L., Jian-Min, Y., Qiao-Ming, Z. (2008). Automated essay scoring using the KNN algorithm. In 2008 International Conference on Computer Science and Software Engineering, (Vol. 1. IEEE, Washington, DC, pp. 735\u2013738)."},{"issue":"2","key":"151_CR11","first-page":"3","volume":"1","author":"R.T. Boyd","year":"1988","unstructured":"Boyd, R.T. (1988). Improving your test-taking skills. Practical Assessment, Research & Evaluation, 1(2), 3.","journal-title":"Practical Assessment, Research & Evaluation"},{"key":"151_CR12","first-page":"819","volume-title":"Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing","author":"J.C. Brown","year":"2005","unstructured":"Brown, J.C., Frishkoff, G.A., Eskenazi, M. (2005). Automatic question generation for vocabulary assessment. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Vancouver, (pp. 819\u2013826)."},{"issue":"1","key":"151_CR13","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1007\/s40593-014-0026-8","volume":"25","author":"S. Burrows","year":"2015","unstructured":"Burrows, S., Gurevych, I., Stein, B. (2015). The eras and trends of automatic short answer grading. International Journal of Artificial Intelligence in Education, 25(1), 60\u2013117.","journal-title":"International Journal of Artificial Intelligence in Education"},{"key":"151_CR14","unstructured":"Carmichael, M., Reid, A., Karpicke, J.D. (2018). Assessing the impact of educational video on student engagement, critical thinking and learning: The Current State of Play, (pp. 1\u201321): A SAGE Whitepaper, Sage Publishing."},{"issue":"1","key":"151_CR15","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1109\/TLT.2018.2889100","volume":"13","author":"D.R. Ch","year":"2018","unstructured":"Ch, D.R., & Saha, S.K. (2018). Automatic multiple choice question generation from text: A survey. IEEE Transactions on Learning Technologies, 13(1), 14\u201325. https:\/\/doi.org\/10.1109\/TLT.2018.2889100.","journal-title":"IEEE Transactions on Learning Technologies"},{"key":"151_CR16","first-page":"1","volume-title":"Proceedings of the COLING\/ACL on Interactive Presentation Sessions","author":"C.-Y. Chen","year":"2006","unstructured":"Chen, C.-Y., Liou, H.-C., Chang, J.S. (2006). Fast\u2013an automatic generation system for grammar tests. In Proceedings of the COLING\/ACL on Interactive Presentation Sessions. Association for Computational Linguistics, Sydney, (pp. 1\u20134)."},{"key":"151_CR17","doi-asserted-by":"crossref","unstructured":"Chen, G., Yang, J., Hauff, C., Houben, G.-J. (2018). Learningq: A large-scale dataset for educational question generation. In Twelfth International AAAI Conference on Web and Social Media, (pp. 481\u2013490).","DOI":"10.1609\/icwsm.v12i1.14987"},{"key":"151_CR18","unstructured":"Clay, B. (2001). A short guide to writing effective test questions. Lawrence: Kansas Curriculum Center, University of Kansas. https:\/\/www.k-state.edu\/ksde\/alp\/resources\/Handout-Module6.pdf."},{"issue":"2-4","key":"151_CR19","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1558\/cj.v14i2-4.15-33","volume":"14","author":"D. Coniam","year":"1997","unstructured":"Coniam, D. (1997). A preliminary inquiry into using corpus word frequency data in the automatic generation of English language cloze tests. Calico Journal, 14(2-4), 15\u201333.","journal-title":"Calico Journal"},{"key":"151_CR20","doi-asserted-by":"publisher","first-page":"168","DOI":"10.1007\/978-3-642-28885-2_19","volume-title":"Computational Processing of the Portuguese Language","author":"R. Correia","year":"2012","unstructured":"Correia, R., Baptista, J., Eskenazi, M., Mamede, N. (2012). Automatic generation of cloze question stems. In Computational Processing of the Portuguese Language. Springer-Verlag, Berlin, (pp. 168\u2013178)."},{"issue":"1","key":"151_CR21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s41239-017-0060-3","volume":"14","author":"B. Das","year":"2017","unstructured":"Das, B., & Majumder, M. (2017). Factual open cloze question generation for assessment of learner\u2019s knowledge. International Journal of Educational Technology in Higher Education, 14(1), 1\u201312.","journal-title":"International Journal of Educational Technology in Higher Education"},{"issue":"6","key":"151_CR22","doi-asserted-by":"publisher","first-page":"1485","DOI":"10.1002\/cae.22163","volume":"27","author":"B. Das","year":"2019","unstructured":"Das, B., Majumder, M., Phadikar, S., Sekh, A.A. (2019). Automatic generation of fill-in-the-blank question with corpus-based distractors for e-assessment to enhance learning. Computer Applications in Engineering Education, 27(6), 1485\u20131495.","journal-title":"Computer Applications in Engineering Education"},{"issue":"2","key":"151_CR23","first-page":"95","volume":"11","author":"G. Deena","year":"2020","unstructured":"Deena, G., Raja, K., PK, N.B., Kannan, K. (2020). Developing the assessment questions automatically to determine the cognitive level of the E-learner using NLP techniques. International Journal of Service Science, Management, Engineering, and Technology (IJSSMET), 11(2), 95\u2013110.","journal-title":"International Journal of Service Science, Management, Engineering, and Technology (IJSSMET)"},{"key":"151_CR24","unstructured":"Dhokrat, A., Gite, H., Mahender, C.N. (2012). Assessment of answers: Online subjective examination. In Proceedings of the Workshop on Question Answering for Complex Domains, (pp. 47\u201356)."},{"issue":"9","key":"151_CR25","doi-asserted-by":"publisher","first-page":"1683","DOI":"10.18520\/cs\/v113\/i09\/1683-1691","volume":"113","author":"M. Divate","year":"2017","unstructured":"Divate, M., & Salgaonkar, A. (2017). Automatic question generation approaches and evaluation techniques. Current Science, 113(9), 1683\u20131691.","journal-title":"Current Science"},{"key":"151_CR26","doi-asserted-by":"publisher","first-page":"1342","DOI":"10.18653\/v1\/P17-1123","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"X. Du","year":"2017","unstructured":"Du, X., Shao, J., Cardie, C. (2017). Learning to Ask: Neural Question Generation for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, (pp. 1342\u20131352)."},{"issue":"4","key":"151_CR27","doi-asserted-by":"publisher","first-page":"63","DOI":"10.4018\/ijdet.2013100105","volume":"11","author":"B. Flanagan","year":"2013","unstructured":"Flanagan, B., Yin, C., Hirokawa, S., Hashimoto, K., Tabata, Y. (2013). An automated method to generate e-learning quizzes from online language learner writing. International Journal of Distance Education Technologies (IJDET), 11(4), 63\u201380.","journal-title":"International Journal of Distance Education Technologies (IJDET)"},{"key":"151_CR28","unstructured":"Gao, H., Mao, J., Zhou, J., Huang, Z., Wang, L., Xu, W. (2015). Are you talking to a machine? Dataset and methods for multilingual image question. In Advances in Neural Information Processing Systems, (pp. 2296\u20132304)."},{"key":"151_CR29","doi-asserted-by":"crossref","unstructured":"Gordon, D., Kembhavi, A., Rastegari, M., Redmon, J., Fox, D., Farhadi, A. (2018). Iqa: Visual question answering in interactive environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 4089\u20134098).","DOI":"10.1109\/CVPR.2018.00430"},{"key":"151_CR30","doi-asserted-by":"publisher","first-page":"192","DOI":"10.1109\/ICITISEE.2016.7803072","volume-title":"2016 1st International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE)","author":"U. Hasanah","year":"2016","unstructured":"Hasanah, U., Permanasari, A.E., Kusumawardani, S.S., Pribadi, F.S. (2016). A review of an information extraction technique approach for automatic short answer grading. In 2016 1st International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE). IEEE, Yogyakarta, (pp. 192\u2013196)."},{"key":"151_CR31","unstructured":"Heaton, J.B. (1990). Classroom testing."},{"key":"151_CR32","unstructured":"Hoshino, A., & Nakagawa, H. (2007). Assisting cloze test making with a web application. In Society for Information Technology & Teacher Education International Conference. Association for the Advancement of Computing in Education (AACE), Waynesville, NC USA, (pp. 2807\u20132814)."},{"key":"151_CR33","doi-asserted-by":"publisher","first-page":"358","DOI":"10.1109\/ICCITECHN.2010.5723884","volume-title":"2010 13th International Conference on Computer and Information Technology (ICCIT)","author":"M.M. Islam","year":"2010","unstructured":"Islam, M.M., & Hoque, A.L. (2010). Automated essay scoring using generalized latent semantic analysis. In 2010 13th International Conference on Computer and Information Technology (ICCIT). IEEE, Dhaka, (pp. 358\u2013363)."},{"key":"151_CR34","doi-asserted-by":"crossref","unstructured":"Jain, U., Zhang, Z., Schwing, A.G. (2017). Creativity: Generating diverse questions using variational autoencoders. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 6485\u20136494).","DOI":"10.1109\/CVPR.2017.575"},{"key":"151_CR35","unstructured":"Jauhar, S.K., Turney, P., Hovy, E. (2015). TabMCQ: A Dataset of General Knowledge Tables and Multiple-choice Questions. https:\/\/www.microsoft.com\/en-us\/research\/publication\/tabmcq-a-dataset-of-general-knowledge-tables-and-multiple-choice-questions\/."},{"key":"151_CR36","doi-asserted-by":"crossref","unstructured":"Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., Girshick, R. (2017). Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 2901\u20132910).","DOI":"10.1109\/CVPR.2017.215"},{"key":"151_CR37","doi-asserted-by":"crossref","unstructured":"Johnson, J., Karpathy, A., Fei-Fei, L. (2016). Densecap: Fully convolutional localization networks for dense captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 4565\u20134574).","DOI":"10.1109\/CVPR.2016.494"},{"key":"151_CR38","doi-asserted-by":"publisher","first-page":"1601","DOI":"10.18653\/v1\/P17-1147","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"M. Joshi","year":"2017","unstructured":"Joshi, M., Choi, E., Weld, D.S., Zettlemoyer, L. (2017). TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, (pp. 1601\u20131611)."},{"issue":"3","key":"151_CR39","first-page":"275","volume":"11","author":"T. Kakkonen","year":"2008","unstructured":"Kakkonen, T., Myller, N., Sutinen, E., Timonen, J. (2008). Comparison of dimension reduction methods for automated essay grading. Journal of Educational Technology & Society, 11(3), 275\u2013288.","journal-title":"Journal of Educational Technology & Society"},{"key":"151_CR40","unstructured":"Klein, T., & Nabi, M. (2019). Learning to Answer by Learning to Ask: Getting the Best of GPT-2 and BERT Worlds. ArXiv, abs\/1911.02365."},{"key":"151_CR41","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1162\/tacl_a_00023","volume":"6","author":"T. Ko\u010disky\u0300","year":"2018","unstructured":"Ko\u010disky\u0300, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K.M., Melis, G., Grefenstette, E. (2018). The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6, 317\u2013328.","journal-title":"Transactions of the Association for Computational Linguistics"},{"issue":"1","key":"151_CR42","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1007\/s40593-019-00186-y","volume":"30","author":"G. Kurdi","year":"2020","unstructured":"Kurdi, G., Leo, J., Parsia, B., Sattler, U., Al-Emari, S. (2020). A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education, 30(1), 121\u2013204.","journal-title":"International Journal of Artificial Intelligence in Education"},{"key":"151_CR43","first-page":"785","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","author":"G. Lai","year":"2017","unstructured":"Lai, G., Xie, Q., Liu, H., Yang, Y., Hovy, E. (2017). RACE: Large-scale ReAding Comprehension Dataset From Examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, (pp. 785\u2013794)."},{"key":"151_CR44","doi-asserted-by":"crossref","unstructured":"Le, N.-T., Kojiri, T., Pinkwart, N. (2014). Automatic question generation for educational applications\u2013the state of art. In Advanced Computational Methods for Knowledge Engineering, (pp. 325\u2013338).","DOI":"10.1007\/978-3-319-06569-4_24"},{"issue":"4","key":"151_CR45","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1023\/A:1025779619903","volume":"37","author":"C. Leacock","year":"2003","unstructured":"Leacock, C., & Chodorow, M. (2003). C-rater: Automated scoring of short-answer questions. Computers and the Humanities, 37(4), 389\u2013405.","journal-title":"Computers and the Humanities"},{"key":"151_CR46","doi-asserted-by":"crossref","unstructured":"Liang, C., Yang, X., Dave, N., Wham, D., Pursel, B., Giles, C.L. (2018). Distractor generation for multiple choice questions using learning to rank. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, (pp. 284\u2013290).","DOI":"10.18653\/v1\/W18-0533"},{"issue":"4","key":"151_CR47","first-page":"377","volume":"6","author":"M. Majumder","year":"2014","unstructured":"Majumder, M., & Saha, S.K. (2014). Automatic selection of informative sentences: The sentences that can generate multiple choice questions. Knowledge Management and E-Learning: An International Journal, 6(4), 377\u2013391.","journal-title":"Knowledge Management and E-Learning: An International Journal"},{"key":"151_CR48","doi-asserted-by":"publisher","first-page":"64","DOI":"10.18653\/v1\/W15-4410","volume-title":"Proceedings of the 2nd Workshop on Natural Language Processing Techniques for Educational Applications","author":"M. Majumder","year":"2015","unstructured":"Majumder, M., & Saha, S.K. (2015). A system for generating multiple choice questions: With a novel approach for sentence selection. In Proceedings of the 2nd Workshop on Natural Language Processing Techniques for Educational Applications. Association for Computational Linguistics, Beijing, (pp. 64\u201372)."},{"issue":"11","key":"151_CR49","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1145\/219717.219748","volume":"38","author":"G.A. Miller","year":"1995","unstructured":"Miller, G.A. (1995). WordNet: A lexical database for English. Communications of the ACM, 38(11), 39\u201341.","journal-title":"Communications of the ACM"},{"issue":"2","key":"151_CR50","doi-asserted-by":"publisher","first-page":"177","DOI":"10.1017\/S1351324906004177","volume":"12","author":"R. Mitkov","year":"2006","unstructured":"Mitkov, R., LE An, H., Karamanis, N. (2006). A computer-aided environment for generating multiple-choice test items. Natural Language Engineering, 12(2), 177\u2013194.","journal-title":"Natural Language Engineering"},{"key":"151_CR51","unstructured":"Mora, I.M., de la Puente, S.P., Nieto, X.G. (2016). Towards automatic generation of question answer pairs from images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, (pp. 1\u20132)."},{"key":"151_CR52","unstructured":"Mostafazadeh, N., Misra, I., Devlin, J., Mitchell, M., He, X., Vanderwende, L. (2016). Generating Natural Questions About an Image. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, (Volume 1: Long Papers), Berlin, Germany, (pp. 1802\u20131813)."},{"key":"151_CR53","first-page":"511","volume-title":"Proceedings of Recent Advances in Natural Language Processing","author":"A. Narendra","year":"2013","unstructured":"Narendra, A., Agarwal, M., Shah, R. (2013). Automatic cloze-questions generation. In Proceedings of Recent Advances in Natural Language Processing. INCOMA Ltd. Shoumen, BULGARIA (ACL 2013), Hissar, (pp. 511\u2013515)."},{"key":"151_CR54","doi-asserted-by":"crossref","unstructured":"Nassif, A.B., Elnagar, A., Shahin, I., Henno, S. (2020). Deep learning for arabic subjective sentiment analysis: Challenges and research opportunities. Applied Soft Computing, 106836.","DOI":"10.1016\/j.asoc.2020.106836"},{"key":"151_CR55","unstructured":"Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., Rosenberg, M., Song, X., Stoica, A., Tiwary, S., Wang, T. (2016). MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv, arXiv:1611.09268. https:\/\/ui.adsabs.harvard.edu\/abs\/2016arXiv161109268B."},{"issue":"1","key":"151_CR56","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1080\/03098770601167922","volume":"31","author":"D. Nicol","year":"2007","unstructured":"Nicol, D. (2007). E-assessment by design: Using multiple-choice tests to good effect. Journal of Further and higher Education, 31(1), 53\u201364.","journal-title":"Journal of Further and higher Education"},{"issue":"2","key":"151_CR57","doi-asserted-by":"publisher","first-page":"337","DOI":"10.1016\/j.compedu.2010.07.013","volume":"56","author":"F. Noorbehbahani","year":"2011","unstructured":"Noorbehbahani, F., & Kardan, A.A. (2011). The automatic assessment of free text answers using a modified BLEU algorithm. Computers & Education, 56(2), 337\u2013345.","journal-title":"Computers & Education"},{"key":"151_CR58","unstructured":"Papasalouros, A., Kanaris, K., Kotis, K. (2008). Automatic generation of multiple choice questions from domain ontologies. In Proceedings of the e-Learning, (pp. 427\u2013434)."},{"key":"151_CR59","first-page":"460","volume-title":"Proceedings of the 22nd International Florida Artificial Intelligence Research Society Conference(FLAIRS)","author":"J. Pino","year":"2009","unstructured":"Pino, J., & Eskenazi, M. (2009). Measuring hint level in open cloze questions. In Proceedings of the 22nd International Florida Artificial Intelligence Research Society Conference(FLAIRS). The AAAI Press, Florida, (pp. 460\u2013465)."},{"key":"151_CR60","first-page":"22","volume-title":"Proceedings of the Workshop on Intelligent Tutoring Systems for Ill-Defined Domains, 9th International Conference on Intelligent Tutoring Systems","author":"J. Pino","year":"2008","unstructured":"Pino, J., Heilman, M., Eskenazi, M. (2008). A selection strategy to improve cloze question quality. In Proceedings of the Workshop on Intelligent Tutoring Systems for Ill-Defined Domains, 9th International Conference on Intelligent Tutoring Systems. Springer, Montreal, (pp. 22\u201334)."},{"key":"151_CR61","doi-asserted-by":"publisher","first-page":"2383","DOI":"10.18653\/v1\/D16-1264","volume-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","author":"P. Rajpurkar","year":"2016","unstructured":"Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, (pp. 2383\u20132392)."},{"key":"151_CR62","doi-asserted-by":"crossref","unstructured":"Ramachandran, L., Cheng, J., Foltz, P. (2015). Identifying patterns for short answer scoring using graph-based lexico-semantic text matching. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, (pp. 97\u2013106).","DOI":"10.3115\/v1\/W15-0612"},{"key":"151_CR63","unstructured":"Ren, M., Kiros, R., Zemel, R. (2015). Exploring models and data for image question answering. In Advances in Neural Information Processing Systems, (pp. 2953\u20132961)."},{"key":"151_CR64","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1007\/978-3-319-27704-2_10","volume-title":"International Computer Assisted Assessment Conference","author":"S. Roy","year":"2015","unstructured":"Roy, S., Narahari, Y., Deshmukh, O.D. (2015). A perspective on computer assisted assessment techniques for short free-text answers. In International Computer Assisted Assessment Conference. Springer, Zeist, (pp. 96\u2013109)."},{"key":"151_CR65","doi-asserted-by":"publisher","first-page":"1479","DOI":"10.1109\/ITSIM.2010.5561484","volume-title":"2010 International Symposium on Information Technology","author":"D.S. Rozali","year":"2010","unstructured":"Rozali, D.S., Hassan, M.F., Zamin, N. (2010). A survey on adaptive qualitative assessment and dynamic questions generation approaches. In 2010 International Symposium on Information Technology, (Vol. 3. IEEE, Kuala Lumpur, pp. 1479\u20131484)."},{"issue":"2","key":"151_CR66","doi-asserted-by":"publisher","first-page":"177","DOI":"10.5087\/dad.2012.208","volume":"3","author":"V. Rus","year":"2012","unstructured":"Rus, V., Wyse, B., Piwek, P., Lintean, M., Stoyanchev, S., Moldovan, C. (2012). A detailed account of the first question generation shared task evaluation challenge. Dialogue & Discourse, 3(2), 177\u2013204.","journal-title":"Dialogue & Discourse"},{"key":"151_CR67","unstructured":"Sakaguchi, K., Heilman, M., Madnani, N. (2015). Effective feature integration for automated short answer scoring. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (pp. 1049\u20131054)."},{"key":"151_CR68","unstructured":"Santoro, A., Hill, F., Barrett, D., Morcos, A., Lillicrap, T. (2018). Measuring abstract reasoning in neural networks. In International Conference on Machine Learning, (pp. 4477\u20134486)."},{"key":"151_CR69","doi-asserted-by":"publisher","first-page":"588","DOI":"10.18653\/v1\/P16-1056","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"I.V. Serban","year":"2016","unstructured":"Serban, I.V., Garc\u00eda-Dur\u00e1n, A., Gulcehre, C., Ahn, S., Chandar, S., Courville, A., Bengio, Y. (2016). Generating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, (pp. 588\u2013598)."},{"key":"151_CR70","first-page":"44","volume":"30","author":"A.-M.S. Shaban","year":"2014","unstructured":"Shaban, A.-M.S. (2014). A comparison between objective and subjective tests. Journal of the College of Languages, 30, 44\u201352.","journal-title":"Journal of the College of Languages"},{"key":"151_CR71","doi-asserted-by":"crossref","unstructured":"Shermis, M.D., & Burstein, J. (2013). Handbook of automated essay evaluation: Current applications and new directions.","DOI":"10.4324\/9780203122761"},{"issue":"1","key":"151_CR72","doi-asserted-by":"publisher","first-page":"1193","DOI":"10.1146\/annurev.neuro.24.1.1193","volume":"24","author":"E.P. Simoncelli","year":"2001","unstructured":"Simoncelli, E.P., & Olshausen, B.A. (2001). Natural image statistics and neural representation. Annual Review of Neuroscience, 24(1), 1193\u20131216.","journal-title":"Annual Review of Neuroscience"},{"key":"151_CR73","doi-asserted-by":"publisher","first-page":"6418","DOI":"10.18653\/v1\/P19-1644","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"A. Suhr","year":"2019","unstructured":"Suhr, A., Zhou, S., Zhang, A., Zhang, I., Bai, H., Artzi, Y. (2019). A Corpus for Reasoning about Natural Language Grounded in Photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, (pp. 6418\u20136428)."},{"key":"151_CR74","doi-asserted-by":"publisher","first-page":"191","DOI":"10.18653\/v1\/W17-2623","volume-title":"Proceedings of the 2nd Workshop on Representation Learning for NLP","author":"A. Trischler","year":"2017","unstructured":"Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., Suleman, K. (2017). NewsQA: A Machine Comprehension Dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP. Association for Computational Linguistics, Vancouver, (pp. 191\u2013200)."},{"key":"151_CR75","doi-asserted-by":"crossref","unstructured":"Welbl, J., Liu, N.F., Gardner, M. (2017). Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, (pp. 94\u2013106).","DOI":"10.18653\/v1\/W17-4413"},{"key":"151_CR76","doi-asserted-by":"crossref","unstructured":"Yu, L., Park, E., Berg, A.C., Berg, T.L. (2015). Visual madlibs: Fill in the blank description generation and question answering. In Proceedings of the IEEE International Conference on Computer Vision, (pp. 2461\u20132469).","DOI":"10.1109\/ICCV.2015.283"},{"key":"151_CR77","first-page":"4235","volume-title":"Proceedings of the 26th International Joint Conference on Artificial Intelligence","author":"S. Zhang","year":"2017","unstructured":"Zhang, S., Qu, L., You, S., Yang, Z., Zhang, J. (2017). Automatic generation of grounded visual questions. In Proceedings of the 26th International Joint Conference on Artificial Intelligence. The AAAI Press, Melbourne, (pp. 4235\u20134243)."},{"key":"151_CR78","doi-asserted-by":"crossref","unstructured":"Zhu, Y., Groth, O., Bernstein, M., Fei-Fei, L. (2016). Visual7w: Grounded question answering in images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 4995\u20135004).","DOI":"10.1109\/CVPR.2016.540"}],"container-title":["Research and Practice in Technology Enhanced Learning"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s41039-021-00151-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s41039-021-00151-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s41039-021-00151-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,21]],"date-time":"2022-12-21T17:00:12Z","timestamp":1671642012000},"score":1,"resource":{"primary":{"URL":"https:\/\/telrp.springeropen.com\/articles\/10.1186\/s41039-021-00151-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,18]]},"references-count":78,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["151"],"URL":"https:\/\/doi.org\/10.1186\/s41039-021-00151-1","relation":{},"ISSN":["1793-7078"],"issn-type":[{"value":"1793-7078","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,18]]},"assertion":[{"value":"10 July 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 February 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 March 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This article does not contain any studies with human participants or animals performed by any of the authors.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}}],"article-number":"5"}}