{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,19]],"date-time":"2026-08-19T17:44:48Z","timestamp":1787161488331,"version":"build-2736575974"},"reference-count":23,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,3,19]],"date-time":"2021-03-19T00:00:00Z","timestamp":1616112000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,3,19]],"date-time":"2021-03-19T00:00:00Z","timestamp":1616112000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Smart Learn. Environ."],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>In this paper, we present a system for automatic evaluation of the quality of a question paper. Question paper plays a major role in educational assessment. The quality of a question paper is crucial to fulfilling the purpose of the assessment. In many education sectors, question papers are prepared manually. A prior analysis of a question paper might help in finding the errors in the question paper, and better achieving the goals of the assessment. In this experiment, we focus on higher education in the technical domain. First, we conducted a student survey to identify the key factors that affect the quality of a question paper. The top factors we identified are question relevance, question difficulty, and time requirement. We explored the strategies to handle these factors and implemented them. We employ various concepts and techniques for the implementation. The system finally assigns a numerical quality score against these factors. The system is evaluated using a set of question papers collected from various sources. The experimental results show that the proposed system is quite promising.<\/jats:p>","DOI":"10.1186\/s40561-021-00148-9","type":"journal-article","created":{"date-parts":[[2021,3,19]],"date-time":"2021-03-19T09:03:48Z","timestamp":1616144628000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Towards development of a system for automatic assessment of the quality of a question paper"],"prefix":"10.1186","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7275-7980","authenticated-orcid":false,"given":"Sujan Kumar","family":"Saha","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,3,19]]},"reference":[{"issue":"1","key":"148_CR1","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1002\/bmb.20362","volume":"38","author":"TR Anderson","year":"2010","unstructured":"Anderson, T. R., & Rogan, J. M. (2010). Bridging the educational research-teaching practice gap: Tools for evaluating the quality of assessment instruments. Biochemistry and Molecular Biology Education, 38(1), 51\u201357.","journal-title":"Biochemistry and Molecular Biology Education"},{"key":"148_CR2","first-page":"1125","volume-title":"Proceedings of COLING 2016","author":"J Araki","year":"2016","unstructured":"Araki, J., Rajagopal, D., Sankaranarayanan, S., Holm, S., Yamakawa, Y., & Mitamura, T. (2016). Generating questions and multiple-choice answers using semantic analysis of texts. In Proceedings of COLING 2016, (pp. 1125\u20131136)."},{"key":"148_CR3","unstructured":"Blei, D. M., Ng, A. Y., Jordan, M. I., & Lafferty, J. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993\u20131022."},{"key":"148_CR4","unstructured":"Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain. David McKay Company."},{"key":"148_CR5","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1007\/s40593-014-0026-8","volume":"25","author":"S Burrows","year":"2015","unstructured":"Burrows, S., Gurevych, I., & Stein, B. (2015). The eras and trends of automatic short answer grading. International Journal of Artificial Intelligence in Education, 25, 60\u2013117.","journal-title":"International Journal of Artificial Intelligence in Education"},{"issue":"1","key":"148_CR6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1162\/COLI_a_00206","volume":"41","author":"Y Chali","year":"2015","unstructured":"Chali, Y., & Hasan, S. A. (2015). Towards topic-to-question generation. Computational Linguistics, 41(1), 1\u201320.","journal-title":"Computational Linguistics"},{"issue":"3","key":"148_CR7","first-page":"273","volume":"20","author":"C Cortes","year":"1995","unstructured":"Cortes, C., & Vapnik, V. N. (1995). \u201cSupport-vector networks\u201d (PDF). Machine Learning., 20(3), 273\u2013297.","journal-title":"Machine Learning."},{"key":"148_CR8","doi-asserted-by":"crossref","unstructured":"Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309\u2013333.","DOI":"10.1207\/S15324818AME1503_5"},{"key":"148_CR9","doi-asserted-by":"crossref","unstructured":"Huang, Z., Liu, Q., Chen, E., Zhao, H., Gao, M., Wei, S., \u2026 Hu, G. (2017). Question difficulty prediction for READING problems in standard tests. In Proceedings of the thirty-first AAAI conference on artificial intelligence (AAAI-17), (pp. 1352\u20131359).","DOI":"10.1609\/aaai.v31i1.10740"},{"key":"148_CR10","doi-asserted-by":"publisher","unstructured":"Kurdi, G., Leo, J., Parsia, B., Sattler, U., & Al-Emari, S. (2019). A systematic review of automatic question generation for educational purposes. Journal of Artificial Intelligence in Education. https:\/\/doi.org\/10.1007\/s40593-019-00186-y.","DOI":"10.1007\/s40593-019-00186-y"},{"key":"148_CR11","first-page":"85","volume-title":"Proceedings of the 2013 conference on empirical methods in natural language processing","author":"J Liu","year":"2013","unstructured":"Liu, J., Wang, Q., Lin, C.-Y., & Hon, H.-W. (2013). Question difficulty estimation in community question answering services. In Proceedings of the 2013 conference on empirical methods in natural language processing, (pp. 85\u201390)."},{"key":"148_CR12","volume-title":"Algorithms for assessing the quality and difficulty of multiple choice exam questions","author":"S Luger","year":"2016","unstructured":"Luger Sarah. (2016) Algorithms for assessing the quality and difficulty of multiple choice exam questions. Ph.D. Thesis, School of Informatics, University of Edinburgh."},{"key":"148_CR13","volume-title":"Proceedings of the first AAAI conference on human computation and crowdsourcing (HCOMP-13)","author":"S Luger","year":"2013","unstructured":"Luger, S., & Bowles, J. (2013). Two methods for measuring question difficulty and discrimination in incomplete Crowdsourced data. In Proceedings of the first AAAI conference on human computation and crowdsourcing (HCOMP-13)."},{"key":"148_CR14","first-page":"640","volume-title":"Proceedings of mining intelligence and knowledge exploration (MIKE), volume 9468 of the series lecture notes in computer science","author":"SK Mishra","year":"2015","unstructured":"Mishra, S. K., Kumar, P., & Saha, S. K. (2015). A support vector machine based system for technical question classification. In Proceedings of mining intelligence and knowledge exploration (MIKE), volume 9468 of the series lecture notes in computer science, (pp. 640\u2013649)."},{"key":"148_CR15","first-page":"752","volume-title":"Proceedings of ACL-2011","author":"M Mohler","year":"2011","unstructured":"Mohler, M., Bunesc, R., & Mihalcea, R. (2011). Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. In Proceedings of ACL-2011, (pp. 752\u2013762)."},{"key":"148_CR16","volume-title":"Proceedings of the 17th European Conference on Machine Learning","author":"A Moschitti","year":"2006","unstructured":"Moschitti, A. (2006). Efficient convolution kernels for dependency and constituent syntactic trees. In Proceedings of the 17th European Conference on Machine Learning."},{"key":"148_CR17","first-page":"511","volume-title":"Proceedings of the International Conference Recent Advances in Natural Language Processing RANLP 2013","author":"A Narendra","year":"2013","unstructured":"Narendra, A., Agarwal, M., & Shah, R. (2013). Automatic cloze-questions generation. In Proceedings of the International Conference Recent Advances in Natural Language Processing RANLP 2013, (pp. 511\u2013515)."},{"issue":"3","key":"148_CR18","doi-asserted-by":"publisher","first-page":"342","DOI":"10.1007\/s40593-019-00180-4","volume":"29","author":"I Pandarova","year":"2019","unstructured":"Pandarova, I., Schmidt, T., Hartig, J., Boubekki, A., Jones, R. D., & Brefeld, U. (2019). Predicting the difficulty of exercise items for dynamic difficulty adaptation in adaptive language tutoring. International Journal of Artificial Intelligence in Education, 29(3), 342\u2013367.","journal-title":"International Journal of Artificial Intelligence in Education"},{"issue":"1","key":"148_CR19","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1109\/TLT.2018.2889100","volume":"13","author":"DCH Rao","year":"2020","unstructured":"Rao, D. C. H., & Saha, S. K. (2020). Automatic multiple choice question generation from text: A survey. IEEE Transaction on Learning Technologies., 13(1), 14\u201325.","journal-title":"IEEE Transaction on Learning Technologies."},{"key":"148_CR20","first-page":"1","volume-title":"Proceedings of the 2012 IEEE Frontiers in Education Conference","author":"LMR Santos","year":"2012","unstructured":"Santos, L. M. R., Fernandez, d. C., Pablo, J., Martin, R. G., Perez, E. V., & Perez, M. J. V. (2012). Automatic classification of question difficulty level: Teachers\u2019 estimation vs. students\u2019 perception. In Proceedings of the 2012 IEEE Frontiers in Education Conference, (pp. 1\u20135)."},{"issue":"1","key":"148_CR21","doi-asserted-by":"publisher","DOI":"10.1186\/s41039-017-0051-y","volume":"12","author":"Y Susanti","year":"2017","unstructured":"Susanti, Y., Tokunaga, T., Nishikawa, H., & Obari, H. (2017). Evaluation of automatically generated english vocabulary questions. Research & Practice in Technology Enhanced Learning, 12(1), Art. no. 11.","journal-title":"Research & Practice in Technology Enhanced Learning"},{"key":"148_CR22","doi-asserted-by":"publisher","unstructured":"Zhang, L., Huang, Y., Yang, X., Yu, S., & Zhuang, F. (2020). An automatic short-answer grading model for semi-open-ended questions. Interactive Learning Environments. https:\/\/doi.org\/10.1080\/10494820.2019.1648300.","DOI":"10.1080\/10494820.2019.1648300"},{"key":"148_CR23","doi-asserted-by":"publisher","unstructured":"Zhang, L., & VanLehn, K. (2016). How do machine-generated questions compare to human-generated questions? RPTEL, 11(7). https:\/\/doi.org\/10.1186\/s41039-016-0031-7.","DOI":"10.1186\/s41039-016-0031-7"}],"container-title":["Smart Learning Environments"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s40561-021-00148-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s40561-021-00148-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s40561-021-00148-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,21]],"date-time":"2022-12-21T19:01:24Z","timestamp":1671649284000},"score":1,"resource":{"primary":{"URL":"https:\/\/slejournal.springeropen.com\/articles\/10.1186\/s40561-021-00148-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,19]]},"references-count":23,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["148"],"URL":"https:\/\/doi.org\/10.1186\/s40561-021-00148-9","relation":{},"ISSN":["2196-7091"],"issn-type":[{"value":"2196-7091","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,19]]},"assertion":[{"value":"29 December 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 March 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 March 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that:\u25cf This article does not contain any studies with human participants or animals performed by any of the authors.\u25cf For this type of study formal consent is not required.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"The authors declare that they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"4"}}