{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,18]],"date-time":"2026-05-18T16:45:43Z","timestamp":1779122743392,"version":"3.51.4"},"reference-count":51,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2023,8,12]],"date-time":"2023-08-12T00:00:00Z","timestamp":1691798400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"MIUR \u201cFondo Departments of Excellence 2018\u20132022\u201d of the DII Department at the University of Brescia, Italy, EU H2020 project AIPlan4EU","award":["101016442"],"award-info":[{"award-number":["101016442"]}]},{"name":"MIUR \u201cFondo Departments of Excellence 2018\u20132022\u201d of the DII Department at the University of Brescia, Italy, EU H2020 project AIPlan4EU","award":["952215"],"award-info":[{"award-number":["952215"]}]},{"name":"EU ICT-48 2020 project TAILOR","award":["101016442"],"award-info":[{"award-number":["101016442"]}]},{"name":"EU ICT-48 2020 project TAILOR","award":["952215"],"award-info":[{"award-number":["952215"]}]},{"name":"IBM Power Systems Academic Initiative","award":["101016442"],"award-info":[{"award-number":["101016442"]}]},{"name":"IBM Power Systems Academic Initiative","award":["952215"],"award-info":[{"award-number":["952215"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>Artificial Intelligence and Natural Language Processing techniques can have a very significant impact on the e-learning sector, with the introduction of chatbots, automatic correctors, or scoring systems. However, integrating such technologies into the business environment in an effective way is not a trivial operation, and it not only requires realising a model with good predictive performance, but also it requires the following: (i) a proper study of the task, (ii) a data collection process, (iii) a real-world evaluation of its utility. Moreover, it is also very important to build an entire IT infrastructure that connects the AI system with the company database, with the human employees, the users, etc. In this work, we present a real-world system, based on the state-of-the-art BERT model, which implements an automatic scoring system for open-ended questions written in Italian. More specifically, these questions pertain to the workplace safety courses which every worker must attend by law, often via e-learning platforms such as the one offered by Mega Italia Media. This article describes how our system has been designed, evaluated, and finally deployed for commercial use with complete integration with the other services provided by the company.<\/jats:p>","DOI":"10.3390\/fi15080268","type":"journal-article","created":{"date-parts":[[2023,8,14]],"date-time":"2023-08-14T10:28:13Z","timestamp":1692008893000},"page":"268","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Real-World Implementation and Integration of an Automatic Scoring System for Workplace Safety Courses in Italian"],"prefix":"10.3390","volume":"15","author":[{"given":"Nicola","family":"Arici","sequence":"first","affiliation":[{"name":"Department of Information Engineering, University of Brescia, Via Branze 38, 25121 Brescia, Italy"},{"name":"Mega Italia Media, Via Roncadelle 70A, 25030 Castel Mella, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alfonso","family":"Gerevini","sequence":"additional","affiliation":[{"name":"Department of Information Engineering, University of Brescia, Via Branze 38, 25121 Brescia, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9843-7854","authenticated-orcid":false,"given":"Matteo","family":"Olivato","sequence":"additional","affiliation":[{"name":"Department of Information Engineering, University of Brescia, Via Branze 38, 25121 Brescia, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Luca","family":"Putelli","sequence":"additional","affiliation":[{"name":"Department of Information Engineering, University of Brescia, Via Branze 38, 25121 Brescia, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Luca","family":"Sigalini","sequence":"additional","affiliation":[{"name":"Mega Italia Media, Via Roncadelle 70A, 25030 Castel Mella, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7785-9492","authenticated-orcid":false,"given":"Ivan","family":"Serina","sequence":"additional","affiliation":[{"name":"Department of Information Engineering, University of Brescia, Via Branze 38, 25121 Brescia, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,8,12]]},"reference":[{"key":"ref_1","first-page":"457","article-title":"A BERT-Based Scoring System for Workplace Safety Courses in Italian","volume":"Volume 13796","author":"Dovier","year":"2022","journal-title":"Lecture Notes in Computer Science, Proceedings of the AIxIA 2022\u2014Advances in Artificial Intelligence\u2014XXIst International Conference of the Italian Association for Artificial Intelligence, AIxIA 2022, Udine, Italy, 28 November\u20132 December 2022"},{"key":"ref_2","first-page":"4171","article-title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","volume":"Volume 1","author":"Burstein","year":"2019","journal-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019"},{"key":"ref_3","unstructured":"Guyon, I., von Luxburg, U., Bengio, S., Wallach, H.M., Fergus, R., Vishwanathan, S.V.N., and Garnett, R. (2017, January 4\u20139). Attention is All you Need. Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA."},{"key":"ref_4","unstructured":"Carpuat, M., de Marneffe, M., and Ru\u00edz, I.V.M. (2022, January 10\u201315). On the Use of Bert for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, USA."},{"key":"ref_5","unstructured":"Inui, K., Jiang, J., Ng, V., and Wan, X. (2019, January 3\u20137). SciBERT: A Pretrained Language Model for Scientific Text. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China."},{"key":"ref_6","unstructured":"Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (2020, January 6\u201312). Unsupervised Data Augmentation for Consistency Training. Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, Virtual Event."},{"key":"ref_7","first-page":"349","article-title":"Deep Learning for Classification of Radiology Reports with a Hierarchical Schema","volume":"Volume 176","author":"Cristani","year":"2020","journal-title":"Procedia Computer Science, Proceedings of the 24th International Conference on Knowledge-Based and Intelligent Information & Engineering Systems KES-2020, Virtual Event, 16\u201318 September 2020"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1017\/S1351324916000383","article-title":"Natural language processing in mental health applications using non-clinical texts","volume":"23","author":"Calvo","year":"2017","journal-title":"Nat. Lang. Eng."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"170","DOI":"10.1016\/j.procs.2018.10.326","article-title":"Customer Perception Analysis Using Deep Learning and NLP","volume":"140","author":"Ramaswamy","year":"2018","journal-title":"Procedia Comput. Sci."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1023","DOI":"10.1108\/INTR-11-2020-0649","article-title":"Asymmetric effect of feature level sentiment on product rating: An application of bigram natural language processing (NLP) analysis","volume":"32","author":"Oh","year":"2022","journal-title":"Internet Res."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Zubani, M., Sigalini, L., Serina, I., Putelli, L., Gerevini, A.E., and Chiari, M. (2022). A Performance Comparison of Different Cloud-Based Natural Language Understanding Services for an Italian e-Learning Platform. Future Internet, 14.","DOI":"10.3390\/fi14020062"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Reddy Karri, S.P., and Santhosh Kumar, B. (2020, January 22\u201324). Deep Learning Techniques for Implementation of Chatbots. Proceedings of the 2020 International Conference on Computer Communication and Informatics (ICCCI), Coimbatore, India.","DOI":"10.1109\/ICCCI48352.2020.9104143"},{"key":"ref_13","first-page":"165","article-title":"Benchmarking Natural Language Understanding Services for Building Conversational Agents","volume":"Volume 714","author":"Marchi","year":"2019","journal-title":"Lecture Notes in Electrical Engineering, Proceedings of the Increasing Naturalness and Flexibility in Spoken Dialogue Interaction\u201410th International Workshop on Spoken Dialogue Systems, IWSDS 2019, Syracuse, Italy, 24\u201326 April 2019"},{"key":"ref_14","unstructured":"Kraus, S. (2019, January 10\u201316). Automated Essay Scoring: A Survey of the State of the Art. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China."},{"key":"ref_15","unstructured":"Haller, S., Aldea, A., Seifert, C., and Strisciuglio, N. (2022). Survey on Automated Short Answer Grading with Deep Learning: From Word Embeddings to Transformers. arXiv."},{"key":"ref_16","unstructured":"Lin, D., Matsumoto, Y., and Mihalcea, R. (2011, January 19\u201324). Learning to Grade Short Answer Questions using Semantic Similarity Measures and Dependency Graph Alignments. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, OR, USA."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"020042","DOI":"10.1063\/1.4976906","article-title":"Automatic short answer scoring using words overlapping methods","volume":"1818","author":"Pribadi","year":"2017","journal-title":"AIP Conf. Proc."},{"key":"ref_18","first-page":"11","article-title":"Short Answer Grading Using String Similarity And Corpus-Based Similarity","volume":"3","author":"Gomaa","year":"2012","journal-title":"Int. J. Adv. Comput. Sci. Appl."},{"key":"ref_19","unstructured":"Suzen, N., Gorban, A.N., Levesley, J., and Mirkes, E.M. (2018). Automatic Short Answer Grading and Feedback Using Text Mining Methods. arXiv, Available online: http:\/\/xxx.lanl.gov\/abs\/1807.10543."},{"key":"ref_20","unstructured":"Bengio, Y., and LeCun, Y. (2013, January 2\u20134). Efficient Estimation of Word Representations in Vector Space. Proceedings of the 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, AZ, USA. Workshop Track Proceedings."},{"key":"ref_21","unstructured":"Jin, P., Zhang, Y., Chen, X., and Xia, Y. (2016, January 9\u201315). Bag-of-embeddings for text classification. Proceedings of the IJCAI, New York, NY, USA."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1080\/19312458.2018.1455817","article-title":"More than bags of words: Sentiment analysis with word embeddings","volume":"12","author":"Rudkowsky","year":"2018","journal-title":"Commun. Methods Meas."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Galke, L., and Scherp, A. (2021). Bag-of-words vs. graph vs. sequence in text classification: Questioning the necessity of text-graphs and the surprising strength of a wide MLP. arXiv.","DOI":"10.18653\/v1\/2022.acl-long.279"},{"key":"ref_24","unstructured":"Inui, K., Jiang, J., Ng, V., and Wan, X. (2019, January 3\u20137). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Bao, W., Bao, W., Du, J., Yang, Y., and Zhao, X. (2018, January 15\u201318). Attentive Siamese LSTM Network for Semantic Textual Similarity Measure. Proceedings of the 2018 International Conference on Asian Language Processing (IALP), Bandung, Indonesia.","DOI":"10.1109\/IALP.2018.8629212"},{"key":"ref_26","unstructured":"Moschitti, A., Pang, B., and Daelemans, W. (2014, January 25\u201329). Glove: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, Doha, Qatar. A meeting of SIGDAT, a Special Interest Group of the ACL."},{"key":"ref_27","first-page":"10","article-title":"Automatic Short Answer Scoring based on Paragraph Embeddings","volume":"9","author":"Hassan","year":"2018","journal-title":"Int. J. Adv. Comput. Sci. Appl."},{"key":"ref_28","first-page":"586","article-title":"Ans2vec: A Scoring System for Short Answers","volume":"Volume 921","author":"Hassanien","year":"2019","journal-title":"Advances in Intelligent Systems and Computing, Proceedings of the International Conference on Advanced Machine Learning Technologies and Applications, AMLTA 2019, Cairo, Egypt, 28\u201330 March 2019"},{"key":"ref_29","unstructured":"Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., and Garnett, R. (2015, January 7\u201312). Skip-Thought Vectors. Proceedings of the Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, Montreal, QC, Canada."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-Term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Prabhudesai, A., and Duong, T.N.B. (2019, January 10\u201313). Automatic Short Answer Grading using Siamese Bidirectional LSTM Based Regression. Proceedings of the IEEE International Conference on Engineering, Technology and Education, TALE 2019, Yogyakarta, Indonesia.","DOI":"10.1109\/TALE48000.2019.9226026"},{"key":"ref_32","unstructured":"Sierra, C. (2017, January 19\u201325). Earth Mover\u2019s Distance Pooling over Siamese LSTMs for Automatic Short Answer Grading. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, VIC, Australia."},{"key":"ref_33","unstructured":"Inui, K., Jiang, J., Ng, V., and Wan, X. (2019, January 3\u20137). Pre-Training BERT on Domain Resources for Short Answer Grading. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China."},{"key":"ref_34","unstructured":"Lun, J., Zhu, J., Tang, Y., and Yang, M. (2020, January 7\u201312). Multiple Data Augmentation Strategies for Improving Performance on Automatic Short Answer Scoring. Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Del Gobbo, E., Guarino, A., Cafarelli, B., and Grilli, L. (2023). GradeAid: A framework for automatic short answers grading in educational contexts\u2014Design, implementation and evaluation. Knowl. Inf. Syst., 1\u201340.","DOI":"10.1007\/s10115-023-01892-9"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"762819","DOI":"10.3389\/fpsyg.2021.762819","article-title":"E-learning research trends in higher education in light of COVID-19: A bibliometric analysis","volume":"12","author":"Brika","year":"2022","journal-title":"Front. Psychol."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"103182","DOI":"10.1016\/j.scs.2021.103182","article-title":"COVID-19, internet, and mobility: The rise of telework, telehealth, e-learning, and e-shopping","volume":"74","author":"Mouratidis","year":"2021","journal-title":"Sustain. Cities Soc."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"2041222","DOI":"10.1080\/2331186X.2022.2041222","article-title":"Assessment of the effects of Covid-19 pandemic on the prospects of e-learning in higher learning institutions: The mediating role of academic innovativeness and technological growth","volume":"9","author":"Jibril","year":"2022","journal-title":"Cogent Educ."},{"key":"ref_39","unstructured":"Inui, K., Jiang, J., Ng, V., and Wan, X. (2019, January 3\u20137). How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China."},{"key":"ref_40","first-page":"2227","article-title":"Deep Contextualized Word Representations","volume":"Volume 1","author":"Walker","year":"2018","journal-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018"},{"key":"ref_41","unstructured":"Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv, Available online: http:\/\/xxx.lanl.gov\/abs\/1907.11692."},{"key":"ref_42","unstructured":"Clark, K., Luong, M., Le, Q.V., and Manning, C.D. (2020, January 26\u201330). ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. Proceedings of the 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. Available online: OpenReview.net."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1002\/pri.66","article-title":"The use and interpretation of the Friedman test in the analysis of ordinal-scale data in repeated measures designs","volume":"1","author":"Sheldon","year":"1996","journal-title":"Physiother. Res. Int."},{"key":"ref_44","unstructured":"Zhang, Z., Wu, Y., Zhao, H., Li, Z., Zhang, S., Zhou, X., and Zhou, X. (2020, January 7\u201312). Semantics-Aware BERT for Language Understanding. Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA."},{"key":"ref_45","first-page":"367","article-title":"Attention-Based Explanation in a Deep Learning Model For Classifying Radiology Reports","volume":"Volume 12721","author":"Tucker","year":"2021","journal-title":"Lecture Notes in Computer Science, Proceedings of the Artificial Intelligence in Medicine\u201419th International Conference on Artificial Intelligence in Medicine, AIME 2021, Virtual Event, 15\u201318 June 2021"},{"key":"ref_46","first-page":"16","article-title":"On the Behaviour of BERT\u2019s Attention for the Classification of Medical Reports","volume":"Volume 3277","author":"Musto","year":"2022","journal-title":"Proceedings of the 3rd Italian Workshop on Explainable Artificial Intelligence Co-Located with 21th International Conference of the Italian Association for Artificial Intelligence(AIxIA 2022)"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Serina, L., Putelli, L., Gerevini, A.E., and Serina, I. (2023). Synonyms, Antonyms and Factual Knowledge in BERT Heads. Future Internet, 15.","DOI":"10.3390\/fi15070230"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"e12432","DOI":"10.1111\/lnc3.12432","article-title":"Five sources of bias in natural language processing","volume":"15","author":"Hovy","year":"2021","journal-title":"Lang. Linguist. Compass"},{"key":"ref_49","first-page":"164","article-title":"Graphical Identification of Gender Bias in BERT with a Weakly Supervised Approach","volume":"Volume 3287","author":"Nozza","year":"2022","journal-title":"Proceedings of the Sixth Workshop on Natural Language for Artificial Intelligence (NL4AI 2022) Co-Located with 21th International Conference of the Italian Association for Artificial Intelligence (AI*IA 2022)"},{"key":"ref_50","first-page":"5356","article-title":"StereoSet: Measuring stereotypical bias in pretrained language models","volume":"Volume 1","author":"Zong","year":"2021","journal-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL\/IJCNLP 2021"},{"key":"ref_51","unstructured":"Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2023, August 08). Lmproving Language Understanding by Generative Pre-Training. Available online: https:\/\/openai.com\/research\/language-unsupervised."}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/15\/8\/268\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:32:08Z","timestamp":1760128328000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/15\/8\/268"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,12]]},"references-count":51,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2023,8]]}},"alternative-id":["fi15080268"],"URL":"https:\/\/doi.org\/10.3390\/fi15080268","relation":{},"ISSN":["1999-5903"],"issn-type":[{"value":"1999-5903","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,12]]}}}