{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T02:52:26Z","timestamp":1760151146890,"version":"build-2065373602"},"reference-count":49,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2022,2,18]],"date-time":"2022-02-18T00:00:00Z","timestamp":1645142400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>During the COVID-19 pandemic, the corporate online training sector has increased exponentially and online course providers had to implement innovative solutions to be more efficient and provide a satisfactory service. This paper considers a real case study in implementing a chatbot, which answers frequently asked questions from learners on an Italian e-learning platform that provides workplace safety courses to several business customers. Having to respond quickly to the increase in the courses activated, the company decided to develop a chatbot using a cloud-based service currently available on the market. These services are based on Natural Language Understanding (NLU) engines, which deal with identifying information such as entities and intentions from the sentences provided as input. To integrate a chatbot in an e-learning platform, we studied the performance of the intent recognition task of the major NLU platforms available on the market with an in-depth comparison, using an Italian dataset provided by the owner of the e-learning platform. We focused on intent recognition, carried out several experiments and evaluated performance in terms of F-score, error rate, response time, and robustness of all the services selected. The chatbot is currently in production, therefore we present a description of the system implemented and its results on the original users\u2019 requests.<\/jats:p>","DOI":"10.3390\/fi14020062","type":"journal-article","created":{"date-parts":[[2022,2,21]],"date-time":"2022-02-21T08:18:05Z","timestamp":1645431485000},"page":"62","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["A Performance Comparison of Different Cloud-Based Natural Language Understanding Services for an Italian e-Learning Platform"],"prefix":"10.3390","volume":"14","author":[{"given":"Matteo","family":"Zubani","sequence":"first","affiliation":[{"name":"Department of Information Engineering, University of Brescia, Via Branze 38, 25121 Brescia, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Luca","family":"Sigalini","sequence":"additional","affiliation":[{"name":"Mega Italia Media, Via Roncadelle 70A, 25030 Castel Mella, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7785-9492","authenticated-orcid":false,"given":"Ivan","family":"Serina","sequence":"additional","affiliation":[{"name":"Department of Information Engineering, University of Brescia, Via Branze 38, 25121 Brescia, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Luca","family":"Putelli","sequence":"additional","affiliation":[{"name":"Department of Information Engineering, University of Brescia, Via Branze 38, 25121 Brescia, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alfonso E.","family":"Gerevini","sequence":"additional","affiliation":[{"name":"Department of Information Engineering, University of Brescia, Via Branze 38, 25121 Brescia, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mattia","family":"Chiari","sequence":"additional","affiliation":[{"name":"Department of Information Engineering, University of Brescia, Via Branze 38, 25121 Brescia, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,2,18]]},"reference":[{"key":"ref_1","unstructured":"Mauldin, M.L. (2022, January 15). CHATTERBOTS, TINYMUDS, and the Turing Test: Entering the Loebner Prize Competition. Available online: https:\/\/www.aaai.org\/Papers\/AAAI\/1994\/AAAI94-003.pdf."},{"key":"ref_2","first-page":"2791","article-title":"A Review on Chatbot Design and Implementation Techniques","volume":"7","author":"Kumar","year":"2020","journal-title":"Int. J. Eng. Technol."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Caldarini, G., Jaf, S., and McGarry, K. (2022). A Literature Survey of Recent Advances in Chatbots. Information, 13.","DOI":"10.3390\/info13010041"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Turing, A.M. (2009). Computing machinery and intelligence. Parsing the Turing Test, Springer.","DOI":"10.1007\/978-1-4020-6710-5_3"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Brandtzaeg, P.B., and F\u00f8lstad, A. (2017). Why People Use Chatbots. Internet Science, Springer International Publishing.","DOI":"10.1007\/978-3-319-70284-1_30"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"e1434","DOI":"10.1002\/widm.1434","article-title":"A critical review of state-of-the-art chatbot designs and applications","volume":"12","author":"Luo","year":"2022","journal-title":"Wiley Interdiscip. Rev. Data Min. Knowl. Discov."},{"key":"ref_7","first-page":"100006","article-title":"Chatbots: History, technology, and applications","volume":"2","author":"Adamopoulou","year":"2020","journal-title":"Mach. Learn. Appl."},{"key":"ref_8","unstructured":"Di Prospero, A., Norouzi, N., Fokaefs, M., and Litoiu, M. (2017, January 6\u20138). Chatbots as assistants: An architectural framework. Proceedings of the 27th Annual International Conference on Computer Science and Software Engineering, Markham, ON, Canada."},{"key":"ref_9","unstructured":"Henderson, M. (2015, January 19\u201320). Machine Learning for Dialog State Tracking: A Review. Proceedings of the First International Workshop on Machine Learning in Spoken Language Processing, Fukushima, Japan."},{"key":"ref_10","first-page":"546","article-title":"A novel approach for ontology-driven information retrieving chatbot for fashion brands","volume":"10","author":"Nazir","year":"2019","journal-title":"Int. J. Adv. Comput. Sci. Appl. IJACSA"},{"key":"ref_11","unstructured":"Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., and Dean, J. (2013). Distributed Representations of Words and Phrases and their Compositionality. Advances in Neural Information Processing Systems 26, Curran Associates, Inc."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Prasomphan, S. (2019, January 12\u201315). Improvement of Chatbot in Trading System for SMEs by Using Deep Neural Network. Proceedings of the 2019 IEEE 4th International Conference on Cloud Computing and Big Data Analysis (ICCCBDA), Chengdu, China.","DOI":"10.1109\/ICCCBDA.2019.8725745"},{"key":"ref_13","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4\u20139). Attention is All you Need. Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA."},{"key":"ref_14","unstructured":"Adiwardana, D., Luong, M., So, D.R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., and Lu, Y. (2020). Towards a Human-like Open-Domain Chatbot. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Thorat, S.A., and Jadhav, V. (2020, January 21\u201323). A review on implementation issues of rule-based chatbot systems. Proceedings of the International Conference on Innovative Computing & Communications (ICICC), Delhi, India.","DOI":"10.2139\/ssrn.3567047"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Lokman, A.S., and Ameedeen, M.A. (2018, January 13\u201314). Modern Chatbot Systems: A Technical Review. Proceedings of the Future Technologies Conference (FTC) 2018, Vancouver, BC, Canada.","DOI":"10.1007\/978-3-030-02683-7_75"},{"key":"ref_17","unstructured":"Canonico, M., and Russis, L.D. (2018, January 18\u201322). A Comparison and Critique of Natural Language Understanding Tools. Proceedings of the CLOUD COMPUTING 2018: The Ninth International Conference on Cloud Computing, GRIDs, and Virtualization, Barcelona, Spain."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Leoni, C., Torre, I., and Vercelli, G. (2020, January 8\u201311). Conversiamo: Improving Italian question answering exploiting IBM watson services. Proceedings of the International Conference on Text, Speech, and Dialogue, Brno, Czech Republic.","DOI":"10.1007\/978-3-030-58323-1_54"},{"key":"ref_19","unstructured":"Tavanapour, N., and Bittner, E.A. (2018, January 13\u201316). Automated facilitation for idea platforms: Design and evaluation of a Chatbot prototype. Proceedings of the 39th International Conference on Information Systems, San Francisco, CA, USA."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Rosruen, N., and Samanchuen, T. (2018, January 12\u201314). Chatbot Utilization for Medical Consultant System. Proceedings of the 3rd Technology Innovation Management and Engineering Science International Conference (TIMES-iCON), Bangkok, Thailand.","DOI":"10.1109\/TIMES-iCON.2018.8621678"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Kadlec, R., Schmid, M., Bajgar, O., and Kleindienst, J. (2016, January 7\u201312). Text Understanding with the Attention Sum Reader Network. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Berlin, Germany.","DOI":"10.18653\/v1\/P16-1086"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Koehn, P. (2020). Neural Machine Translation, Cambridge University Press.","DOI":"10.1017\/9781108608480"},{"key":"ref_23","unstructured":"Xiang, Y., Zhou, X., Chen, Q., Zheng, Z., Tang, B., Wang, X., and Qin, Y. (2016, January 11\u201316). Incorporating label dependency for answer quality tagging in Community Question Answering Via CNN-LSTM-CRF. Proceedings of the COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, Osaka, Japan."},{"key":"ref_24","unstructured":"Bahdanau, D., Cho, K., and Bengio, Y. (2015, January 7\u20139). Neural Machine Translation by Jointly Learning to Align and Translate. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Mullenbach, J., Wiegreffe, S., Duke, J., Sun, J., and Eisenstein, J. (2018, January 1\u20136). Explainable Prediction of Medical Codes from Clinical Text. Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, LA, USA.","DOI":"10.18653\/v1\/N18-1100"},{"key":"ref_26","first-page":"367","article-title":"Attention-Based Explanation in a Deep Learning Model For Classifying Radiology Reports","volume":"Volume 12721","author":"Putelli","year":"2021","journal-title":"Proceedings of the Artificial Intelligence in Medicine-19th International Conference on Artificial Intelligence in Medicine, AIME 2021, Lecture Notes in Computer Science"},{"key":"ref_27","unstructured":"Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019, January 2\u20137). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Qu, C., Yang, L., Qiu, M., Croft, W.B., Zhang, Y., and Iyyer, M. (2019, January 21\u201325). BERT with history answer embedding for conversational question answering. Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, Paris, France.","DOI":"10.1145\/3331184.3331341"},{"key":"ref_29","first-page":"4163","article-title":"TinyBERT: Distilling BERT for Natural Language Understanding","volume":"Volume EMNLP 2020","author":"Jiao","year":"2020","journal-title":"Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2020"},{"key":"ref_30","unstructured":"Wu, C., Wu, F., Qi, T., Huang, Y., and Xie, X. (2021). Fastformer: Additive Attention Can Be All You Need. arXiv."},{"key":"ref_31","first-page":"848","article-title":"Combining Multi-task Learning with Transfer Learning for Biomedical Named Entity Recognition","volume":"Volume 176","author":"Cristani","year":"2020","journal-title":"Knowledge-Based and Intelligent Information & Engineering Systems, Proceedings of the 24th International Conference KES-2020, Virtual Event, 16\u201318 September 2020"},{"key":"ref_32","first-page":"349","article-title":"Deep Learning for Classification of Radiology Reports with a Hierarchical Schema","volume":"Volume 176","author":"Cristani","year":"2020","journal-title":"Knowledge-Based and Intelligent Information & Engineering Systems, Proceedings of the 24th International Conference KES-2020, Virtual Event, 16\u201318 September 2020"},{"key":"ref_33","unstructured":"Mehmood, T., Serina, I., Lavelli, A., and Gerevini, A. (2020, January 25\u201327). Knowledge Distillation Techniques for Biomedical Named Entity Recognition. Proceedings of the 4th Workshop on Natural Language for Artificial Intelligence (NL4AI 2020) Co-Located with the 19th International Conference of the Italian Association for Artificial Intelligence (AI*IA 2020), Virture."},{"key":"ref_34","first-page":"445","article-title":"Applying Self-interaction Attention for Extracting Drug-Drug Interactions","volume":"Volume 11946","author":"Putelli","year":"2019","journal-title":"Proceedings of the AI*IA 2019-Advances in Artificial Intelligence-XVIIIth International Conference of the Italian Association for Artificial Intelligence"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1016\/j.artmed.2018.05.006","article-title":"Automatic classification of radiological reports for clinical care","volume":"91","author":"Gerevini","year":"2018","journal-title":"Artif. Intell. Med."},{"key":"ref_36","unstructured":"Putelli, L., Gerevini, A.E., Lavelli, A., and Serina, I. (2022, January 15). The Impact of Self-Interaction Attention on the Extraction of Drug-Drug Interactions. Available online: http:\/\/ceur-ws.org\/Vol-2481\/paper61.pdf."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"131662","DOI":"10.1109\/ACCESS.2020.3009626","article-title":"Evaluation of sentiment analysis in finance: From lexicons to transformers","volume":"8","author":"Mishev","year":"2020","journal-title":"IEEE Access"},{"key":"ref_38","unstructured":"Mohamed, B.A., Abdelhakim, B.A., and Youness, S. (2018, January 10\u201311). A Deep Learning Model for an Intelligent Chat Bot System: An Application to E-Learning Domain. Proceedings of the Third International Conference on Smart City Applications, Tetouan, Morocco."},{"key":"ref_39","first-page":"150","article-title":"A Context-Centric Chatbot for Cryptocurrency Using the Bidirectional Encoder Representations from Transformers Neural Networks","volume":"15","author":"Xie","year":"2021","journal-title":"Int. J. Econ. Manag. Eng."},{"key":"ref_40","first-page":"105","article-title":"EVALITA4ELG: Italian Benchmark Linguistic Resources, NLP Services and Tools for the ELG Platform","volume":"6","author":"Patti","year":"2020","journal-title":"IJCoL Ital. J. Comput. Linguist."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Bellini, V., Biancofiore, G.M., Di Noia, T., Di Sciascio, E., Narducci, F., and Pomo, C. (2020, January 27\u201329). GUapp: A Conversational Agent for Job Recommendation for the Italian Public Administration. Proceedings of the 2020 IEEE Conference on Evolving and Adaptive Intelligent Systems (EAIS), Bari, Italy.","DOI":"10.1109\/EAIS48028.2020.9122756"},{"key":"ref_42","unstructured":"Polignano, M., Basile, P., De Gemmis, M., Semeraro, G., and Basile, V. (2019, January 13\u201315). Alberto: Italian BERT language understanding model for NLP challenging tasks based on tweets. Proceedings of the 6th Italian Conference on Computational Linguistics, CLiC-it 2019. CEUR, Bari, Italy."},{"key":"ref_43","unstructured":"Polignano, M., Basile, P., De Gemmis, M., and Semeraro, G. (2022, January 15). Hate Speech Detection through AlBERTo Italian Language Understanding Model. Available online: http:\/\/ceur-ws.org\/Vol-2521\/paper-06.pdf."},{"key":"ref_44","unstructured":"Goel, A.K., and Polepeddi, L. (2022, January 15). Jill Watson: A Virtual Teaching Assistant for Online Education. Available online: https:\/\/smartech.gatech.edu\/bitstream\/handle\/1853\/59104\/goelpolepeddi-harvardvolume-v7.1.pdf?sequence=1&isAllowed=y."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Kar, R., and Haldar, R. (2016). Applying Chatbots to the Internet of Things: Opportunities and Architectural Elements. Int. J. Adv. Comput. Sci. Appl., 7.","DOI":"10.14569\/IJACSA.2016.071119"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Braun, D., Hernandez Mendez, A., Matthes, F., and Langen, M. (2017, January 15\u201317). Evaluating Natural Language Understanding Services for Conversational Question Answering Systems. Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, Saarbr\u00fccken, Germany.","DOI":"10.18653\/v1\/W17-5522"},{"key":"ref_47","unstructured":"Liu, X., Eshghi, A., Swietojanski, P., and Rieser, V. (2019). Benchmarking Natural Language Understanding Services for building Conversational Agents. arXiv."},{"key":"ref_48","unstructured":"Zubani, M., Sigalini, L., Serina, I., and Gerevini, A.E. (2020, January 16\u201318). Evaluating different Natural Language Understanding services in a real business case for the Italian language. Proceedings of the Knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 24th International Conference KES2020, Virtual Event."},{"key":"ref_49","first-page":"431","article-title":"Leveraging Multi-task Learning for Biomedical Named Entity Recognition","volume":"Volume 11946","author":"Mehmood","year":"2019","journal-title":"Proceedings of the AI*IA 2019-Advances in Artificial Intelligence-XVIIIth International Conference of the Italian Association for Artificial Intelligence"}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/14\/2\/62\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:22:13Z","timestamp":1760134933000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/14\/2\/62"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,18]]},"references-count":49,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2022,2]]}},"alternative-id":["fi14020062"],"URL":"https:\/\/doi.org\/10.3390\/fi14020062","relation":{},"ISSN":["1999-5903"],"issn-type":[{"type":"electronic","value":"1999-5903"}],"subject":[],"published":{"date-parts":[[2022,2,18]]}}}