{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,12]],"date-time":"2026-03-12T20:03:35Z","timestamp":1773345815824,"version":"3.50.1"},"reference-count":41,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2022,4,26]],"date-time":"2022-04-26T00:00:00Z","timestamp":1650931200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001871","name":"Funda\u00e7\u00e3o para a Ci\u00eancia e Tecnologia","doi-asserted-by":"publisher","award":["DSAIPA\/AI\/0040\/2019"],"award-info":[{"award-number":["DSAIPA\/AI\/0040\/2019"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>SNS24, the Portuguese National Health Contact Center, is a telephone and digital public service that provides clinical services. SNS24 plays an important role in the identification of users\u2019 clinical situations according to their symptoms. Currently, there are a number of possible clinical algorithms defined, and selecting the appropriate clinical algorithm is very important in each telephone triage episode. Decreasing the duration of the phone calls and allowing a faster interaction between citizens and SNS24 service can further improve the performance of the telephone triage service. In this paper, we present a study using deep learning approaches to build classification models, aiming to support the nurses with the clinical algorithm\u2019s choice. Three different deep learning architectures, namely convolutional neural network (CNN), recurrent neural network (RNN), and transformers-based approaches are applied across a total number of 269,654 call records belonging to 51 classes. The CNN, RNN, and transformers-based model each achieve an accuracy of 76.56%, 75.88%, and 78.15% over the test set in the preliminary experiments. Models using the transformers-based architecture are further fine-tuned, achieving an accuracy of 79.67% with Adam and 79.72% with SGD after learning rate fine-tuning; an accuracy of 79.96% with Adam and 79.76% with SGD after epochs fine-tuning; an accuracy of 80.57% with Adam after the batch size fine-tuning. Analysis of similar clinical symptoms is carried out using the fine-tuned neural network model. Comparisons are done over the labels predicted by the neural network model, the support vector machines model, and the original labels from SNS24. These results suggest that using deep learning is an effective and promising approach to aid the clinical triage of the SNS24 phone call services.<\/jats:p>","DOI":"10.3390\/fi14050130","type":"journal-article","created":{"date-parts":[[2022,4,26]],"date-time":"2022-04-26T21:37:53Z","timestamp":1651009073000},"page":"130","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Clinical Trial Classification of SNS24 Calls with Neural Networks"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6720-4831","authenticated-orcid":false,"given":"Hua","family":"Yang","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of \u00c9vora, 7000-671 \u00c9vora, Portugal"},{"name":"Department of Computer Science, Zhongyuan University of Technology, Zhengzhou 450007, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1323-0249","authenticated-orcid":false,"given":"Teresa","family":"Gon\u00e7alves","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of \u00c9vora, 7000-671 \u00c9vora, Portugal"},{"name":"Centro ALGORITMI, Vista Lab, University of \u00c9vora, 7000-671 \u00c9vora, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5086-059X","authenticated-orcid":false,"given":"Paulo","family":"Quaresma","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of \u00c9vora, 7000-671 \u00c9vora, Portugal"},{"name":"Centro ALGORITMI, Vista Lab, University of \u00c9vora, 7000-671 \u00c9vora, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2449-5477","authenticated-orcid":false,"given":"Renata","family":"Vieira","sequence":"additional","affiliation":[{"name":"CIDEHUS, University of \u00c9vora, 7000-809 \u00c9vora, Portugal"}]},{"given":"Rute","family":"Veladas","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of \u00c9vora, 7000-671 \u00c9vora, Portugal"}]},{"given":"C\u00e1tia Sousa","family":"Pinto","sequence":"additional","affiliation":[{"name":"Servi\u00e7os Partilhados do Minist\u00e9rio da Sa\u00fade, 1050-099 Lisboa, Portugal"}]},{"given":"Jo\u00e3o","family":"Oliveira","sequence":"additional","affiliation":[{"name":"Servi\u00e7os Partilhados do Minist\u00e9rio da Sa\u00fade, 1050-099 Lisboa, Portugal"}]},{"given":"Maria Cortes","family":"Ferreira","sequence":"additional","affiliation":[{"name":"Servi\u00e7os Partilhados do Minist\u00e9rio da Sa\u00fade, 1050-099 Lisboa, Portugal"}]},{"given":"J\u00e9ssica","family":"Morais","sequence":"additional","affiliation":[{"name":"Servi\u00e7os Partilhados do Minist\u00e9rio da Sa\u00fade, 1050-099 Lisboa, Portugal"}]},{"given":"Ana Raquel","family":"Pereira","sequence":"additional","affiliation":[{"name":"Servi\u00e7os Partilhados do Minist\u00e9rio da Sa\u00fade, 1050-099 Lisboa, Portugal"}]},{"given":"Nuno","family":"Fernandes","sequence":"additional","affiliation":[{"name":"Servi\u00e7os Partilhados do Minist\u00e9rio da Sa\u00fade, 1050-099 Lisboa, Portugal"}]},{"given":"Carolina","family":"Gon\u00e7alves","sequence":"additional","affiliation":[{"name":"Servi\u00e7os Partilhados do Minist\u00e9rio da Sa\u00fade, 1050-099 Lisboa, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2022,4,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Mackway-Jones, K., Marsden, J., and Windle, J. (2013). Emergency Triage: Manchester Triage Group, John Wiley & Sons.","DOI":"10.1002\/9781118299029"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1347","DOI":"10.1056\/NEJMra1814259","article-title":"Machine learning in medicine","volume":"380","author":"Rajkomar","year":"2019","journal-title":"N. Engl. J. Med."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"bbaa434","DOI":"10.1093\/bib\/bbaa434","article-title":"Machine learning application for patient stratification and phenotype\/genotype investigation in a rare disease","volume":"22","author":"Spiga","year":"2021","journal-title":"Briefings Bioinform."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Sidey-Gibbons, J.A., and Sidey-Gibbons, C.J. (2019). Machine learning in medicine: A practical introduction. BMC Med. Res. Methodol., 19.","DOI":"10.1186\/s12874-019-0681-4"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Shin, B., Chokshi, F.H., Lee, T., and Choi, J.D. (2017, January 14\u201319). Classification of radiology reports using neural attention models. Proceedings of the 2017 International Joint Conference on Neural Networks (IJCNN), Anchorage, AK, USA.","DOI":"10.1109\/IJCNN.2017.7966408"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Wu, H., and Wang, M.D. (2017, January 20\u201323). Infer cause of death for population health using convolutional neural network. Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, Boston, MA, USA.","DOI":"10.1145\/3107411.3107447"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Mullenbach, J., Wiegreffe, S., Duke, J., Sun, J., and Eisenstein, J. (2018). Explainable prediction of medical codes from clinical text. arXiv.","DOI":"10.18653\/v1\/N18-1100"},{"key":"ref_8","first-page":"246","article-title":"Medical text classification using convolutional neural networks","volume":"235","author":"Hughes","year":"2017","journal-title":"Stud. Health Technol. Inform."},{"key":"ref_9","unstructured":"Baker, S., Korhonen, A.L., and Pyysalo, S. (2016). Cancer Hallmark Text Classification Using Convolutional Neural Networks, The COLING 2016 Organizing Committee."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"912","DOI":"10.1109\/TCBB.2020.2994780","article-title":"CNN-RNN based intelligent recommendation for online medical pre-diagnosis support","volume":"18","author":"Zhou","year":"2020","journal-title":"IEEE\/ACM Trans. Comput. Biol. Bioinform."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Gao, X., Xu, X., and Li, D. (2021, January 14\u201316). Accuracy Analysis of Triage Recommendation Based on CNN, RNN and RCNN Models. Proceedings of the 2021 IEEE Asia-Pacific Conference on Image Processing, Electronics and Computers (IPEC), Dalian, China.","DOI":"10.1109\/IPEC51340.2021.9421099"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"3596","DOI":"10.1109\/JBHI.2021.3062322","article-title":"Limitations of Transformers on Clinical Text Classification","volume":"25","author":"Gao","year":"2021","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Behera, B., Kumaravelan, G., and Kumar, P. (2019, January 18\u201320). Performance evaluation of deep learning algorithms in biomedical document classification. Proceedings of the 2019 11th International Conference on Advanced Computing (ICoAC), Chennai, India.","DOI":"10.1109\/ICoAC48765.2019.246843"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Al-Garadi, M.A., Yang, Y.C., Cai, H., Ruan, Y., O\u2019Connor, K., Graciela, G.H., Perrone, J., and Sarker, A. (2021). Text classification models for the automatic detection of nonmedical prescription medication use from social media. BMC Med. Inform. Decis. Mak., 21.","DOI":"10.1186\/s12911-021-01394-0"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Mascio, A., Kraljevic, Z., Bean, D., Dobson, R., Stewart, R., Bendayan, R., and Roberts, A. (2020). Comparative analysis of text classification approaches in electronic health records. arXiv.","DOI":"10.18653\/v1\/2020.bionlp-1.9"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"38767","DOI":"10.1109\/ACCESS.2021.3064000","article-title":"Active Learning for Biomedical Text Classification Based on Automatically Generated Regular Expressions","volume":"9","author":"Flores","year":"2021","journal-title":"IEEE Access"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Veladas, R., Yang, H., Quaresma, P., Gon\u00e7alves, T., Vieira, R., Sousa Pinto, C., Martins, J.P., Oliveira, J., and Cortes Ferreira, M. (2021). Aiding Clinical Triage with Text Classification. EPIA Conference on Artificial Intelligence, Springer.","DOI":"10.1007\/978-3-030-86230-5_7"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Kim, Y. (2014, January 25\u201329). Convolutional Neural Networks for Sentence Classification. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.","DOI":"10.3115\/v1\/D14-1181"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1207\/s15516709cog1402_1","article-title":"Finding structure in time","volume":"14","author":"Elman","year":"1990","journal-title":"Cogn. Sci."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Cho, K., Van Merri\u00ebnboer, B., Bahdanau, D., and Bengio, Y. (2014). On the properties of neural machine translation: Encoder-decoder approaches. arXiv.","DOI":"10.3115\/v1\/W14-4012"},{"key":"ref_22","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Li, Y., and Yang, T. (2017). Word Embedding for Understanding Natural Language: A Survey. Studies in Big Data, Springer.","DOI":"10.1007\/978-3-319-53817-4_4"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018, January 1\u20136). Deep Contextualized Word Representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, LA, USA.","DOI":"10.18653\/v1\/N18-1202"},{"key":"ref_25","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv."},{"key":"ref_26","first-page":"5753","article-title":"Xlnet: Generalized autoregressive pretraining for language understanding","volume":"32","author":"Yang","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_27","unstructured":"Akbik, A., Blythe, D., and Vollgraf, R. (2018, January 20\u201328). Contextual String Embeddings for Sequence Labeling. Proceedings of the COLING 2018, 27th International Conference on Computational Linguistics, Santa Fe, NM, USA."},{"key":"ref_28","unstructured":"Akbik, A., Bergmann, T., Blythe, D., Rasul, K., Schweter, S., and Vollgraf, R. (2019, January 2\u20137). FLAIR: An easy-to-use framework for state-of-the-art NLP. Proceedings of the NAACL 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), Minneapolis, MN, USA."},{"key":"ref_29","unstructured":"Zhang, Y., and Wallace, B. (2015). A sensitivity analysis of (and practitioners\u2019 guide to) convolutional neural networks for sentence classification. arXiv."},{"key":"ref_30","first-page":"1929","article-title":"Dropout: A simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_31","unstructured":"Hendrycks, D., and Gimpel, K. (2022, April 02). Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units. Available online: https:\/\/www.researchgate.net\/publication\/304506026_Bridging_Nonlinearities_and_Stochastic_Regularizers_with_Gaussian_Error_Linear_Units."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Alatawi, H.S., Alhothali, A.M., and Moria, K.M. (2020). Detecting White Supremacist Hate Speech using Domain Specific Word Embedding with Deep Learning and BERT. arXiv.","DOI":"10.1109\/ACCESS.2021.3100435"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Souza, F., Nogueira, R., and Lotufo, R. (2020, January 20\u201323). BERTimbau: Pretrained BERT models for Brazilian Portuguese. Proceedings of the 9th Brazilian Conference on Intelligent Systems, BRACIS, Rio Grande do Sul, Brazil.","DOI":"10.1007\/978-3-030-61377-8_28"},{"key":"ref_34","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_35","unstructured":"Choi, D., Shallue, C.J., Nado, Z., Lee, J., Maddison, C.J., and Dahl, G.E. (2019). On empirical comparisons of optimizers for deep learning. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Vani, S., and Rao, T.M. (2019, January 23\u201325). An experimental approach towards the performance assessment of various optimizers on convolutional neural network. Proceedings of the 2019 3rd International Conference on Trends in Electronics and Informatics (ICOEI), Tirunelveli, India.","DOI":"10.1109\/ICOEI.2019.8862686"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Bottou, L. Large-scale machine learning with stochastic gradient descent. Proceedings of the COMPSTAT\u20192010.","DOI":"10.1007\/978-3-7908-2604-3_16"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Bengio, Y. (2012). Practical recommendations for gradient-based training of deep architectures. Neural Networks: Tricks Trade, Springer.","DOI":"10.1007\/978-3-642-35289-8_26"},{"key":"ref_39","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Smith, L.N. (2017, January 24\u201331). Cyclical learning rates for training neural networks. Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), Santa Rosa, CA, USA.","DOI":"10.1109\/WACV.2017.58"},{"key":"ref_41","unstructured":"Smith, S.L., and Le, Q.V. (2017). A bayesian perspective on generalization and stochastic gradient descent. arXiv."}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/14\/5\/130\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:01:37Z","timestamp":1760137297000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/14\/5\/130"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,26]]},"references-count":41,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,5]]}},"alternative-id":["fi14050130"],"URL":"https:\/\/doi.org\/10.3390\/fi14050130","relation":{},"ISSN":["1999-5903"],"issn-type":[{"value":"1999-5903","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,26]]}}}