{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:43:47Z","timestamp":1760147027085,"version":"build-2065373602"},"reference-count":37,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2023,1,3]],"date-time":"2023-01-03T00:00:00Z","timestamp":1672704000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"FCT\u2014Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia","award":["DSAIPA\/AI\/0040\/2019"],"award-info":[{"award-number":["DSAIPA\/AI\/0040\/2019"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>This paper presents an analysis of the calls made to the Portuguese National Health Contact Center (SNS24) during a three years period. The final goal was to develop a system to help nurse attendants select the appropriate clinical pathway (from 59 options) for each call. It examines several aspects of the calls distribution like age and gender of the user, date and time of the call and final referral, among others and presents comparative results for alternative classification models (SVM and CNN) and different data samples (three months, one and two years data models). For the task of selecting the appropriate pathway, the models, learned on the basis of the available data, achieved F1 values that range between 0.642 (3 months CNN model) and 0.783 (2 years CNN model), with SVM having a more stable performance (between 0.743 and 0.768 for the corresponding data samples). These results are discussed regarding error analysis and possibilities for explaining the system decisions. A final meta evaluation, based on a clinical expert overview, compares the different choices: the nurse attendants (reference ground truth), the expert and the automatic decisions (2 models), revealing a higher agreement between the ML models, followed by their agreement with the clinical expert, and minor agreement with the reference.<\/jats:p>","DOI":"10.3390\/fi15010026","type":"journal-article","created":{"date-parts":[[2023,1,4]],"date-time":"2023-01-04T02:15:42Z","timestamp":1672798542000},"page":"26","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Clinical Screening Prediction in the Portuguese National Health Service: Data Analysis, Machine Learning Models, Explainability and Meta-Evaluation"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1323-0249","authenticated-orcid":false,"given":"Teresa","family":"Gon\u00e7alves","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of \u00c9vora, 7000-671 \u00c9vora, Portugal"},{"name":"Centro Algoritmi, Vista Lab, University of \u00c9vora, 7000-671 \u00c9vora, Portugal"}]},{"given":"Rute","family":"Veladas","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of \u00c9vora, 7000-671 \u00c9vora, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6720-4831","authenticated-orcid":false,"given":"Hua","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of \u00c9vora, 7000-671 \u00c9vora, Portugal"},{"name":"Department of Computer Science, Zhongyuan University of Technology, Zhengzhou 450007, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2449-5477","authenticated-orcid":false,"given":"Renata","family":"Vieira","sequence":"additional","affiliation":[{"name":"CIDEHUS, University of \u00c9vora, 7000-809 \u00c9vora, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5086-059X","authenticated-orcid":false,"given":"Paulo","family":"Quaresma","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of \u00c9vora, 7000-671 \u00c9vora, Portugal"},{"name":"Centro Algoritmi, Vista Lab, University of \u00c9vora, 7000-671 \u00c9vora, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1644-9502","authenticated-orcid":false,"given":"Paulo","family":"Infante","sequence":"additional","affiliation":[{"name":"Department of Mathematics, University of \u00c9vora, 7000-671 \u00c9vora, Portugal"},{"name":"CIMA\u2014Centro de Investiga\u00e7\u00e3o em Matem\u00e1tica e Aplica\u00e7\u00f5es, University of \u00c9vora, 7000-671 \u00c9vora, Portugal"}]},{"given":"C\u00e1tia","family":"Sousa Pinto","sequence":"additional","affiliation":[{"name":"Servi\u00e7os Partilhados do Minist\u00e9rio da Sa\u00fade, 1050-099 Lisboa, Portugal"}]},{"given":"Jo\u00e3o","family":"Oliveira","sequence":"additional","affiliation":[{"name":"Servi\u00e7os Partilhados do Minist\u00e9rio da Sa\u00fade, 1050-099 Lisboa, Portugal"}]},{"given":"Maria","family":"Cortes Ferreira","sequence":"additional","affiliation":[{"name":"Servi\u00e7os Partilhados do Minist\u00e9rio da Sa\u00fade, 1050-099 Lisboa, Portugal"}]},{"given":"J\u00e9ssica","family":"Morais","sequence":"additional","affiliation":[{"name":"Servi\u00e7os Partilhados do Minist\u00e9rio da Sa\u00fade, 1050-099 Lisboa, Portugal"}]},{"given":"Ana Raquel","family":"Pereira","sequence":"additional","affiliation":[{"name":"Servi\u00e7os Partilhados do Minist\u00e9rio da Sa\u00fade, 1050-099 Lisboa, Portugal"}]},{"given":"Nuno","family":"Fernandes","sequence":"additional","affiliation":[{"name":"Servi\u00e7os Partilhados do Minist\u00e9rio da Sa\u00fade, 1050-099 Lisboa, Portugal"}]},{"given":"Carolina","family":"Gon\u00e7alves","sequence":"additional","affiliation":[{"name":"Servi\u00e7os Partilhados do Minist\u00e9rio da Sa\u00fade, 1050-099 Lisboa, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2023,1,3]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Mackway-Jones, K., Marsden, J., and Windle, J. (2013). Emergency Triage: Manchester Triage Group, John Wiley & Sons.","DOI":"10.1002\/9781118299029"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Veladas, R., Yang, H., Quaresma, P., Gon\u00e7alves, T., Vieira, R., Sousa Pinto, C., Martins, J.P., Oliveira, J., and Cortes Ferreira, M. (2021). Aiding Clinical Triage with Text Classification. Proceedings of the EPIA Conference on Artificial Intelligence, Springer.","DOI":"10.1007\/978-3-030-86230-5_7"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Yang, H., Gon\u00e7alves, T., Quaresma, P., Vieira, R., Veladas, R., Pinto, C.S., Oliveira, J., Ferreira, M.C., Morais, J., and Pereira, A.R. (2022). Clinical Trial Classification of SNS24 Calls with Neural Networks. Future Internet, 14.","DOI":"10.3390\/fi14050130"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/j.artmed.2015.04.007","article-title":"An empirical evaluation of supervised learning approaches in assigning diagnosis codes to electronic medical records","volume":"65","author":"Kavuluru","year":"2015","journal-title":"Artif. Intell. Med."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1016\/j.jbi.2015.02.003","article-title":"Efficient and sparse feature selection for biomedical text classification via the elastic net: Application to ICU risk stratification from nursing notes","volume":"54","author":"Marafino","year":"2015","journal-title":"J. Biomed. Inform."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"494","DOI":"10.1016\/j.eswa.2018.09.034","article-title":"Clinical text classification research trends: Systematic literature review and open issues","volume":"116","author":"Mujtaba","year":"2019","journal-title":"Expert Syst. Appl."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1589","DOI":"10.1109\/JBHI.2017.2767063","article-title":"Deep EHR: A survey of recent advances in deep learning techniques for electronic health record (EHR) analysis","volume":"22","author":"Shickel","year":"2017","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Shao, Y., Taylor, S., Marshall, N., Morioka, C., and Zeng-Treitler, Q. (2018, January 10\u201313). Clinical text classification with word embedding features vs. bag-of-words features. In Proceedings of the 2018 IEEE International Conference on Big Data (Big Data), Seattle, WA, USA.","DOI":"10.1109\/BigData.2018.8622345"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Mustafa, A., and Rahimi Azghadi, M. (2021). Automated Machine Learning for Healthcare and Clinical Notes Analysis. Computers, 10.","DOI":"10.3390\/computers10020024"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"835","DOI":"10.1016\/j.procs.2017.11.108","article-title":"Data-driven modeling of clinical pathways using electronic health records","volume":"121","author":"Funkner","year":"2017","journal-title":"Procedia Comput. Sci."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Elbattah, M., Molloy, O., and Zeigler, B.P. (2018, January 9\u201312). Designing Care Pathways Using Simulation Modeling and Machine Learning. Proceedings of the 2018 Winter Simulation Conference (WSC), Gothenburg, Sweden.","DOI":"10.1109\/WSC.2018.8632360"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Ye, X., Soares, F., De Maria, E., G\u00f3mez Vilda, P., Cabitza, F., Fred, A., and Gamboa, H. (2021). Leveraging Clinical Notes for Enhancing Decision-Making Systems with Relevant Patient Information. Proceedings of the Biomedical Engineering Systems and Technologies, Springer International Publishing.","DOI":"10.1007\/978-3-030-72379-8"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Aggarwal, C.C., and Clustering, C.R.D. (2014). Algorithms and Applications, Springer.","DOI":"10.1201\/b15410"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"216","DOI":"10.1162\/NECO_a_00793","article-title":"An empirical overview of the no free lunch theorem and its effect on real-world machine learning classification","volume":"28","author":"Rojas","year":"2016","journal-title":"Neural Comput."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Mascio, A., Kraljevic, Z., Bean, D., Dobson, R., Stewart, R., Bendayan, R., and Roberts, A. (2020). Comparative analysis of text classification approaches in electronic health records. arXiv.","DOI":"10.18653\/v1\/2020.bionlp-1.9"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"216","DOI":"10.1016\/j.ins.2018.09.001","article-title":"An analysis of hierarchical text classification using word embeddings","volume":"471","author":"Stein","year":"2019","journal-title":"Inf. Sci."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1007\/s10462-018-09677-1","article-title":"Survey on supervised machine learning techniques for automatic text classification","volume":"52","author":"Kadhim","year":"2019","journal-title":"Artif. Intell. Rev."},{"key":"ref_18","unstructured":"Baker, S., Korhonen, A.L., and Pyysalo, S. (2016, January 11\u201316). Cancer Hallmark Text Classification Using Convolutional Neural Networks. Proceedings of the Fifth Workshop on Building and Evaluating Resources for Biomedical Text Mining (BioTxtM 2016), Osaka, Japan."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"38767","DOI":"10.1109\/ACCESS.2021.3064000","article-title":"Active Learning for Biomedical Text Classification Based on Automatically Generated Regular Expressions","volume":"9","author":"Flores","year":"2021","journal-title":"IEEE Access"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"3596","DOI":"10.1109\/JBHI.2021.3062322","article-title":"Limitations of Transformers on Clinical Text Classification","volume":"25","author":"Gao","year":"2021","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"103655","DOI":"10.1016\/j.jbi.2020.103655","article-title":"The role of explainability in creating trustworthy artificial intelligence for health care: A comprehensive survey of the terminology, design choices, and evaluation strategies","volume":"113","author":"Markus","year":"2021","journal-title":"J. Biomed. Inform."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12911-020-01332-6","article-title":"Explainability for artificial intelligence in healthcare: A multidisciplinary perspective","volume":"20","author":"Amann","year":"2020","journal-title":"BMC Med. Inform. Decis. Mak."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Li, Y., and Yang, T. (2018). Word Embedding for Understanding Natural Language: A Survey. Guide to Big Data Applications, Springer International Publishing.","DOI":"10.1007\/978-3-319-53817-4_4"},{"key":"ref_24","unstructured":"Burstein, J., Doran, C., and Solorio, T. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, 2\u20137 June 2019, Volume 1 (Long and Short Papers), Association for Computational Linguistics."},{"key":"ref_25","unstructured":"Akbik, A., Blythe, D., and Vollgraf, R. (2018, January 20\u201326). Contextual String Embeddings for Sequence Labeling. Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, NM, USA."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Souza, F., Nogueira, R., and Lotufo, R. (2020, January 20\u201323). BERTimbau: Pretrained BERT models for Brazilian Portuguese. Proceedings of the 9th Brazilian Conference on Intelligent Systems, BRACIS, Rio Grande do Sul, Brazil.","DOI":"10.1007\/978-3-030-61377-8_28"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Kim, Y. (2014, January 25\u201329). Convolutional Neural Networks for Sentence Classification. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.","DOI":"10.3115\/v1\/D14-1181"},{"key":"ref_28","first-page":"1929","article-title":"Dropout: A simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_29","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_30","unstructured":"(2022, December 07). Scikit-learn. Available online: https:\/\/scikit-learn.org\/stable\/."},{"key":"ref_31","unstructured":"(2022, December 07). TensorFlow. Available online: https:\/\/www.tensorflow.org\/."},{"key":"ref_32","unstructured":"(2022, December 07). Pytorch. Available online: https:\/\/pytorch.org\/."},{"key":"ref_33","unstructured":"(2022, December 07). Python. Available online: https:\/\/www.python.org\/."},{"key":"ref_34","unstructured":"(2022, December 07). ELI5. Available online: https:\/\/eli5.readthedocs.io\/en\/latest\/index.html."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1038\/s42256-020-0219-9","article-title":"The carbon impact of artificial intelligence","volume":"2","author":"Dhar","year":"2020","journal-title":"Nat. Mach. Intell."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Bannour, N., Ghannay, S., N\u00e9v\u00e9ol, A., and Ligozat, A.L. (2021, January 7\u201311). Evaluating the carbon footprint of NLP methods: A survey and analysis of existing tools. Proceedings of the EMNLP, Workshop SustaiNLP, Online.","DOI":"10.18653\/v1\/2021.sustainlp-1.2"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1016\/S0167-9473(01)00065-2","article-title":"Stochastic gradient boosting","volume":"38","author":"Friedman","year":"2002","journal-title":"Comput. Stat. Data Anal."}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/15\/1\/26\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T17:57:22Z","timestamp":1760119042000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/15\/1\/26"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,3]]},"references-count":37,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,1]]}},"alternative-id":["fi15010026"],"URL":"https:\/\/doi.org\/10.3390\/fi15010026","relation":{},"ISSN":["1999-5903"],"issn-type":[{"type":"electronic","value":"1999-5903"}],"subject":[],"published":{"date-parts":[[2023,1,3]]}}}