{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T01:11:21Z","timestamp":1760058681601,"version":"build-2065373602"},"reference-count":33,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2025,4,24]],"date-time":"2025-04-24T00:00:00Z","timestamp":1745452800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Data"],"abstract":"<jats:p>In this paper, we use visualization tools to give insight into the performance of six classifiers on multivariate time series data. Five of these classifiers are deep learning models, while the Rocket classifier represents a non-deep learning approach. Our comparison is conducted across fifteen datasets from the UEA repository. Additionally, we apply data engineering techniques to each dataset, allowing us to assess classifier performance concerning the available features and channels within the time series. The results of our experiments indicate that the ROCKET classifier consistently achieves strong performance across most datasets, while the Transformer model underperforms, likely due to the limited number of instances per class in certain datasets.<\/jats:p>","DOI":"10.3390\/data10050058","type":"journal-article","created":{"date-parts":[[2025,4,24]],"date-time":"2025-04-24T08:08:41Z","timestamp":1745482121000},"page":"58","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Using Visualization to Evaluate the Performance of Algorithms for Multivariate Time Series Classification"],"prefix":"10.3390","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3422-7225","authenticated-orcid":false,"given":"Edgar","family":"Acu\u00f1a","sequence":"first","affiliation":[{"name":"Mathematical Science Department, University of Puerto Rico at Mayaguez, Mayaguez PR00681, Puerto Rico"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4254-2305","authenticated-orcid":false,"given":"Roxana","family":"Aparicio","sequence":"additional","affiliation":[{"name":"Department of Industrial Engineering, University of Puerto Rico at Mayaguez, Mayaguez PR00681, Puerto Rico"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2025,4,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"606","DOI":"10.1007\/s10618-016-0483-9","article-title":"The great time series classification bake off: A review and experimental evaluation of recent algorithmic advances","volume":"31","author":"Bagnall","year":"2017","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_2","unstructured":"Bagnall, A., Dau, H.A., Lines, J., Flynn, M., Large, J., Bostrom, A., Southam, P., and Keogh, E. (2018). The UEA multivariate time series classification archive. arXiv."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"917","DOI":"10.1007\/s10618-019-00619-1","article-title":"Deep learning for time series classification: A review","volume":"33","author":"Fawaz","year":"2019","journal-title":"Data Min. Knowl. Disc."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Baldan, F., and Ben\u00edtez, J. (2020). Multivariable times series classification through an interpretable representation. arXiv.","DOI":"10.1016\/j.ins.2021.05.024"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"401","DOI":"10.1007\/s10618-020-00727-3","article-title":"The great multivariate time series classification bake off: A review and experimental evaluation of recent algorithmic advances","volume":"35","author":"Flynn","year":"2021","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Pasos Ruiz, A., and Bagnall, A. (2023). Dimension selection strategies for multivariate time series classification with HIVE-COTEv2.0. Advanced Analytics and Learning on Temporal Data Lecture Notes in Computer Science, Springer International Publishing.","DOI":"10.1007\/978-3-031-24378-3_9"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Ilbert, R., Huang, T.V., and Zhang, Z. (2024, January 13\u201316). Data Augmentation for Multivariate Time Series Classification: An Experimental Study. Proceedings of the IEEE 40th International Conference on Data Engineering Workshops (ICDEW), Utrecht, The Netherlands.","DOI":"10.1109\/ICDEW61823.2024.00023"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1454","DOI":"10.1007\/s10618-020-00701-z","article-title":"ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels","volume":"34","author":"Dempster","year":"2020","journal-title":"Data Min. Knowl. Disc."},{"key":"ref_9","unstructured":"L\u00f6ning, M., Bagnall, A., Ganesh, S., Viktor Kazakov, J.L., and Franz, K. (2019). sktime: A Unified Interface for Machine Learning with Time Series. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Wang, Z., Yan, W., and Oates, T. (2017, January 14\u201319). Time series classification from scratch with deep neural networks: A strong baseline. Proceedings of the International Joint Conference on Neural Networks, Anchorage, AK, USA.","DOI":"10.1109\/IJCNN.2017.7966039"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7\u201312). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1936","DOI":"10.1007\/s10618-020-00710-y","article-title":"InceptionTime: Finding AlexNet for time series classification","volume":"34","author":"Fawaz","year":"2020","journal-title":"Data Min. Knowl. Disc."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"5455","DOI":"10.1007\/s10462-020-09825-6","article-title":"A Survey of the Recent Architectures of Deep Convolutional Neural Networks","volume":"53","author":"Khan","year":"2020","journal-title":"Arti. Intell. Rev."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ymssp.2020.107398","article-title":"1D convolutional neural networks and applications: A Survey","volume":"151","author":"Kiranyaz","year":"2021","journal-title":"Mech. Syst. Signal Process."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Acuna, E., Kendziora, C., Fustenberg, R., Breshike, C.J., and Kendziora, D. (2024, January 21\u201326). Machine learning algorithms for analytes classification based on simulated spectra. Proceedings of the Algorithms, Technologies, and Applications for Multispectral and Hyperspectral Imaging XXX, National Harbor, MA, USA. 130310H.","DOI":"10.1117\/12.3013758"},{"key":"ref_16","first-page":"338","article-title":"Long short-term memory recurrent neural network architectures for large scale acoustic modeling","volume":"2014","author":"Sak","year":"2014","journal-title":"Interspeech"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"6572","DOI":"10.1109\/TITS.2020.2995722","article-title":"Driver profiling using long short term memory (LSTM) and convolutional neural network (CNN) methods","volume":"22","author":"Cura","year":"2020","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Xu, G., Ren, T., Chen, Y., and Che, W.A. (2020). One-Dimensional CNN-LSTM Model for Epileptic Seizure Recognition Using EEG Signal Analysis. Front. Neurosci., 14.","DOI":"10.3389\/fnins.2020.578126"},{"key":"ref_19","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkaoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017, January 4\u20139). Attention Is All You need. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA."},{"key":"ref_20","unstructured":"Qingsong, W., Tian, Z., Chaoli, Z., Weiqi, C., Ziqing, M., Junchi, Y., and Liang, S. (2023, January 19\u201325). Transformers in Time Series: A Survey. Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, Macao, China. Available online: https:\/\/github.com\/qingsongedu\/time-series-transformers-review."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Zerveas, G., Jayaraman, S., Patel, D., Bhamidipaty, A., and Eickhoff, C. (2021, January 14\u201318). A transformer-based framework for multivariate time series representation learning. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, Virtual, Singapore.","DOI":"10.1145\/3447548.3467401"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2559","DOI":"10.1109\/JBHI.2022.3140531","article-title":"Transformer Model for Functional Near-Infrared Spectroscopy Classification","volume":"26","author":"Wang","year":"2022","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1650037","DOI":"10.1142\/S0129065716500374","article-title":"Generalized models for the classification of abnormal movements in daily life and its applicability to epilepsy convulsion recognition","volume":"26","author":"Villar","year":"2016","journal-title":"Int. J. Neural Syst."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"146","DOI":"10.1504\/IJAPR.2017.085315","article-title":"Continuous pattern detection and recognition in stream-a benchmark for online gesture recognition","volume":"4","author":"Ghouaiel","year":"2017","journal-title":"Int. J. Appl. Pattern Recognit."},{"key":"ref_25","unstructured":"Wang, J., Balasubramanian, A., de La Vega, L., Green, J.R., Samal, A., and Prabhakaran, B. (2013, January 21\u201322). Word recognition from continuous articulatory movement time-series data using symbolic representations. Proceedings of the Fourth Workshop on Speech and Language Processing for Assistive Technologies, Grenoble, France."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1038\/18581","article-title":"A spelling device for the paralysed","volume":"398","author":"Birbaumer","year":"1999","journal-title":"Nature"},{"key":"ref_27","unstructured":"Cuturi, M. (July, January 28). Fast global alignment kernels. Proceedings of the 28th International Conference on Machine Learning (ICML-11), Bellevue, WA, USA."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"e215","DOI":"10.1161\/01.CIR.101.23.e215","article-title":"Physiobank, physiotoolkit, and physionet: Components of a new research resource for complex physiologic signals","volume":"101","author":"Goldberger","year":"2000","journal-title":"Circulation"},{"key":"ref_29","unstructured":"Olivetti, R., Kia, M., and Avesani, P. (2024, October 01). DecMeg2014-Decoding the Human Brain. Kaggle. Available online: https:\/\/kaggle.com\/competitions\/decoding-the-human-brain."},{"key":"ref_30","unstructured":"(2024, October 01). Xeno-Canto: Sharing Wildlife Sounds from Around the World. Repository. Available online: https:\/\/xeno-canto.org\/."},{"key":"ref_31","first-page":"157","article-title":"Classifying single trial EEG: Towards brain computer interfacing","volume":"14","author":"Blankertz","year":"2002","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Large, J., Kemsley, E.K., Wellner, N., Goodall, I., and Bagnall, A. (2018, January 3\u20136). Detecting forged alcohol non-invasively through vibrational spectroscopy and machine learning. Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, Melbourne, VIC, Australia.","DOI":"10.1007\/978-3-319-93034-3_24"},{"key":"ref_33","unstructured":"Dias, D., and Peres, S. (2008, January 26\u201331). Algoritmos bio-inspirados aplicados ao reconhecimento de padroes da libras: Enfoque no par\u00e2metro movimento. Proceedings of the 16 Simp\u00f3sio Internacional de Inicia\u00e7ao Cient\u0131fica da Universidade de Sao Paulo, S\u00e3o Paulo, Brasil."}],"container-title":["Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2306-5729\/10\/5\/58\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:20:42Z","timestamp":1760030442000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2306-5729\/10\/5\/58"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,24]]},"references-count":33,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2025,5]]}},"alternative-id":["data10050058"],"URL":"https:\/\/doi.org\/10.3390\/data10050058","relation":{},"ISSN":["2306-5729"],"issn-type":[{"type":"electronic","value":"2306-5729"}],"subject":[],"published":{"date-parts":[[2025,4,24]]}}}