{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T23:18:53Z","timestamp":1781824733552,"version":"3.54.5"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,9,5]],"date-time":"2023-09-05T00:00:00Z","timestamp":1693872000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,9,5]],"date-time":"2023-09-05T00:00:00Z","timestamp":1693872000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001779","name":"Monash University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001779","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Data Min Knowl Disc"],"published-print":{"date-parts":[[2024,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Transformers have demonstrated outstanding performance in many applications of deep learning. When applied to time series data, transformers require effective position encoding to capture the ordering of the time series data. The efficacy of position encoding in time series analysis is not well-studied and remains controversial, e.g., whether it is better to inject absolute position encoding or relative position encoding, or a combination of them. In order to clarify this, we first review existing absolute and relative position encoding methods when applied in time series classification. We then proposed a new absolute position encoding method dedicated to time series data called time Absolute Position Encoding (tAPE). Our new method incorporates the series length and input embedding dimension in absolute position encoding. Additionally, we propose computationally Efficient implementation of Relative Position Encoding (eRPE) to improve generalisability for time series. We then propose a novel multivariate time series classification model combining tAPE\/eRPE and convolution-based input encoding named ConvTran to improve the position and data embedding of time series data. The proposed absolute and relative position encoding methods are simple and efficient. They can be easily integrated into transformer blocks and used for downstream tasks such as forecasting, extrinsic regression, and anomaly detection. Extensive experiments on 32 multivariate time-series datasets show that our model is significantly more accurate than state-of-the-art convolution and transformer-based models. Code and models are open-sourced at <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/Navidfoumani\/ConvTran\">https:\/\/github.com\/Navidfoumani\/ConvTran<\/jats:ext-link>.<\/jats:p>","DOI":"10.1007\/s10618-023-00948-2","type":"journal-article","created":{"date-parts":[[2023,9,5]],"date-time":"2023-09-05T15:02:55Z","timestamp":1693926175000},"page":"22-48","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":201,"title":["Improving position encoding of transformers for multivariate time series classification"],"prefix":"10.1007","volume":"38","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2475-6040","authenticated-orcid":false,"given":"Navid Mohammadi","family":"Foumani","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chang Wei","family":"Tan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Geoffrey I.","family":"Webb","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mahsa","family":"Salehi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,9,5]]},"reference":[{"issue":"3","key":"948_CR1","doi-asserted-by":"publisher","first-page":"606","DOI":"10.1007\/s10618-016-0483-9","volume":"31","author":"A Bagnall","year":"2017","unstructured":"Bagnall A, Lines J, Bostrom A, Large J, Keogh E (2017) The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Min Knowl Disc 31(3):606\u2013660","journal-title":"Data Min Knowl Disc"},{"key":"948_CR2","unstructured":"Bagnall A, Dau HA, Lines J, Flynn M, Large J, Bostrom A, Southam P, Keogh E (2018) The UEA multivariate time series classification archive. arXiv preprint arXiv:1811.00075"},{"key":"948_CR3","doi-asserted-by":"crossref","unstructured":"Bagnall A, Flynn M, Large J, Lines J, Middlehurst M (2020) On the usage and performance of the hierarchical vote collective of transformation-based ensembles version 1.0 (hive-cote v1. 0). In: International workshop on advanced analytics and learning on temporal data, pp 3\u201318","DOI":"10.1007\/978-3-030-65742-0_1"},{"key":"948_CR4","first-page":"3965","volume":"34","author":"Z Dai","year":"2021","unstructured":"Dai Z, Liu H, Le QV, Tan M (2021) Coatnet: marrying convolution and attention for all data sizes. Adv Neural Inf Process Syst 34:3965\u20133977","journal-title":"Adv Neural Inf Process Syst"},{"issue":"5","key":"948_CR5","doi-asserted-by":"publisher","first-page":"1454","DOI":"10.1007\/s10618-020-00701-z","volume":"34","author":"A Dempster","year":"2020","unstructured":"Dempster A, Petitjean F, Webb GI (2020) Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. Data Min Knowl Disc 34(5):1454\u20131495","journal-title":"Data Min Knowl Disc"},{"key":"948_CR6","doi-asserted-by":"crossref","unstructured":"Dempster A, Schmidt DF, Webb GI (2021) Minirocket: A very fast (almost) deterministic transform for time series classification. In: SIGKDD conference on knowledge discovery and data mining, pp 248\u2013257","DOI":"10.1145\/3447548.3467231"},{"key":"948_CR7","first-page":"1","volume":"7","author":"J Dem\u0161ar","year":"2006","unstructured":"Dem\u0161ar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1\u201330","journal-title":"J Mach Learn Res"},{"key":"948_CR8","unstructured":"Devlin J, Chang MW, Lee K, Toutanova K (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805"},{"key":"948_CR9","unstructured":"Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold , Gelly, S, et al (2020) An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929"},{"issue":"3","key":"948_CR10","doi-asserted-by":"publisher","first-page":"733","DOI":"10.1162\/coli_a_00445","volume":"48","author":"P Dufter","year":"2022","unstructured":"Dufter P, Schmitt M, Sch\u00fctze H (2022) Position information in transformers: an overview. Comput Linguist 48(3):733\u2013763","journal-title":"Comput Linguist"},{"issue":"4","key":"948_CR11","doi-asserted-by":"publisher","first-page":"917","DOI":"10.1007\/s10618-019-00619-1","volume":"33","author":"HI Fawaz","year":"2019","unstructured":"Fawaz HI, Forestier G, Weber J, Idoumghar L, Muller P-A (2019) Deep learning for time series classification: a review. Data Min Knowl Disc 33(4):917\u2013963","journal-title":"Data Min Knowl Disc"},{"issue":"6","key":"948_CR12","doi-asserted-by":"publisher","first-page":"1936","DOI":"10.1007\/s10618-020-00710-y","volume":"34","author":"HI Fawaz","year":"2020","unstructured":"Fawaz HI, Lucas B, Forestier G, Pelletier C, Schmidt DF, Weber J, Webb GI, Idoumghar L, Muller P-A, Petitjean F (2020) Inceptiontime: finding alexnet for time series classification. Data Min Knowl Disc 34(6):1936\u20131962","journal-title":"Data Min Knowl Disc"},{"key":"948_CR13","unstructured":"Foumani NM, Miller L, Tan CW, Webb GI, Forestier G, Salehi M (2023) Deep learning for time series classification and extrinsic regression: a current survey. arXiv preprint arXiv:2302.02515"},{"key":"948_CR14","doi-asserted-by":"crossref","unstructured":"Foumani SNM, Tan CW, Salehi M (2021) Disjoint-cnn for multivariate time series classification. In: 2021 International Conference on Data Mining Workshops, pp. 760\u2013769","DOI":"10.1109\/ICDMW53433.2021.00099"},{"key":"948_CR15","doi-asserted-by":"crossref","unstructured":"Hao Y, Cao H (2020) A new attention mechanism to classify multivariate time series. In: International joint conference on artificial intelligence","DOI":"10.24963\/ijcai.2020\/277"},{"key":"948_CR16","unstructured":"Huang CZA, Vaswani A, Uszkoreit J, Shazeer N, Simon I, Hawthorne C, Dai AM, Hoffman MD, Dinculescu M, Eck D (2018) Music transformer. arXiv preprint arXiv:1809.04281"},{"key":"948_CR17","doi-asserted-by":"crossref","unstructured":"Huang Z, Liang D, Xu P, Xiang B (2020) Improve transformer models with better relative position embeddings. arXiv preprint arXiv:2009.13658","DOI":"10.18653\/v1\/2020.findings-emnlp.298"},{"key":"948_CR18","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1016\/j.neunet.2019.04.014","volume":"116","author":"F Karim","year":"2019","unstructured":"Karim F, Majumdar S, Darabi H, Harford S (2019) Multivariate lstm-fcns for time series classification. Neural Netw 116:237\u2013245","journal-title":"Neural Netw"},{"key":"948_CR19","doi-asserted-by":"crossref","unstructured":"Kostas D, Aroca-Ouellette S, Rudzicz F (2021) Bendr: using transformers and a contrastive self-supervised learning task to learn from massive amounts of eeg data. Front Hum Neurosci 15","DOI":"10.3389\/fnhum.2021.653659"},{"key":"948_CR20","doi-asserted-by":"crossref","unstructured":"Liang Y, Cao R, Zheng J, Ren J, Gao L (2021) Learning to remove: towards isotropic pre-trained bert embedding. In: International conference on artificial neural networks, pp 448\u2013459","DOI":"10.1007\/978-3-030-86383-8_36"},{"key":"948_CR21","unstructured":"Liu M, Ren S, Ma S, Jiao J, Chen Y, Wang Z, Song W (2021) Gated transformer networks for multivariate time series classification. arXiv preprint arXiv:2103.14438"},{"key":"948_CR22","doi-asserted-by":"crossref","unstructured":"Lockhart JW, Weiss GM, Xue JC, Gallagher ST, Grosner AB, Pulickal TT (2011) Design considerations for the wisdm smart phone-based sensor mining architecture. In: International workshop on knowledge discovery from sensor data, pp 25\u201333","DOI":"10.1145\/2003653.2003656"},{"key":"948_CR23","doi-asserted-by":"crossref","unstructured":"Luong MT, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025","DOI":"10.18653\/v1\/D15-1166"},{"issue":"11","key":"948_CR24","doi-asserted-by":"publisher","first-page":"3211","DOI":"10.1007\/s10994-021-06057-9","volume":"110","author":"M Middlehurst","year":"2021","unstructured":"Middlehurst M, Large J, Flynn M, Lines J, Bostrom A, Bagnall A (2021) Hive-cote 2.0: a new meta ensemble for time series classification. Mach Learn 110(11):3211\u20133243","journal-title":"Mach Learn"},{"key":"948_CR25","doi-asserted-by":"crossref","unstructured":"Middlehurst M, Large J, Bagnall A (2020) The canonical interval forest (cif) classifier for time series classification. In: 2020 IEEE international conference on big data, pp 188\u2013195","DOI":"10.1109\/BigData50022.2020.9378424"},{"key":"948_CR26","doi-asserted-by":"crossref","unstructured":"Ruiz AP, Flynn M, Large J, Middlehurst M, Bagnall A (2020) The great multivariate time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Mining and Knowledge Discovery, pp 1\u201349","DOI":"10.1007\/s10618-020-00727-3"},{"key":"948_CR27","doi-asserted-by":"crossref","unstructured":"Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510\u20134520","DOI":"10.1109\/CVPR.2018.00474"},{"key":"948_CR28","doi-asserted-by":"crossref","unstructured":"Shaw P, Uszkoreit J, Vaswani A (2018) Self-attention with relative position representations. arXiv preprint arXiv:1803.02155","DOI":"10.18653\/v1\/N18-2074"},{"key":"948_CR29","unstructured":"Tan CW, Dempster A, Bergmeir C, Webb GI (2021) Multirocket: effective summary statistics for convolutional outputs in time series classification. arXiv e-prints, 2102"},{"key":"948_CR30","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser \u0141, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30"},{"key":"948_CR31","doi-asserted-by":"crossref","unstructured":"Wang Z, Yan W, Oates T (2017) Time series classification from scratch with deep neural networks: a strong baseline. In: 2017 International joint conference on neural networks, pp 1578\u20131585","DOI":"10.1109\/IJCNN.2017.7966039"},{"key":"948_CR32","doi-asserted-by":"crossref","unstructured":"Wu K, Peng H, Chen M, Fu J, Chao H (2021) Rethinking and improving relative position encoding for vision transformer. In: IEEE\/CVF international conference on computer vision, pp 10033\u201310041","DOI":"10.1109\/ICCV48922.2021.00988"},{"key":"948_CR33","unstructured":"Yang CHH, Tsai YY, Chen PY (2021) Voice2series: Reprogramming acoustic models for time series classification. In: International Conference on Machine Learning, pp. 11808\u201311819"},{"key":"948_CR34","doi-asserted-by":"crossref","unstructured":"Zerveas G, Jayaraman S, Patel D, Bhamidipaty A, Eickhoff C (2021) A transformer-based framework for multivariate time series representation learning. In: SIGKDD conference on knowledge discovery and data mining, pp 2114\u20132124","DOI":"10.1145\/3447548.3467401"}],"container-title":["Data Mining and Knowledge Discovery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-023-00948-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10618-023-00948-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-023-00948-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,17]],"date-time":"2024-01-17T17:51:17Z","timestamp":1705513877000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10618-023-00948-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,5]]},"references-count":34,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,1]]}},"alternative-id":["948"],"URL":"https:\/\/doi.org\/10.1007\/s10618-023-00948-2","relation":{},"ISSN":["1384-5810","1573-756X"],"issn-type":[{"value":"1384-5810","type":"print"},{"value":"1573-756X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,5]]},"assertion":[{"value":"28 November 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 June 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 September 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no competing interests to declare that are relevant to the content of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}