{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T01:46:02Z","timestamp":1787017562371,"version":"build-2736575974"},"reference-count":48,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2024,2,22]],"date-time":"2024-02-22T00:00:00Z","timestamp":1708560000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Korea Institute of Energy Technology Evaluation and Planning (KETEP) and the Ministry of Trade, Industry & Energy (MOTIE) of the Republic of Korea","award":["20214000000820"],"award-info":[{"award-number":["20214000000820"]}]},{"name":"Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government","award":["RS-2022-00155915"],"award-info":[{"award-number":["RS-2022-00155915"]}]},{"name":"Artificial Intelligence Convergence Innovation Human Resources Development"},{"name":"USA NSF CISE","award":["2038029, 2302720, and 2312758"],"award-info":[{"award-number":["2038029, 2302720, and 2312758"]}]},{"name":"IBM Faculty Award, and a CISCO Edge AI grant"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2024,4,30]]},"abstract":"<jats:p>Many real-world applications require precise and fast time-series forecasting. Recent trends in time-series forecasting models are shifting from LSTM-based models to Transformer-based models. However, the Transformer-based model has a limited ability to represent sequential relationships in time-series data. In addition, the transformer-based model suffers from slow training and inference speed due to the bottleneck incurred by a deep encoder and step-by-step decoder inference. To address these problems, we propose a time-series forecasting optimized Transformer model, called TS-Fastformer. TS-Fastformer introduces three new optimizations: First, we propose a Sub Window Tokenizer for compressing input in a simple manner. The Sub Window Tokenizer reduces the length of input sequences to mitigate the complexity of self-attention and enables both single and multi-sequence learning. Second, we propose Time-series Pre-trained Encoder to extract effective representations through pre-training. This optimization enables TS-Fastformer to capture both seasonal and trend representations as well as to mitigate bottlenecks of conventional transformer models. Third, we propose the Past Attention Decoder to forecast target by incorporating past long short-term dependency patterns. Furthermore, Past Attention Decoder achieves high performance improvement by removing a trend distribution that changes over a long period. We evaluate the efficiency of our model with extensive experiments using seven real-world datasets and compare our model to six representative time-series forecasting approaches. The results show that the proposed TS-Fastformer reduces MSE by 10.1% compared to state-of-the-art model and demonstrates 21.6% faster training time compared to the existing fastest transformer, respectively.<\/jats:p>","DOI":"10.1145\/3630637","type":"journal-article","created":{"date-parts":[[2023,10,30]],"date-time":"2023-10-30T17:55:56Z","timestamp":1698688556000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":25,"title":["TS-Fastformer: Fast Transformer for Time-series Forecasting"],"prefix":"10.1145","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7398-2162","authenticated-orcid":false,"given":"Sangwon","family":"Lee","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, Inha University, Incheo, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5035-8260","authenticated-orcid":false,"given":"Junho","family":"Hong","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Michigan-Dearborn, Dearborn, MI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4138-3082","authenticated-orcid":false,"given":"Ling","family":"Liu","sequence":"additional","affiliation":[{"name":"College of Computing, Georgia Institute of Technology, Atlanta, GA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8533-2829","authenticated-orcid":false,"given":"Wonik","family":"Choi","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Inha University, Incheon, Republic of Korea, and DeepCardio LLC., Incheon, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,2,22]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Jimmy Lei Ba Jamie Ryan Kiros and Geoffrey E. Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016)."},{"key":"e_1_3_2_3_2","unstructured":"Shaojie Bai J. Zico Kolter and Vladlen Koltun. 2018. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 (2018)."},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2936597"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01205"},{"key":"e_1_3_2_6_2","unstructured":"Jacob Devlin Ming-Wei Chang Kenton Lee and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)."},{"key":"e_1_3_2_7_2","unstructured":"Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai Thomas Unterthiner Mostafa Dehghani Matthias Minderer Georg Heigold Sylvain Gelly Jakob Uszkoreit and Neil Houlsby. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)."},{"key":"e_1_3_2_8_2","doi-asserted-by":"crossref","unstructured":"Emadeldeen Eldele Mohamed Ragab Zhenghua Chen Min Wu Chee Keong Kwoh Xiaoli Li and Cuntai Guan. 2021. Time-series representation learning via temporal and contextual contrasting. arXiv preprint arXiv:2106.14112 (2021).","DOI":"10.24963\/ijcai.2021\/324"},{"key":"e_1_3_2_9_2","article-title":"Unsupervised scalable representation learning for multivariate time series","volume":"32","author":"Franceschi Jean-Yves","year":"2019","unstructured":"Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. 2019. Unsupervised scalable representation learning for multivariate time series. Adv. Neural Inf. Process. Syst. 32 (2019).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_10_2","unstructured":"Yoav Goldberg and Omer Levy. 2014. Word2vec Explained: Deriving Mikolov et al.\u2019s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722 (2014)."},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_2_12_2","unstructured":"Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415 (2016)."},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.2967900"},{"key":"e_1_3_2_15_2","first-page":"597","volume-title":"Proceedings of the International Conference on Algorithmic Learning Theory. PMLR","author":"Keles Feyza Duman","year":"2023","unstructured":"Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, and Chinmay Hegde. 2023. On the computational complexity of self-attention. In Proceedings of the International Conference on Algorithmic Learning Theory. PMLR, 597\u2013619."},{"key":"e_1_3_2_16_2","unstructured":"Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)."},{"key":"e_1_3_2_17_2","first-page":"1188","volume-title":"Proceedings of the International Conference on Machine Learning. PMLR","author":"Le Quoc","year":"2014","unstructured":"Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the International Conference on Machine Learning. PMLR, 1188\u20131196."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.3390\/en12132538"},{"key":"e_1_3_2_19_2","article-title":"Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting","volume":"32","author":"Li Shiyang","year":"2019","unstructured":"Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. 2019. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Adv. Neural Inf. Process. Syst. 32 (2019).","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"e_1_3_2_20_2","unstructured":"Yaguang Li Rose Yu Cyrus Shahabi and Yan Liu. 2017. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. arXiv preprint arXiv:1707.01926 (2017)."},{"key":"e_1_3_2_21_2","unstructured":"Jiawei Ma Zheng Shou Alireza Zareian Hassan Mansour Anthony Vetro and Shih-Fu Chang. 2019. CDSA: Cross-dimensional self-attention for multivariate geo-tagged time series imputation. arXiv preprint arXiv:1905.09904 (2019)."},{"key":"e_1_3_2_22_2","unstructured":"Pankaj Malhotra Vishnu T. V. Lovekesh Vig Puneet Agarwal and Gautam Shroff. 2017. TimeNet: Pre-trained deep recurrent neural network for time series classification. arXiv preprint arXiv:1706.08838 (2017)."},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623624"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.measurement.2020.108250"},{"key":"e_1_3_2_25_2","unstructured":"Yuqi Nie Nam H. Nguyen Phanwadee Sinthong and Jayant Kalagnanam. 2022. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730 (2022)."},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.rser.2016.09.025"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/1142473.1142545"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3011982"},{"key":"e_1_3_2_29_2","first-page":"281","volume-title":"Proceedings of the ASME International Mechanical Engineering Congress and Exposition","volume":"43888","author":"Rhudy Matthew","year":"2009","unstructured":"Matthew Rhudy, Brian Bucci, Jeffrey Vipperman, Jeffrey Allanach, and Bruce Abraham. 2009. Microphone array analysis methods using crosscorrelations. In Proceedings of the ASME International Mechanical Engineering Congress and Exposition, Vol. 43888. 281\u2013288."},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00349"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1038\/323533a0"},{"key":"e_1_3_2_32_2","doi-asserted-by":"crossref","unstructured":"Peter Shaw Jakob Uszkoreit and Ashish Vaswani. 2018. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155 (2018).","DOI":"10.18653\/v1\/N18-2074"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11635"},{"key":"e_1_3_2_34_2","first-page":"73","article-title":"Francis Galton's account of the invention of correlation","author":"Stigler Stephen M.","year":"1989","unstructured":"Stephen M. Stigler. 1989. Francis Galton's account of the invention of correlation. Statist. Sci. (1989), 73\u201379.","journal-title":"Statist. Sci."},{"key":"e_1_3_2_35_2","article-title":"Sequence to sequence learning with neural networks","volume":"27","author":"Sutskever Ilya","year":"2014","unstructured":"Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. Adv. Neural Inf. Process. Syst. 27 (2014).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_36_2","unstructured":"Sana Tonekaboni Danny Eytan and Anna Goldenberg. 2021. Unsupervised representation learning for time series with temporal neighborhood coding. arXiv preprint arXiv:2106.00750 (2021)."},{"key":"e_1_3_2_37_2","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Adv. Neural Inf. Process. Syst. 30 (2017).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_38_2","volume-title":"Proceedings of the Conference on Advances in Neural Information Processing Systems","author":"Wang Zhiyuan","year":"2022","unstructured":"Zhiyuan Wang, Xovee Xu, Weifeng Zhang, Goce Trajcevski, Ting Zhong, and Fan Zhou. 2022. Learning latent seasonal-trend representations for time series forecasting. In Proceedings of the Conference on Advances in Neural Information Processing Systems."},{"key":"e_1_3_2_39_2","unstructured":"Gerald Woo Chenghao Liu Doyen Sahoo Akshat Kumar and Steven Hoi. 2022. CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. arXiv preprint arXiv:2202.01575 (2022)."},{"key":"e_1_3_2_40_2","first-page":"793","volume-title":"Proceedings of the International Conference on Artificial Intelligence and Statistics","author":"Wu Lingfei","year":"2018","unstructured":"Lingfei Wu, Ian En-Hsu Yen, Jinfeng Yi, Fangli Xu, Qi Lei, and Michael Witbrock. 2018. Random warping series: A random features method for time-series embedding. In Proceedings of the International Conference on Artificial Intelligence and Statistics. PMLR, 793\u2013802."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.3390\/en13246603"},{"key":"e_1_3_2_42_2","unstructured":"Rose Yu Stephan Zheng Anima Anandkumar and Yisong Yue. 2017. Long-term forecasting using tensor-train rnns. arXiv preprint arXiv:1711.00073 (2017)."},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i8.20881"},{"key":"e_1_3_2_44_2","unstructured":"Ailing Zeng Muxi Chen Lei Zhang and Qiang Xu. 2022. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205.13504 (2022)."},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467401"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i12.17325"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2923006"},{"key":"e_1_3_2_48_2","first-page":"27268","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Zhou Tian","year":"2022","unstructured":"Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proceedings of the International Conference on Machine Learning. PMLR, 27268\u201327286."},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1016\/B978-155860869-6\/50039-1"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3630637","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3630637","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T12:36:32Z","timestamp":1750163792000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3630637"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,22]]},"references-count":48,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,4,30]]}},"alternative-id":["10.1145\/3630637"],"URL":"https:\/\/doi.org\/10.1145\/3630637","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,22]]},"assertion":[{"value":"2022-12-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-10-19","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}