{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,2]],"date-time":"2026-06-02T23:26:52Z","timestamp":1780442812733,"version":"3.54.1"},"reference-count":83,"publisher":"Springer Science and Business Media LLC","issue":"21","license":[{"start":{"date-parts":[[2024,8,20]],"date-time":"2024-08-20T00:00:00Z","timestamp":1724112000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,8,20]],"date-time":"2024-08-20T00:00:00Z","timestamp":1724112000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"NTNU Norwegian University of Science and Technology"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Appl Intell"],"published-print":{"date-parts":[[2024,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This paper addresses the evaluation of multi-step point forecasting models. Currently, deep learning models for multi-step forecasting are evaluated on datasets by selecting one error metric that is aggregated across the time series and the forecast horizon. This approach hides insights that would otherwise be useful for practitioners when evaluating and selecting forecasting models. We propose four novel metrics to provide additional insights when evaluating models: 1) a win-loss metric that shows how models perform across time series in the dataset , allowing the practitioner to check whether the model is superior for all series or just a subset of series. 2) a variance weighted metric that accounts for differences in variance across the seasonal period. It can be used to evaluate models for seasonal datasets such as rush hour traffic prediction, where it is desirable to select the model that performs best during the periods of high uncertainty. 3) a delta horizon metric measuring how much models update their forecast for a period in the future over the forecast horizon. Less change to the forecast means more stability over time and is desirable for most forecasting applications. 4) decomposed errors that relate the forecasting error to trend, seasonality, and noise. Decomposing the errors allows the practitioners to identify for which components the model is making more errors and adjust the model accordingly. To show the applicability of the proposed metrics, we implement four deep learning architectures and conduct experiments on five benchmark datasets. We highlight several use cases for the proposed metrics and discuss the applicability in light of the empirical results.<\/jats:p>","DOI":"10.1007\/s10489-024-05715-4","type":"journal-article","created":{"date-parts":[[2024,8,20]],"date-time":"2024-08-20T05:01:56Z","timestamp":1724130116000},"page":"10490-10515","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Performance metrics for multi-step forecasting measuring win-loss, seasonal variance and forecast stability: an empirical study"],"prefix":"10.1007","volume":"54","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3809-6525","authenticated-orcid":false,"given":"Eivind","family":"Str\u00f8m","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9754-5941","authenticated-orcid":false,"given":"Odd Erik","family":"Gundersen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,8,20]]},"reference":[{"key":"5715_CR1","doi-asserted-by":"publisher","unstructured":"Adya M, Collopy F (1998) How effective are neural networks at forecasting and prediction? A review and evaluation. Journal of Forecasting, 17(5-6):481\u2013495, 9. ISSN 0277-6693. https:\/\/doi.org\/10.1002\/(SICI)1099-131X(1998090)17:5\/6<481::AID-FOR709>3.0.CO;2-Q","DOI":"10.1002\/(SICI)1099-131X(1998090)17:5\/6<481::AID-FOR709>3.0.CO;2-Q"},{"key":"5715_CR2","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1007\/978-1-4612-1694-0_15","volume-title":"Selected Papers of Hirotugu Akaike","author":"H Akaike","year":"1998","unstructured":"Akaike H (1998) Information Theory and an Extension of the Maximum Likelihood Principle. Selected Papers of Hirotugu Akaike. Springer, New York, New York, NY, pp 199\u2013213"},{"key":"5715_CR3","doi-asserted-by":"publisher","unstructured":"Akiba T, Sano S, Yanase T, Ohta T, Koyama M (2019) Optuna: A Next-generation Hyperparameter Optimization Framework. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2623\u20132631, New York, NY, USA, 7. ACM. ISBN 9781450362016. https:\/\/doi.org\/10.1145\/3292500.3330701","DOI":"10.1145\/3292500.3330701"},{"key":"5715_CR4","doi-asserted-by":"publisher","first-page":"142","DOI":"10.1109\/ACOMP.2015.24","volume":"2015","author":"NH An","year":"2016","unstructured":"An NH, Anh DT (2016) Comparison of Strategies for Multi-step-Ahead Prediction of Time Series Using Neural Network. Proceedings - 2015 international conference on advanced computing and applications. ACOMP 2015:142\u2013149. https:\/\/doi.org\/10.1109\/ACOMP.2015.24","journal-title":"ACOMP"},{"key":"5715_CR5","doi-asserted-by":"crossref","unstructured":"Armstrong JS (2001) Evaluating forecasting methods. In: Principles of forecasting, Springer, pp 443\u2013472","DOI":"10.1007\/978-0-306-47630-3_20"},{"key":"5715_CR6","doi-asserted-by":"publisher","unstructured":"Armstrong JS, Fildes R (1995) Correspondence on the selection of error measures for comparisons among forecasting methods. J Forecast 14(1):67\u201371, 1 ISSN 02776693. https:\/\/doi.org\/10.1002\/for.3980140106","DOI":"10.1002\/for.3980140106"},{"key":"5715_CR7","doi-asserted-by":"publisher","unstructured":"Armstrong JS, Collopy F (1992) Error measures for generalizing about forecasting methods: Empirical comparisons. Int J Forecast 8(1):69\u201380, 6 ISSN 01692070. https:\/\/doi.org\/10.1016\/0169-2070(92)90008-W. https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/016920709290008W","DOI":"10.1016\/0169-2070(92)90008-W"},{"key":"5715_CR8","doi-asserted-by":"publisher","unstructured":"Athanasopoulos G, Kourentzes N (2023) On the evaluation of hierarchical forecasts. Int J Forecast 39(4):1502\u20131511 ISSN 0169-2070. https:\/\/doi.org\/10.1016\/j.ijforecast.2022.08.003. https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0169207022001121","DOI":"10.1016\/j.ijforecast.2022.08.003"},{"key":"5715_CR9","unstructured":"Bai S, Kolter JZ, Koltun V (2018) An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. arXiv, 3 2018. ISSN 23318422. http:\/\/arxiv.org\/abs\/1803.01271"},{"key":"5715_CR10","unstructured":"Beitner J (2020) PyTorch Forecasting: Time series forecasting with PyTorch. https:\/\/github.com\/jdb78\/pytorch-forecasting"},{"key":"5715_CR11","doi-asserted-by":"publisher","unstructured":"Taieb SB, Bontempi G, Atiya AF, Sorjamaa A (2012) A review and comparison of strategies for multi-step ahead time series forecasting based on the NN5 forecasting competition. Expert Syst Appl 39(8):7067\u20137083, 6. ISSN 09574174. https:\/\/doi.org\/10.1016\/j.eswa.2012.01.039http:\/\/dx.doi.org\/10.1016\/j.eswa.2012.01.039","DOI":"10.1016\/j.eswa.2012.01.039"},{"key":"5715_CR12","doi-asserted-by":"publisher","unstructured":"B\u00f6se J-H, Flunkert V, Gasthaus J, Januschowski T, Lange D, Salinas D, Schelter S, Seeger M, Wang Y (2017) Probabilistic demand forecasting at scale. Proceedings of the VLDB Endowment, 10(12):1694\u20131705, 8. ISSN 2150-8097. https:\/\/doi.org\/10.14778\/3137765.3137775","DOI":"10.14778\/3137765.3137775"},{"key":"5715_CR13","doi-asserted-by":"publisher","unstructured":"Box GEP, Jenkins GM (1968) Some Recent Advances in Forecasting and Control. Appl Stat 17(2):91. ISSN 00359254. https:\/\/doi.org\/10.2307\/2985674. https:\/\/onlinelibrary.wiley.com\/doi\/10.2307\/2985674","DOI":"10.2307\/2985674"},{"key":"5715_CR14","doi-asserted-by":"publisher","unstructured":"Bustos O, Pomares-Quimbaya A (2020) Stock market movement forecast: A Systematic review. Expert Syst Appl 156, ISSN 09574174. https:\/\/doi.org\/10.1016\/j.eswa.2020.113464","DOI":"10.1016\/j.eswa.2020.113464"},{"key":"5715_CR15","doi-asserted-by":"publisher","unstructured":"Chatfield Chris (1988) Apples, oranges and mean square error. Int J Forecast 4(4):515\u2013518, 1. ISSN 01692070. https:\/\/doi.org\/10.1016\/0169-2070(88)90127-6. https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/0169207088901276","DOI":"10.1016\/0169-2070(88)90127-6"},{"key":"5715_CR16","doi-asserted-by":"crossref","unstructured":"Chatfield Chris (1993) Neural networks: Forecasting breakthrough or passing fad? Int J Forecast 9(1), 1\u20133, 4. ISSN 01692070","DOI":"10.1016\/0169-2070(93)90043-M"},{"key":"5715_CR17","doi-asserted-by":"publisher","unstructured":"Chen C, Twycross J, Garibaldi JM (2017) A new accuracy measure based on bounded relative error for time series forecasting. PLOS ONE, 12(3):3. ISSN 1932-6203. https:\/\/doi.org\/10.1371\/journal.pone.0174202. https:\/\/dx.plos.org\/10.1371\/journal.pone.0174202","DOI":"10.1371\/journal.pone.0174202"},{"key":"5715_CR18","unstructured":"Choi E, Bahadori MT, Kulas JA, Schuetz A, Stewart WF, Sun J (2016) RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism. Adv Neural Inform Process Syst (Nips):3512\u20133520:8. ISSN 10495258"},{"key":"5715_CR19","doi-asserted-by":"publisher","unstructured":"Clements MP, Hendry DF (1993) On the limitations of comparing mean square forecast errors. J Forecast 12(8):617\u2013637, 12. ISSN 02776693. https:\/\/doi.org\/10.1002\/for.3980120802. https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/for.3980120802","DOI":"10.1002\/for.3980120802"},{"key":"5715_CR20","unstructured":"Clements MP, Hendry DF (2001) Explaining the Results of the M3 Forecasting Competition. Int J Forecast 17:550\u2013554. ISSN 0169-2070"},{"key":"5715_CR21","first-page":"3","volume":"6","author":"RB Cleveland","year":"1990","unstructured":"Cleveland RB, Cleveland WS, McRae JE, Terpenning I (1990) STL: A Seasonal-Trend Decomposition Procedure Based on Loess. J Off Stat 6:3\u201373","journal-title":"J Off Stat"},{"key":"5715_CR22","doi-asserted-by":"publisher","unstructured":"Crone SF, Hibon M, Nikolopoulos K (2011) Advances in forecasting with neural networks? Empirical evidence from the NN3 competition on time series prediction. International Journal of Forecasting, 27(3):635\u2013660, 7. ISSN 01692070. https:\/\/doi.org\/10.1016\/j.ijforecast.2011.04.001","DOI":"10.1016\/j.ijforecast.2011.04.001"},{"key":"5715_CR23","unstructured":"Dauphin YN, Fan A, Auli M, Grangier D (2016) Language Modeling with Gated Convolutional Networks. 34th International conference on machine learning, ICML 2017, 2:1551\u20131559, 12"},{"key":"5715_CR24","doi-asserted-by":"publisher","unstructured":"De\u00a0Gooijer JG, Hyndman RJ (2006) 25 years of time series forecasting. Int J Forecast 22(3):443\u2013473, 1. ISSN 01692070. https:\/\/doi.org\/10.1016\/j.ijforecast.2006.01.001. https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/S0169207006000021","DOI":"10.1016\/j.ijforecast.2006.01.001"},{"key":"5715_CR25","unstructured":"Devlin J, Chang M-W, Lee K, Toutanova K (2018) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL HLT 2019 - 2019 conference of the north american chapter of the association for computational linguistics: human language technologies-proceedings of the conference, 1:4171\u20134186, 10"},{"issue":"5","key":"5715_CR26","doi-asserted-by":"publisher","first-page":"937","DOI":"10.1080\/01605682.2021.1892464","volume":"73","author":"F Petropoulos","year":"2022","unstructured":"Petropoulos F, Koutsandreas D, Spiliotis E, Assimakopoulos V (2022) On the selection of forecasting accuracy measures. J Oper Res Soc 73(5):937\u2013954. https:\/\/doi.org\/10.1080\/01605682.2021.1892464","journal-title":"J Oper Res Soc"},{"key":"5715_CR27","unstructured":"Dua D, Graff C (2017) UCI Machine Learning Repository. http:\/\/archive.ics.uci.edu\/ml"},{"key":"5715_CR28","unstructured":"Falcon W, Borovec J, W\u00e4lchli A, Eggert N, Schock J, Jordan J, Skafte N, Ir1dXD, Bereznyuk V, Harris E, Murrell T, Yu P, Pr\u00e6sius S, Addair T, Zhong J, Lipin D, Uchida S, Bapat S, Schr\u00f6ter H, Dayma B, Karnachev A, Kulkarni A, Komatsu S, Martin B, Schiratti J-B, Mary H, Byrne D, Cristo E, cinjon, Bakhtin A (2019) PyTorch Lightning. https:\/\/github.com\/PyTorchLightning\/pytorch-lightning"},{"key":"5715_CR29","doi-asserted-by":"publisher","unstructured":"Fildes R, Ord K (2004) Forecasting Competitions: Their Role in Improving Forecasting Practice and Research. In: A companion to economic forecasting, chapter\u00a015, pp 322\u2013353. Wiley, Ltd,. ISBN 9780470996430. https:\/\/doi.org\/10.1002\/9780470996430.ch15","DOI":"10.1002\/9780470996430.ch15"},{"key":"5715_CR30","doi-asserted-by":"publisher","unstructured":"Fildes R, Hibon M, Makridakis S, Meade N (1998) Generalising about univariate forecasting methods: further empirical evidence. International Journal of Forecasting, 14(3):339\u2013358, 9. ISSN 01692070. https:\/\/doi.org\/10.1016\/S0169-2070(98)00009-0","DOI":"10.1016\/S0169-2070(98)00009-0"},{"key":"5715_CR31","doi-asserted-by":"publisher","unstructured":"Fox I, Ang L, Jaiswal M, Pop-Busui R, Wiens J (2018) Deep Multi-Output Forecasting. In: Proceedings of the 24th ACM SIGKDD International conference on knowledge discovery & data mining, pp 1387\u20131395, New York, NY, USA, 7 ACM. ISBN 9781450355520. https:\/\/doi.org\/10.1145\/3219819.3220102","DOI":"10.1145\/3219819.3220102"},{"key":"5715_CR32","doi-asserted-by":"publisher","unstructured":"Philip\u00a0Hans Franses (2016) A note on the Mean Absolute Scaled Error. Int J Forecast 32(1):20\u201322, 1. ISSN 01692070. https:\/\/doi.org\/10.1016\/j.ijforecast.2015.03.008","DOI":"10.1016\/j.ijforecast.2015.03.008"},{"key":"5715_CR33","unstructured":"Gehring J, Auli M, Grangier D, Yarats D, Dauphin YN (2017) Convolutional Sequence to Sequence Learning. 34th International conference on machine learning, ICML 2017, 3:2029\u20132042, 5"},{"key":"5715_CR34","unstructured":"Heber G, Lunde A, Shephard N, Sheppard K (2009) Oxford-Man Institute\u2019s realized library. https:\/\/realized.oxford-man.ox.ac.uk\/"},{"key":"5715_CR35","doi-asserted-by":"publisher","unstructured":"Hochreiter S, Schmidhuber J (1997) Long Short-Term Memory. Neural Comput 9(8):1735\u20131780, 11. ISSN 0899-7667. https:\/\/doi.org\/10.1162\/neco.1997.9.8.1735. https:\/\/direct.mit.edu\/neco\/article\/9\/8\/1735-1780\/6109","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"5715_CR36","doi-asserted-by":"crossref","unstructured":"Hyndman R, Koehler AB, Ord JK, Snyder RD (2008) Forecasting with Exponential Smoothing: The State Space Approach. Springer Series in Statistics, Springer, Berlin Heidelberg. 9783540719182","DOI":"10.1007\/978-3-540-71918-2"},{"key":"5715_CR37","unstructured":"Hyndman R, Athanasopoulos G (2021) Forecasting: Principles and Practice. OTexts, Australia, 3rd edn"},{"key":"5715_CR38","doi-asserted-by":"publisher","unstructured":"Hyndman RJ (2020)A brief history of forecasting competitions. Int J Forecast, 36(1):7\u201314, 1. ISSN 01692070. https:\/\/doi.org\/10.1016\/j.ijforecast.2019.03.015","DOI":"10.1016\/j.ijforecast.2019.03.015"},{"key":"5715_CR39","doi-asserted-by":"publisher","unstructured":"Hyndman RJ, Khandakar Y (2008) Automatic Time Series Forecasting: The forecast Package for R. J Stat Softw, 27(3):22. ISSN 1548-7660. https:\/\/doi.org\/10.18637\/jss.v027.i03","DOI":"10.18637\/jss.v027.i03"},{"key":"5715_CR40","doi-asserted-by":"publisher","unstructured":"Hyndman RJ, Koehler AB (2006) Another look at measures of forecast accuracy. Int J Forecast, 22(4):679\u2013688. ISSN 01692070. https:\/\/doi.org\/10.1016\/j.ijforecast.2006.03.001","DOI":"10.1016\/j.ijforecast.2006.03.001"},{"key":"5715_CR41","doi-asserted-by":"publisher","unstructured":"Inman RH, Pedro HTC, Coimbra CFM (2013) Solar forecasting methods for renewable energy integration. Prog Energy Combust Sci, 39(6):535\u2013576,12. ISSN 03601285. https:\/\/doi.org\/10.1016\/j.pecs.2013.06.002","DOI":"10.1016\/j.pecs.2013.06.002"},{"key":"5715_CR42","doi-asserted-by":"publisher","unstructured":"Kang Y, Hyndman RJ, Smith-Miles K (2017) Visualising forecasting algorithm performance using time series instance spaces. Int J Forecast, 33(2):345\u2013358, 4. ISSN 01692070. https:\/\/doi.org\/10.1016\/j.ijforecast.2016.09.004","DOI":"10.1016\/j.ijforecast.2016.09.004"},{"key":"5715_CR43","doi-asserted-by":"publisher","unstructured":"Kolassa S (2016) Evaluating predictive count data distributions in retail sales forecasting. Int J Forecast, 32(3):788\u2013803, 7. ISSN 01692070. https:\/\/doi.org\/10.1016\/j.ijforecast.2015.12.004","DOI":"10.1016\/j.ijforecast.2015.12.004"},{"key":"5715_CR44","doi-asserted-by":"publisher","unstructured":"Kolassa S (2020) Why the \u201cbest\u201d point forecast depends on the error or accuracy measure. Int J Forecast, 36(1):208\u2013211, 1. ISSN 01692070. https:\/\/doi.org\/10.1016\/j.ijforecast.2019.02.017","DOI":"10.1016\/j.ijforecast.2019.02.017"},{"key":"5715_CR45","doi-asserted-by":"publisher","unstructured":"Kolassa S (2023) Do we want coherent hierarchical forecasts, or minimal MAPEs or MAEs? (We won\u2019t get both!). Int J Forecast, 39 (4):1512\u20131517. ISSN 0169-2070. https:\/\/doi.org\/10.1016\/j.ijforecast.2022.11.006. https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0169207022001492","DOI":"10.1016\/j.ijforecast.2022.11.006"},{"key":"5715_CR46","unstructured":"Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet Classification with Deep Convolutional Neural Networks. In: Advances in Neural Information Processing Systems, vol 25. Curran Associates, Inc"},{"key":"5715_CR47","doi-asserted-by":"publisher","unstructured":"Lai G, Chang W-C, Yang Y, Liu H (2018a) Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, number July, pages 95\u2013104, New York, NY, USA, 6. ACM. ISBN 9781450356572. https:\/\/doi.org\/10.1145\/3209978.3210006. https:\/\/dl.acm.org\/doi\/10.1145\/3209978.3210006","DOI":"10.1145\/3209978.3210006"},{"key":"5715_CR48","doi-asserted-by":"publisher","unstructured":"Lai ZR, Dai DQ, Ren CX, Huang KK (2018) A peak price tracking-based learning system for portfolio selection. IEEE Trans Neural Netw Learn Syst, 29(7):2823\u20132832. ISSN 21622388. https:\/\/doi.org\/10.1109\/TNNLS.2017.2705658","DOI":"10.1109\/TNNLS.2017.2705658"},{"key":"5715_CR49","unstructured":"Laptev N, Yosinski J, Li LE, Smyl S (2017) Time-series Extreme Event Forecasting with Neural Networks at Uber. In: International conference on machine learning (ICML), pp 1\u20135"},{"key":"5715_CR50","unstructured":"Le Guen V, Thome N (2019) Shape and Time Distortion Loss for Training Deep Time Series Forecasting Models. In: Wallach H, Larochelle H, Beygelzimer A, Alch\u00e9-Buc Fd, Fox E, Garnett R (eds), Adv Neural Inf Process Syst, vol 32. Curran Associates, Inc. https:\/\/proceedings.neurips.cc\/paper\/2019\/file\/466accbac9a66b805ba50e42ad715740-Paper.pdf"},{"key":"5715_CR51","doi-asserted-by":"publisher","unstructured":"Lei M, Shiyan L, Chuanwen J, Hongling L, Yan Z (2009) A review on the forecasting of wind speed and generated power. Renew Sust Energ Rev, 13(4):915\u2013920, 5. ISSN 13640321. https:\/\/doi.org\/10.1016\/j.rser.2008.02.002. https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/S1364032108000282","DOI":"10.1016\/j.rser.2008.02.002"},{"key":"5715_CR52","unstructured":"Li S, Jin X, Xuan Y, Zhou X, Chen W, Wang YX, Yan X (2019) Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Adv Neural Inf Process Syst, 32(NeurIPS). ISSN 10495258"},{"key":"5715_CR53","doi-asserted-by":"crossref","unstructured":"Lim B, Zohren S (2021) Time-series forecasting with deep learning: a survey. Philosophical transactions of the royal society a: mathematical, physical and engineering sciences 379(2194):20200209","DOI":"10.1098\/rsta.2020.0209"},{"key":"5715_CR54","doi-asserted-by":"publisher","unstructured":"Lim B, Ar\u0131k S\u00d6, Loeff N, Pfister T (2021) Temporal Fusion Transformers for interpretable multi-horizon time series forecasting. Int J Forecast, 37(4):1748\u20131764. ISSN 0169-2070. https:\/\/doi.org\/10.1016\/j.ijforecast.2021.03.012. https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0169207021000637","DOI":"10.1016\/j.ijforecast.2021.03.012"},{"issue":"2","key":"5715_CR55","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1002\/for.3980010202","volume":"1","author":"S Makridakis","year":"1982","unstructured":"Makridakis S, Andersen A, Carbone R, Fildes R, Hibon M, Lewandowski R, Newton J, Parzen E, Winkler R (1982) The accuracy of extrapolation (time series) methods: Results of a forecasting competition. J Forecast 1(2):111\u2013153. https:\/\/doi.org\/10.1002\/for.3980010202","journal-title":"J Forecast"},{"key":"5715_CR56","unstructured":"Makridakis S, Spiliotis E, Assimakopoulos V (2020) The M5 accuracy competition: Results, findings and conclusions. Int J Forecast"},{"key":"5715_CR57","doi-asserted-by":"publisher","unstructured":"Makridakis S (1993) Accuracy measures: theoretical and practical concerns. Int J Forecast, 9(4):527\u2013529, 12. ISSN 01692070. https:\/\/doi.org\/10.1016\/0169-2070(93)90079-3. https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/0169207093900793","DOI":"10.1016\/0169-2070(93)90079-3"},{"key":"5715_CR58","doi-asserted-by":"publisher","unstructured":"Makridakis S, Hibon M (2000) The M3-Competition: results, conclusions and implications. Int J Forecast, 16(4):451\u2013476, 10. ISSN 01692070. https:\/\/doi.org\/10.1016\/S0169-2070(00)00057-1. https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/S0169207000000571","DOI":"10.1016\/S0169-2070(00)00057-1"},{"key":"5715_CR59","doi-asserted-by":"publisher","unstructured":"Makridakis S, Spiliotis E, Assimakopoulos V (2018) Statistical and Machine Learning forecasting methods: Concerns and ways forward. PLOS ONE, 13(3), 3. ISSN 1932-6203. https:\/\/doi.org\/10.1371\/journal.pone.0194889. https:\/\/dx.plos.org\/10.1371\/journal.pone.0194889","DOI":"10.1371\/journal.pone.0194889"},{"key":"5715_CR60","doi-asserted-by":"publisher","unstructured":"Mudelsee M (2019) Trend analysis of climate time series: A review of methods. Earth Sci Rev, 190(December 2018):310\u2013322. ISSN 00128252. https:\/\/doi.org\/10.1016\/j.earscirev.2018.12.005","DOI":"10.1016\/j.earscirev.2018.12.005"},{"key":"5715_CR61","doi-asserted-by":"publisher","unstructured":"Parmezan ARS, Souza VMA, Batista GEAPA (2019) Evaluation of statistical and machine learning models for time series prediction: Identifying the state-of-the-art and the best conditions for the use of each model. Inf Sci, 484:302\u2013337, 5. ISSN 00200255. https:\/\/doi.org\/10.1016\/j.ins.2019.01.076","DOI":"10.1016\/j.ins.2019.01.076"},{"key":"5715_CR62","unstructured":"Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Kopf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S (2019) PyTorch: An Imperative Style, High-Performance Deep Learning Library. In: Advances in neural information processing systems 32, pp 8024\u20138035. Curran Associates, Inc. http:\/\/papers.neurips.cc\/paper\/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf"},{"key":"5715_CR63","doi-asserted-by":"publisher","unstructured":"Qin Y, Song D, Chen H, Cheng W, Jiang G, Cottrell GW (2017) A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction. In: Proceedings of the twenty-sixth international joint conference on artificial intelligence, IJCAI-17, pp 2627\u20132633. https:\/\/doi.org\/10.24963\/ijcai.2017\/366","DOI":"10.24963\/ijcai.2017\/366"},{"key":"5715_CR64","unstructured":"Rangapuram SS, Seeger MW, Gasthaus J, Stella L, Wang Y, Januschowski T (2018) Deep State Space Models for Time Series Forecasting. In: Bengio S, Wallach H, Larochelle H, Grauman K, Cesa-Bianchi N, Garnett R (eds) Adv Neural Inf Proc Syst, vol 31. Curran Associates, Inc. https:\/\/proceedings.neurips.cc\/paper\/2018\/file\/5cf68969fb67aa6082363a6d4e6468e2-Paper.pdf"},{"key":"5715_CR65","doi-asserted-by":"publisher","unstructured":"Salinas D, Flunkert V, Gasthaus J (2017) DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks. Int J Forecast, 36(3):1181\u20131191, 4. ISSN 01692070. https:\/\/doi.org\/10.1016\/j.ijforecast.2019.07.001. https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/S0169207019301888","DOI":"10.1016\/j.ijforecast.2019.07.001"},{"issue":"1","key":"5715_CR66","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1080\/00031305.1990.10475690","volume":"44","author":"NC Schwertman","year":"1990","unstructured":"Schwertman NC, Gilks AJ, Cameron J (1990) A Simple Noncalculus Proof That the Median Minimizes the Sum of the Absolute Deviations. Am Stat 44(1):38\u201339. https:\/\/doi.org\/10.1080\/00031305.1990.10475690","journal-title":"Am Stat"},{"key":"5715_CR67","unstructured":"Seeger MW, Salinas D, Flunkert V (2016) Bayesian Intermittent Demand Forecasting for Large Inventories. In: Lee D, Sugiyama M, Luxburg U, Guyon I, Garnett R (eds), Advances in neural information processing systems, vol\u00a029. Curran Associates, Inc. https:\/\/proceedings.neurips.cc\/paper\/2016\/file\/03255088ed63354a54e0e5ed957e9008-Paper.pdf"},{"key":"5715_CR68","doi-asserted-by":"publisher","unstructured":"Silver D, Huang A, Maddison CJ, Guez A, Sifre L, van\u00a0den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M, Dieleman S, Grewe D, Nham J, Kalchbrenner N, Sutskever I, Lillicrap T, Leach M, Kavukcuoglu K, Graepel T, Hassabis D (2016) Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484\u2013489, 1. ISSN 0028-0836. https:\/\/doi.org\/10.1038\/nature16961","DOI":"10.1038\/nature16961"},{"key":"5715_CR69","unstructured":"Sj\u00e4lander M, Jahre M, Tufte G, Reissmann N (2019) EPIC: An Energy-Efficient, High-Performance GPGPU Computing Research Infrastructure"},{"key":"5715_CR70","unstructured":"Sutskever I, Vinyals O, Le QV (2014) Sequence to Sequence Learning with Neural Networks. Adv Neural Inform Process Syst 4(January):3104\u20133112, 9. ISSN 10495258"},{"key":"5715_CR71","doi-asserted-by":"publisher","unstructured":"Taieb SB, Atiya AmirF (2016) A Bias and Variance Analysis for Multistep-Ahead Time Series Forecasting. IEEE Trans Neural Netw Learn Syst 27(1):62\u201376, 1. ISSN 2162-237X. https:\/\/doi.org\/10.1109\/TNNLS.2015.2411629. http:\/\/ieeexplore.ieee.org\/document\/7064712\/","DOI":"10.1109\/TNNLS.2015.2411629"},{"key":"5715_CR72","doi-asserted-by":"publisher","unstructured":"Topol EJ (2019) High-performance medicine: the convergence of human and artificial intelligence. Nature Medicine, 25(1):44\u201356, 2019. ISSN 1546170X. https:\/\/doi.org\/10.1038\/s41591-018-0300-7","DOI":"10.1038\/s41591-018-0300-7"},{"key":"5715_CR73","doi-asserted-by":"publisher","unstructured":"Vallance L, Charbonnier B, Paul N, Dubost S, Blanc P (2017) Towards a standardized procedure to assess solar forecast accuracy: A new ramp and time alignment metric. Solar Energy 150:408\u2013422, 7. ISSN 0038092X. https:\/\/doi.org\/10.1016\/j.solener.2017.04.064https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/S0038092X17303687","DOI":"10.1016\/j.solener.2017.04.064"},{"key":"5715_CR74","doi-asserted-by":"publisher","unstructured":"Van\u00a0Belle J, Crevits R, Verbeke W (2023) Improving forecast stability using deep learning. International Journal of Forecasting, 39(3):1333\u20131350. ISSN 0169-2070. https:\/\/doi.org\/10.1016\/j.ijforecast.2022.06.007https:\/\/www.sciencedirect.com\/science\/article\/pii\/S016920702200098X","DOI":"10.1016\/j.ijforecast.2022.06.007"},{"key":"5715_CR75","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser \u0141, Polosukhin I (2017) Attention is All you Need. In: Guyon I, Von Luxburg U, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds), Advances in neural information processing systems, vol\u00a030. Curran Associates, Inc. https:\/\/proceedings.neurips.cc\/paper\/2017\/file\/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf"},{"key":"5715_CR76","doi-asserted-by":"publisher","unstructured":"Vlahogianni EI, Karlaftis MG, Golias JC (2014) Short-term traffic forecasting: Where we are and where we\u2019re going. Trans Res Part C: merg Technol 43:3\u201319, 6. ISSN 0968090X. https:\/\/doi.org\/10.1016\/j.trc.2014.01.005","DOI":"10.1016\/j.trc.2014.01.005"},{"key":"5715_CR77","unstructured":"Wang Y, Smola A, Maddix D, Gasthaus J, Foster D, Januschowski T (2019) Deep Factors for Forecasting. In: Chaudhuri K, Salakhutdinov R (eds), Proceedings of the 36th international conference on machine learning, vol\u00a097 of Proceedings of machine learning research, pp 6607\u20136617. PMLR, 4. https:\/\/proceedings.mlr.press\/v97\/wang19k.html"},{"key":"5715_CR78","doi-asserted-by":"publisher","unstructured":"Ward JA, Lukowicz P, Gellersen HW (2011) Performance metrics for activity recognition. ACM Trans Intell Syst Technol 2(1):1\u201323, 1. ISSN 2157-6904. https:\/\/doi.org\/10.1145\/1889681.1889687https:\/\/dl.acm.org\/doi\/10.1145\/1889681.1889687","DOI":"10.1145\/1889681.1889687"},{"key":"5715_CR79","unstructured":"Wen R, Torkkola K, Narayanaswamy B, Madeka D (2017) A Multi-Horizon Quantile Recurrent Forecaster. NIPS, 11. ISSN 23318422"},{"key":"5715_CR80","doi-asserted-by":"publisher","unstructured":"Young T, Hazarika D, Poria S, Cambria E (2017) Recent Trends in Deep Learning Based Natural Language Processing. IEEE Computational Intelligence Magazine, 13(3):55\u201375, 8. ISSN 1556-603X. https:\/\/doi.org\/10.1109\/MCI.2018.2840738","DOI":"10.1109\/MCI.2018.2840738"},{"key":"5715_CR81","unstructured":"Yu HF, Rao N, Dhillon IS (2016) Temporal regularized matrix factorization for high-dimensional time series prediction. In: Advances in neural information processing systems, pp 847\u2013855"},{"key":"5715_CR82","doi-asserted-by":"publisher","unstructured":"Zhang J, Florita A, Hodge B-M, Lu S, Hamann HF, Banunarayanan V, Brockway AM (2015) A suite of metrics for assessing the performance of solar power forecasting. Solar Energy, 111:157\u2013175, 1. ISSN 0038092X. https:\/\/doi.org\/10.1016\/j.solener.2014.10.016","DOI":"10.1016\/j.solener.2014.10.016"},{"key":"5715_CR83","doi-asserted-by":"crossref","unstructured":"Zhou H, Zhang S, Peng J, Zhang S, Li J, Xiong H, Zhang W (2021) Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. In: The Thirty-fifth AAAI conference on artificial intelligence, AAAI 2021, virtual conference, AAAI Press, vol\u00a035, pp 11106\u201311115","DOI":"10.1609\/aaai.v35i12.17325"}],"container-title":["Applied Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10489-024-05715-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10489-024-05715-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10489-024-05715-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,18]],"date-time":"2024-09-18T13:57:31Z","timestamp":1726667851000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10489-024-05715-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,20]]},"references-count":83,"journal-issue":{"issue":"21","published-print":{"date-parts":[[2024,11]]}},"alternative-id":["5715"],"URL":"https:\/\/doi.org\/10.1007\/s10489-024-05715-4","relation":{},"ISSN":["0924-669X","1573-7497"],"issn-type":[{"value":"0924-669X","type":"print"},{"value":"1573-7497","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,20]]},"assertion":[{"value":"27 July 2024","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 August 2024","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}