{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T15:00:23Z","timestamp":1777734023703,"version":"3.51.4"},"reference-count":101,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2022,12,2]],"date-time":"2022-12-02T00:00:00Z","timestamp":1669939200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,12,2]],"date-time":"2022-12-02T00:00:00Z","timestamp":1669939200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001779","name":"Monash University","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001779","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100005801","name":"Facebook","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100005801","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"publisher","award":["DE190100045"],"award-info":[{"award-number":["DE190100045"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Data Min Knowl Disc"],"published-print":{"date-parts":[[2023,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Recent trends in the Machine Learning (ML) and in particular Deep Learning (DL) domains have demonstrated that with the availability of massive amounts of time series, ML and DL techniques are competitive in time series forecasting. Nevertheless, the different forms of non-stationarities associated with time series challenge the capabilities of data-driven ML models. Furthermore, due to the domain of forecasting being fostered mainly by statisticians and econometricians over the years, the concepts related to forecast evaluation are not the mainstream knowledge among ML researchers. We demonstrate in our work that as a consequence, ML researchers oftentimes adopt flawed evaluation practices which results in spurious conclusions suggesting methods that are not competitive in reality to be seemingly competitive. Therefore, in this work we provide a tutorial-like compilation of the details associated with forecast evaluation. This way, we intend to impart the information associated with forecast evaluation to fit the context of ML, as means of bridging the knowledge gap between traditional methods of forecasting and adopting current state-of-the-art ML techniques.We elaborate the details of the different problematic characteristics of time series such as non-normality and non-stationarities and how they are associated with common pitfalls in forecast evaluation. Best practices in forecast evaluation are outlined with respect to the different steps such as data partitioning, error calculation, statistical testing, and others. Further guidelines are also provided along selecting valid and suitable error measures depending on the specific characteristics of the dataset at hand.<\/jats:p>","DOI":"10.1007\/s10618-022-00894-5","type":"journal-article","created":{"date-parts":[[2022,12,2]],"date-time":"2022-12-02T17:03:48Z","timestamp":1670000628000},"page":"788-832","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":155,"title":["Forecast evaluation for data scientists: common pitfalls and best practices"],"prefix":"10.1007","volume":"37","author":[{"given":"Hansika","family":"Hewamalage","sequence":"first","affiliation":[]},{"given":"Klaus","family":"Ackermann","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3665-9021","authenticated-orcid":false,"given":"Christoph","family":"Bergmeir","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,12,2]]},"reference":[{"key":"894_CR1","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-306-47630-3","volume-title":"Principles of forecasting: a handbook for researchers and practitioners","author":"J Armstrong","year":"2001","unstructured":"Armstrong J (2001) Evaluating forecasting methods. In: Armstrong JS (ed) Principles of forecasting: a handbook for researchers and practitioners. Kluwer Academic Publishers, Norwell, MA"},{"issue":"2","key":"894_CR2","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1287\/mnsc.19.2.211","volume":"19","author":"JS Armstrong","year":"1972","unstructured":"Armstrong JS, Grohman MC (1972) A comparative study of methods for long-range market forecasting. Manag Sci 19(2):211\u2013221","journal-title":"Manag Sci"},{"key":"894_CR3","doi-asserted-by":"crossref","unstructured":"Arnott R, Harvey C\u00a0R, Markowitz H (2019) A backtesting protocol in the era of machine learning. J Financ Data Sci","DOI":"10.2139\/ssrn.3275654"},{"issue":"3","key":"894_CR4","doi-asserted-by":"publisher","first-page":"606","DOI":"10.1007\/s10618-016-0483-9","volume":"31","author":"A Bagnall","year":"2016","unstructured":"Bagnall A, Lines J, Bostrom A, Large J, Keogh E (2016) The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Min Knowl Disc 31(3):606\u2013660","journal-title":"Data Min Knowl Disc"},{"key":"894_CR5","unstructured":"Balestriero R, Pesenti J, LeCun Y (2021) Learning in high dimension always amounts to extrapolation. arXiv preprint arXiv:2110.09485"},{"key":"894_CR6","unstructured":"Bell F, Smyl S, (2018) Forecasting at uber: an introduction. https:\/\/eng.uber.com\/forecasting-introduction\/"},{"issue":"2","key":"894_CR7","doi-asserted-by":"publisher","first-page":"192","DOI":"10.1016\/j.jfineco.2008.10.006","volume":"94","author":"D Berger","year":"2009","unstructured":"Berger D, Chaboud A, Hjalmarsson E (2009) What drives volatility persistence in the foreign exchange market? J Financ Econ 94(2):192\u2013213","journal-title":"J Financ Econ"},{"key":"894_CR8","doi-asserted-by":"publisher","first-page":"70","DOI":"10.1016\/j.csda.2017.11.003","volume":"120","author":"C Bergmeir","year":"2018","unstructured":"Bergmeir C, Hyndman RJ, Koo B (2018) A note on the validity of cross-validation for evaluating autoregressive time series prediction. Comput Stat Data Anal 120:70\u201383","journal-title":"Comput Stat Data Anal"},{"issue":"1","key":"894_CR9","doi-asserted-by":"publisher","first-page":"177","DOI":"10.1016\/j.csda.2006.02.010","volume":"51","author":"JD Berm\u00fadez","year":"2006","unstructured":"Berm\u00fadez JD, Segura JV, Vercher E (2006) A decision support system methodology for forecasting of time series based on soft computing. Comput Stat Data Anal 51(1):177\u2013191","journal-title":"Comput Stat Data Anal"},{"key":"894_CR10","doi-asserted-by":"crossref","unstructured":"Bojer C\u00a0S, Meldgaard J\u00a0P (2020) Kaggle forecasting competitions: an overlooked learning opportunity. Int J Forecast","DOI":"10.1016\/j.ijforecast.2020.07.007"},{"key":"894_CR11","unstructured":"Brownlee J (2020) Data preparation for machine learning: data cleaning, feature selection, and data transforms in Python. Mach Learn Mastery"},{"issue":"11","key":"894_CR12","doi-asserted-by":"publisher","first-page":"1997","DOI":"10.1007\/s10994-020-05910-7","volume":"109","author":"V Cerqueira","year":"2020","unstructured":"Cerqueira V, Torgo L, Mozeti\u010d I (2020) Evaluating time series forecasting models: an empirical study on performance estimation methods. Mach Learn 109(11):1997\u20132028","journal-title":"Mach Learn"},{"key":"894_CR13","unstructured":"Challu C, Olivares K.\u00a0G, Oreshkin B\u00a0N, Garza, F, Mergenthaler-Canseco M, Dubrawski A (2022) N-hits: neural hierarchical interpolation for time series forecasting. arXiv:2201.12886"},{"issue":"3","key":"894_CR14","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0174202","volume":"12","author":"C Chen","year":"2017","unstructured":"Chen C, Twycross J, Garibaldi JM (2017) A new accuracy measure based on bounded relative error for time series forecasting. PLoS ONE 12(3):e0174202","journal-title":"PLoS ONE"},{"key":"894_CR15","unstructured":"Cox D, Miller H (1965) The Theory of Stochastic Processes"},{"key":"894_CR16","doi-asserted-by":"crossref","unstructured":"Cui Y, Xie J, Zheng K (2021) Historical inertia: a neglected but powerful baseline for long sequence time-series forecasting. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management. CIKM \u201921. Association for Computing Machinery, New York, NY, USA, pp 2965-2969","DOI":"10.1145\/3459637.3482120"},{"issue":"3","key":"894_CR17","doi-asserted-by":"publisher","first-page":"510","DOI":"10.1016\/j.ijforecast.2012.09.002","volume":"29","author":"A Davydenko","year":"2013","unstructured":"Davydenko A, Fildes R (2013) Measuring forecasting accuracy: The case of judgmental adjustments to SKU-level demand forecasts. Int J Forecast 29(3):510\u2013522","journal-title":"Int J Forecast"},{"issue":"1","key":"894_CR18","first-page":"1","volume":"7","author":"J Dem\u0161ar","year":"2006","unstructured":"Dem\u0161ar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7(1):1\u201330","journal-title":"J Mach Learn Res"},{"issue":"1","key":"894_CR19","doi-asserted-by":"publisher","first-page":"134","DOI":"10.1198\/073500102753410444","volume":"20","author":"FX Diebold","year":"2002","unstructured":"Diebold FX, Mariano RS (2002) Comparing predictive accuracy. J Bus Econ Stat 20(1):134\u2013144","journal-title":"J Bus Econ Stat"},{"issue":"4","key":"894_CR20","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1109\/MCI.2015.2471196","volume":"10","author":"G Ditzler","year":"2015","unstructured":"Ditzler G, Roveri M, Alippi C, Polikar R (2015) Learning in nonstationary environments: a survey. IEEE Comput Intell Mag 10(4):12\u201325","journal-title":"IEEE Comput Intell Mag"},{"issue":"293","key":"894_CR21","doi-asserted-by":"publisher","first-page":"52","DOI":"10.1080\/01621459.1961.10482090","volume":"56","author":"OJ Dunn","year":"1961","unstructured":"Dunn OJ (1961) Multiple comparisons among means. J Am Stat Assoc 56(293):52\u201364","journal-title":"J Am Stat Assoc"},{"key":"894_CR22","unstructured":"Du D, Su B, Wei Z (2022) Preformer: predictive transformer with multi-scale segment-wise correlations for long-term time series forecasting. arXiv:2202.11356"},{"key":"894_CR23","doi-asserted-by":"crossref","unstructured":"Du Y, Wang J, Feng W, Pan S, Qin T, Xu R, Wang C (2021) Adarnn: adaptive learning and forecasting of time series. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management. CIKM \u201921. Association for Computing Machinery, New York, NY, USA, pp 402-411","DOI":"10.1145\/3459637.3482315"},{"key":"894_CR24","unstructured":"Engle R\u00a0F (2003) Risk and volatility: econometric models and financial practice. Nobel Lect. https:\/\/www.nobelprize.org\/uploads\/2018\/06\/engle-lecture.pdf"},{"issue":"2","key":"894_CR25","doi-asserted-by":"publisher","first-page":"383","DOI":"10.2307\/2325486","volume":"25","author":"EF Fama","year":"1970","unstructured":"Fama EF (1970) Efficient capital markets: a review of theory and empirical work. J Financ 25(2):383\u2013417","journal-title":"J Financ"},{"issue":"4","key":"894_CR26","doi-asserted-by":"publisher","first-page":"917","DOI":"10.1007\/s10618-019-00619-1","volume":"33","author":"HI Fawaz","year":"2019","unstructured":"Fawaz HI, Forestier G, Weber J, Idoumghar L, Muller P-A (2019) Deep learning for time series classification: a review. Data Min Knowl Discov 33(4):917\u2013963","journal-title":"Data Min Knowl Discov"},{"issue":"200","key":"894_CR27","doi-asserted-by":"publisher","first-page":"675","DOI":"10.1080\/01621459.1937.10503522","volume":"32","author":"M Friedman","year":"1937","unstructured":"Friedman M (1937) The use of ranks to avoid the assumption of normality implicit in the analysis of variance. J Am Stat Assoc 32(200):675\u2013701","journal-title":"J Am Stat Assoc"},{"issue":"205","key":"894_CR28","first-page":"109","volume":"34","author":"M Friedman","year":"1939","unstructured":"Friedman M (1939) A correction: the use of ranks to avoid the assumption of normality implicit in the analysis of variance. J Am Stat Assoc 34(205):109\u2013109","journal-title":"J Am Stat Assoc"},{"issue":"1","key":"894_CR29","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1214\/aoms\/1177731944","volume":"11","author":"M Friedman","year":"1940","unstructured":"Friedman M (1940) A comparison of alternative tests of significance for the problem of $$m$$ rankings. Ann Math Stat 11(1):86\u201392","journal-title":"Ann Math Stat"},{"key":"894_CR30","unstructured":"Fry C, Lichtendahl C (2020) Google practitioner session. In: 40th International Symposium on Forecasting. https:\/\/www.youtube.com\/watch?v=FoUX-muLlB4 &t=3007s"},{"issue":"3","key":"894_CR31","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1007\/s10994-012-5320-9","volume":"90","author":"J Gama","year":"2013","unstructured":"Gama J, Sebastiao R, Rodrigues PP (2013) On evaluating stream learning algorithms. Mach Learn 90(3):317\u2013346","journal-title":"Mach Learn"},{"key":"894_CR32","doi-asserted-by":"crossref","unstructured":"Gama J.\u00a0a, \u017dliobaitundefined I, Bifet A, Pechenizkiy M, Bouchachia A (2014) A survey on concept drift adaptation. ACM Comput Surv 46\u00a0(4)","DOI":"10.1145\/2523813"},{"issue":"3","key":"894_CR33","doi-asserted-by":"publisher","first-page":"663","DOI":"10.1007\/s10618-019-00614-6","volume":"33","author":"H Ghomeshi","year":"2019","unstructured":"Ghomeshi H, Gaber MM, Kovalchuk Y (2019) EACD: evolutionary adaptation to concept drifts in data streams. Data Min Knowl Disc 33(3):663\u2013694","journal-title":"Data Min Knowl Disc"},{"issue":"6","key":"894_CR34","doi-asserted-by":"publisher","first-page":"1545","DOI":"10.1111\/j.1468-0262.2006.00718.x","volume":"74","author":"R Giacomini","year":"2006","unstructured":"Giacomini R, White H (2006) Tests of conditional predictive ability. Econometrica 74(6):1545\u20131578","journal-title":"Econometrica"},{"key":"894_CR35","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2021.107518","volume":"233","author":"R Godahewa","year":"2021","unstructured":"Godahewa R, Bandara K, Webb GI, Smyl S, Bergmeir C (2021) Ensembles of localised models for time series forecasting. Knowl Based Syst 233:107518","journal-title":"Knowl Based Syst"},{"issue":"7","key":"894_CR36","first-page":"2973","volume":"29","author":"LB Godfrey","year":"2018","unstructured":"Godfrey LB, Gashler MS (2018) Neural decomposition of time-series data for effective generalization. IEEE Trans Neural Netw Learn Syst 29(7):2973\u20132985","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"key":"894_CR37","volume-title":"Essentials of econometrics","author":"DN Gujarati","year":"2021","unstructured":"Gujarati DN (2021) Essentials of econometrics. Sage Publications, Christchurch, New Zealand"},{"key":"894_CR38","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2022.118218","volume":"208","author":"Y Guo","year":"2022","unstructured":"Guo Y, Zhang S, Yang J, Yu G, Wang Y (2022) Dual memory scale network for multi-step time series forecasting in thermal environment of aquaculture facility: a case study of recirculating aquaculture water temperature. Expert Syst Appl 208:118218","journal-title":"Expert Syst Appl"},{"issue":"269","key":"894_CR39","first-page":"185","volume":"50","author":"M Halperin","year":"1955","unstructured":"Halperin M, Greenhouse SW, Cornfield J, Zalokar J (1955) Tables of percentage points for the studentized maximum absolute deviate in normal samples. J Am Stat Assoc 50(269):185\u2013195","journal-title":"J Am Stat Assoc"},{"key":"894_CR40","doi-asserted-by":"crossref","unstructured":"H\u00e4m\u00e4l\u00e4inen W, Webb G\u00a0I, (2019) A tutorial on statistically sound pattern discovery. Data Min Knowl Discov 33\u00a0(2): 325\u2013377","DOI":"10.1007\/s10618-018-0590-x"},{"key":"894_CR41","doi-asserted-by":"crossref","unstructured":"Hannun A, Guo C, van\u00a0der Maaten L (2021) Measuring data leakage in machine-learning models with fisher information. In: de\u00a0Campos, C, Maathuis, M\u00a0H (eds) Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence. vol 161, pp 760\u2013770","DOI":"10.24963\/ijcai.2022\/736"},{"key":"894_CR42","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-84858-7","volume-title":"The elements of statistical learning: data mining, inference, and prediction","author":"T Hastie","year":"2009","unstructured":"Hastie T, Tibshirani R, Friedman J (2009) The elements of statistical learning: data mining, inference, and prediction. Springer, New York, NY"},{"issue":"1","key":"894_CR43","doi-asserted-by":"publisher","first-page":"388","DOI":"10.1016\/j.ijforecast.2020.06.008","volume":"37","author":"H Hewamalage","year":"2021","unstructured":"Hewamalage H, Bergmeir C, Bandara K (2021) Recurrent neural networks for time series forecasting: current status and future directions. Int J Forecast 37(1):388\u2013427","journal-title":"Int J Forecast"},{"issue":"4","key":"894_CR44","doi-asserted-by":"publisher","first-page":"800","DOI":"10.1093\/biomet\/75.4.800","volume":"75","author":"Y Hochberg","year":"1988","unstructured":"Hochberg Y (1988) A sharper bonferroni procedure for multiple tests of significance. Biometrika 75(4):800\u2013802","journal-title":"Biometrika"},{"issue":"2","key":"894_CR45","first-page":"65","volume":"6","author":"S Holm","year":"1979","unstructured":"Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat 6(2):65\u201370","journal-title":"Scand J Stat"},{"key":"894_CR46","doi-asserted-by":"crossref","unstructured":"Hyndman R\u00a0J, Athanasopoulos G (2018) Forecasting: principles and Practice, 2nd edn. OTexts. https:\/\/otexts.com\/fpp2\/","DOI":"10.32614\/CRAN.package.fpp2"},{"issue":"4","key":"894_CR47","doi-asserted-by":"publisher","first-page":"679","DOI":"10.1016\/j.ijforecast.2006.03.001","volume":"22","author":"RJ Hyndman","year":"2006","unstructured":"Hyndman RJ, Koehler AB (2006) Another look at measures of forecast accuracy. Int J Forecast 22(4):679\u2013688","journal-title":"Int J Forecast"},{"key":"894_CR48","doi-asserted-by":"crossref","unstructured":"Hyndman R, Kang Y, Talagala T, Wang E, Yang Y (2019) tsfeatures: time series feature extraction. R package version 1.0.0. https:\/\/pkg.robjhyndman.com\/tsfeatures\/","DOI":"10.32614\/CRAN.package.tsfeatures"},{"issue":"1","key":"894_CR49","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1007\/s10618-010-0201-y","volume":"23","author":"E Ikonomovska","year":"2010","unstructured":"Ikonomovska E, Gama J, D\u017eeroski S (2010) Learning model trees from evolving data streams. Data Min Knowl Discov 23(1):128\u2013168","journal-title":"Data Min Knowl Discov"},{"issue":"4","key":"894_CR50","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2382577.2382579","volume":"6","author":"S Kaufman","year":"2012","unstructured":"Kaufman S, Rosset S, Perlich C, Stitelman O (2012) Leakage in data mining: Formulation, detection, and avoidance. ACM Trans Knowl Discov Data 6(4):1\u201321","journal-title":"ACM Trans Knowl Discov Data"},{"issue":"3","key":"894_CR51","doi-asserted-by":"publisher","first-page":"669","DOI":"10.1016\/j.ijforecast.2015.12.003","volume":"32","author":"S Kim","year":"2016","unstructured":"Kim S, Kim H (2016) A new metric of absolute percentage error for intermittent demand forecasts. Int J Forecast 32(3):669\u2013679","journal-title":"Int J Forecast"},{"key":"894_CR52","doi-asserted-by":"crossref","unstructured":"Kolassa S (2020) Why the best point forecast depends on the error or accuracy measure. Int J Forecast 36(1):208\u2013211","DOI":"10.1016\/j.ijforecast.2019.02.017"},{"key":"894_CR53","doi-asserted-by":"publisher","first-page":"180","DOI":"10.1016\/j.ijpe.2014.06.007","volume":"156","author":"N Kourentzes","year":"2014","unstructured":"Kourentzes N (2014) On intermittent demand model optimisation and selection. Int J Prod Econ 156:180\u2013190","journal-title":"Int J Prod Econ"},{"key":"894_CR54","unstructured":"Koutsandreas D, Spiliotis E, Petropoulos F, Assimakopoulos V (2021) Aasures. J Oper Res Soc, 1\u201318"},{"key":"894_CR55","unstructured":"Kunst R (2016) Visualization of distance measures implied by forecast evaluation criteria. In: International Symposium on Forecasting 2016. https:\/\/forecasters.org\/wp-content\/uploads\/gravity_forms\/7-621289a708af3e7af65a7cd487aee6eb\/2016\/07\/Kunst_Robert_ISF2016.pdf"},{"key":"894_CR56","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2021.116163","volume":"190","author":"C Kuranga","year":"2022","unstructured":"Kuranga C, Pillay N (2022) A comparative study of nonlinear regression and autoregressive techniques in hybrid with particle swarm optimization for time-series forecasting. Expert Syst Appl 190:116163","journal-title":"Expert Syst Appl"},{"key":"894_CR57","doi-asserted-by":"crossref","unstructured":"Lai G, Chang W.-C, Yang Y, Liu H (2018) Modeling long- and short-term temporal patterns with deep neural networks. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. SIGIR \u201918. Association for Computing Machinery, New York, NY, USA, pp 95-104","DOI":"10.1145\/3209978.3210006"},{"issue":"2","key":"894_CR58","doi-asserted-by":"publisher","first-page":"843","DOI":"10.1093\/restud\/rdab039","volume":"89","author":"J Li","year":"2022","unstructured":"Li J, Liao Z, Quaedvlieg R (2022) Conditional superior predictive ability. Rev Econ Stud 89(2):843\u2013875","journal-title":"Rev Econ Stud"},{"key":"894_CR59","unstructured":"Li B, Du S, Li T, Hu J, Jia Z (2022a) Draformer: differentially reconstructed attention transformer for time-series forecasting. arXiv:2206.05495"},{"key":"894_CR60","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2020.114443","volume":"168","author":"G Lin","year":"2021","unstructured":"Lin G, Lin A, Cao J (2021) Multidimensional knn algorithm based on eemd and complexity measures in financial time series forecasting. Expert Syst Appl 168:114443","journal-title":"Expert Syst Appl"},{"issue":"8","key":"894_CR61","doi-asserted-by":"publisher","first-page":"2879","DOI":"10.1109\/TNNLS.2019.2934110","volume":"31","author":"S Liu","year":"2020","unstructured":"Liu S, Ji H, Wang MC (2020) Nonpooling convolutional neural network forecasting for seasonal time series with trends. IEEE Trans Neural Netw Learn Syst 31(8):2879\u20132888","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"key":"894_CR62","doi-asserted-by":"crossref","unstructured":"Liu Q, Long L, Peng H, Wang J, Yang Q, Song X, Riscos-N\u00fa\u00f1ez A, P\u00e9rez-Jim\u00e9nez M\u00a0J (2021) Gated spiking neural p systems for time series forecasting. IEEE Trans Neural Netw Learn Syst, 1\u201310","DOI":"10.1109\/TNNLS.2021.3134792"},{"issue":"2","key":"894_CR63","doi-asserted-by":"publisher","first-page":"297","DOI":"10.1093\/biomet\/65.2.297","volume":"65","author":"GM Ljung","year":"1978","unstructured":"Ljung GM, Box GEP (1978) On a measure of lack of fit in time series models. Biometrika 65(2):297\u2013303","journal-title":"Biometrika"},{"issue":"6","key":"894_CR64","doi-asserted-by":"publisher","first-page":"1821","DOI":"10.1007\/s10618-019-00647-x","volume":"33","author":"CH Lubba","year":"2019","unstructured":"Lubba CH, Sethi SS, Knaute P, Schultz SR, Fulcher BD, Jones NS (2019) catch22: CAnonical time-series CHaracteristics. Data Min Knowl Disc 33(6):1821\u20131852","journal-title":"Data Min Knowl Disc"},{"issue":"4","key":"894_CR65","doi-asserted-by":"publisher","first-page":"527","DOI":"10.1016\/0169-2070(93)90079-3","volume":"9","author":"S Makridakis","year":"1993","unstructured":"Makridakis S (1993) Accuracy measures: theoretical and practical concerns. Int J Forecast 9(4):527\u2013529","journal-title":"Int J Forecast"},{"issue":"4","key":"894_CR66","doi-asserted-by":"publisher","first-page":"451","DOI":"10.1016\/S0169-2070(00)00057-1","volume":"16","author":"S Makridakis","year":"2000","unstructured":"Makridakis S, Hibon M (2000) The m3-competition: results, conclusions and implications. Int J Forecast 16(4):451\u2013476","journal-title":"Int J Forecast"},{"issue":"1","key":"894_CR67","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1016\/j.ijforecast.2019.04.014","volume":"36","author":"S Makridakis","year":"2020","unstructured":"Makridakis S, Spiliotis E, Assimakopoulos V (2020) The M4 Competition: 100,000 time series and 61 forecasting methods. Int J Forecast 36(1):54\u201374","journal-title":"Int J Forecast"},{"issue":"4","key":"894_CR68","doi-asserted-by":"publisher","first-page":"1346","DOI":"10.1016\/j.ijforecast.2021.11.013","volume":"38","author":"S Makridakis","year":"2022","unstructured":"Makridakis S, Spiliotis E, Assimakopoulos V (2022) M5 accuracy competition: results, findings, and conclusions. Int J Forecast 38(4):1346\u20131364","journal-title":"Int J Forecast"},{"issue":"1","key":"894_CR69","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1214\/aoms\/1177730491","volume":"18","author":"HB Mann","year":"1947","unstructured":"Mann HB, Whitney DR (1947) On a test of whether one of two random variables is stochastically larger than the other. Ann Math Stat 18(1):50\u201360","journal-title":"Ann Math Stat"},{"key":"894_CR70","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2022.117195","volume":"201","author":"H Moon","year":"2022","unstructured":"Moon H, Lee H, Song B (2022) Mixed pooling of seasonality for time series forecasting: an application to pallet transport data. Expert Syst Appl 201:117195","journal-title":"Expert Syst Appl"},{"key":"894_CR71","unstructured":"Nemenyi P (1963) Distribution-free multiple comparisons. In: Ph.D. thesis, Princeton University"},{"issue":"3","key":"894_CR72","doi-asserted-by":"publisher","first-page":"705","DOI":"10.1016\/j.ijforecast.2021.11.001","volume":"38","author":"F Petropoulos","year":"2022","unstructured":"Petropoulos F et al (2022) Forecasting: theory and practice. Int J Forecast 38(3):705\u2013871","journal-title":"Int J Forecast"},{"issue":"6","key":"894_CR73","doi-asserted-by":"publisher","first-page":"914","DOI":"10.1057\/jors.2014.62","volume":"66","author":"F Petropoulos","year":"2015","unstructured":"Petropoulos F, Kourentzes N (2015) Forecast combinations for intermittent demand. J Oper Res Soc 66(6):914\u2013924","journal-title":"J Oper Res Soc"},{"key":"894_CR74","doi-asserted-by":"publisher","DOI":"10.1016\/j.epsr.2022.108885","volume":"214","author":"P Ran","year":"2023","unstructured":"Ran P, Dong K, Liu X, Wang J (2023) Short-term load forecasting based on ceemdan and transformer. Electric Power Sys Res 214:108885","journal-title":"Electric Power Sys Res"},{"issue":"4","key":"894_CR75","doi-asserted-by":"publisher","first-page":"1063","DOI":"10.1257\/jel.51.4.1063","volume":"51","author":"B Rossi","year":"2013","unstructured":"Rossi B (2013) Exchange rate predictability. J Econ Lit 51(4):1063\u20131119","journal-title":"J Econ Lit"},{"issue":"3","key":"894_CR76","doi-asserted-by":"publisher","first-page":"1181","DOI":"10.1016\/j.ijforecast.2019.07.001","volume":"36","author":"D Salinas","year":"2020","unstructured":"Salinas D, Flunkert V, Gasthaus J, Januschowski T (2020) Deepar: probabilistic forecasting with autoregressive recurrent networks. Int J Forecast 36(3):1181\u20131191","journal-title":"Int J Forecast"},{"key":"894_CR77","doi-asserted-by":"publisher","first-page":"274","DOI":"10.1016\/j.knosys.2018.10.041","volume":"164","author":"R Salles","year":"2019","unstructured":"Salles R, Belloze K, Porto F, Gonzalez PH, Ogasawara E (2019) Nonstationary time series transformation methods: an experimental review. Knowl Based Syst 164:274\u2013291","journal-title":"Knowl Based Syst"},{"key":"894_CR78","unstructured":"Shabani A, Abdi A, Meng L, Sylvain T (2022) Scaleformer: iterative multi-scale refining transformers for time series forecasting. arXiv:2206.04038"},{"issue":"24","key":"894_CR79","first-page":"171","volume":"24","author":"M Shcherbakov","year":"2013","unstructured":"Shcherbakov M, Brebels A, Shcherbakova N, Tyukov A, Janovsky T, Kamaev V (2013) A survey of forecast error measures. World Appl Sci J 24(24):171\u2013176","journal-title":"World Appl Sci J"},{"key":"894_CR80","doi-asserted-by":"publisher","first-page":"302","DOI":"10.1016\/j.neucom.2018.12.084","volume":"396","author":"Z Shen","year":"2020","unstructured":"Shen Z, Zhang Y, Lu J, Xu J, Xiao G (2020) A novel time series forecasting model with deep learning. Neurocomputing 396:302\u2013313","journal-title":"Neurocomputing"},{"issue":"8","key":"894_CR81","doi-asserted-by":"publisher","first-page":"1421","DOI":"10.1007\/s10994-019-05815-0","volume":"108","author":"S-Y Shih","year":"2019","unstructured":"Shih S-Y, Sun F-K, Lee H-Y (2019) Temporal pattern attention for multivariate time series forecasting. Mach Learn 108(8):1421\u20131441","journal-title":"Mach Learn"},{"issue":"2","key":"894_CR82","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1111\/j.2517-6161.1974.tb00994.x","volume":"36","author":"M Stone","year":"1974","unstructured":"Stone M (1974) Cross-validatory choice and assessment of statistical predictions. J R Stat Soc Ser B Methodol 36(2):111\u2013147","journal-title":"J R Stat Soc Ser B Methodol"},{"key":"894_CR83","unstructured":"Suilin A (2017) kaggle-web-traffic. Accessed: 2018-11-19. https:\/\/github.com\/Arturus\/kaggle-web-traffic\/"},{"key":"894_CR84","unstructured":"Sun F-K, Boning D\u00a0S (2022) Fredo: frequency domain-based long-term time series forecasting. arXiv:2205.12301"},{"key":"894_CR85","unstructured":"Svetunkov I (2021) Forecasting and analytics with adam. OpenForecast, (version: [current date]). https:\/\/openforecast.org\/adam\/"},{"issue":"2","key":"894_CR86","doi-asserted-by":"publisher","first-page":"303","DOI":"10.1016\/j.ijforecast.2004.10.001","volume":"21","author":"AA Syntetos","year":"2005","unstructured":"Syntetos AA, Boylan JE (2005) The accuracy of intermittent demand estimates. Int J Forecast 21(2):303\u2013314","journal-title":"Int J Forecast"},{"key":"894_CR87","unstructured":"Talagala T\u00a0S (2020) A tool to detect potential data leaks in forecasting competitions. In: International Symposium on Forecasting 2020. https:\/\/thiyanga.netlify.app\/talk\/isf20-talk\/"},{"issue":"4","key":"894_CR88","doi-asserted-by":"publisher","first-page":"437","DOI":"10.1016\/S0169-2070(00)00065-0","volume":"16","author":"LJ Tashman","year":"2000","unstructured":"Tashman LJ (2000) Out-of-sample tests of forecasting accuracy: an analysis and review. Int J Forecast 16(4):437\u2013450","journal-title":"Int J Forecast"},{"issue":"4","key":"894_CR89","doi-asserted-by":"publisher","first-page":"964","DOI":"10.1007\/s10618-015-0448-4","volume":"30","author":"GI Webb","year":"2016","unstructured":"Webb GI, Hyde R, Cao H, Nguyen HL, Petitjean F (2016) Characterizing concept drift. Data Min Knowl Discov 30(4):964\u2013994","journal-title":"Data Min Knowl Discov"},{"key":"894_CR90","unstructured":"Wong L (2019) Error metrics in time series forecasting. In: International Symposium on Forecasting 2019. https:\/\/isf.forecasters.org\/wp-content\/uploads\/gravity_forms\/2-dd30f7ae09136fa695c552259bdb3f99\/2019\/07\/ISF_2019_slides.pdf"},{"key":"894_CR91","unstructured":"Woo G, Liu C, Sahoo D, Kumar A, Hoi S (2022) Etsformer: exponential smoothing transformers for time-series forecasting. arXiv:2202.01381"},{"key":"894_CR92","doi-asserted-by":"crossref","unstructured":"Wu Z, Pan S, Long G, Jiang J, Chang X, Zhang C (2020) Connecting the dots: Multivariate time series forecasting with graph neural networks. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. KDD \u201920. Association for Computing Machinery, New York, NY, USA, pp 753-763","DOI":"10.1145\/3394486.3403118"},{"key":"894_CR93","unstructured":"Wu H, Xu J, Wang J, Long M (2021) Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting. In: Advances in Neural Information Processing Systems"},{"key":"894_CR94","doi-asserted-by":"crossref","unstructured":"Ye J, Liu Z, Du B, Sun L, Li W, Fu Y, Xiong H (2022) Learning the evolutionary and multi-scale graph structure for multivariate time series forecasting. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. KDD \u201922. Association for Computing Machinery, New York, NY, USA, pp 2296-2306","DOI":"10.1145\/3534678.3539274"},{"key":"894_CR95","unstructured":"Zeng A, Chen M, Zhang L, Xu Q (2022) Are transformers effective for time series forecasting?"},{"key":"894_CR96","doi-asserted-by":"publisher","first-page":"234","DOI":"10.1016\/j.neucom.2021.06.051","volume":"459","author":"X Zhang","year":"2021","unstructured":"Zhang X, He K, Bao Y (2021) Error-feedback stochastic modeling strategy for time series forecasting with convolutional neural networks. Neurocomputing 459:234\u2013248","journal-title":"Neurocomputing"},{"key":"894_CR97","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2022.116772","volume":"198","author":"Y Zhou","year":"2022","unstructured":"Zhou Y, Zhang M, Lin K-P (2022) Time series forecasting by the novel gaussian process wavelet self-join adjacent-feedback loop reservoir model. Expert Syst Appl 198:116772","journal-title":"Expert Syst Appl"},{"key":"894_CR98","unstructured":"Zhou T, Ma Z, wang X, Wen Q, Sun L, Yao T, Yin W, Jin R (2022a) Film: frequency improved legendre memory model for long-term time series forecasting. In: Advances in Neural Information Processing Systems. arXiv:2205.08897"},{"key":"894_CR99","unstructured":"Zhou T, Ma Z, Wen Q, Wang X, Sun L, Jin R (2022b) FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In: Chaudhuri K, Jegelka S, Song L, Szepesvari C, Niu G, Sabato S (eds), Proceedings of the 39th International Conference on Machine Learning. Vol. 162 of Proceedings of Machine Learning Research. PMLR, pp 27268\u201327286"},{"key":"894_CR100","doi-asserted-by":"crossref","unstructured":"Zhou H, Zhang S, Peng J, Zhang S, Li J, Xiong H, Zhang W (2021) Informer: Beyond efficient transformer for long sequence time-series forecasting. In: The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference. vol 35. AAAI Press, pp 11106\u201311115","DOI":"10.1609\/aaai.v35i12.17325"},{"key":"894_CR101","unstructured":"Zhou T, Zhu J, Wang X, Ma Z, Wen Q, Sun L, Jin R (2022c) Treedrnet:a robust deep model for long term time series forecasting. arXiv:2206.12106"}],"container-title":["Data Mining and Knowledge Discovery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-022-00894-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10618-022-00894-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-022-00894-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,9]],"date-time":"2024-10-09T21:56:54Z","timestamp":1728511014000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10618-022-00894-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,2]]},"references-count":101,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,3]]}},"alternative-id":["894"],"URL":"https:\/\/doi.org\/10.1007\/s10618-022-00894-5","relation":{},"ISSN":["1384-5810","1573-756X"],"issn-type":[{"value":"1384-5810","type":"print"},{"value":"1573-756X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,2]]},"assertion":[{"value":"4 April 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 November 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 December 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}