{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,17]],"date-time":"2026-04-17T15:32:37Z","timestamp":1776439957331,"version":"3.51.2"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"10","license":[{"start":{"date-parts":[[2025,9,9]],"date-time":"2025-09-09T00:00:00Z","timestamp":1757376000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,9,9]],"date-time":"2025-09-09T00:00:00Z","timestamp":1757376000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100006752","name":"Universidade do Porto","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100006752","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2025,10]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Accurate evaluation of forecasting models is essential for ensuring reliable predictions. Current practices for evaluating and comparing forecasting models focus on summarising performance into a single score, using metrics such as SMAPE. While convenient, averaging performance over all samples dilutes relevant information about model behaviour under varying conditions. This limitation is especially problematic for time series forecasting, where multiple layers of averaging\u2013across time steps, horizons, and multiple time series in a dataset\u2013can mask relevant performance variations. We address this limitation by proposing ModelRadar, a framework for evaluating univariate time series forecasting models across multiple aspects, such as stationarity, presence of anomalies, or forecasting horizons. We demonstrate the advantages of this framework by comparing 24 forecasting methods, including classical approaches and different machine learning algorithms. PatchTST, a state-of-the-art transformer-based neural network architecture, performs best overall but its superiority varies with forecasting conditions. For instance, concerning the forecasting horizon, we found that PatchTST (and also other neural networks) only outperforms classical approaches for multi-step ahead forecasting. Another relevant insight is that classical approaches such as ETS or Theta are notably more robust in the presence of anomalies. These and other findings highlight the importance of aspect-based model evaluation for both practitioners and researchers. ModelRadar is available as a Python package.<\/jats:p>","DOI":"10.1007\/s10994-025-06877-z","type":"journal-article","created":{"date-parts":[[2025,9,9]],"date-time":"2025-09-09T21:43:34Z","timestamp":1757454214000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Modelradar: aspect-based forecast evaluation"],"prefix":"10.1007","volume":"114","author":[{"given":"Vitor","family":"Cerqueira","sequence":"first","affiliation":[]},{"given":"Luis","family":"Roque","sequence":"additional","affiliation":[]},{"given":"Carlos","family":"Soares","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2025,9,9]]},"reference":[{"issue":"5\u20136","key":"6877_CR1","doi-asserted-by":"publisher","first-page":"594","DOI":"10.1080\/07474938.2010.481556","volume":"29","author":"NK Ahmed","year":"2010","unstructured":"Ahmed, N. K., Atiya, A. F., Gayar, N. E., & El-Shishiny, H. (2010). An empirical comparison of machine learning models for time series forecasting. Econometric Reviews, 29(5\u20136), 594\u2013621.","journal-title":"Econometric Reviews"},{"issue":"4","key":"6877_CR2","doi-asserted-by":"publisher","first-page":"521","DOI":"10.1016\/S0169-2070(00)00066-2","volume":"16","author":"V Assimakopoulos","year":"2000","unstructured":"Assimakopoulos, V., & Nikolopoulos, K. (2000). The theta model: a decomposition approach to forecasting. International Journal of Forecasting, 16(4), 521\u2013530.","journal-title":"International Journal of Forecasting"},{"issue":"3","key":"6877_CR3","doi-asserted-by":"publisher","first-page":"822","DOI":"10.1016\/j.ijforecast.2010.04.009","volume":"27","author":"G Athanasopoulos","year":"2011","unstructured":"Athanasopoulos, G., Hyndman, R. J., Song, H., & Wu, D. C. (2011). The tourism forecasting competition. International Journal of Forecasting, 27(3), 822\u2013844.","journal-title":"International Journal of Forecasting"},{"key":"6877_CR4","unstructured":"Bai, S., Kolter, J. Z., & Koltun, V. (2018). An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271."},{"key":"6877_CR5","doi-asserted-by":"crossref","unstructured":"Bontempi, G., Ben\u00a0Taieb, S., & Le\u00a0Borgne, Y. A. (2013). Machine learning strategies for time series forecasting. In Business Intelligence: Second European Summer School, eBISS 2012, Brussels, Belgium, July 15\u201321, 2012, Tutorial Lectures 2 pp. 62\u201377.","DOI":"10.1007\/978-3-642-36318-4_3"},{"key":"6877_CR6","doi-asserted-by":"crossref","unstructured":"Cerqueira, V., Roque, L., & Soares, C. (2024). Forecasting with deep learning: Beyond average of average of average performance. In International Conference on Discovery Science, pp. 135\u2013149. Springer.","DOI":"10.1007\/978-3-031-78977-9_9"},{"issue":"2","key":"6877_CR7","doi-asserted-by":"publisher","first-page":"415","DOI":"10.1007\/s10844-022-00713-9","volume":"59","author":"V Cerqueira","year":"2022","unstructured":"Cerqueira, V., Torgo, L., & Soares, C. (2022). A case study comparing machine learning with statistical methods for time series forecasting: Size matters. Journal of Intelligent Information Systems, 59(2), 415\u2013433.","journal-title":"Journal of Intelligent Information Systems"},{"issue":"7","key":"6877_CR8","doi-asserted-by":"publisher","first-page":"10073","DOI":"10.1007\/s11063-023-11239-8","volume":"55","author":"V Cerqueira","year":"2023","unstructured":"Cerqueira, V., Torgo, L., & Soares, C. (2023). Model selection for time series forecasting an empirical analysis of multiple estimators. Neural Processing Letters, 55(7), 10073\u201310091.","journal-title":"Neural Processing Letters"},{"key":"6877_CR9","doi-asserted-by":"crossref","unstructured":"Challu, C., Olivares, K. G., Oreshkin, B. N., Ramirez, F. G., Canseco, M. M., & Dubrawski, A. (2023). Nhits: Neural hierarchical interpolation for time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, vol.\u00a037, pp. 6989\u20136997.","DOI":"10.1609\/aaai.v37i6.25854"},{"key":"6877_CR10","unstructured":"Chang, S., Zhang, Y., Han, W., Yu, M., Guo, X., Tan, W., Cui, X., Witbrock, M., Hasegawa-Johnson, M.A., & Huang, T. S. (2017) Dilated recurrent neural networks. Advances in Neural Information Processing Systems, 30."},{"issue":"1","key":"6877_CR11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1002\/for.3980040103","volume":"4","author":"ES Gardner Jr","year":"1985","unstructured":"Gardner, E. S., Jr. (1985). Exponential smoothing: The state of the art. Journal of forecasting, 4(1), 1\u201328.","journal-title":"Journal of forecasting"},{"issue":"107","key":"6877_CR12","first-page":"518","volume":"233","author":"R Godahewa","year":"2021","unstructured":"Godahewa, R., Bandara, K., Webb, G. I., Smyl, S., & Bergmeir, C. (2021). Ensembles of localised models for time series forecasting. Knowledge-Based Systems, 233(107), 518.","journal-title":"Knowledge-Based Systems"},{"key":"6877_CR13","unstructured":"Gong, Z., Tang, Y., & Liang, J. (2023). Patchmixer: A patch-mixing architecture for long-term time series forecasting. arXiv preprint arXiv:2310.00655."},{"key":"6877_CR14","unstructured":"Han, X., Zhang, X., Wu, Y., Zhang, Z., & Wu, Z. (2024). Kan4tsf: Are kan and kan-based models effective for time series forecasting? arXiv preprint arXiv:2408.11306."},{"issue":"2","key":"6877_CR15","doi-asserted-by":"publisher","first-page":"788","DOI":"10.1007\/s10618-022-00894-5","volume":"37","author":"H Hewamalage","year":"2023","unstructured":"Hewamalage, H., Ackermann, K., & Bergmeir, C. (2023). Forecast evaluation for data scientists: Common pitfalls and best practices. Data Mining and Knowledge Discovery, 37(2), 788\u2013832.","journal-title":"Data Mining and Knowledge Discovery"},{"issue":"7","key":"6877_CR16","doi-asserted-by":"publisher","first-page":"1082","DOI":"10.1287\/mnsc.42.7.1082","volume":"42","author":"T Hill","year":"1996","unstructured":"Hill, T., O\u2019Connor, M., & Remus, W. (1996). Neural network models for time series forecasts. Management Science, 42(7), 1082\u20131092.","journal-title":"Management Science"},{"key":"6877_CR17","unstructured":"Hyndman, R. J., & Athanasopoulos, G. (2018) Forecasting: Principles and Practice. OTexts."},{"key":"6877_CR19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v027.i03","volume":"27","author":"RJ Hyndman","year":"2008","unstructured":"Hyndman, R. J., & Khandakar, Y. (2008). Automatic time series forecasting: The forecast package for r. Journal of Statistical Software, 27, 1\u201322.","journal-title":"Journal of Statistical Software"},{"key":"6877_CR18","doi-asserted-by":"crossref","unstructured":"Hyndman, R., Koehler, A. B., Ord, J. K., & Snyder, R. D. (2008). Forecasting with exponential smoothing: The state space approach. Springer Science & Business Media.","DOI":"10.1007\/978-3-540-71918-2"},{"issue":"1","key":"6877_CR20","doi-asserted-by":"publisher","first-page":"167","DOI":"10.1016\/j.ijforecast.2019.05.008","volume":"36","author":"T Januschowski","year":"2020","unstructured":"Januschowski, T., Gasthaus, J., Wang, Y., Salinas, D., Flunkert, V., Bohlke-Schneider, M., & Callot, L. (2020). Criteria for classifying forecasting methods. International Journal of Forecasting, 36(1), 167\u2013177.","journal-title":"International Journal of Forecasting"},{"key":"6877_CR21","unstructured":"Jin, M., Wang, S., Ma, L., Chu, Z., Zhang, J.Y., Shi, X., Chen, P.Y., Liang, Y., Li, Y.F., & Pan, S., et\u00a0al. (2023) Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728."},{"key":"6877_CR22","unstructured":"Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T. Y. (2017) Lightgbm: A highly efficient gradient boosting decision tree. Advances in Neural Information Processing Systems, 30."},{"issue":"5","key":"6877_CR23","doi-asserted-by":"publisher","first-page":"937","DOI":"10.1080\/01605682.2021.1892464","volume":"73","author":"D Koutsandreas","year":"2022","unstructured":"Koutsandreas, D., Spiliotis, E., Petropoulos, F., & Assimakopoulos, V. (2022). On the selection of forecasting accuracy measures. Journal of the Operational Research Society, 73(5), 937\u2013954.","journal-title":"Journal of the Operational Research Society"},{"issue":"2","key":"6877_CR24","doi-asserted-by":"publisher","first-page":"270","DOI":"10.1177\/2515245918771304","volume":"1","author":"JK Kruschke","year":"2018","unstructured":"Kruschke, J. K. (2018). Rejecting or accepting parameter values in Bayesian estimation. Advances in Methods and Practices in Psychological Science, 1(2), 270\u2013280.","journal-title":"Advances in Methods and Practices in Psychological Science"},{"issue":"1\u20133","key":"6877_CR25","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1016\/0304-4076(92)90104-Y","volume":"54","author":"D Kwiatkowski","year":"1992","unstructured":"Kwiatkowski, D., Phillips, P. C., Schmidt, P., & Shin, Y. (1992). Testing the null hypothesis of stationarity against the alternative of a unit root: How sure are we that economic time series have a unit root? Journal of Econometrics, 54(1\u20133), 159\u2013178.","journal-title":"Journal of Econometrics"},{"issue":"4","key":"6877_CR26","doi-asserted-by":"publisher","first-page":"1748","DOI":"10.1016\/j.ijforecast.2021.03.012","volume":"37","author":"B Lim","year":"2021","unstructured":"Lim, B., Ar\u0131k, S. \u00d6., Loeff, N., & Pfister, T. (2021). Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4), 1748\u20131764.","journal-title":"International Journal of Forecasting"},{"key":"6877_CR27","unstructured":"Lin, S., Lin, W., Wu, W., Zhao, F., Mo, R., & Zhang, H. (2023). Segrnn: Segment recurrent neural network for long-term time series forecasting. arXiv preprint arXiv:2308.11200."},{"key":"6877_CR28","unstructured":"Liu, Z., Wang, Y., Vaidya, S., Ruehle, F., Halverson, J., Solja\u010di\u0107, M., Hou, T. LiuY., & Tegmark, M. (2024). Kan: Kolmogorov-arnold networks. arXiv preprint arXiv:2404.19756."},{"issue":"2","key":"6877_CR29","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1002\/for.3980010202","volume":"1","author":"S Makridakis","year":"1982","unstructured":"Makridakis, S., Andersen, A., Carbone, R., Fildes, R., Hibon, M., Lewandowski, R., Newton, J., Parzen, E., & Winkler, R. (1982). The accuracy of extrapolation (time series) methods: Results of a forecasting competition. Journal of Forecasting, 1(2), 111\u2013153.","journal-title":"Journal of Forecasting"},{"issue":"1","key":"6877_CR30","doi-asserted-by":"publisher","first-page":"37","DOI":"10.3233\/RDA-150114","volume":"6","author":"S Makridakis","year":"2016","unstructured":"Makridakis, S., & Bakas, N. (2016). Forecasting and uncertainty: A survey. Risk and Decision Analysis, 6(1), 37\u201364.","journal-title":"Risk and Decision Analysis"},{"issue":"4","key":"6877_CR31","doi-asserted-by":"publisher","first-page":"451","DOI":"10.1016\/S0169-2070(00)00057-1","volume":"16","author":"S Makridakis","year":"2000","unstructured":"Makridakis, S., & Hibon, M. (2000). The m3-competition: Results, conclusions and implications. International Journal of Forecasting, 16(4), 451\u2013476.","journal-title":"International Journal of Forecasting"},{"issue":"4","key":"6877_CR32","doi-asserted-by":"publisher","first-page":"802","DOI":"10.1016\/j.ijforecast.2018.06.001","volume":"34","author":"S Makridakis","year":"2018","unstructured":"Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2018a). The m4 competition: Results, findings, conclusion and way forward. International Journal of Forecasting, 34(4), 802\u2013808.","journal-title":"International Journal of Forecasting"},{"issue":"3","key":"6877_CR33","doi-asserted-by":"publisher","first-page":"e0194,889","DOI":"10.1371\/journal.pone.0194889","volume":"13","author":"S Makridakis","year":"2018","unstructured":"Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2018b). Statistical and machine learning forecasting methods: Concerns and ways forward. PLoS ONE, 13(3), e0194,889.","journal-title":"PLoS ONE"},{"issue":"4","key":"6877_CR34","doi-asserted-by":"publisher","first-page":"1346","DOI":"10.1016\/j.ijforecast.2021.11.013","volume":"38","author":"S Makridakis","year":"2022","unstructured":"Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2022). (2022) M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting, 38(4), 1346\u20131364.","journal-title":"International Journal of Forecasting"},{"key":"6877_CR35","unstructured":"Nie, Y., Nguyen, N.H., Sinthong, P., & Kalagnanam, J. (2022) A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730"},{"key":"6877_CR36","unstructured":"Oreshkin, B.N., Carpov, D., Chapados, N., & Bengio, Y. (2019). N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv:1905.10437"},{"key":"6877_CR37","unstructured":"Rangapuram, S. S., Gasthaus, J., Stella, L., Flunkert, V., Salinas, D., Wang, Y., & Januschowski, T. (2023). Deep non-parametric time series forecaster. arXiv preprint arXiv:2312.14657."},{"key":"6877_CR38","doi-asserted-by":"crossref","unstructured":"Roque, L., Cerqueira, V., Soares, C., & Torgo, L. (2025). Cherry-picking in time series forecasting: How to select datasets to make your model shine. In Proceedings of the AAAI Conference on Artificial Intelligence, vol.\u00a039, pp. 20,192\u201320,199.","DOI":"10.1609\/aaai.v39i19.34224"},{"issue":"3","key":"6877_CR39","doi-asserted-by":"publisher","first-page":"1181","DOI":"10.1016\/j.ijforecast.2019.07.001","volume":"36","author":"D Salinas","year":"2020","unstructured":"Salinas, D., Flunkert, V., Gasthaus, J., & Januschowski, T. (2020). Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3), 1181\u20131191.","journal-title":"International Journal of Forecasting"},{"issue":"1","key":"6877_CR40","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1016\/j.ijforecast.2019.03.017","volume":"36","author":"S Smyl","year":"2020","unstructured":"Smyl, S. (2020). A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. International Journal of Forecasting, 36(1), 75\u201385.","journal-title":"International Journal of Forecasting"},{"issue":"5","key":"6877_CR41","doi-asserted-by":"publisher","first-page":"303","DOI":"10.1177\/003754979105700508","volume":"57","author":"Z Tang","year":"1991","unstructured":"Tang, Z., De Almeida, C., & Fishwick, P. A. (1991). Time series forecasting using neural networks vs. Box-Jenkins methodology. Simulation, 57(5), 303\u2013310.","journal-title":"Simulation"},{"key":"6877_CR42","unstructured":"Triebe, O., Laptev, N., & Rajagopal, R. (2019). Ar-net: A simple auto-regressive neural network for time-series. arXiv preprint arXiv:1911.12436."},{"issue":"3","key":"6877_CR43","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1007\/s10618-005-0039-x","volume":"13","author":"X Wang","year":"2006","unstructured":"Wang, X., Smith, K., & Hyndman, R. (2006). Characteristic-based clustering for time series data. Data Mining and Knowledge Discovery, 13(3), 335\u2013364.","journal-title":"Data Mining and Knowledge Discovery"},{"key":"6877_CR44","doi-asserted-by":"crossref","unstructured":"Yamak, P.T., Yujian, L., & Gadosey, P. K. (2019). A comparison between arima, lstm, and gru for time series forecasting. In Proceedings of the 2019 2nd international conference on algorithms, computing and artificial intelligence, pp. 49\u201355.","DOI":"10.1145\/3377713.3377722"},{"key":"6877_CR45","first-page":"36","volume":"63","author":"L Yardley","year":"2021","unstructured":"Yardley, L., & Petropoulos, F. (2021). Beyond error measures to the utility and cost of the forecasts. Foresight: The International Journal of Applied Forecasting, 63, 36\u201345.","journal-title":"Foresight: The International Journal of Applied Forecasting"},{"key":"6877_CR46","doi-asserted-by":"crossref","unstructured":"Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., & Zhang, W. (2021). Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, vol.\u00a035, pp. 11,106\u201311,115.","DOI":"10.1609\/aaai.v35i12.17325"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-025-06877-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-025-06877-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-025-06877-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,7]],"date-time":"2025-10-07T20:56:30Z","timestamp":1759870590000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-025-06877-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,9]]},"references-count":46,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2025,10]]}},"alternative-id":["6877"],"URL":"https:\/\/doi.org\/10.1007\/s10994-025-06877-z","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,9]]},"assertion":[{"value":"31 March 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 July 2025","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 August 2025","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 September 2025","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no relevant financial or non-financial interests to disclose.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"Not applicable.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}}],"article-number":"229"}}