{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,12]],"date-time":"2026-02-12T15:08:32Z","timestamp":1770908912573,"version":"3.50.1"},"reference-count":40,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2021,3,23]],"date-time":"2021-03-23T00:00:00Z","timestamp":1616457600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001871","name":"Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia","doi-asserted-by":"publisher","award":["UIDB\/50014\/2020"],"award-info":[{"award-number":["UIDB\/50014\/2020"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001871","name":"Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia","doi-asserted-by":"publisher","award":["PD\/BD\/128166\/2016"],"award-info":[{"award-number":["PD\/BD\/128166\/2016"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001871","name":"Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia","doi-asserted-by":"publisher","award":["POCI-01-0145-FEDER-031356 (PTDC\/CCI-BIO\/31356\/2017)"],"award-info":[{"award-number":["POCI-01-0145-FEDER-031356 (PTDC\/CCI-BIO\/31356\/2017)"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"publisher","award":["RGPIN-2020-05023"],"award-info":[{"award-number":["RGPIN-2020-05023"]}],"id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Canada Research Chairs &amp; Natural Sciences and Engineering Research Council of Canada","award":["N\/A"],"award-info":[{"award-number":["N\/A"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Mathematics"],"abstract":"<jats:p>The increasing use of sensor networks has led to an ever larger number of available spatiotemporal datasets. Forecasting applications using this type of data are frequently motivated by important domains such as environmental monitoring. Being able to properly assess the performance of different forecasting approaches is fundamental to achieve progress. However, traditional performance estimation procedures, such as cross-validation, face challenges due to the implicit dependence between observations in spatiotemporal datasets. In this paper, we empirically compare several variants of cross-validation (CV) and out-of-sample (OOS) performance estimation procedures, using both artificially generated and real-world spatiotemporal datasets. Our results show both CV and OOS reporting useful estimates, but they suggest that blocking data in space and\/or in time may be useful in mitigating CV\u2019s bias to underestimate error. Overall, our study shows the importance of considering data dependencies when estimating the performance of spatiotemporal forecasting models.<\/jats:p>","DOI":"10.3390\/math9060691","type":"journal-article","created":{"date-parts":[[2021,3,23]],"date-time":"2021-03-23T14:08:16Z","timestamp":1616508496000},"page":"691","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":["Evaluation Procedures for Forecasting with Spatiotemporal Data"],"prefix":"10.3390","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9708-1123","authenticated-orcid":false,"given":"Mariana","family":"Oliveira","sequence":"first","affiliation":[{"name":"Department of Computer Science, Faculty of Sciences, University of Porto, Rua Campo Alegre 1055, 4169-007 Porto, Portugal"},{"name":"INESC TEC, Rua do Campo Alegre 1055, 4169-007 Porto, Portugal"}]},{"given":"Lu\u00eds","family":"Torgo","sequence":"additional","affiliation":[{"name":"Faculty of Computer Science, Dalhousie University, 6050 University Av., Halifax, NS B3H 1W5, Canada"}]},{"given":"V\u00edtor","family":"Santos Costa","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Faculty of Sciences, University of Porto, Rua Campo Alegre 1055, 4169-007 Porto, Portugal"},{"name":"INESC TEC, Rua do Campo Alegre 1055, 4169-007 Porto, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Liang, Y., Ke, S., Zhang, J., Yi, X., and Zheng, Y. (2018, January 13\u201319). Geoman: Multi-level attention networks for geo-sensory time series prediction. Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI), Stockholm, Sweden.","DOI":"10.24963\/ijcai.2018\/476"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"956","DOI":"10.1109\/TII.2016.2604758","article-title":"Predictive modeling of PV energy production: How to set up the learning task for a better prediction?","volume":"13","author":"Ceci","year":"2016","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1214\/09-SS054","article-title":"A survey of cross-validation procedures for model selection","volume":"4","author":"Arlot","year":"2010","journal-title":"Stat. Surv."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"601","DOI":"10.1109\/TIT.1979.1056087","article-title":"Distribution-free performance bounds for potential function rules","volume":"25","author":"Devroye","year":"1979","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1016\/j.ins.2011.12.028","article-title":"On the use of cross-validation for time series predictor evaluation","volume":"191","author":"Bergmeir","year":"2012","journal-title":"Inf. Sci."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1111\/j.2517-6161.1974.tb00994.x","article-title":"Cross-validatory choice and assessment of statistical predictions","volume":"36","author":"Stone","year":"1974","journal-title":"J. R. Stat. Soc. B"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"320","DOI":"10.1080\/01621459.1975.10479865","article-title":"The predictive sample reuse method with applications","volume":"70","author":"Geisser","year":"1975","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1906","DOI":"10.1214\/aos\/1176348377","article-title":"Comparison of two bandwidth selectors with dependent errors","volume":"19","author":"Chu","year":"1991","journal-title":"Ann. Stat."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"351","DOI":"10.1093\/biomet\/81.2.351","article-title":"A cross-validatory method for dependent data","volume":"81","author":"Burman","year":"1994","journal-title":"Biometrika"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1016\/S0304-4076(00)00030-0","article-title":"Consistent cross-validatory model-selection for dependent data: Hv-block cross-validation","volume":"99","author":"Racine","year":"2000","journal-title":"J. Econom."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.envsoft.2017.12.001","article-title":"Improving performance of spatio-temporal machine learning models using forward feature selection and target-oriented validation","volume":"101","author":"Meyer","year":"2018","journal-title":"Environ. Model. Softw."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"132","DOI":"10.1016\/j.csda.2014.02.001","article-title":"On the usefulness of cross-validation for directional forecast evaluation","volume":"76","author":"Bergmeir","year":"2014","journal-title":"Comput. Stat. Data Anal."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Cerqueira, V., Torgo, L., Smailovi, J., and Mozet\u01d0, I. (2017, January 19\u201321). A Comparative Study of Performance Estimation Methods for Time Series Forecasting. Proceedings of the IEEE International Conference on Data Science and Advanced Analytics (DSAA), Tokyo, Japan.","DOI":"10.1109\/DSAA.2017.7"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Mozeti\u010d, I., Torgo, L., Cerqueira, V., and Smailovi\u0107, J. (2018). How to evaluate sentiment classifiers for Twitter time-ordered data?. PLoS ONE, 13.","DOI":"10.1371\/journal.pone.0194317"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"913","DOI":"10.1111\/ecog.02881","article-title":"Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure","volume":"40","author":"Roberts","year":"2017","journal-title":"Ecography"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Oliveira, M., Torgo, L., and Santos Costa, V. (2018, January 10\u201314). Evaluation Procedures for Forecasting with Spatio-Temporal Data. Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases (ECML\u2013PKDD), Dublin, Ireland.","DOI":"10.1007\/978-3-030-10925-7_43"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Diggle, P. (2002). Analysis of Longitudinal Data, Oxford University Press.","DOI":"10.1093\/oso\/9780198524847.001.0001"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1214\/ss\/1009213287","article-title":"Nonparametric regression with correlated errors","volume":"16","author":"Opsomer","year":"2001","journal-title":"Stat. Sci."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1016\/S0169-2070(00)00065-0","article-title":"Out-of-sample tests of forecasting accuracy: An analysis and review","volume":"16","author":"Tashman","year":"2000","journal-title":"Int. J. Forecast."},{"key":"ref_20","unstructured":"Torgo, L. (2016). Data Mining with R: Learning with Case Studies, Chapman and Hall\/CRC. [2nd ed.]."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1007577530334","article-title":"Prequential and Cross-Validated Regression Estimation","volume":"33","author":"Modha","year":"1998","journal-title":"Mach. Learn."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Snijders, T.A.B. (1988). On Cross-Validation for Predictor Evaluation in Time Series. On Model Uncertainty and Its Statistical Implications, Springer.","DOI":"10.1007\/978-3-642-61564-1_4"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1215","DOI":"10.5194\/cp-12-1215-2016","article-title":"Estimating unbiased transfer-function performances in spatially structured environments","volume":"12","author":"Trachsel","year":"2016","journal-title":"Clim. Past"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"144","DOI":"10.1016\/j.jhydrol.2006.06.028","article-title":"Geostatistical interpolation of hourly precipitation from rain gauges and radar for a large-scale extreme rainfall event","volume":"332","author":"Haberlandt","year":"2007","journal-title":"J. Hydrol."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Appice, A., Pravilovic, S., Malerba, D., and Lanza, A. (2013). Enhancing regression models with spatio-temporal indicator additions. Congress of the Italian Association for Artificial Intelligence, Springer.","DOI":"10.1007\/978-3-319-03524-6_37"},{"key":"ref_26","unstructured":"Ohashi, O., and Torgo, L. (2012, January 27\u201331). Wind speed forecasting using spatio-temporal indicators. Proceedings of the 20th European Conference on Artificial Intelligence (ECAI), Montpellier, France."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1016\/S0022-1694(96)03062-4","article-title":"Spatial modeling of snow water equivalent using covariances estimated from spatial and geomorphic attributes","volume":"190","author":"Carroll","year":"1997","journal-title":"J. Hydrol."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"35","DOI":"10.2307\/1268381","article-title":"A Three-Stage Iterative Procedure for Space-Time Modeling","volume":"22","author":"Pfeifer","year":"1980","journal-title":"Technometrics"},{"key":"ref_29","unstructured":"Cheysson, F. (2020, December 09). Starma: Modelling Space Time AutoRegressive Moving Average (STARMA) Processes. Available online: https:\/\/CRAN.R-project.org\/package=starma."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1080\/13658816.2017.1381338","article-title":"Leveraging correlation across space and time to interpolate geophysical data via CoKriging","volume":"32","author":"Pravilovic","year":"2018","journal-title":"Int. J. Geogr. Inf. Sci."},{"key":"ref_31","unstructured":"Hengl, T. (2020, December 09). GSIF: Global Soil Information Facilities. Available online: https:\/\/CRAN.R-project.org\/package=GSIF."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1016\/j.spasta.2015.04.001","article-title":"Spatio-temporal interpolation of soil water, temperature, and electrical conductivity in 3D+ T: The Cook Agronomy Farm data set","volume":"14","author":"Gasch","year":"2015","journal-title":"Spat. Stat."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v051.i07","article-title":"spacetime: Spatio-Temporal Data in R","volume":"51","author":"Pebesma","year":"2012","journal-title":"J. Stat. Softw."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Zheng, Y., Liu, F., and Hsieh, H.P. (2013, January 11\u201314). U-Air: When Urban Air Quality Inference Meets Big Data. Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), Chicago, IL, USA.","DOI":"10.1145\/2487575.2488188"},{"key":"ref_35","unstructured":"R Core Team (2017). R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing."},{"key":"ref_36","unstructured":"Milborrow, S. (2020, December 09). Earth: Multivariate Adaptive Regression Splines; R Package Version 5.3.0. Available online: https:\/\/CRAN.R-project.org\/package=earth."},{"key":"ref_37","unstructured":"Therneau, T., Atkinson, B., and Ripley, B. (2020, December 09). rpart: Recursive Partitioning and Regression Trees. R Package Version 4.1-15. Available online: http:\/\/CRAN.R-project.org\/package=rpart."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v077.i01","article-title":"ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R","volume":"77","author":"Wright","year":"2017","journal-title":"J. Stat. Softw."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"248","DOI":"10.32614\/RJ-2016-017","article-title":"Scmamp: Statistical Comparison of Multiple Algorithms in Multiple Problems","volume":"8","author":"Calvo","year":"2016","journal-title":"R J."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"551","DOI":"10.1080\/03610918008812173","article-title":"Stationarity and invertibility regions for low order starma models","volume":"9","author":"Pfeifer","year":"1980","journal-title":"Commun. Stat. Comput."}],"container-title":["Mathematics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-7390\/9\/6\/691\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:39:48Z","timestamp":1760161188000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-7390\/9\/6\/691"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,23]]},"references-count":40,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2021,3]]}},"alternative-id":["math9060691"],"URL":"https:\/\/doi.org\/10.3390\/math9060691","relation":{},"ISSN":["2227-7390"],"issn-type":[{"value":"2227-7390","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,23]]}}}