{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T03:46:55Z","timestamp":1775188015978,"version":"3.50.1"},"reference-count":61,"publisher":"Springer Science and Business Media LLC","issue":"1-3","license":[{"start":{"date-parts":[[2022,3,18]],"date-time":"2022-03-18T00:00:00Z","timestamp":1647561600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,3,18]],"date-time":"2022-03-18T00:00:00Z","timestamp":1647561600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Ann Oper Res"],"published-print":{"date-parts":[[2024,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Proper scoring rules are commonly applied to quantify the accuracy of distribution forecasts. Given an observation they assign a scalar score to each distribution forecast, with the lowest expected score attributed to the true distribution. The energy and variogram scores are two rules that have recently gained some popularity in multivariate settings because their computation does not require a forecast to have parametric density function and so they are broadly applicable. Here we conduct a simulation study to compare the discrimination ability between the energy score and three variogram scores. Compared with other studies, our simulation design is more realistic because it is supported by a historical data set containing commodity prices, currencies and interest rates, and our data generating processes include a diverse selection of models with different marginal distributions, dependence structure, and calibration windows. This facilitates a comprehensive comparison of the performance of proper scoring rules in different settings. To compare the scores we use three metrics: the mean relative score, error rate and a generalized discrimination heuristic. Overall, we find that the variogram score with parameter <jats:inline-formula><jats:alternatives><jats:tex-math>$$p=0.5$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mi>p<\/mml:mi>\n                    <mml:mo>=<\/mml:mo>\n                    <mml:mn>0.5<\/mml:mn>\n                  <\/mml:mrow>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula> outperforms the energy score and the other two variogram scores.<\/jats:p>","DOI":"10.1007\/s10479-022-04611-9","type":"journal-article","created":{"date-parts":[[2022,3,18]],"date-time":"2022-03-18T11:09:43Z","timestamp":1647601783000},"page":"857-883","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Evaluating the discrimination ability of proper multi-variate scoring rules"],"prefix":"10.1007","volume":"334","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1247-0184","authenticated-orcid":false,"given":"C.","family":"Alexander","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"M.","family":"Coulon","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Y.","family":"Han","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"X.","family":"Meng","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,3,18]]},"reference":[{"issue":"1","key":"4611_CR1","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1016\/j.ejor.2018.08.006","volume":"273","author":"C Alexander","year":"2019","unstructured":"Alexander, C., Kaeck, A., & Sumawong, A. (2019). A parsimonious parametric model for generating margin requirements for futures. European Journal of Operational Research, 273(1), 31\u201343.","journal-title":"European Journal of Operational Research"},{"key":"4611_CR2","unstructured":"Alexander, C., Meng, X., & Han, Y. (2021). Static and dynamic models for multivariate distribution forecasts: Proper scoring rule tests of factor-quantile vs. multivariate Garch models. arXiv:2004.14108."},{"issue":"2","key":"4611_CR3","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1198\/073500106000000332","volume":"25","author":"G Amisano","year":"2007","unstructured":"Amisano, G., & Giacomini, R. (2007). Comparing density forecasts via weighted likelihood ratio tests. Journal of Business & Economic Statistics, 25(2), 177\u2013190.","journal-title":"Journal of Business & Economic Statistics"},{"key":"4611_CR4","doi-asserted-by":"crossref","first-page":"106113","DOI":"10.1016\/j.jbankfin.2021.106113","volume":"126","author":"D Anghel","year":"2021","unstructured":"Anghel, D. (2021). Data snooping bias in tests of the relative performance of multiple forecasting models. Journal of Banking and Finance, 126, 106113.","journal-title":"Journal of Banking and Finance"},{"key":"4611_CR5","unstructured":"Bank of International Settlements (2016, April). Triennial Central Bank survey: Foreign exchange turnover in April 2016. Technical report."},{"issue":"3","key":"4611_CR6","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1002\/for.1023","volume":"26","author":"Y Bao","year":"2007","unstructured":"Bao, Y., Lee, T.-H., & Salto\u011flu, B. (2007). Comparing density forecast models. Journal of Forecasting, 26(3), 203\u2013225.","journal-title":"Journal of Forecasting"},{"issue":"3","key":"4611_CR7","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1198\/073500104000000523","volume":"23","author":"L Bauwens","year":"2005","unstructured":"Bauwens, L., & Laurent, S. (2005). A new class of multivariate skew densities, with application to generalized autoregressive conditional heteroscedasticity models. Journal of Business & Economic Statistics, 23(3), 346\u2013354.","journal-title":"Journal of Business & Economic Statistics"},{"issue":"2","key":"4611_CR8","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1287\/deca.1070.0089","volume":"4","author":"JE Bickel","year":"2007","unstructured":"Bickel, J. E. (2007). Some comparisons among quadratic, spherical, and logarithmic scoring rules. Decision Analysis, 4(2), 49\u201365.","journal-title":"Decision Analysis"},{"key":"4611_CR9","unstructured":"Bloomberg. (2017, May). The Bloomberg commodity index family: Index methodology. Technical report."},{"issue":"3","key":"4611_CR10","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1016\/0304-4076(86)90063-1","volume":"31","author":"T Bollerslev","year":"1986","unstructured":"Bollerslev, T. (1986). Generalized autoregressive conditional heteroskedasticity. Journal of Econometrics, 31(3), 307\u2013327.","journal-title":"Journal of Econometrics"},{"issue":"3","key":"4611_CR11","doi-asserted-by":"crossref","first-page":"498","DOI":"10.2307\/2109358","volume":"72","author":"T Bollerslev","year":"1990","unstructured":"Bollerslev, T. (1990). Modelling the coherence in short-run nominal exchange rates: A multivariate generalized ARCH model. Review of Economics and Statistics, 72(3), 498\u2013505.","journal-title":"Review of Economics and Statistics"},{"issue":"2","key":"4611_CR12","first-page":"123","volume":"24","author":"L Breiman","year":"1996","unstructured":"Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123\u2013140.","journal-title":"Machine Learning"},{"key":"4611_CR13","unstructured":"Buja, A., Stuetzle, W., & Shen, Y. (2005). Loss functions for binary class probability estimation and classification: Structure and applications."},{"key":"4611_CR14","doi-asserted-by":"crossref","unstructured":"Cajigas, J.-P., & Urga, G. (2006). Dynamic conditional correlation models with asymmetric multivariate Laplace innovations.","DOI":"10.2139\/ssrn.685946"},{"key":"4611_CR15","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1016\/j.jfs.2016.02.002","volume":"23","author":"J Danielsson","year":"2016","unstructured":"Danielsson, J., James, K., Valenzuela, M., & Zer, I. (2016). Model risk of risk models. Journal of Financial Stability, 23, 79\u201391.","journal-title":"Journal of Financial Stability"},{"key":"4611_CR16","doi-asserted-by":"crossref","unstructured":"Dawid, P.A., & Sebastiani, P. (1999). Coherent dispersion criteria for optimal experimental design. Annals of Statistics, 27, 65\u201381.","DOI":"10.1214\/aos\/1018031101"},{"key":"4611_CR17","doi-asserted-by":"crossref","unstructured":"Diebold, F.X., Gunther, T.A., & Tay, A.S. (1998). Evaluating density forecasts, with applications to financial risk management. International Economic Review, 39, 863\u2013883.","DOI":"10.2307\/2527342"},{"issue":"3","key":"4611_CR18","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1080\/07350015.1995.10524599","volume":"13","author":"FX Diebold","year":"1995","unstructured":"Diebold, F. X., & Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business & Economic Statistics, 13(3), 253\u2013263.","journal-title":"Journal of Business & Economic Statistics"},{"issue":"2","key":"4611_CR19","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1016\/j.ijforecast.2019.07.006","volume":"36","author":"C Diks","year":"2020","unstructured":"Diks, C., & Fang, H. (2020). Comparing density forecasts in a risk management context. International Journal of Forecasting, 36(2), 531\u2013551.","journal-title":"International Journal of Forecasting"},{"key":"4611_CR20","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1016\/j.jedc.2014.08.021","volume":"48","author":"C Diks","year":"2014","unstructured":"Diks, C., Panchenko, V., Sokolinskiy, O., & van Dijk, D. (2014). Comparing the accuracy of multivariate density forecasts in selected regions of the copula support. Journal of Economic Dynamics and Control, 48, 79\u201394.","journal-title":"Journal of Economic Dynamics and Control"},{"issue":"2","key":"4611_CR21","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1016\/j.jeconom.2011.04.001","volume":"163","author":"C Diks","year":"2011","unstructured":"Diks, C., Panchenko, V., & Van Dijk, D. (2011). Likelihood-based scoring rules for comparing density forecasts in tails. Journal of Econometrics, 163(2), 215\u2013230.","journal-title":"Journal of Econometrics"},{"key":"4611_CR22","doi-asserted-by":"crossref","unstructured":"Engle, R.F. (1982). Autoregressive conditional heteroscedasticity with estimates of the variance of United Kingdom inflation. Econometrica: Journal of the Econometric Society, 50, 987\u20131007.","DOI":"10.2307\/1912773"},{"issue":"4","key":"4611_CR23","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1257\/jep.15.4.157","volume":"15","author":"RF Engle","year":"2001","unstructured":"Engle, R. F. (2001). GARCH 101: The use of ARCH\/GARCH models in applied econometrics. Journal of Economic Perspectives, 15(4), 157\u2013168.","journal-title":"Journal of Economic Perspectives"},{"issue":"3","key":"4611_CR24","doi-asserted-by":"crossref","first-page":"339","DOI":"10.1198\/073500102288618487","volume":"20","author":"RF Engle","year":"2002","unstructured":"Engle, R. F. (2002). Dynamic conditional correlation: A simple class of multivariate generalized autoregressive conditional heteroskedasticity models. Journal of Business & Economic Statistics, 20(3), 339\u2013350.","journal-title":"Journal of Business & Economic Statistics"},{"issue":"3","key":"4611_CR25","doi-asserted-by":"crossref","first-page":"955","DOI":"10.1175\/MWR-D-14-00210.1","volume":"143","author":"K Feldmann","year":"2015","unstructured":"Feldmann, K., Scheuerer, M., & Thorarinsdottir, T. L. (2015). Spatial postprocessing of ensemble forecasts for temperature using nonhomogeneous Gaussian regression. Monthly Weather Review, 143(3), 955\u2013971.","journal-title":"Monthly Weather Review"},{"issue":"2","key":"4611_CR26","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1111\/j.1467-9868.2007.00587.x","volume":"69","author":"T Gneiting","year":"2007","unstructured":"Gneiting, T., Balabdaoui, F., & Raftery, A. E. (2007). Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(2), 243\u2013268.","journal-title":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)"},{"issue":"477","key":"4611_CR27","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1198\/016214506000001437","volume":"102","author":"T Gneiting","year":"2007","unstructured":"Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359\u2013378.","journal-title":"Journal of the American Statistical Association"},{"issue":"3","key":"4611_CR28","doi-asserted-by":"crossref","first-page":"411","DOI":"10.1198\/jbes.2010.08110","volume":"29","author":"T Gneiting","year":"2011","unstructured":"Gneiting, T., & Ranjan, R. (2011). Comparing density forecasts using threshold-and quantile-weighted scoring rules. Journal of Business & Economic Statistics, 29(3), 411\u2013422.","journal-title":"Journal of Business & Economic Statistics"},{"key":"4611_CR29","doi-asserted-by":"crossref","unstructured":"Granger, C.W.J., & Pesaran, H.M. (2000). A decision theoretic approach to forecast evaluation. In Statistics and finance: An interface (pp. 261\u2013278). World Scientific.","DOI":"10.1142\/9781848160156_0015"},{"issue":"3","key":"4611_CR30","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1175\/1520-0493(2001)129<0550:IORHFV>2.0.CO;2","volume":"129","author":"TM Hamill","year":"2001","unstructured":"Hamill, T. M. (2001). Interpretation of rank histograms for verifying ensemble forecasts. Monthly Weather Review, 129(3), 550\u2013560.","journal-title":"Monthly Weather Review"},{"issue":"4","key":"4611_CR31","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1198\/073500105000000063","volume":"23","author":"PR Hansen","year":"2005","unstructured":"Hansen, P. R. (2005). A test for superior predictive ability. Journal of Business & Economic Statistics, 23(4), 365\u2013380.","journal-title":"Journal of Business & Economic Statistics"},{"issue":"7","key":"4611_CR32","doi-asserted-by":"crossref","first-page":"873","DOI":"10.1002\/jae.800","volume":"20","author":"PR Hansen","year":"2005","unstructured":"Hansen, P. R., & Lunde, A. (2005). A forecast comparison of volatility models: Does anything beat a GARCH(1, 1)? Journal of Applied Econometrics, 20(7), 873\u2013889.","journal-title":"Journal of Applied Econometrics"},{"issue":"1","key":"4611_CR33","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1007\/s10479-015-1846-0","volume":"229","author":"SC Hora","year":"2015","unstructured":"Hora, S. C., & Karde\u015f, E. (2015). Calibration, sharpness and the weighting of experts in a linear opinion pool. Annals of Operations Research, 229(1), 429\u2013450.","journal-title":"Annals of Operations Research"},{"key":"4611_CR34","first-page":"695","volume":"6","author":"A Hyv\u00e4rinen","year":"2005","unstructured":"Hyv\u00e4rinen, A. (2005). Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6, 695\u2013709.","journal-title":"Journal of Machine Learning Research"},{"issue":"2","key":"4611_CR35","first-page":"389","volume":"23","author":"MC Jensen","year":"1968","unstructured":"Jensen, M. C. (1968). The performance of mutual funds in the period 1945\u20131964. The Journal of Finance, 23(2), 389\u2013416.","journal-title":"The Journal of Finance"},{"issue":"4","key":"4611_CR36","doi-asserted-by":"crossref","first-page":"256","DOI":"10.1287\/deca.1110.0216","volume":"8","author":"DJ Johnstone","year":"2011","unstructured":"Johnstone, D. J., Jose, V. R. R., & Winkler, R. L. (2011). Tailored scoring rules for probabilities. Decision Analysis, 8(4), 256\u2013268.","journal-title":"Decision Analysis"},{"key":"4611_CR37","doi-asserted-by":"crossref","unstructured":"Jordan, A., Kr\u00fcger, F., & Lerch, S. (2019). Evaluating probabilistic forecasts with scoringRules. Journal of Statistical Software, 90(12), 1\u201337.","DOI":"10.18637\/jss.v090.i12"},{"issue":"4","key":"4611_CR38","doi-asserted-by":"crossref","first-page":"1267","DOI":"10.5194\/hess-11-1267-2007","volume":"11","author":"F Laio","year":"2007","unstructured":"Laio, F., & Tamea, S. (2007). Verification tools for probabilistic forecasts of continuous hydrological variables. Hydrology and Earth System Sciences Discussions, 11(4), 1267\u20131277.","journal-title":"Hydrology and Earth System Sciences Discussions"},{"issue":"10","key":"4611_CR39","doi-asserted-by":"crossref","first-page":"1781","DOI":"10.1016\/j.jspi.2013.05.012","volume":"143","author":"RL Machete","year":"2013","unstructured":"Machete, R. L. (2013). Contrasting probabilistic scoring rules. Journal of Statistical Planning and Inference, 143(10), 1781\u20131790.","journal-title":"Journal of Statistical Planning and Inference"},{"key":"4611_CR40","doi-asserted-by":"publisher","unstructured":"Mahbobi, M., Kimiagari, S., & Vasudevan, M. (2021). Credit risk classification: an integrated predictive accuracy algorithm using artificial and deep neural networks. Annals of Operations Research, 1\u201329. https:\/\/doi.org\/10.1007\/s10479-021-04114-z.","DOI":"10.1007\/s10479-021-04114-z"},{"key":"4611_CR41","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1086\/294632","volume":"36","author":"BB Mandelbrot","year":"1963","unstructured":"Mandelbrot, B. B. (1963). The variation of certain speculative prices. Journal of Business, 36, 394\u2013419.","journal-title":"Journal of Business"},{"issue":"10","key":"4611_CR42","doi-asserted-by":"crossref","first-page":"1087","DOI":"10.1287\/mnsc.22.10.1087","volume":"22","author":"JE Matheson","year":"1976","unstructured":"Matheson, J. E., & Winkler, R. L. (1976). Scoring rules for continuous probability distributions. Management Science, 22(10), 1087\u20131096.","journal-title":"Management Science"},{"issue":"4","key":"4611_CR43","doi-asserted-by":"crossref","first-page":"292","DOI":"10.1287\/deca.2013.0280","volume":"10","author":"EC Merkle","year":"2013","unstructured":"Merkle, E. C., & Steyvers, M. (2013). Choosing a strictly proper scoring rule. Decision Analysis, 10(4), 292\u2013304.","journal-title":"Decision Analysis"},{"key":"4611_CR44","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1007\/978-3-319-63645-0_24","volume":"84","author":"B Momin","year":"2018","unstructured":"Momin, B., & Chavan, G. (2018). Univariate time series models for forecasting stationary and non-stationary data: A brief review. Smart Innovation, Systems and Technologies, 84, 219\u2013226.","journal-title":"Smart Innovation, Systems and Technologies"},{"key":"4611_CR45","doi-asserted-by":"crossref","unstructured":"Nelson, D.B. (1991). Conditional heteroskedasticity in asset returns: A new approach. Econometrica: Journal of the Econometric Society, 59, 347\u2013370.","DOI":"10.2307\/2938260"},{"issue":"1","key":"4611_CR46","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1214\/12-AOS971","volume":"40","author":"M Parry","year":"2012","unstructured":"Parry, M., Dawid, A. P., Lauritzen, S., et al. (2012). Proper local scoring rules. The Annals of Statistics, 40(1), 561\u2013592.","journal-title":"The Annals of Statistics"},{"key":"4611_CR47","doi-asserted-by":"crossref","unstructured":"Pelagatti, M.M. (2004). Dynamic conditional correlation with elliptical distributions. Available at SSRN 888732.","DOI":"10.2139\/ssrn.888732"},{"issue":"2","key":"4611_CR48","doi-asserted-by":"crossref","first-page":"362","DOI":"10.1016\/j.jbankfin.2009.08.009","volume":"34","author":"C P\u00e9rignon","year":"2010","unstructured":"P\u00e9rignon, C., & Smith, D. (2010). The level and quality of value-at-risk disclosure by commercial banks. Journal of Banking and Finance, 34(2), 362\u2013377.","journal-title":"Journal of Banking and Finance"},{"key":"4611_CR49","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.apenergy.2011.11.004","volume":"96","author":"P Pinson","year":"2012","unstructured":"Pinson, P., & Girard, R. (2012). Evaluating the quality of scenarios of short-term wind power generation. Applied Energy, 96, 12\u201320.","journal-title":"Applied Energy"},{"key":"4611_CR50","unstructured":"Pinson, P., & Tastu, J. (2013). Discrimination ability of the energy score. Technical report."},{"issue":"4","key":"4611_CR51","doi-asserted-by":"crossref","first-page":"1321","DOI":"10.1175\/MWR-D-14-00269.1","volume":"143","author":"M Scheuerer","year":"2015","unstructured":"Scheuerer, M., & Hamill, T. M. (2015). Variogram-based proper scoring rules for probabilistic forecasts of multivariate quantities. Monthly Weather Review, 143(4), 1321\u20131334.","journal-title":"Monthly Weather Review"},{"issue":"1","key":"4611_CR52","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1007\/s10479-018-3108-4","volume":"282","author":"HL Shang","year":"2019","unstructured":"Shang, H. L., Yang, Y., & Kearney, F. (2019). Intraday forecasts of a volatility index: Functional time series methods with dynamic updating. Annals of Operations Research, 282(1), 331\u2013354.","journal-title":"Annals of Operations Research"},{"key":"4611_CR53","doi-asserted-by":"crossref","first-page":"146","DOI":"10.1016\/0001-6918(70)90013-2","volume":"34","author":"C-AS Sta\u00ebl von Holstein","year":"1970","unstructured":"Sta\u00ebl von Holstein, C.-A.S. (1970). Measurement of subjective probability. Acta Psychologica, 34, 146\u2013159.","journal-title":"Acta Psychologica"},{"key":"4611_CR54","unstructured":"Sz\u00e9kely, G.J. (2003). E-statistics: The energy of statistical samples. Bowling Green State University, Department of Mathematics and Statistics Technical Report,\u00a03(5), 1\u201318."},{"key":"4611_CR55","doi-asserted-by":"crossref","unstructured":"Ter\u00e4svirta, T. (2009). An introduction to univariate GARCH models. In Handbook of financial time series (pp. 17\u201342). Springer.","DOI":"10.1007\/978-3-540-71297-8_1"},{"issue":"4\u20136","key":"4611_CR56","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1016\/S0378-4754(99)00030-0","volume":"48","author":"AK Tsui","year":"1999","unstructured":"Tsui, A. K., & Yu, Q. (1999). Constant conditional correlation in a bivariate GARCH model: Evidence from the stock markets of China. Mathematics and Computers in Simulation, 48(4\u20136), 503\u2013509.","journal-title":"Mathematics and Computers in Simulation"},{"issue":"336","key":"4611_CR57","doi-asserted-by":"crossref","first-page":"675","DOI":"10.1080\/01621459.1971.10482329","volume":"66","author":"RL Winkler","year":"1971","unstructured":"Winkler, R. L. (1971). Probabilistic prediction: Some experimental results. Journal of the American Statistical Association, 66(336), 675\u2013685.","journal-title":"Journal of the American Statistical Association"},{"key":"4611_CR58","doi-asserted-by":"crossref","unstructured":"Winkler, R.L. (1977). Rewarding expertise in probability assessment. In Decision making and change in human affairs (pp. 127\u2013140). Springer.","DOI":"10.1007\/978-94-010-1276-8_10"},{"issue":"1","key":"4611_CR59","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/BF02562681","volume":"5","author":"RL Winkler","year":"1996","unstructured":"Winkler, R. L. (1996). Scoring rules and the evaluation of probabilities. Test, 5(1), 1\u201360.","journal-title":"Test"},{"issue":"15","key":"4611_CR60","doi-asserted-by":"crossref","first-page":"3616","DOI":"10.1080\/03610926.2017.1361984","volume":"47","author":"Y Zhang","year":"2018","unstructured":"Zhang, Y., & Nadarajah, S. (2018). A review of backtesting for value at risk. Communications in Statistics\u2014Theory and Methods, 47(15), 3616\u20133639.","journal-title":"Communications in Statistics\u2014Theory and Methods"},{"key":"4611_CR61","unstructured":"Ziel, F., & Berk, K. (2019). Multivariate forecasting evaluation: On sensitive and strictly proper scoring rules. arXiv preprint arXiv:1910.07325."}],"container-title":["Annals of Operations Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10479-022-04611-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10479-022-04611-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10479-022-04611-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,14]],"date-time":"2024-03-14T15:24:08Z","timestamp":1710429848000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10479-022-04611-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,18]]},"references-count":61,"journal-issue":{"issue":"1-3","published-print":{"date-parts":[[2024,3]]}},"alternative-id":["4611"],"URL":"https:\/\/doi.org\/10.1007\/s10479-022-04611-9","relation":{},"ISSN":["0254-5330","1572-9338"],"issn-type":[{"value":"0254-5330","type":"print"},{"value":"1572-9338","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,18]]},"assertion":[{"value":"14 February 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 March 2022","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}