{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T01:19:20Z","timestamp":1760231960169,"version":"build-2065373602"},"reference-count":18,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2022,5,27]],"date-time":"2022-05-27T00:00:00Z","timestamp":1653609600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Estimates based on expert judgements of quantities of interest are commonly used to supplement or replace measurements when the latter are too expensive or impossible to obtain. Such estimates are commonly accompanied by information about the uncertainty of the estimate, such as a credible interval. To be considered well-calibrated, an expert\u2019s credible intervals should cover the true (but unknown) values a certain percentage of time, equal to the percentage specified by the expert. To assess expert calibration, so-called calibration questions may be asked in an expert elicitation exercise; these are questions with known answers used to assess and compare experts\u2019 performance. An approach that is commonly applied to assess experts\u2019 performance by using these questions is to directly compare the stated percentage cover with the actual coverage. We show that this approach has statistical drawbacks when considered in a rigorous hypothesis testing framework. We generalize the test to an equivalence testing framework and discuss the properties of this new proposal. We show that comparisons made on even a modest number of calibration questions have poor power, which suggests that the formal testing of the calibration of experts in an experimental setting may be prohibitively expensive. We contextualise the theoretical findings with a couple of applications and discuss the implications of our findings.<\/jats:p>","DOI":"10.3390\/e24060757","type":"journal-article","created":{"date-parts":[[2022,5,27]],"date-time":"2022-05-27T07:05:07Z","timestamp":1653635107000},"page":"757","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Are Experts Well-Calibrated? An Equivalence-Based Hypothesis Test"],"prefix":"10.3390","volume":"24","author":[{"given":"Gayan","family":"Dharmarathne","sequence":"first","affiliation":[{"name":"Department of Statistics, University of Colombo, Colombo 00700, Sri Lanka"}]},{"given":"Anca M.","family":"Hanea","sequence":"additional","affiliation":[{"name":"Centre of Excellence for Biosecurity Risk Analysis, School of BioSciences, The University of Melbourne, Parkville, VIC 3010, Australia"}]},{"given":"Andrew","family":"Robinson","sequence":"additional","affiliation":[{"name":"Centre of Excellence for Biosecurity Risk Analysis, School of BioSciences, The University of Melbourne, Parkville, VIC 3010, Australia"}]}],"member":"1968","published-online":{"date-parts":[[2022,5,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1093\/reep\/rex022","article-title":"Expert elicitation: Using the classical model to validate experts\u2019 judgments","volume":"12","author":"Colson","year":"2018","journal-title":"Rev. Environ. Econ. Policy"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1080\/00031305.2018.1518265","article-title":"Expert Knowledge Elicitation: Subjective but Scientific","volume":"73","year":"2019","journal-title":"Am. Stat."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Cooke, R. (1991). Experts in Uncertainty: Opinion and Subjective Probability in Science, Oxford University Press.","DOI":"10.1093\/oso\/9780195064650.001.0001"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"512","DOI":"10.1111\/j.1539-6924.2009.01337.x","article-title":"Reducing overconfidence in the interval judgments of experts","volume":"30","author":"Fidler","year":"2010","journal-title":"Risk Anal."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"O\u2019Hagan, A., Buck, C., Daneshkhah, A., Eiser, J., Garthwaite, P., Jenkinson, D., Oakley, J., and Rakow, T. (2006). Uncertain Judgements: Eliciting Experts\u2019 Probabilities, Wiley.","DOI":"10.1002\/0470033312"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"EFSA (2014). Guidance on Expert Knowledge Elicitation in Food and Feed Safety Risk Assessment. Eur. Food Saf. Auth. J., 12, 3734.","DOI":"10.2903\/j.efsa.2014.3734"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"e02075","DOI":"10.1002\/eap.2075","article-title":"Weighting and aggregating expert ecological judgments","volume":"30","author":"Hemming","year":"2020","journal-title":"Ecol. Appl."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"678","DOI":"10.1037\/0033-295X.114.3.678","article-title":"The na\u00efve intuitive statistician: A na\u00efve sampling model of intuitive confidence intervals","volume":"114","author":"Juslin","year":"2007","journal-title":"Psychol. Rev."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1037\/0278-7393.30.2.299","article-title":"Overconfidence in interval estimates","volume":"30","author":"Soll","year":"2004","journal-title":"J. Exp. Psychol. Learn. Mem. Cogn."},{"key":"ref_10","first-page":"455","article-title":"When 90% confidence intervals are 50% certain: On the credibility of credible intervals","volume":"19","author":"Teigen","year":"2005","journal-title":"Appl. Cogn. Psychol. Off. J. Soc. Appl. Res. Mem. Cogn."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"782","DOI":"10.1111\/j.1472-4642.2012.00884.x","article-title":"Evaluating the accuracy and calibration of expert predictions under uncertainty: Predicting the outcomes of ecological research","volume":"18","author":"McBride","year":"2012","journal-title":"Divers. Distrib."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"857","DOI":"10.1017\/S1930297500004277","article-title":"A model-based approach for the analysis of the calibration of probability judgments","volume":"6","author":"Budescu","year":"2011","journal-title":"Judgm. Decis. Mak."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Wellek, S. (2010). Testing Statistical Hypotheses of Equivalence and Noninferiority, CRC.","DOI":"10.1201\/EBK1439808184"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1177\/2515245918770963","article-title":"Equivalence testing for psychological research: A tutorial","volume":"1","author":"Lakens","year":"2018","journal-title":"Adv. Methods Pract. Psychol. Sci."},{"key":"ref_15","unstructured":"Dharmarathne, H.A.S.G. (2020). Exploring the Statistical Aspects of Expert Elicited Experiments. [Ph.D. Thesis, University of Melbourne]."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Hemming, V., Walshe, T.V., Hanea, A.M., Fidler, F., and Burgman, M.A. (2018). Eliciting improved quantitative judgements using the IDEA protocol: A case study in natural resource management. PLoS ONE, 13.","DOI":"10.1371\/journal.pone.0198468"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1111\/2041-210X.12857","article-title":"A practical guide to structured expert elicitation using the IDEA protocol","volume":"9","author":"Hemming","year":"2018","journal-title":"Methods Ecol. Evol."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"364","DOI":"10.1080\/13669877.2014.971334","article-title":"Using expert elicitation to characterise long-term tectonic risks to radioactive waste repositories in Japan","volume":"18","author":"Scourse","year":"2015","journal-title":"J. Risk Res."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/6\/757\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:19:30Z","timestamp":1760138370000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/6\/757"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,27]]},"references-count":18,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2022,6]]}},"alternative-id":["e24060757"],"URL":"https:\/\/doi.org\/10.3390\/e24060757","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2022,5,27]]}}}