{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,31]],"date-time":"2025-12-31T18:37:51Z","timestamp":1767206271252,"version":"build-2238731810"},"reference-count":46,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2018,8,9]],"date-time":"2018-08-09T00:00:00Z","timestamp":1533772800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003246","name":"Nederlandse Organisatie voor Wetenschappelijk Onderzoek","doi-asserted-by":"publisher","award":["NWO-VIDI-452-14-006"],"award-info":[{"award-number":["NWO-VIDI-452-14-006"]}],"id":[{"id":"10.13039\/501100003246","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.mdpi.com"],"crossmark-restriction":true},"short-container-title":["Entropy"],"abstract":"<jats:p>Experts\u2019 beliefs embody a present state of knowledge. It is desirable to take this knowledge into account when making decisions. However, ranking experts based on the merit of their beliefs is a difficult task. In this paper, we show how experts can be ranked based on their knowledge and their level of (un)certainty. By letting experts specify their knowledge in the form of a probability distribution, we can assess how accurately they can predict new data, and how appropriate their level of (un)certainty is. The expert\u2019s specified probability distribution can be seen as a prior in a Bayesian statistical setting. We evaluate these priors by extending an existing prior-data (dis)agreement measure, the Data Agreement Criterion, and compare this approach to using Bayes factors to assess prior specification. We compare experts with each other and the data to evaluate their appropriateness. Using this method, new research questions can be asked and answered, for instance: Which expert predicts the new data best? Is there agreement between my experts and the data? Which experts\u2019 representation is more valid or useful? Can we reach convergence between expert judgement and data? We provided an empirical example ranking (regional) directors of a large financial institution based on their predictions of turnover.<\/jats:p>","DOI":"10.3390\/e20080592","type":"journal-article","created":{"date-parts":[[2018,8,9]],"date-time":"2018-08-09T10:36:31Z","timestamp":1533810991000},"page":"592","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Using the Data Agreement Criterion to Rank Experts\u2019 Beliefs"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8352-7574","authenticated-orcid":false,"given":"Duco","family":"Veen","sequence":"first","affiliation":[{"name":"Department of Methods and Statistics, Utrecht University, 3584 CH 14 Utrecht, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Diederick","family":"Stoel","sequence":"additional","affiliation":[{"name":"ProfitWise International, 1054 HV 237 Amsterdam, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Naomi","family":"Schalken","sequence":"additional","affiliation":[{"name":"Department of Methods and Statistics, Utrecht University, 3584 CH 14 Utrecht, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kees","family":"Mulder","sequence":"additional","affiliation":[{"name":"Department of Methods and Statistics, Utrecht University, 3584 CH 14 Utrecht, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rens","family":"Van de Schoot","sequence":"additional","affiliation":[{"name":"Department of Methods and Statistics, Utrecht University, 3584 CH 14 Utrecht, The Netherlands"},{"name":"Optentia Research Focus Area, North-West University, Vanderbijlpark 1900, South Africa"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2018,8,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A., and Rubin, D.B. (2013). Bayesian Data Analysis, CRC Press.","DOI":"10.1201\/b16018"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Lynch, S.M. (2007). Introduction to Applied Bayesian Statistics and Estimation for Social Scientists, Springer Science & Business Media.","DOI":"10.1007\/978-0-387-71265-9"},{"key":"ref_3","first-page":"5","article-title":"Bayesian probability and statistics in management research","volume":"39","author":"Zyphur","year":"2013","journal-title":"J. Manag."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"705","DOI":"10.1037\/met0000124","article-title":"Using expert knowledge for test linking","volume":"22","author":"Bolsinova","year":"2017","journal-title":"Psychol. Methods"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"O\u2019Hagan, A., Buck, C.E., Daneshkhah, A., Eiser, J.R., Garthwaite, P.H., Jenkinson, D.J., Oakley, J.E., and Rakow, T. (2006). Uncertain Judgements: Eliciting Experts\u2019 Probabilities, John Wiley & Sons.","DOI":"10.1002\/0470033312"},{"key":"ref_6","first-page":"90","article-title":"Application and evaluation of an expert judgment elicitation procedure for correlations","volume":"8","author":"Lek","year":"2017","journal-title":"Front. Psychol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1011","DOI":"10.1080\/02664760802192981","article-title":"Diagnostics of prior-data agreement in applied Bayesian analysis","volume":"35","author":"Bousquet","year":"2008","journal-title":"J. Appl. Stat."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1214\/aoms\/1177729694","article-title":"On information and sufficiency","volume":"22","author":"Kullback","year":"1951","journal-title":"Ann. Math. Stat."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Cooke, R. (1991). Experts in Uncertainty: Opinion and Subjective Probability in Science, Oxford University Press.","DOI":"10.1093\/oso\/9780195064650.001.0001"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Dias, L.C., Morton, A., and Quigley, J. (2018). Elicitation in the classical model. Elicitation, Springer.","DOI":"10.1007\/978-3-319-65052-4"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1002\/pst.1675","article-title":"Advantages of a wholly Bayesian approach to assessing efficacy in early drug development: A case study","volume":"14","author":"Walley","year":"2015","journal-title":"Pharm. Stat."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1615\/Int.J.UncertaintyQuantification.2014011073","article-title":"Bayesian inference for inverse problems occurring in uncertainty analysis","volume":"5","author":"Fu","year":"2015","journal-title":"Int. J. Uncertain. Quantif."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"e2439","DOI":"10.1002\/env.2439","article-title":"An adaptive kriging method for solving nonlinear inverse statistical problems","volume":"28","author":"Fu","year":"2017","journal-title":"Environmetrics"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scales","volume":"20","author":"Cohen","year":"1960","journal-title":"Educ. Psychol. Meas."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Koch, G.G. (2004). Intraclass correlation coefficient. Encycl. Stat. Sci., 6.","DOI":"10.1002\/0471667196.ess1275"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"420","DOI":"10.1037\/0033-2909.86.2.420","article-title":"Intraclass correlations: Uses in assessing rater reliability","volume":"86","author":"Shrout","year":"1979","journal-title":"Psychol. Bull."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1175\/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2","article-title":"Verification of forecasts expressed in terms of probability","volume":"78","author":"Brier","year":"1950","journal-title":"Mon. Weather Rev."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Dias, L.C., Morton, A., and Quigley, J. (2018). Eliciting probabilistic judgements for integrating decision support systems. Elicitation, Springer.","DOI":"10.1007\/978-3-319-65052-4"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"773","DOI":"10.1080\/01621459.1995.10476572","article-title":"Bayes factors","volume":"90","author":"Kass","year":"1995","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"362","DOI":"10.1016\/j.jmp.2008.03.002","article-title":"Bayes factors: Prior sensitivity and model generalizability","volume":"52","author":"Liu","year":"2008","journal-title":"J. Math. Psychol."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1006\/jmps.1999.1278","article-title":"Bayesian model selection and model averaging","volume":"44","author":"Wasserman","year":"2000","journal-title":"J. Math. Psychol."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1111\/j.2517-6161.1979.tb01066.x","article-title":"Reference posterior distributions for Bayesian inference","volume":"41","author":"Bernardo","year":"1979","journal-title":"J. R. Stat. Soc. Ser. B Methodol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1016\/S0378-3758(97)00074-8","article-title":"Noninformative priors do not exist: A discussion with jose m. bernardo","volume":"65","author":"Irony","year":"1997","journal-title":"J. Stat. Inference Plan."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1080\/01621459.1989.10478756","article-title":"Estimating a product of means: Bayesian analysis with reference priors","volume":"84","author":"Berger","year":"1989","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Bernardo, J.M., and Smith, A.F. (1994). Bayesian Theory, John Wiley & Sons.","DOI":"10.1002\/9780470316870"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"905","DOI":"10.1214\/07-AOS587","article-title":"The formal definition of reference priors","volume":"37","author":"Berger","year":"2009","journal-title":"Ann. Stat."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Gelman, A., Simpson, D., and Betancourt, M. (2017). The prior can often only be understood in the context of the likelihood. Entropy, 19.","DOI":"10.3390\/e19100555"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1343","DOI":"10.1080\/01621459.1996.10477003","article-title":"The selection of prior distributions by formal rules","volume":"91","author":"Kass","year":"1996","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_29","first-page":"453","article-title":"An invariant form for the prior probability in estimation problems","volume":"186","author":"Jeffreys","year":"1946","journal-title":"Proc. R. Soc. Lond. Ser. Math. Phys. Sci."},{"key":"ref_30","unstructured":"Jeffreys, S.H. (1961). Theory of Probability, Oxford University Press."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"939","DOI":"10.1109\/PROC.1982.12425","article-title":"On the rationale of maximum-entropy methods","volume":"70","author":"Jaynes","year":"1982","journal-title":"Proc. IEEE"},{"key":"ref_32","unstructured":"Yang, R., and Berger, J.O. (1996). A Catalog of Noninformative Priors, Institute of Statistics and Decision Sciences, Duke University."},{"key":"ref_33","unstructured":"Akaike, H. (1973). Information theory as an extension of the maximum likelihood principle. Second International Symposium on Information Theory, Akademiai Kaido."},{"key":"ref_34","unstructured":"Burnham, K.P., and Anderson, D.R. (2002). Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach, Springer Science & Business Media."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1016\/j.jmp.2015.11.001","article-title":"The philosophy of Bayes factors and the quantification of statistical evidence","volume":"72","author":"Morey","year":"2016","journal-title":"J. Math. Psychol."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1093\/biomet\/83.2.251","article-title":"Approximate Bayes factors and accounting for model uncertainty in generalised linear models","volume":"83","author":"Raftery","year":"1996","journal-title":"Biometrika"},{"key":"ref_37","unstructured":"Dirac, P.A.M. (1947). The Principles of Quantum Mechanics, Clarendon Press."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Barber, D. (2012). Bayesian Reasoning and Machine Learning, Cambridge University Press.","DOI":"10.1017\/CBO9780511804779"},{"key":"ref_39","unstructured":"Bedford, T., French, S., Hanea, A.M., and Nane, G.F. (2018). Dealing with imperfect elicitation results. Expert Judgement in Risk and Decision Analysis, DRDC Centre for Security Studies."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"2110","DOI":"10.3389\/fpsyg.2017.02110","article-title":"Proposal for a Five-Step Method to Elicit Expert Judgement","volume":"8","author":"Veen","year":"2017","journal-title":"Front. Psychol."},{"key":"ref_41","first-page":"359","article-title":"On Bayesian modeling of fat tails and skewness","volume":"93","author":"Steel","year":"1998","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_42","unstructured":"Plummer, M. (2003, January 20\u201322). Rjags: Bayesian Graphical Models Using MCMC. Proceedings of the 3rd International Workshop on Distributed Statistical Computing (DSC 2003), Vienna, Austria."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1214\/ss\/1177011136","article-title":"Inference from iterative simulation using multiple sequences","volume":"7","author":"Gelman","year":"1992","journal-title":"Stat. Sci."},{"key":"ref_44","unstructured":"Stan Development Team (2018, August 08). RStan: The R Interface to Stan. Available online: https:\/\/cran.r-project.org\/web\/packages\/rstan\/."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Gronau, Q.F., and Singmann, H. (2018, August 08). Bridgesampling: Bridge Sampling for Marginal Likelihoods and Bayes Factors. Available online: https:\/\/cran.r-project.org\/web\/packages\/bridgesampling\/.","DOI":"10.32614\/CRAN.package.bridgesampling"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Deza, M.M., and Deza, E. (2009). Encyclopedia of distances. Encyclopedia of Distances, Springer.","DOI":"10.1007\/978-3-642-00234-2"}],"updated-by":[{"DOI":"10.3390\/e21030307","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2018,8,9]],"date-time":"2018-08-09T00:00:00Z","timestamp":1533772800000}}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/20\/8\/592\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,3]],"date-time":"2025-08-03T22:46:03Z","timestamp":1754261163000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/20\/8\/592"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,8,9]]},"references-count":46,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2018,8]]}},"alternative-id":["e20080592"],"URL":"https:\/\/doi.org\/10.3390\/e20080592","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,8,9]]}}}