{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T17:18:54Z","timestamp":1778692734126,"version":"3.51.4"},"reference-count":26,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2022,11,2]],"date-time":"2022-11-02T00:00:00Z","timestamp":1667347200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,11,2]],"date-time":"2022-11-02T00:00:00Z","timestamp":1667347200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100008247","name":"University of Otago","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100008247","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Ann Oper Res"],"published-print":{"date-parts":[[2023,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>An important component of multi-criteria decision analysis (MCDA) in the public sector is the elicitation and aggregation of preference data collected via surveys into the relative importance of the criteria for the decision at hand. These aggregated preference data, usually in the form of mean weights on the criteria, are intended to represent the preferences of the relevant population overall. However, random sampling is often not feasible for public-sector MCDA for logistical reasons, including the expense involved in identifying and recruiting participants. Instead, non-random sampling methods such as convenience, purposive or snowball sampling are widely used. Nonetheless, provided the preference data collected are sufficiently \u2018cohesive\u2019 in terms of the extent to which the weights of the individuals belonging to the various exogenously defined groups in the sample are similar, non-random sampling can still produce externally valid aggregate preference data. We explain a method for measuring cohesiveness using the Kemeny and Hellinger distance measures, which involve measuring the \u2018distance\u2019 of participants\u2019 weights (and the corresponding rankings of the criteria) from each other, <jats:italic>within<\/jats:italic> and <jats:italic>between<\/jats:italic> the groups respectively. As an illustration, these distance measures are applied to data from a MCDA to rank non-communicable diseases according to their overall burden to society. We conclude that the method is useful for evaluating the external validity of preference data obtained from non-random sampling.<\/jats:p>","DOI":"10.1007\/s10479-022-05026-2","type":"journal-article","created":{"date-parts":[[2022,11,2]],"date-time":"2022-11-02T20:23:37Z","timestamp":1667420617000},"page":"939-949","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["External validity of multi-criteria preference data obtained from non-random sampling: measuring cohesiveness within and between groups"],"prefix":"10.1007","volume":"325","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2204-2028","authenticated-orcid":false,"given":"Saeideh","family":"Babashahi","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5790-0710","authenticated-orcid":false,"given":"Paul","family":"Hansen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5261-1119","authenticated-orcid":false,"given":"Ronald","family":"Peeters","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,11,2]]},"reference":[{"key":"5026_CR1","doi-asserted-by":"publisher","first-page":"137","DOI":"10.1016\/j.socscimed.2017.06.024","volume":"188","author":"A Angelis","year":"2017","unstructured":"Angelis, A., & Kanavas, P. (2017). Multiple criteria decision analysis (MCDA) for evaluating new medicines in health technology assessment and beyond: The advance value framework. Social Science & Medicine, 188, 137\u2013156. https:\/\/doi.org\/10.1016\/j.socscimed.2017.06.024","journal-title":"Social Science & Medicine"},{"issue":"2","key":"5026_CR2","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1016\/j.healthpol.2020.12.003","volume":"125","author":"S Babashahi","year":"2021","unstructured":"Babashahi, S., Hansen, P., & Sullivan, T. (2021). Creating a priority list of non-communicable diseases to support health research funding decision-making. Health Policy, 125(2), 221\u2013228. https:\/\/doi.org\/10.1016\/j.healthpol.2020.12.003","journal-title":"Health Policy"},{"issue":"11","key":"5026_CR3","doi-asserted-by":"publisher","first-page":"1283","DOI":"10.1016\/j.jval.2019.06.014","volume":"22","author":"R Baltussen","year":"2019","unstructured":"Baltussen, R., Marsh, K., Thokala, P., Diaby, V., Castro, H., & Cleemput, I. (2019). Multicriteria decision analysis to support health technology assessment agencies: Benefits, limitations, and the way forward. Value in Health, 22(11), 1283\u20131288. https:\/\/doi.org\/10.1016\/j.jval.2019.06.014","journal-title":"Value in Health"},{"key":"5026_CR4","doi-asserted-by":"publisher","first-page":"917","DOI":"10.1007\/s00355-015-0942-z","volume":"46","author":"W Bossert","year":"2016","unstructured":"Bossert, W., Can, B., & D\u2019Ambrosio, C. (2016). Measuring rank mobility with variable population size. Social Choice and Welfare, 46, 917\u2013931. https:\/\/doi.org\/10.1007\/s00355-015-0942-z","journal-title":"Social Choice and Welfare"},{"key":"5026_CR5","doi-asserted-by":"publisher","first-page":"112","DOI":"10.1016\/j.jmateco.2018.04.007","volume":"79","author":"B Can","year":"2018","unstructured":"Can, B., & Storcken, T. (2018). A re-characterization of the Kemeny distance. Journal of Mathematical Economics, 79, 112\u2013116. https:\/\/doi.org\/10.1016\/j.jmateco.2018.04.007","journal-title":"Journal of Mathematical Economics"},{"key":"5026_CR6","doi-asserted-by":"publisher","DOI":"10.4324\/9780203771587","volume-title":"Statistical power analysis for the behavioral sciences","author":"J Cohen","year":"1988","unstructured":"Cohen, J. (1988). Statistical power analysis for the behavioral sciences. New York: Routledge. https:\/\/doi.org\/10.4324\/9780203771587"},{"key":"5026_CR7","doi-asserted-by":"publisher","first-page":"390","DOI":"10.1016\/j.ejor.2003.06.012","volume":"158","author":"Y De Smet","year":"2004","unstructured":"De Smet, Y., & Montano Guzm\u00e1n, L. (2004). Towards multicriteria clustering: An extension of the k-means algorithm. European Journal of Operational Research, 158, 390\u2013398. https:\/\/doi.org\/10.1016\/j.ejor.2003.06.012","journal-title":"European Journal of Operational Research"},{"issue":"1","key":"5026_CR8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.11648\/j.ajtas.20160501.11","volume":"5","author":"I Etikan","year":"2016","unstructured":"Etikan, I., Musa, S., & Alkassim, R. (2016). Comparison of convenience sampling and purposive sampling. American Journal of Theoretical and Applied Statistics, 5(1), 1\u20134. https:\/\/doi.org\/10.11648\/j.ajtas.20160501.11","journal-title":"American Journal of Theoretical and Applied Statistics"},{"issue":"1","key":"5026_CR9","doi-asserted-by":"publisher","first-page":"148","DOI":"10.1214\/aoms\/1177705148","volume":"32","author":"L Goodman","year":"1961","unstructured":"Goodman, L. (1961). Snowball sampling. Annals of Mathematical Statistics, 32(1), 148\u2013170. https:\/\/doi.org\/10.1214\/aoms\/1177705148","journal-title":"Annals of Mathematical Statistics"},{"key":"5026_CR10","doi-asserted-by":"publisher","unstructured":"Hansen, P., & Devlin, N. (2019). Multi-criteria decision analysis (MCDA) in health care decision making. The Oxford Encyclopedia of Health Economics, Oxford University Press. https:\/\/doi.org\/10.1093\/acrefore\/9780190625979.013.98","DOI":"10.1093\/acrefore\/9780190625979.013.98"},{"issue":"3\u20134","key":"5026_CR11","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1002\/mcda.428","volume":"15","author":"P Hansen","year":"2008","unstructured":"Hansen, P., & Ombler, F. (2008). A new method for scoring additive multi-attribute value models using pairwise rankings of alternatives. Multi-Criteria Decision Analysis, 15(3\u20134), 87\u2013107. https:\/\/doi.org\/10.1002\/mcda.428","journal-title":"Multi-Criteria Decision Analysis"},{"key":"5026_CR12","doi-asserted-by":"publisher","first-page":"210","DOI":"10.1515\/crll.1909.136.210","volume":"136","author":"E Hellinger","year":"1909","unstructured":"Hellinger, E. (1909). Neue Begr\u00fcndung der Theorie quadratischer Formen von unendlichvielen Ver\u00e4nderlichen. Journal f\u00fcr die reine und angewandte Mathematik, 136, 210\u2013271. https:\/\/doi.org\/10.1515\/crll.1909.136.210","journal-title":"Journal f\u00fcr die reine und angewandte Mathematik"},{"issue":"2","key":"5026_CR13","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1111\/mono.12296","volume":"82","author":"J Jager","year":"2017","unstructured":"Jager, J., Putnick, D., & Bornstein, M. (2017). More than just convenient: The scientific merits of homogeneous samples. Monographs of the Society for Research in Child Development, 82(2), 13\u201330. https:\/\/doi.org\/10.1111\/mono.12296","journal-title":"Monographs of the Society for Research in Child Development"},{"key":"5026_CR14","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1186\/s13561-015-0048-4","volume":"5","author":"M Kaltoft","year":"2015","unstructured":"Kaltoft, M., Turner, R., Cunich, M., Salkeld, G., Nielsen, J., & Dowie, J. (2015). Addressing preference heterogeneity in public health policy by combining cluster analysis and multi-criteria decision analysis: proof of method. Health Economics Review, 5, 10. https:\/\/doi.org\/10.1186\/s13561-015-0048-4","journal-title":"Health Economics Review"},{"issue":"4","key":"5026_CR15","first-page":"577","volume":"88","author":"J Kemeny","year":"1959","unstructured":"Kemeny, J. (1959). Mathematics without numbers. Daedalus, 88(4), 577\u2013591. https:\/\/www.jstor.org\/stable\/20026529","journal-title":"Daedalus"},{"key":"5026_CR16","doi-asserted-by":"publisher","unstructured":"Marsh, K., Goetghebeur, M., Thokala, P., & Baltussen, R. (2017). Multi-criteria decision analysis to support healthcare decisions. Springer Cham. https:\/\/doi.org\/10.1007\/978-3-319-47540-0","DOI":"10.1007\/978-3-319-47540-0"},{"issue":"2","key":"5026_CR17","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1016\/j.jval.2015.12.016","volume":"19","author":"K Marsh","year":"2016","unstructured":"Marsh, K., IJzerman, M., Thokala, P., Baltussen, R., Boysen, M., Kal\u00f3, Z., L\u00f6nngren, T., Mussen, F., Peacock, S., Watkins, J., & Devlin, N. (2016). Multiple criteria decision analysis for health care decision making\u2014emerging good practices: Report 2 of the ISPOR MCDA emerging good practices task force. Value in Health, 19(2), 125\u2013137. https:\/\/doi.org\/10.1016\/j.jval.2015.12.016","journal-title":"Value in Health"},{"issue":"3","key":"5026_CR18","doi-asserted-by":"publisher","first-page":"494","DOI":"10.1016\/j.ejor.2013.01.016","volume":"277","author":"P Meyer","year":"2013","unstructured":"Meyer, P., & Olteanu, A. L. (2013). Formalizing and solving the problem of clustering in MCDA. European Journal of Operational Research, 277(3), 494\u2013502. https:\/\/doi.org\/10.1016\/j.ejor.2013.01.016","journal-title":"European Journal of Operational Research"},{"key":"5026_CR19","doi-asserted-by":"publisher","first-page":"107850","DOI":"10.1016\/j.patcog.2021.107850","volume":"114","author":"J Rosenfeld","year":"2021","unstructured":"Rosenfeld, J., De Smet, Y., Debeir, O., & Decaestecker, C. (2021). Assessing partially ordered clustering in a multicriteria comparative context. Pattern Recognition, 114, 107850. https:\/\/doi.org\/10.1016\/j.patcog.2021.107850","journal-title":"Pattern Recognition"},{"key":"5026_CR20","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-50417-5_47","volume-title":"Computational Science - ICCS 2020","author":"W Sa\u0142abun","year":"2020","unstructured":"Sa\u0142abun, W., & Urbaniak, K., et al. (2020). A new coefficient of rankings similarity in decision-making problems. In V. Krzhizhanovskaya (Ed.), Computational Science - ICCS 2020.  (Vol. 12138). Cham: Springer. https:\/\/doi.org\/10.1007\/978-3-030-50417-5_47"},{"key":"5026_CR21","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1016\/j.omega.2017.09.001","volume":"80","author":"R Sarrazin","year":"2018","unstructured":"Sarrazin, R., De Smet, Y., & Rosenfeld, J. (2018). An extension of PROMETHEE to interval clustering. Omega, 80, 12\u201321. https:\/\/doi.org\/10.1016\/j.omega.2017.09.001","journal-title":"Omega"},{"issue":"2","key":"5026_CR22","doi-asserted-by":"publisher","first-page":"597","DOI":"10.22237\/jmasm\/1257035100","volume":"8","author":"S Sawilowsky","year":"2009","unstructured":"Sawilowsky, S. (2009). New effect size rules of thumb. Journal of Modern Applied Statistical Methods, 8(2), 597\u2013599. https:\/\/doi.org\/10.22237\/jmasm\/1257035100","journal-title":"Journal of Modern Applied Statistical Methods"},{"key":"5026_CR23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.jval.2015.12.003","volume":"19","author":"P Thokala","year":"2016","unstructured":"Thokala, P., Devlin, N., & Marsh, K. (2016). Multiple criteria decision analysis for health care decision making \u2013 an introduction: Report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value in Health, 19, 1\u201313. https:\/\/doi.org\/10.1016\/j.jval.2015.12.003","journal-title":"Value in Health"},{"key":"5026_CR24","doi-asserted-by":"publisher","unstructured":"Valerio, M., Rodriguez, N., Winkler, P., Lopez, J., Dennison, M., Liang, Y., & Turnr, B. (2016). Comparing two sampling methods to engage hard-to-reach communities in research priority setting. BMC Medical Research Methodology, 16, 146. https:\/\/doi.org\/10.1186\/s12874-016-0242-z","DOI":"10.1186\/s12874-016-0242-z"},{"key":"5026_CR25","doi-asserted-by":"publisher","unstructured":"van Hoeven, L., Janssen, M., Roes, K., & Koffjberg, H. (2015). Aiming for a representative sample: Simulating random versus purposive strategies for hospital selection. BMC Medical Research Methodology, 15, https:\/\/doi.org\/10.1186\/s12874-015-0089-8","DOI":"10.1186\/s12874-015-0089-8"},{"issue":"2","key":"5026_CR26","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1016\/S0377-2217(01)00243-0","volume":"138","author":"C Zopounidis","year":"2002","unstructured":"Zopounidis, C., & Doumpos, M. (2002). Multicriteria classification and sorting methods: A literature review. European Journal of Operational Research, 138(2), 229\u2013246. https:\/\/doi.org\/10.1016\/S0377-2217(01)00243-0","journal-title":"European Journal of Operational Research"}],"container-title":["Annals of Operations Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10479-022-05026-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10479-022-05026-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10479-022-05026-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,1]],"date-time":"2023-06-01T17:32:56Z","timestamp":1685640776000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10479-022-05026-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,2]]},"references-count":26,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,6]]}},"alternative-id":["5026"],"URL":"https:\/\/doi.org\/10.1007\/s10479-022-05026-2","relation":{},"ISSN":["0254-5330","1572-9338"],"issn-type":[{"value":"0254-5330","type":"print"},{"value":"1572-9338","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,2]]},"assertion":[{"value":"12 October 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 November 2022","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Authors have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of interest"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"Not applicable.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}}]}}