{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,29]],"date-time":"2026-07-29T14:54:38Z","timestamp":1785336878600,"version":"3.55.0"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2022,9,27]],"date-time":"2022-09-27T00:00:00Z","timestamp":1664236800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,9,27]],"date-time":"2022-09-27T00:00:00Z","timestamp":1664236800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100005874","name":"Universit\u00e0 degli Studi G. D'Annunzio Chieti Pescara","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005874","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Ann Oper Res"],"published-print":{"date-parts":[[2024,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Expert judgments are widespread in many fields, and the way in which they are collected and the procedure by which they are aggregated are considered crucial steps. From a statistical perspective, expert judgments are subjective data and must be gathered and treated as carefully and scientifically as possible. In the elicitation phase, a multitude of experts is preferable to a single expert, and techniques based on anonymity and iterations, such as Delphi, offer many advantages in terms of reducing distortions, which are mainly related to cognitive biases. There are two approaches to the aggregation of the judgments given by a panel of experts, referred to as <jats:italic>behavioural<\/jats:italic> (implying an interaction between the experts) and <jats:italic>mathematical<\/jats:italic> (involving non-interacting participants and the aggregation of the judgments using a mathematical formula). Both have advantages and disadvantages, and with the mathematical approach, the main problem concerns the subjective choice of an appropriate formula for both normalization and aggregation. We propose a new method for aggregating and processing subjective data collected using the Delphi method, with the aim of obtaining robust rankings of the outputs. This method makes it possible to normalize and aggregate the opinions of a panel of experts, while modelling different sources of uncertainty. We use an uncertainty analysis approach that allows the contemporaneous use of different aggregation and normalization functions, so that the result does not depend on the choice of a specific mathematical formula, thereby solving the problem of choice. Furthermore, we can also model the uncertainty related to the weighting system, which reflects the different expertise of the participants as well as expert opinion accuracy. By combining the Delphi method with the robust ranking procedure, we offer a new protocol covering the elicitation, the aggregation and the processing of subjective data used in the construction of Delphi-based future scenarios. The method is very flexible and can be applied to the aggregation and processing of any subjective judgments, i.e. also those outside the context of futures studies. Finally, we show the validity, reproducibility and potential of the method through its application with regard to the future of Italian families.<\/jats:p>","DOI":"10.1007\/s10479-022-04990-z","type":"journal-article","created":{"date-parts":[[2022,9,27]],"date-time":"2022-09-27T10:02:51Z","timestamp":1664272971000},"page":"1471-1493","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["Robust weighted aggregation of expert opinions in futures studies"],"prefix":"10.1007","volume":"342","author":[{"given":"Marco","family":"Marozzi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mario","family":"Bolzan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9139-1451","authenticated-orcid":false,"given":"Simone","family":"Di Zio","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,9,27]]},"reference":[{"key":"4990_CR1","unstructured":"Ambrosini, E., Rosina, A. (2009). Non \u00e8 un Paese per giovani. L'anomalia italiana: una generazione senza voce. Marsilio, Venezia."},{"key":"4990_CR2","doi-asserted-by":"publisher","DOI":"10.1201\/9781420040906","volume-title":"Elicitation of Expert Opinions for Uncertainty and Risks","author":"BM Ayyub","year":"2001","unstructured":"Ayyub, B. M. (2001). Elicitation of Expert Opinions for Uncertainty and Risks. CRC Press."},{"key":"4990_CR3","unstructured":"Benini, A., Chataigner, P., Noumri, N., Parham, N., Sweeney, J., Tax, L. (2017). The Use of Expert Judgment in Humanitarian Analysis \u2013 Theory, Methods, Applications. Geneva, Assessment Capacities Project \u2013 ACAPS."},{"issue":"1","key":"4990_CR4","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1108\/14636680710727516","volume":"9","author":"P Bishop","year":"2007","unstructured":"Bishop, P., Hines, A., & Collins, T. (2007). The current state of scenario development: An overview of techniques. Foresight, 9(1), 5\u201325.","journal-title":"Foresight"},{"issue":"37","key":"4990_CR5","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3310\/hta25370","volume":"25","author":"L Bojke","year":"2021","unstructured":"Bojke, L., Soares, M. O., Claxton, K., Colson, A., Fox, A., Jackson, C., Jankovic, D., Morton, A., Sharples, L. D., & Taylor, A. (2021). Developing a reference protocol for structured expert elicitation in health-care decision-making: A mixed-methods study. Health Technology Assessment, 25(37), 1\u2013124.","journal-title":"Health Technology Assessment"},{"key":"4990_CR6","unstructured":"Bolzan, M. (2018). Domani in Famiglia: Possibili scenari fra 10 anni. Franco Angeli Editore, Milano."},{"key":"4990_CR7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.techfore.2019.119855","volume":"151","author":"A Bonaccorsi","year":"2020","unstructured":"Bonaccorsi, A., Apreda, R., & Fantoni, G. (2020). Expert biases in technology foresight. Why they are a problem and how to mitigate them. Technological Forecasting & Social Change, 151, 1\u201317.","journal-title":"Technological Forecasting & Social Change"},{"issue":"2","key":"4990_CR8","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1111\/j.1539-6924.1999.tb00399.x","volume":"19","author":"RT Clemen","year":"1999","unstructured":"Clemen, R. T., & Winkler, R. L. (1999). Combining probability distributions from experts in risk analysis. Risk Analysis, 19(2), 187\u2013203.","journal-title":"Risk Analysis"},{"issue":"1","key":"4990_CR9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1093\/reep\/rex022","volume":"12","author":"RA Colson","year":"2018","unstructured":"Colson, R. A., & Cooke, R. M. (2018). Expert elicitation: Using the classical model to validate experts\u2019 judgments. Review of Environmental Economics and Policy, 12(1), 1\u201321.","journal-title":"Review of Environmental Economics and Policy"},{"key":"4990_CR50","doi-asserted-by":"crossref","unstructured":"Cooke, R. M. (1991). Experts in Uncertainty: Opinion and Subjective Probability in Science. New York: Oxford University Press.","DOI":"10.1093\/oso\/9780195064650.001.0001"},{"key":"4990_CR10","doi-asserted-by":"publisher","first-page":"458","DOI":"10.1287\/mnsc.9.3.458","volume":"9","author":"NC Dalkey","year":"1963","unstructured":"Dalkey, N. C., & Helmer, O. (1963). An experimental application of the Delphi method to the use of experts. Management Science, 9, 458\u2013467.","journal-title":"Management Science"},{"key":"4990_CR11","volume-title":"The Art of Conjecture","author":"B de Jouvenel","year":"1967","unstructured":"de Jouvenel, B. (1967). The Art of Conjecture. Weidenfeld and Nicolson."},{"key":"4990_CR12","doi-asserted-by":"publisher","DOI":"10.1016\/j.techfore.2021.121140","volume":"173","author":"S Di Zio","year":"2021","unstructured":"Di Zio, S., Bolzan, M., & Marozzi, M. (2021). Classification of Delphi outputs through robust ranking and fuzzy clustering for Delphi-based scenarios. Technological Forecasting and Social Change, 173, 121140.","journal-title":"Technological Forecasting and Social Change"},{"key":"4990_CR13","first-page":"32","volume-title":"Scenari e Partecipazione","author":"S Di Zio","year":"2019","unstructured":"Di Zio, S., & Pacinelli, A. (2019). Sul controllo della dinamica delle opinioni e della stabilit\u00e0 dei desiderata. In A. Pacinelli, M. Gerarda, & N. Cipolla (Eds.), Scenari e Partecipazione (pp. 32\u201349). Franco Angeli."},{"key":"4990_CR42","unstructured":"Di Zio, S., & Staniscia, B. (2014). Citizen participation and awareness raising in coastal protected areas. A case study from Italy. In A. Montanari (Ed.), Mitigating Conflicts in Coastal Areas Through Science Dissemination: Fostering Dialogue Between Researchers and Stakeholders (pp. 155\u2013197, Cap. 6, Vol. 7.), Sapienza Universit\u00e0 Editrice, Rome."},{"key":"4990_CR14","doi-asserted-by":"publisher","DOI":"10.1016\/j.techfore.2019.119736","volume":"148","author":"C Fritschy","year":"2019","unstructured":"Fritschy, C., & Spinler, S. (2019). The impact of autonomous trucks on business models in the automotive and logistics industry \u2013 A Delphi-based scenario study. Technological Forecasting and Social Change, 148, 119736.","journal-title":"Technological Forecasting and Social Change"},{"key":"4990_CR15","unstructured":"Glenn, J. C., Gordon, T. J. (2009). Futures Research Methodology \u2013 Version 3.0. The Millennium Project, American Council for the United Nations University, Washington, DC."},{"key":"4990_CR16","unstructured":"Gordon, T. J., Helmer, O. (1964). Report in a Long-Range Forecasting Study, RAND corporation Technical Paper P-2982, RAND, Santa Monica, CA, USA."},{"key":"4990_CR17","doi-asserted-by":"publisher","first-page":"234","DOI":"10.1016\/j.techfore.2015.01.013","volume":"95","author":"TJ Gordon","year":"2015","unstructured":"Gordon, T. J., Sharan, Y., & Florescu, E. (2015). Prospects for Lone Wolf and SIMAD terrorism. Technological Forecasting and Social Change, 95, 234\u2013251.","journal-title":"Technological Forecasting and Social Change"},{"key":"4990_CR18","doi-asserted-by":"publisher","unstructured":"Hanea, A. M., Nane, G. F., Bedford, T., French, S. (2021). Expert Judgement in Risk and Decision Analysis. (International Series in Operations Research & Management Science, 293) Springer, Cham, Switzerland. https:\/\/doi.org\/10.1007\/978-3-030-46474-5.","DOI":"10.1007\/978-3-030-46474-5"},{"issue":"1","key":"4990_CR19","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1111\/2041-210X.12857","volume":"9","author":"V Hemming","year":"2018","unstructured":"Hemming, V., Burgman, M. A., Hanea, A. M., McBride, M. F., & Wintle, B. C. (2018). A practical guide to structured expert elicitation using the IDEA protocol. Methods in Ecology and Evolution, 9(1), 169\u2013180.","journal-title":"Methods in Ecology and Evolution"},{"issue":"6","key":"4990_CR20","doi-asserted-by":"publisher","first-page":"1141","DOI":"10.1037\/0022-3514.50.6.1141","volume":"50","author":"DJ Isenberg","year":"1986","unstructured":"Isenberg, D. J. (1986). Group polarization: A critical review and meta-analysis. Journal of Personality and Social Psychology, 50(6), 1141\u20131151.","journal-title":"Journal of Personality and Social Psychology"},{"issue":"7","key":"4990_CR21","doi-asserted-by":"publisher","first-page":"14","DOI":"10.3102\/0013189X033007014","volume":"33","author":"RB Johnson","year":"2004","unstructured":"Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A paradigm whose time has come. Educational Researcher, 33(7), 14\u201326.","journal-title":"Educational Researcher"},{"key":"4990_CR22","doi-asserted-by":"publisher","first-page":"3513","DOI":"10.1080\/002075400422770","volume":"38","author":"K Josien","year":"2000","unstructured":"Josien, K., & Liao, T. W. (2000). Integrated use of fuzzy c-means and fuzzy KNN for GT part family and machine cell formation. International Journal of Production Research, 38, 3513\u20133536.","journal-title":"International Journal of Production Research"},{"key":"4990_CR23","unstructured":"Kosow, H., Ga\u00dfner, R. (2008). Methods of Future and Scenario Analysis: overview, Assessment, and Selection Criteria. Deutsches Institut f\u00fcr Entwicklungspolitik, Bonn."},{"issue":"2","key":"4990_CR24","doi-asserted-by":"publisher","first-page":"110","DOI":"10.1177\/002224378302000202","volume":"20","author":"JC Larr\u00e9ch\u00e9","year":"1983","unstructured":"Larr\u00e9ch\u00e9, J. C., & Moinpour, R. (1983). Managerial judgment in marketing: The concept of expertise. Journal of Marketing Research, 20(2), 110\u2013121.","journal-title":"Journal of Marketing Research"},{"key":"4990_CR25","volume-title":"The Delphi Method: Techniques and Applications","author":"HA Linstone","year":"1975","unstructured":"Linstone, H. A., & Turoff, M. (1975). The Delphi Method: Techniques and Applications. Addison-Wesley Publishing Company."},{"key":"4990_CR26","doi-asserted-by":"publisher","first-page":"759","DOI":"10.1007\/s11205-020-02490-w","volume":"153","author":"M Marozzi","year":"2021","unstructured":"Marozzi, M. (2021). Perceived justifiability towards morally debatable behaviors across Europe. Social Indicators Research, 153, 759\u2013778. https:\/\/doi.org\/10.1007\/s11205-020-02490-w","journal-title":"Social Indicators Research"},{"issue":"2","key":"4990_CR27","doi-asserted-by":"publisher","first-page":"142","DOI":"10.1006\/obhd.1998.2759","volume":"73","author":"C McCauley","year":"1998","unstructured":"McCauley, C. (1998). Group dynamics in Janis\u2019s theory of groupthink: Backward and forward. Organizational Behavior and Human Decision Processes, 73(2), 142\u2013162.","journal-title":"Organizational Behavior and Human Decision Processes"},{"key":"4990_CR28","volume-title":"Eliciting and Analyzing Expert Judgment: A Practical Guide","author":"MA Meyer","year":"1991","unstructured":"Meyer, M. A., & Booker, J. M. (1991). Eliciting and Analyzing Expert Judgment: A Practical Guide. Academic Press Limited."},{"key":"4990_CR29","doi-asserted-by":"publisher","first-page":"1603","DOI":"10.1016\/j.techfore.2011.03.006","volume":"78","author":"M Nowack","year":"2011","unstructured":"Nowack, M., Endrikat, J., & Guenther, E. (2011). Review of Delphi-based scenario studies: Quality and design considerations. Technological Forecasting and Social Change, 78, 1603\u20131615.","journal-title":"Technological Forecasting and Social Change"},{"issue":"sup1","key":"4990_CR30","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1080\/00031305.2018.1518265","volume":"73","author":"A O\u2019Hagan","year":"2019","unstructured":"O\u2019Hagan, A. (2019). Expert knowledge elicitation: Subjective but scientific. The American Statistician, 73(sup1), 69\u201381.","journal-title":"The American Statistician"},{"key":"4990_CR31","doi-asserted-by":"publisher","DOI":"10.1002\/9780470689516","volume-title":"Permutation Tests for Complex Data","author":"F Pesarin","year":"2010","unstructured":"Pesarin, F., & Salmaso, L. (2010). Permutation Tests for Complex Data. Wiley."},{"issue":"4","key":"4990_CR32","doi-asserted-by":"publisher","first-page":"353","DOI":"10.1016\/S0169-2070(99)00018-7","volume":"15","author":"G Rowe","year":"1999","unstructured":"Rowe, G., & Wright, G. (1999). The Delphi technique as a forecasting tool: Issues and analysis. International Journal of Forecasting, 15(4), 353\u2013375. https:\/\/doi.org\/10.1016\/S0169-2070(99)00018-7","journal-title":"International Journal of Forecasting"},{"key":"4990_CR33","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1111\/j.1467-985X.2005.00350.x","volume":"168","author":"M Saisana","year":"2005","unstructured":"Saisana, M., Saltelli, A., & Tarantola, S. (2005). Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators. Journal of the Royal Statistical Society Series A, 168, 307\u2013323. https:\/\/doi.org\/10.1111\/j.1467-985X.2005.00350.x","journal-title":"Journal of the Royal Statistical Society Series A"},{"issue":"1","key":"4990_CR34","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1023\/A:1014301607592","volume":"36","author":"JEM Sale","year":"2002","unstructured":"Sale, J. E. M., Lohfeld, L. H., & Brazil, K. (2002). Revisiting the quantitative\u2013qualitative debate: Implications for mixed-methods research. Quality and Quantity, 36(1), 43\u201353.","journal-title":"Quality and Quantity"},{"key":"4990_CR35","unstructured":"Saltelli, A., Saisana, M. (2010). Uncertainty and Sensitivity Analysis of the 2010 Environmental Performance Index. EUR 24269 EN. Luxembourg (Luxembourg): Publications."},{"issue":"2","key":"4990_CR36","first-page":"25","volume":"36","author":"PJH Schoemaker","year":"1995","unstructured":"Schoemaker, P. J. H. (1995). Scenario planning: A tool for strategic thinking. Sloan Management Review, 36(2), 25\u201340.","journal-title":"Sloan Management Review"},{"key":"4990_CR37","doi-asserted-by":"publisher","first-page":"525","DOI":"10.1108\/FS-11-2018-0095","volume":"5","author":"JWZ Sossa","year":"2019","unstructured":"Sossa, J. W. Z., Halal, W., & Zarta, R. H. (2019). Delphi method: Analysis of rounds, stakeholder and statistical indicators. Foresight, 5, 525\u2013544. https:\/\/doi.org\/10.1108\/FS-11-2018-0095","journal-title":"Foresight"},{"key":"4990_CR38","volume-title":"Expert Judgment in Project Management: Narrowing the Theory-Practice Gap","author":"P Szwed","year":"2016","unstructured":"Szwed, P. (2016). Expert Judgment in Project Management: Narrowing the Theory-Practice Gap. Project Management Institute Inc."},{"key":"4990_CR39","doi-asserted-by":"publisher","first-page":"1124","DOI":"10.1126\/science.185.4157.1124","volume":"185","author":"A Tversky","year":"1974","unstructured":"Tversky, A., & Kahneman, D. (1974). Judgments under uncertainty: Heuristics and biases. Science, 185, 1124\u20131131.","journal-title":"Science"},{"key":"4990_CR40","doi-asserted-by":"publisher","first-page":"811","DOI":"10.1016\/j.enpol.2013.06.110","volume":"61","author":"W Usher","year":"2013","unstructured":"Usher, W., & Strachan, N. (2013). An expert elicitation of climate, energy and economic uncertainties. Energy Policy, 61, 811\u2013821.","journal-title":"Energy Policy"},{"issue":"1","key":"4990_CR41","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1016\/j.ijpe.2010.04.013","volume":"127","author":"HA von der Gracht","year":"2010","unstructured":"von der Gracht, H. A., & Darkow, I.-L. (2010). Scenarios for the logistics services industry. A Delphi-based analysis for 2025. International Journal of Production Economics, 127(1), 46\u201359.","journal-title":"International Journal of Production Economics"}],"container-title":["Annals of Operations Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10479-022-04990-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10479-022-04990-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10479-022-04990-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,25]],"date-time":"2024-11-25T14:18:03Z","timestamp":1732544283000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10479-022-04990-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,27]]},"references-count":43,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,11]]}},"alternative-id":["4990"],"URL":"https:\/\/doi.org\/10.1007\/s10479-022-04990-z","relation":{},"ISSN":["0254-5330","1572-9338"],"issn-type":[{"value":"0254-5330","type":"print"},{"value":"1572-9338","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,27]]},"assertion":[{"value":"12 September 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 September 2022","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}