{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,3]],"date-time":"2026-02-03T18:10:56Z","timestamp":1770142256395,"version":"3.49.0"},"reference-count":74,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T00:00:00Z","timestamp":1721865600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T00:00:00Z","timestamp":1721865600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100010661","name":"Horizon 2020 Framework Programme","doi-asserted-by":"publisher","award":["773718"],"award-info":[{"award-number":["773718"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001871","name":"Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia","doi-asserted-by":"publisher","award":["2021.08977.BD"],"award-info":[{"award-number":["2021.08977.BD"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001871","name":"Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia","doi-asserted-by":"publisher","award":["10.54499\/UIDB\/05037\/2020"],"award-info":[{"award-number":["10.54499\/UIDB\/05037\/2020"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001871","name":"Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia","doi-asserted-by":"publisher","award":["10.54499\/UIDB\/04004\/2020"],"award-info":[{"award-number":["10.54499\/UIDB\/04004\/2020"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005727","name":"Universidade de Coimbra","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005727","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Ann Oper Res"],"published-print":{"date-parts":[[2025,10]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Multicriteria aggregation methods typically require inputs from decision makers concerning the relative importance of the criteria. This work presents an approach to use qualitative information elicited from a panel, which can be applied to compensatory and non-compensatory multicriteria aggregation methods. In particular, it considers the additive multiattribute value function and ELECTRE, two classical methods with well-known differences in the meaning of the criteria weights. Moreover, the proposed protocol makes a distinction between the importance of improving the current situation and the importance of not worsening the current situation. The inputs from the panel are aggregated to define constraints on the importance-related parameters, which can then be used for robustness and stochastic analyses. As a real-world application, a comparison of Integrated Pest Management (IPM) is performed, considering the case of carrots cultivation in a French region. The comparisons are based on a sustainability assessment of the current practices and alternative IPM systems, using data from field trials, laboratory experiments and preferences from stakeholders. Results are robust to weighting choices, thus identifying which changes are recommended.<\/jats:p>","DOI":"10.1007\/s10479-024-06162-7","type":"journal-article","created":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T04:01:54Z","timestamp":1721880114000},"page":"3-24","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Using qualitative information elicited from a panel to obtain robust conclusions: a protocol and an application to improve integrated pest management systems"],"prefix":"10.1007","volume":"353","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1127-1071","authenticated-orcid":false,"given":"Luis C.","family":"Dias","sequence":"first","affiliation":[]},{"given":"Pedro","family":"Marques","sequence":"additional","affiliation":[]},{"given":"Rita","family":"Garcia","sequence":"additional","affiliation":[]},{"given":"Fernanda","family":"de Santo","sequence":"additional","affiliation":[]},{"given":"Rita","family":"Tent\u00fagal","sequence":"additional","affiliation":[]},{"given":"Tiago","family":"Natal-da-Luz","sequence":"additional","affiliation":[]},{"given":"\u00c1lvaro","family":"Sousa","sequence":"additional","affiliation":[]},{"given":"Jos\u00e9 Paulo","family":"Sousa","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7269-5372","authenticated-orcid":false,"given":"Fausto","family":"Freire","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,7,25]]},"reference":[{"key":"6162_CR1","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1051\/cagri\/2020016","volume":"29","author":"N Aouadi","year":"2020","unstructured":"Aouadi, N., Macary, F., & Alonso Ugaglia, A. (2020). \u00c9valuation multicrit\u00e8re des performances socio-\u00e9conomiques et environnementales de syst\u00e8mes viticoles et de sc\u00e9narios de transition agro\u00e9cologique. Cahiers Agricultures, 29, 19. https:\/\/doi.org\/10.1051\/cagri\/2020016","journal-title":"Cahiers Agricultures"},{"key":"6162_CR2","doi-asserted-by":"publisher","DOI":"10.1016\/j.orp.2020.100156","volume":"7","author":"AH Aubert","year":"2020","unstructured":"Aubert, A. H., Esculier, F., & Lienert, J. (2020). Recommendations for online elicitation of swing weights from citizens in environmental decision-making. Operations Research Perspectives, 7, 100156. https:\/\/doi.org\/10.1016\/j.orp.2020.100156","journal-title":"Operations Research Perspectives"},{"issue":"21","key":"6162_CR3","doi-asserted-by":"publisher","first-page":"8720","DOI":"10.1073\/pnas.0702634104","volume":"104","author":"M Balinski","year":"2007","unstructured":"Balinski, M., & Laraki, R. (2007). A theory of measuring, electing, and ranking. Proceedings of the National Academy of Sciences, 104(21), 8720\u20138725. https:\/\/doi.org\/10.1073\/pnas.0702634104","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"02","key":"6162_CR4","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1142\/S0219622012400068","volume":"11","author":"CA Bana e Costa","year":"2012","unstructured":"Bana e Costa, C. A., De Corte, J.-M., & Vansnick, J.-C. (2012). MACBETH. International Journal of Information Technology & Decision Making, 11(02), 359\u2013387. https:\/\/doi.org\/10.1142\/S0219622012400068","journal-title":"International Journal of Information Technology & Decision Making"},{"issue":"1","key":"6162_CR5","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1287\/deca.1080.0104","volume":"5","author":"CA Bana e Costa","year":"2008","unstructured":"Bana e Costa, C. A., Louren\u00e7o, J. C., & Chagas, M. P. (2008). Development of reusable bid evaluation models for the portuguese electric transmission company. Decision Analysis, 5(1), 22\u201342. https:\/\/doi.org\/10.1287\/deca.1080.0104","journal-title":"Decision Analysis"},{"issue":"11","key":"6162_CR6","doi-asserted-by":"publisher","first-page":"1515","DOI":"10.1287\/mnsc.42.11.1515","volume":"42","author":"FH Barron","year":"1996","unstructured":"Barron, F. H., & Barrett, B. E. (1996). Decision quality using ranked attribute weights. Management Science, 42(11), 1515\u20131523. https:\/\/doi.org\/10.1287\/mnsc.42.11.1515","journal-title":"Management Science"},{"issue":"5","key":"6162_CR7","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1002\/mcda.305","volume":"10","author":"ML Bell","year":"2001","unstructured":"Bell, M. L., Hobbs, B. F., Elliott, E. M., Ellis, H., & Robinson, Z. (2001). An evaluation of multi-criteria methods in integrated assessment of climate policy. Journal of Multi-Criteria Decision Analysis, 10(5), 229\u2013256. https:\/\/doi.org\/10.1002\/mcda.305","journal-title":"Journal of Multi-Criteria Decision Analysis"},{"key":"6162_CR8","doi-asserted-by":"crossref","unstructured":"Belton, V., & Stewart, T. J. (2002). Multiple Criteria Decision Analysis: An Integrated Approach. Kluwer.","DOI":"10.1007\/978-1-4615-1495-4"},{"issue":"3","key":"6162_CR9","doi-asserted-by":"publisher","first-page":"283","DOI":"10.1080\/12460125.1997.10511726","volume":"6","author":"V Belton","year":"1997","unstructured":"Belton, V., & Pictet, J. (1997). A framework for group decision using a MCDA Model: Sharing, aggregating or comparing individual information? Journal of Decision Systems, 6(3), 283\u2013303. https:\/\/doi.org\/10.1080\/12460125.1997.10511726","journal-title":"Journal of Decision Systems"},{"issue":"1","key":"6162_CR10","doi-asserted-by":"publisher","first-page":"148","DOI":"10.1016\/j.ejor.2013.01.005","volume":"228","author":"PA Bottomley","year":"2013","unstructured":"Bottomley, P. A., & Doyle, J. R. (2013). Comparing the validity of numerical judgements elicited by direct rating and point allocation: Insights from objectively verifiable perceptual tasks. European Journal of Operational Research, 228(1), 148\u2013157. https:\/\/doi.org\/10.1016\/j.ejor.2013.01.005","journal-title":"European Journal of Operational Research"},{"key":"6162_CR11","doi-asserted-by":"publisher","unstructured":"Bouyssou, D., Marchant, T., Pirlot, M., Tsouki\u00e0s, A., & Vincke, P. (2015). Modelling Preferences. In R. Bisdorff, L. C. Dias, P. Meyer, V. Mousseau, & M. Pirlot (Eds.), Evaluation and Decision Models with Multiple Criteria, Springer Berlin Heidelberg. https:\/\/doi.org\/10.1007\/978-3-662-46816-6_3","DOI":"10.1007\/978-3-662-46816-6_3"},{"key":"6162_CR12","doi-asserted-by":"publisher","first-page":"531","DOI":"10.1016\/S0377-2217(96)00307-4","volume":"103","author":"J Butler","year":"1997","unstructured":"Butler, J., Jia, J., & Dyer, J. (1997). Simulation techniques for the sensitivity analysis of multi-criteria decision models. European Journal of Operational Research, 103, 531\u2013546.","journal-title":"European Journal of Operational Research"},{"key":"6162_CR13","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1016\/j.envsci.2022.09.020","volume":"138","author":"B Cicci\u00f9","year":"2022","unstructured":"Cicci\u00f9, B., Schramm, F., & Schramm, V. B. (2022). Multi-criteria decision making\/aid methods for assessing agricultural sustainability: A literature review. Environmental Science & Policy, 138, 85\u201396. https:\/\/doi.org\/10.1016\/j.envsci.2022.09.020","journal-title":"Environmental Science & Policy"},{"key":"6162_CR14","doi-asserted-by":"publisher","DOI":"10.1016\/j.omega.2020.102261","volume":"96","author":"M Cinelli","year":"2020","unstructured":"Cinelli, M., Kadzi\u0144ski, M., Gonzalez, M., & S\u0142owi\u0144ski, R. (2020). How to support the application of multiple criteria decision analysis? Let us start with a comprehensive taxonomy. Omega, 96, 102261. https:\/\/doi.org\/10.1016\/j.omega.2020.102261","journal-title":"Omega"},{"issue":"2","key":"6162_CR15","doi-asserted-by":"publisher","first-page":"633","DOI":"10.1016\/j.ejor.2022.01.011","volume":"302","author":"M Cinelli","year":"2022","unstructured":"Cinelli, M., Kadzi\u0144ski, M., Miebs, G., Gonzalez, M., & S\u0142owi\u0144ski, R. (2022). Recommending multiple criteria decision analysis methods with a new taxonomy-based decision support system. European Journal of Operational Research, 302(2), 633\u2013651. https:\/\/doi.org\/10.1016\/j.ejor.2022.01.011","journal-title":"European Journal of Operational Research"},{"issue":"3","key":"6162_CR16","doi-asserted-by":"publisher","first-page":"1530","DOI":"10.1016\/j.ejor.2005.11.052","volume":"181","author":"I Contreras","year":"2007","unstructured":"Contreras, I., & M\u00e1rmol, A. M. (2007). A lexicographical compromise method for multiple criteria group decision problems with imprecise information. European Journal of Operational Research, 181(3), 1530\u20131539. https:\/\/doi.org\/10.1016\/j.ejor.2005.11.052","journal-title":"European Journal of Operational Research"},{"issue":"4","key":"6162_CR17","doi-asserted-by":"publisher","first-page":"1464","DOI":"10.1016\/j.dss.2006.06.002","volume":"43","author":"S Damart","year":"2007","unstructured":"Damart, S., Dias, L. C., & Mousseau, V. (2007). Supporting groups in sorting decisions: Methodology and use of a multi-criteria aggregation\/disaggregation DSS. Decision Support Systems, 43(4), 1464\u20131475. https:\/\/doi.org\/10.1016\/j.dss.2006.06.002","journal-title":"Decision Support Systems"},{"issue":"1","key":"6162_CR18","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1016\/j.ejor.2015.08.058","volume":"250","author":"AT de Almeida","year":"2016","unstructured":"de Almeida, A. T., de Almeida, J. A., Costa, A. P. C. S., & de Almeida-Filho, A. T. (2016). A new method for elicitation of criteria weights in additive models: Flexible and interactive tradeoff. European Journal of Operational Research, 250(1), 179\u2013191. https:\/\/doi.org\/10.1016\/j.ejor.2015.08.058","journal-title":"European Journal of Operational Research"},{"key":"6162_CR19","doi-asserted-by":"publisher","unstructured":"Dias, L. C., & Mousseau, V. (2018). Eliciting Multi-Criteria Preferences: ELECTRE Models. In L. C. Dias, A. Morton, & J. Quigley (Eds.), Elicitation - The science and art of structuring judgement (pp. 349\u2013375). Springer. https:\/\/doi.org\/10.1007\/978-3-319-65052-4_14","DOI":"10.1007\/978-3-319-65052-4_14"},{"key":"6162_CR20","doi-asserted-by":"publisher","unstructured":"Dias, L.C. (2021). Sustainability assessment using the ELECTRE TRI multicriteria sorting method. In J. Ren (Ed.), Methods in Sustainability Science (pp. 197\u2013214). Elsevier. https:\/\/doi.org\/10.1016\/C2020-0-00430-5","DOI":"10.1016\/C2020-0-00430-5"},{"key":"6162_CR21","unstructured":"Dias, L., Marques, P., Garcia, R., Freire, F., de Santo, F., Tent\u00fagal, R., da Luz, T. N., Sousa, A., & Sousa, J. P. (2022). Deliverable 6.3 - Multi-Criteria Decision Analysis integrating Risk and Life-Cycle Assessment to compare OPTIMA IPM systems."},{"key":"6162_CR22","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1016\/j.omega.2017.04.004","volume":"76","author":"LC Dias","year":"2018","unstructured":"Dias, L. C., Antunes, C. H., Dantas, G., de Castro, N., & Zamboni, L. (2018). A multi-criteria approach to sort and rank policies based on Delphi qualitative assessments and ELECTRE TRI: The case of smart grids in Brazil. Omega, 76, 100\u2013111. https:\/\/doi.org\/10.1016\/j.omega.2017.04.004","journal-title":"Omega"},{"issue":"9","key":"6162_CR23","doi-asserted-by":"publisher","first-page":"1070","DOI":"10.1057\/palgrave.jors.2601012","volume":"51","author":"LC Dias","year":"2000","unstructured":"Dias, L. C., & Cl\u00edmaco, J. N. (2000). Additive aggregation with variable interdependent parameters: The VIP analysis software. Journal of the Operational Research Society, 51(9), 1070\u20131082. https:\/\/doi.org\/10.1057\/palgrave.jors.2601012","journal-title":"Journal of the Operational Research Society"},{"key":"6162_CR24","doi-asserted-by":"publisher","DOI":"10.1016\/j.marpolbul.2022.113803","volume":"180","author":"LC Dias","year":"2022","unstructured":"Dias, L. C., Cunha, M. C., Watkins, E., & Triantaphyllidis, G. (2022b). A multi-criteria assessment of policies to achieve the objectives of the EU marine litter strategy. Marine Pollution Bulletin, 180, 113803. https:\/\/doi.org\/10.1016\/j.marpolbul.2022.113803","journal-title":"Marine Pollution Bulletin"},{"issue":"2","key":"6162_CR25","doi-asserted-by":"publisher","first-page":"1359","DOI":"10.1007\/s10479-016-2329-7","volume":"312","author":"LC Dias","year":"2022","unstructured":"Dias, L. C., Passeira, C., Mal\u00e7a, J., & Freire, F. (2022c). Integrating life-cycle assessment and multi-criteria decision analysis to compare alternative biodiesel chains. Annals of Operations Research, 312(2), 1359\u20131374. https:\/\/doi.org\/10.1007\/s10479-016-2329-7","journal-title":"Annals of Operations Research"},{"issue":"1\u20132","key":"6162_CR26","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1007\/s10479-022-04903-0","volume":"321","author":"LC Dias","year":"2023","unstructured":"Dias, L. C., & Rocha, H. (2023). A stochastic method for exploiting outranking relations in multicriteria choice problems. Annals of Operations Research, 321(1\u20132), 165\u2013189. https:\/\/doi.org\/10.1007\/s10479-022-04903-0","journal-title":"Annals of Operations Research"},{"key":"6162_CR27","first-page":"450","volume-title":"Handbook of Research Methods and Applications in Environmental Studies","author":"LC Dias","year":"2015","unstructured":"Dias, L. C., Silva, S., & Al\u00e7ada-Almeida, L. (2015). Multi-criteria environmental sustainability assessment with an additive model. In M. Ruth (Ed.), Handbook of Research Methods and Applications in Environmental Studies (pp. 450\u2013472). Edward Elgar Publishing."},{"issue":"3","key":"6162_CR28","doi-asserted-by":"publisher","first-page":"306","DOI":"10.1006\/obhd.1994.1087","volume":"60","author":"W Edwards","year":"1994","unstructured":"Edwards, W., & Barron, F. H. (1994). SMARTS and SMARTER: Improved simple methods for multiattribute utility measurement. Organizational Behavior and Human Decision Processes, 60(3), 306\u2013325. https:\/\/doi.org\/10.1006\/obhd.1994.1087","journal-title":"Organizational Behavior and Human Decision Processes"},{"key":"6162_CR29","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1002\/mcda.1482","volume":"20","author":"J Figueira","year":"2013","unstructured":"Figueira, J., Greco, S., Roy, B., & Slowinski, R. (2013). An overview of ELECTRE methods and their recent extensions. Journal of Multi-Criteria Decision Analysis, 20, 61\u201385.","journal-title":"Journal of Multi-Criteria Decision Analysis"},{"issue":"2","key":"6162_CR30","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1016\/S0377-2217(01)00370-8","volume":"139","author":"J Figueira","year":"2002","unstructured":"Figueira, J., & Roy, B. (2002). Determining the weights of criteria in the ELECTRE type methods with a revised Simos\u2019 procedure. European Journal of Operational Research, 139(2), 317\u2013326. https:\/\/doi.org\/10.1016\/S0377-2217(01)00370-8","journal-title":"European Journal of Operational Research"},{"issue":"1","key":"6162_CR31","doi-asserted-by":"publisher","first-page":"593","DOI":"10.1186\/s12913-023-09550-0","volume":"23","author":"L Freitas","year":"2023","unstructured":"Freitas, L., Vieira, A. C. L., Oliveira, M. D., Monteiro, H., & Bana e Costa, C. A. (2023). Which value aspects are relevant for the evaluation of medical devices? Exploring stakeholders\u2019 views through a Web-Delphi process. BMC Health Services Research, 23(1), 593. https:\/\/doi.org\/10.1186\/s12913-023-09550-0","journal-title":"BMC Health Services Research"},{"issue":"2","key":"6162_CR32","doi-asserted-by":"publisher","first-page":"227","DOI":"10.1016\/S0195-9255(99)00036-0","volume":"20","author":"P Girardin","year":"2000","unstructured":"Girardin, P., Bockstaller, C., & Van der Werf, H. (2000). Assessment of potential impacts of agricultural practices on the environment. Environmental Impact Assessment Review, 20(2), 227\u2013239. https:\/\/doi.org\/10.1016\/S0195-9255(99)00036-0","journal-title":"Environmental Impact Assessment Review"},{"issue":"1\u20132","key":"6162_CR33","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1016\/S1161-0301(97)00014-2","volume":"8","author":"C Giupponi","year":"1998","unstructured":"Giupponi, C. (1998). Environmental evaluation of alternative cropping systems with impact indices of pollution. European Journal of Agronomy, 8(1\u20132), 71\u201382. https:\/\/doi.org\/10.1016\/S1161-0301(97)00014-2","journal-title":"European Journal of Agronomy"},{"key":"6162_CR34","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4939-3094-4","volume-title":"Multiple Criteria Decision Analysis - State of the Art Surveys","author":"S Greco","year":"2016","unstructured":"Greco, S., Ehrgott, M., & Figueira, J. R. (2016). Multiple Criteria Decision Analysis - State of the Art Surveys. Springer-Verlag."},{"issue":"19","key":"6162_CR35","doi-asserted-by":"publisher","first-page":"3578","DOI":"10.1016\/j.scitotenv.2011.06.022","volume":"409","author":"IB Huang","year":"2011","unstructured":"Huang, I. B., Keisler, J., & Linkov, I. (2011). Multi-criteria decision analysis in environmental sciences: Ten years of applications and trends. The Science of the Total Environment, 409(19), 3578\u20133594. https:\/\/doi.org\/10.1016\/j.scitotenv.2011.06.022","journal-title":"The Science of the Total Environment"},{"key":"6162_CR36","doi-asserted-by":"crossref","unstructured":"Ishizaka, A., & Nemery, P. (2013). Multi-criteria Decision Analysis: Methods and Software. Wiley.","DOI":"10.1002\/9781118644898"},{"key":"6162_CR37","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1016\/S0377-2217(00)00035-7","volume":"130","author":"E Jacquet-Lagr\u00e8ze","year":"2001","unstructured":"Jacquet-Lagr\u00e8ze, E., & Siskos, Y. (2001). Preference disaggregation: 20 years of MCDA experience. European Journal of Operational Research, 130, 233\u2013245.","journal-title":"European Journal of Operational Research"},{"issue":"12","key":"6162_CR38","doi-asserted-by":"publisher","first-page":"1217","DOI":"10.1111\/j.1365-2929.2004.02012.x","volume":"38","author":"S Jamieson","year":"2004","unstructured":"Jamieson, S. (2004). Likert scales: How to (ab)use them. Medical Education, 38(12), 1217\u20131218. https:\/\/doi.org\/10.1111\/j.1365-2929.2004.02012.x","journal-title":"Medical Education"},{"issue":"1","key":"6162_CR39","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1016\/j.ejor.2013.01.022","volume":"228","author":"M Kadzi\u0144ski","year":"2013","unstructured":"Kadzi\u0144ski, M., & Tervonen, T. (2013). Robust multi-criteria ranking with additive value models and holistic pair-wise preference statements. European Journal of Operational Research, 228(1), 169\u2013180. https:\/\/doi.org\/10.1016\/j.ejor.2013.01.022","journal-title":"European Journal of Operational Research"},{"key":"6162_CR40","doi-asserted-by":"publisher","first-page":"126","DOI":"10.1016\/j.omega.2014.06.004","volume":"55","author":"M Kadzi\u0144ski","year":"2015","unstructured":"Kadzi\u0144ski, M., Tervonen, T., & Figueira, J.R. (2015). Robust multi-criteria sorting with the outranking preference model and characteristic profiles. Omega, 55, 126\u2013140. https:\/\/doi.org\/10.1016\/j.omega.2014.06.004","journal-title":"Omega"},{"issue":"5","key":"6162_CR41","doi-asserted-by":"publisher","first-page":"257","DOI":"10.1002\/mcda.306","volume":"10","author":"J Kangas","year":"2001","unstructured":"Kangas, J., Kangas, A., Leskinen, P., & Pyk\u00e4l\u00e4inen, J. (2001). MCDM methods in strategic planning of forestry on state-owned lands in Finland: Applications and experiences. Journal of Multi-Criteria Decision Analysis, 10(5), 257\u2013271. https:\/\/doi.org\/10.1002\/mcda.306","journal-title":"Journal of Multi-Criteria Decision Analysis"},{"key":"6162_CR42","volume-title":"Value-Focused Thinking","author":"RL Keeney","year":"1992","unstructured":"Keeney, R. L. (1992). Value-Focused Thinking. Harvard University Press."},{"key":"6162_CR43","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781139174084","volume-title":"Decisions with multiple objectives\u2013preferences and value tradeoffs","author":"RL Keeney","year":"1993","unstructured":"Keeney, R. L., & Raiffa, H. (1993). Decisions with multiple objectives\u2013preferences and value tradeoffs. Cambridge University Press."},{"key":"6162_CR44","doi-asserted-by":"publisher","unstructured":"Lahdelma, R. (1998). S M A A - Stochastic multiobjective acceptability analysis. European Journal of Operational Research, 106(1), 137\u2013143. https:\/\/doi.org\/10.1016\/S0377-2217(97)00163-X","DOI":"10.1016\/S0377-2217(97)00163-X"},{"key":"6162_CR45","doi-asserted-by":"publisher","first-page":"137","DOI":"10.1016\/S0377-2217(97)00163-X","volume":"106","author":"R Lahdelma","year":"1998","unstructured":"Lahdelma, R., Hokkanen, J., & Salminen, P. (1998). SMAA - Stochastic multiobjective acceptability analysis. European Journal of Operational Research, 106, 137\u2013143.","journal-title":"European Journal of Operational Research"},{"issue":"2","key":"6162_CR46","doi-asserted-by":"publisher","first-page":"454","DOI":"10.1016\/S0377-2217(01)00276-4","volume":"141","author":"R Lahdelma","year":"2002","unstructured":"Lahdelma, R., & Salminen, P. (2002). Pseudo-criteria versus linear utility function in stochastic multi-criteria acceptability analysis. European Journal of Operational Research, 141(2), 454\u2013469. https:\/\/doi.org\/10.1016\/S0377-2217(01)00276-4","journal-title":"European Journal of Operational Research"},{"key":"6162_CR47","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1016\/j.ins.2014.02.125","volume":"271","author":"H Liao","year":"2014","unstructured":"Liao, H., Xu, Z., & Zeng, X.-J. (2014). Distance and similarity measures for hesitant fuzzy linguistic term sets and their application in multi-criteria decision making. Information Sciences, 271, 125\u2013142. https:\/\/doi.org\/10.1016\/j.ins.2014.02.125","journal-title":"Information Sciences"},{"key":"6162_CR48","doi-asserted-by":"publisher","DOI":"10.1016\/j.indic.2021.100149","volume":"12","author":"A Lindfors","year":"2021","unstructured":"Lindfors, A. (2021). Assessing sustainability with multi-criteria methods: A methodologically focused literature review. Environmental and Sustainability Indicators, 12, 100149. https:\/\/doi.org\/10.1016\/j.indic.2021.100149","journal-title":"Environmental and Sustainability Indicators"},{"key":"6162_CR49","doi-asserted-by":"publisher","DOI":"10.1201\/9780429326448","volume-title":"Multi-criteria decision analysis: Case studies in engineering and the environment","author":"I Linkov","year":"2020","unstructured":"Linkov, I., Moberg, E., Trump, B. D., Yatsalo, B., & Keisler, J. M. (2020). Multi-criteria decision analysis: Case studies in engineering and the environment. CRC Press."},{"issue":"2\/3","key":"6162_CR50","doi-asserted-by":"publisher","first-page":"178","DOI":"10.1504\/IJMCDM.2013.053725","volume":"3","author":"F Macary","year":"2013","unstructured":"Macary, F., Dias, J. A., Uny, D., & Probst, A. (2013). Assessment of the effects of best environmental practices on reducing pesticide contamination in surface water, using multi-criteria modelling combined with a GIS. International Journal of Multicriteria Decision Making, 3(2\/3), 178. https:\/\/doi.org\/10.1504\/IJMCDM.2013.053725","journal-title":"International Journal of Multicriteria Decision Making"},{"issue":"1\u20132","key":"6162_CR51","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1007\/s40070-013-0016-3","volume":"3","author":"M Marttunen","year":"2015","unstructured":"Marttunen, M., Mustajoki, J., Dufva, M., & Karjalainen, T. (2015). How to design and realize participation of stakeholders in MCDA processes? A framework for selecting an appropriate approach. EURO Journal on Decision Processes, 3(1\u20132), 187\u2013214. https:\/\/doi.org\/10.1007\/s40070-013-0016-3","journal-title":"EURO Journal on Decision Processes"},{"key":"6162_CR52","doi-asserted-by":"publisher","unstructured":"Matsatsinis, N. F., Grigoroudis, E., & Siskos, E. (2018). Disaggregation Approach to Value Elicitation. In L. C. Dias, A. Morton, & J. Quigley (Eds.), Elicitation - The Science and Art of Structuring Judgement (pp. 313\u2013348). Spinger. https:\/\/doi.org\/10.1007\/978-3-319-65052-4_13","DOI":"10.1007\/978-3-319-65052-4_13"},{"key":"6162_CR53","doi-asserted-by":"publisher","unstructured":"Morton, A. (2018). Multiattribute Value Elicitation. In L. C Dias, A. Morton, & J. Quigley (Eds.), Elicitation: The Science and Art of Structuring Judgement (pp. 287\u2013311). Springer International Publishing. https:\/\/doi.org\/10.1007\/978-3-319-65052-4_12","DOI":"10.1007\/978-3-319-65052-4_12"},{"issue":"11","key":"6162_CR54","doi-asserted-by":"publisher","first-page":"902","DOI":"10.1071\/CP17064","volume":"68","author":"E Pelzer","year":"2017","unstructured":"Pelzer, E., Bourlet, C., Carlsson, G., Lopez-Bellido, R. J., Jensen, E. S., & Jeuffroy, M.-H. (2017). Design, assessment and feasibility of legume-based cropping systems in three European regions. Crop and Pasture Science, 68(11), 902. https:\/\/doi.org\/10.1071\/CP17064","journal-title":"Crop and Pasture Science"},{"issue":"1","key":"6162_CR55","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1007\/s10479-007-0183-3","volume":"154","author":"LD Phillips","year":"2007","unstructured":"Phillips, L. D., & Bana e Costa, C. A. (2007). Transparent prioritisation, budgeting and resource allocation with multi-criteria decision analysis and decision conferencing. Annals of Operations Research, 154(1), 51\u201368. https:\/\/doi.org\/10.1007\/s10479-007-0183-3","journal-title":"Annals of Operations Research"},{"key":"6162_CR56","doi-asserted-by":"publisher","DOI":"10.1016\/j.agsy.2021.103153","volume":"191","author":"C Puech","year":"2021","unstructured":"Puech, C., Brulaire, A., Paraiso, J., & Faloya, V. (2021). Collective design of innovative agroecological cropping systems for the industrial vegetable sector. Agricultural Systems, 191, 103153. https:\/\/doi.org\/10.1016\/j.agsy.2021.103153","journal-title":"Agricultural Systems"},{"key":"6162_CR57","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1016\/j.omega.2014.11.009","volume":"53","author":"J Rezaei","year":"2015","unstructured":"Rezaei, J. (2015). Best-worst multi-criteria decision-making method. Omega, 53, 49\u201357. https:\/\/doi.org\/10.1016\/j.omega.2014.11.009","journal-title":"Omega"},{"issue":"3","key":"6162_CR58","doi-asserted-by":"publisher","first-page":"463","DOI":"10.1016\/S0921-8009(01)00245-2","volume":"39","author":"D Rigby","year":"2001","unstructured":"Rigby, D., Woodhouse, P., Young, T., & Burton, M. (2001). Constructing a farm level indicator of sustainable agricultural practice. Ecological Economics, 39(3), 463\u2013478. https:\/\/doi.org\/10.1016\/S0921-8009(01)00245-2","journal-title":"Ecological Economics"},{"issue":"4","key":"6162_CR59","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1016\/j.eiar.2009.10.002","volume":"30","author":"GS Rodrigues","year":"2010","unstructured":"Rodrigues, G. S., Rodrigues, I. A., de Almeida Buschinelli, C. C., & De Barros, I. (2010). Integrated farm sustainability assessment for the environmental management of rural activities. Environmental Impact Assessment Review, 30(4), 229\u2013239. https:\/\/doi.org\/10.1016\/j.eiar.2009.10.002","journal-title":"Environmental Impact Assessment Review"},{"issue":"31","key":"6162_CR60","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1007\/BF00134132","volume":"1","author":"B Roy","year":"1991","unstructured":"Roy, B. (1991). The Outranking Approach and the Foundations of Electre Methods. Theory and Decision, 1(31), 49\u201373.","journal-title":"Theory and Decision"},{"key":"6162_CR61","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-2500-1","volume-title":"Multicriteria Methodology for Decision Aiding","author":"B Roy","year":"1996","unstructured":"Roy, B. (1996). Multicriteria Methodology for Decision Aiding. Kluwer Academic."},{"issue":"2","key":"6162_CR62","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1002\/(SICI)1099-1360(199606)5:2<145::AID-MCDA99>3.0.CO;2-5","volume":"5","author":"B Roy","year":"1996","unstructured":"Roy, B., & Mousseau, V. (1996). A theoretical framework for analysing the notion of relative importance of criteria. Journal of Multi-Criteria Decision Analysis, 5(2), 145\u2013159. https:\/\/doi.org\/10.1002\/(SICI)1099-1360(199606)5:2%3c145::AID-MCDA99%3e3.0.CO;2-5","journal-title":"Journal of Multi-Criteria Decision Analysis"},{"key":"6162_CR63","doi-asserted-by":"publisher","DOI":"10.1504\/IJSSCI.2008.017590","author":"TL Saaty","year":"2008","unstructured":"Saaty, T. L. (2008). Decision making with the analytic hierarchy process. International Journal of Services Sciences. https:\/\/doi.org\/10.1504\/IJSSCI.2008.017590","journal-title":"International Journal of Services Sciences"},{"issue":"3","key":"6162_CR64","doi-asserted-by":"publisher","first-page":"485","DOI":"10.1016\/S0377-2217(96)00370-0","volume":"104","author":"P Salminen","year":"1998","unstructured":"Salminen, P., Hokkanen, J., & Lahdelma, R. (1998). Comparing multicriteria methods in the context of environmental problems. European Journal of Operational Research, 104(3), 485\u2013496. https:\/\/doi.org\/10.1016\/S0377-2217(96)00370-0","journal-title":"European Journal of Operational Research"},{"issue":"3","key":"6162_CR65","doi-asserted-by":"publisher","first-page":"545","DOI":"10.1109\/TSMCA.2009.2014555","volume":"39","author":"P Sarabando","year":"2009","unstructured":"Sarabando, P., & Dias, L. C. (2009). Multiattribute choice with ordinal information: A comparison of different decision rules. IEEE Transactions on Systems, Man, and Cybernetics - Part a: Systems and Humans, 39(3), 545\u2013554. https:\/\/doi.org\/10.1109\/TSMCA.2009.2014555","journal-title":"IEEE Transactions on Systems, Man, and Cybernetics - Part a: Systems and Humans"},{"issue":"2","key":"6162_CR66","doi-asserted-by":"publisher","first-page":"472","DOI":"10.1016\/j.ejor.2012.01.055","volume":"220","author":"N Schuwirth","year":"2012","unstructured":"Schuwirth, N., Reichert, P., & Lienert, J. (2012). Methodological aspects of multi-criteria decision analysis for policy support\u202f: A case study on pharmaceutical removal from hospital wastewater. European Journal of Operational Research, 220(2), 472\u2013483. https:\/\/doi.org\/10.1016\/j.ejor.2012.01.055","journal-title":"European Journal of Operational Research"},{"key":"6162_CR67","doi-asserted-by":"publisher","unstructured":"Siskos, Y., Grigoroudis, E., & Matsatsinis, N. F. (2016). UTA Methods. In S. Greco, M. Ehrgott, & J. R. Figueira (Eds.), Multiple Criteria Decision Analysis - State of the Art Surveys (pp. 315\u2013362). Springer-Verlag. https:\/\/doi.org\/10.1007\/978-1-4939-3094-4_9","DOI":"10.1007\/978-1-4939-3094-4_9"},{"issue":"5","key":"6162_CR68","doi-asserted-by":"publisher","first-page":"1786","DOI":"10.1111\/itor.12512","volume":"26","author":"O Sobrie","year":"2019","unstructured":"Sobrie, O., Mousseau, V., & Pirlot, M. (2019). Learning monotone preferences using a majority rule sorting model. International Transactions in Operational Research, 26(5), 1786\u20131809. https:\/\/doi.org\/10.1111\/itor.12512","journal-title":"International Transactions in Operational Research"},{"key":"6162_CR69","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclepro.2021.129291","volume":"325","author":"E Soul\u00e9","year":"2021","unstructured":"Soul\u00e9, E., Michonneau, P., Michel, N., & Bockstaller, C. (2021). Environmental sustainability assessment in agricultural systems: A conceptual and methodological review. Journal of Cleaner Production, 325, 129291. https:\/\/doi.org\/10.1016\/j.jclepro.2021.129291","journal-title":"Journal of Cleaner Production"},{"issue":"4","key":"6162_CR70","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1017\/S0889189300005403","volume":"8","author":"DC Taylor","year":"1993","unstructured":"Taylor, D. C., Mohamed, Z. A., Shamsudin, M. N., Mohayidin, M. G., & Chiew, E. F. C. (1993). Creating a farmer sustainability index: A Malaysian case study. American Journal of Alternative Agriculture, 8(4), 175\u2013184. https:\/\/doi.org\/10.1017\/S0889189300005403","journal-title":"American Journal of Alternative Agriculture"},{"issue":"2","key":"6162_CR71","doi-asserted-by":"publisher","first-page":"500","DOI":"10.1016\/j.ejor.2005.12.037","volume":"178","author":"T Tervonen","year":"2007","unstructured":"Tervonen, T., & Lahdelma, R. (2007). Implementing stochastic multicriteria acceptability analysis. European Journal of Operational Research, 178(2), 500\u2013513. https:\/\/doi.org\/10.1016\/j.ejor.2005.12.037","journal-title":"European Journal of Operational Research"},{"issue":"3","key":"6162_CR72","doi-asserted-by":"publisher","first-page":"738","DOI":"10.3390\/su11030738","volume":"11","author":"P Trivino-Tarradas","year":"2019","unstructured":"Trivino-Tarradas, P., Gomez-Ariza, M., Basch, G., & Gonzalez-Sanchez, E. (2019). Sustainability assessment of annual and permanent crops: The Inspia Model. Sustainability, 11(3), 738. https:\/\/doi.org\/10.3390\/su11030738","journal-title":"Sustainability"},{"key":"6162_CR73","doi-asserted-by":"crossref","unstructured":"Tzeng, G.-H., & Huang, J.-J. (2011). Multiple attribute decision making: Methods and applications. Springer.","DOI":"10.1201\/b11032"},{"issue":"4","key":"6162_CR74","doi-asserted-by":"publisher","first-page":"527","DOI":"10.1080\/01488376.2017.1329775","volume":"43","author":"H Wu","year":"2017","unstructured":"Wu, H., & Leung, S.-O. (2017). Can likert scales be treated as interval scales?\u2014A simulation study. Journal of Social Service Research, 43(4), 527\u2013532. https:\/\/doi.org\/10.1080\/01488376.2017.1329775","journal-title":"Journal of Social Service Research"}],"container-title":["Annals of Operations Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10479-024-06162-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10479-024-06162-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10479-024-06162-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,29]],"date-time":"2025-09-29T15:10:50Z","timestamp":1759158650000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10479-024-06162-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,25]]},"references-count":74,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,10]]}},"alternative-id":["6162"],"URL":"https:\/\/doi.org\/10.1007\/s10479-024-06162-7","relation":{},"ISSN":["0254-5330","1572-9338"],"issn-type":[{"value":"0254-5330","type":"print"},{"value":"1572-9338","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,25]]},"assertion":[{"value":"2 January 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 July 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 July 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no financial or non-financial competing interests that are directly or indirectly related to this work.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}