{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,5,2]],"date-time":"2025-05-02T13:55:55Z","timestamp":1746194155589,"version":"3.37.3"},"reference-count":37,"publisher":"Springer Science and Business Media LLC","issue":"2-3","license":[{"start":{"date-parts":[[2024,4,17]],"date-time":"2024-04-17T00:00:00Z","timestamp":1713312000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,4,17]],"date-time":"2024-04-17T00:00:00Z","timestamp":1713312000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Spanish Ministry of Science","award":["PID2020-115429GB-I00"],"award-info":[{"award-number":["PID2020-115429GB-I00"]}]},{"name":"Regional Government of Andaluc\u00eda","award":["SEJ-532","P18-RT-1566"],"award-info":[{"award-number":["SEJ-532","P18-RT-1566"]}]},{"DOI":"10.13039\/100009473","name":"University of M\u00e1laga","doi-asserted-by":"crossref","award":["B1-2020-18"],"award-info":[{"award-number":["B1-2020-18"]}],"id":[{"id":"10.13039\/100009473","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100002341","name":"Academy of Finland","doi-asserted-by":"publisher","award":["322221"],"award-info":[{"award-number":["322221"]}],"id":[{"id":"10.13039\/501100002341","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Spanish Ministry of Science","award":["PID2019-104263RB-C42"],"award-info":[{"award-number":["PID2019-104263RB-C42"]}]},{"DOI":"10.13039\/501100005222","name":"University of Jyv\u00e4skyl\u00e4","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005222","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Ann Oper Res"],"published-print":{"date-parts":[[2024,7]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In multiobjective optimization problems, Pareto optimal solutions representing different tradeoffs cannot be ordered without incorporating preference information of a decision maker (DM). In interactive methods, the DM takes an active part in the solution process and provides preference information iteratively. Between iterations, the DM can learn how achievable the preferences are, learn about the tradeoffs, and adjust the preferences. Different interactive methods have been proposed in the literature, but the question of how to select the best-suited method for a problem to be solved remains partly open. We propose an experimental design for evaluating interactive methods according to several desirable properties related to the cognitive load experienced by the DM, the method\u2019s ability to capture preferences and its responsiveness to changes in the preferences, the DM\u2019s satisfaction in the overall solution process, and their confidence in the final solution. In the questionnaire designed, we connect each questionnaire item to be asked with a relevant research question characterizing these desirable properties of interactive methods. We also conduct a between-subjects experiment to compare three interactive methods and report interesting findings. In particular, we find out that trade-off-free methods may be more suitable for exploring the whole set of Pareto optimal solutions, while classification-based methods seem to work better for fine-tuning the preferences to find the final solution.<\/jats:p>","DOI":"10.1007\/s10479-024-05941-6","type":"journal-article","created":{"date-parts":[[2024,4,17]],"date-time":"2024-04-17T13:02:03Z","timestamp":1713358923000},"page":"835-856","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["An experimental design for comparing interactive methods based on their desirable properties"],"prefix":"10.1007","volume":"338","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3643-2342","authenticated-orcid":false,"given":"Bekir","family":"Afsar","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0763-0297","authenticated-orcid":false,"given":"Johanna","family":"Silvennoinen","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2612-009X","authenticated-orcid":false,"given":"Francisco","family":"Ruiz","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0543-8055","authenticated-orcid":false,"given":"Ana B.","family":"Ruiz","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4673-7388","authenticated-orcid":false,"given":"Giovanni","family":"Misitano","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1013-4689","authenticated-orcid":false,"given":"Kaisa","family":"Miettinen","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,4,17]]},"reference":[{"issue":"4","key":"5941_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3448301","volume":"54","author":"B Afsar","year":"2021","unstructured":"Afsar, B., Miettinen, K., & Ruiz, F. (2021). Assessing the performance of interactive multiobjective optimization methods: A survey. ACM Computing Surveys, 54(4), 1\u201327. Art. no. 85.","journal-title":"ACM Computing Surveys"},{"issue":"11","key":"5941_CR2","doi-asserted-by":"publisher","first-page":"2327","DOI":"10.1080\/01605682.2022.2141145","volume":"74","author":"B Afsar","year":"2023","unstructured":"Afsar, B., Silvennoinen, J., Misitano, G., Ruiz, F., Ruiz, A. B., & Miettinen, K. (2023). Designing empirical experiments to compare interactive multiobjective optimization methods. Journal of the Operational Research Society, 74(11), 2327\u20132338.","journal-title":"Journal of the Operational Research Society"},{"key":"5941_CR3","doi-asserted-by":"publisher","first-page":"405","DOI":"10.1007\/978-3-540-88908-3_15","volume-title":"Multiobjective Optimization: Interactive and Evolutionary Approaches","author":"V Belton","year":"2008","unstructured":"Belton, V., Branke, J., Eskelinen, P., Greco, S., Molina, J., Ruiz, F., & S\u0142owi\u0144ski, R. (2008). Interactive multiobjective optimization from a learning perspective. In J. Branke, K. Deb, K. Miettinen, & R. Slowinski (Eds.), Multiobjective Optimization: Interactive and Evolutionary Approaches (pp. 405\u2013434). Berlin: Springer."},{"issue":"2","key":"5941_CR4","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1016\/0377-2217(85)90224-3","volume":"22","author":"K Brockhoff","year":"1985","unstructured":"Brockhoff, K. (1985). Experimental test of MCDM algorithms in a modular approach. European Journal of Operational Research, 22(2), 159\u2013166.","journal-title":"European Journal of Operational Research"},{"issue":"9","key":"5941_CR5","doi-asserted-by":"publisher","first-page":"1050","DOI":"10.1057\/jors.1994.170","volume":"45","author":"JT Buchanan","year":"1994","unstructured":"Buchanan, J. T. (1994). An experimental evaluation of interactive MCDM methods and the decision making process. Journal of the Operational Research Society, 45(9), 1050\u20131059.","journal-title":"Journal of the Operational Research Society"},{"issue":"3","key":"5941_CR6","doi-asserted-by":"publisher","first-page":"353","DOI":"10.1016\/0377-2217(87)90248-7","volume":"29","author":"JT Buchanan","year":"1987","unstructured":"Buchanan, J. T., & Daellenbach, H. G. (1987). A comparative evaluation of interactive solution methods for multiple objective decision models. European Journal of Operational Research, 29(3), 353\u2013359.","journal-title":"European Journal of Operational Research"},{"issue":"4","key":"5941_CR7","doi-asserted-by":"publisher","first-page":"293","DOI":"10.1207\/s1532690xci0804_2","volume":"8","author":"P Chandler","year":"1991","unstructured":"Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and Instruction, 8(4), 293\u2013332.","journal-title":"Cognition and Instruction"},{"key":"5941_CR8","volume-title":"Multiobjective Decision Making: Theory and Methodology","author":"V Chankong","year":"1983","unstructured":"Chankong, V., & Haimes, Y. Y. (1983). Multiobjective Decision Making: Theory and Methodology. North-Holland."},{"key":"5941_CR9","volume-title":"Quasi-experimentation: Design and Analysis Issues for Field Settings","author":"TD Cook","year":"1979","unstructured":"Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and Analysis Issues for Field Settings. Houghton Mifflin Company."},{"key":"5941_CR10","doi-asserted-by":"publisher","first-page":"904","DOI":"10.1177\/154193120605000909","volume":"50","author":"SG Hart","year":"2006","unstructured":"Hart, S. G. (2006). NASA-task load index (NASA-TLX); 20 years later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 50, 904\u2013908.","journal-title":"Proceedings of the Human Factors and Ergonomics Society Annual Meeting"},{"key":"5941_CR11","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1016\/S0166-4115(08)62386-9","volume":"52","author":"SG Hart","year":"1988","unstructured":"Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (task load index): Results of empirical and theoretical research. Advances in Psychology, 52, 139\u2013183.","journal-title":"Advances in Psychology"},{"issue":"2","key":"5941_CR12","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1504\/IJLSM.2023.134404","volume":"46","author":"R Heikkinen","year":"2023","unstructured":"Heikkinen, R., Sipila, J., Ojalehto, V., & Miettinen, K. (2023). Flexible data driven inventory management with interactive multiobjective lot size optimization. International Journal of Logistics Systems and Management, 46(2), 206\u2013235.","journal-title":"International Journal of Logistics Systems and Management"},{"key":"5941_CR13","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-45511-7","volume-title":"Multiple Objective Decision Making - Methods and Applications: A State-of-the-Art Survey","author":"C-L Hwang","year":"1979","unstructured":"Hwang, C.-L., & Masud, A. S. M. (1979). Multiple Objective Decision Making - Methods and Applications: A State-of-the-Art Survey. Springer."},{"issue":"4","key":"5941_CR14","doi-asserted-by":"publisher","first-page":"396","DOI":"10.9734\/BJAST\/2015\/14975","volume":"7","author":"A Joshi","year":"2015","unstructured":"Joshi, A., Kale, S., Chandel, S., & Pal, D. K. (2015). Likert scale: Explored and explained. British Journal of Applied Science & Technology, 7(4), 396\u2013403.","journal-title":"British Journal of Applied Science & Technology"},{"issue":"1","key":"5941_CR15","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1016\/0377-2217(86)90162-1","volume":"26","author":"M Kok","year":"1986","unstructured":"Kok, M. (1986). The interface with decision makers and some experimental results in interactive multiple objective programming methods. European Journal of Operational Research, 26(1), 96\u2013107.","journal-title":"European Journal of Operational Research"},{"key":"5941_CR16","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1007\/978-3-662-22160-0_21","volume-title":"Methodology and Software for Interactive Decision Support","author":"P Korhonen","year":"1989","unstructured":"Korhonen, P., & Wallenius, J. (1989). Observations regarding choice behaviour in interactive multiple criteria decision-making environments: An experimental investigation. In A. Lewandowski & I. Stanchev (Eds.), Methodology and Software for Interactive Decision Support (pp. 163\u2013170). Berlin: Springer."},{"issue":"260","key":"5941_CR17","doi-asserted-by":"publisher","first-page":"583","DOI":"10.1080\/01621459.1952.10483441","volume":"47","author":"WH Kruskal","year":"1952","unstructured":"Kruskal, W. H., & Wallis, W. A. (1952). Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association, 47(260), 583\u2013621.","journal-title":"Journal of the American Statistical Association"},{"issue":"1","key":"5941_CR18","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1080\/10447319509526110","volume":"7","author":"JR Lewis","year":"1995","unstructured":"Lewis, J. R. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7(1), 57\u201378.","journal-title":"International Journal of Human-Computer Interaction"},{"issue":"140","key":"5941_CR19","first-page":"5","volume":"22","author":"R Likert","year":"1932","unstructured":"Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 22(140), 5\u201355.","journal-title":"Archives of Psychology"},{"key":"5941_CR20","doi-asserted-by":"crossref","unstructured":"L\u00f3pez-Ib\u00e1\u00f1ez, M., & Knowles, J. (2015). Machine decision makers as a laboratory for interactive EMO. In A. Gaspar-Cunha, C. Henggeler Antunes, & C. Coello Coello (Eds.) Evolutionary Multi-criterion Optimization, 8th International Conference, Proceedings, Part II (pp. 295\u2013309). Springer.","DOI":"10.1007\/978-3-319-15892-1_20"},{"issue":"1","key":"5941_CR21","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1007\/s00291-008-0154-3","volume":"33","author":"M Luque","year":"2011","unstructured":"Luque, M., Ruiz, F., & Miettinen, K. (2011). Global formulation for interactive multiobjective optimization. OR Spectrum, 33(1), 27\u201348.","journal-title":"OR Spectrum"},{"issue":"3","key":"5941_CR22","doi-asserted-by":"publisher","first-page":"17:1","DOI":"10.1145\/2808234","volume":"5","author":"D Meignan","year":"2015","unstructured":"Meignan, D., Knust, S., Frayret, J. M., Pesant, G., & Gaud, N. (2015). A review and taxonomy of interactive optimization methods in operations research. ACM Transactions on Interactive Intelligent Systems, 5(3), 17:1-17:43.","journal-title":"ACM Transactions on Interactive Intelligent Systems"},{"key":"5941_CR23","volume-title":"Nonlinear Multiobjective Optimization","author":"K Miettinen","year":"1999","unstructured":"Miettinen, K. (1999). Nonlinear Multiobjective Optimization. Kluwer Academic Publishers."},{"key":"5941_CR24","first-page":"931","volume-title":"Multiple Criteria Decision Analysis: State of the Art Surveys","author":"K Miettinen","year":"2016","unstructured":"Miettinen, K., Hakanen, J., & Podkopaev, D. (2016). Interactive nonlinear multiobjective optimization methods. In S. Greco, M. Ehrgott, & J. Figueira (Eds.), Multiple Criteria Decision Analysis: State of the Art Surveys (Vol. 2, pp. 931\u2013980). Springer."},{"issue":"3","key":"5941_CR25","doi-asserted-by":"publisher","first-page":"909","DOI":"10.1016\/j.ejor.2004.07.052","volume":"170","author":"K Miettinen","year":"2006","unstructured":"Miettinen, K., & M\u00e4kel\u00e4, M. M. (2006). Synchronous approach in interactive multiobjective optimization. European Journal of Operational Research, 170(3), 909\u2013922.","journal-title":"European Journal of Operational Research"},{"issue":"1","key":"5941_CR26","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1007\/s11573-015-0786-0","volume":"86","author":"K Miettinen","year":"2016","unstructured":"Miettinen, K., & Ruiz, F. (2016). NAUTILUS framework: Towards trade-off-free interaction in multiobjective optimization. Journal of Business Economics, 86(1), 5\u201321.","journal-title":"Journal of Business Economics"},{"key":"5941_CR27","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1007\/978-3-540-88908-3_2","volume-title":"Multiobjective Optimization: Interactive and Evolutionary Approaches","author":"K Miettinen","year":"2008","unstructured":"Miettinen, K., Ruiz, F., & Wierzbicki, A. P. (2008). Introduction to multiobjective optimization: Interactive approaches. In J. Branke, K. Deb, K. Miettinen, & R. S\u0142owi\u0144ski (Eds.), Multiobjective Optimization: Interactive and Evolutionary Approaches (pp. 27\u201357). Springer."},{"key":"5941_CR28","doi-asserted-by":"publisher","first-page":"148277","DOI":"10.1109\/ACCESS.2021.3123825","volume":"9","author":"G Misitano","year":"2021","unstructured":"Misitano, G., Saini, B. S., Afsar, B., Shavazipour, B., & Miettinen, K. (2021). DESDEO: The modular and open source framework for interactive multiobjective optimization. IEEE Access, 9, 148277\u2013148295.","journal-title":"IEEE Access"},{"issue":"4","key":"5941_CR29","doi-asserted-by":"publisher","first-page":"880","DOI":"10.1111\/j.1540-5915.1988.tb00309.x","volume":"19","author":"R Narasimhan","year":"1988","unstructured":"Narasimhan, R., & Vickery, S. K. (1988). An experimental evaluation of articulation of preferences in multiple criterion decision-making (MCDM) methods. Decision Sciences, 19(4), 880\u2013888.","journal-title":"Decision Sciences"},{"key":"5941_CR30","doi-asserted-by":"publisher","first-page":"218","DOI":"10.1016\/j.ejor.2015.04.027","volume":"246","author":"AB Ruiz","year":"2015","unstructured":"Ruiz, A. B., Sindhya, K., Miettinen, K., Ruiz, F., & Luque, M. (2015). E-NAUTILUS: A decision support system for complex multiobjective optimization problems based on the NAUTILUS method. European Journal of Operational Research, 246, 218\u2013231.","journal-title":"European Journal of Operational Research"},{"issue":"1","key":"5941_CR31","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1007\/s10479-010-0831-x","volume":"197","author":"F Ruiz","year":"2012","unstructured":"Ruiz, F., Luque, M., & Miettinen, K. (2012). Improving the computational efficiency in a global formulation (GLIDE) for interactive multiobjective optimization. Annals of Operations Research, 197(1), 47\u201370.","journal-title":"Annals of Operations Research"},{"key":"5941_CR32","volume-title":"Multiple Criteria Optimization: Theory, Computation, and Applications","author":"RE Steuer","year":"1986","unstructured":"Steuer, R. E. (1986). Multiple Criteria Optimization: Theory, Computation, and Applications. Wiley."},{"issue":"2","key":"5941_CR33","doi-asserted-by":"publisher","first-page":"257","DOI":"10.1207\/s15516709cog1202_4","volume":"12","author":"J Sweller","year":"1988","unstructured":"Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257\u2013285.","journal-title":"Cognitive Science"},{"issue":"3","key":"5941_CR34","doi-asserted-by":"publisher","first-page":"251","DOI":"10.1023\/A:1022193728205","volume":"10","author":"J Sweller","year":"1998","unstructured":"Sweller, J., Van Merrienboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10(3), 251\u2013296.","journal-title":"Educational Psychology Review"},{"issue":"12","key":"5941_CR35","doi-asserted-by":"publisher","first-page":"1387","DOI":"10.1287\/mnsc.21.12.1387","volume":"21","author":"J Wallenius","year":"1975","unstructured":"Wallenius, J. (1975). Comparative evaluation of some interactive approaches to multicriterion optimization. Management Science, 21(12), 1387\u20131396.","journal-title":"Management Science"},{"key":"5941_CR36","doi-asserted-by":"publisher","DOI":"10.4135\/9781412983488","volume-title":"Basic Content Analysis","author":"RP Weber","year":"1990","unstructured":"Weber, R. P. (1990). Basic Content Analysis. Sage."},{"key":"5941_CR37","doi-asserted-by":"publisher","first-page":"468","DOI":"10.1007\/978-3-642-48782-8_32","volume-title":"Multiple Criteria Decision Making, Theory and Applications","author":"AP Wierzbicki","year":"1980","unstructured":"Wierzbicki, A. P. (1980). The use of reference objectives in multiobjective optimization. In G. Fandel & T. Gal (Eds.), Multiple Criteria Decision Making, Theory and Applications (pp. 468\u2013486). Springer."}],"container-title":["Annals of Operations Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10479-024-05941-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10479-024-05941-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10479-024-05941-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,29]],"date-time":"2024-07-29T14:15:56Z","timestamp":1722262556000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10479-024-05941-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,17]]},"references-count":37,"journal-issue":{"issue":"2-3","published-print":{"date-parts":[[2024,7]]}},"alternative-id":["5941"],"URL":"https:\/\/doi.org\/10.1007\/s10479-024-05941-6","relation":{},"ISSN":["0254-5330","1572-9338"],"issn-type":[{"type":"print","value":"0254-5330"},{"type":"electronic","value":"1572-9338"}],"subject":[],"published":{"date-parts":[[2024,4,17]]},"assertion":[{"value":"17 April 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 March 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 April 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no Conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}