{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T16:26:56Z","timestamp":1783096016787,"version":"3.54.6"},"reference-count":49,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2020,3,4]],"date-time":"2020-03-04T00:00:00Z","timestamp":1583280000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61872182"],"award-info":[{"award-number":["61872182"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100005416","name":"Norges Forskningsr\u00e5d","doi-asserted-by":"crossref","award":["286898\/F20"],"award-info":[{"award-number":["286898\/F20"]}],"id":[{"id":"10.13039\/501100005416","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100002241","name":"Japan Science and Technology Agency","doi-asserted-by":"crossref","award":["JPMJER1603"],"award-info":[{"award-number":["JPMJER1603"]}],"id":[{"id":"10.13039\/501100002241","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2020,4,30]]},"abstract":"<jats:p>Search-Based Software Engineering (SBSE) researchers who apply multi-objective search algorithms (MOSAs) often assess the quality of solutions produced by MOSAs with one or more quality indicators (QIs). However, SBSE lacks evidence providing insights on commonly used QIs, especially about agreements among them and their relations with SBSE problems and applied MOSAs. Such evidence about QIs agreements is essential to understand relationships among QIs, identify redundant QIs, and consequently devise guidelines for SBSE researchers to select appropriate QIs for their specific contexts. To this end, we conducted an extensive empirical evaluation to provide insights on commonly used QIs in the context of SBSE, by studying agreements among QIs with and without considering differences of SBSE problems and MOSAs. In addition, by defining a systematic process based on three common ways of comparing MOSAs in SBSE, we present additional observations that were automatically produced based on the results of our empirical evaluation. These observations can be used by SBSE researchers to gain a better understanding of the commonly used QIs in SBSE, in particular, regarding their agreements. Finally, based on the results, we also provide a set of guidelines for SBSE researchers to select appropriate QIs for their particular context.<\/jats:p>","DOI":"10.1145\/3375636","type":"journal-article","created":{"date-parts":[[2020,3,4]],"date-time":"2020-03-04T12:33:55Z","timestamp":1583325235000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":34,"title":["Quality Indicators in Search-based Software Engineering"],"prefix":"10.1145","volume":"29","author":[{"given":"Shaukat","family":"Ali","sequence":"first","affiliation":[{"name":"Simula Research Laboratory, Fornebu, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6253-4062","authenticated-orcid":false,"given":"Paolo","family":"Arcaini","sequence":"additional","affiliation":[{"name":"National Institute of Informatics, Tokyo, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dipesh","family":"Pradhan","sequence":"additional","affiliation":[{"name":"Simula Research Laboratory, Fornebu, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8090-6627","authenticated-orcid":false,"given":"Safdar Aqeel","family":"Safdar","sequence":"additional","affiliation":[{"name":"Simula Research Laboratory, Fornebu, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3262-5577","authenticated-orcid":false,"given":"Tao","family":"Yue","sequence":"additional","affiliation":[{"name":"Nanjing University of Aeronautics and Astronautics and Simula Research Laboratory, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,3,4]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2008.12.005"},{"key":"e_1_2_1_2_1","volume-title":"Categorical Data Analysis","author":"Agresti Alan"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2009.52"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1985793.1985795"},{"key":"e_1_2_1_5_1","volume-title":"Silvia Regina Vergilio, and Aurora Pozo.","author":"Guez Assun\u00e7\u00e3o Wesley Klewerton","year":"2013"},{"key":"e_1_2_1_6_1","volume-title":"Threats to validity in search-based software engineering empirical studies. RelaTe-DIA 5 (01","author":"Barros M\u00e1rcio","year":"2011"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33666-9_50"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-22183-0_32"},{"key":"e_1_2_1_10_1","volume-title":"Multi-Objective Optimization Using Evolutionary Algorithms","author":"Deb Kalyanmoy"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/4235.996017"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.advengsoft.2011.05.014"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-87700-4_66"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/SSBSE.2009.21"},{"key":"e_1_2_1_15_1","first-page":"1","article-title":"Multiobjective optimization and multiple constraint handling with evolutionary algorithms. I. A unified formulation","volume":"28","author":"Fonseca Carlos M.","year":"1998","journal-title":"IEEE Trans. Syst. Man Cybern. Part A Syst. Humans"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/38.1-2.141"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2017.03.012"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0950-5849(01)00189-6"},{"key":"e_1_2_1_19_1","volume-title":"Article 11 (Dec.","author":"Harman Mark","year":"2012"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1162\/106365600568167"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183399.3183405"},{"key":"e_1_2_1_23_1","volume-title":"Article 26 (March","author":"Li Miqing","year":"2019"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 2014 IEEE Congress on Evolutionary Computation (CEC). 387--396","author":"Lopez-Herrejon R. E."},{"key":"e_1_2_1_25_1","volume-title":"C (May","author":"Lopez-Herrejon Roberto E.","year":"2015"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2013.10.015"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2011.100"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCDM.2009.4938830"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1002\/int.20358"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2908812.2908850"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-47443-4_11"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2018.00015"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2803055"},{"key":"e_1_2_1_34_1","volume-title":"C (June","author":"Ravber Miha","year":"2017"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3071178.3071261"},{"key":"e_1_2_1_36_1","volume-title":"Handbook of Parametric and Nonparametric Statistical Procedures (5 Ed.)","author":"Sheskin David J."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3092703.3092709"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSMC.1995.537993"},{"key":"e_1_2_1_39_1","first-page":"2","article-title":"Multiobjective evolutionary algorithms: Analyzing the state-of-the-art","volume":"8","author":"Van Veldhuizen David A.","year":"2000","journal-title":"Evol. Comput."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2014.02.154"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2014.08.024"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884880"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2015.7381833"},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the IEEE Congress on Evolutionary Computation (CEC\u201908)","author":"Wang Zai","year":"2008"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2576768.2598309"},{"key":"e_1_2_1_46_1","volume-title":"Retrieved","author":"Zhang Yuanyuan","year":"2017"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2006.1688406"},{"key":"e_1_2_1_48_1","unstructured":"E. Zitzler M. Laumanns and L. Thiele. 2001. SPEA2: Improving the strength Pareto evolutionary algorithm for multiobjective optimization. In Evolutionary Methods for Design Optimization and Control with Applications to Industrial Problems (19--21 Sept. 2001) K. C. Giannakoglou D. T. Tsahalis J. P\u00e9riaux K. D. Papailiou and T. Fogarty (Eds.). International Center for Numerical Methods in Engineering Athens Greece 95--100.  E. Zitzler M. Laumanns and L. Thiele. 2001. SPEA2: Improving the strength Pareto evolutionary algorithm for multiobjective optimization. In Evolutionary Methods for Design Optimization and Control with Applications to Industrial Problems (19--21 Sept. 2001) K. C. Giannakoglou D. T. Tsahalis J. P\u00e9riaux K. D. Papailiou and T. Fogarty (Eds.). International Center for Numerical Methods in Engineering Athens Greece 95--100."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/4235.797969"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3375636","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3375636","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3375636","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:38:15Z","timestamp":1750199895000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3375636"}},"subtitle":["An Empirical Evaluation"],"short-title":[],"issued":{"date-parts":[[2020,3,4]]},"references-count":49,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,4,30]]}},"alternative-id":["10.1145\/3375636"],"URL":"https:\/\/doi.org\/10.1145\/3375636","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3,4]]},"assertion":[{"value":"2019-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-12-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-03-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}