{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,25]],"date-time":"2026-07-25T04:23:20Z","timestamp":1784953400120,"version":"3.55.0"},"reference-count":68,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2021,10,13]],"date-time":"2021-10-13T00:00:00Z","timestamp":1634083200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100004837","name":"Spanish Ministry of Science and Innovation","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004837","id-type":"DOI","asserted-by":"crossref"}]},{"name":"FCT - Foundation for Science and Technology"},{"name":"CISUC","award":["UID\/CEC\/00326\/2020"],"award-info":[{"award-number":["UID\/CEC\/00326\/2020"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Evol. Learn. Optim."],"published-print":{"date-parts":[[2021,12,31]]},"abstract":"<jats:p>\n            Experimental studies are prevalent in\n            <jats:bold>Evolutionary Computation<\/jats:bold>\n            (\n            <jats:bold>EC<\/jats:bold>\n            ), and concerns about the reproducibility and replicability of such studies have increased in recent times, reflecting similar concerns in other scientific fields. In this article, we discuss, within the context of EC, the different types of reproducibility and suggest a classification that refines the badge system of the\n            <jats:bold>Association of Computing Machinery<\/jats:bold>\n            (\n            <jats:bold>ACM<\/jats:bold>\n            ) adopted by ACM\n            <jats:bold>Transactions on Evolutionary Learning and Optimization<\/jats:bold>\n            (\n            <jats:bold>TELO<\/jats:bold>\n            ). We identify cultural and technical obstacles to reproducibility in the EC field. Finally, we provide guidelines and suggest tools that may help to overcome some of these reproducibility obstacles.\n          <\/jats:p>","DOI":"10.1145\/3466624","type":"journal-article","created":{"date-parts":[[2021,10,13]],"date-time":"2021-10-13T22:05:35Z","timestamp":1634162735000},"page":"1-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":40,"title":["Reproducibility in Evolutionary Computation"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9974-1295","authenticated-orcid":false,"given":"Manuel","family":"L\u00f3pez-ib\u00e1\u00f1ez","sequence":"first","affiliation":[{"name":"University of M\u00e1laga, M\u00e1laga, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4343-5878","authenticated-orcid":false,"given":"Juergen","family":"Branke","sequence":"additional","affiliation":[{"name":"University of Warwick, Coventry, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lu\u00eds","family":"Paquete","sequence":"additional","affiliation":[{"name":"University of Coimbra, CISUC, Department of Informatics Engineering, Coimbra, Portugal"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,10,13]]},"reference":[{"key":"e_1_3_2_2_1","volume-title":"https:\/\/aaai.org\/Conferences\/AAAI-21\/reproducibility-checklist\/","year":"2021","unstructured":"AAAI. 2021. 35th AAAI Conference on Artificial Intelligence: Reproducibility Checklist. Retrieved June 6, 2021 https:\/\/aaai.org\/Conferences\/AAAI-21\/reproducibility-checklist\/."},{"key":"e_1_3_2_3_1","volume-title":"https:\/\/www.acm.org\/publications\/policies\/artifact-review-and-badging-current","year":"2020","unstructured":"ACM. 2020. Artifact Review and Badging Version 1.1. Retrieved on August 17, 2021 from https:\/\/www.acm.org\/publications\/policies\/artifact-review-and-badging-current."},{"key":"e_1_3_2_4_1","doi-asserted-by":"publisher","DOI":"10.1038\/533452a"},{"key":"e_1_3_2_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-43505-2_56"},{"key":"e_1_3_2_6_1","doi-asserted-by":"publisher","DOI":"10.5555\/1941931"},{"key":"e_1_3_2_7_1","volume-title":"arXiv:2007.03488 [cs.NE]","author":"Bartz-Beielstein Thomas","year":"2020","unstructured":"Thomas Bartz-Beielstein, Carola Doerr, Daan van den Berg, Jakob Bossek, Sowmya Chandrasekaran, Tome Eftimov, Andreas Fischbach, Pascal Kerschke, William La Cava, Manuel L\u00f3pez-Ib\u00e1\u00f1ez, Katherine M. Malan, Jason H. Moore, Boris Naujoks, Patryk Orzechowski, Vanessa Volz, Markus Wagner, and Thomas Weise. 2020. Benchmarking in optimization: Best practice and open issues. arXiv:2007.03488 [cs.NE]. https:\/\/arxiv.org\/abs\/2007.03488."},{"key":"e_1_3_2_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33206-7_10"},{"key":"e_1_3_2_9_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco_a_00217"},{"key":"e_1_3_2_10_1","doi-asserted-by":"publisher","DOI":"10.5555\/1610387"},{"issue":"6","key":"e_1_3_2_11_1","doi-asserted-by":"crossref","first-page":"445","DOI":"10.3233\/AIC-170537","article-title":"Performance robustness of AI planners in the 2014 international planning competition","volume":"31","author":"Bocchese Andrea F.","year":"2018","unstructured":"Andrea F. Bocchese, Chris Fawcett, Mauro Vallati, Alfonso E. Gerevini, and Holger H. Hoos. 2018. Performance robustness of AI planners in the 2014 international planning competition. AI Communications 31, 6 (Dec. 2018), 445\u2013463. https:\/\/doi.org\/10.3233\/AIC-170537","journal-title":"AI Communications"},{"key":"e_1_3_2_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/0167-6377(94)90065-5"},{"key":"e_1_3_2_13_1","doi-asserted-by":"publisher","DOI":"10.1002\/9780470743386"},{"key":"e_1_3_2_14_1","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1007\/978-3-319-15934-8_13","volume-title":"Evolutionary Multi-Criterion Optimization, EMO 2015 Part I","author":"Brockhoff Dimo","year":"2015","unstructured":"Dimo Brockhoff. 2015. A bug in the multiobjective optimizer IBEA: Salutary lessons for code release and a performance re-assessment. In Evolutionary Multi-Criterion Optimization, EMO 2015 Part I, Ant\u00f3nio Gaspar-Cunha, Carlos Henggeler Antunes, and Carlos A. Coello Coello (Eds.). Lecture Notes in Computer Science, Vol. 9018. Springer, Heidelberg, Germany, 187\u2013201. https:\/\/doi.org\/10.1007\/978-3-319-15934-8_13"},{"key":"e_1_3_2_15_1","doi-asserted-by":"crossref","first-page":"1782","DOI":"10.1145\/3319619.3326899","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO Companion 2019","author":"Buzdalov Maxim","year":"2019","unstructured":"Maxim Buzdalov. 2019. Towards better estimation of statistical significance when comparing evolutionary algorithms. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO Companion 2019, Manuel L\u00f3pez-Ib\u00e1\u00f1ez, Anne Auger, and Thomas St\u00fctzle (Eds.). ACM Press, New York, NY, 1782\u20131788. https:\/\/doi.org\/10.1145\/3319619.3326899"},{"key":"e_1_3_2_16_1","volume-title":"Journal of Heuristics","author":"Campelo Felipe","year":"2020","unstructured":"Felipe Campelo and Elizabeth F. Wanner. 2020. Sample size calculations for the experimental comparison of multiple algorithms on multiple problem instances. Journal of Heuristics (2020). https:\/\/doi.org\/10.1007\/s10732-020-09454-w"},{"key":"e_1_3_2_17_1","doi-asserted-by":"crossref","unstructured":"Marco Chiarandini and Yuri Goegebeur. 2010. Mixed models for the analysis of optimization algorithms. See Bartz-Beielstein et\u00a0al. 2010 225\u2013264. https:\/\/doi.org\/10.1007\/978-3-642-02538-9","DOI":"10.1007\/978-3-642-02538-9_10"},{"key":"e_1_3_2_18_1","doi-asserted-by":"crossref","first-page":"601","DOI":"10.1190\/1.1822162","volume-title":"SEG Technical Program Expanded Abstracts 1992","author":"Claerbout Jon","year":"1992","unstructured":"Jon Claerbout and Martin Karrenbach. 1992. Electronic documents give reproducible research a new meaning. In SEG Technical Program Expanded Abstracts 1992. Society of Exploration Geophysicists, 601\u2013604. https:\/\/doi.org\/10.1190\/1.1822162"},{"key":"e_1_3_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3360311"},{"key":"e_1_3_2_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/208443"},{"key":"e_1_3_2_21_1","volume-title":"Understanding the New Statistics\u2014Effect Sizes, Confidence Intervals, and Meta-Analysis","author":"Cumming Jeff","year":"2012","unstructured":"Jeff Cumming. 2012. Understanding the New Statistics\u2014Effect Sizes, Confidence Intervals, and Meta-Analysis. Taylor & Francis."},{"key":"e_1_3_2_22_1","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-44427-7","volume-title":"Swarm Intelligence","author":"Dorigo Marco","year":"2016","unstructured":"Marco Dorigo. 2016. Swarm intelligence: A few things you need to know if you want to publish in this journal. Swarm Intelligence (Nov. 2016). Retrieved on August 17, 2021 from https:\/\/static.springer.com\/sgw\/documents\/1593723\/application\/pdf\/Additional_submission_instructions.pdf."},{"key":"e_1_3_2_23_1","volume-title":"Proceedings of the Evaluation Methods for Machine Learning Workshop at the 26th ICML","author":"Drummond Chris","year":"2009","unstructured":"Chris Drummond. 2009. Replicability is not reproducibility: Nor is it good science. In Proceedings of the Evaluation Methods for Machine Learning Workshop at the 26th ICML. Retrieved on August 17, 2021 from http:\/\/www.site.uottawa.ca\/cdrummon\/pubs\/ICMLws09.pdf."},{"key":"e_1_3_2_24_1","first-page":"141","volume-title":"Reproducibility \u2014Principles, Problems, Practices and Prospects","author":"Ehm Werner","year":"2016","unstructured":"Werner Ehm. 2016. Reproducibility from the perspective of meta-analysis. In Reproducibility \u2014Principles, Problems, Practices and Prospects, Harald Atmanspacher and Sabine Maasen (Eds.). Wiley, 141\u2013168."},{"key":"e_1_3_2_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2002.1006991"},{"key":"e_1_3_2_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11192-011-0494-7"},{"key":"e_1_3_2_27_1","first-page":"519","volume-title":"17th International Conference on Information Technology-New Generations (ITNG\u201920)","author":"Cacho Jorge Ram\u00f3n Fonseca","year":"2020","unstructured":"Jorge Ram\u00f3n Fonseca Cacho and Kazem Taghva. 2020. The state of reproducible research in computer science. In 17th International Conference on Information Technology-New Generations (ITNG\u201920), Shahram Latifi (Ed.). Springer International Publishing, 519\u2013524. https:\/\/doi.org\/10.1007\/978-3-030-43020-7_68"},{"issue":"617","key":"e_1_3_2_28_1","doi-asserted-by":"crossref","first-page":"617","DOI":"10.1007\/s10732-008-9080-4","article-title":"A study on the use of non-parametric tests for analyzing the evolutionary algorithms\u2019 behaviour: A case study on the CEC\u20192005 Special Session on Real Parameter Optimization","volume":"15","author":"Garc\u00eda Salvador","year":"2009","unstructured":"Salvador Garc\u00eda, Daniel Molina, Manuel Lozano, and Francisco Herrera. 2009. A study on the use of non-parametric tests for analyzing the evolutionary algorithms\u2019 behaviour: A case study on the CEC\u20192005 Special Session on Real Parameter Optimization. Journal of Heuristics 15, 617 (2009), 617\u2013644. https:\/\/doi.org\/10.1007\/s10732-008-9080-4","journal-title":"Journal of Heuristics"},{"key":"e_1_3_2_30_1","doi-asserted-by":"publisher","DOI":"10.1098\/rsos.171511"},{"issue":"3","key":"e_1_3_2_31_1","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1609\/aimag.v39i3.2816","article-title":"On reproducible AI: Towards reproducible research, open science, and digital scholarship in AI publications","volume":"39","author":"Gundersen Odd Erik","year":"2018","unstructured":"Odd Erik Gundersen, Yolanda Gil, and David W. Aha. 2018. On reproducible AI: Towards reproducible research, open science, and digital scholarship in AI publications. AI Magazine 39, 3 (Sept. 2018), 56\u201368. https:\/\/doi.org\/10.1609\/aimag.v39i3.2816","journal-title":"AI Magazine"},{"key":"e_1_3_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2743015"},{"key":"e_1_3_2_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09851-6"},{"key":"e_1_3_2_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02430364"},{"key":"e_1_3_2_35_1","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1007\/978-3-642-25566-3_40","volume-title":"5th International Conference on Learning and Intelligent Optimization (LION 5)","author":"Hutter Frank","year":"2011","unstructured":"Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. 2011. Sequential model-based optimization for general algorithm configuration. In 5th International Conference on Learning and Intelligent Optimization (LION 5), Carlos A. Coello Coello (Ed.). Lecture Notes in Computer Science, Vol. 6683. Springer, Heidelberg, Germany, 507\u2013523. https:\/\/doi.org\/10.1007\/978-3-642-25566-3_40"},{"issue":"1","key":"e_1_3_2_36_1","article-title":"The configurable SAT solver challenge (CSSC)","volume":"243","author":"Hutter Frank","year":"2017","unstructured":"Frank Hutter, Marius Thomas Lindauer, Adrian Balint, Sam Bayless, Holger H. Hoos, and Kevin Leyton-Brown. 2017. The configurable SAT solver challenge (CSSC). Artificial Intelligence 243, 1\u201325 (2017).","journal-title":"Artificial Intelligence"},{"key":"e_1_3_2_37_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.0020124"},{"key":"e_1_3_2_38_1","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1090\/dimacs\/059\/11","volume-title":"Data Structures, Near Neighbor Searches, and Methodology: Fifth and Sixth DIMACS Implementation Challenges","author":"Johnson David S.","year":"2002","unstructured":"David S. Johnson. 2002. A theoretician\u2019s guide to the experimental analysis of algorithms. In Data Structures, Near Neighbor Searches, and Methodology: Fifth and Sixth DIMACS Implementation Challenges, M. H. Goldwasser, David S. Johnson, and Catherine C. McGeoch (Eds.). American Mathematical Society, Providence, RI, 215\u2013250."},{"key":"e_1_3_2_39_1","volume-title":"http:\/\/www.springer.com\/journal\/10732","author":"Journal of Heuristics","year":"2015","unstructured":"Journal of Heuristics 2015. Policies on Heuristic Search Research. Retrieved June 10, 2015 from http:\/\/www.springer.com\/journal\/10732."},{"key":"e_1_3_2_40_1","doi-asserted-by":"publisher","DOI":"10.1057\/jors.2015.77"},{"key":"e_1_3_2_41_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15327957pspr0203_4"},{"key":"e_1_3_2_42_1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511612398"},{"key":"e_1_3_2_43_1","volume-title":"https:\/\/ijcai-21.org\/wp-content\/uploads\/2020\/12\/20201226-IJCAI-Reproducibility.pdf","author":"Liu Zhiyuan","year":"2021","unstructured":"Zhiyuan Liu and Jian Tang. 2021. IJCAI 2021 Reproducibility Guidelines, 35th International Joint Conference on Artificial Intelligence. Retrieved on August 17, 2021 from https:\/\/ijcai-21.org\/wp-content\/uploads\/2020\/12\/20201226-IJCAI-Reproducibility.pdf."},{"key":"e_1_3_2_44_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.orp.2016.09.002"},{"key":"e_1_3_2_45_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cosrev.2010.09.009"},{"key":"e_1_3_2_46_1","doi-asserted-by":"publisher","DOI":"10.5555\/2159557"},{"key":"e_1_3_2_47_1","volume-title":"arXiv:1807.02811","author":"Melis G\u00e1bor","year":"2017","unstructured":"G\u00e1bor Melis, Chris Dyer, and Phil Blunsom. 2017. On the state of the art of evaluation in neural language models. arXiv:1807.02811. http:\/\/arxiv.org\/abs\/1707.05589."},{"key":"e_1_3_2_48_1","volume-title":"Design and Analysis of Experiments (8th ed.)","author":"Montgomery Douglas C.","year":"2012","unstructured":"Douglas C. Montgomery. 2012. Design and Analysis of Experiments (8th ed.). John Wiley & Sons, New York, NY."},{"key":"e_1_3_2_49_1","doi-asserted-by":"publisher","DOI":"10.1162\/evco_a_00262"},{"key":"e_1_3_2_50_1","first-page":"450","volume-title":"ICGA","author":"Nagata Yuichi","year":"1997","unstructured":"Yuichi Nagata and Shigenobu Kobayashi. 1997. Edge assembly crossover: A high-power genetic algorithm for the traveling salesman problem. In ICGA, Thomas B\u00e4ck (Ed.). Morgan Kaufmann Publishers, San Francisco, CA, 450\u2013457."},{"key":"e_1_3_2_51_1","first-page":"628","volume-title":"Proceedings of the IEEE 1999 IEEE International Conference on Systems, Man, and Cybernetics","author":"Nagata Yuichi","year":"1999","unstructured":"Yuichi Nagata and Shigenobu Kobayashi. 1999. An analysis of edge assembly crossover for the traveling salesman problem. In Proceedings of the IEEE 1999 IEEE International Conference on Systems, Man, and Cybernetics (SMC\u201999), Koji Ito, Fumio Harashima, and Kazuo Tanie (Eds.). IEEE Press, 628\u2013633. https:\/\/doi.org\/10.1109\/icsmc.1999.823285"},{"key":"e_1_3_2_52_1","doi-asserted-by":"publisher","DOI":"10.1287\/ijoc.1120.0506"},{"key":"e_1_3_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/360018.360022"},{"key":"e_1_3_2_54_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.aab2374"},{"key":"e_1_3_2_55_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1708274114"},{"issue":"6251","key":"e_1_3_2_56_1","doi-asserted-by":"crossref","first-page":"aac4716","DOI":"10.1126\/science.aac4716","article-title":"Estimating the reproducibility of psychological science","volume":"349","author":"Collaboration Open Science","year":"2015","unstructured":"Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science 349, 6251 (2015), aac4716. https:\/\/doi.org\/10.1126\/science.aac4716","journal-title":"Science"},{"key":"e_1_3_2_57_1","series-title":"Operations Research\/Computer Science Interfaces","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1007\/978-0-387-71921-4_17","volume-title":"Metaheuristics: Progress in Complex Systems Optimization","author":"Paquete Lu\u00eds","year":"2007","unstructured":"Lu\u00eds Paquete, Thomas St\u00fctzle, and Manuel L\u00f3pez-Ib\u00e1\u00f1ez. 2007. Using experimental design to analyze stochastic local search algorithms for multiobjective problems. In Metaheuristics: Progress in Complex Systems Optimization, Karl F. Doerner, Michel Gendreau, Peter Greistorfer, Walter J. Gutjahr, Richard F. Hartl, and Marc Reimann (Eds.). Operations Research\/Computer Science Interfaces, Vol. 39. Springer, New York, NY, 325\u2013344. https:\/\/doi.org\/10.1007\/978-0-387-71921-4_17"},{"key":"e_1_3_2_58_1","volume-title":"https:\/\/www.cs.mcgill.ca\/ jpineau\/ReproducibilityChecklist-v2.0.pdf","author":"Pineau Joelle","year":"2020","unstructured":"Joelle Pineau and Koustuv Sinha. 2020. The Machine Learning Reproducibility Checklist (v2.0). Retrieved on August 17, 2021 from https:\/\/www.cs.mcgill.ca\/ jpineau\/ReproducibilityChecklist-v2.0.pdf."},{"key":"e_1_3_2_59_1","volume-title":"arXiv:2003.12206 [cs.LG]","author":"Pineau Joelle","year":"2020","unstructured":"Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivi\u00e8re, Alina Beygelzimer, Florence d\u2019Alch\u00e9 Buc, Emily Fox, and Hugo Larochelle. 2020. Improving reproducibility in machine learning research (a report from the NeurIPS 2019 reproducibility program). arXiv:2003.12206 [cs.LG]."},{"key":"e_1_3_2_60_1","article-title":"Reproducibility vs. Replicability: A brief history of a confused terminology","volume":"11","author":"Plesser Hans E.","year":"2018","unstructured":"Hans E. Plesser. 2018. Reproducibility vs. Replicability: A brief history of a confused terminology. Frontiers in Neuroinformatics 11 (Jan. 2018). https:\/\/doi.org\/10.3389\/fninf.2017.00076","journal-title":"Frontiers in Neuroinformatics"},{"key":"e_1_3_2_61_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2008.03.007"},{"key":"e_1_3_2_62_1","volume-title":"Psychological Science","author":"Simmons Joseph P.","year":"2011","unstructured":"Joseph P. Simmons, Leif D. Nelson, and Uri Simonsohn. 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science (2011). https:\/\/ssrn.com\/abstract=1850704."},{"key":"e_1_3_2_63_1","first-page":"1","volume-title":"Proceedings of the 2010 Congress on Evolutionary Computation","author":"Smit Selmar K.","year":"2010","unstructured":"Selmar K. Smit and Agoston E. Eiben. 2010. Beating the \u2018world champion\u2019 evolutionary algorithm via REVAC tuning. In Proceedings of the 2010 Congress on Evolutionary Computation (CEC\u201910), Hisao Ishibuchi et\u00a0al. (Eds.). IEEE Press, Piscataway, NJ, 1\u20138. https:\/\/doi.org\/10.1109\/CEC.2010.5586026"},{"issue":"1","key":"e_1_3_2_64_1","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1111\/itor.12443","article-title":"A critical analysis of the \u201cimproved Clarke and Wright savings algorithm\u201d","volume":"26","author":"S\u00f6rensen Kenneth","year":"2017","unstructured":"Kenneth S\u00f6rensen, Florian Arnold, and Daniel Palhazi Cuervo. 2017. A critical analysis of the \u201cimproved Clarke and Wright savings algorithm\u201d. International Transactions in Operational Research 26, 1 (2017), 54\u201363. https:\/\/doi.org\/10.1111\/itor.12443","journal-title":"International Transactions in Operational Research"},{"key":"e_1_3_2_65_1","volume-title":"Edge","author":"Stodden Victoria","year":"2014","unstructured":"Victoria Stodden. 2014. What scientific idea is ready for retirement? Reproducibility. Edge (2014). Retrieved on August 17, 2021 from https:\/\/www.edge.org\/annual-question\/2014\/response\/25340."},{"key":"e_1_3_2_66_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.aah6168"},{"key":"e_1_3_2_67_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1708290115"},{"key":"e_1_3_2_68_1","volume-title":"The Turing Way: A Handbook for Reproducible Data Science","author":"Community The Turing Way","year":"2019","unstructured":"The Turing Way Community, Becky Arnold, Louise Bowler, Sarah Gibson, Patricia Herterich, Rosie Higman, Anna Krystalli, Alexander Morley, Martin O\u2019Reilly, and Kirstie Whitaker. 2019. The Turing Way: A Handbook for Reproducible Data Science. Zenodo. https:\/\/doi.org\/10.5281\/zenodo.3233986"},{"issue":"2","key":"e_1_3_2_69_1","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1016\/j.ejor.2020.10.045","article-title":"Meta-analysis of metaheuristics: Quantifying the effect of adaptiveness in adaptive large neighborhood search","volume":"292","author":"Turke\u0161 Renata","year":"2021","unstructured":"Renata Turke\u0161, Kenneth S\u00f6rensen, and Lars Magnus Hvattum. 2021. Meta-analysis of metaheuristics: Quantifying the effect of adaptiveness in adaptive large neighborhood search. European Journal of Operational Research 292, 2 (2021), 423\u2013442. https:\/\/doi.org\/10.1016\/j.ejor.2020.10.045","journal-title":"European Journal of Operational Research"},{"key":"e_1_3_2_70_1","doi-asserted-by":"publisher","DOI":"10.5555\/800253.807694"}],"container-title":["ACM Transactions on Evolutionary Learning and Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3466624","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3466624","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:24:52Z","timestamp":1750195492000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3466624"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,13]]},"references-count":68,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,12,31]]}},"alternative-id":["10.1145\/3466624"],"URL":"https:\/\/doi.org\/10.1145\/3466624","relation":{},"ISSN":["2688-299X","2688-3007"],"issn-type":[{"value":"2688-299X","type":"print"},{"value":"2688-3007","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,13]]},"assertion":[{"value":"2020-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-05-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-10-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}