{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T06:43:29Z","timestamp":1782801809513,"version":"3.54.5"},"reference-count":58,"publisher":"MIT Press","issue":"1","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Reproducibility of experiments is a complex task in stochastic methods such as evolutionary algorithms or metaheuristics in general. Many works from the literature give general guidelines to favor reproducibility. However, none of them provide both a practical set of steps or software tools to help in this process. In this article, we propose a practical methodology to favor reproducibility in optimization problems tackled with stochastic methods. This methodology is divided into three main steps, where the researcher is assisted by software tools which implement state-of-the-art techniques related to this process. The methodology has been applied to study the double-row facility layout problem (DRFLP) where we propose a new algorithm able to obtain better results than the state-of-the-art methods. To this aim, we have also replicated the previous methods in order to complete the study with a new set of larger instances. All the produced artifacts related to the methodology and the study of the target problem are available in Zenodo.<\/jats:p>","DOI":"10.1162\/evco_a_00317","type":"journal-article","created":{"date-parts":[[2022,11,17]],"date-time":"2022-11-17T22:30:07Z","timestamp":1668724207000},"page":"69-104","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":10,"title":["A Practical Methodology for Reproducible Experimentation: An Application to the Double-Row Facility Layout Problem"],"prefix":"10.1162","volume":"32","author":[{"given":"Ra\u00fal","family":"Mart\u00edn-Santamar\u00eda","sequence":"first","affiliation":[{"name":"Department of Computer Science and Statistics, Universidad Rey Juan Carlos, M\u00f3stoles, 28933, Spain raul.martin@urjc.es"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sergio","family":"Cavero","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Statistics, Universidad Rey Juan Carlos, M\u00f3stoles, 28933, Spain sergio.cavero@urjc.es"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alberto","family":"Herr\u00e1n","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Statistics, Universidad Rey Juan Carlos, M\u00f3stoles, 28933, Spain alberto.herran@urjc.es"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Abraham","family":"Duarte","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Statistics, Universidad Rey Juan Carlos, M\u00f3stoles, 28933, Spain abraham.duarte@urjc.es"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"J. Manuel","family":"Colmenar","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Statistics, Universidad Rey Juan Carlos, M\u00f3stoles, 28933, Spain josemanuel.colmenar@urjc.es"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2024,3,1]]},"reference":[{"key":"2024030114125783400_B1","article-title":"Artifact review and badging, version 2.0","author":"ACM","year":"2021"},{"issue":"2","key":"2024030114125783400_B2","doi-asserted-by":"publisher","first-page":"508","DOI":"10.1016\/j.ejor.2004.12.021","article-title":"On the exact solution of a facility layout problem","volume":"173","author":"Amaral","year":"2006","journal-title":"European Journal of Operational Research"},{"issue":"12","key":"2024030114125783400_B3","doi-asserted-by":"crossref","first-page":"3325","DOI":"10.1016\/j.cor.2012.04.016","article-title":"The corridor allocation problem","volume":"39","author":"Amaral","year":"2012","journal-title":"Computers & Operations Research"},{"issue":"2","key":"2024030114125783400_B4","doi-asserted-by":"publisher","first-page":"407","DOI":"10.1007\/s11590-011-0426-8","article-title":"Optimal solutions for the double row layout problem","volume":"7","author":"Amaral","year":"2013","journal-title":"Optimization Letters"},{"issue":"1","key":"2024030114125783400_B5","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1080\/00207543.2018.1457811","article-title":"A mixed-integer programming formulation for the double row layout of machines in manufacturing systems","volume":"57","author":"Amaral","year":"2019","journal-title":"International Journal of Production Research"},{"key":"2024030114125783400_B6","first-page":"1","article-title":"A heuristic approach for the double row layout problem","author":"Amaral","year":"2020","journal-title":"Annals of Operations Research"},{"issue":"4","key":"2024030114125783400_B7","doi-asserted-by":"publisher","first-page":"1407","DOI":"10.1007\/s11590-020-01651-7","article-title":"A mixed-integer programming formulation of the double row layout problem based on a linear extension of a partial order","volume":"15","author":"Amaral","year":"2021","journal-title":"Optimization Letters"},{"issue":"4","key":"2024030114125783400_B8","doi-asserted-by":"publisher","first-page":"611","DOI":"10.1287\/ijoc.1080.0270","article-title":"Computing globally optimal solutions for single-row layout problems using semidefinite programming and cutting planes","volume":"20","author":"Anjos","year":"2008","journal-title":"INFORMS Journal on Computing"},{"key":"2024030114125783400_B9","article-title":"The Turing Way: A handbook for reproducible data science","author":"Arnold","year":"2019","journal-title":"Zenodo"},{"key":"2024030114125783400_B10","author":"Arthur","year":"2006","journal-title":"k-means++: The advantages of careful seeding"},{"issue":"1","key":"2024030114125783400_B11","article-title":"Handbook of evolutionary computation","volume":"97","author":"B\u00e4ck","year":"1997","journal-title":"Release"},{"issue":"7604","key":"2024030114125783400_B12","doi-asserted-by":"crossref","DOI":"10.1038\/533452a","article-title":"1,500 scientists lift the lid on reproducibility","volume":"533","author":"Baker","year":"2016","journal-title":"Nature"},{"key":"2024030114125783400_B13","article-title":"Benchmarking in optimization: Best practice and open issues","author":"Bartz-Beielstein","year":"2020"},{"issue":"2","key":"2024030114125783400_B14","doi-asserted-by":"publisher","first-page":"353","DOI":"10.1051\/ita:2006009","article-title":"Towards a theory of practice in metaheuristics design: A machine learning perspective","volume":"40","author":"Birattari","year":"2006","journal-title":"RAIRO-Theoretical Informatics and Applications"},{"key":"2024030114125783400_B15","doi-asserted-by":"crossref","first-page":"324","DOI":"10.1145\/3205651.3205658","article-title":"Bayesian inference for algorithm ranking analysis","author":"Calvo","year":"2018","journal-title":"Proceedings of the Genetic and Evolutionary Computation Conference Companion"},{"key":"2024030114125783400_B16","doi-asserted-by":"crossref","first-page":"1789","DOI":"10.1145\/3319619.3326888","article-title":"Bayesian performance analysis for black-box optimization benchmarking","author":"Calvo","year":"2019","journal-title":"Proceedings of the Genetic and Evolutionary Computation Conference Companion"},{"key":"2024030114125783400_B17","doi-asserted-by":"crossref","DOI":"10.1016\/j.cie.2019.106244","article-title":"A mixed integer programming model for a double row layout problem","volume":"140","author":"Chae","year":"2020","journal-title":"Computers & Industrial Engineering"},{"issue":"3","key":"2024030114125783400_B18","doi-asserted-by":"publisher","first-page":"709","DOI":"10.1080\/00207540802192126","article-title":"The double row layout problem","volume":"48","author":"Chung","year":"2010","journal-title":"International Journal of Production Research"},{"key":"2024030114125783400_B19","doi-asserted-by":"publisher","DOI":"10.15626\/MP.2018.892","article-title":"Computational reproducibility via containers in psychology","volume":"3","author":"Clyburne-Sherin","year":"2019","journal-title":"Meta-Psychology"},{"issue":"4","key":"2024030114125783400_B20","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1007\/BF02023001","article-title":"A tabu search procedure for multicommodity location\/allocation with balancing requirements","volume":"41","author":"Crainic","year":"1993","journal-title":"Annals of Operations Research"},{"issue":"2","key":"2024030114125783400_B21","doi-asserted-by":"publisher","first-page":"443","DOI":"10.1016\/j.ejor.2021.02.044","article-title":"A mixed-integer linear programming approach for the T-row and the multi-bay facility layout problem","volume":"295","author":"Dahlbeck","year":"2021","journal-title":"European Journal of Operational Research"},{"key":"2024030114125783400_B22","doi-asserted-by":"publisher","DOI":"10.1016\/j.orp.2021.100186","article-title":"Acviz: A tool for the visual analysis of the configuration of algorithms with irace","volume":"8","author":"De Souza","year":"2021","journal-title":"Operations Research Perspectives"},{"key":"2024030114125783400_B23","first-page":"582","article-title":"A critical note on experimental research methodology in EC","author":"Eiben","year":"2002","journal-title":"Proceedings of the 2002 Congress on Evolutionary Computation"},{"key":"2024030114125783400_B24","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1016\/0167-6377(89)90002-3","article-title":"A probabilistic heuristic for a computationally difficult set covering problem","volume":"8","author":"Feo","year":"1989","journal-title":"Operations Research Letters"},{"key":"2024030114125783400_B25","first-page":"37","volume-title":"An introduction to Tabu Search","author":"Gendreau","year":"2003"},{"key":"2024030114125783400_B26","author":"Gent","year":"1997","journal-title":"How not to do it"},{"key":"2024030114125783400_B27","volume-title":"Handbook of metaheuristics","author":"Glover","year":"2006"},{"issue":"3","key":"2024030114125783400_B28","doi-asserted-by":"crossref","first-page":"893","DOI":"10.1016\/j.ejor.2021.03.027","article-title":"An efficient variable neighborhood search for the space-free multi-row facility layout problem","volume":"295","author":"Herr\u00e1n","year":"2021","journal-title":"European Journal of Operational Research"},{"issue":"6","key":"2024030114125783400_B29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s42452-020-2734-z","article-title":"Exact and heuristic approaches for a new circular layout problem","volume":"2","author":"Hungerl\u00e4nder","year":"2020","journal-title":"SN Applied Sciences"},{"key":"2024030114125783400_B30","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1007\/978-3-642-25566-3_40","article-title":"Sequential model-based optimization for general algorithm configuration","author":"Hutter","year":"2011","journal-title":"International Conference on Learning and Intelligent Optimization"},{"key":"2024030114125783400_B31","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1613\/jair.2861","article-title":"Paramils: An automatic algorithm configuration framework","volume":"36","author":"Hutter","year":"2009","journal-title":"Journal of Artificial Intelligence Research"},{"issue":"7","key":"2024030114125783400_B32","doi-asserted-by":"publisher","first-page":"1129","DOI":"10.1002\/1520-6750(199510)42:7&lt;1129::AID-NAV3220420711&gt;3.0.CO;2-M","article-title":"Note: A local-search heuristic for large set-covering problems","volume":"42","author":"Jacobs","year":"1995","journal-title":"Naval Research Logistics (NRL)"},{"key":"2024030114125783400_B33","article-title":"Experimental analysis of algorithms","volume":"59","author":"Johnson","year":"2002","journal-title":"Data Structures, Near Neighbor Searches, and Methodology: Fifth and Sixth DIMACS Implementation Challenges: Papers Related to the DIMACS Challenge on Dictionaries and Priority Queues (1995-1996) and the DIMACS Challenge on Near Neighbor Searches (1998-1999)"},{"key":"2024030114125783400_B34","doi-asserted-by":"crossref","DOI":"10.1002\/0470013192.bsa501","article-title":"Principal component analysis","volume-title":"Encyclopedia of statistics in behavioral science","author":"Jolliffe","year":"2005"},{"issue":"2065","key":"2024030114125783400_B35","doi-asserted-by":"publisher","DOI":"10.1098\/rsta.2015.0202","article-title":"Principal component analysis: A review and recent developments","volume":"374","author":"Jolliffe","year":"2016","journal-title":"Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences"},{"issue":"4","key":"2024030114125783400_B36","doi-asserted-by":"publisher","first-page":"676","DOI":"10.1057\/jors.2015.77","article-title":"Good laboratory practice for optimization research","volume":"67","author":"Kendall","year":"2016","journal-title":"Journal of the Operational Research Society"},{"issue":"4","key":"2024030114125783400_B37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3466624","article-title":"Reproducibility in evolutionary computation","volume":"1","author":"L\u00f3pez-Ib\u00e1\u00f1ez","year":"2021","journal-title":"ACM Transactions on Evolutionary Learning and Optimization"},{"key":"2024030114125783400_B38","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1016\/j.orp.2016.09.002","article-title":"The irace package: Iterated racing for automatic algorithm configuration","volume":"3","author":"L\u00f3pez-Ib\u00e1\u00f1ez","year":"2016","journal-title":"Operations Research Perspectives"},{"issue":"1","key":"2024030114125783400_B39","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1016\/j.ejor.2011.04.018","article-title":"Iterated greedy for the maximum diversity problem","volume":"214","author":"Lozano","year":"2011","journal-title":"European Journal of Operational Research"},{"issue":"2","key":"2024030114125783400_B40","first-page":"467","article-title":"Two strategies based on meta-heuristic algorithms for parallel row ordering problem (prop)","volume":"10","author":"Maadi","year":"2017","journal-title":"Iranian Journal of Management Studies"},{"issue":"14","key":"2024030114125783400_B41","first-page":"281","article-title":"Some methods for classification and analysis of multivariate observations","volume":"1","author":"MacQueen","year":"1967","journal-title":"Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability"},{"key":"2024030114125783400_B42","first-page":"177","article-title":"A memetic approach for the traveling salesman problem implementation of a computational ecology for combinatorial optimization on message-passing systems","volume":"1","author":"Moscato","year":"1992","journal-title":"Parallel Computing and Transputer Applications"},{"issue":"2","key":"2024030114125783400_B43","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1007\/s10462-010-9165-y","article-title":"A review of instance selection methods","volume":"34","author":"Olvera-L\u00f3pez","year":"2010","journal-title":"Artificial Intelligence Review"},{"key":"2024030114125783400_B44","article-title":"Improving reproducibility in machine learning research: A report from the neurIPS 2019 reproducibility program","volume":"22","author":"Pineau","year":"2021","journal-title":"Journal of Machine Learning Research"},{"key":"2024030114125783400_B45","doi-asserted-by":"publisher","DOI":"10.3389\/fninf.2017.00076","article-title":"Reproducibility vs. replicability: A brief history of a confused terminology","volume":"11","author":"Plesser","year":"2018","journal-title":"Frontiers in Neuroinformatics"},{"key":"2024030114125783400_B46","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2021.107659","article-title":"Solving the regenerator location problem with an iterated greedy approach","volume":"111","author":"Quintana","year":"2021","journal-title":"Applied Soft Computing"},{"key":"2024030114125783400_B47","article-title":"Empirical standards for software engineering research","author":"Ralph","year":"2020"},{"key":"2024030114125783400_B48","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.knosys.2016.05.030","article-title":"Grasp with path relinking for the single row facility layout problem","volume":"106","author":"Rubio-S\u00e1nchez","year":"2016","journal-title":"Knowledge-Based Systems"},{"key":"2024030114125783400_B49","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1016\/j.omega.2018.03.004","article-title":"Iterated greedy methods for the distributed permutation flowshop scheduling problem","volume":"83","author":"Ruiz","year":"2019","journal-title":"Omega"},{"issue":"3","key":"2024030114125783400_B50","doi-asserted-by":"publisher","first-page":"2033","DOI":"10.1016\/j.ejor.2005.12.009","article-title":"A simple and effective iterated greedy algorithm for the permutation flowshop scheduling problem","volume":"177","author":"Ruiz","year":"2007","journal-title":"European Journal of Operational Research"},{"issue":"4","key":"2024030114125783400_B51","first-page":"193","article-title":"Scatter search algorithm for single row layout problem in FMS","volume":"3","author":"Satheesh Kumar","year":"2008","journal-title":"Advances in Production Engineering & Management"},{"key":"2024030114125783400_B52","first-page":"166","article-title":"Finding a \u201ckneedle\u201d in a haystack: Detecting knee points in system behavior","author":"Satopaa","year":"2011","journal-title":"31st International Conference on Distributed Computing Systems Workshops"},{"issue":"1","key":"2024030114125783400_B53","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1007\/s11590-018-1263-9","article-title":"An improved mixed-integer programming model for the double row layout of facilities","volume":"13","author":"Secchin","year":"2019","journal-title":"Optimization Letters"},{"issue":"5","key":"2024030114125783400_B54","doi-asserted-by":"publisher","first-page":"812","DOI":"10.1287\/opre.17.5.812","article-title":"One-dimensional space allocation: An ordering algorithm","volume":"17","author":"Simmons","year":"1969","journal-title":"Operations Research"},{"key":"2024030114125783400_B55","doi-asserted-by":"publisher","DOI":"10.1201\/b16868","volume-title":"Implementing reproducible research","author":"Stodden","year":"2014"},{"key":"2024030114125783400_B56","doi-asserted-by":"crossref","first-page":"541","DOI":"10.1007\/978-3-319-91086-4_17","article-title":"Automated design of metaheuristic algorithms","volume-title":"Handbook of metaheuristics","author":"St\u00fctzle","year":"2019"},{"issue":"6","key":"2024030114125783400_B57","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3459664","article-title":"Machine learning into metaheuristics: A survey and taxonomy","volume":"54","author":"Talbi","year":"2021","journal-title":"ACM Computing Surveys (CSUR)"},{"issue":"1","key":"2024030114125783400_B58","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/sdata.2016.18","article-title":"The fair guiding principles for scientific data management and stewardship","volume":"3","author":"Wilkinson","year":"2016","journal-title":"Scientific Data"}],"container-title":["Evolutionary Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/evco\/article-pdf\/32\/1\/69\/2329006\/evco_a_00317.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/evco\/article-pdf\/32\/1\/69\/2329006\/evco_a_00317.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,1]],"date-time":"2024-03-01T14:13:42Z","timestamp":1709302422000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/evco\/article\/32\/1\/69\/113683\/A-Practical-Methodology-for-Reproducible"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":58,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,3,1]]},"published-print":{"date-parts":[[2024,3,1]]}},"URL":"https:\/\/doi.org\/10.1162\/evco_a_00317","relation":{},"ISSN":["1530-9304"],"issn-type":[{"value":"1530-9304","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}