{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,19]],"date-time":"2025-09-19T13:15:42Z","timestamp":1758287742705,"version":"3.44.0"},"reference-count":20,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2025,5,19]],"date-time":"2025-05-19T00:00:00Z","timestamp":1747612800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,5,19]],"date-time":"2025-05-19T00:00:00Z","timestamp":1747612800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Johannes Kepler University Linz"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J Softw Tools Technol Transfer"],"published-print":{"date-parts":[[2025,8]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>We present Simsala, an easy-to-install and easy-to-use collection of scripts supporting benchmarking on clusters of compute nodes. While designed with applications for benchmarking solving technologies like SAT and extensions, our solution can be easily transferred to many other application scenarios as well. In this work, we discuss the design objectives behind Simsala, provide some implementation details, and illustrate with case studies how Simsala has been applied in the past for extensive evaluations.<\/jats:p>","DOI":"10.1007\/s10009-025-00811-3","type":"journal-article","created":{"date-parts":[[2025,5,19]],"date-time":"2025-05-19T07:18:17Z","timestamp":1747639097000},"page":"403-409","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Reproducible and hackable software benchmarking with(out) compute clusters"],"prefix":"10.1007","volume":"27","author":[{"given":"Maximilian","family":"Heisinger","sequence":"first","affiliation":[]},{"given":"Martina","family":"Seidl","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2025,5,19]]},"reference":[{"issue":"1","key":"811_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10009-017-0469-y","volume":"21","author":"D. Beyer","year":"2019","unstructured":"Beyer, D., L\u00f6we, S., Wendler, P.: Reliable benchmarking: requirements and solutions. Int. J. Softw. Tools Technol. Transf. 21(1), 1\u201329 (2019). https:\/\/doi.org\/10.1007\/s10009-017-0469-y","journal-title":"Int. J. Softw. Tools Technol. Transf."},{"key":"811_CR2","unstructured":"Biere, A., Jussila, T.: RunLim. http:\/\/fmv.jku.at\/runlim\/ (2022)"},{"key":"811_CR3","volume-title":"Proc. of the 23rd Int. Conf. on Automated Deduction","author":"A. Biere","year":"2011","unstructured":"Biere, A., Lonsing, F., Seidl, M.: Blocked clause elimination for QBF. In: Proc. of the 23rd Int. Conf. on Automated Deduction (2011)"},{"key":"811_CR4","unstructured":"Dominik, C., et\u00a0al.: Org Mode. https:\/\/orgmode.org (2022)"},{"key":"811_CR5","doi-asserted-by":"publisher","unstructured":"Heisinger, M., Seidl, M.: Simsala, Artifact for Paper \u201cReproducible and Hackable Software Benchmarking with(out) Compute Clusters\u201d (2023). https:\/\/doi.org\/10.5281\/zenodo.7794224","DOI":"10.5281\/zenodo.7794224"},{"key":"811_CR6","volume-title":"Proc. of the 16th Int. Conf. on Intelligent Computer Mathematics","author":"S. Heisinger","year":"2023","unstructured":"Heisinger, S., Seidl, M.: True crafted formula families for benchmarking quantified satisfiability solvers. In: Proc. of the 16th Int. Conf. on Intelligent Computer Mathematics (2023)"},{"key":"811_CR7","volume-title":"Proc. of the 23rd Conf. on Theory and Applications of Satisfiability Testing (SAT)","author":"M. Heisinger","year":"2020","unstructured":"Heisinger, M., Fleury, M., Biere, A.: Distributed cube and conquer with paracooba. In: Proc. of the 23rd Conf. on Theory and Applications of Satisfiability Testing (SAT) (2020)"},{"key":"811_CR8","doi-asserted-by":"publisher","unstructured":"Heisinger, M., Seidl, M., Biere, A.: Artifact for Paper ParaQooba: a Fast and Flexible Framework for Parallel and Distributed QBF Solving (2022). https:\/\/doi.org\/10.5281\/zenodo.7554207","DOI":"10.5281\/zenodo.7554207"},{"key":"811_CR9","volume-title":"Proc. of the Workshop on Practical Aspects of Automated Reasoning (FLoC\/IJCAR)","author":"M. Heisinger","year":"2022","unstructured":"Heisinger, M., Seidl, M., Biere, A.: QuAPI: adding assumptions to non-assuming SAT & QBF solvers. In: Proc. of the Workshop on Practical Aspects of Automated Reasoning (FLoC\/IJCAR) (2022)."},{"key":"811_CR10","volume-title":"Proc. of the 29th Conf. on Tools and Algorithms for the Construction and Analysis of Systems (TACAS)","author":"M. Heisinger","year":"2023","unstructured":"Heisinger, M., Seidl, M., Biere, A.: Paraqooba: a fast and flexible framework for parallel and distributed QBF solving. In: Proc. of the 29th Conf. on Tools and Algorithms for the Construction and Analysis of Systems (TACAS) (2023)"},{"key":"811_CR11","unstructured":"Hipp, R.D., et\u00a0al.: SQLite. https:\/\/www.sqlite.org\/index.html (2022)"},{"issue":"3","key":"811_CR12","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1109\/MCSE.2007.55","volume":"9","author":"J.D. Hunter","year":"2007","unstructured":"Hunter, J.D., Matplotlib: a 2d graphics environment. Comput. Sci. Eng. 9(3), 90\u201395 (2007). https:\/\/doi.org\/10.1109\/MCSE.2007.55","journal-title":"Comput. Sci. Eng."},{"key":"811_CR13","volume-title":"DB Browser for SQLite","author":"M. Kleusberg","year":"2022","unstructured":"Kleusberg, M., et al.: DB Browser for SQLite (2022). https:\/\/sqlitebrowser.org"},{"key":"811_CR14","first-page":"87","volume-title":"Positioning and Power in Academic Publishing: Players, Agents and Agendas","author":"T. Kluyver","year":"2016","unstructured":"Kluyver, T., Ragan-Kelley, B., P\u00e9rez, F., Granger, B., Bussonnier, M., Frederic, J., Kelley, K., Hamrick, J., Grout, J., Corlay, S., Ivanov, P., Avila, D., Abdalla, S., Willing, C., development team, J.: Jupyter notebooks - a publishing format for reproducible computational workflows. In: Positioning and Power in Academic Publishing: Players, Agents and Agendas, pp.\u00a087\u201390 (2016)"},{"key":"811_CR15","volume-title":"Proc. of the 26rd Conf. on Theory and Applications of Satisfiability Testing (SAT)","author":"A. Plank","year":"2023","unstructured":"Plank, A., Seidl, M.: Qmusext: a minimal (un)satisfiable core extractor for quantified Boolean formulas. In: Proc. of the 26rd Conf. on Theory and Applications of Satisfiability Testing (SAT) (2023)"},{"key":"811_CR16","volume-title":"29th Int. Conf. on Principles and Practice of Constraint Programming, CP","author":"A. Plank","year":"2023","unstructured":"Plank, A., M\u00f6hle, S., Seidl, M.: Enumerative level-2 solution counting for quantified Boolean formulas. In: 29th Int. Conf. on Principles and Practice of Constraint Programming, CP (2023)"},{"key":"811_CR17","doi-asserted-by":"publisher","first-page":"367","DOI":"10.1007\/978-3-319-08587-6_28","volume-title":"Proc. of 7th Int. Conf. on Automated Reasoning (IJCAR 2014)","author":"A. Stump","year":"2014","unstructured":"Stump, A., Sutcliffe, G., Tinelli, C.: Starexec: a cross-community infrastructure for logic solving. In: Proc. of 7th Int. Conf. on Automated Reasoning (IJCAR 2014), pp.\u00a0367\u2013373 (2014). https:\/\/doi.org\/10.1007\/978-3-319-08587-6_28"},{"issue":"1","key":"811_CR18","first-page":"42","volume":"36","author":"O. Tange","year":"2011","unstructured":"Tange, O.: GNU parallel - the command-line power tool. ;login: The USENIX Magazine 36(1), 42\u201347 (2011). http:\/\/www.gnu.org\/s\/parallel","journal-title":";login: The USENIX Magazine"},{"key":"811_CR19","unstructured":"Williams, T., Kelley, C.: many others: Gnuplot 5.4: an interactive plotting program http:\/\/gnuplot.sourceforge.net\/ (2022)"},{"key":"811_CR20","volume-title":"JSSPP","author":"A.B. Yoo","year":"2003","unstructured":"Yoo, A.B., Jette, M.A., Grondona, M.: Slurm: simple Linux utility for resource management. In: JSSPP (2003)"}],"container-title":["International Journal on Software Tools for Technology Transfer"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10009-025-00811-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10009-025-00811-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10009-025-00811-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,18]],"date-time":"2025-09-18T08:57:45Z","timestamp":1758185865000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10009-025-00811-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,19]]},"references-count":20,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,8]]}},"alternative-id":["811"],"URL":"https:\/\/doi.org\/10.1007\/s10009-025-00811-3","relation":{},"ISSN":["1433-2779","1433-2787"],"issn-type":[{"type":"print","value":"1433-2779"},{"type":"electronic","value":"1433-2787"}],"subject":[],"published":{"date-parts":[[2025,5,19]]},"assertion":[{"value":"29 April 2025","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 May 2025","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}