{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,19]],"date-time":"2026-02-19T03:36:20Z","timestamp":1771472180000,"version":"3.50.1"},"reference-count":38,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2024,7,30]],"date-time":"2024-07-30T00:00:00Z","timestamp":1722297600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,7,30]],"date-time":"2024-07-30T00:00:00Z","timestamp":1722297600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Government of Spain","award":["PID2022-139551NB-I00"],"award-info":[{"award-number":["PID2022-139551NB-I00"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2024,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Building past snapshots of a software project has shown to be of interest both for researchers and practitioners. However, little attention has been devoted specifically to tests available in those past snapshots, which are fundamental for the maintenance of old versions still in production. The aim of this study is to determine to which extent tests of past snapshots can be executed successfully, which would mean these past snapshots are still testable. Given a software project, we build all its past snapshots from source code, including tests, and then run the tests. When tests do not result in success, we also record the reasons, allowing us to determine factors that make tests fail. We run this method on a total of 86 Java projects. On average, for 52.53% of the project snapshots on which tests can be built, all tests pass. However, on average, 94.14% of tests pass in previous snapshots when we take into account the percentage of tests passing in the snapshots used for building those tests. In real software projects, successfully running tests in past snapshots is not something that we can take for granted: we have found that in a large proportion of the projects we studied this does not happen frequently. We have found that the building from source code is the main limitation when running tests on past snapshots. However, we have found some projects for which tests run successfully in a very large fraction of past snapshots, which allows us to identify good practices. We also provide a framework and metrics to quantify testability (the extent to which we are able to run tests of a snapshot with a success result) of past snapshots from several points of view, which simplifies new analyses on this matter, and could help to measure how any project performs in this respect.<\/jats:p>","DOI":"10.1007\/s10664-024-10530-z","type":"journal-article","created":{"date-parts":[[2024,7,30]],"date-time":"2024-07-30T08:02:11Z","timestamp":1722326531000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Testing the past: can we still run tests in past snapshots for Java projects?"],"prefix":"10.1007","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8138-9702","authenticated-orcid":false,"given":"Michel","family":"Maes-Bermejo","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Micael","family":"Gallego","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Francisco","family":"Gort\u00e1zar","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gregorio","family":"Robles","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jesus M.","family":"Gonzalez-Barahona","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,7,30]]},"reference":[{"issue":"12\u20131990","key":"10530_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/IEEESTD.1990.101064","volume":"610","author":"Ieee standard glossary of software engineering terminology","year":"1990","unstructured":"Ieee standard glossary of software engineering terminology (1990) IEEE Std 610(12\u20131990):1\u201384. https:\/\/doi.org\/10.1109\/IEEESTD.1990.101064","journal-title":"IEEE Std"},{"key":"10530_CR2","unstructured":"Iso standard 9126-1. https:\/\/www.iso.org\/standard\/22749.html (Accessed May 20, 2024) (2001)"},{"key":"10530_CR3","unstructured":"Iso standard 25010. https:\/\/www.iso.org\/standard\/35733.html (Accessed May 20, 2024) (2011)"},{"key":"10530_CR4","unstructured":"Software testability. https:\/\/www.geeksforgeeks.org\/software-testability\/ (2023) (accessed September 8, 2023)"},{"key":"10530_CR5","doi-asserted-by":"crossref","unstructured":"Ayewah N, Pugh W, Morgenthaler JD, Penix J, Zhou Y (2007) Using findbugs on production software. In: Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion, pp. 805\u2013806. ACM","DOI":"10.1145\/1297846.1297897"},{"key":"10530_CR6","unstructured":"Bartel A (2019) Dos software security: Is there anyone left to patch a 25-year old vulnerability? https:\/\/www.abartel.net\/static\/p\/hacklu2019_build_engine.pdf"},{"key":"10530_CR7","doi-asserted-by":"publisher","unstructured":"Behnamghader P, Meemeng P, Fostiropoulos I, Huang D, Srisopha K, Boehm B (2018) A scalable and efficient approach for compiling and analyzing commit history. In: Proceedings of the 12th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM \u201918. ACM, Association for Computing Machinery, New York, NY, USA https:\/\/doi.org\/10.1145\/3239235.3239237","DOI":"10.1145\/3239235.3239237"},{"key":"10530_CR8","doi-asserted-by":"crossref","unstructured":"Beller M, Gousios G, & Zaidman A (2017) Oops, my tests broke the build: An explorative analysis of travis ci with github. In: 2017 IEEE\/ACM 14th International Conference on Mining Software Repositories (MSR), pp. 356\u2013367. IEEE","DOI":"10.1109\/MSR.2017.62"},{"issue":"9","key":"10530_CR9","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1145\/182987.184077","volume":"37","author":"RV Binder","year":"1994","unstructured":"Binder RV (1994) Design for testability in object-oriented systems. Commun ACM 37(9):87\u2013101","journal-title":"Commun ACM"},{"issue":"9","key":"10530_CR10","doi-asserted-by":"publisher","first-page":"1219","DOI":"10.1016\/j.jss.2006.02.036","volume":"79","author":"M Bruntink","year":"2006","unstructured":"Bruntink M, van Deursen A (2006) An empirical study into class testability. J Syst Softw 79(9):1219\u20131232","journal-title":"J Syst Softw"},{"key":"10530_CR11","doi-asserted-by":"crossref","unstructured":"Bruntink M, Van\u00a0Deursen A (2004) Predicting class testability using object-oriented metrics. In: Source Code Analysis and Manipulation, Fourth IEEE International Workshop on, pp. 136\u2013145. IEEE","DOI":"10.1109\/SCAM.2004.16"},{"key":"10530_CR12","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1016\/j.infsof.2018.12.003","volume":"108","author":"V Garousi","year":"2019","unstructured":"Garousi V, Felderer M, K\u0131l\u0131\u00e7aslan FN (2019) A survey on software testability. Inf Softw Technol 108:35\u201364","journal-title":"Inf Softw Technol"},{"key":"10530_CR13","doi-asserted-by":"publisher","unstructured":"He, J., Min, S., Ogudu, K., Shoga, M., Polak, A., Fostiropoulos, I., Boehm, B.W., Behnamghader, P (2020) The characteristics and impact of uncompilable code changes on software quality evolution. In: 20th IEEE International Conference on Software Quality, Reliability and Security, QRS 2020, Macau, China, December 11-14, 2020, pp. 418\u2013429. IEEE, IEEE https:\/\/doi.org\/10.1109\/QRS51102.2020.00061","DOI":"10.1109\/QRS51102.2020.00061"},{"key":"10530_CR14","doi-asserted-by":"publisher","unstructured":"Just R, Jalali D, Ernst MD (2014) Defects4j: A database of existing faults to enable controlled testing studies for java programs. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, p. 437-440. ACM, Association for Computing Machinery, New York, NY, USA https:\/\/doi.org\/10.1145\/2610384.2628055","DOI":"10.1145\/2610384.2628055"},{"key":"10530_CR15","doi-asserted-by":"publisher","unstructured":"Karampatsis RM, Sutton C (2020) How often do single-statement bugs occur? the manysstubs4j dataset. In: Proceedings of the 17th International Conference on Mining Software Repositories, MSR \u201920, p. 573-577. ACM, Association for Computing Machinery, New York, NY, USA https:\/\/doi.org\/10.1145\/3379597.3387491","DOI":"10.1145\/3379597.3387491"},{"key":"10530_CR16","doi-asserted-by":"publisher","unstructured":"Kovalenko V, Palomba F, Bacchelli A (2018) Mining file histories: should we consider branches? In: M.\u00a0Huchard, C.\u00a0K\u00e4stner, G.\u00a0Fraser (eds.) Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018, pp. 202\u2013213. ACM, ACM.https:\/\/doi.org\/10.1145\/3238147.3238169","DOI":"10.1145\/3238147.3238169"},{"key":"10530_CR17","doi-asserted-by":"publisher","unstructured":"Lou Y, Chen J, Zhang L, Hao D, Zhang L (2019) History-driven build failure fixing: How far are we? In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019, p. 43-54. ACM, Association for Computing Machinery, New York, NY, USA https:\/\/doi.org\/10.1145\/3293882.3330578","DOI":"10.1145\/3293882.3330578"},{"key":"10530_CR18","doi-asserted-by":"publisher","unstructured":"Macho C, McIntosh S, Pinzger M (2018) Automatically repairing dependency-related build breakage. In: R.\u00a0Oliveto, M.D. Penta, D.C. Shepherd (eds.) 25th International Conference on Software Analysis, Evolution and Reengineering, SANER 2018, Campobasso, Italy, March 20-23, 2018, pp. 106\u2013117. IEEE, IEEE Computer Society https:\/\/doi.org\/10.1109\/SANER.2018.8330201","DOI":"10.1109\/SANER.2018.8330201"},{"issue":"3","key":"10530_CR19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10664-022-10117-6","volume":"27","author":"M Maes-Bermejo","year":"2022","unstructured":"Maes-Bermejo M, Gallego M, Gort\u00e1zar F, Robles G, Gonzalez-Barahona JM (2022) Revisiting the building of past snapshots-a replication and reproduction study. Empir Softw Eng 27(3):1\u201326","journal-title":"Empir Softw Eng"},{"issue":"3","key":"10530_CR20","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1007\/s10664-024-10479-z","volume":"29","author":"M Maes-Bermejo","year":"2024","unstructured":"Maes-Bermejo M, Serebrenik A, Gallego M, Gort\u00e1zar F, Robles G, Gonz\u00e1lez Barahona JM (2024) Hunting bugs: Towards an automated approach to identifying which change caused a bug through regression testing. Empir Softw Eng 29(3):66","journal-title":"Empir Softw Eng"},{"key":"10530_CR21","doi-asserted-by":"crossref","unstructured":"Mannan UA, Ahmed I, Almurshed RAM, Dig D, Jensen C (2016) Understanding code smells in android applications. In: 2016 IEEE\/ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft), pp. 225\u2013236. IEEE","DOI":"10.1145\/2897073.2897094"},{"key":"10530_CR22","doi-asserted-by":"publisher","unstructured":"Querel L, Rigby PC (2021) Warning-introducing commits vs bug-introducing commits: A tool, statistical models, and a preliminary user study. In: 29th IEEE\/ACM International Conference on Program Comprehension, ICPC 2021, Madrid, Spain, May 20-21, 2021, pp. 433\u2013443. IEEE, IEEE https:\/\/doi.org\/10.1109\/ICPC52881.2021.00051","DOI":"10.1109\/ICPC52881.2021.00051"},{"key":"10530_CR23","doi-asserted-by":"publisher","unstructured":"Rausch T, Hummer W, Leitner P, Schulte S (2017) An empirical analysis of build failures in the continuous integration workflows of java-based open-source software. In: Proceedings of the 14th International Conference on Mining Software Repositories, MSR \u201917, pp. 345\u2013355. IEEE, IEEE Press, Piscataway, NJ, USA https:\/\/doi.org\/10.1109\/MSR.2017.54","DOI":"10.1109\/MSR.2017.54"},{"key":"10530_CR24","unstructured":"Rodr\u00edguez-P\u00e9rez G, Nagappan M, Robles G (2020a) Watch out for extrinsic bugs! a case study of their impact in just-in-time bug prediction models on the openstack project. IEEE Transactions on Software Engineering"},{"key":"10530_CR25","doi-asserted-by":"crossref","unstructured":"Rodr\u00edguez-P\u00e9rez G, Robles G, Serebrenik A, Zaidman A, Germ\u00e1n DM, Gonzalez-Barahona JM (2020b) How bugs are born: a model to identify how bugs are introduced in software components. Empirical Software Engineering pp. 1\u201347","DOI":"10.1007\/s10664-019-09781-y"},{"key":"10530_CR26","doi-asserted-by":"crossref","unstructured":"Rodr\u00edguez-P\u00e9rez G, Zaidman A, Serebrenik A, Robles G, Gonz\u00e1lez-Barahona JM (2018) What if a bug has a different origin? making sense of bugs without an explicit bug introducing change. In: Proceedings of the 12th ACM\/IEEE international symposium on empirical software engineering and measurement, pp. 1\u20134. ACM","DOI":"10.1145\/3239235.3267436"},{"key":"10530_CR27","doi-asserted-by":"crossref","unstructured":"Rosen C, Grawi B, Shihab E (2015) Commit guru: analytics and risk prediction of software commits. In: Proceedings of the 2015 10th joint meeting on foundations of software engineering, pp. 966\u2013969. ACM","DOI":"10.1145\/2786805.2803183"},{"key":"10530_CR28","doi-asserted-by":"publisher","unstructured":"Santos RE, Bener A, Baldassarre MT, VC\u00a0Magalh\u00e3es C, Correia-Neto JS, da\u00a0Silva FQ (2019) Mind the gap: Are practitioners and researchers in software testing speaking the same language? In: 2019 IEEE\/ACM Joint 7th International Workshop on Conducting Empirical Studies in Industry (CESI) and 6th International Workshop on Software Engineering Research and Industrial Practice (SER &IP), pp. 10\u201317 https:\/\/doi.org\/10.1109\/CESSER-IP.2019.00010","DOI":"10.1109\/CESSER-IP.2019.00010"},{"key":"10530_CR29","doi-asserted-by":"crossref","unstructured":"Staats M, Whalen MW, Heimdahl MP (2011) Better testing through oracle selection (nier track). In: Proceedings of the 33rd International Conference on Software Engineering, pp. 892\u2013895","DOI":"10.1145\/1985793.1985936"},{"issue":"3","key":"10530_CR30","doi-asserted-by":"publisher","first-page":"86","DOI":"10.3390\/data5030086","volume":"5","author":"M Sul\u00edr","year":"2020","unstructured":"Sul\u00edr M, Ba\u010d\u00edkov\u00e1 M, Madeja M, Chodarev S, Juh\u00e1r J (2020) Large-scale dataset of local java software build results. Data 5(3):86","journal-title":"Data"},{"key":"10530_CR31","doi-asserted-by":"publisher","unstructured":"Sul\u00edr M, Porub\u00e4n J (2016) A quantitative study of java software buildability. In: Proceedings of the 7th International Workshop on Evaluation and Usability of Programming Languages and Tools, PLATEAU 2016, pp. 17\u201325. ACM, ACM, New York, NY, USA https:\/\/doi.org\/10.1145\/3001878.3001882","DOI":"10.1145\/3001878.3001882"},{"key":"10530_CR32","doi-asserted-by":"crossref","unstructured":"Terragni V, Salza P, Pezz\u00e8 M (2020) Measuring software testability modulo test quality. In: Proceedings of the 28th International Conference on Program Comprehension, pp. 241\u2013251. ACM","DOI":"10.1145\/3387904.3389273"},{"key":"10530_CR33","unstructured":"Tian Y (2017) Mining software repositories for automatic software bug management from bug triaging to patch backporting. Ph.D. thesis, Singapore Management University, Singapore"},{"issue":"6","key":"10530_CR34","doi-asserted-by":"publisher","first-page":"5137","DOI":"10.1007\/s10664-020-09880-1","volume":"25","author":"A Trautsch","year":"2020","unstructured":"Trautsch A, Herbold S, Grabowski J (2020) A longitudinal study of static analysis warning evolution and the effects of pmd on software quality in apache open source projects. Empir Softw Eng 25(6):5137\u20135192","journal-title":"Empir Softw Eng"},{"issue":"4","key":"10530_CR35","first-page":"e1838","volume":"29","author":"M Tufano","year":"2017","unstructured":"Tufano M, Palomba F, Bavota G, Di Penta M, Oliveto R, De Lucia A, Poshyvanyk D (2017) There and back again: Can you compile that snapshot? J Softw: Evol Process 29(4):e1838","journal-title":"J Softw: Evol Process"},{"key":"10530_CR36","doi-asserted-by":"crossref","unstructured":"Whalen M, Gay G, You D, Heimdahl MP, Staats M (2013) Observable modified condition\/decision coverage. In: 2013 35th International Conference on Software Engineering (ICSE), pp. 102\u2013111. IEEE","DOI":"10.1109\/ICSE.2013.6606556"},{"key":"10530_CR37","doi-asserted-by":"crossref","unstructured":"Yamashita K, McIntosh S, Kamei Y, Hassan AE, Ubayashi N (2015) Revisiting the applicability of the pareto principle to core development teams in open source software projects. In: Proceedings of the 14th International Workshop on Principles of Software Evolution, pp. 46\u201355. ACM","DOI":"10.1145\/2804360.2804366"},{"key":"10530_CR38","doi-asserted-by":"publisher","unstructured":"Zhu Q, Zaidman A, Panichella A (2021) How to kill them all: An exploratory study on the impact of code observability on mutation testing. J Syst Softw 173:110864 https:\/\/doi.org\/10.1016\/j.jss.2020.110864https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0164121220302545","DOI":"10.1016\/j.jss.2020.110864"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-024-10530-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-024-10530-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-024-10530-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,25]],"date-time":"2024-11-25T01:54:13Z","timestamp":1732499653000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-024-10530-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,30]]},"references-count":38,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,9]]}},"alternative-id":["10530"],"URL":"https:\/\/doi.org\/10.1007\/s10664-024-10530-z","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,30]]},"assertion":[{"value":"9 July 2024","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 July 2024","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"This research has not involved human participants and\/or animals.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declared that they have no conflict of interest","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}}],"article-number":"127"}}