{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T23:10:28Z","timestamp":1789081828292,"version":"build-2803163510"},"reference-count":61,"publisher":"Public Library of Science (PLoS)","issue":"5","license":[{"start":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T00:00:00Z","timestamp":1779840000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"<jats:p>Modern software development relies on automated build systems that compile and test code whenever developers make changes. Predicting whether these builds will succeed or fail before execution could save computational resources and developer time. However, many machine learning models for build prediction suffer from temporal data leakage, a methodological flaw where the model inadvertently uses information that would only be available after the build completes, producing artificially inflated accuracy that fails in real-world deployment. This study develops a three-type taxonomy to systematically identify and prevent such leakage: (1) Direct Outcome Encoding (using the build result itself as a feature), (2) Execution-Dependent Metrics (information generated during build execution), and (3) Future Information Leakage (using data from chronologically later builds). Applying this taxonomy reveals that prior studies reporting 95\u201399% accuracy likely used contaminated features, while realistic accuracy is substantially lower. The methodology is validated on 175,706 builds from two open-source CI\/CD platforms spanning 10 years: TravisTorrent (100,000 builds, 2013\u20132017) and GHALogs (75,706 workflows, 2023). Removing leaky features reduces accuracy by 15.07 percentage points on TravisTorrent (97.8% to 82.73%) but only 0.48 points on GHALogs (83.77% to 83.30%), revealing that modern GitHub Actions\u2019 tight integration with repositories enables accurate prediction from static project metadata alone. Using only legitimately available pre-build features, Random Forest classifiers achieve 82.73% (TravisTorrent) and 83.30% (GHALogs) accuracy, sufficient for practical deployment. Surprisingly, project maturity and build history prove more predictive than code complexity metrics, suggesting organizational factors outweigh code quality. The models generalize across programming languages (Java, Ruby, Python, JavaScript) with minimal performance variation. Open-source tools for detecting temporal leakage in any software prediction task are provided.<\/jats:p>","DOI":"10.1371\/journal.pone.0340167","type":"journal-article","created":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T17:38:57Z","timestamp":1779903537000},"page":"e0340167","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":2,"title":["A taxonomy for detecting and preventing temporal data leakage in machine learning-based build prediction: A dual-platform empirical validation"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-2638-1592","authenticated-orcid":true,"given":"Lalit Narayan","family":"Mishra","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amit","family":"Rangari","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sandesh","family":"Nagrare","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Saroj Kumar","family":"Nayak","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,5,27]]},"reference":[{"issue":"2","key":"pone.0340167.ref001","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1109\/MS.2025.3525768","article-title":"DevOps 2.0","volume":"42","author":"C Ebert","year":"2025","journal-title":"IEEE Softw"},{"key":"pone.0340167.ref002","doi-asserted-by":"crossref","unstructured":"Vassallo C, Schermann G, Zampetti F, Romano D, Leitner P, Zaidman A, et al. A Tale of CI Build Failures: An Open Source and a Financial Organization Perspective. In: Proceedings of the 33rd IEEE International Conference on Software Maintenance and Evolution (ICSME). Shanghai, China; 2017. p. 183\u201393.","DOI":"10.1109\/ICSME.2017.67"},{"key":"pone.0340167.ref003","doi-asserted-by":"crossref","unstructured":"Rausch T, Hummer W, Leitner P, Schulte S. An Empirical Analysis of Build Failures in the Continuous Integration Workflows of Java-Based Open-Source Software. In: Proceedings of the 14th International Conference on Mining Software Repositories (MSR). Buenos Aires, Argentina; 2017. p. 345\u201355.","DOI":"10.1109\/MSR.2017.54"},{"issue":"5","key":"pone.0340167.ref004","first-page":"426","article-title":"Usage, Costs, and Benefits of Continuous Integration in Open-Source Projects","volume":"43","author":"M Hilton","year":"2016","journal-title":"IEEE Trans Softw Eng"},{"issue":"1","key":"pone.0340167.ref005","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10515-021-00319-5","article-title":"Improving the prediction of continuous integration build failures using deep learning","volume":"29","author":"I Saidani","year":"2022","journal-title":"Autom Softw Eng"},{"key":"pone.0340167.ref006","doi-asserted-by":"crossref","unstructured":"Beller M, Gousios G, Zaidman A. TravisTorrent: Synthesizing Travis CI and GitHub for Full-Stack Research on Continuous Integration. In: Proceedings of the 14th International Conference on Mining Software Repositories (MSR). Buenos Aires, Argentina; 2017. p. 447\u201350.","DOI":"10.1109\/MSR.2017.24"},{"issue":"4","key":"pone.0340167.ref007","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2382577.2382579","article-title":"Leakage in data mining","volume":"6","author":"S Kaufman","year":"2012","journal-title":"ACM Trans Knowl Discov Data"},{"issue":"9","key":"pone.0340167.ref008","doi-asserted-by":"crossref","first-page":"100804","DOI":"10.1016\/j.patter.2023.100804","article-title":"Leakage and the reproducibility crisis in machine-learning-based science","volume":"4","author":"S Kapoor","year":"2023","journal-title":"Patterns (NY)"},{"issue":"3","key":"pone.0340167.ref009","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1049\/iet-sen.2017.0148","article-title":"Progress on approaches to software defect prediction","volume":"12","author":"Z Li","year":"2018","journal-title":"IET Software"},{"issue":"10","key":"pone.0340167.ref010","doi-asserted-by":"crossref","first-page":"1054","DOI":"10.1109\/TSE.2018.2822270","article-title":"How do static and dynamic test case prioritization techniques perform on modern software systems? An extensive study on GitHub projects","volume":"46","author":"Q Luo","year":"2019","journal-title":"IEEE Trans Softw Eng"},{"key":"pone.0340167.ref011","doi-asserted-by":"crossref","unstructured":"Bacchelli A, Bird C. Expectations, Outcomes, and Challenges of Modern Code Review. In: Proceedings of the 2013 International Conference on Software Engineering. 2013. p. 712\u201321.","DOI":"10.1109\/ICSE.2013.6606617"},{"key":"pone.0340167.ref012","unstructured":"Decan A, Mens T. GHALogs: A Large-Scale Multi-Language Dataset of GitHub Actions Workflow Runs. In: Proceedings of the 22nd International Conference on Mining Software Repositories (MSR). IEEE; 2025. Dataset available at Zenodo: https:\/\/doi.org\/10.5281\/zenodo.10154920"},{"issue":"1","key":"pone.0340167.ref013","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2379776.2379787","article-title":"Search-based software engineering","volume":"45","author":"M Harman","year":"2012","journal-title":"ACM Comput Surv"},{"issue":"3","key":"pone.0340167.ref014","doi-asserted-by":"crossref","first-page":"1630","DOI":"10.1109\/TSE.2022.3173346","article-title":"Machine\/Deep Learning for Software Engineering: A Systematic Literature Review","volume":"49","author":"S Wang","year":"2023","journal-title":"IEEE Trans Softw Eng"},{"key":"pone.0340167.ref015","doi-asserted-by":"crossref","first-page":"107187","DOI":"10.1016\/j.infsof.2023.107187","article-title":"Fault-insertion and fault-fixing behavioural patterns in Apache Software Foundation Projects","volume":"158","author":"M Ortu","year":"2023","journal-title":"Inf Softw Technol"},{"key":"pone.0340167.ref016","doi-asserted-by":"crossref","unstructured":"Grillmeyer D, Hadry M, Lesch V, Borst V, Leppich R, Bauer A, et al. Quantifying Data Leakage in Failure Prediction Tasks. In: Proceedings of the 16th ACM\/SPEC International Conference on Performance Engineering (ICPE). London, United Kingdom: ACM; 2025. p. 185\u201396.","DOI":"10.1145\/3676151.3719368"},{"issue":"3","key":"pone.0340167.ref017","doi-asserted-by":"crossref","first-page":"3615","DOI":"10.1007\/s10586-023-04170-z","article-title":"Software defect prediction using a bidirectional LSTM network combined with oversampling techniques","volume":"27","author":"NAA Khleel","year":"2023","journal-title":"Cluster Comput"},{"issue":"4","key":"pone.0340167.ref018","doi-asserted-by":"crossref","first-page":"1748","DOI":"10.1016\/j.ijforecast.2021.03.012","article-title":"Temporal Fusion Transformers for interpretable multi-horizon time series forecasting","volume":"37","author":"B Lim","year":"2021","journal-title":"Int J Forecast"},{"issue":"3","key":"pone.0340167.ref019","first-page":"456","article-title":"Cross\u2010Project Defect Prediction Using Transfer Learning with Long Short\u2010Term Memory Networks","volume":"18","author":"H Tao","year":"2024","journal-title":"IET Software"},{"key":"pone.0340167.ref020","article-title":"Enhancing software defect prediction: a framework with improved feature selection and ensemble machine learning","volume":"10","author":"M Ali","year":"2024","journal-title":"PeerJ Comput Sci"},{"issue":"4","key":"pone.0340167.ref021","article-title":"Ensemble learning: A survey","volume":"8","author":"O Sagi","year":"2018","journal-title":"WIREs Data Min Knowl Discov"},{"key":"pone.0340167.ref022","doi-asserted-by":"crossref","unstructured":"Ghotra B, McIntosh S, Hassan AE. Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models. In: Proceedings of the 37th IEEE International Conference on Software Engineering (ICSE). Florence, Italy; 2015. p. 789\u2013800. doi:10.1109\/ICSE.2015.91","DOI":"10.1109\/ICSE.2015.91"},{"issue":"4","key":"pone.0340167.ref023","first-page":"456","article-title":"Stock Price Time Series Data Forecasting Using the Light Gradient Boosting Machine (LightGBM) Model","volume":"7","author":"AD Hartanto","year":"2023","journal-title":"Int J Inf Visual"},{"key":"pone.0340167.ref024","doi-asserted-by":"crossref","first-page":"106622","DOI":"10.1016\/j.knosys.2020.106622","article-title":"AutoML: A survey of the state-of-the-art","volume":"212","author":"X He","year":"2021","journal-title":"Knowl-Based Syst"},{"issue":"12","key":"pone.0340167.ref025","doi-asserted-by":"crossref","DOI":"10.1109\/TSE.2024.3462251","article-title":"The Effects of Computational Resources on Flaky Tests","volume":"50","author":"D Silva","year":"2024","journal-title":"IEEE Trans Softw Eng"},{"issue":"2","key":"pone.0340167.ref026","doi-asserted-by":"crossref","first-page":"1095","DOI":"10.1007\/s10664-019-09785-8","article-title":"An empirical characterization of bad practices in continuous integration","volume":"25","author":"F Zampetti","year":"2020","journal-title":"Empir Softw Eng"},{"key":"pone.0340167.ref027","doi-asserted-by":"crossref","first-page":"996","DOI":"10.1145\/3643771","article-title":"RavenBuild: Context, Relevance, and Dependency Aware Build Outcome Prediction","volume":"1","author":"G Sun","year":"2024","journal-title":"Proc ACM Softw Eng"},{"issue":"2","key":"pone.0340167.ref028","first-page":"1031","article-title":"Random forest model for software build time prediction on CI\/CD pipeline","volume":"33","author":"WH Seow","year":"2025","journal-title":"Pertanika J Sci Technol"},{"key":"pone.0340167.ref029","doi-asserted-by":"crossref","unstructured":"Bouzenia I, Pradel M. Resource Usage and Optimization Opportunities in Workflows of GitHub Actions. In: Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering (ICSE). Lisbon, Portugal: ACM; 2024. p. 25:1\u201325:12.","DOI":"10.1145\/3597503.3623303"},{"key":"pone.0340167.ref030","doi-asserted-by":"crossref","unstructured":"Bernardo JH, Da Costa DA, de Medeiros SQ, Kulesza U. How do Machine Learning Projects use Continuous Integration Practices? An Empirical Study on GitHub Actions. In: Proceedings of the 21st International Conference on Mining Software Repositories (MSR). Lisbon, Portugal: ACM; 2024. p. 665\u201376.","DOI":"10.1145\/3643991.3644915"},{"key":"pone.0340167.ref031","doi-asserted-by":"crossref","unstructured":"Cardoen G, Mens T, Decan A. A dataset of GitHub Actions workflow histories. In: Proceedings of the 21st International Conference on Mining Software Repositories (MSR) - Data Showcase Track. Lisbon, Portugal: ACM; 2024. p. 677\u201381. Dataset available at Zenodo https:\/\/doi.org\/10.5281\/zenodo.10259013","DOI":"10.1145\/3643991.3644867"},{"key":"pone.0340167.ref032","doi-asserted-by":"crossref","unstructured":"Saito S. Understanding Key Business Processes for Business Process Outsourcing Transition. In: 2019 ACM\/IEEE 14th International Conference on Global Software Engineering (ICGSE), 2019. p. 35\u20139.","DOI":"10.1109\/ICGSE.2019.00021"},{"issue":"4","key":"pone.0340167.ref033","doi-asserted-by":"crossref","first-page":"308","DOI":"10.1109\/TSE.1976.233837","article-title":"A Complexity Measure","volume":"2","author":"TJ McCabe","year":"1976","journal-title":"IEEE Trans Softw Eng"},{"key":"pone.0340167.ref034","doi-asserted-by":"crossref","unstructured":"Hassan AE. Predicting faults using the complexity of code changes. In: 2009 IEEE 31st International Conference on Software Engineering (ICSE). Vancouver, Canada; 2009. p. 78\u201388.","DOI":"10.1109\/ICSE.2009.5070510"},{"key":"pone.0340167.ref035","doi-asserted-by":"crossref","unstructured":"Nagappan N, Ball T, Zeller A. Mining metrics to predict component failures. In: Proceedings of the 28th international conference on Software engineering (ICSE). 2006. p. 452\u201361.","DOI":"10.1145\/1134285.1134349"},{"issue":"5","key":"pone.0340167.ref036","first-page":"3874","article-title":"Code and commit metrics of developer productivity: a study on team leaders perceptions","volume":"25","author":"E Oliveira","year":"2020","journal-title":"Empir Softw Eng"},{"key":"pone.0340167.ref037","volume-title":"Continuous delivery: reliable software releases through build, test, and deployment automation","author":"J Humble","year":"2010"},{"key":"pone.0340167.ref038","volume-title":"State of Tech Talent Report: Survey-based Insights into the Current State of Technical Talent Acquisition, Retention, and Management Globally","author":"A Lawson","year":"2024"},{"issue":"1","key":"pone.0340167.ref039","first-page":"6756","article-title":"Evaluation metrics and statistical tests for machine learning","volume":"14","author":"O Rainio","year":"2024","journal-title":"Sci Rep"},{"issue":"3","key":"pone.0340167.ref040","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0118432","article-title":"The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets","volume":"10","author":"T Saito","year":"2015","journal-title":"PLoS One"},{"issue":"1","key":"pone.0340167.ref041","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1186\/s12859-023-05156-9","article-title":"Evaluation of a decided sample size in machine learning applications","volume":"24","author":"D Rajput","year":"2023","journal-title":"BMC Bioinform"},{"key":"pone.0340167.ref042","doi-asserted-by":"crossref","first-page":"1658","DOI":"10.1007\/978-3-642-04898-2_616","article-title":"Wilcoxon-Signed-Rank Test","volume-title":"International Encyclopedia of Statistical Science","author":"D Rey","year":"2011"},{"key":"pone.0340167.ref043","doi-asserted-by":"crossref","DOI":"10.1214\/09-SS054","article-title":"A survey of cross-validation procedures for model selection","volume":"4","author":"S Arlot","year":"2010","journal-title":"Statist Surv"},{"issue":"8","key":"pone.0340167.ref044","doi-asserted-by":"crossref","first-page":"721","DOI":"10.1109\/TSE.2002.1027796","article-title":"Preliminary Guidelines for Empirical Research in Software Engineering","volume":"28","author":"BA Kitchenham","year":"2002","journal-title":"IEEE Trans Softw Eng"},{"key":"pone.0340167.ref045","doi-asserted-by":"crossref","unstructured":"Hassan AE, Xie T. Software intelligence. In: Proceedings of the FSE\/SDP workshop on Future of software engineering research. 2010. p. 161\u20136.","DOI":"10.1145\/1882362.1882397"},{"key":"pone.0340167.ref046","doi-asserted-by":"crossref","unstructured":"Gousios G, Pinzger M, Deursen Av. An exploratory study of the pull-based software development model. In: Proceedings of the 36th International Conference on Software Engineering. 2014. p. 345\u201355.","DOI":"10.1145\/2568225.2568260"},{"key":"pone.0340167.ref047","unstructured":"Wu H, Xu J, Wang J, Long M. Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). vol. 34. 2021. p. 22419\u201330."},{"key":"pone.0340167.ref048","volume-title":"Sampling Techniques. Classic reference on statistical sampling theory and sample size determination","author":"WG Cochran","year":"1977","edition":"3"},{"key":"pone.0340167.ref049","doi-asserted-by":"crossref","unstructured":"Feurer M, Hutter F. Hyperparameter Optimization. Autom Mach Learn. 2019. p. 3\u201333. https:\/\/doi.org\/10.1007\/978-3-030-05318-5_1","DOI":"10.1007\/978-3-030-05318-5_1"},{"issue":"7604","key":"pone.0340167.ref050","doi-asserted-by":"crossref","first-page":"452","DOI":"10.1038\/533452a","article-title":"1,500 scientists lift the lid on reproducibility","volume":"533","author":"M Baker","year":"2016","journal-title":"Nature"},{"issue":"2","key":"pone.0340167.ref051","doi-asserted-by":"crossref","first-page":"175","DOI":"10.3758\/BF03193146","article-title":"G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences","volume":"39","author":"F Faul","year":"2007","journal-title":"Behav Res Methods"},{"key":"pone.0340167.ref052","doi-asserted-by":"crossref","unstructured":"Santana FA, Cordeiro AFR, Oliveira E. Dublin Core for Recording Metadata of Experiments in Software Engineering: A Survey. arXiv preprint arXiv:230316989. 2023. Under review.","DOI":"10.5753\/eres.2023.237790"},{"key":"pone.0340167.ref053","doi-asserted-by":"crossref","unstructured":"Zhao G, Georgiou S, Hassan S, Zou Y, Truong D, Corbin T. Enhancing Performance Bug Prediction Using Performance Code Metrics. In: Proceedings of the 21st International Conference on Mining Software Repositories (MSR). Lisbon, Portugal; 2024. p. 50\u201362.","DOI":"10.1145\/3643991.3644920"},{"key":"pone.0340167.ref054","volume-title":"A review of software change impact analysis. Technical Report","author":"S Lehnert","year":"2011"},{"issue":"6","key":"pone.0340167.ref055","doi-asserted-by":"crossref","first-page":"757","DOI":"10.1109\/TSE.2012.70","article-title":"A large-scale empirical study of just-in-time quality assurance","volume":"39","author":"Y Kamei","year":"2013","journal-title":"IIEEE Trans Software Eng"},{"issue":"1","key":"pone.0340167.ref056","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1109\/TSE.2007.256943","article-title":"A Systematic Review of Software Development Cost Estimation Studies","volume":"33","author":"M Jorgensen","year":"2007","journal-title":"IIEEE Trans Software Eng"},{"key":"pone.0340167.ref057","doi-asserted-by":"crossref","unstructured":"Qi X, Chen J, Deng L. CPP\u00b3: Hierarchical Cross-Platform Power\/Performance Prediction Using Transfer Learning. In: Proceedings of the International Conference on Algorithms and Architectures for Parallel Processing (ICA3PP). vol. 13777 of Lecture Notes in Computer Science. Springer; 2023. p. 123\u201337.","DOI":"10.1007\/978-3-031-22677-9_7"},{"key":"pone.0340167.ref058","doi-asserted-by":"crossref","unstructured":"Li J, Ahmed I. Commit message matters: investigating impact and evolution of commit message quality. In: Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering (ICSE). Melbourne, Australia; 2023. p. 806\u201317.","DOI":"10.1109\/ICSE48619.2023.00076"},{"key":"pone.0340167.ref059","unstructured":"Das A, Kong W, Leach A, Mathur S, Sen R, Yu R. A decoder-only foundation model for time-series forecasting. In: Proceedings of the International Conference on Machine Learning (ICML). Vienna, Austria; 2024. p. 567\u201389. https:\/\/doi.org\/10.48550\/arXiv.2310.10688"},{"key":"pone.0340167.ref060","doi-asserted-by":"crossref","unstructured":"Feng Z, Guo D, Tang D, Duan N, Feng X, Gong M, et al. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In: Findings of the Association for Computational Linguistics: EMNLP 2020. 2020. p. 1536\u201347.","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"pone.0340167.ref061","doi-asserted-by":"crossref","first-page":"106775","DOI":"10.1016\/j.knosys.2021.106775","article-title":"A survey on federated learning","volume":"216","author":"C Zhang","year":"2021","journal-title":"Knowl-Based Syst"}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0340167","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T17:39:13Z","timestamp":1779903553000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0340167"}},"subtitle":[],"editor":[{"given":"Elochukwu","family":"Ukwandu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,5,27]]},"references-count":61,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2026,5,27]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0340167","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,27]]}}}