{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T13:16:49Z","timestamp":1775135809635,"version":"3.50.1"},"reference-count":65,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2020,4,20]],"date-time":"2020-04-20T00:00:00Z","timestamp":1587340800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61502205"],"award-info":[{"award-number":["61502205"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61872167"],"award-info":[{"award-number":["61872167"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["U1836116"],"award-info":[{"award-number":["U1836116"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002858","name":"China Postdoctoral Science Foundation","doi-asserted-by":"publisher","award":["2019T120396"],"award-info":[{"award-number":["2019T120396"]}],"id":[{"id":"10.13039\/501100002858","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Senior Personnel Scientific Research Foundation of Jiangsu University","award":["14JDG039"],"award-info":[{"award-number":["14JDG039"]}]},{"name":"Young Backbone Teacher Cultivation Project of Jiangsu University"},{"name":"Postgraduate Research & Practice Innovation Program of Jiangsu Province","award":["KYCX19_1614"],"award-info":[{"award-number":["KYCX19_1614"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,5,19]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Combinatorial interaction testing (CIT) aims at constructing a covering array (CA) of all value combinations at a specific interaction strength, to detect faults that are caused by the interaction of parameters. CIT has been widely used in different applications, with many algorithms and tools having been proposed to support CA construction. To date, however, there appears to have been no studies comparing different CA constructors when only some of the CA test cases are executed. In this paper, we present an investigation of five popular CA constructors: ACTS, Jenny, PICT, CASA and TCA. We conducted empirical studies examining the five programs, focusing on interaction coverage and fault detection. The experimental results show that when there is no preference or special justification for using other CA constructors, then Jenny is recommended\u2014because it achieves better interaction coverage and fault detection than the other four constructors in many cases. Our results also show that when using ACTS or CASA, their CAs must be prioritized before testing. The main reason for this is that these CAs can result in considerable interaction coverage or fault detection capabilities when executing a large number of test cases; however, they may also produce the lowest rates of fault detection and interaction coverage.<\/jats:p>","DOI":"10.1093\/comjnl\/bxaa020","type":"journal-article","created":{"date-parts":[[2020,2,11]],"date-time":"2020-02-11T12:08:38Z","timestamp":1581422918000},"page":"762-788","source":"Crossref","is-referenced-by-count":5,"title":["Covering Array Constructors: An Experimental Analysis of Their Interaction Coverage and Fault Detection"],"prefix":"10.1093","volume":"64","author":[{"given":"Rubing","family":"Huang","sequence":"first","affiliation":[{"name":"School of Computer Science and Communication Engineering, and Jiangsu Key Laboratory of Security Technology for Industrial Cyberspace, Jiangsu University, Zhenjiang, Jiangsu 212013, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Haibo","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang 202013, Jiangsu, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yunan","family":"Zhou","sequence":"additional","affiliation":[{"name":"School of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang 202013, Jiangsu, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tsong","family":"Yueh Chen","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Software Engineering, Swinburne University of Technology, Hawthorn 3122, VIC, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dave","family":"Towey","sequence":"additional","affiliation":[{"name":"School of Computer Science, University of Nottingham Ningbo China, Ningbo 315100, Zhejiang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Man","family":"Fai Lau","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Software Engineering, Swinburne University of Technology, Hawthorn 3122, VIC, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sebastian","family":"Ng","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Software Engineering, Swinburne University of Technology, Hawthorn 3122, VIC, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Robert","family":"Merkel","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology, Monash University, Clayton 3168, VIC, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jinfu","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang 202013, Jiangsu, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2020,4,20]]},"reference":[{"key":"2021052810194807400_ref1","author":"CA Constructor Tools"},{"key":"2021052810194807400_ref2","author":"GNU FTP Server"},{"key":"2021052810194807400_ref3","volume-title":"Advanced Combinatorial Testing System.","author":"ACTS"},{"key":"2021052810194807400_ref4","volume-title":"Covering Arrays by Simulated Annealing.","author":"CASA"},{"key":"2021052810194807400_ref5","author":"cloc: Count Lines of Code"},{"key":"2021052810194807400_ref6","author":"jenny: A freely available CA constructor"},{"key":"2021052810194807400_ref7","article-title":"Pairwise Independent Combinatorial Testing","author":"PICT"},{"key":"2021052810194807400_ref8","author":"R: The R project for statistical computing"},{"key":"2021052810194807400_ref9","author":"TCA: Two-model Covering Array"},{"key":"2021052810194807400_ref10","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1016\/j.infsof.2015.05.005","article-title":"Achievement of minimized combinatorial test suite for configuration-aware software functional testing using the cuckoo search algorithm","volume":"66","author":"Ahmed","year":"2015","journal-title":"Inf. Softw. Technol."},{"key":"2021052810194807400_ref11","doi-asserted-by":"crossref","first-page":"25706","DOI":"10.1109\/ACCESS.2017.2771562","article-title":"Constrained interaction testing: a systematic literature study","volume":"5","author":"Ahmed","year":"2017","journal-title":"IEEE Access"},{"key":"2021052810194807400_ref12","doi-asserted-by":"crossref","first-page":"1330","DOI":"10.1016\/j.asoc.2011.11.029","article-title":"Application of particle swarm optimization to uniform and variable strength covering array construction","volume":"12","author":"Ahmed","year":"2012","journal-title":"Appl. Soft Comput."},{"key":"2021052810194807400_ref13","doi-asserted-by":"crossref","first-page":"553","DOI":"10.1016\/j.infsof.2012.01.002","article-title":"Design and implementation of a harmony-search-based variable-strength t-way testing strategy with constraints support","volume":"54","author":"Alsewari","year":"2012","journal-title":"Inf. Softw. Technol."},{"key":"2021052810194807400_ref14","doi-asserted-by":"crossref","first-page":"608","DOI":"10.1109\/TSE.2006.83","article-title":"Using mutation analysis for assessing and comparing testing coverage criteria","volume":"32","author":"Andrews","year":"2006","journal-title":"IEEE Trans. Softw. Eng."},{"key":"2021052810194807400_ref15","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1002\/stvr.1486","article-title":"A hitchhiker\u2019s guide to statistical tests for assessing randomized algorithms in software engineering","volume":"24","author":"Arcuri","year":"2014","journal-title":"Softw. Test., Verif. Rel."},{"key":"2021052810194807400_ref16","article-title":"A Framework of Greedy Methods for Constructing Interaction Test Suites","volume-title":"Proc. 27th Int. Conf. Software Engineering (ICSE\u201905), pp. 146\u2013155, St.Louis, Missouri, USA, 15 May","author":"Bryce","year":"2005"},{"key":"2021052810194807400_ref17","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1002\/stvr.365","article-title":"The density algorithm for pairwise interaction testing","volume":"17","author":"Bryce","year":"2007","journal-title":"Softw. Test. Verif. Rel."},{"key":"2021052810194807400_ref18","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1002\/stvr.393","article-title":"A density-based greedy algorithm for higher strength covering arrays","volume":"19","author":"Bryce","year":"2009","journal-title":"Softw. Test. Verif. Rel."},{"key":"2021052810194807400_ref19","doi-asserted-by":"crossref","DOI":"10.1145\/1985793.1985931","article-title":"Tuple Density: A New Metric for Combinatorial Test Suites (NIER Track)","volume-title":"Proc. 33rd Int. Conf. Software Engineering (ICSE\u201911), pp. 876\u2013879, Waikiki, Honolulu, HI, USA, 21 May","author":"Chen","year":"2011"},{"key":"2021052810194807400_ref20","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.jss.2009.02.022","article-title":"Adaptive random testing: the art of test case diversity","volume":"83","author":"Chen","year":"2010","journal-title":"J. Syst. Softw."},{"key":"2021052810194807400_ref21","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1109\/APSEC.2009.18","article-title":"Variable strength interaction testing with an ant colony system approach","volume-title":"Proc. 16th Asia-Pacific Software Engineering Conference (APSEC\u201909)","author":"Chen","year":"2009"},{"key":"2021052810194807400_ref22","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1109\/32.605761","article-title":"The AETG system: an approach to testing based on combinatorial design","volume":"23","author":"Cohen","year":"1997","journal-title":"IEEE Trans. Softw. Eng."},{"key":"2021052810194807400_ref23","first-page":"38","article-title":"Constructing Test Suites for Interaction Testing","volume-title":"Proc. 25th Int. Conf. Software Engineering (ICSE\u201903)","author":"Cohen","year":"2003"},{"key":"2021052810194807400_ref24","first-page":"419","article-title":"Pairwise Testing in Real World: Practical Extensions to Test Case Generators","volume-title":"Proc. 24th Pacific Northwest Software Quality Conference (PNSQC\u201906)","author":"Czerwonka","year":"2006"},{"key":"2021052810194807400_ref25","doi-asserted-by":"crossref","first-page":"234","DOI":"10.1080\/00401706.1998.10485524","article-title":"Factor-covering designs for testing software","volume":"40","author":"Dalal","year":"1998","journal-title":"Technometrics"},{"key":"2021052810194807400_ref26","doi-asserted-by":"crossref","first-page":"733","DOI":"10.1109\/TSE.2006.92","article-title":"On the use of mutation faults in empirical assessments of test case prioritization techniques","volume":"32","author":"Do","year":"2006","journal-title":"IEEE Trans. Softw. Eng."},{"key":"2021052810194807400_ref27","author":"Garvin","year":"2009"},{"key":"2021052810194807400_ref28","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1007\/s10664-010-9135-7","article-title":"Evaluating improvements to a meta-heuristic search for constrained interaction testing","volume":"16","author":"Garvin","year":"2011","journal-title":"Empir. Softw. Eng."},{"key":"2021052810194807400_ref29","author":"Ghandehari","year":"2014"},{"key":"2021052810194807400_ref30","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1002\/stvr.319","article-title":"Combination testing strategies: a survey","volume":"15","author":"Grindal","year":"2005","journal-title":"Softw. Test. Verif. Rel."},{"key":"2021052810194807400_ref31","doi-asserted-by":"crossref","DOI":"10.1002\/0471028959.sof268","article-title":"Random testing","volume-title":"Encyclopedia of Software Engineering","author":"Hamlet","year":"2002"},{"key":"2021052810194807400_ref32","doi-asserted-by":"crossref","first-page":"490","DOI":"10.1109\/TSE.2015.2496939","article-title":"To be optimal or not in test-case prioritization","volume":"42","author":"Hao","year":"2016","journal-title":"IEEE Trans. Softw. Eng."},{"key":"2021052810194807400_ref33","article-title":"Comparing White-Box and Black-Box Test Prioritization","volume-title":"Proc. 38th Int. Conf. Software Engineering (ICSE\u201916), pp. 523\u2013534, Austin, TX, USA, 14 May","author":"Henard","year":"22  2016"},{"key":"2021052810194807400_ref34","doi-asserted-by":"crossref","DOI":"10.1109\/TSE.2019.2942921","article-title":"A survey on adaptive random testing","author":"Huang","year":"2019","journal-title":"IEEE Trans. Softw. Eng."},{"key":"2021052810194807400_ref35","article-title":"Poster: An Experimental Analysis of Fault Detection Capabilities of Covering Array Constructors","volume-title":"Proc. 40th Int. Conf. Software Engineering Companion (ICSE-C\u201918), pp. 141\u2013142, Gothenburg, Sweden, 27 May","author":"Huang","year":"2018"},{"key":"2021052810194807400_ref36","doi-asserted-by":"crossref","first-page":"830","DOI":"10.1587\/transinf.E97.D.830","article-title":"New metrics for prioritized interaction test suites","volume":"97-D","author":"Huang","year":"2014","journal-title":"IEICE Trans. Inf. Syst."},{"key":"2021052810194807400_ref37","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1109\/TSE.2010.62","article-title":"An analysis and survey of the development of mutation testing","volume":"37","author":"Jia","year":"2011","journal-title":"IEEE Trans. Softw. Eng."},{"key":"2021052810194807400_ref38","first-page":"540","article-title":"Learning Combinatorial Interaction Test Generation Strategies Using Hyperheuristic Search","volume-title":"Proc. 37th Int. Conf. Software Engineering (ICSE\u201915)","author":"Yue Jia","year":"2015"},{"key":"2021052810194807400_ref39","doi-asserted-by":"crossref","first-page":"323","DOI":"10.1109\/ISSRE.2014.15","article-title":"An Orchestrated Survey of Available Algorithms and Tools for Combinatorial Testing","author":"Khalsa","year":"2014","journal-title":"Proc. 25th Int. Symposium on Software Reliability Engineering (ISSRE\u201914)"},{"key":"2021052810194807400_ref40","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1016\/S0950-5849(01)00222-1","article-title":"Non-specification-based approaches to logic testing for software","volume":"44","author":"Kobayashi","year":"2002","journal-title":"Inf. Softw. Technol."},{"key":"2021052810194807400_ref41","first-page":"83","article-title":"Random vs. Combinatorial Methods for Discrete Event Simulation of a Grid Computer Network","volume-title":"Proc. 3rd MODSIM World Conference","author":"Kuhn","year":"2009"},{"key":"2021052810194807400_ref42","first-page":"352","article-title":"Combinatorial Coverage Measurement Concepts and Applications","volume-title":"Proc. 6th Int. Conf. Software Testing, Verification and Validation Workshops (ICSTW\u201913)","author":"Kuhn","year":"2013"},{"key":"2021052810194807400_ref43","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1002\/stvr.381","article-title":"IPOG\/IPOD: efficient test generation for multi-way software testing","volume":"18","author":"Lei","year":"2008","journal-title":"Softw. Test., Verif. Rel."},{"key":"2021052810194807400_ref44","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1109\/TSE.2007.38","article-title":"Search algorithms for regression test case prioritization","volume":"33","author":"Li","year":"2007","journal-title":"IEEE Trans. Softw. Eng."},{"key":"2021052810194807400_ref45","author":"Lin","year":"2015"},{"key":"2021052810194807400_ref46","doi-asserted-by":"crossref","first-page":"11:1","DOI":"10.1145\/1883612.1883618","article-title":"A survey of combinatorial testing","volume":"43","author":"Nie","year":"2011","journal-title":"ACM Comput. Surv."},{"key":"2021052810194807400_ref47","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1016\/j.infsof.2015.02.008","article-title":"Combinatorial testing, random testing, and adaptive random testing for detecting interaction triggered failures","volume":"62","author":"Nie","year":"2015","journal-title":"Inf. Softw. Technol."},{"key":"2021052810194807400_ref48","first-page":"17","article-title":"Constraints: The Future of Combinatorial Interaction Testing","volume-title":"Proc. 8th Int. Workshop on Search-Based Software Testing (SBST\u201915)","author":"Petke","year":"2015"},{"key":"2021052810194807400_ref49","author":"Petke","year":"26  2013"},{"key":"2021052810194807400_ref50","doi-asserted-by":"crossref","first-page":"901","DOI":"10.1109\/TSE.2015.2421279","article-title":"Practical combinatorial interaction testing: empirical findings on efficiency and early fault detection","volume":"41","author":"Petke","year":"2015","journal-title":"IEEE Trans. Softw. Eng."},{"key":"2021052810194807400_ref51","author":"Qu","year":"2007"},{"key":"2021052810194807400_ref52","author":"Qu","year":"2013"},{"key":"2021052810194807400_ref53","doi-asserted-by":"crossref","first-page":"929","DOI":"10.1109\/32.962562","article-title":"Prioritizing test cases for regression testing","volume":"27","author":"Rothermel","year":"2001","journal-title":"IEEE Trans. Softw. Eng."},{"key":"2021052810194807400_ref54","author":"Schroeder","year":"2004"},{"key":"2021052810194807400_ref55","first-page":"151","article-title":"Effective Testing of Factor Combinations","volume-title":"Proc. 3rd Int. Conference on Software Testing, Analysis and Review (STARy94)","author":"Sherwood","year":"1994"},{"key":"2021052810194807400_ref56","first-page":"72","article-title":"Using Artificial Life Techniques to Generate Test Cases for Combinatorial Testing","volume-title":"Proc. 28th Annual Int. Computer Software and Applications Conference (COMPSAC\u201904)","author":"Shiba","year":"2004"},{"key":"2021052810194807400_ref57","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1109\/32.979992","article-title":"A test generation strategy for pairwise testing","volume":"28","author":"Tai","year":"2002","journal-title":"IEEE Trans. Softw. Eng."},{"key":"2021052810194807400_ref58","first-page":"101","article-title":"A critique and improvment of the cl common language effect size statistics of Mcgraw and Wong","volume":"25","author":"Vargha","year":"2000","journal-title":"J. Educ. Behav. Stat."},{"key":"2021052810194807400_ref59","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1016\/j.jspi.2008.05.020","article-title":"Tabu search for covering arrays using permutation vectors","volume":"139","author":"Walker","year":"2009","journal-title":"J. Stat. Plan. Inference"},{"key":"2021052810194807400_ref60","first-page":"304","article-title":"A Measure for Component Interaction Test Coverage","volume-title":"Proc. ACS\/IEEE Int. Conf. Computer Systems and Applications (AICCSA\u201901)","author":"Williams","year":"2001"},{"key":"2021052810194807400_ref61","doi-asserted-by":"crossref","first-page":"575","DOI":"10.1109\/TEVC.2014.2362532","article-title":"A discrete particle swarm optimization for covering array generation","volume":"19","author":"Wu","year":"2015","journal-title":"IEEE Trans. Evol. Comput."},{"key":"2021052810194807400_ref62","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1109\/ISSRE.1997.630850","article-title":"Automatic Test Generation Using Checkpoint Encoding and Antirandom Testing","volume-title":"Proc. 8th Int. Symposium on Software Reliability Engineering (ISSRE\u201997)","author":"Yin","year":"1997"},{"key":"2021052810194807400_ref63","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/j.asoc.2016.03.021","article-title":"A Tabu search hyper-heuristic strategy for t-way test suite generation","volume":"44","author":"Zamli","year":"2016","journal-title":"Appl. Soft Comput. J."},{"key":"2021052810194807400_ref64","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1016\/j.ins.2017.03.007","article-title":"An experimental study of hyper-heuristic selection and acceptance mechanism for combinatorial t-way test suite generation","volume":"399","author":"Zamli","year":"2017","journal-title":"Inf. Sci."},{"key":"2021052810194807400_ref65","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-662-43429-1","article-title":"Automatic Generation of Combinatorial Test Data","volume-title":"Springer Briefs in Computer Science","author":"Zhang","year":"2014"}],"container-title":["The Computer Journal"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/comjnl\/article-pdf\/64\/5\/762\/38335286\/bxaa020.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/comjnl\/article-pdf\/64\/5\/762\/38335286\/bxaa020.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,5,28]],"date-time":"2021-05-28T11:58:50Z","timestamp":1622203130000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/comjnl\/article\/64\/5\/762\/5822659"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,4,20]]},"references-count":65,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2020,4,20]]},"published-print":{"date-parts":[[2021,5,19]]}},"URL":"https:\/\/doi.org\/10.1093\/comjnl\/bxaa020","relation":{},"ISSN":["0010-4620","1460-2067"],"issn-type":[{"value":"0010-4620","type":"print"},{"value":"1460-2067","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,5]]},"published":{"date-parts":[[2020,4,20]]}}}