{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T21:15:21Z","timestamp":1783804521253,"version":"3.55.0"},"reference-count":35,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2015,12,2]],"date-time":"2015-12-02T00:00:00Z","timestamp":1449014400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2015,12,2]]},"abstract":"<jats:p>Several techniques and tools have been proposed for the automatic generation of test cases. Usually, these tools are evaluated in terms of fault-revealing or coverage capability, but their impact on the manual debugging activity is not considered. The question is whether automatically generated test cases are equally effective in supporting debugging as manually written tests.<\/jats:p>\n          <jats:p>We conducted a family of three experiments (five replications) with humans (in total, 55 subjects) to assess whether the features of automatically generated test cases, which make them less readable and understandable (e.g., unclear test scenarios, meaningless identifiers), have an impact on the effectiveness and efficiency of debugging. The first two experiments compare different test case generation tools (Randoop vs. EvoSuite). The third experiment investigates the role of code identifiers in test cases (obfuscated vs. original identifiers), since a major difference between manual and automatically generated test cases is that the latter contain meaningless (obfuscated) identifiers.<\/jats:p>\n          <jats:p>We show that automatically generated test cases are as useful for debugging as manual test cases. Furthermore, we find that, for less experienced developers, automatic tests are more useful on average due to their lower static and dynamic complexity.<\/jats:p>","DOI":"10.1145\/2768829","type":"journal-article","created":{"date-parts":[[2015,12,4]],"date-time":"2015-12-04T13:43:07Z","timestamp":1449236587000},"page":"1-38","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":42,"title":["Do Automatically Generated Test Cases Make Debugging Easier? An Experimental Assessment of Debugging Effectiveness and Efficiency"],"prefix":"10.1145","volume":"25","author":[{"given":"Mariano","family":"Ceccato","sequence":"first","affiliation":[{"name":"Fondazione Bruno Kessler, Trento, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alessandro","family":"Marchetto","sequence":"additional","affiliation":[{"name":"Fondazione Bruno Kessler, Trento, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Leonardo","family":"Mariani","sequence":"additional","affiliation":[{"name":"University of Milano Bicocca, Milano, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cu D.","family":"Nguyen","sequence":"additional","affiliation":[{"name":"Fondazione Bruno Kessler and SnT Centre, University of Luxembourg, Luxembourg"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paolo","family":"Tonella","sequence":"additional","affiliation":[{"name":"Fondazione Bruno Kessler, Trento, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2015,12,2]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1062455.1062530"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2008.12"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1831708.1831715"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.49"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2008.69"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1455518.1455522"},{"key":"e_1_2_2_7_1","volume-title":"Proceedings of the 34th International Conference on Software Engineering (ICSE'12)","author":"Ceccato M.","unstructured":"M. Ceccato , A. Marchetto , L. Mariani , C. D. Nguyen , and P. Tonella . 2012. An empirical study about the effectiveness of debugging when random test cases are used . In Proceedings of the 34th International Conference on Software Engineering (ICSE'12) . 452--462. M. Ceccato, A. Marchetto, L. Mariani, C. D. Nguyen, and P. Tonella. 2012. An empirical study about the effectiveness of debugging when random test cases are used. In Proceedings of the 34th International Conference on Software Engineering (ICSE'12). 452--462."},{"key":"e_1_2_2_8_1","unstructured":"M. Ceccato C. D. Nguyen A. Marchetto L. Mariani and P. Tonella. 2013. A family of experiments to assess the impact of automated test case generation on the accuracy and efficiency of debugging data analysis of five replications. Tech. rep. FBK TR-FBK-SE-2013-2. https:\/\/se.fbk.eu\/technical-reports.  M. Ceccato C. D. Nguyen A. Marchetto L. Mariani and P. Tonella. 2013. A family of experiments to assess the impact of automated test case generation on the accuracy and efficiency of debugging data analysis of five replications. Tech. rep. FBK TR-FBK-SE-2013-2. https:\/\/se.fbk.eu\/technical-reports."},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1273463.1273476"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1967.1053964"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-005-3861-2"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1984.5010257"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/120807.120821"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2011.19"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1831708.1831728"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2010.5609691"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1065010.1065036"},{"key":"e_1_2_2_18_1","volume-title":"Proceedings of the Network and Distributed System Security Symposium (NDSS'08)","author":"Godefroid P.","unstructured":"P. Godefroid , M. Y. Levin , and D. A. Molnar . 2008. Automated whitebox fuzz testing . In Proceedings of the Network and Distributed System Security Symposium (NDSS'08) . P. Godefroid, M. Y. Levin, and D. A. Molnar. 2008. Automated whitebox fuzz testing. In Proceedings of the Network and Distributed System Security Symposium (NDSS'08)."},{"key":"e_1_2_2_19_1","first-page":"65","article-title":"A simple sequentially rejective multiple test procedure","volume":"6","author":"Holm S.","year":"1979","unstructured":"S. Holm . 1979 . A simple sequentially rejective multiple test procedure . Scandinav. J. Statist. 6 , 2, 65 -- 70 . S. Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinav. J. Statist. 6, 2, 65--70.","journal-title":"Scandinav. J. Statist."},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2008.03.007"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2009.5314240"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/581339.581397"},{"key":"e_1_2_2_23_1","volume-title":"Questionnaire Design, Interviewing and Attitude Measurement","author":"Oppenheim A. N.","unstructured":"A. N. Oppenheim . 1992. Questionnaire Design, Interviewing and Attitude Measurement . Pinter , London . A. N. Oppenheim. 1992. Questionnaire Design, Interviewing and Attitude Measurement. Pinter, London."},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1297846.1297902"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2001420.2001445"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2008.01.007"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1062455.1062523"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1095430.1081750"},{"key":"e_1_2_2_29_1","volume-title":"Proceedings of the 2nd International Conference on Tests and Proofs (TAP'08)","author":"Tillmann N.","unstructured":"N. Tillmann and J. De Halleux . 2008. Pex: White box test generation for .NET . In Proceedings of the 2nd International Conference on Tests and Proofs (TAP'08) . Springer, 134--153. N. Tillmann and J. De Halleux. 2008. Pex: White box test generation for .NET. In Proceedings of the 2nd International Conference on Tests and Proofs (TAP'08). Springer, 134--153."},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1007512.1007528"},{"key":"e_1_2_2_31_1","volume-title":"Information Retrieval","author":"Van Rijsbergen C. J.","unstructured":"C. J. Van Rijsbergen . 1979. Information Retrieval , 2 nd ed. Butterworths , London . C. J. Van Rijsbergen. 1979. Information Retrieval, 2nd ed. Butterworths, London.","edition":"2"},{"key":"e_1_2_2_32_1","volume-title":"Proceedings of the 1st Workshop on Empirical Studies of Programmers. Ablex Publishing","author":"Weiser M.","unstructured":"M. Weiser and J. Lyle . 1986. Experiments on slicing-based debugging aids . In Proceedings of the 1st Workshop on Empirical Studies of Programmers. Ablex Publishing , Norwood, NJ, 187--197. M. Weiser and J. Lyle. 1986. Experiments on slicing-based debugging aids. In Proceedings of the 1st Workshop on Empirical Studies of Programmers. Ablex Publishing, Norwood, NJ, 187--197."},{"key":"e_1_2_2_33_1","doi-asserted-by":"crossref","unstructured":"C. Wohlin P. Runeson M. Host M. C. Ohlsson B. Regnell and A. Wesslen. 2012. Experimentation in Software Engineering. Springer.   C. Wohlin P. Runeson M. Host M. C. Ohlsson B. Regnell and A. Wesslen. 2012. Experimentation in Software Engineering. Springer.","DOI":"10.1007\/978-3-642-29044-2"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1368088.1368116"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.988498"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2768829","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2768829","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T05:07:37Z","timestamp":1750223257000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2768829"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,12,2]]},"references-count":35,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2015,12,2]]}},"alternative-id":["10.1145\/2768829"],"URL":"https:\/\/doi.org\/10.1145\/2768829","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,12,2]]},"assertion":[{"value":"2014-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-04-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-12-02","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}