{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T21:00:36Z","timestamp":1774731636304,"version":"3.50.1"},"reference-count":42,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2015,9,2]],"date-time":"2015-09-02T00:00:00Z","timestamp":1441152000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Research Fund, Luxembourg","award":["FNR\/P10\/03"],"award-info":[{"award-number":["FNR\/P10\/03"]}]},{"DOI":"10.13039\/501100000266","name":"EPSRC","doi-asserted-by":"crossref","award":["EP\/K030353\/1 (\u201cEXOGEN\u201d) and EP\/I010386\/1 (\u201cRE-COST: REducing the Cost of Oracles in Software Testing\u201d)"],"award-info":[{"award-number":["EP\/K030353\/1 (\u201cEXOGEN\u201d) and EP\/I010386\/1 (\u201cRE-COST: REducing the Cost of Oracles in Software Testing\u201d)"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Google Focused Research Award on \u201cTest Amplification\u201d"},{"DOI":"10.13039\/501100005416","name":"Norwegian Research Council","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005416","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2015,9,2]]},"abstract":"<jats:p>\n            Work on automated test generation has produced several tools capable of generating test data which achieves high structural coverage over a program. In the absence of a specification, developers are expected to manually construct or verify the test oracle for each test input. Nevertheless, it is assumed that these generated tests ease the task of testing for the developer, as testing is reduced to checking the results of tests. While this assumption has persisted for decades, there has been no conclusive evidence to date confirming it. However, the limited adoption in industry indicates this assumption may not be correct, and calls into question the practical value of test generation tools. To investigate this issue, we performed two controlled experiments comparing a total of 97 subjects split between writing tests manually and writing tests with the aid of an automated unit test generation tool, E\n            <jats:sc>vo<\/jats:sc>\n            S\n            <jats:sc>uite<\/jats:sc>\n            . We found that, on one hand, tool support leads to clear improvements in commonly applied quality metrics such as code coverage (up to 300% increase). However, on the other hand, there was no measurable improvement in the number of bugs actually found by developers. Our results not only cast some doubt on how the research community evaluates test generation tools, but also point to improvements and future work necessary before automated test generation tools will be widely adopted by practitioners.\n          <\/jats:p>","DOI":"10.1145\/2699688","type":"journal-article","created":{"date-parts":[[2015,9,8]],"date-time":"2015-09-08T12:36:15Z","timestamp":1441715775000},"page":"1-49","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":83,"title":["Does Automated Unit Test Generation Really Help Software Testers? A Controlled Empirical Study"],"prefix":"10.1145","volume":"24","author":[{"given":"Gordon","family":"Fraser","sequence":"first","affiliation":[{"name":"University of Sheffield, Sheffield, UK"}]},{"given":"Matt","family":"Staats","sequence":"additional","affiliation":[{"name":"University of Luxembourg, Luxembourg"}]},{"given":"Phil","family":"McMinn","sequence":"additional","affiliation":[{"name":"University of Sheffield, Sheffield, UK"}]},{"given":"Andrea","family":"Arcuri","sequence":"additional","affiliation":[{"name":"Simula Research Laboratory, Lysaker, Norway"}]},{"given":"Frank","family":"Padberg","sequence":"additional","affiliation":[{"name":"Karlsruhe Institute of Technology, Karlsruhe, Germany"}]}],"member":"320","published-online":{"date-parts":[[2015,9,2]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2013.11"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1062455.1062530"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1486"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2010.54"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2076021.2048117"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1002\/spe.602"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/SSBSE.2010.25"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2006.92"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2025113.2025179"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2012.92"},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the ACM\/IEEE International Conference on Software Engineering (ICSE). 178--188","author":"Fraser G.","unstructured":"G. Fraser and A. Arcuri . 2012b. Sound empirical evidence in software testing . In Proceedings of the ACM\/IEEE International Conference on Software Engineering (ICSE). 178--188 . G. Fraser and A. Arcuri. 2012b. Sound empirical evidence in software testing. In Proceedings of the ACM\/IEEE International Conference on Software Engineering (ICSE). 178--188."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2012.14"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2483760.2483774"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2011.53"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2011.93"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2379776.2379787"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2009.71"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467-9280.00272"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1868321.1868326"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2011.6100138"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2002.1027796"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10710-010-9112-3"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2010.07.026"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 24th International IEEE Symposium on Proceedings of Software Reliability Engineering (ISSRE). 380--389","author":"Li N.","unstructured":"N. Li , X. Meng , J. Offutt , and L. Deng . 2013. Is bytecode instrumentation as good as source code instrumentation: An empirical study with industrial tools (Experience Report) . In Proceedings of the 24th International IEEE Symposium on Proceedings of Software Reliability Engineering (ISSRE). 380--389 . N. Li, X. Meng, J. Offutt, and L. Deng. 2013. Is bytecode instrumentation as good as source code instrumentation: An empirical study with industrial tools (Experience Report). In Proceedings of the 24th International IEEE Symposium on Proceedings of Software Reliability Engineering (ISSRE). 380--389."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.v14:2"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/800027.808424"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1572272.1572280"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1297846.1297902"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2001420.2001445"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1858996.1859035"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2013.13"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/SEAA.2012.42"},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the European Conference on Research and Advanced Technology for Digital Libraries. 357--367","author":"Sautter G.","unstructured":"G. Sautter , K. B\u00f6hm , F. Padberg , and W. Tichy . 2007. Empirical Evaluation of Semi-Automated XML Annotation of Text Documents with the GoldenGATE Editor . In Proceedings of the European Conference on Research and Advanced Technology for Digital Libraries. 357--367 . G. Sautter, K. B\u00f6hm, F. Padberg, and W. Tichy. 2007. Empirical Evaluation of Semi-Automated XML Annotation of Text Documents with the GoldenGATE Editor. In Proceedings of the European Conference on Research and Advanced Technology for Digital Libraries. 357--367."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.799955"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2005.97"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.5555\/2337223.2337326"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2338965.2336776"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.5555\/1792786.1792798"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/1007512.1007528"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0950-5849(01)00190-2"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1985793.1985820"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.435"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2699688","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2699688","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T06:16:59Z","timestamp":1750227419000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2699688"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,9,2]]},"references-count":42,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2015,9,2]]}},"alternative-id":["10.1145\/2699688"],"URL":"https:\/\/doi.org\/10.1145\/2699688","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,9,2]]},"assertion":[{"value":"2014-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2014-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-09-02","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}