{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,19]],"date-time":"2026-03-19T14:48:46Z","timestamp":1773931726980,"version":"3.50.1"},"reference-count":72,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2015,12,2]],"date-time":"2015-12-02T00:00:00Z","timestamp":1449014400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000266","name":"EPSRC","doi-asserted-by":"crossref","award":["EP\/I010386\/1"],"award-info":[{"award-number":["EP\/I010386\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2015,12,2]]},"abstract":"<jats:p>Despite industry advice to the contrary, there has been little work that has sought to test that a relational database's schema has correctly specified integrity constraints. These critically important constraints ensure the coherence of data in a database, defending it from manipulations that could violate requirements such as \u201cusernames must be unique\u201d or \u201cthe host name cannot be missing or unknown.\u201d This article is the first to propose coverage criteria, derived from logic coverage criteria, that establish different levels of testing for the formulation of integrity constraints in a database schema. These range from simple criteria that mandate the testing of successful and unsuccessful INSERT statements into tables to more advanced criteria that test the formulation of complex integrity constraints such as multi-column PRIMARY KEYs and arbitrary CHECK constraints. Due to different vendor interpretations of the structured query language (SQL) specification with regard to how integrity constraints should actually function in practice, our criteria crucially account for the underlying semantics of the database management system (DBMS). After formally defining these coverage criteria and relating them in a subsumption hierarchy, we present two approaches for automatically generating tests that satisfy the criteria. We then describe the results of an empirical study that uses mutation analysis to investigate the fault-finding capability of data generated when our coverage criteria are applied to a wide variety of relational schemas hosted by three well-known and representative DBMSs\u2014HyperSQL, PostgreSQL, and SQLite. In addition to revealing the complementary fault-finding capabilities of the presented criteria, the results show that mutation scores range from as low as just 12% of mutants being killed with the simplest of criteria to 96% with the most advanced.<\/jats:p>","DOI":"10.1145\/2818639","type":"journal-article","created":{"date-parts":[[2015,12,4]],"date-time":"2015-12-04T13:43:07Z","timestamp":1449236587000},"page":"1-49","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":30,"title":["The Effectiveness of Test Coverage Criteria for Relational Database Schema Integrity Constraints"],"prefix":"10.1145","volume":"25","author":[{"given":"Phil","family":"Mcminn","sequence":"first","affiliation":[{"name":"University of Sheffield, Sheffield, UK"}]},{"given":"Chris J.","family":"Wright","sequence":"additional","affiliation":[{"name":"University of Sheffield, Sheffield, UK"}]},{"given":"Gregory M.","family":"Kapfhammer","sequence":"additional","affiliation":[{"name":"Allegheny College, Meadville, PA"}]}],"member":"320","published-online":{"date-parts":[[2015,12,2]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2013.11"},{"key":"e_1_2_1_2_1","volume-title":"Sadalage","author":"Ambler Scott","year":"2006","unstructured":"Scott Ambler and Pramod J . Sadalage . 2006 . Refactoring Databases : Evolutionary Database Design. Pearson Education . Scott Ambler and Pramod J. Sadalage. 2006. Refactoring Databases: Evolutionary Database Design. Pearson Education."},{"key":"e_1_2_1_3_1","doi-asserted-by":"crossref","unstructured":"P. Ammann and J. Offutt. 2008. Introduction to Software Testing. Cambridge University Press.   P. Ammann and J. Offutt. 2008. Introduction to Software Testing. Cambridge University Press.","DOI":"10.1017\/CBO9780511809163"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1062455.1062530"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2010.17"},{"key":"e_1_2_1_6_1","volume-title":"Parameter tuning or default values? An empirical investigation in search-based software engineering. Empirical Softw. Eng. 18, 3","author":"Arcuri Andrea","year":"2013","unstructured":"Andrea Arcuri and Gordon Fraser . 2013. Parameter tuning or default values? An empirical investigation in search-based software engineering. Empirical Softw. Eng. 18, 3 ( 2013 ). Andrea Arcuri and Gordon Fraser. 2013. Parameter tuning or default values? An empirical investigation in search-based software engineering. Empirical Softw. Eng. 18, 3 (2013)."},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 10th Working Conference on Mining Software Repositories.","author":"Bacchelli Alberto","year":"2013","unstructured":"Alberto Bacchelli . 2013 . Mining challenge 2013: Stack overflow . In Proceedings of the 10th Working Conference on Mining Software Repositories. Alberto Bacchelli. 2013. Mining challenge 2013: Stack overflow. In Proceedings of the 10th Working Conference on Mining Software Repositories."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.5555\/1325851.1325993"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 12th International Conference on Music Information Retrieval.","author":"Bertin-Mahieux Thierry","year":"2011","unstructured":"Thierry Bertin-Mahieux , Daniel P. W. Ellis , Brian Whitman , and Paul Lamere . 2011 . The million song dataset . In Proceedings of the 12th International Conference on Music Information Retrieval. Thierry Bertin-Mahieux, Daniel P. W. Ellis, Brian Whitman, and Paul Lamere. 2011. The million song dataset. In Proceedings of the 12th International Conference on Music Information Retrieval."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.5555\/1083592.1083719"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2006.190"},{"key":"e_1_2_1_12_1","volume-title":"Amazon: Our cloud powered Obama's campaign. Network World","author":"Butler Brandon","year":"2012","unstructured":"Brandon Butler . 2012 . Amazon: Our cloud powered Obama's campaign. Network World (2012). Brandon Butler. 2012. Amazon: Our cloud powered Obama's campaign. Network World (2012)."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2005.27"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/347324.348954"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1049\/sej.1994.0025"},{"key":"e_1_2_1_16_1","unstructured":"John J. Chilenski and L. A. Richey. 1997. Definition for a Masking Form of Modified Condition Decision Coverage. Technical Report. Boeing.  John J. Chilenski and L. A. Richey. 1997. Definition for a Masking Form of Modified Condition Decision Coverage. Technical Report. Boeing."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2011.03.011"},{"key":"e_1_2_1_18_1","volume-title":"IEE Proc. Softw. 150","author":"Clark John A.","year":"2003","unstructured":"John A. Clark , Jos Javier Dolado , Mark Harman , Robert M. Hierons , Bryan F. Jones , M. Lumkin , Brian S. Mitchell , Spiros Mancoridis , K. Rees , Marc Roper , and Martin J. Shepperd . 2003. Reformulating software engineering as a search problem . IEE Proc. Softw. 150 , 3 ( 2003 ). John A. Clark, Jos Javier Dolado, Mark Harman, Robert M. Hierons, Bryan F. Jones, M. Lumkin, Brian S. Mitchell, Spiros Mancoridis, K. Rees, Marc Roper, and Martin J. Shepperd. 2003. Reformulating software engineering as a search problem. IEE Proc. Softw. 150, 3 (2003)."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2011.6100056"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2002951.2002955"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/362384.362685"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1808266.1808276"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/C-M.1978.218136"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2006.92"},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the 19th Digital Avionics Systems Conference.","author":"Dupuy A.","unstructured":"A. Dupuy and N. Leveson . 2000. An empirical evaluation of the MC\/DC coverage criterion on the HETE-2 satellite software . In Proceedings of the 19th Digital Avionics Systems Conference. A. Dupuy and N. Leveson. 2000. An empirical evaluation of the MC\/DC coverage criterion on the HETE-2 satellite software. In Proceedings of the 19th Digital Avionics Systems Conference."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2342356.2342360"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9299-z"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2001.922739"},{"key":"e_1_2_1_29_1","unstructured":"Szymon Guz. 2011. Basic mistakes in database testing. http:\/\/java.dzone.com\/articles\/basic-mistakes-database. (Last accessed January 2014).  Szymon Guz. 2011. Basic mistakes in database testing. http:\/\/java.dzone.com\/articles\/basic-mistakes-database. (Last accessed January 2014)."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-006-0028-8"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2006.27"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2010.13"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.5555\/1182635.1164254"},{"key":"e_1_2_1_34_1","volume-title":"Advancing declarative query in the long tail of science. Data Eng. Bullet. 35, 3","author":"Howe Bill","year":"2012","unstructured":"Bill Howe and Daniel Halperin . 2012. Advancing declarative query in the long tail of science. Data Eng. Bullet. 35, 3 ( 2012 ). Bill Howe and Daniel Halperin. 2012. Advancing declarative query in the long tail of science. Data Eng. Bullet. 35, 3 (2012)."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-002-0081-x"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.62"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635929"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2012.08.024"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2012.206"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2013.47"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/940071.940086"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1342211.1342228"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tcs.2014.12.009"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2008.34"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1858996.1859063"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.18293\/SEKE2015-205"},{"key":"e_1_2_1_48_1","volume-title":"Proceedings of the 27th International Conference on Software Engineering and Knowledge Engineering.","author":"Kinneer Cody","year":"2015","unstructured":"Cody Kinneer , Gregory M. Kapfhammer , Chris J. Wright , and Phil McMinn . 2015 b. expOse: Inferring worst-case time complexity by automatic empirical study . In Proceedings of the 27th International Conference on Software Engineering and Knowledge Engineering. Cody Kinneer, Gregory M. Kapfhammer, Chris J. Wright, and Phil McMinn. 2015b. expOse: Inferring worst-case time complexity by automatic empirical study. In Proceedings of the 27th International Conference on Software Engineering and Knowledge Engineering."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.57624"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2014.43"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-009-0157-y"},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the Workshop on Interfaces and Abstractions for Scientific Data Storage.","author":"Loebman Sarah","unstructured":"Sarah Loebman , Dylan Nunley , Yongchul Kwon , Bill Howe , Magdalena Balazinska , and Jeffrey P. Gardner . 2009. Analyzing massive astrophysical datasets: Can Pig\/Hadoop or a relational DBMS help? In Proceedings of the Workshop on Interfaces and Abstractions for Scientific Data Storage. Sarah Loebman, Dylan Nunley, Yongchul Kwon, Bill Howe, Magdalena Balazinska, and Jeffrey P. Gardner. 2009. Analyzing massive astrophysical datasets: Can Pig\/Hadoop or a relational DBMS help? In Proceedings of the Workshop on Interfaces and Abstractions for Scientific Data Storage."},{"key":"e_1_2_1_53_1","volume-title":"The Theory of Relational Databases","author":"Maier D.","unstructured":"D. Maier . 1983. The Theory of Relational Databases . Computer Science Press . D. Maier. 1983. The Theory of Relational Databases. Computer Science Press."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.v14:2"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2012.94"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2011.6100152"},{"key":"e_1_2_1_57_1","volume-title":"Practical Issues in Database Management: A Reference for the Thinking Practitioner","author":"Pascal Fabian","unstructured":"Fabian Pascal . 2000. Practical Issues in Database Management: A Reference for the Thinking Practitioner . Addison-Wesley . Fabian Pascal. 2000. Practical Issues in Database Management: A Reference for the Thinking Practitioner. Addison-Wesley."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/1317471.1317478"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/2491411.2491431"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2011.5767876"},{"key":"e_1_2_1_61_1","volume-title":"Automatic generation of valid and invalid test data for string validation routines using web searches and regular expressions. Sci. Comput. Program. 97, 4","author":"Shahbaz Muzammil","year":"2015","unstructured":"Muzammil Shahbaz , Phil McMinn , and Mark Stevenson . 2015. Automatic generation of valid and invalid test data for string validation routines using web searches and regular expressions. Sci. Comput. Program. 97, 4 ( 2015 ). Muzammil Shahbaz, Phil McMinn, and Mark Stevenson. 2015. Automatic generation of valid and invalid test data for string validation routines using web searches and regular expressions. Sci. Comput. Program. 97, 4 (2015)."},{"key":"e_1_2_1_62_1","unstructured":"Abraham Silberschatz Henry F. Korth and S. Sudarshan. 2010. Database System Concepts (6th ed.). McGraw-Hill.  Abraham Silberschatz Henry F. Korth and S. Sudarshan. 2010. Database System Concepts (6th ed.). McGraw-Hill."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.5555\/645924.671199"},{"key":"e_1_2_1_64_1","volume-title":"Proceedings of the Testing: Academic and Industrial Conference - Practice and Research Techniques and the International Workshop on Mutation Analysis.","author":"Smith B. H.","unstructured":"B. H. Smith and L. Williams . 2007. An empirical evaluation of the MuJava mutation operators . In Proceedings of the Testing: Academic and Industrial Conference - Practice and Research Techniques and the International Workshop on Mutation Analysis. B. H. Smith and L. Williams. 2007. An empirical evaluation of the MuJava mutation operators. In Proceedings of the Testing: Academic and Industrial Conference - Practice and Research Techniques and the International Workshop on Mutation Analysis."},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/1629175.1629197"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.is.2013.07.004"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2006.06.009"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.v20:3"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/1134285.1134301"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2013.15"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2014.26"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.462"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/267580.267590"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2818639","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2818639","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T05:42:49Z","timestamp":1750225369000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2818639"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,12,2]]},"references-count":72,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2015,12,2]]}},"alternative-id":["10.1145\/2818639"],"URL":"https:\/\/doi.org\/10.1145\/2818639","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,12,2]]},"assertion":[{"value":"2015-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-12-02","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}