{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T01:47:03Z","timestamp":1775094423554,"version":"3.50.1"},"reference-count":31,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2012,11,1]],"date-time":"2012-11-01T00:00:00Z","timestamp":1351728000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2012,11]]},"abstract":"<jats:p>\n            <jats:italic>Context<\/jats:italic>\n            . Researchers proposing a new metric have the burden of proof to demonstrate to the research community that the metric is acceptable in its intended use. This burden of proof is provided through the multi-faceted, scientific, and objective process of software metrics validation. Over the last 40 years, however, researchers have debated what constitutes a \u201cvalid\u201d metric.\n          <\/jats:p>\n          <jats:p>\n            <jats:italic>Aim<\/jats:italic>\n            . The debate over what constitutes a valid metric centers on software metrics validation criteria. The objective of this article is to guide researchers in making sound contributions to the field of software engineering metrics by providing a practical summary of the metrics validation criteria found in the academic literature.\n          <\/jats:p>\n          <jats:p>\n            <jats:italic>Method<\/jats:italic>\n            . We conducted a systematic literature review that began with 2,288 papers and ultimately focused on 20 papers. After extracting 47 unique validation criteria from these 20 papers, we performed a comparative analysis to explore the relationships amongst the criteria.\n          <\/jats:p>\n          <jats:p>\n            <jats:italic>Results<\/jats:italic>\n            . Our 47 validation criteria represent a diverse view of what constitutes a valid metric. We present an analysis of the criteria's categorization, conflicts, common themes, and philosophical motivations behind the validation criteria.\n          <\/jats:p>\n          <jats:p>\n            <jats:italic>Conclusions<\/jats:italic>\n            . Although the 47 validation criteria are not conflict-free, the diversity of motivations and philosophies behind the validation criteria indicates that metrics validation is complex. Researchers proposing new metrics should consider the applicability of the validation criteria in terms of our categorization and analysis. Rather than arbitrarily choosing validation criteria for each metric, researchers should choose criteria that can confirm that the metric is appropriate for its intended use. We conclude that metrics validation criteria provide answers to questions that researchers have about the merits and limitations of a metric.\n          <\/jats:p>","DOI":"10.1145\/2377656.2377661","type":"journal-article","created":{"date-parts":[[2013,2,5]],"date-time":"2013-02-05T13:19:41Z","timestamp":1360070381000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":45,"title":["Validating software metrics"],"prefix":"10.1145","volume":"21","author":[{"given":"Andrew","family":"Meneely","sequence":"first","affiliation":[{"name":"North Carolina State University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ben","family":"Smith","sequence":"additional","affiliation":[{"name":"North Carolina State University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Laurie","family":"Williams","sequence":"additional","affiliation":[{"name":"North Carolina State University, Raleigh, NC"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2013,2,7]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2008.130"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/0164-1212(90)90050-V"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1984.5010301"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2006.07.009"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00125812"},{"key":"e_1_2_1_6_1","unstructured":"Briand L. Emam K. E. and Morasca S. 1995. Theoretical and empirical validation of software product measures. Tech. rep. &num;ISERN-95-04. International Software Engineering Research Network.  Briand L. Emam K. E. and Morasca S. 1995. Theoretical and empirical validation of software product measures. Tech. rep. &num;ISERN-95-04. International Software Engineering Research Network."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/0164-1212(90)90043-L"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/800283.811113"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1049\/sej.1993.0002"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/PROC.1980.11813"},{"key":"e_1_2_1_11_1","volume-title":"A methodology for validating software product metrics. Tech. rep. NCR\/ERC-1076","author":"El Emam K.","unstructured":"El Emam , K. 2000. A methodology for validating software product metrics. Tech. rep. NCR\/ERC-1076 , National Research Council of Canada , Ottawa, Ontario , Canada. El Emam, K. 2000. A methodology for validating software product metrics. Tech. rep. NCR\/ERC-1076, National Research Council of Canada, Ottawa, Ontario, Canada."},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the 5th International Software Metrics Symposium.","author":"Elbaum S. G.","unstructured":"Elbaum , S. G. and Munson , J. C . 1998. Getting a handle on the fault injection process: validation of measurement tools . In Proceedings of the 5th International Software Metrics Symposium. Elbaum, S. G. and Munson, J. C. 1998. Getting a handle on the fault injection process: validation of measurement tools. In Proceedings of the 5th International Software Metrics Symposium."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.268921"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.4370010204"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0164-1212(99)00035-7"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/336512.336588"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.5555\/1018439.1021893"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/235969.235994"},{"key":"e_1_2_1_19_1","first-page":"12","volume":"610","author":"IEEE","year":"1990","unstructured":"IEEE 1990 . IEEE Standard 610 . 12 - 1990 . IEEE Standard Glossary of Software Engineering Terminology. IEEE 1990. IEEE Standard 610.12-1990. IEEE Standard Glossary of Software Engineering Terminology.","journal-title":"IEEE Standard"},{"key":"e_1_2_1_20_1","unstructured":"ISO\/IEC 1991. ISO\/IEC 9126: Information technology - software product evaluation - quality characteristics and guidelines for their use.  ISO\/IEC 1991. ISO\/IEC 9126: Information technology - software product evaluation - quality characteristics and guidelines for their use."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/2.312055"},{"key":"e_1_2_1_22_1","volume-title":"Procedures for performing systematic reviews. Joint Tech. rep","author":"Kitchenham B.","unstructured":"Kitchenham , B. 2004. Procedures for performing systematic reviews. Joint Tech. rep ., Keele University Tech . rep. TR\/SE-0401 and NICTA Tech. rep. 0400011T. Kitchenham, B. 2004. Procedures for performing systematic reviews. Joint Tech. rep., Keele University Tech. rep. TR\/SE-0401 and NICTA Tech. rep. 0400011T."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2009.06.041"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.489070"},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the 3rd International Workshop on Metamodels, Schemas, Grammars, and Ontologies for Reverse Engineering (ATEM'06)","author":"Lincke R.","unstructured":"Lincke , R. and Lowe , W . 2006. Foundations for defining software metrics . In Proceedings of the 3rd International Workshop on Metamodels, Schemas, Grammars, and Ontologies for Reverse Engineering (ATEM'06) . Lincke, R. and Lowe, W. 2006. Foundations for defining software metrics. In Proceedings of the 3rd International Workshop on Metamodels, Schemas, Grammars, and Ontologies for Reverse Engineering (ATEM'06)."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1976.233837"},{"key":"e_1_2_1_27_1","volume-title":"Statistical Research Methods in Life Sciences","author":"Rao P. V.","unstructured":"Rao , P. V. 2007. Statistical Research Methods in Life Sciences . Duxbury Press , Pacific Grove, CA . Rao, P. V. 2007. Statistical Research Methods in Life Sciences. Duxbury Press, Pacific Grove, CA."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/181610.181625"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.1991.145385"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.135774"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.6178"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2377656.2377661","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2377656.2377661","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T09:34:33Z","timestamp":1750239273000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2377656.2377661"}},"subtitle":["A spectrum of philosophies"],"short-title":[],"issued":{"date-parts":[[2012,11]]},"references-count":31,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2012,11]]}},"alternative-id":["10.1145\/2377656.2377661"],"URL":"https:\/\/doi.org\/10.1145\/2377656.2377661","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2012,11]]},"assertion":[{"value":"2010-09-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2011-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2013-02-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}