{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T06:26:19Z","timestamp":1769754379703,"version":"3.49.0"},"reference-count":37,"publisher":"Springer Science and Business Media LLC","issue":"1","content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Bioinformatics"],"published-print":{"date-parts":[[2009,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Background<\/jats:title><jats:p>Recent advances in experimental and computational technologies have fueled the development of many sophisticated bioinformatics programs. The correctness of such programs is crucial as incorrectly computed results may lead to wrong biological conclusion or misguide downstream experimentation. Common software testing procedures involve executing the target program with a set of test inputs and then verifying the correctness of the test outputs. However, due to the complexity of many bioinformatics programs, it is often difficult to verify the correctness of the test outputs. Therefore our ability to perform systematic software testing is greatly hindered.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>We propose to use a novel software testing technique, metamorphic testing (MT), to test a range of bioinformatics programs. Instead of requiring a mechanism to verify whether an individual test output is correct, the MT technique verifies whether a pair of test outputs conform to a set of domain specific properties, called metamorphic relations (MRs), thus greatly increases the number and variety of test cases that can be applied. To demonstrate how MT is used in practice, we applied MT to test two open-source bioinformatics programs, namely GNLab and SeqMap. In particular we show that MT is simple to implement, and is effective in detecting faults in a real-life program and some artificially fault-seeded programs. Further, we discuss how MT can be applied to test programs from various domains of bioinformatics.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>This paper describes the application of a simple, effective and automated technique to systematically test a range of bioinformatics programs. We show how MT can be implemented in practice through two real-life case studies. Since many bioinformatics programs, particularly those for large scale simulation and data analysis, are hard to test systematically, their developers may benefit from using MT as part of the testing strategy. Therefore our work represents a significant step towards software reliability in bioinformatics.<\/jats:p><\/jats:sec>","DOI":"10.1186\/1471-2105-10-24","type":"journal-article","created":{"date-parts":[[2009,1,20]],"date-time":"2009-01-20T07:14:37Z","timestamp":1232435677000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":111,"title":["An innovative approach for testing bioinformatics programs using metamorphic testing"],"prefix":"10.1186","volume":"10","author":[{"given":"Tsong Yueh","family":"Chen","sequence":"first","affiliation":[]},{"given":"Joshua WK","family":"Ho","sequence":"additional","affiliation":[]},{"given":"Huai","family":"Liu","sequence":"additional","affiliation":[]},{"given":"Xiaoyuan","family":"Xie","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2009,1,19]]},"reference":[{"key":"2754_CR1","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1109\/ICSE.2004.1317409","volume-title":"In Proceedings of International Conference on Software Engineering","author":"J Baker","year":"2004","unstructured":"Baker J, Thornton J: Software Engineering Challenges in Bioinformatics. In Proceedings of International Conference on Software Engineering. 2004, 12-15."},{"key":"2754_CR2","doi-asserted-by":"publisher","DOI":"10.1002\/0470030143","volume-title":"Bioinformatics Software Engineering: Delivering Effective Applications","author":"P Weston","year":"2004","unstructured":"Weston P: Bioinformatics Software Engineering: Delivering Effective Applications. 2004, Wiley"},{"key":"2754_CR3","volume-title":"Software Testing Techniques","author":"B Beizer","year":"1990","unstructured":"Beizer B: Software Testing Techniques . 1990, Van Nostrand Reinhold"},{"key":"2754_CR4","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511809163","volume-title":"Introduction to Software Testing","author":"P Ammann","year":"2008","unstructured":"Ammann P, Offutt J: Introduction to Software Testing. 2008, Cambridge University Press"},{"key":"2754_CR5","first-page":"128","volume-title":"Proceedings of the 5th International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel\/Distributed Computing (SNPD '04)","author":"T Chen","year":"2004","unstructured":"Chen T, Kuo FC, Liu Y, Tang A: Metamorphic Testing and Testing with Special Values. Proceedings of the 5th International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel\/Distributed Computing (SNPD '04). 2004, 128-134."},{"key":"2754_CR6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/S0950-5849(02)00129-5","volume":"45","author":"TY Chen","year":"2003","unstructured":"Chen TY, Tse TH, Zhou ZQ: Fault-based testing without the need of oracles. Inform Software Tech. 2003, 45: 1-9.","journal-title":"Inform Software Tech"},{"issue":"17","key":"2754_CR7","doi-asserted-by":"publisher","first-page":"1963","DOI":"10.1093\/bioinformatics\/btn319","volume":"24","author":"FT Bergmann","year":"2008","unstructured":"Bergmann FT, Sauro HM: Comparing simulation results of SBML capable simulators. Bioinformatics. 2008, 24 (17): 1963-1965.","journal-title":"Bioinformatics"},{"issue":"2","key":"2754_CR8","doi-asserted-by":"publisher","first-page":"285","DOI":"10.1093\/bioinformatics\/btm566","volume":"24","author":"TW Evans","year":"2008","unstructured":"Evans TW, Gillespie CS, Wilkinson DJ: The SBML discrete stochastic models test suite. Bioinformatics. 2008, 24 (2): 285-286.","journal-title":"Bioinformatics"},{"key":"2754_CR9","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1109\/TSE.1986.6312924","volume":"12","author":"JC Knight","year":"1986","unstructured":"Knight JC, Leveson NG: An experimental evaluation of the assumption of independence in multiversion programmings. IEEE Transactions on Software Engineering. 1986, 12: 96-109.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"2754_CR10","volume-title":"Tech Rep HKUST-CS98-01","author":"TY Chen","year":"1998","unstructured":"Chen TY, Cheung SC, Yiu SM: Metamorphic testing: a new approach for generating next test cases. Tech Rep HKUST-CS98-01. 1998, Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong"},{"key":"2754_CR11","first-page":"327","volume-title":"Proceedings of the 26th Annual International Computer Software and Applications Conference (COMPSAC 2002)","author":"TY Chen","year":"2002","unstructured":"Chen TY, Feng J, Tse TH: Metamorphic testing of programs on partial differential equations: a case study. Proceedings of the 26th Annual International Computer Software and Applications Conference (COMPSAC 2002). 2002, IEEE Computer Society Press, Los Alamitos, California, 327-333."},{"key":"2754_CR12","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1109\/CMPSAC.2003.1245319","volume-title":"Proceedings of the 27th Annual International Computer Software and Applications Conference (COMPSAC 2003)","author":"A Gotlieb","year":"2003","unstructured":"Gotlieb A, Botella B: Automated metamorphic testing. Proceedings of the 27th Annual International Computer Software and Applications Conference (COMPSAC 2003). 2003, IEEE Computer Society Press, Los Alamitos, California, 34-40."},{"key":"2754_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1083231.1083236","volume-title":"In Proceedings of the First Workshop on End-User Software Engineering","author":"TY Chen","year":"2005","unstructured":"Chen TY, Kuo FC, Zhou ZQ: An Effective Testing Method for End-User Programmer. In Proceedings of the First Workshop on End-User Software Engineering. 2005, 1-5."},{"key":"2754_CR14","volume-title":"GNLab: computational pipeline for large-scale gene network analysis","author":"JWK Ho","year":"2008","unstructured":"Ho JWK, Charleston MA: GNLab: computational pipeline for large-scale gene network analysis. 2008, [http:\/\/www.cs.usyd.edu.au\/~mcharles\/software\/gnlab\/index.html]"},{"issue":"20","key":"2754_CR15","doi-asserted-by":"publisher","first-page":"2395","DOI":"10.1093\/bioinformatics\/btn429","volume":"24","author":"H Jiang","year":"2008","unstructured":"Jiang H, Wong WH: SeqMap: mapping massive amount of oligonucleotides to the genome. Bioinformatics. 2008, 24 (20): 2395-2396.","journal-title":"Bioinformatics"},{"key":"2754_CR16","first-page":"44","volume-title":"Proceedings of the Eighth International Conference on Systems Biology","author":"JWK Ho","year":"2007","unstructured":"Ho JWK, Charleston MA: Modeling the Evolution of Gene Regulatory Networks. Proceedings of the Eighth International Conference on Systems Biology. 2007, 44-"},{"issue":"1","key":"2754_CR17","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1038\/ng873","volume":"31","author":"N Guelzim","year":"2002","unstructured":"Guelzim N, Bottani S, Bourgine P, K\u00e9p\u00e8s F: Topological and causal structure of the yeast transcriptional regulatory network. Nat Genet. 2002, 31 (1): 60-63.","journal-title":"Nat Genet"},{"key":"2754_CR18","doi-asserted-by":"publisher","first-page":"D394","DOI":"10.1093\/nar\/gkj156","volume":"34 (Database is","author":"H Salgado","year":"2006","unstructured":"Salgado H, Gamma-Castro S, Peralta-Gil M, D\u00edaz-Peredo E, S\u00e1nchez-Solano F, Santo-Zavaleta A, Mart\u00ednez-Flores I, Jim\u00e9nez-Jacinto V, Bonavides-Martinez C, Segura-Salazar J, Mart\u00ednez-Antonio A, Collado-Vides J: RegulonDB (version 5.0): Escherichia coli K-12 transcriptional regulatory network, operon organization, and growth conditions. Nucleic Acids Res. 2006, 34 (Database issue): D394-D397.","journal-title":"Nucleic Acids Res"},{"issue":"8","key":"2754_CR19","doi-asserted-by":"publisher","first-page":"1470","DOI":"10.1261\/rna.1070208","volume":"14","author":"Y Xing","year":"2008","unstructured":"Xing Y, Stoilov P, Kapur K, Han A, Jiang H, Shen S, Black DL, Wong WH: MADS: a new and improved method for analysis of differential alternative splicing by exon-tiling microarrays. RNA. 2008, 14 (8): 1470-1479.","journal-title":"RNA"},{"key":"2754_CR20","first-page":"402","volume-title":"In Proceedings of the 27th International Conference on Software Engineering (ICSE05)","author":"JH Andrews","year":"2005","unstructured":"Andrews JH, Briand LC, Labiche Y: Is mutation an appropriate tool for testing experiments?. In Proceedings of the 27th International Conference on Software Engineering (ICSE05). 2005, 402-411."},{"key":"2754_CR21","doi-asserted-by":"publisher","first-page":"152","DOI":"10.1109\/WST.1988.5370","volume-title":"In Proceedings of the 2nd Workshop on Software Testing, Verification and Analysis","author":"MR Woodward","year":"1988","unstructured":"Woodward MR, Halewood K: From weak to strong, deal or alive? An analysis of some mutation testing issues. In Proceedings of the 2nd Workshop on Software Testing, Verification and Analysis. 1988, 152-158."},{"key":"2754_CR22","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1145\/566172.566202","volume-title":"Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2002)","author":"TY Chen","year":"2002","unstructured":"Chen TY, Tse TH, Zhou ZQ: Semi-proving: an integrated method based on global symbolic evaluation and metamorphic testing. Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2002). 2002, ACM Press, New York, 191-195."},{"key":"2754_CR23","first-page":"267","volume-title":"In Proceedings of the 3rd IFIP Central and Eastern European Conference on Software Engineering Techniques (CEE-SET 08)","author":"AC Barus","year":"2008","unstructured":"Barus AC, Chen TY, Grant D, Kuo FC, Lau MF: Testing of Heuristic Methods: A Case Study of Greedy Algorithm. In Proceedings of the 3rd IFIP Central and Eastern European Conference on Software Engineering Techniques (CEE-SET 08). 2008, 267-280."},{"key":"2754_CR24","first-page":"867","volume-title":"In Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE08)","author":"C Murphy","year":"2008","unstructured":"Murphy C, Kaiser G, Hu L, Wu L: Properties of Machine Learning Applications for Use in Metamorphic Testing. In Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE08). 2008, 867-872."},{"key":"2754_CR25","first-page":"569","volume-title":"Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC 2004)","author":"TY Chen","year":"2004","unstructured":"Chen TY, Huang DH, Tse TH, Zhou ZQ: Case studies on the selection of useful relations in metamorphic testing. Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC 2004). 2004, Madrid, Spain: Polytechnic University of Madrid, 569-583."},{"key":"2754_CR26","volume-title":"Inferring Phylogenies","author":"J Felsenstein","year":"2004","unstructured":"Felsenstein J: Inferring Phylogenies. 2004, Sinauer Associates"},{"key":"2754_CR27","doi-asserted-by":"crossref","first-page":"Article3","DOI":"10.2202\/1544-6115.1027","volume":"3","author":"GK Smyth","year":"2004","unstructured":"Smyth GK: Linear Models and Empirical Bayes Methods for Assessing Differential Expression in Microarray Experiments. Stat Appl Genet Mol Biol. 2004, 3: Article3-","journal-title":"Stat Appl Genet Mol Biol"},{"issue":"13","key":"2754_CR28","doi-asserted-by":"publisher","first-page":"i390","DOI":"10.1093\/bioinformatics\/btn142","volume":"24","author":"JWK Ho","year":"2008","unstructured":"Ho JWK, Stefani M, dos Remedios CG, Charleston MA: Differential variability analysis of gene expression and its application to human diseases. Bioinformatics. 2008, 24 (13): i390-i398.","journal-title":"Bioinformatics"},{"issue":"25","key":"2754_CR29","doi-asserted-by":"publisher","first-page":"14863","DOI":"10.1073\/pnas.95.25.14863","volume":"95","author":"MB Eisen","year":"1998","unstructured":"Eisen MB, Spellman PT, Browndagger PO, Botstein D: Cluster analysis and display of genome-wide expression patterns. Proc Natl Acad Sci U S A. 1998, 95 (25): 14863-14868.","journal-title":"Proc Natl Acad Sci U S A"},{"issue":"43","key":"2754_CR30","doi-asserted-by":"publisher","first-page":"15545","DOI":"10.1073\/pnas.0506580102","volume":"102","author":"A Subramanian","year":"2005","unstructured":"Subramanian A, Tamayo P, Mootha VK, Mukherjee S, Ebert BL, Gillette MA, Paulovich A, Pomeroy SL, Golub TR, Lander ES, Mesirov JP: Gene set enrichment analysis: A knowledge-based approach for interpreting genome-wide expression profiles. Proc Natl Acad Sci U S A. 2005, 102 (43): 15545-15550.","journal-title":"Proc Natl Acad Sci U S A"},{"issue":"3-4","key":"2754_CR31","doi-asserted-by":"publisher","first-page":"601","DOI":"10.1089\/106652700750050961","volume":"7","author":"N Friedman","year":"2000","unstructured":"Friedman N, Linial M, Nachman I, Pe'er D: Using Bayesian Networks to Analyze Expression Data. J Comput Biol. 2000, 7 (3-4): 601-620.","journal-title":"J Comput Biol"},{"issue":"1","key":"2754_CR32","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1093\/nar\/27.1.29","volume":"27","author":"H Ogata","year":"1999","unstructured":"Ogata H, Goto S, Sato K, Fujibuchi W, Bono H, Kanehisa M: KEGG: Kyoto Encyclopedia of Genes and Genomes. Nucleic Acids Res. 1999, 27 (1): 29-34.","journal-title":"Nucleic Acids Res"},{"key":"2754_CR33","first-page":"D334","volume":"33 (Database is","author":"I Keseler","year":"2005","unstructured":"Keseler I, Collado-Vides J, Gama-Castro S, Ingraham J, Paley S, Saulsen I, Peralta-Gil M, Karp P: EcoCyc: a comprehensive database resource for Escherichia coli. Nucleic Acids Res. 2005, 33 (Database issue): D334-D337.","journal-title":"Nucleic Acids Res"},{"key":"2754_CR34","doi-asserted-by":"publisher","first-page":"D449","DOI":"10.1093\/nar\/gkh086","volume":"32 (Database is","author":"L Salwinski","year":"2004","unstructured":"Salwinski L, Miller CS, Smith AJ, Pettit FK, Bowie JU, Eisenberg D: The Database of Interacting Proteins: 2004 update. Nucleic Acids Res. 2004, 32 (Database issue): D449-D451.","journal-title":"Nucleic Acids Res"},{"key":"2754_CR35","first-page":"iv","volume":"40","author":"AV Hill","year":"1910","unstructured":"Hill AV: The possible effects of the aggregation of the molecules of haemoglobin on its dissociation curves. J Physiol. 1910, 40: iv-vii.","journal-title":"J Physiol"},{"issue":"4","key":"2754_CR36","first-page":"377","volume":"13","author":"JHS Hofmeyr","year":"1997","unstructured":"Hofmeyr JHS, Cornish-Bowden A: The reversible hill equation: how to incorporate cooperative enzymes into metabolic models. Comput Appl Biosci. 1997, 13 (4): 377-385.","journal-title":"Comput Appl Biosci"},{"key":"2754_CR37","doi-asserted-by":"crossref","first-page":"ii122","DOI":"10.1093\/bioinformatics\/btg1069","volume":"19 Suppl 2","author":"P Mendes","year":"2003","unstructured":"Mendes P, Sha W, Ye K: Artificial gene networks for objective comparison of analysis algorithms. Bioinformatics. 2003, 19 Suppl 2: ii122-ii129.","journal-title":"Bioinformatics"}],"container-title":["BMC Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/1471-2105-10-24.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,23]],"date-time":"2023-05-23T17:18:32Z","timestamp":1684862312000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/1471-2105-10-24"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2009,1,19]]},"references-count":37,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2009,12]]}},"alternative-id":["2754"],"URL":"https:\/\/doi.org\/10.1186\/1471-2105-10-24","relation":{},"ISSN":["1471-2105"],"issn-type":[{"value":"1471-2105","type":"electronic"}],"subject":[],"published":{"date-parts":[[2009,1,19]]},"assertion":[{"value":"29 May 2008","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 January 2009","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 January 2009","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"24"}}