{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,11]],"date-time":"2025-11-11T12:58:55Z","timestamp":1762865935782},"reference-count":39,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2016,10,2]],"date-time":"2016-10-02T00:00:00Z","timestamp":1475366400000},"content-version":"vor","delay-in-days":1663,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/3.0"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2012,5,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Motivation: Analyses and algorithmic predictions based on high-throughput data are essential for the success of systems biology in academic and industrial settings. Organizations, such as companies and academic consortia, conduct large multi-year scientific studies that entail the collection and analysis of thousands of individual experiments, often over many physical sites and with internal and outsourced components. To extract maximum value, the interested parties need to verify the accuracy and reproducibility of data and methods before the initiation of such large multi-year studies. However, systematic and well-established verification procedures do not exist for automated collection and analysis workflows in systems biology which could lead to inaccurate conclusions.<\/jats:p>\n               <jats:p>Results: We present here, a review of the current state of systems biology verification and a detailed methodology to address its shortcomings. This methodology named \u2018Industrial Methodology for Process Verification in Research\u2019 or IMPROVER, consists on evaluating a research program by dividing a workflow into smaller building blocks that are individually verified. The verification of each building block can be done internally by members of the research program or externally by \u2018crowd-sourcing\u2019 to an interested community. www.sbvimprover.com<\/jats:p>\n               <jats:p>Implementation: This methodology could become the preferred choice to verify systems biology research workflows that are becoming increasingly complex and sophisticated in industrial and academic settings.<\/jats:p>\n               <jats:p>Contact: \u00a0gustavo@us.ibm.com<\/jats:p>","DOI":"10.1093\/bioinformatics\/bts116","type":"journal-article","created":{"date-parts":[[2012,3,16]],"date-time":"2012-03-16T00:38:48Z","timestamp":1331858328000},"page":"1193-1201","source":"Crossref","is-referenced-by-count":47,"title":["Industrial methodology for process verification in research (IMPROVER): toward systems biology verification"],"prefix":"10.1093","volume":"28","author":[{"given":"Pablo","family":"Meyer","sequence":"first","affiliation":[{"name":"1 IBM Computational Biology Center, Yorktown Heights, 10598 NY, USA, 2Phillip Morris Products SA, Research and Development, 2000, Neuch\u00e2tel, Switzerland and 3IBM Life Sciences Division,8802, Zurich, Switzerland"}]},{"given":"Julia","family":"Hoeng","sequence":"additional","affiliation":[{"name":"1 IBM Computational Biology Center, Yorktown Heights, 10598 NY, USA, 2Phillip Morris Products SA, Research and Development, 2000, Neuch\u00e2tel, Switzerland and 3IBM Life Sciences Division,8802, Zurich, Switzerland"}]},{"given":"J. Jeremy","family":"Rice","sequence":"additional","affiliation":[{"name":"1 IBM Computational Biology Center, Yorktown Heights, 10598 NY, USA, 2Phillip Morris Products SA, Research and Development, 2000, Neuch\u00e2tel, Switzerland and 3IBM Life Sciences Division,8802, Zurich, Switzerland"}]},{"given":"Raquel","family":"Norel","sequence":"additional","affiliation":[{"name":"1 IBM Computational Biology Center, Yorktown Heights, 10598 NY, USA, 2Phillip Morris Products SA, Research and Development, 2000, Neuch\u00e2tel, Switzerland and 3IBM Life Sciences Division,8802, Zurich, Switzerland"}]},{"given":"J\u00f6rg","family":"Sprengel","sequence":"additional","affiliation":[{"name":"1 IBM Computational Biology Center, Yorktown Heights, 10598 NY, USA, 2Phillip Morris Products SA, Research and Development, 2000, Neuch\u00e2tel, Switzerland and 3IBM Life Sciences Division,8802, Zurich, Switzerland"}]},{"given":"Katrin","family":"Stolle","sequence":"additional","affiliation":[{"name":"1 IBM Computational Biology Center, Yorktown Heights, 10598 NY, USA, 2Phillip Morris Products SA, Research and Development, 2000, Neuch\u00e2tel, Switzerland and 3IBM Life Sciences Division,8802, Zurich, Switzerland"}]},{"given":"Thomas","family":"Bonk","sequence":"additional","affiliation":[{"name":"1 IBM Computational Biology Center, Yorktown Heights, 10598 NY, USA, 2Phillip Morris Products SA, Research and Development, 2000, Neuch\u00e2tel, Switzerland and 3IBM Life Sciences Division,8802, Zurich, Switzerland"}]},{"given":"Stephanie","family":"Corthesy","sequence":"additional","affiliation":[{"name":"1 IBM Computational Biology Center, Yorktown Heights, 10598 NY, USA, 2Phillip Morris Products SA, Research and Development, 2000, Neuch\u00e2tel, Switzerland and 3IBM Life Sciences Division,8802, Zurich, Switzerland"}]},{"given":"Ajay","family":"Royyuru","sequence":"additional","affiliation":[{"name":"1 IBM Computational Biology Center, Yorktown Heights, 10598 NY, USA, 2Phillip Morris Products SA, Research and Development, 2000, Neuch\u00e2tel, Switzerland and 3IBM Life Sciences Division,8802, Zurich, Switzerland"}]},{"given":"Manuel C.","family":"Peitsch","sequence":"additional","affiliation":[{"name":"1 IBM Computational Biology Center, Yorktown Heights, 10598 NY, USA, 2Phillip Morris Products SA, Research and Development, 2000, Neuch\u00e2tel, Switzerland and 3IBM Life Sciences Division,8802, Zurich, Switzerland"}]},{"given":"Gustavo","family":"Stolovitzky","sequence":"additional","affiliation":[{"name":"1 IBM Computational Biology Center, Yorktown Heights, 10598 NY, USA, 2Phillip Morris Products SA, Research and Development, 2000, Neuch\u00e2tel, Switzerland and 3IBM Life Sciences Division,8802, Zurich, Switzerland"}]}],"member":"286","published-online":{"date-parts":[[2012,3,14]]},"reference":[{"key":"2023012512230323700_B1"},{"key":"2023012512230323700_B2","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1126\/science.1162115","article-title":"Reviewing peer review","volume":"321","author":"Alberts","year":"2008","journal-title":"Science"},{"key":"2023012512230323700_B3","volume-title":"Language, Truth, and Logic.","author":"Ayer","year":"1936"},{"issue":"Suppl. 9","key":"2023012512230323700_B4","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1002\/prot.22591","article-title":"Assessment of CASP8 structure predictions for template free targets","volume":"77","author":"Ben-David","year":"2009","journal-title":"Proteins"},{"key":"2023012512230323700_B5","doi-asserted-by":"crossref","first-page":"1502","DOI":"10.1016\/S0140-6736(08)61620-7","article-title":"Multiple sclerosis","volume":"372","author":"Compston","year":"2008","journal-title":"Lancet"},{"key":"2023012512230323700_B6","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1145\/1143844.1143874","article-title":"The relationship between Precision-Recall and ROC curves","volume-title":"Proceedings of the 23rd International Conference on Machine learning.","author":"Davis","year":"2006"},{"key":"2023012512230323700_B7","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1093\/bib\/bbq078","article-title":"Validation of gene regulatory networks: scientific and inferential","volume":"12","author":"Dougherty","year":"2011","journal-title":"Brief Bioinform."},{"key":"2023012512230323700_B8","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1016\/S0076-6879(10)70012-4","article-title":"High-quality binary interactome mapping","volume":"470","author":"Dreze","year":"2010","journal-title":"Methods Enzymol."},{"key":"2023012512230323700_B9","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1145\/240455.240464","article-title":"The KDD process for extracting useful knowledge from volumes of data","volume":"39","author":"Fayyad","year":"1996","journal-title":"Commun. ACM"},{"key":"2023012512230323700_B10","doi-asserted-by":"crossref","first-page":"631","DOI":"10.1038\/nature04532","article-title":"Proteome survey reveals modularity of the yeast cell machinery","volume":"440","author":"Gavin","year":"2006","journal-title":"Nature"},{"issue":"Suppl. 1","key":"2023012512230323700_B11","doi-asserted-by":"crossref","first-page":"S1","DOI":"10.1186\/1471-2105-6-S1-S1","article-title":"Overview of BioCreAtIvE: critical assessment of information extraction for biology","volume":"6","author":"Hirschman","year":"2005","journal-title":"BMC Bioinformatics"},{"key":"2023012512230323700_B12","doi-asserted-by":"crossref","first-page":"454","DOI":"10.1016\/S0140-6736(05)17878-7","article-title":"Microarrays and molecular research: noise discovery?","volume":"365","author":"Ioannidis","year":"2005","journal-title":"Lancet"},{"key":"2023012512230323700_B13","doi-asserted-by":"crossref","first-page":"4569","DOI":"10.1073\/pnas.061034498","article-title":"A comprehensive two-hybrid analysis to explore the yeast protein interactome. In","volume":"98","author":"Ito","year":"2001","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"2023012512230323700_B14","doi-asserted-by":"crossref","first-page":"1990","DOI":"10.1093\/bioinformatics\/btq323","article-title":"Over-optimism in bioinformatics: an illustration","volume":"26","author":"Jelizarow","year":"2010","journal-title":"Bioinformatics"},{"key":"2023012512230323700_B15","volume-title":"The structure of scientific revolutions.","author":"Kuhn","year":"1962"},{"key":"2023012512230323700_B16","first-page":"504","article-title":"Master regulators used as breast cancer metastasis classifier","author":"Lim","year":"2009","journal-title":"Pac. Symp. Biocomput."},{"key":"2023012512230323700_B17","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1038\/469286a","article-title":"Peer review: trial by Twitter","volume":"469","author":"Mandavilli","year":"2011","journal-title":"Nature"},{"key":"2023012512230323700_B18","doi-asserted-by":"crossref","first-page":"6286","DOI":"10.1073\/pnas.0913357107","article-title":"Revealing strengths and weaknesses of methods for gene network inference","volume":"107","author":"Marbach","year":"2010","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023012512230323700_B19","doi-asserted-by":"crossref","first-page":"943","DOI":"10.1038\/ng1422","article-title":"Towards sound epistemological foundations of statistical methods for high-dimensional biology","volume":"36","author":"Mehta","year":"2004","journal-title":"Nat. Genet."},{"key":"2023012512230323700_B20","doi-asserted-by":"crossref","first-page":"811","DOI":"10.1038\/nbt.1968","article-title":"Verification of systems biology research in the age of collaborative competition","volume":"29","author":"Meyer","year":"2011","journal-title":"Nat. Biotech."},{"key":"2023012512230323700_B21","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1179\/016164106X98080","article-title":"Multiple sclerosis as a vascular disease","volume":"28","author":"Minagar","year":"2006","journal-title":"Neurol. Res."},{"issue":"Suppl 2","key":"2023012512230323700_B22","doi-asserted-by":"crossref","first-page":"S3","DOI":"10.1186\/gb-2008-9-s2-s3","article-title":"Overview of BioCreative II gene normalization","volume":"9","author":"Morgan","year":"2008","journal-title":"Genome Biol."},{"key":"2023012512230323700_B23","doi-asserted-by":"crossref","first-page":"ii","DOI":"10.1002\/prot.340230303","article-title":"A large-scale experiment to assess protein structure prediction methods","volume":"23","author":"Moult","year":"1995","journal-title":"Prot. Struct. Func. Bioinform."},{"key":"2023012512230323700_B24","doi-asserted-by":"crossref","first-page":"422","DOI":"10.1016\/S0958-1669(96)80118-2","article-title":"The current state of the art in protein structure prediction","volume":"7","author":"Moult","year":"1996","journal-title":"Current Opinion in Biotechnology"},{"key":"2023012512230323700_B25","doi-asserted-by":"crossref","first-page":"537","DOI":"10.1038\/msb.2011.70","article-title":"The self-assessment trap: can we all be better than average?","volume":"7","author":"Norel","year":"2011","journal-title":"Mol. Syst. Biol."},{"key":"2023012512230323700_B26","volume-title":"OECD Good Laboratory Practice - Principles and Guidance for Compliance Monitoring","author":"Organisation for Economic Cooperation and Development","year":"1998"},{"key":"2023012512230323700_B27","doi-asserted-by":"crossref","first-page":"2817","DOI":"10.1056\/NEJMoa041588","article-title":"A multigene assay to predict recurrence of tamoxifen-treated, node-negative breast cancer","volume":"351","author":"Paik","year":"2004","journal-title":"New Engl. J. Med."},{"key":"2023012512230323700_B28","volume-title":"The Logic of Scientific Discovery.","author":"Popper","year":"1959"},{"key":"2023012512230323700_B29","doi-asserted-by":"crossref","first-page":"e9202","DOI":"10.1371\/journal.pone.0009202","article-title":"Towards a rigorous assessment of systems biology models: the DREAM3 challenges","volume":"5","author":"Prill","year":"2010","journal-title":"PLoS One"},{"key":"2023012512230323700_B30","first-page":"mr7","article-title":"Crowdsourcing network inference: the DREAM predictive signaling network challenge","volume":"4","author":"Prill","year":"2011","journal-title":"Sci. Signal."},{"key":"2023012512230323700_B31","doi-asserted-by":"crossref","first-page":"559","DOI":"10.1086\/519795","article-title":"PLINK: a tool set for whole-genome association and population-based linkage analyses","volume":"81","author":"Purcell","year":"2007","journal-title":"Am. J. Hum. Genet."},{"key":"2023012512230323700_B32","doi-asserted-by":"crossref","first-page":"911","DOI":"10.1056\/NEJMoa044396","article-title":"Natalizumab plus interferon beta-1a for relapsing multiple sclerosis","volume":"354","author":"Rudick","year":"2006","journal-title":"New Engl. J. Med."},{"key":"2023012512230323700_B33","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1016\/S0167-7799(02)01985-6","article-title":"The history of the peer-review process","volume":"20","author":"Spier","year":"2002","journal-title":"Trends Biotechnol."},{"key":"2023012512230323700_B34","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1196\/annals.1407.021","article-title":"Dialogue on reverse-engineering assessment and methods","volume":"1115","author":"Stolovitzky","year":"2007","journal-title":"Ann. NY. Acad. Sci."},{"key":"2023012512230323700_B35","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1111\/j.1749-6632.2009.04497.x","article-title":"Lessons from the DREAM2 challenges","volume":"1158","author":"Stolovitzky","year":"2009","journal-title":"Ann.NY. Acad. Sci."},{"key":"2023012512230323700_B36","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1038\/nrclinonc.2010.60","article-title":"What should physicians look for in evaluating prognostic gene-expression signatures?","volume":"7","author":"Subramanian","year":"2010","journal-title":"Nat. Rev. Clin. Oncol."},{"key":"2023012512230323700_B37","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1016\/S1369-5274(00)00094-1","article-title":"Systematic and large-scale two-hybrid screens","volume":"3","author":"Uetz","year":"2000","journal-title":"Curr. Opin. Microbiol."},{"key":"2023012512230323700_B38","doi-asserted-by":"crossref","first-page":"1999","DOI":"10.1056\/NEJMoa021967","article-title":"A gene-expression signature as a predictor of survival in breast cancer","volume":"347","author":"van de Vijver","year":"2002","journal-title":"New Engl. J. Med."},{"key":"2023012512230323700_B39","doi-asserted-by":"crossref","first-page":"242","DOI":"10.1016\/j.sbi.2004.02.003","article-title":"Prediction of protein-protein interactions: the CAPRI experiment, its evaluation and implications","volume":"14","author":"Wodak","year":"2004","journal-title":"Curr. Opin. Struct. Biol."}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/28\/9\/1193\/48875202\/bioinformatics_28_9_1193.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/28\/9\/1193\/48875202\/bioinformatics_28_9_1193.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,25]],"date-time":"2023-01-25T15:52:58Z","timestamp":1674661978000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/28\/9\/1193\/311774"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,3,14]]},"references-count":39,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2012,5,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/bts116","relation":{},"ISSN":["1367-4811","1367-4803"],"issn-type":[{"value":"1367-4811","type":"electronic"},{"value":"1367-4803","type":"print"}],"subject":[],"published-other":{"date-parts":[[2012,5,1]]},"published":{"date-parts":[[2012,3,14]]}}}