{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,27]],"date-time":"2025-09-27T10:43:48Z","timestamp":1758969828031},"reference-count":62,"publisher":"Oxford University Press (OUP)","issue":"22","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2013,11,15]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Motivation: Validation and reproducibility of results is a central and pressing issue in genomics. Several recent embarrassing incidents involving the irreproducibility of high-profile studies have illustrated the importance of this issue and the need for rigorous methods for the assessment of reproducibility.<\/jats:p><jats:p>Results: Here, we describe an existing statistical model that is very well suited to this problem. We explain its utility for assessing the reproducibility of validation experiments, and apply it to a genome-scale study of adenosine deaminase acting on RNA (ADAR)-mediated RNA editing in Drosophila. We also introduce a statistical method for planning validation experiments that will obtain the tightest reproducibility confidence limits, which, for a fixed total number of experiments, returns the optimal number of replicates for the study.<\/jats:p><jats:p>Availability: Downloadable software and a web service for both the analysis of data from a reproducibility study and for the optimal design of these studies is provided at http:\/\/ccmbweb.ccv.brown.edu\/reproducibility.html<\/jats:p><jats:p>Contact: Charles_Lawrence@Brown.edu<\/jats:p><jats:p>Supplementary information: \u00a0Supplementary data are available at Bioinformatics online.<\/jats:p>","DOI":"10.1093\/bioinformatics\/btt508","type":"journal-article","created":{"date-parts":[[2013,9,19]],"date-time":"2013-09-19T01:05:55Z","timestamp":1379552755000},"page":"2844-2851","source":"Crossref","is-referenced-by-count":7,"title":["Assessing the validity and reproducibility of genome-scale predictions"],"prefix":"10.1093","volume":"29","author":[{"given":"Lauren A.","family":"Sugden","sequence":"first","affiliation":[{"name":"1 Center for Computational Molecular Biology and the Division of Applied Mathematics, Brown University, Providence, RI 02912, USA, 2St. Laurent Institute, 317 New Boston St, Woburn, MA 01801, USA and 3Department of Molecular Biology, Cell Biology and Biochemistry, Brown University, Providence, RI 02912, USA"}]},{"given":"Michael R.","family":"Tackett","sequence":"additional","affiliation":[{"name":"1 Center for Computational Molecular Biology and the Division of Applied Mathematics, Brown University, Providence, RI 02912, USA, 2St. Laurent Institute, 317 New Boston St, Woburn, MA 01801, USA and 3Department of Molecular Biology, Cell Biology and Biochemistry, Brown University, Providence, RI 02912, USA"}]},{"given":"Yiannis A.","family":"Savva","sequence":"additional","affiliation":[{"name":"1 Center for Computational Molecular Biology and the Division of Applied Mathematics, Brown University, Providence, RI 02912, USA, 2St. Laurent Institute, 317 New Boston St, Woburn, MA 01801, USA and 3Department of Molecular Biology, Cell Biology and Biochemistry, Brown University, Providence, RI 02912, USA"}]},{"given":"William A.","family":"Thompson","sequence":"additional","affiliation":[{"name":"1 Center for Computational Molecular Biology and the Division of Applied Mathematics, Brown University, Providence, RI 02912, USA, 2St. Laurent Institute, 317 New Boston St, Woburn, MA 01801, USA and 3Department of Molecular Biology, Cell Biology and Biochemistry, Brown University, Providence, RI 02912, USA"}]},{"given":"Charles E.","family":"Lawrence","sequence":"additional","affiliation":[{"name":"1 Center for Computational Molecular Biology and the Division of Applied Mathematics, Brown University, Providence, RI 02912, USA, 2St. Laurent Institute, 317 New Boston St, Woburn, MA 01801, USA and 3Department of Molecular Biology, Cell Biology and Biochemistry, Brown University, Providence, RI 02912, USA"}]}],"member":"286","published-online":{"date-parts":[[2013,9,17]]},"reference":[{"key":"2023012810473963400_btt508-B1","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1534\/genetics.110.114983","article-title":"Statistical design and analysis of RNA sequencing data","volume":"185","author":"Auer","year":"2010","journal-title":"Genetics"},{"key":"2023012810473963400_btt508-B2","doi-asserted-by":"crossref","first-page":"1477","DOI":"10.1093\/bioinformatics\/btg173","article-title":"Differential expression in SAGE: accounting for normal between-library variation","volume":"19","author":"Baggerly","year":"2003","journal-title":"Bioinformatics"},{"key":"2023012810473963400_btt508-B3","doi-asserted-by":"crossref","first-page":"955","DOI":"10.1038\/ng.175","article-title":"Genome-wide association defines more than 30 distinct susceptibility loci for Crohn\u2019s disease","volume":"40","author":"Barrett","year":"2008","journal-title":"Nat. Genet."},{"key":"2023012810473963400_btt508-B4","doi-asserted-by":"crossref","first-page":"735","DOI":"10.1177\/1087057110374994","article-title":"Factors affecting reproducibility between genome-scale siRNA-based screens","volume":"15","author":"Barrows","year":"2010","journal-title":"J. Biomol. Screen."},{"key":"2023012810473963400_btt508-B5","doi-asserted-by":"crossref","first-page":"1089","DOI":"10.1016\/0092-8674(88)90253-X","article-title":"An unwinding activity that covalently modifies its double-stranded RNA substrate","volume":"55","author":"Bass","year":"1988","journal-title":"Cell"},{"key":"2023012810473963400_btt508-B6","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1038\/483531a","article-title":"Drug development: raise standards for preclinical cancer research","volume":"483","author":"Begley","year":"2012","journal-title":"Nature"},{"key":"2023012810473963400_btt508-B7","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1038\/nmeth.1333","article-title":"A HUPO test sample study reveals common problems in mass spectrometry-based proteomics","volume":"6","author":"Bell","year":"2009","journal-title":"Nat. Meth."},{"key":"2023012810473963400_btt508-B8","doi-asserted-by":"crossref","first-page":"1609","DOI":"10.1093\/bioinformatics\/18.12.1609","article-title":"Calculation of the minimum number of replicate spots required for detection of significant gene expression fold change in microarray experiments","volume":"18","author":"Black","year":"2002","journal-title":"Bioinformatics"},{"key":"2023012810473963400_btt508-B9","doi-asserted-by":"crossref","first-page":"556","DOI":"10.1093\/bib\/bbp034","article-title":"Stability and aggregation of ranked gene lists","volume":"10","author":"Boulesteix","year":"2009","journal-title":"Brief. Bioinform."},{"key":"2023012810473963400_btt508-B10","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1038\/nrn3475","article-title":"Power failure: why small sample size undermines the reliability of neuroscience","volume":"14","author":"Button","year":"2013","journal-title":"Nat. Rev. Neurosci."},{"key":"2023012810473963400_btt508-B11","doi-asserted-by":"crossref","first-page":"e1002600","DOI":"10.1371\/journal.pgen.1002600","article-title":"Critical evaluation of imprinted gene expression by RNA-seq: a new perspective","volume":"8","author":"DeVeale","year":"2012","journal-title":"PLoS Genet."},{"key":"2023012810473963400_btt508-B12","doi-asserted-by":"crossref","first-page":"806","DOI":"10.1038\/nbt.2335","article-title":"Further confirmation needed","volume":"30","author":"Editorial","year":"2012","journal-title":"Nat. Biotechnol."},{"key":"2023012810473963400_btt508-B13","doi-asserted-by":"crossref","first-page":"406","DOI":"10.1038\/487406a","article-title":"Error prone: biologists must realize the piffalls of work on massive amounts of data","volume":"487","author":"Editorial","year":"2012","journal-title":"Nature"},{"key":"2023012810473963400_btt508-B14","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1201\/9780429258480","article-title":"Hierarchical models","volume-title":"Bayesian Data Analysis","author":"Gelman","year":"2003","edition":"2nd edn"},{"key":"2023012810473963400_btt508-B15","doi-asserted-by":"crossref","first-page":"1721","DOI":"10.1093\/bioinformatics\/bts260","article-title":"Identifying differentially expressed ranscripts from RNA-seq data with biological variation","volume":"28","author":"Glaus","year":"2012","journal-title":"Bioinformatics"},{"key":"2023012810473963400_btt508-B16","doi-asserted-by":"crossref","first-page":"643","DOI":"10.1126\/science.1190830","article-title":"High-resolution analysis of parent-of-origin allelic expression in the mouse brain","volume":"329","author":"Gregg","year":"2010","journal-title":"Science"},{"key":"2023012810473963400_btt508-B17","doi-asserted-by":"crossref","first-page":"832","DOI":"10.1126\/science.1086763","article-title":"Nervous system targets of RNA editing identified by comparative genomics","volume":"301","author":"Hoopengardner","year":"2003","journal-title":"Science"},{"key":"2023012810473963400_btt508-B18","doi-asserted-by":"crossref","first-page":"182","DOI":"10.1101\/gr.112466.110","article-title":"Genome-wide analysis of promoter architecture in Drosophila melanogaster","volume":"21","author":"Hoskins","year":"2011","journal-title":"Genome Res."},{"key":"2023012810473963400_btt508-B19","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1186\/jbiol104","article-title":"\u2018Validation\u2019 in genome-scale research","volume":"8","author":"Hughes","year":"2009","journal-title":"J. Biol."},{"key":"2023012810473963400_btt508-B20","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1038\/ng.1037","article-title":"Rare and functional SIAE variants are not associated with autoimmune disease risk in up to 66,924 individuals of European ancestry","volume":"44","author":"Hunt","year":"2012","journal-title":"Nat. Genet."},{"key":"2023012810473963400_btt508-B21","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1038\/ng.295","article-title":"Repeatability of published microarray gene expression analyses","volume":"41","author":"Ioannidis","year":"2009","journal-title":"Nat. Genet."},{"key":"2023012810473963400_btt508-B22","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1038\/nmeth.1827","article-title":"Visualizing adenosine-to-inosine RNA editing in the drosophila nervous system","volume":"9","author":"Jepson","year":"2012","journal-title":"Nat. Meth."},{"key":"2023012810473963400_btt508-B23","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1038\/nbt.1619","article-title":"Analyzing omics data using hierarchical models","volume":"28","author":"Ji","year":"2010","journal-title":"Nat. Biotech."},{"key":"2023012810473963400_btt508-B24","doi-asserted-by":"crossref","first-page":"8961","DOI":"10.1073\/pnas.161273698","article-title":"Bootstrapping cluster analysis: assessing the reliability of conclusions from microarray experiments","volume":"98","author":"Kerr","year":"2001","journal-title":"PNAS"},{"key":"2023012810473963400_btt508-B25","doi-asserted-by":"crossref","first-page":"1302","DOI":"10.1126\/science.1209658","article-title":"Comment on Widespread RNA and DNA Sequence Differences in the Human Transcriptome","volume":"335","author":"Kleinman","year":"2012","journal-title":"Science"},{"key":"2023012810473963400_btt508-B26","doi-asserted-by":"crossref","first-page":"832","DOI":"10.1038\/nbt1217","article-title":"A sequence-oriented comparison of gene expression measurements across different hybridization-based technologies","volume":"24","author":"Kuo","year":"2006","journal-title":"Nat. Biotechnol."},{"key":"2023012810473963400_btt508-B27","doi-asserted-by":"crossref","first-page":"733","DOI":"10.1038\/nrg2825","article-title":"Tackling the widespread and critical impact of batch effects in high-throughput data","volume":"11","author":"Leek","year":"2010","journal-title":"Nat. Rev. Genet."},{"key":"2023012810473963400_btt508-B28","doi-asserted-by":"crossref","first-page":"1752","DOI":"10.1214\/11-AOAS466","article-title":"Measuring reproducibility of high-throughput experiments","volume":"5","author":"Li","year":"2011","journal-title":"Ann. Appl. Stat."},{"key":"2023012810473963400_btt508-B29","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1126\/science.1207018","article-title":"Widespread RNA and DNA sequence differences in the Human Transcriptome","volume":"333","author":"Li","year":"2011","journal-title":"Science"},{"key":"2023012810473963400_btt508-B30","doi-asserted-by":"crossref","first-page":"e27","DOI":"10.1371\/journal.pbio.0060027","article-title":"Transcription factors bind thousands of active and inactive regions in the Drosophila blastoderm","volume":"6","author":"Li","year":"2008","journal-title":"PLoS Biol."},{"key":"2023012810473963400_btt508-B31","doi-asserted-by":"crossref","first-page":"1302","DOI":"10.1126\/science.1210624","article-title":"Comment on widespread RNA and DNA sequence differences in the human transcriptome","volume":"335","author":"Lin","year":"2012","journal-title":"Science"},{"key":"2023012810473963400_btt508-B32","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1038\/487427a","article-title":"Face up to false positives","volume":"487","author":"MacArthur","year":"2012","journal-title":"Nature"},{"key":"2023012810473963400_btt508-B33","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1038\/477511a","article-title":"Why animal research needs to improve","volume":"477","author":"Macleod","year":"2011","journal-title":"Nature"},{"key":"2023012810473963400_btt508-B34","doi-asserted-by":"crossref","first-page":"1151","DOI":"10.1038\/nbt1239","article-title":"The microarray quality control (MAQC) project shows inter- and intraplatform reproducibility of gene expression measurements","volume":"24","author":"MAQC Consortium","year":"2006","journal-title":"Nat. Biotechnol."},{"key":"2023012810473963400_btt508-B35","doi-asserted-by":"crossref","first-page":"1462","DOI":"10.1093\/bioinformatics\/18.11.1462","article-title":"Methods for assessing reproducibility of clustering patterns observed in analysis of microarray data","volume":"18","author":"McShane","year":"2002","journal-title":"Bioinformatics"},{"key":"2023012810473963400_btt508-B36","doi-asserted-by":"crossref","first-page":"617","DOI":"10.1073\/pnas.0705554105","article-title":"Required sample size and nonreplicability thresholds for heterogeneous genetic associations","volume":"105","author":"Moonesinghe","year":"2008","journal-title":"PNAS"},{"key":"2023012810473963400_btt508-B37","doi-asserted-by":"crossref","first-page":"e1000814","DOI":"10.1371\/journal.pgen.1000814","article-title":"A comprehensive map of insulator elements for the Drosophila genome","volume":"6","author":"N\u00e8gre","year":"2010","journal-title":"PLoS Genet."},{"key":"2023012810473963400_btt508-B38","doi-asserted-by":"crossref","first-page":"3523","DOI":"10.1002\/j.1460-2075.1991.tb04916.x","article-title":"Substrate specificity of the dsRNA unwinding\/modifying activity","volume":"10","author":"Nishikura","year":"1991","journal-title":"EMBO J."},{"key":"2023012810473963400_btt508-B39","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1146\/annurev-biochem-060208-105251","article-title":"Functions and regulation of RNA editing by ADAR deaminases","volume":"79","author":"Nishikura","year":"2010","journal-title":"Annu. Rev. Biochem."},{"key":"2023012810473963400_btt508-B40","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1093\/biostatistics\/kxn036","article-title":"Optimal multistage designs \u2013 a general framework for efficient genome-wide association studies","volume":"10","author":"Pahl","year":"2009","journal-title":"Biostatistics"},{"key":"2023012810473963400_btt508-B41","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1016\/S0092-8674(00)00049-0","article-title":"A-to-I Pre-mRNA editing in Drosophila is primarily involved in adult nervous system function and integrity","volume":"102","author":"Palladino","year":"2000","journal-title":"Cell"},{"key":"2023012810473963400_btt508-B42","doi-asserted-by":"crossref","DOI":"10.1186\/gb-2002-3-5-research0022","article-title":"How many replicates of arrays are required to detect gene expression changes in microarray experiments? A mixture model approach","volume":"3","author":"Pan","year":"2002","journal-title":"Genome Biol."},{"key":"2023012810473963400_btt508-B43","doi-asserted-by":"crossref","first-page":"1302","DOI":"10.1126\/science.1210484","article-title":"Comment on widespread RNA and DNA sequence differences in the human transcriptome","volume":"335","author":"Pickrell","year":"2012","journal-title":"Science"},{"key":"2023012810473963400_btt508-B44","doi-asserted-by":"crossref","first-page":"712","DOI":"10.1038\/nrd3439-c1","article-title":"Believe it or not: how much can we rely on published data on potential drug targets? Nat","volume":"10","author":"Prinz","year":"2011","journal-title":"Rev. Drug Discov."},{"key":"2023012810473963400_btt508-B45","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1038\/496007a","article-title":"If a job is worth doing, it is worth doing twice","volume":"496","author":"Russell","year":"2013","journal-title":"Nature"},{"key":"2023012810473963400_btt508-B46","doi-asserted-by":"crossref","first-page":"e56515","DOI":"10.1371\/journal.pone.0056515","article-title":"Priming intelligent behavior: an elusive phenomenon","volume":"8","author":"Shanks","year":"2013","journal-title":"PLoS One"},{"key":"2023012810473963400_btt508-B62","doi-asserted-by":"crossref","DOI":"10.1038\/nsmb.2675","article-title":"Genome-wide analysis of A-to-I RNA editing via single molecule sequencing in Drosophila","author":"St. Laurent","year":"2013","journal-title":"Nat. Struct. Mol. Biol"},{"key":"2023012810473963400_btt508-B47","doi-asserted-by":"crossref","first-page":"1922","DOI":"10.1261\/rna.254306","article-title":"RNA editing in Drosophila melanogaster: new targets and functional consequences","volume":"12","author":"Stapleton","year":"2006","journal-title":"RNA"},{"key":"2023012810473963400_btt508-B48","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1111\/1467-9868.00346","article-title":"A direct approach to false discovery rates","volume":"64","author":"Storey","year":"2002","journal-title":"J. R. Stat. Soc. Ser. B Stat. Methodol."},{"key":"2023012810473963400_btt508-B49","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1038\/nature09115","article-title":"Functionally defective germline variants of sialic acid acetylesterase in autoimmunity","volume":"466","author":"Surolia","year":"2010","journal-title":"Nature"},{"key":"2023012810473963400_btt508-B50","doi-asserted-by":"crossref","first-page":"106","DOI":"10.1186\/1471-2105-7-106","article-title":"A simple method for assessing sample sizes in microarray experiments","volume":"7","author":"Tibshirani","year":"2005","journal-title":"BMC Bioinformatics"},{"key":"2023012810473963400_btt508-B51","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1186\/1471-2164-6-77","article-title":"A generally applicable validation scheme for the assessment of factors involved in reproducibility and quality of DNA-microarray data","volume":"6","author":"Van Hijum","year":"2005","journal-title":"BMC Genomics"},{"key":"2023012810473963400_btt508-B52","doi-asserted-by":"crossref","first-page":"180","DOI":"10.1038\/492180a","article-title":"Know when your numbers are significant","volume":"492","author":"Vaux","year":"2012","journal-title":"Nature"},{"key":"2023012810473963400_btt508-B53","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1186\/1471-2105-5-119","article-title":"Bayesian model accounting for within-class biological variability in Serial Analysis of Gene Expression (SAGE)","volume":"5","author":"V\u00eancio","year":"2004","journal-title":"BMC Bioinformatics"},{"key":"2023012810473963400_btt508-B54","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1186\/1471-2164-5-87","article-title":"Sample size for detecting differentially expressed genes in microarray experiments","volume":"5","author":"Wei","year":"2004","journal-title":"BMC Genomics"},{"key":"2023012810473963400_btt508-B55","doi-asserted-by":"crossref","first-page":"S15","DOI":"10.1186\/1752-0509-5-S2-S15","article-title":"Extended local similarity analysis (eLSA) of microbial community and other time series data with replicates","volume":"5","author":"Xia","year":"2011","journal-title":"BMC Syst. Biol."},{"key":"2023012810473963400_btt508-B56","doi-asserted-by":"crossref","first-page":"693","DOI":"10.1142\/S0219720006002120","article-title":"Similarities of ordered gene lists","volume":"4","author":"Yang","year":"2006","journal-title":"J. Bioinform. Comput. Biol."},{"key":"2023012810473963400_btt508-B57","doi-asserted-by":"crossref","first-page":"298","DOI":"10.1038\/485298a","article-title":"Replication studies: Bad copy","volume":"485","author":"Yong","year":"2012","journal-title":"Nature"},{"key":"2023012810473963400_btt508-B58","doi-asserted-by":"crossref","first-page":"191","DOI":"10.2217\/14622416.10.2.191","article-title":"Meta-analysis in genome-wide association studies","volume":"10","author":"Zeggini","year":"2009","journal-title":"Pharmacogenomics"},{"key":"2023012810473963400_btt508-B59","doi-asserted-by":"crossref","first-page":"638","DOI":"10.1038\/ng.120","article-title":"Meta-analysis of genome-wide association data and large-scale replication identifies additional susceptibility loci for type 2 diabetes","volume":"40","author":"Zeggini","year":"2008","journal-title":"Nat. Genet."},{"key":"2023012810473963400_btt508-B60","doi-asserted-by":"crossref","first-page":"1512","DOI":"10.1038\/ng.2007.26","article-title":"RNA polymerase stalling at developmental control genes in the Drosophila melanogaster embyo","volume":"39","author":"Zeitlinger","year":"2007","journal-title":"Nat. Genet."},{"key":"2023012810473963400_btt508-B61","doi-asserted-by":"crossref","first-page":"1662","DOI":"10.1093\/bioinformatics\/btp295","article-title":"Evaluating reproducibility of differential expression discoveries in microarray studies by considering correlated molecular changes","volume":"25","author":"Zhang","year":"2009","journal-title":"Bioinformatics"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/29\/22\/2844\/48895216\/bioinformatics_29_22_2844.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/29\/22\/2844\/48895216\/bioinformatics_29_22_2844.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,17]],"date-time":"2024-05-17T23:33:21Z","timestamp":1715988801000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/29\/22\/2844\/316252"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,9,17]]},"references-count":62,"journal-issue":{"issue":"22","published-print":{"date-parts":[[2013,11,15]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btt508","relation":{},"ISSN":["1367-4811","1367-4803"],"issn-type":[{"value":"1367-4811","type":"electronic"},{"value":"1367-4803","type":"print"}],"subject":[],"published-other":{"date-parts":[[2013,11,15]]},"published":{"date-parts":[[2013,9,17]]}}}