{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,29]],"date-time":"2026-06-29T18:48:02Z","timestamp":1782758882873,"version":"3.54.5"},"reference-count":45,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,9,7]],"date-time":"2021-09-07T00:00:00Z","timestamp":1630972800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,9,7]],"date-time":"2021-09-07T00:00:00Z","timestamp":1630972800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Bioinformatics"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Background<\/jats:title><jats:p>Assessing the reproducibility of measurements is an important first step for improving the reliability of downstream analyses of high-throughput metabolomics experiments. We define a metabolite to be reproducible when it demonstrates consistency across replicate experiments. Similarly, metabolites which are not consistent across replicates can be labeled as irreproducible. In this work, we introduce and evaluate the use (Ma)ximum (R)ank (R)eproducibility (MaRR) to examine reproducibility in mass spectrometry-based metabolomics experiments. We examine reproducibility across technical or biological samples in three different mass spectrometry metabolomics (MS-Metabolomics) data sets.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>We apply MaRR, a nonparametric approach that detects the change from reproducible to irreproducible signals using a maximal rank statistic. The advantage of using MaRR over model-based methods that it does not make parametric assumptions on the underlying distributions or dependence structures of reproducible metabolites. Using three MS Metabolomics data sets generated in the multi-center Genetic Epidemiology of Chronic Obstructive Pulmonary Disease (COPD) study, we applied the MaRR procedure after data processing to explore reproducibility across technical or biological samples. Under realistic settings of MS-Metabolomics data, the MaRR procedure effectively controls the False Discovery Rate (FDR) when there was a gradual reduction in correlation between replicate pairs for less highly ranked signals. Simulation studies also show that the MaRR procedure tends to have high power for detecting reproducible metabolites in most situations except for smaller values of proportion of reproducible metabolites. Bias (i.e., the difference between the estimated and the true value of reproducible signal proportions) values for simulations are also close to zero. The results reported from the real data show a higher level of reproducibility for technical replicates compared to biological replicates across all the three different datasets. In summary, we demonstrate that the MaRR procedure application can be adapted to various experimental designs, and that the nonparametric approach performs consistently well.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>This research was motivated by reproducibility, which has proven to be a major obstacle in the use of genomic findings to advance clinical practice. In this paper, we developed a data-driven approach to assess the reproducibility of MS-Metabolomics data sets. The methods described in this paper are implemented in the open-source R package<jats:italic>marr<\/jats:italic>, which is freely available from Bioconductor at<jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"http:\/\/bioconductor.org\/packages\/marr\">http:\/\/bioconductor.org\/packages\/marr<\/jats:ext-link>.<\/jats:p><\/jats:sec>","DOI":"10.1186\/s12859-021-04336-9","type":"journal-article","created":{"date-parts":[[2021,9,7]],"date-time":"2021-09-07T17:05:18Z","timestamp":1631034318000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":38,"title":["Reproducibility of mass spectrometry based metabolomics data"],"prefix":"10.1186","volume":"22","author":[{"given":"Tusharkanti","family":"Ghosh","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daisy","family":"Philtron","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weiming","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Katerina","family":"Kechris","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Debashis","family":"Ghosh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,9,7]]},"reference":[{"key":"4336_CR1","doi-asserted-by":"crossref","unstructured":"Maniscalco M, Fuschillo S, Paris D, Cutignano A, Sanduzzi A, Motta A. Clinical metabolomics of exhaled breath condensate in chronic respiratory diseases. In: Advances in clinical chemistry, vol. 88. Elsevier; 2019. p. 121\u2013149.","DOI":"10.1016\/bs.acc.2018.10.002"},{"issue":"3","key":"4336_CR2","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1007\/s11306-019-1505-6","volume":"15","author":"GV Sarode","year":"2019","unstructured":"Sarode GV, Kim K, Kieffer DA, Shibata NM, Litwin T, Czlonkowska A, Medici V. Metabolomics profiles of patients with Wilson disease reveal a distinct metabolic signature. Metabolomics. 2019;15(3):43.","journal-title":"Metabolomics"},{"key":"4336_CR3","doi-asserted-by":"publisher","first-page":"150","DOI":"10.3389\/fnins.2019.00150","volume":"13","author":"L-L Liu","year":"2019","unstructured":"Liu L-L, Lin Y, Chen W, Tong M-L, Luo X, Lin L-R, Zhang H, Yan J-H, Niu J-J, Yang T-C. Metabolite profiles of the cerebrospinal fluid in neurosyphilis patients determined by untargeted metabolomics analysis. Front Neurosci. 2019;13:150.","journal-title":"Front Neurosci"},{"key":"4336_CR4","doi-asserted-by":"publisher","first-page":"188","DOI":"10.3389\/fpls.2019.00188","volume":"10","author":"CF Sanchez Arcos","year":"2019","unstructured":"Sanchez Arcos CF, Kai M, Svato\u0161 A, Gershenzon J, Kunert G. Metabolomic approaches reveal differences in pea aphid host plant chemistry before and after infestation with different pea aphid host races. Front Plant Sci. 2019;10:188.","journal-title":"Front Plant Sci"},{"issue":"1","key":"4336_CR5","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1038\/nprot.2012.135","volume":"8","author":"EJ Want","year":"2013","unstructured":"Want EJ, Masson P, Michopoulos F, Wilson ID, Theodoridis G, Plumb RS, Shockcor J, Loftus N, Holmes E, Nicholson JK. Global metabolic profiling of animal and human tissues via UPLC-MS. Nat Protoc. 2013;8(1):17.","journal-title":"Nat Protoc"},{"issue":"9","key":"4336_CR6","doi-asserted-by":"publisher","first-page":"106","DOI":"10.1007\/s11306-017-1242-7","volume":"13","author":"R Spicer","year":"2017","unstructured":"Spicer R, Salek RM, Moreno P, Ca\u00f1ueto D, Steinbeck C. Navigating freely-available software tools for metabolomics analysis. Metabolomics. 2017;13(9):106.","journal-title":"Metabolomics"},{"issue":"3","key":"4336_CR7","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1007\/s11306-019-1501-x","volume":"15","author":"JW Allwood","year":"2019","unstructured":"Allwood JW, Xu Y, Martinez-Martin P, Palau R, Cowan A, Goodacre R, Marshall A, Stewart D, Howarth C. Rapid UHPLC-MS metabolite profiling and phenotypic assays reveal genotypic impacts of nitrogen supplementation in oats. Metabolomics. 2019;15(3):42.","journal-title":"Metabolomics"},{"issue":"8","key":"4336_CR8","doi-asserted-by":"publisher","first-page":"195982","DOI":"10.1242\/jeb.195982","volume":"222","author":"KE Lohr","year":"2019","unstructured":"Lohr KE, Camp EF, Kuzhiumparambil U, Lutz A, Leggat W, Patterson JT, Suggett DJ. Resolving coral photoacclimation dynamics through coupled photophysiological and metabolomic profiling. J Exp Biol. 2019;222(8):195982.","journal-title":"J Exp Biol"},{"issue":"19","key":"4336_CR9","doi-asserted-by":"publisher","first-page":"10397","DOI":"10.1021\/acs.analchem.7b02380","volume":"89","author":"NG Mahieu","year":"2017","unstructured":"Mahieu NG, Patti GJ. Systems-level annotation of a metabolomics data set reduces 25000 features to fewer than 1000 unique metabolites. Anal Chem. 2017;89(19):10397\u2013406.","journal-title":"Anal Chem"},{"issue":"14","key":"4336_CR10","doi-asserted-by":"publisher","first-page":"2311","DOI":"10.1002\/elps.200900584","volume":"31","author":"M Sugimoto","year":"2010","unstructured":"Sugimoto M, Hirayama A, Robert M, Abe S, Soga T, Tomita M. Prediction of metabolite identity from accurate mass, migration time prediction and isotopic pattern information in CE-TOFMS data. Electrophoresis. 2010;31(14):2311\u20138.","journal-title":"Electrophoresis"},{"issue":"2","key":"4336_CR11","doi-asserted-by":"publisher","first-page":"236","DOI":"10.1016\/j.jchromb.2008.04.031","volume":"871","author":"W Lu","year":"2008","unstructured":"Lu W, Bennett BD, Rabinowitz JD. Analytical strategies for LC\u2013MS-based targeted metabolomics. J Chromatogr B. 2008;871(2):236\u201342.","journal-title":"J Chromatogr B"},{"issue":"1","key":"4336_CR12","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1186\/1758-2946-2-9","volume":"2","author":"CD DeHaven","year":"2010","unstructured":"DeHaven CD, Evans AM, Dai H, Lawton KA. Organization of GC\/MS and LC\/MS metabolomics data into chemical libraries. J Cheminform. 2010;2(1):9.","journal-title":"J Cheminform"},{"issue":"3","key":"4336_CR13","doi-asserted-by":"publisher","first-page":"1752","DOI":"10.1214\/11-AOAS466","volume":"5","author":"Q Li","year":"2011","unstructured":"Li Q, Brown JB, Huang H, Bickel PJ, et al. Measuring reproducibility of high-throughput experiments. Ann Appl Stat. 2011;5(3):1752\u201379.","journal-title":"Ann Appl Stat"},{"issue":"4","key":"4336_CR14","doi-asserted-by":"publisher","first-page":"666","DOI":"10.1111\/j.1755-0998.2009.02819.x","volume":"10","author":"DL Porazinska","year":"2010","unstructured":"Porazinska DL, Sung W, Giblin-Davis RM, Thomas WK. Reproducibility of read numbers in high-throughput sequencing analysis of nematode community composition and structure. Mol Ecol Resour. 2010;10(4):666\u201376.","journal-title":"Mol Ecol Resour"},{"issue":"9","key":"4336_CR15","doi-asserted-by":"publisher","first-page":"1509","DOI":"10.1101\/gr.079558.108","volume":"18","author":"JC Marioni","year":"2008","unstructured":"Marioni JC, Mason CE, Mane SM, Stephens M, Gilad Y. RNA-seq: an assessment of technical reproducibility and comparison with gene expression arrays. Genome Res. 2008;18(9):1509\u201317.","journal-title":"Genome Res"},{"key":"4336_CR16","doi-asserted-by":"crossref","unstructured":"AC\u2019t Hoen P, Friedl\u00e4nder MR, Alml\u00f6f J, Sammeth M, Pulyakhina I, Anvar SY, Laros JF, Buermans HP, Karlberg O, Br\u00e4nnvall M, et al. Reproducibility of high-throughput MRNA and small RNA sequencing across laboratories. Nat Biotechnol 2013;31(11):1015.","DOI":"10.1038\/nbt.2702"},{"issue":"46","key":"4336_CR17","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1073\/pnas.1010604107","volume":"107","author":"W Talloen","year":"2010","unstructured":"Talloen W, Hochreiter S, Bijnens L, Kasim A, Shkedy Z, Amaratunga D, G\u00f6hlmann H. Filtering data from high-throughput experiments based on measurement reliability. Proc Natl Acad Sci. 2010;107(46):173\u20134.","journal-title":"Proc Natl Acad Sci"},{"issue":"1","key":"4336_CR18","doi-asserted-by":"publisher","first-page":"465","DOI":"10.1186\/1471-2105-9-465","volume":"9","author":"S Arvidsson","year":"2008","unstructured":"Arvidsson S, Kwasniewski M, Ria\u00f1o-Pach\u00f3n DM, Mueller-Roeber B. Quantprime-a flexible tool for reliable high-throughput primer design for quantitative PCR. BMC Bioinform. 2008;9(1):465.","journal-title":"BMC Bioinform"},{"issue":"5","key":"4336_CR19","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1038\/nmeth756","volume":"2","author":"RA Irizarry","year":"2005","unstructured":"Irizarry RA, Warren D, Spencer F, Kim IF, Biswal S, Frank BC, Gabrielson E, Garcia JG, Geoghegan J, Germino G, et al. Multiple-laboratory comparison of microarray platforms. Nat Methods. 2005;2(5):345\u201350.","journal-title":"Nat Methods"},{"issue":"3","key":"4336_CR20","doi-asserted-by":"publisher","first-page":"803","DOI":"10.1111\/biom.12832","volume":"74","author":"Q Li","year":"2018","unstructured":"Li Q, Zhang F. A regression framework for assessing covariate effects on the reproducibility of high-throughput experiments. Biometrics. 2018;74(3):803\u201313.","journal-title":"Biometrics"},{"key":"4336_CR21","unstructured":"Zhang F, Shen F, Yang T, Li Q. Segmented correspondence curve regression model for quantifying reproducibility of high-throughput experiments. arXiv:1807.00943 2018."},{"issue":"W1","key":"4336_CR22","doi-asserted-by":"publisher","first-page":"486","DOI":"10.1093\/nar\/gky310","volume":"46","author":"J Chong","year":"2018","unstructured":"Chong J, Soufan O, Li C, Caraus I, Li S, Bourque G, Wishart DS, Xia J. Metaboanalyst 4.0: towards more transparent and integrative metabolomics analysis. Nucleic Acids Res. 2018;46(W1):486\u201394.","journal-title":"Nucleic Acids Res"},{"issue":"1","key":"4336_CR23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12859-019-2871-9","volume":"20","author":"C Schiffman","year":"2019","unstructured":"Schiffman C, Petrick L, Perttula K, Yano Y, Carlsson H, Whitehead T, Metayer C, Hayes J, Rappaport S, Dudoit S. Filtering procedures for untargeted LC\u2013MS metabolomics data. BMC Bioinform. 2019;20(1):1\u201310.","journal-title":"BMC Bioinform"},{"issue":"9","key":"4336_CR24","doi-asserted-by":"publisher","first-page":"1493","DOI":"10.1093\/bioinformatics\/btu813","volume":"31","author":"F Giacomoni","year":"2015","unstructured":"Giacomoni F, Le Corguille G, Monsoor M, Landi M, Pericard P, P\u00e9t\u00e9ra M, Duperier C, Tremblay-Franco M, Martin J-F, Jacob D, et al. Workflow4metabolomics: a collaborative research infrastructure for computational metabolomics. Bioinformatics. 2015;31(9):1493\u20135.","journal-title":"Bioinformatics"},{"issue":"11","key":"4336_CR25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s11306-017-1280-1","volume":"13","author":"R Patterson","year":"2017","unstructured":"Patterson R, Kirpich A, Koelmel J, Kalavalapalli S, Morse A, Cusi K, Sunny N, McIntyre L, Garrett T, Yost R. Improved experimental data processing for UHPLC-HRMS\/MS lipidomics applied to nonalcoholic fatty liver disease. Metabolomics. 2017;13(11):1\u201311.","journal-title":"Metabolomics"},{"issue":"523","key":"4336_CR26","doi-asserted-by":"publisher","first-page":"1028","DOI":"10.1080\/01621459.2017.1397521","volume":"113","author":"D Philtron","year":"2018","unstructured":"Philtron D, Lyu Y, Li Q, Ghosh D. Maximum rank reproducibility: a nonparametric approach to assessing reproducibility in replicate experiments. J Am Stat Assoc. 2018;113(523):1028\u201339.","journal-title":"J Am Stat Assoc"},{"issue":"1","key":"4336_CR27","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1111\/j.1467-9868.2004.00439.x","volume":"66","author":"JD Storey","year":"2004","unstructured":"Storey JD, Taylor JE, Siegmund D. Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach. J R Stat Soc Ser B (Stat Methodol). 2004;66(1):187\u2013205.","journal-title":"J R Stat Soc Ser B (Stat Methodol)"},{"issue":"3","key":"4336_CR28","doi-asserted-by":"publisher","first-page":"499","DOI":"10.1111\/1467-9868.00347","volume":"64","author":"C Genovese","year":"2002","unstructured":"Genovese C, Wasserman L. Operating characteristics and extensions of the false discovery rate procedure. J R Stat Soc Ser B (Stat Methodol). 2002;64(3):499\u2013517.","journal-title":"J R Stat Soc Ser B (Stat Methodol)"},{"key":"4336_CR29","unstructured":"Heller R, Gur H. False discovery rate controlling procedures for discrete tests. arXiv:1112.4627 2011."},{"issue":"1","key":"4336_CR30","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1111\/j.2517-6161.1995.tb02031.x","volume":"57","author":"Y Benjamini","year":"1995","unstructured":"Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Ser B (Methodol). 1995;57(1):289\u2013300.","journal-title":"J R Stat Soc Ser B (Methodol)"},{"issue":"1","key":"4336_CR31","doi-asserted-by":"publisher","first-page":"32","DOI":"10.3109\/15412550903499522","volume":"7","author":"EA Regan","year":"2011","unstructured":"Regan EA, Hokanson JE, Murphy JR, Make B, Lynch DA, Beaty TH, Curran-Everett D, Silverman EK, Crapo JD. Genetic epidemiology of COPD (COPDGene) study design. COPD: J Chron Obstruct Pulmon Dis. 2011;7(1):32\u201343. https:\/\/doi.org\/10.3109\/15412550903499522.","journal-title":"COPD: J Chron Obstruct Pulmon Dis"},{"issue":"1","key":"4336_CR32","doi-asserted-by":"publisher","first-page":"17132","DOI":"10.1038\/s41598-018-35372-w","volume":"8","author":"CI Cruickshank-Quinn","year":"2018","unstructured":"Cruickshank-Quinn CI, Jacobson S, Hughes G, Powell RL, Petrache I, Kechris K, Bowler R, Reisdorph N. Metabolomics and transcriptomics pathway approach reveals outcome-specific perturbations in COPD. Sci Rep. 2018;8(1):17132.","journal-title":"Sci Rep"},{"issue":"16","key":"4336_CR33","doi-asserted-by":"publisher","first-page":"6656","DOI":"10.1021\/ac901536h","volume":"81","author":"AM Evans","year":"2009","unstructured":"Evans AM, DeHaven CD, Barrett T, Mitchell M, Milgram E. Integrated, nontargeted ultrahigh performance liquid chromatography\/electrospray ionization tandem mass spectrometry platform for the identification and relative quantification of the small-molecule complement of biological systems. Anal Chem. 2009;81(16):6656\u201367.","journal-title":"Anal Chem"},{"issue":"1","key":"4336_CR34","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1093\/bioinformatics\/btt589","volume":"30","author":"G Hughes","year":"2013","unstructured":"Hughes G, Cruickshank-Quinn C, Reisdorph R, Lutz S, Petrache I, Reisdorph N, Bowler R, Kechris K. MSPrep\u2013summarization, normalization and diagnostics for processing of mass spectrometry-based metabolomic data. Bioinformatics. 2013;30(1):133\u20134.","journal-title":"Bioinformatics"},{"issue":"2","key":"4336_CR35","doi-asserted-by":"publisher","first-page":"567","DOI":"10.1021\/ac051495j","volume":"78","author":"S Bijlsma","year":"2006","unstructured":"Bijlsma S, Bobeldijk I, Verheij ER, Ramaker R, Kochhar S, Macdonald IA, Van Ommen B, Smilde AK. Large-scale human metabolomics studies: a strategy for data (pre-) processing and validation. Anal Chem. 2006;78(2):567\u201374.","journal-title":"Anal Chem"},{"issue":"9","key":"4336_CR36","doi-asserted-by":"publisher","first-page":"1164","DOI":"10.1093\/bioinformatics\/btm069","volume":"23","author":"W Stacklies","year":"2007","unstructured":"Stacklies W, Redestig H, Scholz M, Walther D, Selbig J. pcaMethods\u2014a bioconductor package providing PCA methods for incomplete data. Bioinformatics. 2007;23(9):1164\u20137.","journal-title":"Bioinformatics"},{"issue":"24","key":"4336_CR37","doi-asserted-by":"publisher","first-page":"3050","DOI":"10.1002\/elps.201500352","volume":"36","author":"EG Armitage","year":"2015","unstructured":"Armitage EG, Godzien J, Alonso-Herranz V, L\u00f3pez-Gonz\u00e1lvez \u00c1, Barbas C. Missing value imputation strategies for metabolomics data. Electrophoresis. 2015;36(24):3050\u201360.","journal-title":"Electrophoresis"},{"key":"4336_CR38","unstructured":"Hastie T, Tibshirani R, Sherlock G, Eisen M, Brown P, Botstein D. Imputing missing data for gene expression arrays;1999."},{"issue":"1","key":"4336_CR39","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12859-019-3110-0","volume":"20","author":"M Kokla","year":"2019","unstructured":"Kokla M, Virtanen J, Kolehmainen M, Paananen J, Hanhineva K. Random forest-based imputation outperforms other methods for imputing LC-MS metabolomics data: a comparative study. BMC Bioinform. 2019;20(1):1\u201311.","journal-title":"BMC Bioinform."},{"issue":"1","key":"4336_CR40","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1007\/s11306-011-0366-4","volume":"8","author":"O Hrydziuszko","year":"2012","unstructured":"Hrydziuszko O, Viant MR. Missing values in mass spectrometry based metabolomics: an undervalued step in the data processing pipeline. Metabolomics. 2012;8(1):161\u201374.","journal-title":"Metabolomics"},{"issue":"5","key":"4336_CR41","doi-asserted-by":"publisher","first-page":"1993","DOI":"10.1021\/pr501138h","volume":"14","author":"B-JM Webb-Robertson","year":"2015","unstructured":"Webb-Robertson B-JM, Wiberg HK, Matzke MM, Brown JN, Wang J, McDermott JE, Smith RD, Rodland KD, Metz TO, Pounds JG, et al. Review, evaluation, and discussion of the challenges of missing value imputation for mass spectrometry-based label-free global proteomics. J Proteome Res. 2015;14(5):1993\u20132001.","journal-title":"J Proteome Res"},{"key":"4336_CR42","doi-asserted-by":"crossref","unstructured":"Reinke SN, Gallart-Ayala H, G\u00f3mez C, Checa A, Fauland A, Naz S, Kamleh MA, Djukanovi\u0107 R, Hinks TS, Wheelock CE. Metabolomics analysis identifies different metabotypes of asthma severity. Eur Respir J 2017;49(3).","DOI":"10.1183\/13993003.01740-2016"},{"issue":"3","key":"4336_CR43","doi-asserted-by":"publisher","first-page":"962","DOI":"10.1214\/009053604000000265","volume":"32","author":"D Donoho","year":"2004","unstructured":"Donoho D, Jin J, et al. Higher criticism for detecting sparse heterogeneous mixtures. Ann Stat. 2004;32(3):962\u201394.","journal-title":"Ann Stat"},{"key":"4336_CR44","unstructured":"Lin X. Reproducibility and replicability in large scale genetic studies. In: Committee on reproducibility and replicability in science, National Academies of Sciences. Engineering, and Medicine;2018."},{"key":"4336_CR45","unstructured":"Ghosh T, McGrath M, Philtron D, Kechris K, Ghosh DM, Maximum rank reproducibility. In: R\/Bioconductor package version 1.00.02; 2021."}],"container-title":["BMC Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-021-04336-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12859-021-04336-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-021-04336-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,7]],"date-time":"2024-09-07T23:42:38Z","timestamp":1725752558000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/s12859-021-04336-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,7]]},"references-count":45,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["4336"],"URL":"https:\/\/doi.org\/10.1186\/s12859-021-04336-9","relation":{},"ISSN":["1471-2105"],"issn-type":[{"value":"1471-2105","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,7]]},"assertion":[{"value":"15 November 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 August 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 September 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"No ethics approval was required for the study.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not Applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"423"}}