{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,17]],"date-time":"2026-01-17T09:30:15Z","timestamp":1768642215295,"version":"3.49.0"},"reference-count":88,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2019,5,15]],"date-time":"2019-05-15T00:00:00Z","timestamp":1557878400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>A statistical hypothesis test is one of the most eminent methods in statistics. Its pivotal role comes from the wide range of practical problems it can be applied to and the sparsity of data requirements. Being an unsupervised method makes it very flexible in adapting to real-world situations. The availability of high-dimensional data makes it necessary to apply such statistical hypothesis tests simultaneously to the test statistics of the underlying covariates. However, if applied without correction this leads to an inevitable increase in Type 1 errors. To counteract this effect, multiple testing procedures have been introduced to control various types of errors, most notably the Type 1 error. In this paper, we review modern multiple testing procedures for controlling either the family-wise error (FWER) or the false-discovery rate (FDR). We emphasize their principal approach allowing categorization of them as (1) single-step vs. stepwise approaches, (2) adaptive vs. non-adaptive approaches, and (3) marginal vs. joint multiple testing procedures. We place a particular focus on procedures that can deal with data with a (strong) correlation structure because real-world data are rarely uncorrelated. Furthermore, we also provide background information making the often technically intricate methods accessible for interdisciplinary data scientists.<\/jats:p>","DOI":"10.3390\/make1020039","type":"journal-article","created":{"date-parts":[[2019,5,15]],"date-time":"2019-05-15T11:37:40Z","timestamp":1557920260000},"page":"653-683","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Large-Scale Simultaneous Inference with Hypothesis Testing: Multiple Testing Procedures in Practice"],"prefix":"10.3390","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0745-5641","authenticated-orcid":false,"given":"Frank","family":"Emmert-Streib","sequence":"first","affiliation":[{"name":"Predictive Society and Data Analytics Lab, Faculty of Information Technology and Communication Sciences, Tampere University, 33100 Tampere, Finland"},{"name":"Institute of Biosciences and Medical Technology, 33520 Tampere, Finland"}]},{"given":"Matthias","family":"Dehmer","sequence":"additional","affiliation":[{"name":"Institute for Intelligent Production, Faculty for Management, University of Applied Sciences Upper Austria, Steyr Campus, Steyr 4400, Austria"},{"name":"Department of Mechatronics and Biomedical Computer Science, University for Health Sciences, Medical Informatics and Technology, Tirol 6060, Austria"},{"name":"College of Computer and Control Engineering, Nankai University, Tianjin 300071, China"}]}],"member":"1968","published-online":{"date-parts":[[2019,5,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1093\/nsr\/nwt032","article-title":"Challenges of big data analysis","volume":"1","author":"Fan","year":"2014","journal-title":"Natl. Sci. Rev."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1089\/big.2013.1508","article-title":"Data science and its relationship to big data and data-driven decision making","volume":"1","author":"Provost","year":"2013","journal-title":"Big Data"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Hayashi, C. (1998). What is data science? Fundamental concepts and a heuristic example. Data Science, Classification, and Related Methods, Springer.","DOI":"10.1007\/978-4-431-65950-1_3"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1111\/j.1751-5823.2001.tb00477.x","article-title":"Data science: An action plan for expanding the technical areas of the field of statistics","volume":"69","author":"Cleveland","year":"2001","journal-title":"Int. Stat. Rev."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1080\/00031305.2015.1077729","article-title":"Data Science in Statistics Curricula: Preparing Students to \u2018Think with Data\u2019","volume":"69","author":"Hardin","year":"2015","journal-title":"Am. Stat."},{"key":"ref_6","first-page":"12","article-title":"The process of analyzing data is the emergent feature of data science","volume":"7","author":"Moutari","year":"2016","journal-title":"Front. Genet."},{"key":"ref_7","first-page":"235","article-title":"Defining Data Science by a Data-Driven Quantification of the Community","volume":"1","author":"Dehmer","year":"2019","journal-title":"Mach. Learn. Knowl. Extract."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Vapnik, V.N. (1995). The Nature of Statistical Learning Theory, Springer.","DOI":"10.1007\/978-1-4757-2440-0"},{"key":"ref_9","unstructured":"Lehman, E. (2005). Testing Statistical Hypotheses, Springer."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Dudoit, S., and Van Der Laan, M.J. (2007). Multiple Testing Procedures With Applications to Genomics, Springer Science & Business Media.","DOI":"10.1007\/978-0-387-49317-6"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1135","DOI":"10.1038\/nbt1209-1135","article-title":"How does multiple testing correction work?","volume":"27","author":"Noble","year":"2009","journal-title":"Nat. Biotechnol."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Efron, B. (2010). Large-Scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction, Cambridge University Press.","DOI":"10.1017\/CBO9780511761362"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1408","DOI":"10.1198\/016214506000000339","article-title":"Exceedance Control of the False Discovery Proportion","volume":"101","author":"Genovese","year":"2006","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1111\/1467-9868.00346","article-title":"A direct approach to false discovery rates","volume":"64","author":"Storey","year":"2002","journal-title":"J. R. Stat. Soc. Ser. B"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1214\/07-AOAS102","article-title":"Control of the mean number of false discoveries, Bonferroni and stability of multiple testing","volume":"1","author":"Gordon","year":"2007","journal-title":"Ann. Appl. Stat."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"499","DOI":"10.1111\/1467-9868.00347","article-title":"Operating characteristics and extensions of the false discovery rate procedure","volume":"64","author":"Genovese","year":"2002","journal-title":"J. Royal Stat. Soc. Ser. B (Stat. Methodol.)"},{"key":"ref_17","first-page":"3","article-title":"Teoria statistica delle classi e calcolo delle probabilita","volume":"8","author":"Bonferroni","year":"1936","journal-title":"Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commerciali di Firenze"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1093\/biomet\/69.3.493","article-title":"Plots of p-values to evaluate many tests simultaneously","volume":"69","author":"Schweder","year":"1982","journal-title":"Biometrika"},{"key":"ref_19","first-page":"125","article-title":"Controlling the false discovery rate: a practical and powerful approach to multiple testing","volume":"57","author":"Benjamini","year":"1995","journal-title":"J. R. Stat. Soc. Ser. B (Methodol.)"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"R1","DOI":"10.1152\/ajpregu.2000.279.1.R1","article-title":"Multiple comparisons: Philosophies and illustrations","volume":"279","year":"2000","journal-title":"Am. J. Physiol.-Regul. Integr. Comparat. Physiol."},{"key":"ref_21","first-page":"626","article-title":"Rectangular confidence regions for the means of multivariate normal distributions","volume":"62","year":"1967","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_22","unstructured":"Westfall, P.H., and Young, S.S. (1993). Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment, John Wiley & Sons."},{"key":"ref_23","first-page":"65","article-title":"A simple sequentially rejective multiple test procedure","volume":"6","author":"Holm","year":"1979","journal-title":"Scand. J. Stat."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"800","DOI":"10.1093\/biomet\/75.4.800","article-title":"A sharper Bonferroni procedure for multiple tests of significance","volume":"75","author":"Hochberg","year":"1988","journal-title":"Biometrika"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1165","DOI":"10.1214\/aos\/1013699998","article-title":"The control of the false discovery rate in multiple testing under dependency","volume":"29","author":"Benjamini","year":"2001","journal-title":"Ann. Stat."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"491","DOI":"10.1093\/biomet\/93.3.491","article-title":"Adaptive linear step-up procedures that control the false discovery rate","volume":"93","author":"Benjamini","year":"2006","journal-title":"Biometrika"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1007\/s11749-008-0126-6","article-title":"Control of the false discovery rate under dependence using the bootstrap and subsampling","volume":"17","author":"Romano","year":"2008","journal-title":"Test"},{"key":"ref_28","first-page":"303-14","article-title":"Multiple hypothesis testing: A review","volume":"68","author":"Austin","year":"2014","journal-title":"J. Indian Soc. Agric. Stat."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"13","DOI":"10.2202\/1544-6115.1040","article-title":"Multiple Testing. Part I. Single-Step Procedures for Control of General Type I Error Rates","volume":"3","author":"Dudoit","year":"2004","journal-title":"Stat. Appl. Genet. Mol. Biol."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"716","DOI":"10.1002\/bimj.200710473","article-title":"Resampling-Based Empirical Bayes Multiple Testing Procedures for Controlling Generalized Tail Probability and Expected Value Error Rates: Focus on the False Discovery Rate and Simulation Study","volume":"50","author":"Dudoit","year":"2008","journal-title":"Biometrical J."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Emmert-Streib, F., and Dehmer, M. (2010). Multiple Testing Methods. Medical Biostatistics for Complex Diseases, John Wiley & Sons, Ltd.. Available online: https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/9783527630332.ch3.","DOI":"10.1002\/9783527630332"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Kim, K.I., and van de Wiel, M. (2008). Effects of dependence in high-dimensional multiple testing problems. BMC Bioinform., 9.","DOI":"10.1186\/1471-2105-9-114"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"2665","DOI":"10.1016\/j.csda.2011.03.016","article-title":"Estimation of the proportion of true null hypotheses in high-dimensional data under dependence","volume":"55","author":"Friguet","year":"2011","journal-title":"Comput. Stat. Data Anal."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1080\/01621459.2014.999157","article-title":"Large-scale multiple testing of correlations","volume":"111","author":"Cai","year":"2016","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_35","unstructured":"R Development Core Team (2008). R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Hochberg, J., and Tamhane, A. (1987). Multiple Comparison Procedures, John Wiley & Sons.","DOI":"10.1002\/9780470316672"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"751","DOI":"10.1093\/biomet\/73.3.751","article-title":"An improved Bonferroni procedure for multiple tests of significance","volume":"73","author":"Simes","year":"1986","journal-title":"Biometrika"},{"key":"ref_38","unstructured":"Genz, A., Bretz, F., Miwa, T., Mi, X., Leisch, F., Scheipl, F., and Hothorn, T. (2008, August 23). mvtnorm: Multivariate Normal and t Distributions. R Package Version 1.0-9. Available online: https:\/\/cran.r-project.org\/web\/packages\/mvtnorm\/index.html."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Genz, A., and Bretz, F. (2009). Computation of Multivariate Normal and t Probabilities, Springer. Lecture Notes in Statistics.","DOI":"10.1007\/978-3-642-01689-9"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"17","DOI":"10.3389\/fams.2019.00017","article-title":"Constrained covariance matrices with a biologically realistic structure: Comparison of methods for generating high-dimensional Gaussian graphical models","volume":"5","author":"Tripathi","year":"2019","journal-title":"Front. Appl. Math. Stat."},{"key":"ref_41","unstructured":"Tripathi, S., and Emmert-Streib, F. (2008, August 23). Mvgraphnorm: Multivariate Gaussian Graphical Models. R Package Version 1.0.0. Available online: https:\/\/cran.r-project.org\/web\/packages\/mvgraphnorm\/index.html."},{"key":"ref_42","unstructured":"Blanchard, G., Dickhaus, T., Hack, N., Konietschke, F., Rohmeyer, K., Rosenblatt, J., Scheer, M., and Werft, W. (2010, January 1\u20133). \u03bcTOSS-Multiple hypothesis testing in an open software system. Proceedings of the First Workshop on Applications of Pattern Analysis, Windsor, UK."},{"key":"ref_43","unstructured":"Pollard, K., Dudoit, S., and van der Laan, M. (2019, May 25). Multiple Testing Procedures: R Multtest Package and Applications to Genomics. UC Berkeley Division of Biostatistics Working Paper Series. Technical Report, Working Paper 164. Available online: http:\/\/www.bepress.com\/ucbbiostat\/paper164."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1002\/bimj.201700316","article-title":"Hommel\u2019s procedure in linear time","volume":"61","author":"Meijer","year":"2019","journal-title":"Biometrical J."},{"key":"ref_45","first-page":"1","article-title":"Neural correlates of interspecies perspective taking in the post-mortem atlantic salmon: An argument for proper multiple comparisons correction","volume":"1","author":"Bennett","year":"2011","journal-title":"J. Serendipitous Unexpected Results"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1093\/scan\/nsp053","article-title":"The principled control of false positives in neuroimaging","volume":"4","author":"Bennett","year":"2009","journal-title":"Soc. Cognit. Affect. Neurosci."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"419","DOI":"10.1191\/0962280203sm341ra","article-title":"Controlling the familywise error rate in functional neuroimaging: A comparative review","volume":"12","author":"Nichols","year":"2003","journal-title":"Stat. Methods Med. Res."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"M110.004374","DOI":"10.1074\/mcp.M110.004374","article-title":"Multiple hypothesis testing in proteomics: A strategy for experimental work","volume":"10","author":"Diz","year":"2011","journal-title":"Mol. Cell. Proteomics"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1214\/ss\/1056397487","article-title":"Multiple hypothesis testing in microarray experiments","volume":"18","author":"Dudoit","year":"2003","journal-title":"Stat. Sci."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"1946","DOI":"10.1002\/sim.6082","article-title":"Multiple hypothesis testing in genomics","volume":"33","author":"Goeman","year":"2014","journal-title":"Stat. Med."},{"key":"ref_51","first-page":"567","article-title":"On multiple-testing correction in genome-wide association studies","volume":"32","author":"Moskvina","year":"2008","journal-title":"Genet. Epidemiol. Off. Publ. Int. Genet. Epidemiol. Soc."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"108","DOI":"10.3905\/jpm.2014.40.5.108","article-title":"Evaluating trading strategies","volume":"40","author":"Harvey","year":"2014","journal-title":"J. Portfolio Manag."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"3492","DOI":"10.1086\/324109","article-title":"Controlling the false-discovery rate in astrophysical data analysis","volume":"122","author":"Miller","year":"2001","journal-title":"Astron. J."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Cranmer, K. (2006). Statistical challenges for searches for new physics at the LHC. Statistical Problems in Particle Physics, Astrophysics and Cosmology, World Scientific.","DOI":"10.1142\/9781860948985_0026"},{"key":"ref_55","first-page":"1867","article-title":"New FDR bounds for discrete and heterogeneous tests","volume":"12","author":"Durand","year":"2018","journal-title":"Electronic J. Stat."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"60","DOI":"10.2307\/1165312","article-title":"On the adaptive control of the false discovery rate in multiple testing with independent statistics","volume":"25","author":"Benjamini","year":"2000","journal-title":"J. Educ. Behav. Stat."},{"key":"ref_57","first-page":"135","article-title":"On methods controlling the false discovery rate","volume":"70","author":"Sarkar","year":"2008","journal-title":"Sankhy\u0101 Indian J. Stat. Ser. A (2008-)"},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1146\/annurev.ps.46.020195.003021","article-title":"Multiple hypothesis testing","volume":"46","author":"Shaffer","year":"1995","journal-title":"Annu. Rev. Psychol."},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Dmitrienko, A., Tamhane, A.C., and Bretz, F. (2009). Multiple Testing Problems in Pharmaceutical Statistics, CRC Press.","DOI":"10.1201\/9781584889854"},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1093\/biomet\/75.2.383","article-title":"A stagewise rejective multiple test procedure based on a modified Bonferroni test","volume":"75","author":"Hommel","year":"1988","journal-title":"Biometrika"},{"key":"ref_61","first-page":"745","article-title":"Multiple testing with minimal assumptions","volume":"50","author":"Westfall","year":"2008","journal-title":"Biometrical J. J. Math. Methods Biosci."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"3782","DOI":"10.1214\/10-AOS829","article-title":"The sequential rejection principle of familywise error control","volume":"38","author":"Goeman","year":"2010","journal-title":"Ann. Stat."},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/BF02595811","article-title":"Resampling-based multiple testing for microarray data analysis","volume":"12","author":"Ge","year":"2003","journal-title":"TEST"},{"key":"ref_64","doi-asserted-by":"crossref","unstructured":"Rempala, G.A., and Yang, Y. (2013). On permutation procedures for strong control in multiple testing with gene expression data. Stat. Interface, 6.","DOI":"10.4310\/SII.2013.v6.n1.a8"},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"1827","DOI":"10.1214\/009053606000000425","article-title":"On the Benjamini\u2013Hochberg method","volume":"34","author":"Ferreira","year":"2006","journal-title":"Ann. Stat."},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1111\/j.1467-9868.2011.01001.x","article-title":"Adaptive and dynamic adaptive procedures for false discovery rate control and estimation","volume":"74","author":"Liang","year":"2012","journal-title":"J. R. Stat. Soc. Ser. B (Stat. Methodol.)"},{"key":"ref_67","first-page":"2837","article-title":"Adaptive false discovery rate control under independence and dependence","volume":"10","author":"Blanchard","year":"2009","journal-title":"J. Mach. Learn. Res."},{"key":"ref_68","doi-asserted-by":"crossref","first-page":"1450018","DOI":"10.1142\/S0219720014500188","article-title":"Comparative analysis of false discovery rate methods in constructing metabolic association networks","volume":"12","author":"Koo","year":"2014","journal-title":"J. Bioinform. Comput. Biol."},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"5","DOI":"10.22237\/jmasm\/1430453040","article-title":"Are per-family type I error rates relevant in social and behavioral science?","volume":"14","author":"Frane","year":"2015","journal-title":"J. Mod. Appl. Stat. Methods"},{"key":"ref_70","doi-asserted-by":"crossref","first-page":"1187","DOI":"10.1080\/10543406.2011.607751","article-title":"On using the bootstrap for multiple comparisons","volume":"21","author":"Westfall","year":"2011","journal-title":"J. Biopharm. Stat."},{"key":"ref_71","doi-asserted-by":"crossref","unstructured":"Li, D., and Dye, T.D. (2013). Power and stability properties of resampling-based multiple testing procedures with applications to gene oncology studies. Comput. Math. Methods Med., 2013.","DOI":"10.1155\/2013\/610297"},{"key":"ref_72","doi-asserted-by":"crossref","unstructured":"De Matos Simoes, R., Dehmer, M., and Emmert-Streib, F. (2013). Interfacing cellular networks of S. cerevisiae and E. coli: Connecting dynamic and genetic information. BMC Genom., 14.","DOI":"10.1186\/1471-2164-14-324"},{"key":"ref_73","doi-asserted-by":"crossref","unstructured":"Emmert-Streib, F., Moutari, S., and Dehmer, M. (2019). A comprehensive survey of error measures for evaluating binary decision making in data science. Wiley Interdiscip. Rev. Data Min. Knowl. Discov., e1303.","DOI":"10.1002\/widm.1303"},{"key":"ref_74","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1111\/j.1467-9868.2004.00439.x","article-title":"Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: A unified approach","volume":"66","author":"Storey","year":"2004","journal-title":"J. R. Stat. Soc. Ser. B (Stat. Methodol.)"},{"key":"ref_75","doi-asserted-by":"crossref","first-page":"619","DOI":"10.1214\/07-AOS586","article-title":"An adaptive step-down procedure with proven FDR control under independence","volume":"37","author":"Gavrilov","year":"2009","journal-title":"Ann. Stat."},{"key":"ref_76","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1093\/biomet\/93.3.509","article-title":"False discovery control with p-value weighting","volume":"93","author":"Genovese","year":"2006","journal-title":"Biometrika"},{"key":"ref_77","doi-asserted-by":"crossref","first-page":"801","DOI":"10.1214\/13-AOAS707","article-title":"Testing the disjunction hypothesis using Voronoi diagrams with applications to genetics","volume":"8","author":"Phillips","year":"2014","journal-title":"Ann. Appl. Stat."},{"key":"ref_78","doi-asserted-by":"crossref","first-page":"3369","DOI":"10.1214\/11-AOS946","article-title":"Asymptotic optimality of the Westfall\u2013Young permutation procedure for multiple testing under dependence","volume":"39","author":"Meinshausen","year":"2011","journal-title":"Ann. Stat."},{"key":"ref_79","doi-asserted-by":"crossref","first-page":"598","DOI":"10.1214\/09-AOS734","article-title":"Balanced control of generalized error rates","volume":"38","author":"Romano","year":"2010","journal-title":"Ann. Stat."},{"key":"ref_80","doi-asserted-by":"crossref","first-page":"1165","DOI":"10.2307\/41703503","article-title":"Business intelligence and analytics: from big data to big impact","volume":"36","author":"Chen","year":"2012","journal-title":"MIS Q."},{"key":"ref_81","doi-asserted-by":"crossref","first-page":"897","DOI":"10.1016\/j.jbusres.2015.07.001","article-title":"Big Data consumer analytics and the transformation of marketing","volume":"69","author":"Erevelles","year":"2016","journal-title":"J. Bus. Res."},{"key":"ref_82","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1016\/j.bdr.2015.01.006","article-title":"Significance and challenges of big data research","volume":"2","author":"Jin","year":"2015","journal-title":"Big Data Res."},{"key":"ref_83","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1038\/455028a","article-title":"Big data: How do your data grow?","volume":"455","author":"Lynch","year":"2008","journal-title":"Nature"},{"key":"ref_84","doi-asserted-by":"crossref","first-page":"216","DOI":"10.1068\/b36093","article-title":"An assessment of the effectiveness of multiple hypothesis testing for geographical anomaly detection","volume":"38","author":"Brunsdon","year":"2011","journal-title":"Environ. Plan. B Plan. Des."},{"key":"ref_85","doi-asserted-by":"crossref","first-page":"59","DOI":"10.21314\/JRMV.2010.062","article-title":"Validation of credit default probabilities using multiple-testing procedures","volume":"4","year":"2010","journal-title":"J. Risk Model Validat."},{"key":"ref_86","doi-asserted-by":"crossref","unstructured":"Stevens, J.R., Al Masud, A., and Suyundikov, A. (2017). A comparison of multiple testing adjustment methods with block-correlation positively-dependent tests. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0176124"},{"key":"ref_87","doi-asserted-by":"crossref","first-page":"278","DOI":"10.1111\/j.2041-210X.2010.00061.x","article-title":"Using false discovery rates for multiple comparisons in ecology and evolution","volume":"2","author":"Pike","year":"2011","journal-title":"Methods Ecol. Evol."},{"key":"ref_88","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1111\/j.1467-9868.2010.00746.x","article-title":"Discovering the false discovery rate","volume":"72","author":"Benjamini","year":"2010","journal-title":"J. R. Stat. Soc. Ser. B (Stat. Methodol.)"}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/1\/2\/39\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T12:52:13Z","timestamp":1760187133000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/1\/2\/39"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,5,15]]},"references-count":88,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2019,6]]}},"alternative-id":["make1020039"],"URL":"https:\/\/doi.org\/10.3390\/make1020039","relation":{},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,5,15]]}}}