{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,3]],"date-time":"2026-03-03T06:44:42Z","timestamp":1772520282476,"version":"3.50.1"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1010061","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2022,5,12]],"date-time":"2022-05-12T00:00:00Z","timestamp":1652313600000}}],"reference-count":47,"publisher":"Public Library of Science (PLoS)","issue":"5","license":[{"start":{"date-parts":[[2022,5,2]],"date-time":"2022-05-02T00:00:00Z","timestamp":1651449600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"national institutes of health","doi-asserted-by":"publisher","award":["CA227942"],"award-info":[{"award-number":["CA227942"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>\n                    While hierarchical experimental designs are near-ubiquitous in neuroscience and biomedical research, researchers often do not take the structure of their datasets into account while performing statistical hypothesis tests. Resampling-based methods are a flexible strategy for performing these analyses but are difficult due to the lack of open-source software to automate test construction and execution. To address this, we present\n                    <jats:italic>Hierarch<\/jats:italic>\n                    , a Python package to perform hypothesis tests and compute confidence intervals on hierarchical experimental designs. Using a combination of permutation resampling and bootstrap aggregation,\n                    <jats:italic>Hierarch<\/jats:italic>\n                    can be used to perform hypothesis tests that maintain nominal Type I error rates and generate confidence intervals that maintain the nominal coverage probability without making distributional assumptions about the dataset of interest.\n                    <jats:italic>Hierarch<\/jats:italic>\n                    makes use of the Numba JIT compiler to reduce\n                    <jats:italic>p-<\/jats:italic>\n                    value computation times to under one second for typical datasets in biomedical research.\n                    <jats:italic>Hierarch<\/jats:italic>\n                    also enables researchers to construct user-defined resampling plans that take advantage of\n                    <jats:italic>Hierarch\u2019s<\/jats:italic>\n                    Numba-accelerated functions.\n                  <\/jats:p>","DOI":"10.1371\/journal.pcbi.1010061","type":"journal-article","created":{"date-parts":[[2022,5,2]],"date-time":"2022-05-02T13:35:25Z","timestamp":1651498525000},"page":"e1010061","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":15,"title":["Analyzing nested experimental designs\u2014A user-friendly resampling method to determine experimental significance"],"prefix":"10.1371","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6422-149X","authenticated-orcid":true,"given":"Rishikesh U.","family":"Kulkarni","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9855-6194","authenticated-orcid":true,"given":"Catherine L.","family":"Wang","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4482-2754","authenticated-orcid":true,"given":"Carolyn R.","family":"Bertozzi","sequence":"additional","affiliation":[]}],"member":"340","published-online":{"date-parts":[[2022,5,2]]},"reference":[{"key":"pcbi.1010061.ref001","doi-asserted-by":"crossref","first-page":"10601","DOI":"10.1523\/JNEUROSCI.0362-10.2010","article-title":"A Study of Clustered Data and Approaches to Its Analysis","volume":"30","author":"S Galbraith","year":"2010","journal-title":"J Neurosci"},{"key":"pcbi.1010061.ref002","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1177\/0013164416678980","article-title":"Using Cluster Bootstrapping to Analyze Nested Data With a Few Clusters.","volume":"78","author":"FL Huang","year":"2018","journal-title":"Educ Psychol Meas."},{"key":"pcbi.1010061.ref003","doi-asserted-by":"crossref","first-page":"e0146721","DOI":"10.1371\/journal.pone.0146721","article-title":"Analyzing Clustered Data: Why and How to Account for Multiple Observations Nested within a Study Participant?","volume":"11","author":"EL Moen","year":"2016","journal-title":"PLOS ONE."},{"key":"pcbi.1010061.ref004","article-title":"Application of the hierarchical bootstrap to multi-level data in neuroscience","author":"V Saravanan","year":"2020","journal-title":"ArXiv200707797 Q-Bio."},{"key":"pcbi.1010061.ref005","doi-asserted-by":"crossref","DOI":"10.3389\/fpsyg.2011.00074","article-title":"Data with Hierarchical Structure: Impact of Intraclass Correlation and Sample Size on Type-I Error.","volume":"2","author":"SC Musca","year":"2011","journal-title":"Front Psychol"},{"key":"pcbi.1010061.ref006","doi-asserted-by":"crossref","first-page":"103","DOI":"10.3389\/fnhum.2018.00103","article-title":"Powerful Statistical Inference for Nested Data Using Sufficient Summary Statistics.","volume":"12","author":"I Dowding","year":"2018","journal-title":"Front Hum Neurosci."},{"key":"pcbi.1010061.ref007","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1007\/s10654-016-0149-3","article-title":"Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations","volume":"31","author":"S Greenland","year":"2016","journal-title":"Eur J Epidemiol"},{"key":"pcbi.1010061.ref008","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-73186-5","volume-title":"Handbook of multilevel analysis","author":"J de Leeuw","year":"2008"},{"key":"pcbi.1010061.ref009","first-page":"401","volume-title":"Resampling Multilevel Models","author":"R van der Leeden","year":"2008"},{"key":"pcbi.1010061.ref010","first-page":"86","article-title":"Sufficient Sample Sizes for Multilevel Modeling.","volume":"1","author":"CJM Maas","year":"2005","journal-title":"Methodol Eur J Res Methods Behav Soc Sci."},{"key":"pcbi.1010061.ref011","doi-asserted-by":"crossref","first-page":"1141","DOI":"10.1111\/2041-210X.13434","article-title":"Robustness of linear mixed-effects models to violations of distributional assumptions.","volume":"11","author":"H Schielzeth","year":"2020","journal-title":"Methods Ecol Evol"},{"key":"pcbi.1010061.ref012","doi-asserted-by":"crossref","first-page":"3151","DOI":"10.1080\/00949655.2018.1504945","article-title":"The effect of number of clusters and cluster size on statistical power and Type I error rates when testing random effects variance components in multilevel linear and logistic regression models.","volume":"88","author":"PC Austin","year":"2018","journal-title":"J Stat Comput Simul"},{"key":"pcbi.1010061.ref013","doi-asserted-by":"crossref","first-page":"1319","DOI":"10.1002\/(SICI)1097-0258(20000530)19:10<1319::AID-SIM490>3.0.CO;2-0","article-title":"Pros and cons of permutation tests in clinical trials.","volume":"19","author":"VW Berger","year":"2000","journal-title":"Stat Med."},{"key":"pcbi.1010061.ref014","doi-asserted-by":"crossref","first-page":"19151","DOI":"10.1073\/pnas.1915454117","article-title":"When possible, report a Fisher-exact P value and display its underlying null randomization distribution","volume":"117","author":"M -a. C Bind","year":"2020","journal-title":"Proc Natl Acad Sci"},{"key":"pcbi.1010061.ref015","article-title":"Permutation Tests at Nonparametric Rates.","author":"M Bertanha","year":"2021","journal-title":"ArXiv210213638 Econ Math Stat."},{"key":"pcbi.1010061.ref016","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1016\/S0167-7152(97)00043-6","article-title":"Studentized permutation tests for non-i.i.d. hypotheses and the generalized Behrens-Fisher problem","volume":"36","author":"A. Janssen","year":"1997","journal-title":"Stat Probab Lett"},{"key":"pcbi.1010061.ref017","doi-asserted-by":"crossref","first-page":"266","DOI":"10.1016\/0197-2456(93)90225-3","article-title":"Tightening the clinical trial.","volume":"14","author":"JW Tukey","year":"1993","journal-title":"Control Clin Trials"},{"key":"pcbi.1010061.ref018","doi-asserted-by":"crossref","DOI":"10.22237\/jmasm\/1020255120","article-title":"Parametric Analyses In Randomized Clinical Trials.","volume":"1","author":"V Berger","year":"2002","journal-title":"J Mod Appl Stat Methods."},{"key":"pcbi.1010061.ref019","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1038\/s41586-020-2649-2","article-title":"Array programming with NumPy","volume":"585","author":"CR Harris","year":"2020","journal-title":"Nature"},{"key":"pcbi.1010061.ref020","first-page":"1","volume-title":"Numba: a LLVM-based Python JIT compiler. Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC","author":"SK Lam","year":"2015"},{"key":"pcbi.1010061.ref021","volume-title":"When Should You Adjust Standard Errors for Clustering?","author":"National Bureau of Economic Research","year":"2017"},{"key":"pcbi.1010061.ref022","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1016\/j.neuroimage.2015.05.092","article-title":"Multi-level block permutation.","volume":"123","author":"AM Winkler","year":"2015","journal-title":"NeuroImage"},{"key":"pcbi.1010061.ref023","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1214\/aos\/1176344552","article-title":"Bootstrap Methods: Another Look at the Jackknife.","volume":"7","author":"B. Efron","year":"1979","journal-title":"Ann Stat."},{"key":"pcbi.1010061.ref024","doi-asserted-by":"crossref","first-page":"1268","DOI":"10.2307\/2532271","article-title":"Bootstrap Hypothesis Testing Procedures.","volume":"49","author":"H Becher","year":"1993","journal-title":"Biometrics"},{"key":"pcbi.1010061.ref025","doi-asserted-by":"crossref","first-page":"687","DOI":"10.1080\/01621459.1988.10478649","article-title":"Prepivoting Test Statistics: A Bootstrap View of Asymptotic Refinements","volume":"83","author":"R. Beran","year":"1988","journal-title":"J Am Stat Assoc"},{"key":"pcbi.1010061.ref026","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1214\/aos\/1176345338","article-title":"The Bayesian Bootstrap.","volume":"9","author":"DB Rubin","year":"1981","journal-title":"Ann Stat"},{"key":"pcbi.1010061.ref027","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1007\/BF00058655","article-title":"Bagging predictors.","volume":"24","author":"L. Breiman","year":"1996","journal-title":"Mach Learn"},{"key":"pcbi.1010061.ref028","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1080\/00949650215733","article-title":"Permutation tests for multi-factorial analysis of variance.","volume":"73","author":"M Anderson","year":"2003","journal-title":"J Stat Comput Simul"},{"key":"pcbi.1010061.ref029","volume-title":"Randomization, Bootstrap and Monte Carlo Methods in Biology.","author":"BFJ Manly","year":"2017","edition":"3"},{"key":"pcbi.1010061.ref030","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1214\/13-AOS1090","article-title":"Exact and asymptotically robust permutation tests.","volume":"41","author":"E Chung","year":"2013","journal-title":"Ann Stat."},{"key":"pcbi.1010061.ref031","doi-asserted-by":"crossref","first-page":"1358","DOI":"10.1214\/12-EJS714","article-title":"A studentized permutation test for the nonparametric Behrens-Fisher problem in paired data.","volume":"6","author":"F Konietschke","year":"2012","journal-title":"Electron J Stat"},{"key":"pcbi.1010061.ref032","doi-asserted-by":"crossref","first-page":"5192","DOI":"10.1016\/j.csda.2006.05.024","article-title":"A studentized permutation test for the non-parametric Behrens\u2013Fisher problem.","volume":"51","author":"K Neubert","year":"2007","journal-title":"Comput Stat Data Anal"},{"key":"pcbi.1010061.ref033","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1007\/BF02741303","article-title":"A Monte Carlo comparison of studentized bootstrap and permutation tests for heteroscedastic two-sample problems.","volume":"20","author":"A Janssen","year":"2005","journal-title":"Comput Stat."},{"key":"pcbi.1010061.ref034","doi-asserted-by":"crossref","first-page":"1211","DOI":"10.1080\/01621459.2016.1202117","article-title":"Robust Permutation Tests For Correlation And Regression Coefficients","volume":"112","author":"CJ DiCiccio","year":"2017","journal-title":"J Am Stat Assoc"},{"key":"pcbi.1010061.ref035","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1080\/00031305.2016.1154108","article-title":"The ASA Statement on p-Values: Context, Process, and Purpose.","volume":"70","author":"RL Wasserstein","year":"2016","journal-title":"Am Stat."},{"key":"pcbi.1010061.ref036","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1111\/tri.13535","article-title":"To test or to estimate? P-values versus effect sizes.","volume":"33","author":"D Dunkler","year":"2020","journal-title":"Transpl Int."},{"key":"pcbi.1010061.ref037","doi-asserted-by":"crossref","first-page":"485","DOI":"10.1214\/aos\/1176347995","article-title":"Coverage Probabilities of Bootstrap-Confidence Intervals for Quantiles.","volume":"19","author":"M Falk","year":"1991","journal-title":"Ann Stat."},{"key":"pcbi.1010061.ref038","first-page":"1453","article-title":"On the Number of Bootstrap Simulations Required to Construct a Confidence Interval.","volume":"14","author":"P. Hall","year":"1986","journal-title":"Ann Stat."},{"key":"pcbi.1010061.ref039","doi-asserted-by":"crossref","first-page":"945","DOI":"10.1177\/0013164406288161","article-title":"Confidence Interval Coverage for Cohen\u2019s Effect Size Statistic.","volume":"66","author":"J Algina","year":"2006","journal-title":"Educ Psychol Meas."},{"key":"pcbi.1010061.ref040","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1016\/j.csda.2003.11.020","article-title":"A systematic comparison of methods for combining p-values from independent tests.","volume":"47","author":"TM Loughin","year":"2004","journal-title":"Comput Stat Data Anal"},{"key":"pcbi.1010061.ref041","doi-asserted-by":"crossref","first-page":"827","DOI":"10.1111\/1467-9868.00267","article-title":"Improved Small Sample Inference in the Mixed Linear Model: Bartlett Correction and Adjusted Likelihood.","volume":"62","author":"DM Zucker","year":"2000","journal-title":"J R Stat Soc Ser B Stat Methodol."},{"key":"pcbi.1010061.ref042","doi-asserted-by":"crossref","first-page":"1494","DOI":"10.3758\/s13428-016-0809-y","article-title":"Evaluating significance in linear mixed-effects models in R.","volume":"49","author":"SG Luke","year":"2017","journal-title":"Behav Res Methods"},{"key":"pcbi.1010061.ref043","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1016\/j.neuroimage.2014.01.060","article-title":"Permutation inference for the general linear model.","volume":"92","author":"AM Winkler","year":"2014","journal-title":"NeuroImage"},{"key":"pcbi.1010061.ref044","doi-asserted-by":"crossref","first-page":"d549","DOI":"10.1136\/bmj.d549","article-title":"Interpretation of random effects meta-analyses","volume":"342","author":"RD Riley","year":"2011","journal-title":"BMJ"},{"key":"pcbi.1010061.ref045","first-page":"127","article-title":"Why Permutation Tests Are Superior to t and F Tests in Biomedical Research.","volume":"52","author":"J Ludbrook","year":"1998","journal-title":"Am Stat"},{"key":"pcbi.1010061.ref046","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1093\/genetics\/142.1.285","article-title":"Permutation Tests for Multiple Loci Affecting a Quantitative Character","volume":"142","author":"RW Doerge","year":"1996","journal-title":"Genetics"},{"key":"pcbi.1010061.ref047","doi-asserted-by":"crossref","first-page":"2280","DOI":"10.1002\/pmic.201300361","article-title":"IQuant: An automated pipeline for quantitative proteomics based upon isobaric tags","volume":"14","author":"B Wen","year":"2014","journal-title":"PROTEOMICS"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1010061","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2022,5,12]],"date-time":"2022-05-12T00:00:00Z","timestamp":1652313600000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1010061","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,5,12]],"date-time":"2022-05-12T13:56:05Z","timestamp":1652363765000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1010061"}},"subtitle":[],"editor":[{"given":"Dina","family":"Schneidman-Duhovny","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2022,5,2]]},"references-count":47,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,5,2]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1010061","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.06.29.450439","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,2]]}}}