{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T17:01:57Z","timestamp":1791392517989,"version":"4.3.3"},"reference-count":41,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2018,11,8]],"date-time":"2018-11-08T00:00:00Z","timestamp":1541635200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,2,1]]},"abstract":"<jats:title>Summary<\/jats:title>\n                  <jats:p>Factor analysis and principal component analysis are used in many application areas. The first step, choosing the number of components, remains a serious challenge. Our work proposes improved methods for this important problem. One of the most popular state of the art methods is parallel analysis (PA), which compares the observed factor strengths with simulated strengths under a noise-only model. The paper proposes improvements to PA. We first derandomize it, proposing deterministic PA, which is faster and more reproducible than PA. Both PA and deterministic PA are prone to a shadowing phenomenon in which a strong factor makes it difficult to detect smaller but more interesting factors. We propose deflation to counter shadowing. We also propose to raise the decision threshold to improve estimation accuracy. We prove several consistency results for our methods, and test them in simulations. We also illustrate our methods on data from the human genome diversity project, where they significantly improve the accuracy.<\/jats:p>","DOI":"10.1111\/rssb.12301","type":"journal-article","created":{"date-parts":[[2018,11,8]],"date-time":"2018-11-08T09:00:11Z","timestamp":1541667611000},"page":"163-183","source":"Crossref","is-referenced-by-count":65,"title":["Deterministic Parallel Analysis: An Improved Method for Selecting Factors and Principal Components"],"prefix":"10.1093","volume":"81","author":[{"given":"Edgar","family":"Dobriban","sequence":"first","affiliation":[{"name":"University of Pennsylvania , Philadelphia , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Art B.","family":"Owen","sequence":"additional","affiliation":[{"name":"Stanford University \u00a0 USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2018,11,8]]},"reference":[{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"1203","DOI":"10.3982\/ECTA8968","article-title":"Eigenvalue ratio test for the number of factors","volume":"81","author":"Ahn","year":"2013","journal-title":"Econometrica"},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1214\/11-AOS966","article-title":"Statistical analysis of factor models of high dimension","volume":"40","author":"Bai","year":"2012","journal-title":"Ann. Statist."},{"key":"2023022002401374500_","doi-asserted-by":"crossref","DOI":"10.1561\/0800000002","volume-title":"Large Dimensional Factor Analysis","author":"Bai","year":"2008"},{"key":"2023022002401374500_","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4419-0661-8","volume-title":"Spectral Analysis of Large Dimensional Random Matrices","author":"Bai","year":"2010"},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"1643","DOI":"10.1214\/009117905000000233","article-title":"Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices","volume":"33","author":"Baik","year":"2005","journal-title":"Ann. Probab."},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1016\/j.jmva.2012.04.019","article-title":"The singular values and vectors of low rank perturbations of large rectangular random matrices","volume":"111","author":"Benaych-George","year":"2012","journal-title":"J. Multiv. Anal."},{"key":"2023022002401374500_","volume-title":"Algorithms for Minimization without Derivatives","author":"Brent","year":"1973"},{"key":"2023022002401374500_","volume-title":"Confirmatory Factor Analysis for Applied Research","author":"Brown","year":"2014"},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1207\/s15327906mbr2704_2","article-title":"Remarks on parallel analysis","volume":"27","author":"Buja","year":"1992","journal-title":"Multiv. Behav. Res."},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1126\/science.296.5566.261b","article-title":"A human genome diversity cell line panel","volume":"296","author":"Cann","year":"2002","journal-title":"Science"},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1207\/s15327906mbr0102_10","article-title":"The scree test for the number of factors","volume":"1","author":"Cattell","year":"1966","journal-title":"Multiv. Behav. Res."},{"key":"2023022002401374500_","article-title":"Efficient computation of limit spectra of sample covariance matrices","volume":"4","author":"Dobriban","year":"2015","journal-title":"Rand Matr. Theory Appl."},{"key":"2023022002401374500_","volume-title":"Factor selection by permutation","author":"Dobriban","year":"2017"},{"key":"2023022002401374500_","volume-title":"Optimal prediction in the linearly transformed spiked model","author":"Dobriban","year":"2017"},{"key":"2023022002401374500_","volume-title":"Removing unwanted variation from high dimensional data with negative controls","author":"Gagnon-Bartsch","year":"2013"},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"5040","DOI":"10.1109\/TIT.2014.2323359","article-title":"The optimal hard threshold for singular values is 4\/sqrt(3)","volume":"60","author":"Gavish","year":"2014","journal-title":"IEEE Trans. Inform. Theory"},{"key":"2023022002401374500_","volume-title":"Unifying and generalizing methods for removing unwanted variation based on negative controls","author":"Gerard","year":"2017"},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"377","DOI":"10.1177\/0013164495055003002","article-title":"An improvement on Horn\u2019s parallel analysis methodology for selecting the correct number of factors to retain","volume":"55","author":"Glorfeld","year":"1995","journal-title":"Educ. Psychol. Measmnt"},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"2264","DOI":"10.1214\/15-AOP1022","article-title":"Large complex correlated Wishart matrices: fluctuations and asymptotic independence at the edges","volume":"44","author":"Hachem","year":"2016","journal-title":"Ann. Probab."},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1007\/BF02289447","article-title":"A rationale and test for the number of factors in factor analysis","volume":"30","author":"Horn","year":"1965","journal-title":"Psychometrika"},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1214\/aos\/1009210544","article-title":"On the distribution of the largest eigenvalue in principal components analysis","volume":"29","author":"Johnstone","year":"2001","journal-title":"Ann. Statist."},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1177\/001316446002000116","article-title":"The application of electronic computers to factor analysis","volume":"20","author":"Kaiser","year":"1960","journal-title":"Educ. Psychol. Measmnt"},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"3786","DOI":"10.1214\/16-AAP1193","article-title":"Tracy\u2013Widom distribution for the largest eigenvalue of real sample covariance matrices with general population","volume":"26","author":"Lee","year":"2016","journal-title":"Ann. Appl. Probab."},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"e161","DOI":"10.1371\/journal.pgen.0030161","article-title":"Capturing heterogeneity in gene expression studies by surrogate variable analysis","volume":"3","author":"Leek","year":"2007","journal-title":"PLOS Genet."},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"18718","DOI":"10.1073\/pnas.0808709105","article-title":"A general framework for multiple testing dependence","volume":"105","author":"Leek","year":"2008","journal-title":"Proc. Natn. Acad. Sci. USA"},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"1100","DOI":"10.1126\/science.1153717","article-title":"Worldwide human relationships inferred from genome-wide patterns of variation","volume":"319","author":"Li","year":"2008","journal-title":"Science"},{"key":"2023022002401374500_","volume-title":"Factor Analysis in Chemistry","author":"Malinowski","year":"2002"},{"key":"2023022002401374500_","first-page":"507","article-title":"Distribution of eigenvalues for some sets of random matrices","volume":"114","author":"Marchenko","year":"1967","journal-title":"Mat. Sb."},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"3002","DOI":"10.1109\/TIT.2014.2311661","article-title":"Optshrink: an algorithm for improved low-rank signal matrix denoising by optimal, data-driven singular value shrinkage","volume":"60","author":"Nadakuditi","year":"2014","journal-title":"IEEE Trans. Inform. Theory"},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"2625","DOI":"10.1109\/TSP.2008.917356","article-title":"Sample eigenvalue based detection of high-dimensional signals in white noise using relatively few samples","volume":"56","author":"Nadakuditi","year":"2008","journal-title":"IEEE Trans. Signl Process."},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1214\/15-STS539","article-title":"Bi-cross-validation for factor analysis","volume":"31","author":"Owen","year":"2016","journal-title":"Statist. Sci."},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"e190","DOI":"10.1371\/journal.pgen.0020190","article-title":"Population structure and eigenanalysis","volume":"2","author":"Patterson","year":"2006","journal-title":"PLOS Genet"},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jspi.2013.09.005","article-title":"Random matrix theory in statistics: a review","volume":"150","author":"Paul","year":"2014","journal-title":"J. Statist. Planng Inf."},{"key":"2023022002401374500_","volume-title":"Cross-validation for unsupervised learning","author":"Perry","year":"2009"},{"key":"2023022002401374500_","first-page":"603","article-title":"A rotation test to verify latent structure","volume":"11","author":"Perry","year":"2010","journal-title":"J. Mach. Learn. Res."},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1006\/jmva.1995.1058","article-title":"Analysis of the limiting spectral distribution of large dimensional random matrices","volume":"54","author":"Silverstein","year":"1995","journal-title":"J. Multiv. Anal."},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"1664","DOI":"10.1214\/12-AOAS561","article-title":"Multiple hypothesis testing adjusted for latent variables, with an application to the AGEMAP gene expression data","volume":"6","author":"Sun","year":"2012","journal-title":"Ann. Appl. Statist."},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"e2937","DOI":"10.1002\/cem.2937","article-title":"Selecting the number of factors in principal component analysis by permutation testing\u2014numerical and practical aspects","volume":"31","author":"Vitale","year":"2017","journal-title":"J. Chemometr."},{"key":"2023022002401374500_","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9781107588080","volume-title":"Large Sample Covariance Matrices and High-dimensional Data Analysis","author":"Yao","year":"2015"},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"439","DOI":"10.1111\/biom.12767","article-title":"Eigenvalue significance testing for genetic association","volume":"74","author":"Zhou","year":"2017","journal-title":"Biometrics"},{"key":"2023022002401374500_","doi-asserted-by":"crossref","first-page":"432","DOI":"10.1037\/0033-2909.99.3.432","article-title":"Comparison of five rules for determining the number of components to retain","volume":"99","author":"Zwick","year":"1986","journal-title":"Psychol. Bull."}],"container-title":["Journal of the Royal Statistical Society Series B: Statistical Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1111%2Frssb.12301","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/81\/1\/163\/49269756\/jrsssb_81_1_163.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/81\/1\/163\/49269756\/jrsssb_81_1_163.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,6]],"date-time":"2023-09-06T18:05:54Z","timestamp":1694023554000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jrsssb\/article\/81\/1\/163\/7048361"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,11,8]]},"references-count":41,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2018,11,8]]},"published-print":{"date-parts":[[2019,2,1]]}},"URL":"https:\/\/doi.org\/10.1111\/rssb.12301","relation":{},"ISSN":["1369-7412","1467-9868"],"issn-type":[{"value":"1369-7412","type":"print"},{"value":"1467-9868","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,2]]},"published":{"date-parts":[[2018,11,8]]}}}