{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T09:40:52Z","timestamp":1790761252717,"version":"4.1.0"},"reference-count":45,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2023,3,9]],"date-time":"2023-03-09T00:00:00Z","timestamp":1678320000000},"content-version":"vor","delay-in-days":128,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/N031938\/1"],"award-info":[{"award-number":["EP\/N031938\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/P031447\/1"],"award-info":[{"award-number":["EP\/P031447\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/T02772X\/1"],"award-info":[{"award-number":["EP\/T02772X\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100010663","name":"H2020 European Research Council","doi-asserted-by":"publisher","award":["101019498"],"award-info":[{"award-number":["101019498"]}],"id":[{"id":"10.13039\/100010663","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["DMS-2015366"],"award-info":[{"award-number":["DMS-2015366"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,11,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>We study the problem of high-dimensional Principal Component Analysis (PCA) with missing observations. In a simple, homogeneous observation model, we show that an existing observed-proportion weighted (OPW) estimator of the leading principal components can (nearly) attain the minimax optimal rate of convergence, which exhibits an interesting phase transition. However, deeper investigation reveals that, particularly in more realistic settings where the observation probabilities are heterogeneous, the empirical performance of the OPW estimator can be unsatisfactory; moreover, in the noiseless case, it fails to provide exact recovery of the principal components. Our main contribution, then, is to introduce a new method, which we call primePCA, that is designed to cope with situations where observations may be missing in a heterogeneous manner. Starting from the OPW estimator, primePCA iteratively projects the observed entries of the data matrix onto the column space of our current estimate to impute the missing entries, and then updates our estimate by computing the leading right singular space of the imputed data matrix. We prove that the error of primePCA converges to zero at a geometric rate in the noiseless case, and when the signal strength is not too small. An important feature of our theoretical guarantees is that they depend on average, as opposed to worst-case, properties of the missingness mechanism. Our numerical studies on both simulated and real data reveal that primePCA exhibits very encouraging performance across a wide range of scenarios, including settings where the data are not Missing Completely At Random.<\/jats:p>","DOI":"10.1111\/rssb.12550","type":"journal-article","created":{"date-parts":[[2022,11,21]],"date-time":"2022-11-21T02:08:02Z","timestamp":1668996482000},"page":"2000-2031","source":"Crossref","is-referenced-by-count":37,"title":["High-Dimensional Principal Component Analysis with Heterogeneous Missingness"],"prefix":"10.1093","volume":"84","author":[{"given":"Ziwei","family":"Zhu","sequence":"first","affiliation":[{"name":"Statistical Laboratory , , Cambridge , UK"},{"name":"University of Cambridge , , Cambridge , UK"},{"name":"Department of Statistics , , Ann Arbor , Michigan USA"},{"name":"University of Michigan , , Ann Arbor , Michigan USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tengyao","family":"Wang","sequence":"additional","affiliation":[{"name":"Statistical Laboratory , , Cambridge , UK"},{"name":"University of Cambridge , , Cambridge , UK"},{"name":"Department of Statistics , , London , UK"},{"name":"London School of Economics , , London , UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Richard J.","family":"Samworth","sequence":"additional","affiliation":[{"name":"Statistical Laboratory , , Cambridge , UK"},{"name":"University of Cambridge , , Cambridge , UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2022,11,20]]},"reference":[{"key":"2023030906295745100_","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1080\/01621459.1957.10501379","article-title":"Maximum likelihood estimates for a multivariate normal distribution when some observations are missing","volume":"52","author":"Anderson","year":"1957","journal-title":"Journal of the American Statistical Association"},{"key":"2023030906295745100_","doi-asserted-by":"crossref","first-page":"i","DOI":"10.1002\/j.2333-8504.1964.tb00689.x","article-title":"The use of special matrix operators in statistical calculus","volume":"2","author":"Beaton","year":"1964","journal-title":"ETS Research Bulletin Series"},{"key":"2023030906295745100_","doi-asserted-by":"crossref","first-page":"939","DOI":"10.1111\/rssb.12196","article-title":"Linear and conic programming estimators in high dimensional errors-in-variables models","volume":"79","author":"Belloni","year":"2017","journal-title":"Journal of the Royal Statistical Society. Series B: Statistical Methodology"},{"key":"2023030906295745100_","first-page":"3074","article-title":"Sparse PCA: optimal rates and adaptive estimation","volume":"41","author":"Cai","year":"2013","journal-title":"The Annals of\u2009Statistics"},{"key":"2023030906295745100_","first-page":"55","article-title":"Minimax rate-optimal estimation of high-dimensional covariance matrices with incomplete data","volume":"150","author":"Cai","year":"2016","journal-title":"The\u2009Journal\u2009of\u2009Multivariate\u2009Analysis"},{"key":"2023030906295745100_","volume-title":"High-dimensional linear discriminant analysis: optimality, adaptive algorithm, and missing data. arXiv:1804.03018","author":"Cai","year":"2018"},{"key":"2023030906295745100_","doi-asserted-by":"crossref","first-page":"11:1","DOI":"10.1145\/1970392.1970395","article-title":"Robust principal component analysis?","volume":"58","author":"Cand\u00e8s","year":"2011","journal-title":"Journal of the ACM"},{"key":"2023030906295745100_","doi-asserted-by":"crossref","first-page":"925","DOI":"10.1109\/JPROC.2009.2035722","article-title":"Matrix completion with noise","volume":"98","author":"Cand\u00e8s","year":"2010","journal-title":"Proceedings of the IEEE"},{"key":"2023030906295745100_","doi-asserted-by":"crossref","first-page":"717","DOI":"10.1007\/s10208-009-9045-5","article-title":"Exact matrix completion via convex optimization","volume":"9","author":"Cand\u00e8s","year":"2009","journal-title":"Foundations of\u2009Computational Mathematics (FoCM)"},{"key":"2023030906295745100_","volume-title":"Nonconvex optimization meets low-rank matrix factorization: an overview. arXiv preprint arXiv:1809.09573","author":"Chi","year":"2018"},{"key":"2023030906295745100_","first-page":"1921","article-title":"Asymptotic theory for estimating the singular vectors and values of a partially-observed low rank matrix with noise","volume":"27","author":"Cho","year":"2017","journal-title":"Statistica Sinica"},{"key":"2023030906295745100_","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1137\/0707001","article-title":"The rotation of eigenvectors by a perturbation III","volume":"7","author":"Davis","year":"1970","journal-title":"The\u2009SIAM\u2009Journal on Numerical Analysis"},{"key":"2023030906295745100_","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/j.2517-6161.1977.tb01600.x","article-title":"Maximum likelihood from incomplete data via the EM algorithm","volume":"39","author":"Dempster","year":"1977","journal-title":"Journal\u2009of the Royal\u2009Statistical Society. Series B: Statistical\u2009Methodology"},{"key":"2023030906295745100_","doi-asserted-by":"crossref","first-page":"657","DOI":"10.1007\/s11258-014-0406-z","article-title":"Principal component analysis with missing values: a comparative survey of methods","volume":"216","author":"Dray","year":"2015","journal-title":"Plant Ecology"},{"key":"2023030906295745100_","author":"Elsener","year":"2018"},{"key":"2023030906295745100_","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1111\/rssb.12016","article-title":"Large covariance estimation by thresholding principal orthogonal complements","volume":"75","author":"Fan","year":"2013","journal-title":"Journal of the Royal Statistical Society. Series B: Statistical Methodology"},{"key":"2023030906295745100_","first-page":"185","volume-title":"Incomplete data in sample surveys, Vol. 2: theory and bibliographies","author":"Ford","year":"1983"},{"key":"2023030906295745100_","first-page":"1","article-title":"Achieving optimal misclassification proportion in stochastic block models","volume":"18","author":"Gao","year":"2016","journal-title":"Journal of Machine Learning Research"},{"key":"2023030906295745100_","first-page":"3367","article-title":"Matrix completion and low-rank SVD via fast alternating least squares","volume":"16","author":"Hastie","year":"2015","journal-title":"Journal of Machine Learning Research"},{"key":"2023030906295745100_","doi-asserted-by":"crossref","first-page":"682","DOI":"10.1198\/jasa.2009.0121","article-title":"On consistency and sparsity for principal components analysis in high dimensions","volume":"104","author":"Johnstone","year":"2009","journal-title":"Journal of the American Statistical Association"},{"key":"2023030906295745100_","first-page":"1","article-title":"Handling missing values in exploratory multivariate data analysis methods","volume":"153","author":"Josse","year":"2012","journal-title":"Journal de la\u2009soci\u00e9t\u00e9 fran\u00e7aise de statistique"},{"key":"2023030906295745100_","first-page":"28","article-title":"Gestion des donne\u00e9s manquantes en analyse en composantes principales","volume":"150","author":"Josse","year":"2009","journal-title":"Journal de la\u2009soci\u00e9t\u00e9 fran\u00e7aise de statistique"},{"key":"2023030906295745100_","doi-asserted-by":"crossref","first-page":"2980","DOI":"10.1109\/TIT.2010.2046205","article-title":"Matrix completion from a few entries","volume":"56","author":"Keshavan","year":"2010","journal-title":"IEEE Transactions on Information Theory"},{"key":"2023030906295745100_","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1007\/BF02295279","article-title":"Weighted least squares fitting using ordinary least squares algorithms","volume":"62","author":"Kiers","year":"1997","journal-title":"Psychometrika"},{"key":"2023030906295745100_","first-page":"2302","article-title":"Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion","volume":"39","author":"Koltchinskii","year":"2011","journal-title":"The Annals of\u2009Statistics"},{"key":"2023030906295745100_","volume-title":"Statistical analysis with missing data","author":"Little","year":"2019"},{"key":"2023030906295745100_","doi-asserted-by":"crossref","first-page":"1429","DOI":"10.1214\/18-EJS1427","article-title":"High-dimensional robust precision matrix estimation: cellwise corruption under \u03f5-contamination","volume":"12","author":"Loh","year":"2018","journal-title":"The Electronic Journal of Statistics"},{"key":"2023030906295745100_","first-page":"1637","article-title":"High-dimensional regression with noisy and missing data: provable guarantees with nonconvexity","volume":"40","author":"Loh","year":"2012","journal-title":"The Annals of\u2009Statistics"},{"key":"2023030906295745100_","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1007\/978-3-0348-0490-5_20","volume-title":"High dimensional probability VI","author":"Lounici","year":"2013"},{"key":"2023030906295745100_","doi-asserted-by":"crossref","first-page":"1029","DOI":"10.3150\/12-BEJ487","article-title":"High-dimensional covariance matrix estimation with missing observations","volume":"20","author":"Lounici","year":"2014","journal-title":"Bernoulli"},{"key":"2023030906295745100_","first-page":"2287","article-title":"Spectral regularization algorithms for learning large incomplete matrices","volume":"11","author":"Mazumder","year":"2010","journal-title":"Journal of Machine Learning Research"},{"key":"2023030906295745100_","first-page":"1665","article-title":"Restricted strong convexity and weighted matrix completion: optimal bounds with noise","volume":"13","author":"Negahban","year":"2012","journal-title":"Journal of Machine Learning Research"},{"key":"2023030906295745100_","first-page":"1617","article-title":"Asymptotics of sample eigenstructure for a large dimensional spiked covariance model","volume":"17","author":"Paul","year":"2007","journal-title":"Statistica Sinica"},{"key":"2023030906295745100_","first-page":"1878","article-title":"Spectral clustering and the high-dimensional stochastic blockmodel","volume":"39","author":"Rohe","year":"2011","journal-title":"The Annals of\u2009Statistics"},{"key":"2023030906295745100_","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1093\/biomet\/63.3.581","article-title":"Inference and missing data","volume":"63","author":"Rubin","year":"1976","journal-title":"Biometrika"},{"key":"2023030906295745100_","volume-title":"Multiple imputation for nonresponse in surveys","author":"Rubin","year":"2004"},{"key":"2023030906295745100_","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/BF02289451","article-title":"A generalized solution of the orthogonal Procrustes problem","volume":"31","author":"Sch\u00f6nemann","year":"1966","journal-title":"Psychometrika"},{"key":"2023030906295745100_","first-page":"257","article-title":"What is meant by \u201cmissing at random\u201d?","volume":"28","author":"Seaman","year":"2013","journal-title":"Statistical\u2009Science"},{"key":"2023030906295745100_","first-page":"1747","article-title":"The statistics and mathematics of high dimension low sample size asymptotics","volume":"26","author":"Shen","year":"2016","journal-title":"Statistica Sinica"},{"key":"2023030906295745100_","doi-asserted-by":"crossref","DOI":"10.1017\/9781108231596","volume-title":"High-dimensional probability: an introduction with applications in data science","author":"Vershynin","year":"2018"},{"key":"2023030906295745100_","volume-title":"Spectral methods and computational trade-offs in high-dimensional statistical inference","author":"Wang","year":"2016"},{"key":"2023030906295745100_","first-page":"1342","article-title":"Asymptotics of empirical eigen-structure for ultra-high dimensional spiked covariance model","volume":"45","author":"Wang","year":"2017","journal-title":"The Annals of\u2009Statistics"},{"key":"2023030906295745100_","first-page":"29","article-title":"Nonlinear iterative partial least squares (NIPALS) estimation procedures","volume":"43","author":"Wold","year":"1969","journal-title":"Bulletin\u2009of the International Statistical Institute"},{"key":"2023030906295745100_","volume-title":"Heteroskedastic PCA: algorithm, optimality, and applications. arXiv:1810.08316","author":"Zhang","year":"2018"},{"key":"2023030906295745100_","volume-title":"primePCA: projected refinement for imputation of missing entries in principal component analysis. R package, version 1.2. Available from","author":"Zhu","year":"2019"}],"container-title":["Journal of the Royal Statistical Society Series B: Statistical Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/rssb.12550","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1111\/rssb.12550","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/84\/5\/2000\/49458913\/jrsssb_84_5_2000.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/84\/5\/2000\/49458913\/jrsssb_84_5_2000.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,9]],"date-time":"2024-10-09T01:06:26Z","timestamp":1728435986000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jrsssb\/article\/84\/5\/2000\/7072923"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,1]]},"references-count":45,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,11,20]]},"published-print":{"date-parts":[[2022,11,1]]}},"URL":"https:\/\/doi.org\/10.1111\/rssb.12550","relation":{},"ISSN":["1369-7412","1467-9868"],"issn-type":[{"value":"1369-7412","type":"print"},{"value":"1467-9868","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,11]]},"published":{"date-parts":[[2022,11,1]]}}}