{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T20:18:36Z","timestamp":1790626716033,"version":"4.1.0"},"reference-count":41,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2015,7,7]],"date-time":"2015-07-07T00:00:00Z","timestamp":1436227200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"Agency for Children and Families of the US Department of Health and Human Services","award":["90YR0049\/02"],"award-info":[{"award-number":["90YR0049\/02"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2016,6,1]]},"abstract":"<jats:title>Summary<\/jats:title>\n                  <jats:p>Applied researchers are increasingly interested in whether and how treatment effects vary in randomized evaluations, especially variation that is not explained by observed covariates. We propose a model-free approach for testing for the presence of such unexplained variation. To use this randomization-based approach, we must address the fact that the average treatment effect, which is generally the object of interest in randomized experiments, actually acts as a nuisance parameter in this setting. We explore potential solutions and advocate for a method that guarantees valid tests in finite samples despite this nuisance. We also show how this method readily extends to testing for heterogeneity beyond a given model, which can be useful for assessing the sufficiency of a given scientific theory. We finally apply our method to the National Head Start impact study, which is a large-scale randomized evaluation of a Federal preschool programme, finding that there is indeed significant unexplained treatment effect variation.<\/jats:p>","DOI":"10.1111\/rssb.12124","type":"journal-article","created":{"date-parts":[[2015,7,7]],"date-time":"2015-07-07T07:18:56Z","timestamp":1436253536000},"page":"655-671","source":"Crossref","is-referenced-by-count":88,"title":["Randomization Inference for Treatment Effect Variation"],"prefix":"10.1093","volume":"78","author":[{"given":"Peng","family":"Ding","sequence":"first","affiliation":[{"name":"Harvard University , Cambridge , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Avi","family":"Feller","sequence":"additional","affiliation":[{"name":"Harvard University , Cambridge , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luke","family":"Miratrix","sequence":"additional","affiliation":[{"name":"Harvard University , Cambridge , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2015,7,7]]},"reference":[{"key":"2023021711192646300_rssb12124-cit-0001","doi-asserted-by":"crossref","first-page":"284","DOI":"10.1198\/016214502753479419","article-title":"Bootstrap tests for distributional treatment effects in instrumental variable models","volume":"97","author":"Abadie","year":"2002","journal-title":"J. Am. Statist. Ass."},{"key":"2023021711192646300_rssb12124-cit-0002","first-page":"63","article-title":"Goodness-of-fit tests when parameters are estimated","volume":"66","author":"Babu","year":"2004","journal-title":"Sankhya A"},{"key":"2023021711192646300_rssb12124-cit-0003","doi-asserted-by":"crossref","first-page":"1012","DOI":"10.1080\/01621459.1994.10476836","article-title":"P values maximized over a confidence set for the nuisance parameter","volume":"89","author":"Berger","year":"1994","journal-title":"J. Am. Statist. Ass."},{"key":"2023021711192646300_rssb12124-cit-0004","doi-asserted-by":"crossref","first-page":"371","DOI":"10.1214\/07-AOAS124","article-title":"Interpretation of interaction: a review","volume":"1","author":"Berrington de Gonz\u00e1lez","year":"2007","journal-title":"Ann. Appl. Statist."},{"key":"2023021711192646300_rssb12124-cit-0005","volume-title":"Experimental evidence on distributional effects of Head Start","author":"Bitler","year":"2013"},{"key":"2023021711192646300_rssb12124-cit-0006","doi-asserted-by":"crossref","first-page":"988","DOI":"10.1257\/aer.96.4.988","article-title":"What mean impacts miss: distributional effects of welfare reform experiments","volume":"96","author":"Bitler","year":"2006","journal-title":"Am. Econ. Rev."},{"key":"2023021711192646300_rssb12124-cit-0007","volume-title":"Can variation in subgroups' average treatment effects explain treatment effect heterogeneity?: evidence from a social experiment","author":"Bitler","year":"2010"},{"key":"2023021711192646300_rssb12124-cit-0008","volume-title":"Quantifying variation in Head Start effects on young children's cognitive and socio-emotional skills using data from the Head Start Impact Study","author":"Bloom","year":"2015"},{"key":"2023021711192646300_rssb12124-cit-0009","first-page":"253","article-title":"Subsampling inference on quantile regression processes","volume":"67","author":"Chernozhukov","year":"2005","journal-title":"Sankhya"},{"key":"2023021711192646300_rssb12124-cit-0010","doi-asserted-by":"crossref","first-page":"1","DOI":"10.2307\/1403235","article-title":"Interaction","volume":"52","author":"Cox","year":"1984","journal-title":"Int. Statist. Rev."},{"key":"2023021711192646300_rssb12124-cit-0011","doi-asserted-by":"crossref","first-page":"389","DOI":"10.1162\/rest.90.3.389","article-title":"Nonparametric tests for treatment effect heterogeneity","volume":"90","author":"Crump","year":"2008","journal-title":"Rev. Econ. Statist."},{"key":"2023021711192646300_rssb12124-cit-0012","volume-title":"A paradox from randomization-based causal inference","author":"Ding","year":"2014"},{"key":"2023021711192646300_rssb12124-cit-00100","volume-title":"Technical Report","author":"Ding","year":"2015"},{"key":"2023021711192646300_rssb12124-cit-0013","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1016\/j.jeconom.2008.05.012","article-title":"Heterogeneous impacts in PROGRESA","volume":"145","author":"Djebbari","year":"2008","journal-title":"J. Econmetr."},{"key":"2023021711192646300_rssb12124-cit-0014","doi-asserted-by":"crossref","first-page":"421","DOI":"10.1093\/biomet\/63.3.421","article-title":"Plotting with confidence: graphical comparisons of two populations","volume":"63","author":"Doksum","year":"1976","journal-title":"Biometrika"},{"key":"2023021711192646300_rssb12124-cit-0015","doi-asserted-by":"crossref","DOI":"10.1137\/1.9781611970586","volume-title":"Distribution Theory for Tests Based on the Sample Distribution Function","author":"Durbin","year":"1973"},{"key":"2023021711192646300_rssb12124-cit-0016","volume-title":"Compared to what?: variations in the impacts of Head Start by alternative care-type settings","author":"Feller","year":"2014"},{"key":"2023021711192646300_rssb12124-cit-0017","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1111\/j.1468-0262.2007.00738.x","article-title":"Efficient semiparametric estimation of quantile treatment effects","volume":"75","author":"Firpo","year":"2007","journal-title":"Econometrica"},{"key":"2023021711192646300_rssb12124-cit-0018","doi-asserted-by":"crossref","first-page":"953","DOI":"10.3982\/ECTA6822","article-title":"Unconditional quantile regressions","volume":"77","author":"Firpo","year":"2009","journal-title":"Econometrica"},{"key":"2023021711192646300_rssb12124-cit-0019","volume-title":"The Design of Experiments","author":"Fisher","year":"1935"},{"key":"2023021711192646300_rssb12124-cit-0020","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1002\/0470090456","volume-title":"Applied Bayesian Modeling and Causal Inference from an Incomplete Data Perspective","author":"Gelman","year":"2004"},{"key":"2023021711192646300_rssb12124-cit-0021","doi-asserted-by":"crossref","first-page":"487","DOI":"10.2307\/2971729","article-title":"Making the most out of programme evaluations and social experiments: accounting for heterogeneity in programme impacts","volume":"64","author":"Heckman","year":"1997","journal-title":"Rev. Econ. Stud."},{"key":"2023021711192646300_rssb12124-cit-0022","doi-asserted-by":"crossref","first-page":"33","DOI":"10.2307\/2344652","article-title":"Post stratification","volume":"142","author":"Holt","year":"1979","journal-title":"J. R. Statist. Soc. A"},{"key":"2023021711192646300_rssb12124-cit-0023","doi-asserted-by":"crossref","first-page":"1583","DOI":"10.1111\/1468-0262.00342","article-title":"Inference on the quantile regression process","volume":"70","author":"Koenker","year":"2002","journal-title":"Econometrica"},{"key":"2023021711192646300_rssb12124-cit-0024","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1214\/12-AOAS583","article-title":"Agnostic notes on regression adjustments to experimental data: reexamining Freedman's critique","volume":"7","author":"Lin","year":"2013","journal-title":"Ann. Appl. Statist."},{"key":"2023021711192646300_rssb12124-cit-0025","doi-asserted-by":"crossref","first-page":"735","DOI":"10.1111\/j.1467-937X.2005.00350.x","article-title":"Consistent testing for stochastic dominance under general sampling schemes","volume":"72","author":"Linton","year":"2005","journal-title":"Rev. Econ. Stud."},{"key":"2023021711192646300_rssb12124-cit-0026","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1111\/j.1467-9868.2012.01048.x","article-title":"Adjusting treatment effect estimates by post-stratification in randomized experiments","volume":"75","author":"Miratrix","year":"2013","journal-title":"J. R. Statist. Soc. B"},{"key":"2023021711192646300_rssb12124-cit-0027","doi-asserted-by":"crossref","first-page":"1263","DOI":"10.1214\/12-AOS1008","article-title":"Rerandomization to improve covariate balance in experiments","volume":"40","author":"Morgan","year":"2012","journal-title":"Ann. Statist."},{"key":"2023021711192646300_rssb12124-cit-0028","first-page":"465","article-title":"On the application of probability theory to agricultural experiments","volume":"5","author":"Neyman","year":"1990","journal-title":"Statist. Sci."},{"key":"2023021711192646300_rssb12124-cit-00101","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1198\/jasa.2011.tm10356","article-title":"Randomization-based inference within principal strata","volume":"106","author":"Nolen","year":"2011","journal-title":"J. Am. Statist. Ass."},{"key":"2023021711192646300_rssb12124-cit-0029","first-page":"195","article-title":"Generalizing from unrepresentative experiments: a stratified propensity score approach","volume":"63","author":"O'Muircheartaigh","year":"2014","journal-title":"Appl. Statist."},{"key":"2023021711192646300_rssb12124-cit-0030","volume-title":"Head Start Impact Study: Final Report","author":"Puma","year":"2010"},{"key":"2023021711192646300_rssb12124-cit-0031","doi-asserted-by":"crossref","first-page":"560","DOI":"10.1111\/j.0006-341X.1999.00560.x","article-title":"Reduced sensitivity to hidden bias at upper quantiles in observational studies with dilated treatment effects","volume":"55","author":"Rosenbaum","year":"1999","journal-title":"Biometrics"},{"key":"2023021711192646300_rssb12124-cit-0032","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1093\/biomet\/88.1.219","article-title":"Effects attributable to treatment: inference in experiments and observational studies with a discrete pivot","volume":"88","author":"Rosenbaum","year":"2001","journal-title":"Biometrika"},{"key":"2023021711192646300_rssb12124-cit-0033","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4757-3692-2","volume-title":"Observational Studies","author":"Rosenbaum","year":"2002"},{"key":"2023021711192646300_rssb12124-cit-0034","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1214\/ss\/1042727942","article-title":"Covariance adjustment in randomized experiments and observational studies","volume":"17","author":"Rosenbaum","year":"2002","journal-title":"Statist. Sci."},{"key":"2023021711192646300_rssb12124-cit-0035","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4419-1213-8","volume-title":"Design of Observational Studies","author":"Rosenbaum","year":"2010"},{"key":"2023021711192646300_rssb12124-cit-0036","doi-asserted-by":"crossref","first-page":"1017","DOI":"10.1111\/j.1541-0420.2010.01535.x","article-title":"A new u-statistic with superior design sensitivity in matched observational studies","volume":"67","author":"Rosenbaum","year":"2011","journal-title":"Biometrics"},{"key":"2023021711192646300_rssb12124-cit-0037","doi-asserted-by":"crossref","first-page":"688","DOI":"10.1037\/h0037350","article-title":"Estimating causal effects of treatments in randomized and nonrandomized studies","volume":"66","author":"Rubin","year":"1974","journal-title":"J. Educ. Psychol."},{"key":"2023021711192646300_rssb12124-cit-0038","first-page":"591","article-title":"Comment on \u201cRandomization analysis of experimental data: the Fisher Randomization Test\u201d","volume":"75","author":"Rubin","year":"1980","journal-title":"J. Am. Statist. Ass."},{"key":"2023021711192646300_rssb12124-cit-0039","doi-asserted-by":"crossref","first-page":"322","DOI":"10.1198\/016214504000001880","article-title":"Causal inference using potential outcomes: design, modeling, decisions","volume":"100","author":"Rubin","year":"2005","journal-title":"J. Am. Statist. Ass."}],"container-title":["Journal of the Royal Statistical Society Series B: Statistical Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1111%2Frssb.12124","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/78\/3\/655\/49236389\/jrsssb_78_3_655.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/78\/3\/655\/49236389\/jrsssb_78_3_655.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,9]],"date-time":"2024-06-09T17:56:10Z","timestamp":1717955770000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jrsssb\/article\/78\/3\/655\/7040906"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,7,7]]},"references-count":41,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2015,7,7]]},"published-print":{"date-parts":[[2016,6,1]]}},"URL":"https:\/\/doi.org\/10.1111\/rssb.12124","relation":{},"ISSN":["1369-7412","1467-9868"],"issn-type":[{"value":"1369-7412","type":"print"},{"value":"1467-9868","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2016,6]]},"published":{"date-parts":[[2015,7,7]]}}}