{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T02:00:33Z","timestamp":1775613633776,"version":"3.50.1"},"reference-count":48,"publisher":"Wiley","issue":"6","license":[{"start":{"date-parts":[[2019,10,29]],"date-time":"2019-10-29T00:00:00Z","timestamp":1572307200000},"content-version":"vor","delay-in-days":58,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["J of Applied Econometrics"],"published-print":{"date-parts":[[2019,9]]},"abstract":"<jats:title>Summary<\/jats:title><jats:p>We consider two recent suggestions for how to perform an empirically motivated Monte Carlo study to help select a treatment effect estimator under unconfoundedness. We show theoretically that neither is likely to be informative except under restrictive conditions that are unlikely to be satisfied in many contexts. To test empirical relevance, we also apply the approaches to a real\u2010world setting where estimator performance is known. Both approaches are worse than random at selecting estimators that minimize absolute bias. They are better when selecting estimators that minimize mean squared error. However, using a simple bootstrap is at least as good and often better. For now, researchers would be best advised to use a range of estimators and compare estimates for robustness.<\/jats:p>","DOI":"10.1002\/jae.2724","type":"journal-article","created":{"date-parts":[[2019,6,25]],"date-time":"2019-06-25T20:33:34Z","timestamp":1561494814000},"page":"893-910","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Mostly harmless simulations? Using Monte Carlo studies for estimator selection"],"prefix":"10.1002","volume":"34","author":[{"given":"Arun","family":"Advani","sequence":"first","affiliation":[{"name":"Department of Economics University of Warwick  Warwick UK"},{"name":"Centre for Competitive Advantage in the Global Economy University of Warwick  Warwick UK"},{"name":"Institute for Fiscal Studies  London UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Toru","family":"Kitagawa","sequence":"additional","affiliation":[{"name":"Department of Economics University College London  London UK"},{"name":"Centre for Microdata Methods and Practice  London UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tymon","family":"S\u0142oczy\u0144ski","sequence":"additional","affiliation":[{"name":"Department of Economics and International Business School Brandeis University  Waltham Massachusetts United States"},{"name":"IZA\u2014Institute of Labor Economics  Bonn Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"311","published-online":{"date-parts":[[2019,10,29]]},"reference":[{"key":"e_1_2_9_2_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-economics-080217-053402"},{"key":"e_1_2_9_3_1","doi-asserted-by":"publisher","DOI":"10.1177\/1536867X0400400307"},{"key":"e_1_2_9_4_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1468-0262.2006.00655.x"},{"key":"e_1_2_9_5_1","doi-asserted-by":"publisher","DOI":"10.1198\/jbes.2009.07333"},{"key":"e_1_2_9_6_1","doi-asserted-by":"publisher","DOI":"10.3982\/ECTA11293"},{"key":"e_1_2_9_7_1","doi-asserted-by":"publisher","DOI":"10.1002\/sim.3854"},{"key":"e_1_2_9_8_1","doi-asserted-by":"publisher","DOI":"10.1162\/003355304772839588"},{"key":"e_1_2_9_9_1","doi-asserted-by":"publisher","DOI":"10.1353\/jhr.2009.0009"},{"key":"e_1_2_9_10_1","doi-asserted-by":"publisher","DOI":"10.1080\/07350015.2018.1476247"},{"key":"e_1_2_9_11_1","doi-asserted-by":"publisher","DOI":"10.1515\/jem-2017-0005"},{"key":"e_1_2_9_12_1","unstructured":"Busso M. DiNardo J. &McCrary J(2009).Finite sample properties of semi\u2010parametric estimators of average treatment effects. Unpublished manuscript."},{"key":"e_1_2_9_13_1","doi-asserted-by":"publisher","DOI":"10.1162\/REST_a_00431"},{"key":"e_1_2_9_14_1","doi-asserted-by":"publisher","DOI":"10.1086\/692397"},{"key":"e_1_2_9_15_1","doi-asserted-by":"publisher","DOI":"10.1162\/rest.90.3.414"},{"key":"e_1_2_9_16_1","doi-asserted-by":"publisher","DOI":"10.1214\/009053607000000947"},{"key":"e_1_2_9_17_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1999.10473858"},{"key":"e_1_2_9_18_1","doi-asserted-by":"publisher","DOI":"10.1162\/003465302317331982"},{"key":"e_1_2_9_19_1","doi-asserted-by":"publisher","DOI":"10.1162\/REST_a_00318"},{"key":"e_1_2_9_20_1","doi-asserted-by":"publisher","DOI":"10.1162\/REST_a_00504"},{"key":"e_1_2_9_21_1","doi-asserted-by":"publisher","DOI":"10.1162\/003465304323023697"},{"key":"e_1_2_9_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.csda.2017.05.007"},{"key":"e_1_2_9_23_1","doi-asserted-by":"publisher","DOI":"10.1093\/restud\/rdr047"},{"key":"e_1_2_9_24_1","doi-asserted-by":"publisher","DOI":"10.1080\/07350015.2015.1038544"},{"key":"e_1_2_9_25_1","doi-asserted-by":"publisher","DOI":"10.2307\/2998560"},{"key":"e_1_2_9_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jeconom.2006.07.011"},{"key":"e_1_2_9_27_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1989.10478848"},{"key":"e_1_2_9_28_1","doi-asserted-by":"publisher","DOI":"10.1111\/1468-0262.00442"},{"key":"e_1_2_9_29_1","doi-asserted-by":"publisher","DOI":"10.1080\/07350015.2015.1017644"},{"key":"e_1_2_9_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jeconom.2012.11.006"},{"key":"e_1_2_9_31_1","doi-asserted-by":"publisher","DOI":"10.1257\/jel.47.1.5"},{"key":"e_1_2_9_32_1","doi-asserted-by":"publisher","DOI":"10.1177\/1536867X0800800401"},{"key":"e_1_2_9_33_1","doi-asserted-by":"publisher","DOI":"10.1002\/jae.1181"},{"key":"e_1_2_9_34_1","doi-asserted-by":"publisher","DOI":"10.1257\/aer.101.3.532"},{"key":"e_1_2_9_35_1","first-page":"604","article-title":"Evaluating the econometric evaluations of training programs with experimental data","volume":"76","author":"LaLonde R. J.","year":"1986","journal-title":"American Economic Review"},{"key":"e_1_2_9_36_1","doi-asserted-by":"publisher","DOI":"10.1080\/07474938.2017.1318509"},{"key":"e_1_2_9_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.labeco.2013.01.004"},{"key":"e_1_2_9_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00181-011-0481-0"},{"key":"e_1_2_9_39_1","volume-title":"PSMATCH2: Stata module to perform full Mahalanobis and propensity score matching, common support graphing, and covariate imbalance testing","author":"Leuven E.","year":"2003"},{"key":"e_1_2_9_40_1","doi-asserted-by":"publisher","DOI":"10.1002\/sim.1903"},{"key":"e_1_2_9_41_1","doi-asserted-by":"publisher","DOI":"10.1198\/jbes.2009.06045"},{"key":"e_1_2_9_42_1","unstructured":"Schuler A. Jung K. Tibshirani R. Hastie T. &Shah N.(2017).Synth\u2010validation: Selecting the best causal inference method for a given dataset (Working paper). Palo Alto CA: Stanford University."},{"key":"e_1_2_9_43_1","unstructured":"S\u0142oczy\u0144ski T.(2018).A general weighted average representation of the ordinary and two\u2010stage least squares estimands (Working paper).Waltham MA:Brandeis University."},{"key":"e_1_2_9_44_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0266466617000056"},{"key":"e_1_2_9_45_1","doi-asserted-by":"publisher","DOI":"10.1257\/aer.91.2.112"},{"key":"e_1_2_9_46_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jeconom.2004.04.011"},{"key":"e_1_2_9_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jeconom.2007.02.002"},{"key":"e_1_2_9_48_1","doi-asserted-by":"publisher","DOI":"10.1162\/003465304323023705"},{"key":"e_1_2_9_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.econlet.2007.05.010"}],"container-title":["Journal of Applied Econometrics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fjae.2724","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/jae.2724","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/jae.2724","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/jae.2724","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,19]],"date-time":"2023-08-19T08:59:57Z","timestamp":1692435597000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/jae.2724"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9]]},"references-count":48,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2019,9]]}},"alternative-id":["10.1002\/jae.2724"],"URL":"https:\/\/doi.org\/10.1002\/jae.2724","archive":["Portico"],"relation":{},"ISSN":["0883-7252","1099-1255"],"issn-type":[{"value":"0883-7252","type":"print"},{"value":"1099-1255","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,9]]},"assertion":[{"value":"2018-09-21","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-03-27","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-10-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}