{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T17:49:33Z","timestamp":1777916973188,"version":"3.51.4"},"reference-count":62,"publisher":"SAGE Publications","issue":"3","license":[{"start":{"date-parts":[[2021,1,20]],"date-time":"2021-01-20T00:00:00Z","timestamp":1611100800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"DOI":"10.13039\/100000002","name":"national institutes of health","doi-asserted-by":"publisher","award":["SRF-2013-06-016"],"award-info":[{"award-number":["SRF-2013-06-016"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Med Decis Making"],"published-print":{"date-parts":[[2021,4]]},"abstract":"<jats:p>Much evidence in comparative effectiveness research is based on observational studies. Researchers who conduct observational studies typically assume that there are no unobservable differences between the treatment groups under comparison. Treatment effectiveness is estimated after adjusting for observed differences between comparison groups. However, estimates of treatment effectiveness may be biased because of misspecification of the statistical model. That is, if the method of treatment effect estimation imposes unduly strong functional form assumptions, treatment effect estimates may be inaccurate, leading to inappropriate recommendations about treatment decisions. We compare the performance of a wide variety of treatment effect estimation methods for the average treatment effect. We do so within the context of the REFLUX study from the United Kingdom. In REFLUX, participants were enrolled in either an randomized controlled trial (RCT) or an observational study arm. In the RCT, patients were randomly assigned to either surgery or medical management. In the patient preference arm, participants selected to either have surgery or medical management. We attempt to recover the treatment effect estimate from the RCT using the data from the patient preference arms of the study. We vary the method of treatment effect estimation and record which methods are successful and which are not. We apply more than 20 different methods, including standard regression models as well as advanced machine learning methods. We find that simple propensity score matching methods provide the least accurate estimates versus the RCT benchmark. We find variation in performance across the other methods, with some, but not all recovering the experimental benchmark. We conclude that future studies should use multiple methods of estimation to fully represent uncertainty according to the choice of estimation approach.<\/jats:p>","DOI":"10.1177\/0272989x20986545","type":"journal-article","created":{"date-parts":[[2021,1,21]],"date-time":"2021-01-21T01:03:51Z","timestamp":1611191031000},"page":"340-353","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":1,"title":["Comparing the Performance of Statistical Adjustment Methods by Recovering the Experimental Benchmark from the REFLUX Trial"],"prefix":"10.1177","volume":"41","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3859-2713","authenticated-orcid":false,"given":"Luke","family":"Keele","sequence":"first","affiliation":[{"name":"University of Pennsylvania, Philadelphia, PA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stephen","family":"O\u2019Neill","sequence":"additional","affiliation":[{"name":"Department of Health Services Research and Policy, London School of Hygiene and Tropical Medicine, London, UK"},{"name":"Department of Health Services Research and Policy, London School of Hygiene and Tropical Medicine, London, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Richard","family":"Grieve","sequence":"additional","affiliation":[{"name":"Department of Health Services Research and Policy, London School of Hygiene and Tropical Medicine, London, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2021,1,20]]},"reference":[{"key":"bibr1-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.2105\/AJPH.2018.304337"},{"issue":"396","key":"bibr2-0272989X20986545","volume":"81","author":"Holland PW.","year":"1986","journal-title":"J Am Stat Assoc"},{"issue":"396","key":"bibr3-0272989X20986545","volume":"81","author":"Rubin DB.","year":"1986","journal-title":"J Am Stat Assoc"},{"issue":"2","key":"bibr4-0272989X20986545","volume":"128","author":"Cochran WG.","year":"1965","journal-title":"J R Stat Soc Ser A"},{"key":"bibr5-0272989X20986545","first-page":"43","volume-title":"Evaluation Studies","volume":"5","author":"Barnow BS","year":"1980"},{"issue":"1","key":"bibr6-0272989X20986545","volume":"29","author":"Rubin DB.","year":"1973","journal-title":"Biometrics"},{"issue":"4","key":"bibr7-0272989X20986545","volume":"84","author":"Rosenbaum PR.","year":"1989","journal-title":"J Am Stat Assoc"},{"key":"bibr8-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1999.10473858"},{"key":"bibr9-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1111\/1468-0262.00442"},{"key":"bibr10-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/asn055"},{"key":"bibr11-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1097\/00001648-200009000-00011"},{"issue":"3","key":"bibr12-0272989X20986545","volume":"2","author":"Hirano K","year":"2001","journal-title":"Health Serv Outcomes Res Methodol"},{"key":"bibr13-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1162\/003465304323023651"},{"issue":"429","key":"bibr14-0272989X20986545","volume":"90","author":"Robins JM","year":"1995","journal-title":"J Am Stat Assoc"},{"issue":"448","key":"bibr15-0272989X20986545","volume":"94","author":"Scharfstein DO","year":"1999","journal-title":"J Am Stat Assoc"},{"key":"bibr16-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1111\/j.1541-0420.2005.00377.x"},{"key":"bibr17-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1037\/1082-989X.9.4.403"},{"key":"bibr18-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1080\/00273171.2011.570161"},{"key":"bibr19-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2017.1319839"},{"key":"bibr20-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.2202\/1544-6115.1240"},{"issue":"4","key":"bibr21-0272989X20986545","volume":"76","author":"Lalonde R.","year":"1986","journal-title":"Am Econ Rev"},{"key":"bibr22-0272989X20986545","unstructured":"National Institute for Health Care Excellence. Gastro-oesophageal reflux disease and dyspepsia in adults: investigation and management. NICE Clinical Guidelines, No. 184. London: National Institute for Health Care Excellence; 2020."},{"key":"bibr23-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.3310\/hta12310"},{"key":"bibr24-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.3310\/hta17220"},{"key":"bibr25-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.3109\/07853890109002087"},{"key":"bibr26-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1093\/aje\/kwz100"},{"key":"bibr27-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1214\/14-STS486"},{"key":"bibr28-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1510507113"},{"key":"bibr29-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1097\/EDE.0000000000001231"},{"issue":"4","key":"bibr30-0272989X20986545","volume":"5","author":"Neyman J.","year":"1923","journal-title":"Section 9. Stat Sci"},{"key":"bibr31-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1037\/h0037350"},{"key":"bibr32-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1097\/EDE.0b013e3181bd5638"},{"key":"bibr33-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1214\/ss\/1177010894"},{"key":"bibr34-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511803161"},{"key":"bibr35-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1214\/08-AOAS187"},{"key":"bibr36-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/70.1.41"},{"issue":"1","key":"bibr37-0272989X20986545","first-page":"49","volume":"2","author":"Mahalanobis PC.","year":"1936","journal-title":"Proc Natl Inst Sci (Calcutta)"},{"key":"bibr38-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.2307\/2529981"},{"key":"bibr39-0272989X20986545","volume-title":"Semiparametric Regression for the Social Sciences","author":"Keele L.","year":"2008"},{"key":"bibr40-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.3368\/jhr.50.2.373"},{"key":"bibr41-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1214\/09-AOAS285"},{"key":"bibr42-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1198\/jcgs.2010.08162"},{"key":"bibr43-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1214\/19-BA1195"},{"key":"bibr44-0272989X20986545","volume-title":"Generalized Random Forests","author":"Athey S","year":"2016"},{"key":"bibr45-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4419-9782-1"},{"key":"bibr46-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1002\/sim.6322"},{"key":"bibr47-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-3692-2"},{"key":"bibr48-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1214\/13-AOAS713"},{"issue":"27","key":"bibr49-0272989X20986545","volume":"35","author":"Resa M","year":"2016","journal-title":"Stat Med"},{"key":"bibr50-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1198\/jbes.2009.07333"},{"key":"bibr51-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781139025751"},{"issue":"1","key":"bibr52-0272989X20986545","volume":"10","author":"Wong VC","year":"2017","journal-title":"J Res Educ Effect"},{"key":"bibr53-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1016\/j.brat.2017.01.005"},{"key":"bibr54-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1037\/a0019623"},{"key":"bibr55-0272989X20986545","volume-title":"Practical Propensity Score Methods Using R","author":"Leite W.","year":"2016"},{"key":"bibr56-0272989X20986545","volume-title":"Propensity Score Analysis: Statistical Methods and Applications","author":"Guo S","year":"2015"},{"key":"bibr57-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4419-1213-8"},{"key":"bibr58-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2012.703874"},{"key":"bibr59-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2014.997879"},{"key":"bibr60-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1162\/REST_a_00318"},{"key":"bibr61-0272989X20986545","doi-asserted-by":"publisher","DOI":"10.1017\/pan.2019.11"},{"key":"bibr62-0272989X20986545","first-page":"180503743","author":"Keele LJ","year":"2019","journal-title":"arXiv preprint"}],"container-title":["Medical Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0272989X20986545","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/0272989X20986545","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0272989X20986545","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T18:00:55Z","timestamp":1777658455000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/0272989X20986545"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,1,20]]},"references-count":62,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,4]]}},"alternative-id":["10.1177\/0272989X20986545"],"URL":"https:\/\/doi.org\/10.1177\/0272989x20986545","relation":{},"ISSN":["0272-989X","1552-681X"],"issn-type":[{"value":"0272-989X","type":"print"},{"value":"1552-681X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,1,20]]}}}