{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T22:24:57Z","timestamp":1783376697241,"version":"3.54.6"},"reference-count":52,"publisher":"Institute for Operations Research and the Management Sciences (INFORMS)","issue":"7","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Management Science"],"published-print":{"date-parts":[[2026,7]]},"abstract":"<jats:p>Large-scale online platforms launch hundreds of randomized experiments (also known as A\/B tests) every day to iterate their operations and marketing strategies. The combinations of these treatments are typically not exhaustively tested, which triggers an important question of both academic and practical interest. Without observing the outcomes of all treatment combinations, how does one estimate the causal effect of any treatment combination and identify the optimal treatment combination? We develop a novel framework combining deep learning and doubly robust estimation to estimate the causal effect of any treatment combination for each user on the platform when observing only a small subset of treatment combinations. Our proposed framework (called debiased deep learning ( DeDL )) exploits Neyman orthogonality and combines interpretable and flexible structural layers in deep learning. We show theoretically that this framework yields efficient, consistent, and asymptotically normal estimators under mild assumptions, thus allowing for identifying the best treatment combination when observing only a few combinations. To empirically validate our method, we collaborated with a large-scale video-sharing platform and implemented our framework for three experiments involving three treatments, where each combination of treatments is tested. When observing only a subset of treatment combinations, our DeDL approach significantly outperforms other benchmarks to accurately estimate and infer the average treatment effect of any treatment combination and to identify the optimal treatment combination.<\/jats:p>\n                  <jats:p>This paper was accepted by Vivek Farias, data science.<\/jats:p>\n                  <jats:p>Funding: R. Zhang is grateful for financial support from the Hong Kong Research Grants Council General Research Fund [Grants 14502722, 14503224, and 14504123].<\/jats:p>\n                  <jats:p>Supplemental Material: The online appendix and data files are available at https:\/\/doi.org\/10.1287\/mnsc.2024.04625 .<\/jats:p>","DOI":"10.1287\/mnsc.2024.04625","type":"journal-article","created":{"date-parts":[[2025,10,15]],"date-time":"2025-10-15T18:31:00Z","timestamp":1760553060000},"page":"5611-5634","source":"Crossref","is-referenced-by-count":4,"title":["Deep Learning-Based Causal Inference for Large-Scale Combinatorial Experiments: Theory and Empirical Evidence"],"prefix":"10.1287","volume":"72","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9914-7966","authenticated-orcid":false,"given":"Zikun","family":"Ye","sequence":"first","affiliation":[{"name":"Michael G. Foster School of Business, University of Washington, Seattle, Washington 98195"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-4566-8148","authenticated-orcid":false,"given":"Zhiqi","family":"Zhang","sequence":"additional","affiliation":[{"name":"Olin Business School, Washington University in St. Louis, St. Louis, Missouri 63130"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4544-775X","authenticated-orcid":false,"given":"Dennis J.","family":"Zhang","sequence":"additional","affiliation":[{"name":"Olin Business School, Washington University in St. Louis, St. Louis, Missouri 63130"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6105-6994","authenticated-orcid":false,"given":"Heng","family":"Zhang","sequence":"additional","affiliation":[{"name":"W. P. Carey School of Business, Arizona State University, Tempe, Arizona 85287"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0284-164X","authenticated-orcid":false,"given":"Renyu","family":"Zhang","sequence":"additional","affiliation":[{"name":"Chinese University of Hong Kong Business School, The Chinese University of Hong Kong, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"109","reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.1137\/20M131309X"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.1515\/9781400829828"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.1257\/aer.20190159"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1510489113"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.1111\/rssb.12268"},{"key":"B6","unstructured":"Bertsimas D, Imai K, Li ML (2022) Distributionally robust causal inference with observational data. Preprint, submitted October 15, https:\/\/arxiv.org\/abs\/2210.08326."},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.2022.4583"},{"key":"B8","volume-title":"Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building","author":"Box GEP","year":"1978"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.2014.2069"},{"key":"B10","doi-asserted-by":"crossref","unstructured":"Candogan O, Chen C, Niazadeh R (2024) Correlated cluster-based randomized experiments: Robust variance minimization.\n                      Management Sci.\n                      70(6):4069\u20134086.","DOI":"10.1287\/mnsc.2021.02741"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.3982\/ECTA18515"},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.1111\/ectj.12097"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.1287\/opre.2017.1629"},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.1080\/07350015.2021.1895815"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.1111\/rssb.12085"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.1257\/aeri.20180150"},{"key":"B17","doi-asserted-by":"publisher","DOI":"10.1257\/app.20160213"},{"key":"B18","doi-asserted-by":"publisher","DOI":"10.1080\/07350015.2020.1811102"},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.1093\/ectj\/utac003"},{"key":"B20","first-page":"14001","volume":"34","author":"Farias V","year":"2021","journal-title":"Adv. Neural Inform. Processing Systems"},{"key":"B21","unstructured":"Farrell MH, Liang T, Misra S (2020) Deep learning for individual heterogeneity: An automatic inference framework. Preprint, submitted October 28, https:\/\/arxiv.org\/abs\/2010.14694."},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.3982\/ECTA16901"},{"key":"B23","doi-asserted-by":"crossref","unstructured":"Goli A, Lambrecht A, Yoganarasimhan H (2024) A bias correction approach for interference in ranking experiments.\n                      Marketing Sci.\n                      43(3):590\u2013614.","DOI":"10.1287\/mksc.2022.0046"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.1287\/mksc.2022.1413"},{"key":"B25","first-page":"8637","volume":"34","author":"Guo Y","year":"2021","journal-title":"Adv. Neural Inform. Processing Systems"},{"key":"B26","doi-asserted-by":"publisher","DOI":"10.1016\/0893-6080(89)90020-8"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781139025751"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.2021.4247"},{"key":"B29","first-page":"6921","volume":"31","author":"Kallus N","year":"2018","journal-title":"Adv. Neural Inform. Processing Systems"},{"key":"B30","unstructured":"Kingma DP (2014) Adam: A method for stochastic optimization. Preprint, submitted December 22, https:\/\/arxiv.org\/abs\/1412.6980."},{"key":"B31","doi-asserted-by":"crossref","unstructured":"Knaus MC (2022) Double machine learning-based programme evaluation under unconfoundedness.\n                      Econometrics J.\n                      25(3):602\u2013627.","DOI":"10.1093\/ectj\/utac015"},{"issue":"5","key":"B32","first-page":"74","volume":"95","author":"Kohavi R","year":"2017","journal-title":"Harvard Bus. Rev."},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.1017\/9781108653985"},{"key":"B34","doi-asserted-by":"crossref","unstructured":"Lee MR, Shen M (2018) Winner\u2019s curse: Bias estimation for total effects of features in online controlled experiments.\n                      Proc. 24th ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining\n                      (ACM, New York), 491\u2013499.","DOI":"10.1145\/3219819.3219905"},{"key":"B35","doi-asserted-by":"crossref","unstructured":"Lim AE, Shanthikumar JG, Shen ZM (2006) Model uncertainty, robust optimization, and learning.\n                      Models, Methods, and Applications for Innovative Decision Making\n                      (INFORMS, Cantonsville, MD), 66\u201394.","DOI":"10.1287\/educ.1063.0021"},{"key":"B36","first-page":"6466","volume":"34","author":"Nandy P","year":"2021","journal-title":"Adv. Neural Inform. Processing Systems"},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.2307\/2951752"},{"key":"B38","doi-asserted-by":"crossref","unstructured":"Pashley NE, Bind MAC (2023) Causal inference for multiple treatments using fractional factorial designs.\n                      Canadian J. Statist.\n                      51(2):444\u2013468.","DOI":"10.1002\/cjs.11734"},{"key":"B40","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.2023.4914"},{"key":"B41","doi-asserted-by":"crossref","unstructured":"Tang D, Agarwal A, O\u2019Brien D, Meyer M (2010) Overlapping experiment infrastructure: More, better, faster experimentation.\n                      Proc. 16th ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining\n                      (ACM, New York), 17\u201326.","DOI":"10.1145\/1835804.1835810"},{"key":"B39","doi-asserted-by":"crossref","unstructured":"Tang J, Qi Z, Fang E, Shi C (2025) Offline feature-based pricing under censored demand: A causal inference approach.\n                      Manufacturing Service Oper. Management\n                      27(2):535\u2013553.","DOI":"10.1287\/msom.2024.1061"},{"key":"B42","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2017.1319839"},{"key":"B43","volume-title":"Econometric Analysis of Cross Section and Panel Data","author":"Wooldridge JM","year":"2010"},{"key":"B44","volume-title":"Experiments: Planning, Analysis, and Optimization","author":"Wu CFJ","year":"2011"},{"key":"B45","doi-asserted-by":"crossref","unstructured":"Xie H, Aurisset J (2016) Improving the sensitivity of online controlled experiments: Case studies at netflix.\n                      Proc. 22nd ACM SIGKDD International Conference Knowledge Discovery Data Mining\n                      (ACM, New York), 645\u2013654.","DOI":"10.1145\/2939672.2939733"},{"key":"B46","unstructured":"Xiong T, Wang Y, Zheng S (2020) Orthogonal traffic assignment in online overlapping A\/B tests. EasyChair technical report, Tencent Inc., Shenzhen, China."},{"key":"B47","unstructured":"Xiong R, Chin A, Taylor S (2023) Bias-variance tradeoffs for designing simultaneous temporal experiments.\n                      The KDD\u201923 Workshop Causal Discovery, Prediction Decision\n                      (PMLR, New York), 115\u2013131."},{"key":"B48","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2017.07.002"},{"key":"B49","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.2022.4550"},{"key":"B50","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.2022.4622"},{"key":"B51","doi-asserted-by":"publisher","DOI":"10.1214\/21-AOS2156"},{"key":"B52","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.2019.3316"}],"container-title":["Management Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pubsonline.informs.org\/doi\/pdf\/10.1287\/mnsc.2024.04625","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T10:24:56Z","timestamp":1783333496000},"score":1,"resource":{"primary":{"URL":"https:\/\/pubsonline.informs.org\/doi\/10.1287\/mnsc.2024.04625"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7]]},"references-count":52,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2026,7]]}},"alternative-id":["10.1287\/mnsc.2024.04625"],"URL":"https:\/\/doi.org\/10.1287\/mnsc.2024.04625","relation":{},"ISSN":["0025-1909","1526-5501"],"issn-type":[{"value":"0025-1909","type":"print"},{"value":"1526-5501","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7]]}}}