{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T17:38:01Z","timestamp":1790357881157,"version":"4.1.0"},"reference-count":76,"publisher":"Wiley","issue":"3","license":[{"start":{"date-parts":[[2021,9,1]],"date-time":"2021-09-01T00:00:00Z","timestamp":1630454400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["AI Magazine"],"published-print":{"date-parts":[[2021,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In recent years, a new line of research has taken an <jats:italic>interventional<\/jats:italic> view of recommender systems, where recommendations are viewed as actions that the system takes to have a desired effect. This interventional view has led to the development of <jats:italic>counterfactual<\/jats:italic> inference techniques for evaluating and optimizing recommendation policies. This article explains how these techniques enable unbiased offline evaluation and learning despite biased data, and how they can inform considerations of fairness and equity in recommender systems.<\/jats:p>","DOI":"10.1609\/aimag.v42i3.18141","type":"journal-article","created":{"date-parts":[[2022,1,11]],"date-time":"2022-01-11T21:03:24Z","timestamp":1641935004000},"page":"19-30","source":"Crossref","is-referenced-by-count":17,"title":["Recommendations as treatments"],"prefix":"10.1002","volume":"42","author":[{"given":"Thorsten","family":"Joachims","sequence":"first","affiliation":[{"name":"Cornell University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ben","family":"London","sequence":"additional","affiliation":[{"name":"Amazon"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yi","family":"Su","sequence":"additional","affiliation":[{"name":"Cornell University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adith","family":"Swaminathan","sequence":"additional","affiliation":[{"name":"Microsoft Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lequn","family":"Wang","sequence":"additional","affiliation":[{"name":"Cornell University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2021,9]]},"reference":[{"key":"e_1_2_8_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-019-09256-1"},{"key":"e_1_2_8_3_1","doi-asserted-by":"crossref","unstructured":"Abebe R. S.Barocas J.Kleinberg K.Levy M.Raghavan andD. G.Robinson.2020. \u201cRoles for Computing in Social Change.\u201d InConference on Fairness Accountability and Transparency 252\u201360.","DOI":"10.1145\/3351095.3372871"},{"key":"e_1_2_8_4_1","unstructured":"Agarwal A. S.Bird M.Cozowicz L.Hoang J.Langford S.Lee J.Li et al.2016. Making contextual decisions with low technical debt.ArXiv Preprint arXiv:1606.03966."},{"key":"e_1_2_8_5_1","doi-asserted-by":"crossref","unstructured":"Agarwal A. S.Basu T.Schnabel andT.Joachims.2017. \u201cEffective Evaluation Using Logged Bandit Feedback from Multiple Loggers.\u201d InACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD).","DOI":"10.1145\/3097983.3098155"},{"key":"e_1_2_8_6_1","unstructured":"Agarwal N. A.Suresh F.Yu S.Kumar andB.McMahan.2018. \u201ccpSGD: Communication\u2010efficient and Differentially\u2010private Distributed SGD.\u201d InAdvances in Neural Information Processing Systems."},{"key":"e_1_2_8_7_1","doi-asserted-by":"crossref","unstructured":"Agarwal A. I.Zaitsev X.Wang C.Li M.Najork andT.Joachims.2019. \u201cEstimating Position Bias Without Intrusive Interventions.\u201d InInternational Conference on Web Search and Data Mining.","DOI":"10.1145\/3289600.3291017"},{"key":"e_1_2_8_8_1","unstructured":"Agrawal S. andN.Goyal.2013. \u201cThompson Sampling for Contextual Bandits with Linear Payoffs.\u201d InInternational Conference on Machine Learning."},{"key":"e_1_2_8_9_1","first-page":"2016","article-title":"Machine bias","volume":"23","author":"Angwin J.","year":"2016","journal-title":"ProPublica, May"},{"key":"e_1_2_8_10_1","doi-asserted-by":"publisher","DOI":"10.1137\/S0097539701398375"},{"key":"e_1_2_8_11_1","doi-asserted-by":"crossref","unstructured":"Beutel A. J.Chen T.Doshi H.Qian L.Wei Y.Wu L.Heldt et al.2019. \u201cFairness in Recommendation Ranking Through Pairwise Comparisons.\u201d InACM SIGKDD International Conference on Knowledge Discovery and Data Mining 2212\u201320.","DOI":"10.1145\/3292500.3330745"},{"key":"e_1_2_8_12_1","unstructured":"Bibaut A. I.Malenica N.Vlassis andM.Van Der Laan.2019. \u201cMore Efficient Off\u2010Policy Evaluation Through Regularized Targeted Learning.\u201d InInternational Conference on Machine Learning 654\u20133."},{"key":"e_1_2_8_13_1","doi-asserted-by":"crossref","unstructured":"Biega A. J. K. P.Gummadi andG.Weikum.2018. \u201cEquity of Attention: Amortizing Individual Fairness in Rankings.\u201d InACM SIGIR Conference on Research and Development in Information Retrieval 405\u201314.","DOI":"10.1145\/3209978.3210063"},{"key":"e_1_2_8_14_1","first-page":"3207","article-title":"Counterfactual Reasoning and Learning Systems: The Example of Computational Advertising","volume":"14","author":"Bottou L.","year":"2013","journal-title":"Journal of Machine Learning Research"},{"issue":"1","key":"e_1_2_8_15_1","first-page":"3207","article-title":"Counterfactual Reasoning and Learning Systems: The Example of Computational Advertising","volume":"14","author":"Bottou L.","year":"2013","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_8_16_1","unstructured":"Celis L. E. D.Straszak andN. K.Vishnoi.2017. Ranking with fairness constraints.ArXiv Preprint arXiv:1704.06840."},{"key":"e_1_2_8_17_1","unstructured":"Cesa\u2010Bianchi N. C.Gentile G.Neu andG.Lugosi.2017. \u201cBoltzmann Exploration Done Right.\u201d InAdvances in Neural Information Processing Systems."},{"key":"e_1_2_8_18_1","doi-asserted-by":"crossref","unstructured":"Chen M. A.Beutel P.Covington S.Jain F.Belletti andE. H.Chi.2019a. \u201cTop\u2010k Off\u2010Policy Correction for a Reinforce Recommender System.\u201d InInternational Conference on Web Search and Data Mining 456\u201364.","DOI":"10.1145\/3289600.3290999"},{"key":"e_1_2_8_19_1","unstructured":"Chen M. R.Gummadi C.Harris andD.Schuurmans.2019b. \u201cSurrogate Objectives for Batch Policy Optimization in One\u2010Step Decision Making.\u201d InAdvances in Neural Information Processing Systems 8827\u201337."},{"key":"e_1_2_8_20_1","doi-asserted-by":"publisher","DOI":"10.1287\/mksc.2019.1154"},{"key":"e_1_2_8_21_1","doi-asserted-by":"crossref","unstructured":"Dimakopoulou M. N.Vlassis andT.Jebara.2019. \u201cMarginal Posterior Sampling for Slate Bandits.\u201d InInternational Joint Conference on Artificial Intelligence 2223\u20139.","DOI":"10.24963\/ijcai.2019\/308"},{"key":"e_1_2_8_22_1","unstructured":"Dud\u00edk M. J.Langford&L.Li2011. \u201cDoubly Robust Policy Evaluation and Learning.\u201d In International Conference on Machine Learning."},{"key":"e_1_2_8_23_1","doi-asserted-by":"crossref","unstructured":"Evans D. S. andR.Schmalensee.2016.Matchmakers: The New Economics of Multi\u2010Sided Platforms. Harvard Business Review Press.","DOI":"10.2139\/ssrn.2793021"},{"key":"e_1_2_8_24_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i04.5797"},{"key":"e_1_2_8_25_1","unstructured":"Fox.2004.House M.D. (Season 1 Episode 18). According tohttps:\/\/en.wikiquote.org\/wiki\/House(Season 1)."},{"key":"e_1_2_8_26_1","unstructured":"Geyer R. T.Klein andM.Nabi.2017. \u201cDifferentially Private Federated Learning: A Client Level Perspective.\u201dCoRRabs\/1712.07557."},{"key":"e_1_2_8_27_1","doi-asserted-by":"crossref","unstructured":"Gilotte A. C.Calauz\u00e9nes T.Nedelec A.Abraham andS.Doll\u00e9.2018. \u201cOffline a\/b testing for recommender systems.\u201d InInternational Conference on Web Search and Data Mining 198\u2013206.","DOI":"10.1145\/3159652.3159687"},{"key":"e_1_2_8_28_1","doi-asserted-by":"crossref","unstructured":"Gruson A. P.Chandar C.Charbuillet J.McInerney S.Hansen D.Tardieu andB.Carterette.2019. \u201cOffline Evaluation to Make Decisions about Playlist Recommendation Algorithms.\u201d InACM International Conference on Web Search and Data Mining 420\u20138.","DOI":"10.1145\/3289600.3291027"},{"key":"e_1_2_8_29_1","doi-asserted-by":"crossref","unstructured":"Holstein K. J. WortmanVaughan H.Daum\u00e9III M.Dud\u00edk&H.Wallach2019. \u201cImproving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?.\u201d In ACM CHI Conference on Human Factors in Computing Systems 1\u201316.","DOI":"10.1145\/3290605.3300830"},{"key":"e_1_2_8_30_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1952.10483446"},{"key":"e_1_2_8_31_1","doi-asserted-by":"publisher","DOI":"10.1198\/106186008X320456"},{"key":"e_1_2_8_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-020-09274-4"},{"key":"e_1_2_8_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1229179.1229181"},{"key":"e_1_2_8_34_1","unstructured":"Joachims T. A.Swaminathan andM.deRijke.2018. \u201cDeep Learning with Logged Bandit Feedback.\u201d InInternational Conference on Learning Representations."},{"key":"e_1_2_8_35_1","doi-asserted-by":"crossref","unstructured":"Joachims T. A.Swaminathan andT.Schnabel.2017. \u201cUnbiased Learning\u2010to\u2010Rank with Biased Feedback.\u201d InACM Conference on Web Search and Data Mining 781\u20139.","DOI":"10.1145\/3018661.3018699"},{"issue":"534","key":"e_1_2_8_36_1","first-page":"1","article-title":"More Efficient Policy Learning Via Optimal Retargeting","volume":"116","author":"Kallus N.","year":"2020","journal-title":"Journal of the American Statistical Association"},{"key":"e_1_2_8_37_1","doi-asserted-by":"crossref","unstructured":"Kohavi R. A.Deng B.Frasca R.Longbotham T.Walker andY.Xu.2012. \u201cTrustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained.\u201d InACM SIGKDD International Conference on Knowledge Discovery and Data Mining 786\u201394.","DOI":"10.1145\/2339530.2339653"},{"key":"e_1_2_8_38_1","doi-asserted-by":"crossref","unstructured":"Li L. W.Chu J.Langford andX.Wang.2011. \u201cUnbiased Offline Evaluation of Contextual\u2010Bandit\u2010Based News Article Recommendation Algorithms.\u201d InInternational Conference on Web Search and Data Mining 297\u2013306.","DOI":"10.1145\/1935826.1935878"},{"key":"e_1_2_8_39_1","doi-asserted-by":"crossref","unstructured":"Liu L. T. S.Dean E.Rolf M.Simchowitz andM.Hardt.2018. \u201cDelayed Impact of Fair Machine Learning.\u201d InInternational Conference on Machine Learning.","DOI":"10.24963\/ijcai.2019\/862"},{"key":"e_1_2_8_40_1","unstructured":"Liu A. H.Liu A.Anandkumar andY.Yue.2019. \u201cTriply Robust Off\u2010Policy Evaluation.\u201dCoRRabs\/1911.05811."},{"key":"e_1_2_8_41_1","unstructured":"London B. andT.Sandler.2019. \u201cBayesian Counterfactual Risk Minimization.\u201d InInternational Conference on Machine Learning."},{"key":"e_1_2_8_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-018-9209-6"},{"key":"e_1_2_8_43_1","doi-asserted-by":"crossref","unstructured":"Ma J. Z.Zhao X.Yi J.Yang M.Chen J.Tang L.Hong andE. H.Chi.2020a. \u201cOff\u2010policy Learning in Two\u2010Stage Recommender Systems.\u201d InThe Web Conference 463\u201373.","DOI":"10.1145\/3366423.3380130"},{"key":"e_1_2_8_44_1","doi-asserted-by":"crossref","unstructured":"Ma Y. B.Narayanaswamy H.Lin andH.Ding.2020b. \u201cTemporal\u2010contextual Recommendation in Real\u2010Time.\u201d InACM SIGKDD International Conference on Knowledge Discovery and Data Mining 2291\u20139.","DOI":"10.1145\/3394486.3403278"},{"key":"e_1_2_8_45_1","unstructured":"Ma Y. Y. \u2010 X.Wang andB.Narayanaswamy.2019. \u201cImitation\u2010regularized Offline Learning.\u201d InInternational Conference on Artificial Intelligence and Statistics 2956\u201365."},{"key":"e_1_2_8_46_1","doi-asserted-by":"crossref","unstructured":"McInerney J. B.Brost P.Chandar R.Mehrotra andB.Carterette.2020. \u201cCounterfactual Evaluation of Slate Recommendations With Sequential Reward Interactions.\u201d InACM SIGKDD International Conference on Knowledge Discovery and Data Mining 1779\u201388.","DOI":"10.1145\/3394486.3403229"},{"key":"e_1_2_8_47_1","unstructured":"McMahan H. D.Ramage K.Talwar andL.Zhang.2018. \u201cLearning Differentially Private Recurrent Language Models.\u201d InInternational Conference on Learning Representations."},{"key":"e_1_2_8_48_1","doi-asserted-by":"crossref","unstructured":"Mehrotra R. J.McInerney H.Bouchard M.Lalmas andF.Diaz.2018. \u201cTowards a Fair Marketplace: Counterfactual Evaluation of The Trade\u2010Off Between Relevance Fairness & Satisfaction in Recommendation Systems.\u201d InACM International Conference on Information and Knowledge Management 2243\u201351.","DOI":"10.1145\/3269206.3272027"},{"key":"e_1_2_8_49_1","doi-asserted-by":"crossref","unstructured":"Morik M. A.Singh J.Hong andT.Joachims.2020. \u201cControlling Fairness and Bias in Dynamic Learning\u2010To\u2010Rank.\u201d InACM SIGIR Conference on Research and Development in Information Retrieval.","DOI":"10.1145\/3397271.3401100"},{"key":"e_1_2_8_50_1","unstructured":"Azure personalizer service. Available athttps:\/\/docs.microsoft.com\/en\u2010in\/azure\/cognitive\u2010services\/personalizer\/."},{"key":"e_1_2_8_51_1","unstructured":"Nezik A. \u2010 K.2019. \u201cWenn maschinen kalt entscheiden.\u201dDie Zeit44.https:\/\/www.zeit.de\/2019\/44\/algorithmen\u2010software\u2010diskriminierung\u2010arbeitsmarkt\u2010assistenz\u2010system."},{"key":"e_1_2_8_52_1","doi-asserted-by":"crossref","unstructured":"Robertson S. E.1977. \u201cThe Probability Ranking Principle In IR.\u201dJournal of Documentation.","DOI":"10.1108\/eb026647"},{"key":"e_1_2_8_53_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/70.1.41"},{"key":"e_1_2_8_54_1","unstructured":"Seldin Y. andA.Slivkins.2014. \u201cOne Practical Algorithm for Both Stochastic and Adversarial Bandits.\u201d InInternational Conference on Machine Learning."},{"key":"e_1_2_8_55_1","doi-asserted-by":"crossref","unstructured":"Singh A. andT.Joachims.2018. \u201cFairness of Exposure in Rankings.\u201d InACM SIGKDD International Conference on Knowledge Discovery and Data Mining.","DOI":"10.1145\/3219819.3220088"},{"key":"e_1_2_8_56_1","unstructured":"Singh A. andT.Joachims.2019. \u201cPolicy Learning for Fairness in Rankings.\u201d InAdvances in Neural Information Processing Systems."},{"key":"e_1_2_8_57_1","unstructured":"Strehl A. J.Langford L.Li andS.Kakade.2010a. \u201cLearning from Logged Implicit Exploration Data.\u201d InAdvances in Neural Information Processing Systems."},{"key":"e_1_2_8_58_1","unstructured":"Strehl A. J.Langford L.Li andS. M.Kakade.2010b. \u201cLearning from logged implicit exploration data.\u201d InAdvances in Neural Information Processing Systems 2217\u201325."},{"key":"e_1_2_8_59_1","unstructured":"Su Y. L.Wang M.Santacatterina andT.Joachims.2019. \u201cCAB: Continuous Adaptive Blending for Policy Evaluation And Learning.\u201d InInternational Conference on Machine Learning."},{"key":"e_1_2_8_60_1","unstructured":"Su Y. M.Dimakopoulou A.Krishnamurthy&M.Dud\u00edk2020. \u201cDoubly robust off\u2010policy evaluation with shrinkage.\u201d In International Conference on Machine Learning 9167\u201376."},{"key":"e_1_2_8_61_1","doi-asserted-by":"crossref","unstructured":"Swaminathan A. andT.Joachims.2015a. \u201cCounterfactual Risk Minimization: Learning From Logged Bandit Feedback.\u201d InInternational Conference on Machine Learning 814\u201323.","DOI":"10.1145\/2740908.2742564"},{"key":"e_1_2_8_62_1","unstructured":"Swaminathan A. andT.Joachims.2015b. \u201cThe Self\u2010Normalized Estimator for Counterfactual Learning.\u201d InAdvances in Neural Information Processing Systems."},{"key":"e_1_2_8_63_1","unstructured":"Swaminathan A. A.Krishnamurthy A.Agarwal M.Dud\u00edk J.Langford D.Jose andI.Zitouni.2017. \u201cOff\u2010policy Evaluation for Slate Recommendation.\u201d InAdvances in Neural Information Processing Systems 3635\u201345."},{"key":"e_1_2_8_64_1","doi-asserted-by":"crossref","unstructured":"Thomas P. G.Theocharous andM.Ghavamzadeh.2015. \u201cHigh\u2010confidence Off\u2010policy Evaluation.\u201d InAAAI Conference on Artificial Intelligence.","DOI":"10.1609\/aaai.v29i1.9541"},{"key":"e_1_2_8_65_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/25.3-4.285"},{"key":"e_1_2_8_66_1","unstructured":"Vlassis N. A.Bibaut M.Dimakopoulou andT.Jebara.2019. \u201cOn the Design of Estimators for Bandit Off\u2010Policy Evaluation.\u201d InInternational Conference on Machine Learning."},{"key":"e_1_2_8_67_1","doi-asserted-by":"crossref","unstructured":"Vondrat'k J.2008. \u201cOptimal Approximation for the Submodular Welfare Problem in the Value Oracle Model.\u201d InACM Symposium on Theory of Computing 67\u201374.","DOI":"10.1145\/1374376.1374389"},{"key":"e_1_2_8_68_1","doi-asserted-by":"crossref","unstructured":"Wang L.&T.Joachims2021. \u201cUser Fairness Item Fairness and Diversity for Rankings in Two\u2010Sided Markets.\u201d ACM International Conference on the Theory of Information Retrieval.","DOI":"10.1145\/3471158.3472260"},{"key":"e_1_2_8_69_1","doi-asserted-by":"crossref","unstructured":"Wang X. N.Golbandi M.Bendersky D.Metzler andM.Najork.2018. \u201cPosition bias Estimation for Unbiased Learning to Rank in Personal Search.\u201d InACM International Conference on Web Search and Data Mining 610\u20138.","DOI":"10.1145\/3159652.3159732"},{"key":"e_1_2_8_70_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v35i1.2504"},{"key":"e_1_2_8_71_1","unstructured":"Wu H. andM.Wang.2018. \u201cVariance Regularized Counterfactual Risk Minimization Via Variational Divergence Minimization.\u201d InInternational Conference on Machine Learning 5353\u201362."},{"key":"e_1_2_8_72_1","doi-asserted-by":"crossref","unstructured":"Xiao L. Z.Min Z.Yongfeng G.Zhaoquan L.Yiqun andM.Shaoping.2017. \u201cFairness\u2010aware Group Recommendation with Pareto\u2010Efficiency.\u201d InACM Conference on Recommender Systems 107\u20135.","DOI":"10.1145\/3109859.3109887"},{"key":"e_1_2_8_73_1","unstructured":"Yadav H. Z.Du andT.Joachims.2021. \u201cFair Learning\u2010to\u2010Rank From Implicit Feedback.\u201d InACM SIGIR Conference on Research and Development in Information Retrieval."},{"key":"e_1_2_8_74_1","unstructured":"Yao S. andB.Huang.2017. \u201cBeyond Parity: Fairness Objectives for Collaborative Filtering.\u201d InAdvances in Neural Information Processing Systems 2921\u201330."},{"key":"e_1_2_8_75_1","unstructured":"Yin M. andY. \u2010 X.Wang.2020. \u201cAsymptotically Efficient Off\u2010Policy Evaluation for Tabular Reinforcement Learning.\u201d InInternational Conference on Artificial Intelligence and Statistics 3948\u201358."},{"key":"e_1_2_8_76_1","unstructured":"Yue Y. andC.Guestrin.2011. \u201cLinear Submodular Bandits and their Application to Diversified Retrieval.\u201d InAdvances in Neural Information Processing Systems 2483\u201391."},{"key":"e_1_2_8_77_1","doi-asserted-by":"crossref","unstructured":"Zehlike M. F.Bonchi C.Castillo S.Hajian M.Megahed andR.Baeza\u2010Yates.2017. \u201cFa* ir: A fair top\u2010k ranking algorithm.\u201d InACM Conference on Information and Knowledge Management 1569\u201378.","DOI":"10.1145\/3132847.3132938"}],"container-title":["AI Magazine"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/ojs.aaai.org\/index.php\/aimagazine\/article\/download\/18141\/18875","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1609\/aimag.v42i3.18141","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,14]],"date-time":"2023-09-14T16:31:16Z","timestamp":1694709076000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1609\/aimag.v42i3.18141"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9]]},"references-count":76,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,9]]}},"alternative-id":["10.1609\/aimag.v42i3.18141"],"URL":"https:\/\/doi.org\/10.1609\/aimag.v42i3.18141","archive":["Portico"],"relation":{},"ISSN":["0738-4602","2371-9621"],"issn-type":[{"value":"0738-4602","type":"print"},{"value":"2371-9621","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9]]}}}