{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T05:16:44Z","timestamp":1784179004484,"version":"3.55.0"},"reference-count":82,"publisher":"Association for Computing Machinery (ACM)","license":[{"start":{"date-parts":[[2023,1,12]],"date-time":"2023-01-12T00:00:00Z","timestamp":1673481600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"abstract":"<jats:p>Digital experiments are routinely used to test the value of a treatment relative to a status quo control setting \u2014 for instance, a new search relevance algorithm for a website or a new results layout for a mobile app. As digital experiments have become increasingly pervasive in organizations and a wide variety of research areas, their growth has prompted a new set of challenges for experimentation platforms. One challenge is that experiments often focus on the average treatment effect (ATE) without explicitly considering differences across major sub-groups \u2014 heterogeneous treatment effect (HTE). This is especially problematic because ATEs have decreased in many organizations as the more obvious benefits have already been realized. However, questions abound regarding the pervasiveness of user HTEs and how best to detect them. We propose a framework for detecting and analyzing user HTEs in digital experiments. Our framework combines an array of user characteristics with double machine learning. Analysis of 27 real-world experiments spanning 1.76 billion sessions and simulated data demonstrates the effectiveness of our detection method relative to existing techniques. We also find that transaction, demographic, engagement, satisfaction, and lifecycle characteristics exhibit statistically significant HTEs in 10% to 20% of our real-world experiments, underscoring the importance of considering user heterogeneity when analyzing experiment results, otherwise personalized features and experiences cannot happen, thus reducing effectiveness. In terms of the number of experiments and user sessions, we are not aware of any study that has examined user HTEs at this scale. Our findings have important implications for information retrieval, user modeling, platforms, and digital experience contexts, in which online experiments are often used to evaluate the effectiveness of design artifacts.<\/jats:p>","DOI":"10.1145\/3578931","type":"journal-article","created":{"date-parts":[[2023,1,12]],"date-time":"2023-01-12T15:46:25Z","timestamp":1673538385000},"update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Examining User Heterogeneity in Digital Experiments"],"prefix":"10.1145","author":[{"given":"Sriram","family":"Somanchi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ahmed","family":"Abbasi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ken","family":"Kelley","sequence":"additional","affiliation":[{"name":"University of Notre Dame, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Dobolyi","sequence":"additional","affiliation":[{"name":"University of Colorado, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ted Tao","family":"Yuan","sequence":"additional","affiliation":[{"name":"eBay, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,1,12]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Sentiment analysis in multiple languages: Feature selection for opinion classification in web forums. ACM transactions on information systems (TOIS) 26, 3","author":"Abbasi Ahmed","year":"2008","unstructured":"Ahmed Abbasi , Hsinchun Chen , and Arab Salem . 2008. Sentiment analysis in multiple languages: Feature selection for opinion classification in web forums. ACM transactions on information systems (TOIS) 26, 3 ( 2008 ), 1\u201334. Ahmed Abbasi, Hsinchun Chen, and Arab Salem. 2008. Sentiment analysis in multiple languages: Feature selection for opinion classification in web forums. ACM transactions on information systems (TOIS) 26, 3 (2008), 1\u201334."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2015.19"},{"key":"e_1_2_1_3_1","doi-asserted-by":"crossref","first-page":"899","DOI":"10.1037\/amp0000190","article-title":"Big data in psychology: A framework for research advancement","volume":"73","author":"Adjerid Idris","year":"2021","unstructured":"Idris Adjerid and Ken Kelley . 2021 . Big data in psychology: A framework for research advancement . American Psychologist 73 (2021), 899 \u2013\u2013 917 . Idris Adjerid and Ken Kelley. 2021. Big data in psychology: A framework for research advancement. American Psychologist 73(2021), 899\u2013\u2013917.","journal-title":"American Psychologist"},{"key":"e_1_2_1_4_1","volume-title":"Deep learning for adverse event detection from web search","author":"Ahmad Faizan","year":"2022","unstructured":"Faizan Ahmad , Ahmed Abbasi , Brent Kitchens , Donald\u00a0 A Adjeroh , and Daniel Zeng . 2022. Deep learning for adverse event detection from web search . IEEE Transactions on Knowledge and Data Engineering ( 2022 ). Faizan Ahmad, Ahmed Abbasi, Brent Kitchens, Donald\u00a0A Adjeroh, and Daniel Zeng. 2022. Deep learning for adverse event detection from web search. IEEE Transactions on Knowledge and Data Engineering (2022)."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3365211"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3322128"},{"key":"e_1_2_1_7_1","doi-asserted-by":"crossref","unstructured":"Barry\u00a0C. Arnold. 2015. Pareto Distribution. John Wiley & Sons Ltd 1\u201310.  Barry\u00a0C. Arnold. 2015. Pareto Distribution. John Wiley & Sons Ltd 1\u201310.","DOI":"10.1002\/9781118445112.stat01100.pub2"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1510489113"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1214\/18-AOS1709"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3106372"},{"key":"e_1_2_1_11_1","volume-title":"Customer event history for churn prediction: How long is long enough?Expert Systems with Applications 39, 18","author":"Ballings Michel","year":"2012","unstructured":"Michel Ballings and Dirk Van\u00a0den Poel . 2012. Customer event history for churn prediction: How long is long enough?Expert Systems with Applications 39, 18 ( 2012 ), 13517\u201313522. Michel Ballings and Dirk Van\u00a0den Poel. 2012. Customer event history for churn prediction: How long is long enough?Expert Systems with Applications 39, 18 (2012), 13517\u201313522."},{"key":"e_1_2_1_12_1","unstructured":"Yihan Bao Shichao Han and Yong Wang. 2021. Treatment Effect Detection with Controlled FDR under Dependence for Large-Scale Experiments. arXiv preprint arXiv:2110.07279(2021).  Yihan Bao Shichao Han and Yong Wang. 2021. Treatment Effect Detection with Controlled FDR under Dependence for Large-Scale Experiments. arXiv preprint arXiv:2110.07279(2021)."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1214\/15-AOS1337"},{"key":"e_1_2_1_14_1","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1177\/1536867X0500500303","article-title":"Estimation of Marginal Effects using Margeff","volume":"5","author":"Bartus Tam\u00e1s","year":"2005","unstructured":"Tam\u00e1s Bartus . 2005 . Estimation of Marginal Effects using Margeff . The Stata Journal 5 , 3 (2005), 309 \u2013 329 . https:\/\/doi.org\/10.1177\/1536867X0500500303 10.1177\/1536867X0500500303 Tam\u00e1s Bartus. 2005. Estimation of Marginal Effects using Margeff. The Stata Journal 5, 3 (2005), 309\u2013329. https:\/\/doi.org\/10.1177\/1536867X0500500303","journal-title":"The Stata Journal"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1995.tb02031.x"},{"key":"e_1_2_1_16_1","unstructured":"Yoav Benjamini and Daniel Yekutieli. 2001. The control of the false discovery rate in multiple testing under dependency. Annals of statistics(2001) 1165\u20131188.  Yoav Benjamini and Daniel Yekutieli. 2001. The control of the false discovery rate in multiple testing under dependency. Annals of statistics(2001) 1165\u20131188."},{"key":"e_1_2_1_17_1","volume-title":"Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet 327, 8476","author":"Bland J\u00a0Martin","year":"1986","unstructured":"J\u00a0Martin Bland and Douglas\u00a0 G Altman . 1986. Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet 327, 8476 ( 1986 ), 307\u2013310. J\u00a0Martin Bland and Douglas\u00a0G Altman. 1986. Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet 327, 8476 (1986), 307\u2013310."},{"key":"e_1_2_1_18_1","volume-title":"Random forests. Machine learning 45, 1","author":"Breiman Leo","year":"2001","unstructured":"Leo Breiman . 2001. Random forests. Machine learning 45, 1 ( 2001 ), 5\u201332. Leo Breiman. 2001. Random forests. Machine learning 45, 1 (2001), 5\u201332."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2015.32"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1257\/aer.p20171038"},{"key":"e_1_2_1_21_1","doi-asserted-by":"crossref","unstructured":"Victor Chernozhukov Denis Chetverikov Mert Demirer Esther Duflo Christian Hansen Whitney Newey and James Robins. 2016. Double\/debiased machine learning for treatment and causal parameters. arXiv preprint arXiv:1608.00060(2016).  Victor Chernozhukov Denis Chetverikov Mert Demirer Esther Duflo Christian Hansen Whitney Newey and James Robins. 2016. Double\/debiased machine learning for treatment and causal parameters. arXiv preprint arXiv:1608.00060(2016).","DOI":"10.3386\/w23564"},{"key":"e_1_2_1_22_1","volume-title":"Causal inference","author":"Cunningham Scott","unstructured":"Scott Cunningham . 2021. Causal inference . Yale University Press . Scott Cunningham. 2021. Causal inference. Yale University Press."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433413"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3057278"},{"key":"e_1_2_1_25_1","unstructured":"Boyan Duan Larry Wasserman and Aaditya Ramdas. 2021. Interactive identification of individuals with positive treatment effect while controlling false discoveries. arXiv preprint arXiv:2102.10778(2021).  Boyan Duan Larry Wasserman and Aaditya Ramdas. 2021. Interactive identification of individuals with positive treatment effect while controlling false discoveries. arXiv preprint arXiv:2102.10778(2021)."},{"key":"e_1_2_1_26_1","doi-asserted-by":"crossref","unstructured":"William Fithian and Lihua Lei. 2022. Conditional calibration for false discovery rate control under dependence. Annals of statistics (to appear)(2022).  William Fithian and Lihua Lei. 2022. Conditional calibration for false discovery rate control under dependence. Annals of statistics (to appear)(2022).","DOI":"10.1214\/21-AOS2137"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2382438.2382443"},{"key":"e_1_2_1_28_1","first-page":"1","article-title":"Learning to respond with your favorite stickers: A framework of unifying multi-modality and user preference in multi-turn dialog","volume":"39","author":"Gao Shen","year":"2021","unstructured":"Shen Gao , Xiuying Chen , Li Liu , Dongyan Zhao , and Rui Yan . 2021 . Learning to respond with your favorite stickers: A framework of unifying multi-modality and user preference in multi-turn dialog . ACM Transactions on Information Systems (TOIS) 39 , 2 (2021), 1 \u2013 32 . Shen Gao, Xiuying Chen, Li Liu, Dongyan Zhao, and Rui Yan. 2021. Learning to respond with your favorite stickers: A framework of unifying multi-modality and user preference in multi-turn dialog. ACM Transactions on Information Systems (TOIS) 39, 2 (2021), 1\u201332.","journal-title":"ACM Transactions on Information Systems (TOIS)"},{"key":"e_1_2_1_29_1","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1016\/j.jretai.2011.03.001","article-title":"Dynamics of customer response to promotional and relational direct mailings from an apparel retailer: The moderating role of relationship strength","volume":"87","author":"G\u00e1zquez-Abad Juan\u00a0Carlos","year":"2011","unstructured":"Juan\u00a0Carlos G\u00e1zquez-Abad , Marie\u00a0H\u00e9l\u00e8ne De\u00a0Canni\u00e9re , and Francisco\u00a0 J Mart\u00ednez-L\u00f3pez . 2011 . Dynamics of customer response to promotional and relational direct mailings from an apparel retailer: The moderating role of relationship strength . Journal of Retailing 87 , 2 (2011), 166 \u2013 181 . Juan\u00a0Carlos G\u00e1zquez-Abad, Marie\u00a0H\u00e9l\u00e8ne De\u00a0Canni\u00e9re, and Francisco\u00a0J Mart\u00ednez-L\u00f3pez. 2011. Dynamics of customer response to promotional and relational direct mailings from an apparel retailer: The moderating role of relationship strength. Journal of Retailing 87, 2 (2011), 166\u2013181.","journal-title":"Journal of Retailing"},{"key":"e_1_2_1_30_1","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1080\/07421222.2017.1334465","article-title":"Whose and what social media complaints have happier resolutions? Evidence from Twitter","volume":"34","author":"Gunarathne Priyanga","year":"2017","unstructured":"Priyanga Gunarathne , Huaxia Rui , and Abraham Seidmann . 2017 . Whose and what social media complaints have happier resolutions? Evidence from Twitter . Journal of Management Information Systems 34 , 2 (2017), 314 \u2013 340 . Priyanga Gunarathne, Huaxia Rui, and Abraham Seidmann. 2017. Whose and what social media complaints have happier resolutions? Evidence from Twitter. Journal of Management Information Systems 34, 2 (2017), 314\u2013340.","journal-title":"Journal of Management Information Systems"},{"key":"e_1_2_1_31_1","volume-title":"Recommender systems handbook","author":"Gunawardana Asela","unstructured":"Asela Gunawardana and Guy Shani . 2015. Evaluating recommender systems . In Recommender systems handbook . Springer , 265\u2013308. Asela Gunawardana and Guy Shani. 2015. Evaluating recommender systems. In Recommender systems handbook. Springer, 265\u2013308."},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1012\u20131023","author":"Guo Yue","year":"2022","unstructured":"Yue Guo , Yi Yang , and Ahmed Abbasi . 2022 . Auto-Debias: Debiasing Masked Language Models with Automated Biased Prompts . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1012\u20131023 . Yue Guo, Yi Yang, and Ahmed Abbasi. 2022. Auto-Debias: Debiasing Masked Language Models with Automated Biased Prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1012\u20131023."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331651.3331655"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2738036"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.2307\/2347785"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3472538.3472550"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.5555\/2896454.2896457"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1095872.1095876"},{"key":"e_1_2_1_39_1","volume-title":"Web analytics 2.0: The art of online accountability and science of customer centricity","author":"Kaushik Avinash","unstructured":"Avinash Kaushik . 2009. Web analytics 2.0: The art of online accountability and science of customer centricity . John Wiley & Sons . Avinash Kaushik. 2009. Web analytics 2.0: The art of online accountability and science of customer centricity. John Wiley & Sons."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1037\/1082-989X.8.3.305"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0028086"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1080\/07421222.2018.1451957"},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1\u20131.","author":"Kohavi Ron","year":"2015","unstructured":"Ron Kohavi . 2015 . Online controlled experiments: Lessons from running a\/b\/n tests for 12 years . In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1\u20131. Ron Kohavi. 2015. Online controlled experiments: Lessons from running a\/b\/n tests for 12 years. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1\u20131."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2488217"},{"key":"e_1_2_1_45_1","volume-title":"Trustworthy online controlled experiments: A practical guide to a\/b testing","author":"Kohavi Ron","unstructured":"Ron Kohavi , Diane Tang , and Ya Xu. 2020. Trustworthy online controlled experiments: A practical guide to a\/b testing . Cambridge University Press . Ron Kohavi, Diane Tang, and Ya Xu. 2020. Trustworthy online controlled experiments: A practical guide to a\/b testing. Cambridge University Press."},{"key":"e_1_2_1_46_1","first-page":"1","article-title":"Online randomized controlled experiments at scale: lessons and extensions to medicine","volume":"21","author":"Kohavi Ron","year":"2020","unstructured":"Ron Kohavi , Diane Tang , Ya Xu , Lars\u00a0 G Hemkens , and John\u00a0 PA Ioannidis . 2020 . Online randomized controlled experiments at scale: lessons and extensions to medicine . Trials 21 , 1 (2020), 1 \u2013 9 . Ron Kohavi, Diane Tang, Ya Xu, Lars\u00a0G Hemkens, and John\u00a0PA Ioannidis. 2020. Online randomized controlled experiments at scale: lessons and extensions to medicine. Trials 21, 1 (2020), 1\u20139.","journal-title":"Trials"},{"key":"e_1_2_1_47_1","volume-title":"The surprising power of online experiments. Harvard business review 95, 5","author":"Kohavi Ron","year":"2017","unstructured":"Ron Kohavi and Stefan Thomke . 2017. The surprising power of online experiments. Harvard business review 95, 5 ( 2017 ), 74\u201382. Ron Kohavi and Stefan Thomke. 2017. The surprising power of online experiments. Harvard business review 95, 5 (2017), 74\u201382."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1804597116"},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 3598\u20133609","author":"Lalor P","year":"2022","unstructured":"John\u00a0 P Lalor , Yi Yang , Kendall Smith , Nicole Forsgren , and Ahmed Abbasi . 2022 . Benchmarking intersectional biases in NLP . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 3598\u20133609 . John\u00a0P Lalor, Yi Yang, Kendall Smith, Nicole Forsgren, and Ahmed Abbasi. 2022. Benchmarking intersectional biases in NLP. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 3598\u20133609."},{"key":"e_1_2_1_50_1","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1177\/0022242920911628","article-title":"Path to purpose? How online customer journeys differ for hedonic versus utilitarian purchases","volume":"84","author":"Li Jingjing","year":"2020","unstructured":"Jingjing Li , Ahmed Abbasi , Amar Cheema , and Linda\u00a0 B Abraham . 2020 . Path to purpose? How online customer journeys differ for hedonic versus utilitarian purchases . Journal of Marketing 84 , 4 (2020), 127 \u2013 146 . Jingjing Li, Ahmed Abbasi, Amar Cheema, and Linda\u00a0B Abraham. 2020. Path to purpose? How online customer journeys differ for hedonic versus utilitarian purchases. Journal of Marketing 84, 4 (2020), 127\u2013146.","journal-title":"Journal of Marketing"},{"key":"e_1_2_1_51_1","volume-title":"TheoryOn: A design framework and system for unlocking behavioral knowledge through ontology learning. MIS Quarterly 44, 4","author":"Li Jingjing","year":"2020","unstructured":"Jingjing Li , Kai Larsen , and Ahmed Abbasi . 2020. TheoryOn: A design framework and system for unlocking behavioral knowledge through ontology learning. MIS Quarterly 44, 4 ( 2020 ). Jingjing Li, Kai Larsen, and Ahmed Abbasi. 2020. TheoryOn: A design framework and system for unlocking behavioral knowledge through ontology learning. MIS Quarterly 44, 4 (2020)."},{"key":"e_1_2_1_52_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3470565","article-title":"Profiling users for question answering communities via flow-based constrained co-embedding model","volume":"40","author":"Liang Shangsong","year":"2021","unstructured":"Shangsong Liang , Yupeng Luo , and Zaiqiao Meng . 2021 . Profiling users for question answering communities via flow-based constrained co-embedding model . ACM Transactions on Information Systems (TOIS) 40 , 2 (2021), 1 \u2013 38 . Shangsong Liang, Yupeng Luo, and Zaiqiao Meng. 2021. Profiling users for question answering communities via flow-based constrained co-embedding model. ACM Transactions on Information Systems (TOIS) 40, 2 (2021), 1\u201338.","journal-title":"ACM Transactions on Information Systems (TOIS)"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2016.04.008"},{"key":"e_1_2_1_54_1","volume-title":"Ana\u00a0S Camanho, and Jo\u00e3o\u00a0Falc\u00e3o e Cunha.","author":"Migu\u00e9is L","year":"2012","unstructured":"Vera\u00a0 L Migu\u00e9is , Dirk Van\u00a0den Poel , Ana\u00a0S Camanho, and Jo\u00e3o\u00a0Falc\u00e3o e Cunha. 2012 . Modeling partial customer churn: On the value of first product-category purchase sequences. Expert systems with applications 39, 12 (2012), 11250\u201311256. Vera\u00a0L Migu\u00e9is, Dirk Van\u00a0den Poel, Ana\u00a0S Camanho, and Jo\u00e3o\u00a0Falc\u00e3o e Cunha. 2012. Modeling partial customer churn: On the value of first product-category purchase sequences. Expert systems with applications 39, 12 (2012), 11250\u201311256."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1509\/jmkr.38.1.131.18832"},{"key":"e_1_2_1_56_1","volume-title":"Modeling online browsing and path analysis using clickstream data. Marketing science 23, 4","author":"Montgomery L","year":"2004","unstructured":"Alan\u00a0 L Montgomery , Shibo Li , Kannan Srinivasan , and John\u00a0 C Liechty . 2004. Modeling online browsing and path analysis using clickstream data. Marketing science 23, 4 ( 2004 ), 579\u2013595. Alan\u00a0L Montgomery, Shibo Li, Kannan Srinivasan, and John\u00a0C Liechty. 2004. Modeling online browsing and path analysis using clickstream data. Marketing science 23, 4 (2004), 579\u2013595."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3447679"},{"key":"e_1_2_1_58_1","volume-title":"Quasi-oracle estimation of heterogeneous treatment effects. Biometrika 108, 2 (09","author":"Nie X","year":"2020","unstructured":"X Nie and S Wager . 2020. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika 108, 2 (09 2020 ), 299\u2013319. https:\/\/doi.org\/10.1093\/biomet\/asaa076 arXiv:https:\/\/academic.oup.com\/biomet\/article-pdf\/108\/2\/299\/37938939\/asaa076.pdf 10.1093\/biomet X Nie and S Wager. 2020. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika 108, 2 (09 2020), 299\u2013319. https:\/\/doi.org\/10.1093\/biomet\/asaa076 arXiv:https:\/\/academic.oup.com\/biomet\/article-pdf\/108\/2\/299\/37938939\/asaa076.pdf"},{"key":"e_1_2_1_59_1","doi-asserted-by":"crossref","unstructured":"Judea Pearl. 2009. Causal inference in statistics: An overview. Statistics surveys 3(2009) 96\u2013146.  Judea Pearl. 2009. Causal inference in statistics: An overview. Statistics surveys 3(2009) 96\u2013146.","DOI":"10.1214\/09-SS057"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1509\/jmkg.67.1.77.18589"},{"key":"e_1_2_1_61_1","first-page":"1","article-title":"Preventing false discovery of heterogeneous treatment effect subgroups in randomized trials","volume":"19","author":"Rigdon Joseph","year":"2018","unstructured":"Joseph Rigdon , Michael Baiocchi , and Sanjay Basu . 2018 . Preventing false discovery of heterogeneous treatment effect subgroups in randomized trials . Trials 19 , 1 (2018), 1 \u2013 15 . Joseph Rigdon, Michael Baiocchi, and Sanjay Basu. 2018. Preventing false discovery of heterogeneous treatment effect subgroups in randomized trials. Trials 19, 1 (2018), 1\u201315.","journal-title":"Trials"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.2307\/1912705"},{"key":"e_1_2_1_63_1","volume-title":"Relevance Assessments for Web Search Evaluation: Should We Randomise or Prioritise the Pooled Documents?ACM Transactions on Information Systems (TOIS) 40, 4","author":"Sakai Tetsuya","year":"2022","unstructured":"Tetsuya Sakai , Sijie Tao , and Zhaohao Zeng . 2022. Relevance Assessments for Web Search Evaluation: Should We Randomise or Prioritise the Pooled Documents?ACM Transactions on Information Systems (TOIS) 40, 4 ( 2022 ), 1\u201335. Tetsuya Sakai, Sijie Tao, and Zhaohao Zeng. 2022. Relevance Assessments for Web Search Evaluation: Should We Randomise or Prioritise the Pooled Documents?ACM Transactions on Information Systems (TOIS) 40, 4 (2022), 1\u201335."},{"key":"e_1_2_1_64_1","volume-title":"Counting your customers: Who-are they and what will they do next?Management science 33, 1","author":"Schmittlein C","year":"1987","unstructured":"David\u00a0 C Schmittlein , Donald\u00a0 G Morrison , and Richard Colombo . 1987. Counting your customers: Who-are they and what will they do next?Management science 33, 1 ( 1987 ), 1\u201324. David\u00a0C Schmittlein, Donald\u00a0G Morrison, and Richard Colombo. 1987. Counting your customers: Who-are they and what will they do next?Management science 33, 1 (1987), 1\u201324."},{"key":"e_1_2_1_65_1","volume-title":"A unified approach to false discovery rate estimation. BMC bioinformatics 9, 1","author":"Strimmer Korbinian","year":"2008","unstructured":"Korbinian Strimmer . 2008. A unified approach to false discovery rate estimation. BMC bioinformatics 9, 1 ( 2008 ), 1\u201314. Korbinian Strimmer. 2008. A unified approach to false discovery rate estimation. BMC bioinformatics 9, 1 (2008), 1\u201314."},{"key":"e_1_2_1_66_1","volume-title":"Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 4072\u20134073","author":"Syrgkanis Vasilis","year":"2021","unstructured":"Vasilis Syrgkanis , Greg Lewis , Miruna Oprescu , Maggie Hei , Keith Battocchi , Eleanor Dillon , Jing Pan , Yifeng Wu , Paul Lo , Huigang Chen , et\u00a0al. 2021 . Causal inference and machine learning in practice with econml and causalml: Industrial use cases at microsoft, tripadvisor, uber . In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 4072\u20134073 . Vasilis Syrgkanis, Greg Lewis, Miruna Oprescu, Maggie Hei, Keith Battocchi, Eleanor Dillon, Jing Pan, Yifeng Wu, Paul Lo, Huigang Chen, et\u00a0al. 2021. Causal inference and machine learning in practice with econml and causalml: Industrial use cases at microsoft, tripadvisor, uber. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 4072\u20134073."},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1080\/07350015.2016.1172013"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835804.1835810"},{"key":"e_1_2_1_69_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3464301","article-title":"What and How long: Prediction of Mobile App Engagement","volume":"40","author":"Tian Yuan","year":"2021","unstructured":"Yuan Tian , Ke Zhou , and Dan Pelleg . 2021 . What and How long: Prediction of Mobile App Engagement . ACM Transactions on Information Systems (TOIS) 40 , 1 (2021), 1 \u2013 38 . Yuan Tian, Ke Zhou, and Dan Pelleg. 2021. What and How long: Prediction of Mobile App Engagement. ACM Transactions on Information Systems (TOIS) 40, 1 (2021), 1\u201338.","journal-title":"ACM Transactions on Information Systems (TOIS)"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33015183"},{"key":"e_1_2_1_71_1","volume-title":"Towards a Searcher\u2019s Learning Objective. ACM Transactions on Information Systems (TOIS) 40, 4","author":"Urgo Kelsey","year":"2022","unstructured":"Kelsey Urgo and Jaime Arguello . 2022. Understanding the \u201cPathway \u201d Towards a Searcher\u2019s Learning Objective. ACM Transactions on Information Systems (TOIS) 40, 4 ( 2022 ), 1\u201342. Kelsey Urgo and Jaime Arguello. 2022. Understanding the \u201cPathway\u201d Towards a Searcher\u2019s Learning Objective. ACM Transactions on Information Systems (TOIS) 40, 4 (2022), 1\u201342."},{"key":"e_1_2_1_72_1","volume-title":"Visualizing data using t-SNE.Journal of machine learning research 9, 11","author":"Maaten Laurens Van\u00a0der","year":"2008","unstructured":"Laurens Van\u00a0der Maaten and Geoffrey Hinton . 2008. Visualizing data using t-SNE.Journal of machine learning research 9, 11 ( 2008 ). Laurens Van\u00a0der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE.Journal of machine learning research 9, 11 (2008)."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2017.1319839"},{"key":"e_1_2_1_74_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3490476","article-title":"Personalized and Explainable Employee Training Course Recommendations: A Bayesian Variational Approach","volume":"40","author":"Wang Chao","year":"2021","unstructured":"Chao Wang , Hengshu Zhu , Peng Wang , Chen Zhu , Xi Zhang , Enhong Chen , and Hui Xiong . 2021 . Personalized and Explainable Employee Training Course Recommendations: A Bayesian Variational Approach . ACM Transactions on Information Systems (TOIS) 40 , 4 (2021), 1 \u2013 32 . Chao Wang, Hengshu Zhu, Peng Wang, Chen Zhu, Xi Zhang, Enhong Chen, and Hui Xiong. 2021. Personalized and Explainable Employee Training Course Recommendations: A Bayesian Variational Approach. ACM Transactions on Information Systems (TOIS) 40, 4 (2021), 1\u201332.","journal-title":"ACM Transactions on Information Systems (TOIS)"},{"key":"e_1_2_1_75_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3490478","article-title":"Combining Graph Convolutional Neural Networks and Label Propagation","volume":"40","author":"Wang Hongwei","year":"2021","unstructured":"Hongwei Wang and Jure Leskovec . 2021 . Combining Graph Convolutional Neural Networks and Label Propagation . ACM Transactions on Information Systems (TOIS) 40 , 4 (2021), 1 \u2013 27 . Hongwei Wang and Jure Leskovec. 2021. Combining Graph Convolutional Neural Networks and Label Propagation. ACM Transactions on Information Systems (TOIS) 40, 4 (2021), 1\u201327.","journal-title":"ACM Transactions on Information Systems (TOIS)"},{"key":"e_1_2_1_76_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3463913","article-title":"HyperSoRec: Exploiting hyperbolic user and item representations with multiple aspects for social-aware recommendation","volume":"40","author":"Wang Hao","year":"2021","unstructured":"Hao Wang , Defu Lian , Hanghang Tong , Qi Liu , Zhenya Huang , and Enhong Chen . 2021 . HyperSoRec: Exploiting hyperbolic user and item representations with multiple aspects for social-aware recommendation . ACM Transactions on Information Systems (TOIS) 40 , 2 (2021), 1 \u2013 28 . Hao Wang, Defu Lian, Hanghang Tong, Qi Liu, Zhenya Huang, and Enhong Chen. 2021. HyperSoRec: Exploiting hyperbolic user and item representations with multiple aspects for social-aware recommendation. ACM Transactions on Information Systems (TOIS) 40, 2 (2021), 1\u201328.","journal-title":"ACM Transactions on Information Systems (TOIS)"},{"key":"e_1_2_1_77_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3472955","article-title":"Dynamic structural role node embedding for user modeling in evolving networks","volume":"40","author":"Wang Lili","year":"2021","unstructured":"Lili Wang , Chenghan Huang , Ying Lu , Weicheng Ma , Ruibo Liu , and Soroush Vosoughi . 2021 . Dynamic structural role node embedding for user modeling in evolving networks . ACM Transactions on Information Systems (TOIS) 40 , 3 (2021), 1 \u2013 21 . Lili Wang, Chenghan Huang, Ying Lu, Weicheng Ma, Ruibo Liu, and Soroush Vosoughi. 2021. Dynamic structural role node embedding for user modeling in evolving networks. ACM Transactions on Information Systems (TOIS) 40, 3 (2021), 1\u201321.","journal-title":"ACM Transactions on Information Systems (TOIS)"},{"key":"e_1_2_1_78_1","first-page":"1","article-title":"Attributed collaboration network embedding for academic relationship mining","volume":"15","author":"Wang Wei","year":"2020","unstructured":"Wei Wang , Jiaying Liu , Tao Tang , Suppawong Tuarob , Feng Xia , Zhiguo Gong , and Irwin King . 2020 . Attributed collaboration network embedding for academic relationship mining . ACM Transactions on the Web (TWEB) 15 , 1 (2020), 1 \u2013 20 . Wei Wang, Jiaying Liu, Tao Tang, Suppawong Tuarob, Feng Xia, Zhiguo Gong, and Irwin King. 2020. Attributed collaboration network embedding for academic relationship mining. ACM Transactions on the Web (TWEB) 15, 1 (2020), 1\u201320.","journal-title":"ACM Transactions on the Web (TWEB)"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939733"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219860"},{"key":"e_1_2_1_82_1","first-page":"1","article-title":"Clarifying ambiguous keywords with personal word embeddings for personalized search","volume":"40","author":"Yao Jing","year":"2021","unstructured":"Jing Yao , Zhicheng Dou , and Ji-Rong Wen . 2021 . Clarifying ambiguous keywords with personal word embeddings for personalized search . ACM Transactions on Information Systems (TOIS) 40 , 3 (2021), 1 \u2013 29 . Jing Yao, Zhicheng Dou, and Ji-Rong Wen. 2021. Clarifying ambiguous keywords with personal word embeddings for personalized search. ACM Transactions on Information Systems (TOIS) 40, 3 (2021), 1\u201329.","journal-title":"ACM Transactions on Information Systems (TOIS)"},{"key":"e_1_2_1_83_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3495530","article-title":"Jointly Predicting Future Content in Multiple Social Media Sites Based on Multi-task Learning","volume":"40","author":"Zhang Peng","year":"2022","unstructured":"Peng Zhang , Baoxi Liu , Tun Lu , Xianghua Ding , Hansu Gu , and Ning Gu . 2022 . Jointly Predicting Future Content in Multiple Social Media Sites Based on Multi-task Learning . ACM Transactions on Information Systems (TOIS) 40 , 4 (2022), 1 \u2013 28 . Peng Zhang, Baoxi Liu, Tun Lu, Xianghua Ding, Hansu Gu, and Ning Gu. 2022. Jointly Predicting Future Content in Multiple Social Media Sites Based on Multi-task Learning. ACM Transactions on Information Systems (TOIS) 40, 4 (2022), 1\u201328.","journal-title":"ACM Transactions on Information Systems (TOIS)"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3578931","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3578931","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:08:52Z","timestamp":1750183732000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3578931"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,12]]},"references-count":82,"alternative-id":["10.1145\/3578931"],"URL":"https:\/\/doi.org\/10.1145\/3578931","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,12]]},"assertion":[{"value":"2022-04-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-12-17","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-01-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"3578931"}}