{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T08:33:44Z","timestamp":1789115624386,"version":"build-2803163510"},"publisher-location":"New York, NY, USA","reference-count":40,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,4,25]],"date-time":"2022-04-25T00:00:00Z","timestamp":1650844800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,4,25]]},"DOI":"10.1145\/3487553.3524215","type":"proceedings-article","created":{"date-parts":[[2022,8,16]],"date-time":"2022-08-16T22:41:30Z","timestamp":1660689690000},"page":"99-104","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["Beyond NDCG: Behavioral Testing of Recommender Systems with RecList"],"prefix":"10.1145","author":[{"given":"Patrick John","family":"Chia","sequence":"first","affiliation":[{"name":"Coveo, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jacopo","family":"Tagliabue","sequence":"additional","affiliation":[{"name":"Coveo Labs, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Federico","family":"Bianchi","sequence":"additional","affiliation":[{"name":"Bocconi University, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chloe","family":"He","sequence":"additional","affiliation":[{"name":"Stanford University, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brian","family":"Ko","sequence":"additional","affiliation":[{"name":"KOSA AI, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,8,16]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645752"},{"key":"e_1_3_2_1_2_1","volume-title":"Retrieved","author":"Arora Nidhi","year":"2021","unstructured":"Nidhi Arora , Daniel Ensslen , Lars Fiedler , Wei\u00a0Wei Liu , Kelsey Robinson , Eli Stein , and Gustavo Sch\u00fcler . 2021 . The value of getting personalization right\u2014or wrong\u2014is multiplying . Retrieved November 15, 2021 from https:\/\/www.mckinsey.com\/business-functions\/marketing-and-sales\/our-insights\/the-value-of-getting-personalization-right-or-wrong-is-multiplying Nidhi Arora, Daniel Ensslen, Lars Fiedler, Wei\u00a0Wei Liu, Kelsey Robinson, Eli Stein, and Gustavo Sch\u00fcler. 2021. The value of getting personalization right\u2014or wrong\u2014is multiplying. Retrieved November 15, 2021 from https:\/\/www.mckinsey.com\/business-functions\/marketing-and-sales\/our-insights\/the-value-of-getting-personalization-right-or-wrong-is-multiplying"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.1996.536464"},{"key":"e_1_3_2_1_4_1","unstructured":"David Berg Ravi\u00a0Kiran Chirravuri Romain Cledat Savin Goyal Ferras Hamad and Ville Tuulos. 2019. Open-Sourcing Metaflow a Human-Centric Framework for Data Science. https:\/\/netflixtechblog.com\/open-sourcing-metaflow-a-human-centric-framework-for-data-science-fa72e04a5d9  David Berg Ravi\u00a0Kiran Chirravuri Romain Cledat Savin Goyal Ferras Hamad and Ville Tuulos. 2019. Open-Sourcing Metaflow a Human-Centric Framework for Data Science. https:\/\/netflixtechblog.com\/open-sourcing-metaflow-a-human-centric-framework-for-data-science-fa72e04a5d9"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219891"},{"key":"e_1_3_2_1_6_1","unstructured":"Federico Bianchi J. Tagliabue Bingqing Yu Luca Bigon and Ciro Greco. 2020. Fantastic Embeddings and How to Align Them: Zero-Shot Inference in a Multi-Shop Scenario. ArXiv abs\/2007.14906(2020).  Federico Bianchi J. Tagliabue Bingqing Yu Luca Bigon and Ciro Greco. 2020. Fantastic Embeddings and How to Align Them: Zero-Shot Inference in a Multi-Shop Scenario. ArXiv abs\/2007.14906(2020)."},{"key":"e_1_3_2_1_7_1","volume-title":"Proceedings of The 4th Workshop on e-Commerce and NLP. Association for Computational Linguistics, Online, 1\u201312","author":"Bianchi Federico","year":"2021","unstructured":"Federico Bianchi , Bingqing Yu , and Jacopo Tagliabue . 2021 . BERT Goes Shopping: Comparing Distributional Models for Product Representations . In Proceedings of The 4th Workshop on e-Commerce and NLP. Association for Computational Linguistics, Online, 1\u201312 . https:\/\/doi.org\/10.18653\/v1\/2021.ecnlp-1.1 10.18653\/v1 Federico Bianchi, Bingqing Yu, and Jacopo Tagliabue. 2021. BERT Goes Shopping: Comparing Distributional Models for Product Representations. In Proceedings of The 4th Workshop on e-Commerce and NLP. Association for Computational Linguistics, Online, 1\u201312. https:\/\/doi.org\/10.18653\/v1\/2021.ecnlp-1.1"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3462832"},{"key":"e_1_3_2_1_9_1","unstructured":"Patrick\u00a0John Chia Bingqing Yu and Jacopo Tagliabue. 2021. \u201dAre you sure?\u201d: Preliminary Insights from Scaling Product Comparisons to Multiple Shops. ArXiv abs\/2107.03256(2021).  Patrick\u00a0John Chia Bingqing Yu and Jacopo Tagliabue. 2021. \u201dAre you sure?\u201d: Preliminary Insights from Scaling Product Comparisons to Multiple Shops. ArXiv abs\/2107.03256(2021)."},{"key":"e_1_3_2_1_10_1","volume-title":"Retrieved","author":"Cloud Google","year":"2021","unstructured":"Google Cloud . 2021 . Implementing Recommendations AI . Retrieved November 17, 2021 from https:\/\/cloud.google.com\/retail\/recommendations-ai\/docs\/overview Google Cloud. 2021. Implementing Recommendations AI. Retrieved November 17, 2021 from https:\/\/cloud.google.com\/retail\/recommendations-ai\/docs\/overview"},{"key":"e_1_3_2_1_11_1","unstructured":"Comet.ML. 2021. Comet.ML home page. https:\/\/www.comet.ml\/  Comet.ML. 2021. Comet.ML home page. https:\/\/www.comet.ml\/"},{"key":"e_1_3_2_1_12_1","unstructured":"Gabriel de Souza Pereira\u00a0Moreira Sara Rabhi Ronay Ak Md\u00a0Yasin Kabir and Even Oldridge. 2021. Transformers with multi-modal features and post-fusion context for e-commerce session-based recommendation. ArXiv abs\/2107.05124(2021).  Gabriel de Souza Pereira\u00a0Moreira Sara Rabhi Ronay Ak Md\u00a0Yasin Kabir and Even Oldridge. 2021. Transformers with multi-modal features and post-fusion context for e-commerce session-based recommendation. ArXiv abs\/2107.05124(2021)."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330839"},{"key":"e_1_3_2_1_14_1","article-title":"The MovieLens Datasets","volume":"5","author":"Harper Maxwell","year":"2015","unstructured":"F.\u00a0 Maxwell Harper and Joseph\u00a0 A. Konstan . 2015 . The MovieLens Datasets : History and Context. ACM Trans. Interact. Intell. Syst. 5 , 4, Article 19 (Dec. 2015), 19\u00a0pages. https:\/\/doi.org\/10.1145\/2827872 10.1145\/2827872 F.\u00a0Maxwell Harper and Joseph\u00a0A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Trans. Interact. Intell. Syst. 5, 4, Article 19 (Dec. 2015), 19\u00a0pages. https:\/\/doi.org\/10.1145\/2827872","journal-title":"History and Context. ACM Trans. Interact. Intell. Syst."},{"key":"e_1_3_2_1_15_1","unstructured":"Bal\u00e1zs Hidasi Alexandros Karatzoglou Linas Baltrunas and Domonkos Tikk. 2016. Session-based Recommendations with Recurrent Neural Networks. CoRR abs\/1511.06939(2016).  Bal\u00e1zs Hidasi Alexandros Karatzoglou Linas Baltrunas and Domonkos Tikk. 2016. Session-based Recommendations with Recurrent Neural Networks. CoRR abs\/1511.06939(2016)."},{"key":"e_1_3_2_1_16_1","volume-title":"Will the Global Village Fracture Into Tribes? Recommender Systems and Their Effects on Consumer Fragmentation. Management Science 60 (04","author":"Hosanagar Kartik","year":"2014","unstructured":"Kartik Hosanagar , Daniel Fleder , Dokyun Lee , and Andreas Buja . 2014. Will the Global Village Fracture Into Tribes? Recommender Systems and Their Effects on Consumer Fragmentation. Management Science 60 (04 2014 ), 805\u2013823. https:\/\/doi.org\/10.1287\/mnsc.2013.1808 10.1287\/mnsc.2013.1808 Kartik Hosanagar, Daniel Fleder, Dokyun Lee, and Andreas Buja. 2014. Will the Global Village Fracture Into Tribes? Recommender Systems and Their Effects on Consumer Fragmentation. Management Science 60 (04 2014), 805\u2013823. https:\/\/doi.org\/10.1287\/mnsc.2013.1808"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3109859.3109872"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2339530.2339653"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"crossref","unstructured":"Denis Kotkov Jari Veijalainen and Shuaiqiang Wang. 2016. Challenges of Serendipity in Recommender Systems. In WEBIST.  Denis Kotkov Jari Veijalainen and Shuaiqiang Wang. 2016. Challenges of Serendipity in Recommender Systems. In WEBIST.","DOI":"10.5220\/0005879802510256"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3412235"},{"key":"e_1_3_2_1_21_1","volume-title":"The Impact of Product Recommendations. Retrieved","author":"Garcia Krista","year":"2021","unstructured":"Krista Garcia . 2018. The Impact of Product Recommendations. Retrieved November 9, 2021 from https:\/\/www.emarketer.com\/content\/the-impact-of-product-recommendations Krista Garcia. 2018. The Impact of Product Recommendations. Retrieved November 9, 2021 from https:\/\/www.emarketer.com\/content\/the-impact-of-product-recommendations"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3460231.3474602"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-018-9209-6"},{"key":"e_1_3_2_1_24_1","unstructured":"Tomas Mikolov Kai Chen Gregory\u00a0S. Corrado and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. In ICLR.  Tomas Mikolov Kai Chen Gregory\u00a0S. Corrado and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. In ICLR."},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3412263"},{"key":"e_1_3_2_1_26_1","volume-title":"Workshop.","author":"Nassif Houssam","year":"2018","unstructured":"Houssam Nassif , Kemal\u00a0Oral Cansizlar , Mitchell Goodman , and SVN Vishwanathan . 2018 . Diversifying Music Recommendations. In ICML \u201916 Workshop. Houssam Nassif, Kemal\u00a0Oral Cansizlar, Mitchell Goodman, and SVN Vishwanathan. 2018. Diversifying Music Recommendations. In ICML \u201916 Workshop."},{"key":"e_1_3_2_1_27_1","unstructured":"Alec Radford Jong\u00a0Wook Kim Chris Hallacy Aditya Ramesh Gabriel Goh Sandhini Agarwal Girish Sastry Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In ICML.  Alec Radford Jong\u00a0Wook Kim Chris Hallacy Aditya Ramesh Gabriel Goh Sandhini Agarwal Girish Sastry Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In ICML."},{"key":"e_1_3_2_1_28_1","volume-title":"MultiRec: A Multi-Relational Approach for Unique Item Recommendation in Auction Systems. Fourteenth ACM Conference on Recommender Systems","author":"Rashed Ahmed","year":"2020","unstructured":"Ahmed Rashed , Shayan Jawed , Lars Schmidt-Thieme , and Andre Hintsches . 2020 . MultiRec: A Multi-Relational Approach for Unique Item Recommendation in Auction Systems. Fourteenth ACM Conference on Recommender Systems (2020). Ahmed Rashed, Shayan Jawed, Lars Schmidt-Thieme, and Andre Hintsches. 2020. MultiRec: A Multi-Relational Approach for Unique Item Recommendation in Auction Systems. Fourteenth ACM Conference on Recommender Systems (2020)."},{"key":"e_1_3_2_1_29_1","volume-title":"Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. In ACL.","author":"Ribeiro Marco\u00a0T\u00falio","year":"2020","unstructured":"Marco\u00a0T\u00falio Ribeiro , Tongshuang\u00a0(Sherry) Wu, Carlos Guestrin , and Sameer Singh . 2020 . Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. In ACL. Marco\u00a0T\u00falio Ribeiro, Tongshuang\u00a0(Sherry) Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. In ACL."},{"key":"e_1_3_2_1_30_1","volume-title":"RecSysOps: Best Practices for Operating a Large-Scale Recommender System","author":"Saberian Mohammad","unstructured":"Mohammad Saberian and Justin Basilico . 2021. RecSysOps: Best Practices for Operating a Large-Scale Recommender System . Association for Computing Machinery , New York, NY, USA , 590\u2013591. https:\/\/doi.org\/10.1145\/3460231.3474620 10.1145\/3460231.3474620 Mohammad Saberian and Justin Basilico. 2021. RecSysOps: Best Practices for Operating a Large-Scale Recommender System. Association for Computing Machinery, New York, NY, USA, 590\u2013591. https:\/\/doi.org\/10.1145\/3460231.3474620"},{"key":"e_1_3_2_1_31_1","volume-title":"Retrieved","author":"Statista Research Department","year":"2020","unstructured":"Statista Research Department . 2020 . Global retail e-commerce sales 2014-2023 . Retrieved November 29, 2020 from https:\/\/www.statista.com\/statistics\/379046\/worldwide-retail-e-commerce-sales\/ Statista Research Department. 2020. Global retail e-commerce sales 2014-2023. Retrieved November 29, 2020 from https:\/\/www.statista.com\/statistics\/379046\/worldwide-retail-e-commerce-sales\/"},{"key":"e_1_3_2_1_32_1","volume-title":"You Do Not Need a Bigger Boat: Recommendations at Reasonable Scale in a (Mostly) Serverless and Open Stack","author":"Tagliabue Jacopo","unstructured":"Jacopo Tagliabue . 2021. You Do Not Need a Bigger Boat: Recommendations at Reasonable Scale in a (Mostly) Serverless and Open Stack . Association for Computing Machinery , New York, NY, USA , 598\u2013600. https:\/\/doi.org\/10.1145\/3460231.3474604 10.1145\/3460231.3474604 Jacopo Tagliabue. 2021. You Do Not Need a Bigger Boat: Recommendations at Reasonable Scale in a (Mostly) Serverless and Open Stack. Association for Computing Machinery, New York, NY, USA, 598\u2013600. https:\/\/doi.org\/10.1145\/3460231.3474604"},{"key":"e_1_3_2_1_33_1","volume-title":"SIGIR 2021 E-Commerce Workshop Data Challenge. In SIGIR eCom","author":"Tagliabue Jacopo","year":"2021","unstructured":"Jacopo Tagliabue , Ciro Greco , Jean-Francis Roy , Federico Bianchi , Giovanni Cassani , Bingqing Yu , and Patrick\u00a0John Chia . 2021 . SIGIR 2021 E-Commerce Workshop Data Challenge. In SIGIR eCom 2021. Jacopo Tagliabue, Ciro Greco, Jean-Francis Roy, Federico Bianchi, Giovanni Cassani, Bingqing Yu, and Patrick\u00a0John Chia. 2021. SIGIR 2021 E-Commerce Workshop Data Challenge. In SIGIR eCom 2021."},{"key":"e_1_3_2_1_34_1","volume-title":"The Embeddings That Came in From the Cold: Improving Vectors for New and Rare Products with Content-Based Inference. In Fourteenth ACM Conference on Recommender Systems","author":"Tagliabue Jacopo","year":"2020","unstructured":"Jacopo Tagliabue , Bingqing Yu , and Federico Bianchi . 2020 . The Embeddings That Came in From the Cold: Improving Vectors for New and Rare Products with Content-Based Inference. In Fourteenth ACM Conference on Recommender Systems ( Virtual Event, Brazil) (RecSys \u201920). Association for Computing Machinery, New York, NY, USA, 577\u2013578. https:\/\/doi.org\/10.1145\/3383313.3411477 10.1145\/3383313.3411477 Jacopo Tagliabue, Bingqing Yu, and Federico Bianchi. 2020. The Embeddings That Came in From the Cold: Improving Vectors for New and Rare Products with Content-Based Inference. In Fourteenth ACM Conference on Recommender Systems(Virtual Event, Brazil) (RecSys \u201920). Association for Computing Machinery, New York, NY, USA, 577\u2013578. https:\/\/doi.org\/10.1145\/3383313.3411477"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959160"},{"key":"e_1_3_2_1_36_1","unstructured":"Menghan Wang Yujie Lin Guli Lin Keping Yang and Xiao-Ming Wu. 2020. M2GRL: A Multi-task Multi-view Graph Representation Learning Framework for Web-scale Recommender Systems. CoRR abs\/2005.10110(2020). arxiv:2005.10110https:\/\/arxiv.org\/abs\/2005.10110  Menghan Wang Yujie Lin Guli Lin Keping Yang and Xiao-Ming Wu. 2020. M2GRL: A Multi-task Multi-view Graph Representation Learning Framework for Web-scale Recommender Systems. CoRR abs\/2005.10110(2020). arxiv:2005.10110https:\/\/arxiv.org\/abs\/2005.10110"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3214306"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331267"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"crossref","unstructured":"Yixin Wang Dawen Liang Laurent Charlin and David\u00a0M. Blei. 2020. Causal Inference for Recommender Systems. In recsys2020.  Yixin Wang Dawen Liang Laurent Charlin and David\u00a0M. Blei. 2020. Causal Inference for Recommender Systems. In recsys2020.","DOI":"10.1145\/3383313.3412225"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3344257"}],"event":{"name":"WWW '22: The ACM Web Conference 2022","location":"Virtual Event, Lyon France","acronym":"WWW '22","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web"]},"container-title":["Companion Proceedings of the Web Conference 2022"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3487553.3524215","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3487553.3524215","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:33Z","timestamp":1750188633000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3487553.3524215"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,25]]},"references-count":40,"alternative-id":["10.1145\/3487553.3524215","10.1145\/3487553"],"URL":"https:\/\/doi.org\/10.1145\/3487553.3524215","relation":{},"subject":[],"published":{"date-parts":[[2022,4,25]]},"assertion":[{"value":"2022-08-16","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}