{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,12]],"date-time":"2026-06-12T03:16:26Z","timestamp":1781234186201,"version":"3.54.1"},"reference-count":131,"publisher":"Association for Computing Machinery (ACM)","issue":"1","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Recomm. Syst."],"published-print":{"date-parts":[[2026,3,31]]},"abstract":"<jats:p>\n            Although originally developed to evaluate\n            <jats:italic toggle=\"yes\">sets<\/jats:italic>\n            of items, recall is often used to evaluate\n            <jats:italic toggle=\"yes\">rankings<\/jats:italic>\n            of items, including those produced by recommender, retrieval, and other machine learning systems. The application of recall without a formal evaluative motivation has led to criticism of recall as a vague or inappropriate measure. In light of this debate, we reflect on the measurement of recall in rankings from a formal perspective. Our analysis is composed of three tenets: recall, robustness, and lexicographic evaluation. First, we formally define \u201crecall orientation\u201d as the sensitivity of a metric to a user interested in finding every relevant item. Second, we analyze recall orientation from the perspective of robustness with respect to possible content consumers and providers, connecting recall to recent conversations about fair ranking. Finally, we extend this conceptual and theoretical treatment of recall by developing a practical preference-based evaluation method based on lexicographic comparison. Through extensive empirical analysis across multiple recommendation and retrieval tasks, we establish that our new evaluation method, lexirecall, has convergent validity (i.e., it is correlated with existing recall metrics) and exhibits substantially higher sensitivity in terms of discriminative power and stability in the presence of missing labels. Our conceptual, theoretical, and empirical analysis substantially deepens our understanding of recall and motivates its adoption through connections to robustness and fairness.\n          <\/jats:p>","DOI":"10.1145\/3728373","type":"journal-article","created":{"date-parts":[[2025,4,4]],"date-time":"2025-04-04T07:13:34Z","timestamp":1743750814000},"page":"1-50","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Recall, Robustness, and Lexicographic Evaluation"],"prefix":"10.1145","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2345-1288","authenticated-orcid":false,"given":"Fernando","family":"Diaz","sequence":"first","affiliation":[{"name":"Carnegie Mellon University","place":["Pittsburgh, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2467-0108","authenticated-orcid":false,"given":"Michael D.","family":"Ekstrand","sequence":"additional","affiliation":[{"name":"Information Science, Drexel University","place":["Philadelphia, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5270-5550","authenticated-orcid":false,"given":"Bhaskar","family":"Mitra","sequence":"additional","affiliation":[{"name":"Research, Microsoft Corp","place":["Montr\u00e9al, Canada"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,7,29]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484081"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/0022-0531(88)90148-2"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1108\/eb026722"},{"key":"e_1_3_3_5_2","volume-title":"Proceedings of the 28th Text REtrieval Conference (TREC\u201919)","author":"Biega Asia J.","year":"2019","unstructured":"Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, and Sebastian Kohlmeier. 2019. Overview of the TREC 2019 fair ranking track. In Proceedings of the 28th Text REtrieval Conference (TREC\u201919)."},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210063"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630300505"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(82)90033-4"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484034"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3371844"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/1008992.1009000"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3412259"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010037"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2396782"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/2094072.2094076"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-0716-2197-4_16"},{"key":"e_1_3_3_17_2","volume-title":"Advances in Neural Information Processing Systems","author":"Chandar Praveen","year":"2020","unstructured":"Praveen Chandar, Fernando Diaz, and Brian St. Thomas. 2020. Beyond accuracy: Grounding evaluation metrics for human-machine learning systems. In Advances in Neural Information Processing Systems."},{"key":"e_1_3_3_18_2","first-page":"1248","volume-title":"Proceedings of the 16th ACM International Conference on Web Search and Data Mining (WSDM\u201923)","author":"Clarke Charles L. A.","year":"2023","unstructured":"Charles L. A. Clarke, Fernando Diaz, and Negar Arabzadeh. 2023. Preference-based offline evaluation. In Proceedings of the 16th ACM International Conference on Web Search and Data Mining (WSDM\u201923). Association for Computing Machinery, New York, NY, 1248\u20131251."},{"key":"e_1_3_3_19_2","volume-title":"Proceedings of the 22nd Text Retrieval Conference Proceedings (TREC\u201913)","author":"Collins-Thompson Kevyn","year":"2013","unstructured":"Kevyn Collins-Thompson, Paul Bennett, Fernando Diaz, Charlie Clarke, and Ellen Voorhees. 2013. TREC 2013 web track overview. In Proceedings of the 22nd Text Retrieval Conference Proceedings (TREC\u201913). NIST."},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1177\/0305735617751050"},{"key":"e_1_3_3_21_2","volume-title":"Research Synthesis and Meta-analysis: A Step-by-step Approach","author":"Cooper Harris M.","year":"2016","unstructured":"Harris M. Cooper. 2016. Research Synthesis and Meta-analysis: A Step-by-step Approach. SAGE, Thousand Oaks, CA."},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1016\/0020-0271(71)90024-6"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1002\/asi.5090190108"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911510"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080812"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080657"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.2307\/2297061"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/2808194.2809491"},{"key":"e_1_3_3_29_2","volume-title":"Proceedings of the 10th International Workshop on Evaluating Information Access co-located with the 17th NTCIR Conference on the Evaluation of Information Access Technologies (NTCIR\u201923)","author":"Diaz Fernando","year":"2023","unstructured":"Fernando Diaz. 2023. Best-case retrieval evaluation: Improving the sensitivity of reciprocal rank with lexicographic precision. In Proceedings of the 10th International Workshop on Evaluating Information Access co-located with the 17th NTCIR Conference on the Evaluation of Information Access Technologies (NTCIR\u201923)."},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3532033"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3411962"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/1242572.1242651"},{"key":"e_1_3_3_33_2","article-title":"Robustness in deep learning for computer vision: Mind the gap?","volume":"2112","author":"Drenkow Nathan","year":"2021","unstructured":"Nathan Drenkow, Numair Sani, Ilya Shpitser, and Mathias Unberath. 2021. Robustness in deep learning for computer vision: Mind the gap? CoRR abs\/2112.00639.","journal-title":"CoRR"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8691.2006.00392.x"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3412778"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1561\/9781638280415"},{"key":"e_1_3_3_37_2","volume-title":"Proceedings of the 30th International Florida Artificial Intelligence Research Society Conference","author":"Ekstrand Michael D.","year":"2017","unstructured":"Michael D. Ekstrand and Vaibhav Mahant. 2017. Sturgeon and the cool kids: Problems with random decoys for Top-N recommender evaluation. In Proceedings of the 30th International Florida Artificial Intelligence Research Society Conference."},{"key":"e_1_3_3_38_2","first-page":"172","volume-title":"Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research)","volume":"81","author":"Ekstrand Michael D.","year":"2018","unstructured":"Michael D. Ekstrand, Mucun Tian, Ion Madrazo Azpiazu, Jennifer D. Ekstrand, Oghenemaro Anuyah, David McNeill, and Maria Soledad Pera. 2018. All the cool kids, how do they fit in?: Popularity and demographic biases in recommender evaluation and effectiveness. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research), Sorelle A. Friedler and Christo Wilson (Eds.), Vol. 81. PMLR, New York, NY, 172\u2013186."},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/2808194.2809452"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3523227.3551476"},{"key":"e_1_3_3_41_2","volume-title":"Advances in Neural Information Processing Systems","author":"Flach Peter","year":"2015","unstructured":"Peter Flach and Meelis Kull. 2015. Precision-recall-gain curves: PR analysis done right. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc."},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1093\/aristotelian\/56.1.167"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445423"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210012"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1093\/oxfordhb\/9780199603510.013.018"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199210)43:9<602::AID-ASI3>3.0.CO;2-Q"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.5555\/231880.231887"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-45442-5_21"},{"key":"e_1_3_3_49_2","first-page":"1929","volume-title":"Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research)","volume":"80","author":"Hashimoto Tatsunori","year":"2018","unstructured":"Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. 2018. Fairness without demographics in repeated loss minimization. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research), Jennifer Dy and Andreas Krause (Eds.), Vol. 80. PMLR, Stockholmsm\u00e4ssan, Stockholm Sweden, 1929\u20131938."},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287584"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/963770.963772"},{"key":"e_1_3_3_52_2","article-title":"Bridging language and items for retrieval and recommendation","author":"Hou Yupeng","year":"2024","unstructured":"Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley. 2024. Bridging language and items for retrieval and recommendation. arXiv:2403.03952. Retrieved from https:\/\/arxiv.org\/abs\/2403.03952","journal-title":"arXiv:2403.03952"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/WI-IAT59888.2023.00018"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445901"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3637528.3671687"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/1185877.1185883"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/959258.959260"},{"key":"e_1_3_3_58_2","volume-title":"Justice and Equity","author":"Kolm Serge-Christophe","year":"2002","unstructured":"Serge-Christophe Kolm. 2002. Justice and Equity. MIT Press."},{"key":"e_1_3_3_59_2","unstructured":"Liu Leqi Fatma Kilinc-Karzan Zachary C. Lipton and Alan L. Montgomery. 2021. Rebounding bandits for modeling satiation effects. In Proceedings of the 35th International Conference on Neural Information Processing Systems. ACM New York NY."},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3482415"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/2600428.2609618"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-018-9209-6"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-53817-3"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-012-9209-9"},{"key":"e_1_3_3_65_2","unstructured":"Iain Mackie Shubham Chatterjee and Jeffrey Dalton. 2023. Generative and pseudo-relevant feedback for sparse dense and learned sparse retrieval. arxiv:cs.IR\/2305.07477. Retrieved from https:\/\/arxiv.org\/abs\/2305.07477"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835551"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-15998-5_10"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3041021.3054197"},{"key":"e_1_3_3_69_2","unstructured":"Omid Memarrast Ashkan Rezaei Rizal Fathony and Brian D. Ziebart. 2021. Fairness for robust learning to rank. arXiv:2112.06288. Retrieved from https:\/\/arxiv.org\/abs\/2112.06288"},{"key":"e_1_3_3_70_2","first-page":"6987","volume-title":"Proceedings of the 37th International Conference on Machine Learning (ICML\u201920)","volume":"119","author":"Mladenov Martin","year":"2020","unstructured":"Martin Mladenov, Elliot Creager, Omer Ben-Porat, Kevin Swersky, Richard S. Zemel, and Craig Boutilier. 2020. Optimizing long-term social welfare in recommender systems: A constrained matching approach. In Proceedings of the 37th International Conference on Machine Learning (ICML\u201920), Vol. 119. PMLR, 6987\u20136998."},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/1278366.1278372"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-45068-6_1"},{"key":"e_1_3_3_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/1416950.1416952"},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210005"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401099"},{"key":"e_1_3_3_76_2","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/2954.001.0001"},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-99736-6_43"},{"key":"e_1_3_3_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401102"},{"key":"e_1_3_3_79_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-009-9124-x"},{"key":"e_1_3_3_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/1281192.1281254"},{"issue":"2","key":"e_1_3_3_81_2","first-page":"141","article-title":"Some reasons for the maximin criterion","volume":"64","author":"Rawls John","year":"1974","unstructured":"John Rawls. 1974. Some reasons for the maximin criterion. Am. Econ. Rev. 64, 2 (1974), 141\u2013146.","journal-title":"Am. Econ. Rev."},{"key":"e_1_3_3_82_2","doi-asserted-by":"publisher","DOI":"10.2307\/j.ctv31xf5v0"},{"key":"e_1_3_3_83_2","doi-asserted-by":"publisher","DOI":"10.1108\/eb026647"},{"key":"e_1_3_3_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390453"},{"key":"e_1_3_3_85_2","volume-title":"Information Storage and Retrieval","author":"Rocchio Joseph John","year":"1964","unstructured":"Joseph John Rocchio. 1964. Performance indices for document retrieval systems. In Information Storage and Retrieval, Gerald Salton (Ed.). Number ISR-8. Harvard University, Chapter III."},{"key":"e_1_3_3_86_2","volume-title":"Proceedings of the 24th Text REtrieval Conference TREC\u201915)","volume":"500","author":"Roegiest Adam","year":"2015","unstructured":"Adam Roegiest, Gordon V. Cormack, Charles L. A. Clarke, and Maura R. Grossman. 2015. TREC 2015 total recall track overview. In Proceedings of the 24th Text REtrieval Conference TREC\u201915), Ellen M. Voorhees and Angela Ellis (Eds.), Vol. 500-319. National Institute of Standards and Technology (NIST)."},{"key":"e_1_3_3_87_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1195"},{"key":"e_1_3_3_88_2","doi-asserted-by":"publisher","DOI":"10.1108\/JD-02-2014-0031"},{"key":"e_1_3_3_89_2","doi-asserted-by":"publisher","DOI":"10.5555\/3327345.3327429"},{"key":"e_1_3_3_90_2","volume-title":"A Further Note on Evaluation Metrics for the Task of Finding One Highly Relevant Document","author":"Sakai Tetsuya","year":"2006","unstructured":"Tetsuya Sakai. 2006. A Further Note on Evaluation Metrics for the Task of Finding One Highly Relevant Document. Technical Report 33(2006-DD-054). Toshiba Corporate R&D Center."},{"key":"e_1_3_3_91_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-54798-0_6"},{"key":"e_1_3_3_92_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-008-9059-7"},{"key":"e_1_3_3_93_2","volume-title":"Proceedings of the 2nd International Workshop on Evaluating Information Access (EVIA\u201908)","author":"Sakai Tetsuya","year":"2008","unstructured":"Tetsuya Sakai and Stephen Robertson. 2008. Modelling a user population for designing information retrieval metrics. In Proceedings of the 2nd International Workshop on Evaluating Information Access (EVIA\u201908)."},{"key":"e_1_3_3_94_2","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835542"},{"key":"e_1_3_3_95_2","doi-asserted-by":"publisher","DOI":"10.5555\/2800520"},{"key":"e_1_3_3_96_2","volume-title":"Collective Choice and Social Welfare","author":"Sen Amartya","year":"1970","unstructured":"Amartya Sen. 1970. Collective Choice and Social Welfare. Holden-Day."},{"key":"e_1_3_3_97_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00135080"},{"key":"e_1_3_3_98_2","doi-asserted-by":"publisher","DOI":"10.2307\/1913949"},{"key":"e_1_3_3_99_2","doi-asserted-by":"publisher","DOI":"10.1145\/3461702.3462592"},{"key":"e_1_3_3_100_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781139136617"},{"key":"e_1_3_3_101_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220088"},{"key":"e_1_3_3_102_2","doi-asserted-by":"publisher","DOI":"10.1145\/1321440.1321528"},{"key":"e_1_3_3_103_2","doi-asserted-by":"publisher","DOI":"10.1145\/2124295.2124348"},{"key":"e_1_3_3_104_2","volume-title":"Report on the Need for and Provision for an \u2018IDEAL\u2019 Information Retrieval Test Collection","author":"Jones Karen Sp\u00e4rck","year":"1975","unstructured":"Karen Sp\u00e4rck Jones and Cornelis J. van Rijsbergen. 1975. Report on the Need for and Provision for an \u2018IDEAL\u2019 Information Retrieval Test Collection. British Library Research and Development Report 5266. Computer Laboratory, University of Cambridge."},{"key":"e_1_3_3_105_2","doi-asserted-by":"publisher","DOI":"10.1177\/0022242921997082"},{"key":"e_1_3_3_106_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460231.3478848"},{"key":"e_1_3_3_107_2","doi-asserted-by":"publisher","DOI":"10.1145\/1721831.1721835"},{"key":"e_1_3_3_108_2","article-title":"Commentary on achieving quality in the E-Discovery process","volume":"10","author":"Conference The Sedona","year":"2009","unstructured":"The Sedona Conference. 2009. Commentary on achieving quality in the E-Discovery process. Sedona Conf. J. 10 (2009).","journal-title":"Sedona Conf. J."},{"key":"e_1_3_3_109_2","doi-asserted-by":"publisher","DOI":"10.1145\/3343413.3378004"},{"key":"e_1_3_3_110_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-53817-3_6"},{"key":"e_1_3_3_111_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-53817-3_5"},{"key":"e_1_3_3_112_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-020-09377-x"},{"key":"e_1_3_3_113_2","volume-title":"Proceedings of the 13th Text REtrieval Conference (TREC\u201904)","author":"Voorhees E. M.","year":"2004","unstructured":"E. M. Voorhees. 2004. Overview of the TREC 2004 robust track. In Proceedings of the 13th Text REtrieval Conference (TREC\u201904)."},{"key":"e_1_3_3_114_2","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291017"},{"key":"e_1_3_3_115_2","doi-asserted-by":"publisher","DOI":"10.6028\/NIST.SP.500-240.overview-overview"},{"key":"e_1_3_3_116_2","doi-asserted-by":"publisher","DOI":"10.1145\/3523227.3546780"},{"key":"e_1_3_3_117_2","doi-asserted-by":"publisher","DOI":"10.1145\/3406522.3446019"},{"key":"e_1_3_3_118_2","volume-title":"Proceedings of the 1st Workshop on Normative Design and Evaluation of Recommender Systems","author":"Vrijenhoek Sanne","year":"2023","unstructured":"Sanne Vrijenhoek, Lien Michiels, Johannes Kruse, Jordi Viader Guerrero, Alain Starke, and Nava Tintarev (Eds.). 2023. Proceedings of the 1st Workshop on Normative Design and Evaluation of Recommender Systems."},{"key":"e_1_3_3_119_2","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348385"},{"key":"e_1_3_3_120_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11002-013-9238-1"},{"key":"e_1_3_3_121_2","doi-asserted-by":"publisher","DOI":"10.1145\/3485447.3512255"},{"key":"e_1_3_3_122_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.676"},{"key":"e_1_3_3_123_2","doi-asserted-by":"publisher","DOI":"10.5555\/3495724.3496858"},{"key":"e_1_3_3_124_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0377-2217(96)00217-2"},{"key":"e_1_3_3_125_2","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3371818"},{"key":"e_1_3_3_126_2","first-page":"15269","volume-title":"Advances in Neural Information Processing Systems 32","author":"Zhang Xueru","year":"2019","unstructured":"Xueru Zhang, Mohammadmahdi Khaliligarekani, Cem Tekin, and mingyan liu. 2019. Group retention when using machine learning in sequential decision making: The interplay between user dynamics and fairness. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d\u2019Alch\u00e9-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 15269\u201315278."},{"key":"e_1_3_3_127_2","doi-asserted-by":"publisher","DOI":"10.1145\/3545796"},{"key":"e_1_3_3_128_2","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864759"},{"key":"e_1_3_3_129_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3462948"},{"key":"e_1_3_3_130_2","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291014"},{"key":"e_1_3_3_131_2","doi-asserted-by":"publisher","DOI":"10.1145\/1670598.1670600"},{"key":"e_1_3_3_132_2","doi-asserted-by":"publisher","DOI":"10.1145\/3388640"}],"container-title":["ACM Transactions on Recommender Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3728373","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,29]],"date-time":"2025-07-29T12:47:15Z","timestamp":1753793235000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3728373"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,29]]},"references-count":131,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,3,31]]}},"alternative-id":["10.1145\/3728373"],"URL":"https:\/\/doi.org\/10.1145\/3728373","relation":{},"ISSN":["2770-6699"],"issn-type":[{"value":"2770-6699","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,29]]},"assertion":[{"value":"2024-03-07","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-04","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-29","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}