{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T02:34:16Z","timestamp":1783737256845,"version":"3.55.0"},"reference-count":66,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2025,3,22]],"date-time":"2025-03-22T00:00:00Z","timestamp":1742601600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"the MSIT (Ministry of Science, ICT), Korea","award":["2022-00155958"],"award-info":[{"award-number":["2022-00155958"]}]},{"name":"IITP"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Recomm. Syst."],"published-print":{"date-parts":[[2025,9,30]]},"abstract":"<jats:p>In recent years, recommender systems have achieved remarkable performance by using ensembles of heterogeneous models. However, this approach is costly due to the resources and inference latency proportional to the number of models, creating a bottleneck for production. Our work aims at transfering the ensemble knowledge of heterogeneous teachers to a lightweight student model using knowledge distillation (KD), reducing inference costs while maintaining high accuracy. We find that the efficacy of distillation decreases when transferring knowledge from heterogeneous teachers. To address this, we propose a new KD framework, named HetComp, that guides the student model by transferring easy-to-hard sequences of knowledge generated from teachers\u2019 trajectories. HetComp uses dynamic knowledge construction to provide progressively difficult ranking knowledge and adaptive knowledge transfer to gradually transfer finer-grained ranking information. Although HetComp improves accuracy, it exacerbates popularity bias, resulting in a high popularity lift. To mitigate this issue, we introduce two strategies that leverage models\u2019 disagreement knowledge (i.e., dissensus) for heterogeneous comparison. Our experiments demonstrate that HetComp significantly enhances distillation quality and the student model\u2019s generalization capabilities. Furthermore, we provide extensive experimental results supporting the effectiveness of our dissensus-based debiasing techniques in mitigating the popularity lift caused by HetComp.<\/jats:p>","DOI":"10.1145\/3649443","type":"journal-article","created":{"date-parts":[[2024,2,23]],"date-time":"2024-02-23T12:05:03Z","timestamp":1708689903000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Unbiased, Effective, and Efficient Distillation from Heterogeneous Models for Recommender Systems"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5528-1426","authenticated-orcid":false,"given":"Seongku","family":"Kang","sequence":"first","affiliation":[{"name":"UIUC, Urbana, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8813-3179","authenticated-orcid":false,"given":"Wonbin","family":"Kweon","sequence":"additional","affiliation":[{"name":"POSTECH, Pohang, South Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0694-6385","authenticated-orcid":false,"given":"Dongha","family":"Lee","sequence":"additional","affiliation":[{"name":"Yonsei University, Seoul, South Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3108-5601","authenticated-orcid":false,"given":"Jianxun","family":"Lian","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8608-8482","authenticated-orcid":false,"given":"Xing","family":"Xie","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7510-0255","authenticated-orcid":false,"given":"Hwanjo","family":"Yu","sequence":"additional","affiliation":[{"name":"POSTECH, Pohang, South Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,3,22]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3418487"},{"key":"e_1_3_2_3_2","unstructured":"Samira Abnar Mostafa Dehghani and Willem Zuidema. 2020. Transferring inductive biases through knowledge distillation. arXiv:2006.00555. Retrieved from https:\/\/arxiv.org\/abs\/2006.00555"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553380"},{"key":"e_1_3_2_5_2","first-page":"4750","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Cazenavette George","year":"2022","unstructured":"George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. 2022. Dataset distillation by matching training trajectories. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 4750\u20134759."},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539597.3570477"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3564284"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539452"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11783"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467132"},{"key":"e_1_3_2_11_2","volume-title":"Proceedings of the NeurIPS","author":"Ding Jingtao","year":"2020","unstructured":"Jingtao Ding, Yuhan Quan, Quanming Yao, Yong Li, and Depeng Jin. 2020. Simplify and robustify negative sampling for implicit collaborative filtering. In Proceedings of the NeurIPS."},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2017-614"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401063"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052569"},{"key":"e_1_3_2_15_2","unstructured":"Geoffrey Hinton Oriol Vinyals and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)."},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052639"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v29i1.9608"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00143"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3412005"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2021.08.060"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467319"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3357384.3357914"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3543507.3583209"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3485447.3512070"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2021.107958"},{"key":"e_1_3_2_26_2","volume-title":"NIPS","unstructured":"M. Kumar, Benjamin Packer, and Daphne Koller. 2010. Self-paced learning for latent variable models. In NIPS, Vol. 23, Curran Associates, Inc."},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3449878"},{"key":"e_1_3_2_28_2","volume-title":"Advances in Neural Information Processing Systems","author":"lan Xu","year":"2018","unstructured":"Xu lan, Xiatian Zhu, and Shaogang Gong. 2018. Knowledge distillation by on-the-fly native ensemble. In Advances in Neural Information Processing Systems, 7527--7537."},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3462935"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3482093"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3587272"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186150"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3582002"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3568030"},{"key":"e_1_3_2_35_2","article-title":"An easy-to-hard learning paradigm for multiple classes and multiple labels","author":"Liu Weiwei","year":"2017","unstructured":"Weiwei Liu, Ivor W. Tsang, and Klaus-Robert M\u00fcller. 2017. An easy-to-hard learning paradigm for multiple classes and multiple labels. The Journal of Machine Learning Research 18, 94 (2017), 1--38. http:\/\/jmlr.org\/papers\/v18\/16-212.html","journal-title":"The Journal of Machine Learning Research"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2020.07.048"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401094"},{"key":"e_1_3_2_38_2","volume-title":"Analyzing and Modeling Rank Data","author":"Marden John I.","year":"1996","unstructured":"John I. Marden. 1996. Analyzing and Modeling Rank Data. CRC Press."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3365375"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2019.01.012"},{"key":"e_1_3_2_41_2","volume-title":"Proceedings of the International Conference on Artificial Intelligence and Statistics","author":"Reddi Sashank","year":"2021","unstructured":"Sashank Reddi, Rama Kumar Pasumarthi, Aditya Menon, Ankit Singh Rawat, Felix Yu, Seungyeon Kim, Andreas Veit, and Sanjiv Kumar. 2021. Rankdistil: Knowledge distillation for ranking. In Proceedings of the International Conference on Artificial Intelligence and Statistics. PMLR."},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/2556195.2556248"},{"key":"e_1_3_2_43_2","volume-title":"UAI","author":"Rendle Steffen","year":"2009","unstructured":"Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In UAI, 452--461."},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3523227.3546757"},{"key":"e_1_3_2_45_2","unstructured":"Adriana Romero Nicolas Ballas Samira Ebrahimi Kahou Antoine Chassang Carlo Gatta and Yoshua Bengio. 2014. Fitnets: Hints for thin deep nets. arXiv:1412.6550. Retrieved from https:\/\/arxiv.org\/abs\/1412.6550"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/2740908.2742726"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-86523-8_36"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3568392"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220021"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/667"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3449898"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2021.3069908"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467289"},{"key":"e_1_3_2_54_2","volume-title":"Advances in Neural Information Processing Systems","author":"Weimer Markus","year":"2007","unstructured":"Markus Weimer, Alexandros Karatzoglou, Quoc Le, and Alex Smola. 2007. COFI RANK - maximum margin matrix factorization for collaborative ranking. In Advances in Neural Information Processing Systems, John C. Platt, Daphne Koller, Yoram Singer, and Sam T. Roweis (Eds.). 1593--1600."},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-acl.387"},{"key":"e_1_3_2_56_2","article-title":"When do curricula work?","author":"Wu Xiaoxia","year":"2021","unstructured":"Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. 2021. When do curricula work? In ICLR. https:\/\/openreview.net\/forum?id=tW4QEInpni","journal-title":"ICLR"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/1390156.1390306"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531775"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58558-7_15"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098135"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i16.17680"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531791"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3543507.3583461"},{"key":"e_1_3_2_64_2","doi-asserted-by":"crossref","unstructured":"Zihao Zhao Jiawei Chen Sheng Zhou Xiangnan He Xuezhi Cao Fuzheng Zhang and Wei Wu. 2023. Popularity bias is not always evil: Disentangling benign and harmful bias for recommendation. IEEE Trans. on Knowl. and Data Eng. 35 10 (Oct 2023) 9920--9931.","DOI":"10.1109\/TKDE.2022.3218994"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3412704"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-emnlp.111"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313678"}],"container-title":["ACM Transactions on Recommender Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3649443","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3649443","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:03:21Z","timestamp":1750291401000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3649443"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,22]]},"references-count":66,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,9,30]]}},"alternative-id":["10.1145\/3649443"],"URL":"https:\/\/doi.org\/10.1145\/3649443","relation":{},"ISSN":["2770-6699"],"issn-type":[{"value":"2770-6699","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,22]]},"assertion":[{"value":"2023-07-12","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-29","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}