{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T16:16:16Z","timestamp":1783700176510,"version":"3.55.0"},"reference-count":32,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2025,5,9]],"date-time":"2025-05-09T00:00:00Z","timestamp":1746748800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2025,7,31]]},"abstract":"<jats:p>\n            The data sparsity problem has been a long-standing obstacle towards achieving better recommendation performance since it is miserable to estimate the user\u2019s interests from limited historical behaviors. The pre-training paradigm, i.e., learning universal knowledge across a wide spectrum of domains, has increasingly become a new de-facto practice in many fields, especially for adaption to new domains. The merit of this superior generalizability renders it a natural choice to tackle the data sparsity problem for various recommendation scenarios. Hence, several efforts mainly follow masked language modeling or simple data augmentation via contrastive learning to build a pre-trained recommendation model. Our recent work (namely\n            <jats:sc>Miracle<\/jats:sc>\n            ) suggests that the common treatment utilizing the masked language modeling is not sufficient for pre-training a recommender system, since a user\u2019s intent could be more complex than predicting the next word or item. The encouraging results demonstrate that the multi-interest modeling could significantly push the frontier of recommender system pre-training. Nevertheless, how to accommodate the temporal dynamics of the user interests seems to be underexplored under both single vector representation and multi-interest schemes. In this article, we aim to incorporate sophisticated temporal information modeling with the current advance in this line. More specifically, we extend\n            <jats:sc>Miracle<\/jats:sc>\n            by further considering relative position information and two kinds of relative time interval information jointly when performing multi-interest learning. Then, a sequential process for interest refinement is proposed to learn the subtle nuances of how interests change and shift along the timeline, leading to a more precise representation of user interests. Our extensive experiments on multiple real-world datasets validate the effectiveness of the proposed solution, demonstrating a significant improvement over current state-of-the-art models on these benchmarks. The code is available at\n            <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/WHUIR\/Horae\">https:\/\/github.com\/WHUIR\/Horae<\/jats:ext-link>\n            .\n          <\/jats:p>","DOI":"10.1145\/3727645","type":"journal-article","created":{"date-parts":[[2025,4,1]],"date-time":"2025-04-01T16:18:01Z","timestamp":1743524281000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["HORAE: Temporal Multi-Interest Pre-training for Sequential Recommendation"],"prefix":"10.1145","volume":"43","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-2151-3408","authenticated-orcid":false,"given":"Shirui","family":"Hu","sequence":"first","affiliation":[{"name":"Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, Wuhan, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4575-9419","authenticated-orcid":false,"given":"Weichang","family":"Wu","sequence":"additional","affiliation":[{"name":"Ant Group, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-8335-1157","authenticated-orcid":false,"given":"Zuoli","family":"Tang","sequence":"additional","affiliation":[{"name":"Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, Wuhan, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3611-0901","authenticated-orcid":false,"given":"Zhaoxin","family":"Huan","sequence":"additional","affiliation":[{"name":"Ant Group, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9807-9479","authenticated-orcid":false,"given":"Lin","family":"Wang","sequence":"additional","affiliation":[{"name":"Ant Group, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8055-0245","authenticated-orcid":false,"given":"Xiaolu","family":"Zhang","sequence":"additional","affiliation":[{"name":"Ant Group, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6033-6102","authenticated-orcid":false,"given":"Jun","family":"Zhou","sequence":"additional","affiliation":[{"name":"Ant Group, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6755-871X","authenticated-orcid":false,"given":"Lixin","family":"Zou","sequence":"additional","affiliation":[{"name":"Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, Wuhan, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3144-6374","authenticated-orcid":false,"given":"Chenliang","family":"Li","sequence":"additional","affiliation":[{"name":"Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, Wuhan, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,5,9]]},"reference":[{"key":"e_1_3_2_2_2","article-title":"Layer normalization","author":"Ba Jimmy Lei","year":"2016","unstructured":"Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv:1607.06450. Retrieved from https:\/\/arxiv.org\/abs\/1607.06450","journal-title":"arXiv:1607.06450. Retrieved from"},{"key":"e_1_3_2_3_2","doi-asserted-by":"crossref","unstructured":"Yukuo Cen Jianwei Zhang Xu Zou Chang Zhou Hongxia Yang and Jie Tang. 2020. Controllable multi-interest framework for recommendation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining 2942\u20132951.","DOI":"10.1145\/3394486.3403344"},{"key":"e_1_3_2_4_2","doi-asserted-by":"crossref","unstructured":"Zheng Chai Zhihong Chen Chenliang Li Rong Xiao Houyi Li Jiawei Wu Jingxu Chen and Haihong Tang. 2022. User-aware multi-interest learning for candidate matching in recommenders. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval 1326\u20131335.","DOI":"10.1145\/3477495.3532073"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2023.3324312"},{"key":"e_1_3_2_6_2","article-title":"Bert: Pre-training of deep bidirectional transformers for language understanding","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. Retrieved from https:\/\/arxiv.org\/abs\/1810.04805","journal-title":"arXiv:1810.04805. Retrieved from"},{"key":"e_1_3_2_7_2","unstructured":"Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 770\u2013778."},{"key":"e_1_3_2_8_2","article-title":"Session-based recommendations with recurrent neural networks","author":"Hidasi Bal\u00e1zs","year":"2015","unstructured":"Bal\u00e1zs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2015. Session-based recommendations with recurrent neural networks. arXiv:1511.06939. Retrieved from https:\/\/arxiv.org\/abs\/1511.06939","journal-title":"arXiv:1511.06939. Retrieved from"},{"key":"e_1_3_2_9_2","doi-asserted-by":"crossref","unstructured":"Yupeng Hou Shanlei Mu Wayne Xin Zhao Yaliang Li Bolin Ding and Ji-Rong Wen. 2022. Towards universal sequence representation learning for recommender systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining 585\u2013593.","DOI":"10.1145\/3534678.3539381"},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","unstructured":"Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recommendation. In Proceedings of the 2018 IEEE International Conference on Data Mining (ICDM). IEEE 197\u2013206.","DOI":"10.1109\/ICDM.2018.00035"},{"key":"e_1_3_2_11_2","article-title":"Adam: A method for stochastic optimization","author":"Kingma Diederik P.","year":"2014","unstructured":"Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980. Retrieved from https:\/\/arxiv.org\/abs\/1412.6980","journal-title":"arXiv:1412.6980. Retrieved from"},{"key":"e_1_3_2_12_2","article-title":"On the sentence embeddings from pre-trained language models","author":"Li Bohan","year":"2020","unstructured":"Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. arXiv:2011.05864. Retrieved from https:\/\/arxiv.org\/abs\/2011.05864","journal-title":"arXiv:2011.05864. Retrieved from"},{"key":"e_1_3_2_13_2","unstructured":"Chao Li Zhiyuan Liu Mengmeng Wu Yuchi Xu Huan Zhao Pipei Huang Guoliang Kang Qiwei Chen Wei Li and Dik Lun Lee. 2019. Multi-interest network with dynamic routing for recommendation at Tmall. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management 2615\u20132623."},{"key":"e_1_3_2_14_2","unstructured":"Jiacheng Li Yujie Wang and Julian McAuley. 2020. Time interval aware self-attention for sequential recommendation. In Proceedings of the 13th International Conference on Web Search and Data Mining 322\u2013330."},{"key":"e_1_3_2_15_2","article-title":"Contrastive self-supervised sequential recommendation with robust augmentation","author":"Liu Zhiwei","year":"2021","unstructured":"Zhiwei Liu, Yongjun Chen, Jia Li, Philip S. Yu, Julian McAuley, and Caiming Xiong. 2021. Contrastive self-supervised sequential recommendation with robust augmentation. arXiv:2108.06479. Retrieved from https:\/\/arxiv.org\/abs\/2108.06479","journal-title":"arXiv:2108.06479. Retrieved from"},{"key":"e_1_3_2_16_2","unstructured":"Jianmo Ni Jiacheng Li and Julian McAuley. 2019. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) 188\u2013197."},{"key":"e_1_3_2_17_2","doi-asserted-by":"crossref","unstructured":"Yuqi Qin Pengfei Wang and Chenliang Li. 2021. The world is binary: Contrastive learning for denoising next basket recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval 859\u2013868.","DOI":"10.1145\/3404835.3462836"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3473339"},{"key":"e_1_3_2_19_2","doi-asserted-by":"crossref","unstructured":"Ruihong Qiu Jingjing Li Zi Huang and Hongzhi Yin. 2019. Rethinking the item order in session-based recommendation with graph neural networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management 579\u2013588.","DOI":"10.1145\/3357384.3358010"},{"key":"e_1_3_2_20_2","unstructured":"Ruihong Qiu Hongzhi Yin Zi Huang and Tong Chen. 2020. Gag: Global attributed graph neural network for streaming session-based recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval 669\u2013678."},{"key":"e_1_3_2_21_2","doi-asserted-by":"crossref","unstructured":"Zhaopeng Qiu Xian Wu Jingyue Gao and Wei Fan. 2021. U-BERT: Pre-training user representations for improved recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence Vol. 35 4320\u20134327.","DOI":"10.1609\/aaai.v35i5.16557"},{"key":"e_1_3_2_22_2","doi-asserted-by":"crossref","unstructured":"Steffen Rendle Christoph Freudenthaler and Lars Schmidt-Thieme. 2010. Factorizing personalized Markov chains for next-basket recommendation. In Proceedings of the 19th International Conference on World Wide Web 811\u2013820.","DOI":"10.1145\/1772690.1772773"},{"key":"e_1_3_2_23_2","article-title":"Dynamic routing between capsules","volume":"30","author":"Sabour Sara","year":"2017","unstructured":"Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton. 2017. Dynamic routing between capsules. In Proceedings of the Advances in Neural Information Processing Systems 30.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_24_2","article-title":"Outrageously large neural networks: The sparsely-gated mixture-of-experts layer","author":"Shazeer Noam","year":"2017","unstructured":"Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv:1701.06538. Retrieved from https:\/\/arxiv.org\/abs\/1701.06538","journal-title":"arXiv:1701.06538. Retrieved from"},{"key":"e_1_3_2_25_2","article-title":"Roformer: Enhanced transformer with rotary position embedding","author":"Su Jianlin","year":"2021","unstructured":"Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2021. Roformer: Enhanced transformer with rotary position embedding. arXiv:2104.09864. Retrieved from https:\/\/arxiv.org\/abs\/2104.09864","journal-title":"arXiv:2104.09864. Retrieved from"},{"key":"e_1_3_2_26_2","doi-asserted-by":"crossref","unstructured":"Fei Sun Jun Liu Jian Wu Changhua Pei Xiao Lin Wenwu Ou and Peng Jiang. 2019. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management 1441\u20131450.","DOI":"10.1145\/3357384.3357895"},{"key":"e_1_3_2_27_2","unstructured":"Qiaoyu Tan Jianwei Zhang Jiangchao Yao Ninghao Liu Jingren Zhou Hongxia Yang and Xia Hu. 2021. Sparse-interest network for sequential recommendation. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining 598\u2013606."},{"key":"e_1_3_2_28_2","doi-asserted-by":"crossref","unstructured":"Zuoli Tang Lin Wang Lixin Zou Xiaolu Zhang Jun Zhou and Chenliang Li. 2023. Towards multi-interest pre-training with sparse capsule network. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval 311\u2013320.","DOI":"10.1145\/3539618.3591778"},{"key":"e_1_3_2_29_2","doi-asserted-by":"crossref","unstructured":"Yu Tian Jianxin Chang Yanan Niu Yang Song and Chenliang Li. 2022. When multi-level meets multi-interest: A multi-grained neural model for sequential recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval 1632\u20131641.","DOI":"10.1145\/3477495.3532081"},{"key":"e_1_3_2_30_2","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems 30.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.5555\/3367471.3367642"},{"key":"e_1_3_2_32_2","doi-asserted-by":"crossref","unstructured":"Kun Zhou Hui Wang Wayne Xin Zhao Yutao Zhu Sirui Wang Fuzheng Zhang Zhongyuan Wang and Ji-Rong Wen. 2020. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management 1893\u20131902.","DOI":"10.1145\/3340531.3411954"},{"key":"e_1_3_2_33_2","unstructured":"Simiao Zuo Haoming Jiang Zichong Li Tuo Zhao and Hongyuan Zha. 2020. Transformer Hawkes process. In Proceedings of the International Conference on Machine Learning. PMLR 11692\u201311702."}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3727645","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3727645","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:57:29Z","timestamp":1750298249000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3727645"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,9]]},"references-count":32,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,7,31]]}},"alternative-id":["10.1145\/3727645"],"URL":"https:\/\/doi.org\/10.1145\/3727645","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,9]]},"assertion":[{"value":"2024-07-11","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-28","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}