{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T00:56:26Z","timestamp":1778028986914,"version":"3.51.4"},"reference-count":74,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2019,3,16]],"date-time":"2019-03-16T00:00:00Z","timestamp":1552694400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100004543","name":"China Scholarship Council","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004543","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Ahold Delhaize"},{"name":"Innovation Center for Artificial Intelligence"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2019,7,31]]},"abstract":"<jats:p>Academic search engines have been widely used to access academic papers, where users\u2019 information needs are explicitly represented as search queries. Some modern recommender systems have taken one step further by predicting users\u2019 information needs without the presence of an explicit query. In this article, we examine an academic paper recommender that sends out paper recommendations in email newsletters, based on the users\u2019 browsing history on the academic search engine. Specifically, we look at users who regularly browse papers on the search engine, and we sign up for the recommendation newsletters for the first time. We address the task of reranking the recommendation candidates that are generated by a production system for such users.<\/jats:p>\n          <jats:p>We face the challenge that the users on whom we focus have not interacted with the recommender system before, which is a common scenario that every recommender system encounters when new users sign up. We propose an approach to reranking candidate recommendations that utilizes both paper content and user behavior. The approach is designed to suit the characteristics unique to our academic recommendation setting. For instance, content similarity measures can be used to find the closest match between candidate recommendations and the papers previously browsed by the user. To this end, we use a knowledge graph derived from paper metadata to compare entity similarities (papers, authors, and journals) in the embedding space. Since the users on whom we focus have no prior interactions with the recommender system, we propose a model to learn a mapping from users\u2019 browsed articles to user clicks on the recommendations. We combine both content and behavior into a hybrid reranking model that outperforms the production baseline significantly, providing a relative 13% increase in Mean Average Precision and 28% in Precision@1.<\/jats:p>\n          <jats:p>Moreover, we provide a detailed analysis of the model components, highlighting where the performance boost comes from. The obtained insights reveal useful components for the reranking process and can be generalized to other academic recommendation settings as well, such as the utility of graph embedding similarity. Also, recent papers browsed by users provide stronger evidence for recommendation than historical ones.<\/jats:p>","DOI":"10.1145\/3312528","type":"journal-article","created":{"date-parts":[[2019,3,18]],"date-time":"2019-03-18T12:09:30Z","timestamp":1552910970000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":47,"title":["Personalised Reranking of Paper Recommendations Using Paper Content and User Behavior"],"prefix":"10.1145","volume":"37","author":[{"given":"Xinyi","family":"Li","sequence":"first","affiliation":[{"name":"University of Amsterdam, The Netherlands and National University of Defense Technology, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yifan","family":"Chen","sequence":"additional","affiliation":[{"name":"University of Amsterdam, The Netherlands and National University of Defense Technology, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Benjamin","family":"Pettit","sequence":"additional","affiliation":[{"name":"Elsevier, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Maarten De","family":"Rijke","sequence":"additional","affiliation":[{"name":"University of Amsterdam, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,3,16]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the 4th Italian Information Retrieval Workshop (CEUR Workshop Proceedings)","volume":"964","author":"Aiolli Fabio","year":"2013","unstructured":"Fabio Aiolli . 2013 . A preliminary study on a recommender system for the million songs dataset challenge . In Proceedings of the 4th Italian Information Retrieval Workshop (CEUR Workshop Proceedings) , Vol. 964 . CEUR-WS.org, 73--83. Retrieved from http:\/\/ceur-ws.org\/Vol-964\/paper12.pdf. Fabio Aiolli. 2013. A preliminary study on a recommender system for the million songs dataset challenge. In Proceedings of the 4th Italian Information Retrieval Workshop (CEUR Workshop Proceedings), Vol. 964. CEUR-WS.org, 73--83. Retrieved from http:\/\/ceur-ws.org\/Vol-964\/paper12.pdf."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.5555\/3200334.3200389"},{"key":"e_1_2_1_3_1","unstructured":"Antoine Bordes Nicolas Usunier Alberto Garcia-Duran Jason Weston and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Adv. Neural Info. Process. Syst. 2787--2795.   Antoine Bordes Nicolas Usunier Alberto Garcia-Duran Jason Weston and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Adv. Neural Info. Process. Syst. 2787--2795."},{"key":"e_1_2_1_4_1","first-page":"23","article-title":"From RankNet to LambdaRank to LambdaMart: An overview","volume":"11","author":"Burges Christopher J. C.","year":"2010","unstructured":"Christopher J. C. Burges . 2010 . From RankNet to LambdaRank to LambdaMart: An overview . Learning 11 , 23 -- 581 (2010), 81. Christopher J. C. Burges. 2010. From RankNet to LambdaRank to LambdaMart: An overview. Learning 11, 23--581 (2010), 81.","journal-title":"Learning"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623663"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.5555\/2503308.2503357"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2014.112"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959185"},{"key":"e_1_2_1_9_1","unstructured":"Fan R. K. Chung. 1997. Spectral Graph Theory. Number 92. American Mathematical Soc.  Fan R. K. Chung. 1997. Spectral Graph Theory. Number 92. American Mathematical Soc."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959190"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864721"},{"key":"e_1_2_1_12_1","unstructured":"Van Dang. 2018. The Lemur Project-Wiki-RankLib. Lemur Project. Retrieved from https:\/\/sourceforge.net\/projects\/lemur\/.  Van Dang. 2018. The Lemur Project-Wiki-RankLib. Lemur Project. Retrieved from https:\/\/sourceforge.net\/projects\/lemur\/."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864770"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/963770.963776"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2021068"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080730"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864740"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2700495"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-27302-5_2"},{"key":"e_1_2_1_20_1","unstructured":"Google Scholar. 2018. Retrieved from https:\/\/scholar.google.com\/.  Google Scholar. 2018. Retrieved from https:\/\/scholar.google.com\/."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935926"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080777"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3057148.3057152"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 29th AAAI Conference on Artificial Intelligence. AAAI, 2404--2410","author":"Huang Wenyi","unstructured":"Wenyi Huang , Zhaohui Wu , Liang Chen , Prasenjit Mitra , and C. Lee Giles . 2015. A neural probabilistic model for context-based citation recommendation . In Proceedings of the 29th AAAI Conference on Artificial Intelligence. AAAI, 2404--2410 . Wenyi Huang, Zhaohui Wu, Liang Chen, Prasenjit Mitra, and C. Lee Giles. 2015. A neural probabilistic model for context-based citation recommendation. In Proceedings of the 29th AAAI Conference on Artificial Intelligence. AAAI, 2404--2410."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2365952.2366004"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1150402.1150429"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2487589"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611974348.15"},{"key":"e_1_2_1_29_1","first-page":"265","article-title":"Exploring behavior of e-journal users in science and technology: Transaction log analysis of elsevier\u2019s sciencedirect OnSite in Taiwan. Library Info","volume":"24","author":"Ke Hao-Ren","year":"2002","unstructured":"Hao-Ren Ke , Rolf Kwakkelaar , Yu-Min Tai , and Li-Chun Chen . 2002 . Exploring behavior of e-journal users in science and technology: Transaction log analysis of elsevier\u2019s sciencedirect OnSite in Taiwan. Library Info . Sci. Res. 24 , 3 (2002), 265 -- 291 . Hao-Ren Ke, Rolf Kwakkelaar, Yu-Min Tai, and Li-Chun Chen. 2002. Exploring behavior of e-journal users in science and technology: Transaction log analysis of elsevier\u2019s sciencedirect OnSite in Taiwan. Library Info. Sci. Res. 24, 3 (2002), 265--291.","journal-title":"Sci. Res."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2910896.2910922"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00799-016-0170-x"},{"key":"e_1_2_1_32_1","volume-title":"Kingma and Jimmy Ba","author":"Diederik","year":"2014","unstructured":"Diederik P. Kingma and Jimmy Ba . 2014 . Adam : A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)."},{"key":"e_1_2_1_33_1","volume-title":"Recommendation on academic networks using direction aware citation analysis. arXiv preprint arXiv:1205.1143","author":"K\u00fc\u00e7\u00fcktun\u00e7 Onur","year":"2012","unstructured":"Onur K\u00fc\u00e7\u00fcktun\u00e7 , Erik Saule , Kamer Kaya , and \u00dcmit V. \u00c7ataly\u00fcrek . 2012. Recommendation on academic networks using direction aware citation analysis. arXiv preprint arXiv:1205.1143 ( 2012 ). Onur K\u00fc\u00e7\u00fcktun\u00e7, Erik Saule, Kamer Kaya, and \u00dcmit V. \u00c7ataly\u00fcrek. 2012. Recommendation on academic networks using direction aware citation analysis. arXiv preprint arXiv:1205.1143 (2012)."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661829.2661947"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1135777.1135926"},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the 5th International Workshop on Bibliometric-enhanced Information Retrieval.","author":"Li Xinyi","year":"2017","unstructured":"Xinyi Li and Maarten de Rijke . 2017 . Academic search in response to major scientific events . In Proceedings of the 5th International Workshop on Bibliometric-enhanced Information Retrieval. Xinyi Li and Maarten de Rijke. 2017. Academic search in response to major scientific events. In Proceedings of the 5th International Workshop on Bibliometric-enhanced Information Retrieval."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-56608-5_12"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2018.10.019"},{"key":"e_1_2_1_39_1","first-page":"3","article-title":"Investigating queries and search failures in academic search. Info","volume":"53","author":"Li Xinyi","year":"2017","unstructured":"Xinyi Li , Bob J. A. Schijvenaars , and Maarten de Rijke . 2017 . Investigating queries and search failures in academic search. Info . Process. Manage. 53 , 3 (May 2017), 666--683. Xinyi Li, Bob J. A. Schijvenaars, and Maarten de Rijke. 2017. Investigating queries and search failures in academic search. Info. Process. Manage. 53, 3 (May 2017), 666--683.","journal-title":"Process. Manage."},{"key":"e_1_2_1_40_1","volume-title":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence","volume":"15","author":"Lin Yankai","year":"2015","unstructured":"Yankai Lin , Zhiyuan Liu , Maosong Sun , Yang Liu , and Xuan Zhu . 2015 . Learning entity and relation embeddings for knowledge graph completion . In Proceedings of the 32nd AAAI Conference on Artificial Intelligence , Vol. 15 . 2181--2187. Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, Vol. 15. 2181--2187."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2003.1167344"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2015.2481320"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935877"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3041021.3054265"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2382438.2382439"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1998076.1998132"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2011.134"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.21669"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2016.0051"},{"key":"e_1_2_1_50_1","volume-title":"Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence","author":"Pennock David M.","unstructured":"David M. Pennock , Eric Horvitz , Steve Lawrence , and C. Lee Giles . 2000. Collaborative filtering by personality diagnosis: A hybrid memory-and model-based approach . In Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence . Morgan Kaufmann Publishers Inc ., 473--480. David M. Pennock, Eric Horvitz, Steve Lawrence, and C. Lee Giles. 2000. Collaborative filtering by personality diagnosis: A hybrid memory-and model-based approach. In Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence. Morgan Kaufmann Publishers Inc., 473--480."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23277"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23623"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2168752.2168771"},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence. AUAI Press, 452--461","author":"Rendle Steffen","year":"2009","unstructured":"Steffen Rendle , Christoph Freudenthaler , Zeno Gantner , and Lars Schmidt-Thieme . 2009 . BPR: Bayesian personalized ranking from implicit feedback . In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence. AUAI Press, 452--461 . Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence. AUAI Press, 452--461."},{"key":"e_1_2_1_55_1","doi-asserted-by":"crossref","unstructured":"Francesco Ricci Lior Rokach and Bracha Shapira (Eds.). 2015. Recommender Systems Handbook. Springer.   Francesco Ricci Lior Rokach and Bracha Shapira (Eds.). 2015. Recommender Systems Handbook. Springer.","DOI":"10.1007\/978-1-4899-7637-6"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/352871.352887"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/371920.372071"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645751"},{"key":"e_1_2_1_59_1","unstructured":"ScienceDirect. 2015. Retrieved from https:\/\/sciencedirect.com.  ScienceDirect. 2015. Retrieved from https:\/\/sciencedirect.com."},{"key":"e_1_2_1_60_1","unstructured":"ScienceDirect. 2016. Retrieved from https:\/\/www.elsevier.com\/solutions\/sciencedirect\/features.  ScienceDirect. 2016. Retrieved from https:\/\/www.elsevier.com\/solutions\/sciencedirect\/features."},{"key":"e_1_2_1_61_1","unstructured":"Semantic Scholar. 2018. Retrieved from https:\/\/www.semanticscholar.org\/.  Semantic Scholar. 2018. Retrieved from https:\/\/www.semanticscholar.org\/."},{"key":"e_1_2_1_62_1","volume-title":"Can I have more of these please? assisting researchers in finding similar research papers from a seed basket of papers","author":"Raamkumar Aravind Sesagiri","unstructured":"Aravind Sesagiri Raamkumar , Schubert Foo , and Natalie Pang . 2018. Can I have more of these please? assisting researchers in finding similar research papers from a seed basket of papers . Emerald Publishing Limited . Aravind Sesagiri Raamkumar, Schubert Foo, and Natalie Pang. 2018. Can I have more of these please? assisting researchers in finding similar research papers from a seed basket of papers. Emerald Publishing Limited."},{"key":"e_1_2_1_63_1","volume-title":"Proceedings of the WSDM 2014 Workshop on Web Search Click Data. ACM.","author":"Song Guocong","year":"2014","unstructured":"Guocong Song . 2014 . Point-wise approach for yandex personalized web search challenge . In Proceedings of the WSDM 2014 Workshop on Web Search Click Data. ACM. Guocong Song. 2014. Point-wise approach for yandex personalized web search challenge. In Proceedings of the WSDM 2014 Workshop on Web Search Click Data. ACM."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277868"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/1816123.1816129"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/2835776.2835849"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2008.71"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/996350.996402"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/2020408.2020480"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/2835776.2835837"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.12785\/amis\/080463"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052558"},{"key":"e_1_2_1_73_1","volume-title":"Proceedings of the 25th International Joint Conference on Artificial Intelligence. IJCAI\/AAAI Press, 2378--2384","author":"Zhao Feipeng","year":"2016","unstructured":"Feipeng Zhao and Yuhong Guo . 2016 . Improving top-N recommendation with heterogeneous loss . In Proceedings of the 25th International Joint Conference on Artificial Intelligence. IJCAI\/AAAI Press, 2378--2384 . Feipeng Zhao and Yuhong Guo. 2016. Improving top-N recommendation with heterogeneous loss. In Proceedings of the 25th International Joint Conference on Artificial Intelligence. IJCAI\/AAAI Press, 2378--2384."},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911500"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3312528","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3312528","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:54:28Z","timestamp":1750204468000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3312528"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,3,16]]},"references-count":74,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2019,7,31]]}},"alternative-id":["10.1145\/3312528"],"URL":"https:\/\/doi.org\/10.1145\/3312528","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,3,16]]},"assertion":[{"value":"2018-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-02-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-03-16","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}