{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:16:15Z","timestamp":1750306575644,"version":"3.41.0"},"reference-count":49,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2016,2,17]],"date-time":"2016-02-17T00:00:00Z","timestamp":1455667200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"DFG in the project \u201cInfo 2.0 -- Informationelle Selbstbestimmung im Web 2.0\u201d"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2016,4]]},"abstract":"<jats:p>\n            Social bookmarking systems have established themselves as an important part in today\u2019s Web. In such systems,\n            <jats:italic>tag recommender systems<\/jats:italic>\n            support users during the posting of a resource by suggesting suitable tags. Tag recommender algorithms have often been evaluated in offline benchmarking experiments. Yet, the particular setup of such experiments has rarely been analyzed. In particular, since the recommendation quality usually suffers from difficulties such as the sparsity of the data or the cold-start problem for new resources or users, datasets have often been pruned to so-called\n            <jats:italic>cores<\/jats:italic>\n            (specific subsets of the original datasets), without much consideration of the implications on the benchmarking results.\n          <\/jats:p>\n          <jats:p>\n            In this article, we generalize the notion of a core by introducing the new notion of a\n            <jats:italic>set-core<\/jats:italic>\n            , which is independent of any graph structure, to overcome a structural drawback in the previous constructions of cores on tagging data. We show that problems caused by some types of cores can be eliminated using set-cores. Further, we present a thorough analysis of tag recommender benchmarking setups using cores. To that end, we conduct a large-scale experiment on four real-world datasets, in which we analyze the influence of different cores on the evaluation of recommendation algorithms. We can show that the results of the comparison of different recommendation approaches depends on the selection of core type and level. For the benchmarking of tag recommender algorithms, our results suggest that the evaluation must be set up more carefully and should not be based on one arbitrarily chosen core type and level.\n          <\/jats:p>","DOI":"10.1145\/2700485","type":"journal-article","created":{"date-parts":[[2016,2,22]],"date-time":"2016-02-22T13:07:16Z","timestamp":1456146436000},"page":"1-33","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["The Role of Cores in Recommender Benchmarking for Social Bookmarking Systems"],"prefix":"10.1145","volume":"7","author":[{"given":"Stephan","family":"Doerfel","sequence":"first","affiliation":[{"name":"KDE Group, ITeG, University of Kassel, Kassel, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Robert","family":"J\u00e4schke","sequence":"additional","affiliation":[{"name":"L3S Research Center, Hannover, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gerd","family":"Stumme","sequence":"additional","affiliation":[{"name":"KDE Group, ITeG, University of Kassel, Germany, and L3S Research Center, Hannover, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2016,2,17]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2151163.2151166"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/APVIS.2007.329304"},{"key":"e_1_2_1_3_1","volume-title":"K-core decomposition of Internet graphs: Hierarchies, self-similarity and measurement biases. CoRR cs\/0511007","author":"Alvarez-Hamelin Jos\u00e9 Ignacio","year":"2005","unstructured":"Jos\u00e9 Ignacio Alvarez-Hamelin , Luca Dall\u2019Asta , Alain Barrat , and Alessandro Vespignani . 2005. K-core decomposition of Internet graphs: Hierarchies, self-similarity and measurement biases. CoRR cs\/0511007 ( 2005 ). http:\/\/arxiv.org\/abs\/cs\/0511007. Jos\u00e9 Ignacio Alvarez-Hamelin, Luca Dall\u2019Asta, Alain Barrat, and Alessandro Vespignani. 2005. K-core decomposition of Internet graphs: Hierarchies, self-similarity and measurement biases. CoRR cs\/0511007 (2005). http:\/\/arxiv.org\/abs\/cs\/0511007."},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the Mining Social Data Workshop. ECAI, 21--25","author":"Angelova Ralitsa","year":"2008","unstructured":"Ralitsa Angelova , Marek Lipczak , Evangelos Milios , and Pawe\u0142 Pra\u0142at; at. 2008 . Characterizing a social bookmarking and tagging network . In Proceedings of the Mining Social Data Workshop. ECAI, 21--25 . Ralitsa Angelova, Marek Lipczak, Evangelos Milios, and Pawe\u0142 Pra\u0142at; at. 2008. Characterizing a social bookmarking and tagging network. In Proceedings of the Mining Social Data Workshop. ECAI, 21--25."},{"key":"e_1_2_1_5_1","unstructured":"Pierpaolo Basile Domenico Gendarmi Filippo Lanubile and Giovanni Semeraro. 2007. Recommending smart tags in a social bookmarking system. In Bridging the Gap Between Semantic Web and Web 2.0 (SemNet\u201907). 22--29.  Pierpaolo Basile Domenico Gendarmi Filippo Lanubile and Giovanni Semeraro. 2007. Recommending smart tags in a social bookmarking system. In Bridging the Gap Between Semantic Web and Web 2.0 (SemNet\u201907). 22--29."},{"key":"e_1_2_1_6_1","volume-title":"Zaver\u0161nik","author":"Batagelj Vladimir","year":"2002","unstructured":"Vladimir Batagelj and Matja\u017e M . Zaver\u0161nik . 2002 . Generalized cores. CoRR cs.DS\/0202039 (2002). http:\/\/arxiv.org\/abs\/cs\/0202039 Vladimir Batagelj and Matja\u017e M. Zaver\u0161nik. 2002. Generalized cores. CoRR cs.DS\/0202039 (2002). http:\/\/arxiv.org\/abs\/cs\/0202039"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11634-010-0079-y"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the European Conference of Complex Systems.","author":"Baur Michael","year":"2007","unstructured":"Michael Baur , Marco Gaertler , Robert G\u00f6rke , Marcus Krug , and Dorothea Wagner . 2007 . Generating graphs with predefined K-core structure . In Proceedings of the European Conference of Complex Systems. Michael Baur, Marco Gaertler, Robert G\u00f6rke, Marcus Krug, and Dorothea Wagner. 2007. Generating graphs with predefined K-core structure. In Proceedings of the European Conference of Complex Systems."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-7552(98)00110-X"},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the International Conference on Weblogs and Social Media.","author":"Byde Andrew","year":"2007","unstructured":"Andrew Byde , Hui Wan , and Steve Cayzer . 2007 . Personalized tag recommendations via tagging and content-based similarity metrics . In Proceedings of the International Conference on Weblogs and Social Media. Andrew Byde, Hui Wan, and Steve Cayzer. 2007. Personalized tag recommendations via tagging and content-based similarity metrics. In Proceedings of the International Conference on Weblogs and Social Media."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864721"},{"volume-title":"Graph Theory (3rd (electronic edition) ed.)","author":"Diestel Reinhard","key":"e_1_2_1_12_1","unstructured":"Reinhard Diestel . 2005. Graph Theory (3rd (electronic edition) ed.) . Springer-Verlag , Berlin , I--XVI, 1--344 pages. Reinhard Diestel. 2005. Graph Theory (3rd (electronic edition) ed.). Springer-Verlag, Berlin, I--XVI, 1--344 pages."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2507157.2507222"},{"key":"e_1_2_1_14_1","unstructured":"Folke Eisterlehner Andreas Hotho and Robert J\u00e4schke (Eds.). 2009. ECML PKDD Discovery Challenge 2009 (DC\u201909). CEUR-WS.org Vol. 497.  Folke Eisterlehner Andreas Hotho and Robert J\u00e4schke (Eds.). 2009. ECML PKDD Discovery Challenge 2009 (DC\u201909). CEUR-WS.org Vol. 497."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASONAM.2011.65"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/963770.963772"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/11762256_31"},{"key":"e_1_2_1_18_1","unstructured":"Andreas Hotho Beate Krause Dominik Benz and Robert J\u00e4schke (Eds.). 2008. ECML PKDD Discovery Challenge 2008 (RSDC\u201908).  Andreas Hotho Beate Krause Dominik Benz and Robert J\u00e4schke (Eds.). 2008. ECML PKDD Discovery Challenge 2008 (RSDC\u201908)."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1002\/9781118032718"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/3120747.3120802"},{"volume-title":"Recommender Systems for the Social Web","author":"J\u00e4schke Robert","key":"e_1_2_1_21_1","unstructured":"Robert J\u00e4schke , Andreas Hotho , Folke Mitzlaff , and Gerd Stumme . 2012. Challenges in tag recommendations for collaborative tagging systems . In Recommender Systems for the Social Web . Intelligent Systems Reference Library, Vol . 32. Springer , Berlin, 65--87. DOI:http:\/\/dx.doi.org\/10.1007\/978-3-642-25694-3_3 10.1007\/978-3-642-25694-3_3 Robert J\u00e4schke, Andreas Hotho, Folke Mitzlaff, and Gerd Stumme. 2012. Challenges in tag recommendations for collaborative tagging systems. In Recommender Systems for the Social Web. Intelligent Systems Reference Library, Vol. 32. Springer, Berlin, 65--87. DOI:http:\/\/dx.doi.org\/10.1007\/978-3-642-25694-3_3"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/1487691.1487696"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/502585.502627"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/324133.324140"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1639714.1639726"},{"key":"e_1_2_1_26_1","series-title":"Lecture Notes in Business Information Processing","volume-title":"E-Commerce and Web Technologies, Christian Huemer and Thomas Setzer (Eds.)","author":"Kubatz Marius","unstructured":"Marius Kubatz , Fatih Gedikli , and Dietmar Jannach . 2011. LocalRank\u2014neighborhood-based , fast computation of tag recommendations . In E-Commerce and Web Technologies, Christian Huemer and Thomas Setzer (Eds.) . Lecture Notes in Business Information Processing , Vol. 85 . Springer , Berlin , 258--269. DOI:http:\/\/dx.doi.org\/10.1007\/978-3-642-23014-1_22 10.1007\/978-3-642-23014-1_22 Marius Kubatz, Fatih Gedikli, and Dietmar Jannach. 2011. LocalRank\u2014neighborhood-based, fast computation of tag recommendations. In E-Commerce and Web Technologies, Christian Huemer and Thomas Setzer (Eds.). Lecture Notes in Business Information Processing, Vol. 85. Springer, Berlin, 258--269. DOI:http:\/\/dx.doi.org\/10.1007\/978-3-642-23014-1_22"},{"key":"e_1_2_1_27_1","unstructured":"Marek Lipczak Yeming Hu Yael Kollet and Evangelos Milios. 2009. Tag sources for recommendation in collaborative tagging systems In Folke Eisterlehner Andreas Hotho and Robert J\u00e4schke (Eds.). ECMB PKDD Discovery Challenge 2009 (DC\u201909) 157--172.  Marek Lipczak Yeming Hu Yael Kollet and Evangelos Milios. 2009. Tag sources for recommendation in collaborative tagging systems In Folke Eisterlehner Andreas Hotho and Robert J\u00e4schke (Eds.). ECMB PKDD Discovery Challenge 2009 (DC\u201909) 157--172."},{"volume-title":"Social Media Retrieval and Mining, Shuigeng Zhou and Zhiang Wu (Eds.). Communications in Computer and Information Science","author":"Ma Feichao","key":"e_1_2_1_28_1","unstructured":"Feichao Ma , Wenqing Wang , and Zhihong Deng . 2013. TagRank: A new tag recommendation algorithm and recommender enhancement with data fusion techniques . In Social Media Retrieval and Mining, Shuigeng Zhou and Zhiang Wu (Eds.). Communications in Computer and Information Science , Vol. 387 . Springer , Berlin , 80--91. DOI:http:\/\/dx.doi.org\/10.1007\/978-3-642-41629-3_7 10.1007\/978-3-642-41629-3_7 Feichao Ma, Wenqing Wang, and Zhihong Deng. 2013. TagRank: A new tag recommendation algorithm and recommender enhancement with data fusion techniques. In Social Media Retrieval and Mining, Shuigeng Zhou and Zhiang Wu (Eds.). Communications in Computer and Information Science, Vol. 387. Springer, Berlin, 80--91. DOI:http:\/\/dx.doi.org\/10.1007\/978-3-642-41629-3_7"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458205"},{"volume-title":"Introduction to Information Retrieval","author":"Manning Christopher D.","key":"e_1_2_1_30_1","unstructured":"Christopher D. Manning , Prabhakar Raghavan , and Hinrich Sch\u00fctze . 2008. Introduction to Information Retrieval . Cambridge University Press, New York , NY. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Sch\u00fctze. 2008. Introduction to Information Retrieval. Cambridge University Press, New York, NY."},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 18th National Conference on Artificial Intelligence. AAAI, 187--192","author":"Melville Prem","year":"2002","unstructured":"Prem Melville , Raymond J. Mooney , and Ramadass Nagarajan . 2002 . Content-boosted collaborative filtering for improved recommendations . In Proceedings of the 18th National Conference on Artificial Intelligence. AAAI, 187--192 . Prem Melville, Raymond J. Mooney, and Ramadass Nagarajan. 2002. Content-boosted collaborative filtering for improved recommendations. In Proceedings of the 18th National Conference on Artificial Intelligence. AAAI, 187--192."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1135777.1135961"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1093\/jigpal\/jzq036"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the 11th International Conference Wirtschaftsinformatik. 87--101","author":"Nanopoulos Alexandros","year":"2013","unstructured":"Alexandros Nanopoulos , Dimitrios Rafailidis , and Ioannis Karydis . 2013 . Matrix factorization with content relationships for media personalization . In Proceedings of the 11th International Conference Wirtschaftsinformatik. 87--101 . Alexandros Nanopoulos, Dimitrios Rafailidis, and Ioannis Karydis. 2013. Matrix factorization with content relationships for media personalization. In Proceedings of the 11th International Conference Wirtschaftsinformatik. 87--101."},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the 17th Conference on Uncertainty in Artificial Intelligence (UAI\u201901)","author":"Popescul Alexandrin","year":"2001","unstructured":"Alexandrin Popescul , Lyle H. Ungar , David M. Pennock , and Steve Lawrence . 2001 . Probabilistic models for unified collaborative and content-based recommendation in sparse-data environments . In Proceedings of the 17th Conference on Uncertainty in Artificial Intelligence (UAI\u201901) . Morgan Kaufmann, San Francisco, CA, 437--444. Alexandrin Popescul, Lyle H. Ungar, David M. Pennock, and Steve Lawrence. 2001. Probabilistic models for unified collaborative and content-based recommendation in sparse-data environments. In Proceedings of the 17th Conference on Uncertainty in Artificial Intelligence (UAI\u201901). Morgan Kaufmann, San Francisco, CA, 437--444."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043962"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.4304\/jetwi.3.2.168-176"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI\u201909)","author":"Rendle Steffen","year":"2009","unstructured":"Steffen Rendle , Christoph Freudenthaler , Zeno Gantner , and Schmidt-Thieme Lars . 2009 . BPR: Bayesian personalized ranking from implicit feedback . In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI\u201909) . AUAI Press, Arlington, Virginia, 452--461. Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Schmidt-Thieme Lars. 2009. BPR: Bayesian personalized ranking from implicit feedback. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI\u201909). AUAI Press, Arlington, Virginia, 452--461."},{"key":"e_1_2_1_39_1","volume-title":"Factor models for tag recommendation in BibSonomy","author":"Rendle Steffen","year":"2009","unstructured":"Steffen Rendle and Lars Schmidt-Thieme . 2009. Factor models for tag recommendation in BibSonomy . In Folke Eisterlehner, Andreas Hotho, and Robert J\u00e4schke (Eds.). ECMB PKDD Discovery Challenge 2009 (DC\u2019 09), 235--242. Steffen Rendle and Lars Schmidt-Thieme. 2009. Factor models for tag recommendation in BibSonomy. In Folke Eisterlehner, Andreas Hotho, and Robert J\u00e4schke (Eds.). ECMB PKDD Discovery Challenge 2009 (DC\u201909), 235--242."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/371920.372071"},{"volume-title":"ACM WEBKDD Workshop.","author":"Sarwar Badrul M.","key":"e_1_2_1_41_1","unstructured":"Badrul M. Sarwar , George Karypis , Joseph A. Konstan , and John T. Riedl . 2000. Application of dimensionality reduction in recommender system -- A case study . In ACM WEBKDD Workshop. Badrul M. Sarwar, George Karypis, Joseph A. Konstan, and John T. Riedl. 2000. Application of dimensionality reduction in recommender system -- A case study. In ACM WEBKDD Workshop."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/564376.564421"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1016\/0378-8733(83)90028-X"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505625"},{"volume-title":"Evaluating recommendation systems","author":"Shani Guy","key":"e_1_2_1_45_1","unstructured":"Guy Shani and Asela Gunawardana . 2011. Evaluating recommendation systems . In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, Bracha Shapira, and Paul B. Kantor (Eds.). Springer , New York, NY , 257--297. DOI:http:\/\/dx.doi.org\/10.1007\/978-0-387-85820-3_8 10.1007\/978-0-387-85820-3_8 Guy Shani and Asela Gunawardana. 2011. Evaluating recommendation systems. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, Bracha Shapira, and Paul B. Kantor (Eds.). Springer, New York, NY, 257--297. DOI:http:\/\/dx.doi.org\/10.1007\/978-0-387-85820-3_8"},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of the International Conference on Weblogs and Social Media (ICWSM\u201907)","author":"Sood Sanjay","year":"2007","unstructured":"Sanjay Sood , Sara Owsley , Kristian Hammond , and Larry Birnbaum . 2007 . TagAssist: Automatic tag suggestion for blog posts . In Proceedings of the International Conference on Weblogs and Social Media (ICWSM\u201907) . Sanjay Sood, Sara Owsley, Kristian Hammond, and Larry Birnbaum. 2007. TagAssist: Automatic tag suggestion for blog posts. In Proceedings of the International Conference on Weblogs and Social Media (ICWSM\u201907)."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2007.01.045"},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the Collaborative Web Tagging Workshop at the WWW","author":"Xu Zhichen","year":"2006","unstructured":"Zhichen Xu , Yun Fu , Jianchang Mao , and Difu Su . 2006 . Towards the semantic web: Collaborative tag suggestions . In Proceedings of the Collaborative Web Tagging Workshop at the WWW 2006. Edinburgh, Scotland. Zhichen Xu, Yun Fu, Jianchang Mao, and Difu Su. 2006. Towards the semantic web: Collaborative tag suggestions. In Proceedings of the Collaborative Web Tagging Workshop at the WWW 2006. Edinburgh, Scotland."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11227-009-0299-0"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2700485","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2700485","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T06:13:34Z","timestamp":1750227214000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2700485"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,2,17]]},"references-count":49,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2016,4]]}},"alternative-id":["10.1145\/2700485"],"URL":"https:\/\/doi.org\/10.1145\/2700485","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"type":"print","value":"2157-6904"},{"type":"electronic","value":"2157-6912"}],"subject":[],"published":{"date-parts":[[2016,2,17]]},"assertion":[{"value":"2014-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2014-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-02-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}