{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,11]],"date-time":"2026-02-11T20:53:04Z","timestamp":1770843184297,"version":"3.50.1"},"reference-count":44,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2013,12,1]],"date-time":"2013-12-01T00:00:00Z","timestamp":1385856000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2013,12]]},"abstract":"<jats:p>The accuracy of collaborative-filtering recommender systems largely depends on three factors: the quality of the rating prediction algorithm, and the quantity and quality of available ratings. While research in the field of recommender systems often concentrates on improving prediction algorithms, even the best algorithms will fail if they are fed poor-quality data during training, that is, garbage in, garbage out. Active learning aims to remedy this problem by focusing on obtaining better-quality data that more aptly reflects a user's preferences. However, traditional evaluation of active learning strategies has two major flaws, which have significant negative ramifications on accurately evaluating the system's performance (prediction error, precision, and quantity of elicited ratings). (1) Performance has been evaluated for each user independently (ignoring system-wide improvements). (2) Active learning strategies have been evaluated in isolation from unsolicited user ratings (natural acquisition).<\/jats:p>\n          <jats:p>In this article we show that an elicited rating has effects across the system, so a typical user-centric evaluation which ignores any changes of rating prediction of other users also ignores these cumulative effects, which may be more influential on the performance of the system as a whole (system centric). We propose a new evaluation methodology and use it to evaluate some novel and state-of-the-art rating elicitation strategies. We found that the system-wide effectiveness of a rating elicitation strategy depends on the stage of the rating elicitation process, and on the evaluation measures (MAE, NDCG, and Precision). In particular, we show that using some common user-centric strategies may actually degrade the overall performance of a system. Finally, we show that the performance of many common active learning strategies changes significantly when evaluated concurrently with the natural acquisition of ratings in recommender systems.<\/jats:p>","DOI":"10.1145\/2542182.2542195","type":"journal-article","created":{"date-parts":[[2014,1,2]],"date-time":"2014-01-02T13:09:43Z","timestamp":1388668183000},"page":"1-33","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":32,"title":["Active learning strategies for rating elicitation in collaborative filtering"],"prefix":"10.1145","volume":"5","author":[{"given":"Mehdi","family":"Elahi","sequence":"first","affiliation":[{"name":"Free University of Bozen-Bolzano, Bolzano, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Francesco","family":"Ricci","sequence":"additional","affiliation":[{"name":"Free University of Bozen-Bolzano, Bolzano, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Neil","family":"Rubens","sequence":"additional","affiliation":[{"name":"University of Electro-Communications, Tokyo, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2014,1,3]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"The Long Tail","author":"Anderson C.","unstructured":"Anderson , C. 2006. The Long Tail . Random House Business . Anderson, C. 2006. The Long Tail. Random House Business."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-012-5297-4"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the 24th Annual Conference on Neural Information Processing Systems (NIPS'10)","author":"Bonilla E. V.","unstructured":"Bonilla , E. V. , Guo , S. , and Sanner , S . 2010. Gaussian process preference elicitation . In Proceedings of the 24th Annual Conference on Neural Information Processing Systems (NIPS'10) . 262--270. Bonilla, E. V., Guo, S., and Sanner, S. 2010. Gaussian process preference elicitation. In Proceedings of the 24th Annual Conference on Neural Information Processing Systems (NIPS'10). 262--270."},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 19th Conference in Uncertainty in Artificial Intelligence (UAI'03)","author":"Boutilier C.","unstructured":"Boutilier , C. , Zemel , R. S. , and Marlin , B. M . 2003. Active collaborative filtering . In Proceedings of the 19th Conference in Uncertainty in Artificial Intelligence (UAI'03) . 98--106. Boutilier, C., Zemel, R. S., and Marlin, B. M. 2003. Active collaborative filtering. In Proceedings of the 19th Conference in Uncertainty in Artificial Intelligence (UAI'03). 98--106."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807342.1807377"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864753"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/604045.604052"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-011-9108-6"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864721"},{"key":"e_1_2_1_10_1","doi-asserted-by":"crossref","unstructured":"Desrosiers C. and Karypis G. 2011. A comprehensive survey of neighborhood-based recommendation methods. In Recommender Systems Handbook F. Ricci L. Rokach B. Shapira and P. B. Kantor Eds. Springer 107--144.  Desrosiers C. and Karypis G. 2011. A comprehensive survey of neighborhood-based recommendation methods. In Recommender Systems Handbook F. Ricci L. Rokach B. Shapira and P. B. Kantor Eds. Springer 107--144.","DOI":"10.1007\/978-0-387-85820-3_4"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.5555\/2021855.2021899"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings 12th International Conference on E-Commerce and Web Technologies (EC-Web'11)","author":"Elahi M.","unstructured":"Elahi , M. , Repsys , V. , and Ricci , F . 2011a. Rating elicitation strategies for collaborative filtering . In Proceedings 12th International Conference on E-Commerce and Web Technologies (EC-Web'11) . 160--171. Elahi, M., Repsys, V., and Ricci, F. 2011a. Rating elicitation strategies for collaborative filtering. In Proceedings 12th International Conference on E-Commerce and Web Technologies (EC-Web'11). 160--171."},{"key":"e_1_2_1_13_1","unstructured":"Elahi M. Ricci F. and Repsys V. 2011b. System-wide effectiveness of active learning in collaborative filtering. In Proceedings of the International Workshop on Social Web Mining co-located with International Joint Conference on Artificial Intelligence. F. Bonchi W. Buntine R. Gavald and S. Gu Eds. http:\/\/users.cecs.anu.edu.au\/&sim;sguo\/swm2011_submission_4.pdf.  Elahi M. Ricci F. and Repsys V. 2011b. System-wide effectiveness of active learning in collaborative filtering. In Proceedings of the International Workshop on Social Web Mining co-located with International Joint Conference on Artificial Intelligence. F. Bonchi W. Buntine R. Gavald and S. Gu Eds. http:\/\/users.cecs.anu.edu.au\/&sim;sguo\/swm2011_submission_4.pdf."},{"key":"e_1_2_1_14_1","unstructured":"Funk S. 2006. Netflix update: Try this at home. http:\/\/sifter.org\/&sim;simon\/journal\/20061211.html.  Funk S. 2006. Netflix update: Try this at home. http:\/\/sifter.org\/&sim;simon\/journal\/20061211.html."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1871437.1871734"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935910"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-13278-0_51"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390352"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/312624.312682"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/963770.963772"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511763113"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of the 20th Conference in Uncertainty in Artificial Intelligence (UAI'04)","author":"Jin R.","unstructured":"Jin , R. and Si , L . 2004. A bayesian approach toward active learning for collaborative filtering . In Proceedings of the 20th Conference in Uncertainty in Artificial Intelligence (UAI'04) . 278--285. Jin, R. and Si, L. 2004. A bayesian approach toward active learning for collaborative filtering. In Proceedings of the 20th Conference in Uncertainty in Artificial Intelligence (UAI'04). 278--285."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401944"},{"key":"e_1_2_1_25_1","doi-asserted-by":"crossref","unstructured":"Koren Y. and Bell R. 2011. Advances in collaborative filtering. In Recommender Systems Handbook F. Ricci L. Rokach B. Shapira and P. Kantor Eds. Springer 145--186.  Koren Y. and Bell R. 2011. Advances in collaborative filtering. In Recommender Systems Handbook F. Ricci L. Rokach B. Shapira and P. Kantor Eds. Springer 145--186.","DOI":"10.1007\/978-0-387-85820-3_5"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043943"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390351"},{"key":"e_1_2_1_28_1","volume-title":"Introduction to Information Retrieval","author":"Manning C.","unstructured":"Manning , C. 2008. Introduction to Information Retrieval . Cambridge University Press . Manning, C. 2008. Introduction to Information Retrieval. Cambridge University Press."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.5555\/2283696.2283843"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the International Conference on User Modeling. 178--187","author":"Mcnee S. M.","unstructured":"Mcnee , S. M. , Lam , S. K. , Konstan , J. A. , and Riedl , J . 2003. Interfaces for eliciting new user preferences in recommender systems . In Proceedings of the International Conference on User Modeling. 178--187 . Mcnee, S. M., Lam, S. K., Konstan, J. A., and Riedl, J. 2003. Interfaces for eliciting new user preferences in recommender systems. In Proceedings of the International Conference on User Modeling. 178--187."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/604045.604094"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v29i4.2200"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/502716.502737"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1540276.1540302"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1297231.1297236"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/245108.245121"},{"key":"e_1_2_1_37_1","doi-asserted-by":"crossref","unstructured":"Ricci F. Rokach L. and Shapira B. 2011a. Introduction to recommender systems handbook. In Recommender Systems Handbook F. Ricci L. Rokach B. Shapira and P. Kantor Eds. Springer 1--35.  Ricci F. Rokach L. and Shapira B. 2011a. Introduction to recommender systems handbook. In Recommender Systems Handbook F. Ricci L. Rokach B. Shapira and P. Kantor Eds. Springer 1--35.","DOI":"10.1007\/978-0-387-85820-3_1"},{"key":"e_1_2_1_38_1","volume-title":"Eds","author":"Ricci F.","year":"2011","unstructured":"Ricci , F. , Rokach , L. , Shapira , B. , and Kantor , P. B. , Eds . 2011 b. Recommender Systems Handbook. Springer . Ricci, F., Rokach, L., Shapira, B., and Kantor, P. B., Eds. 2011b. Recommender Systems Handbook. Springer."},{"key":"e_1_2_1_39_1","doi-asserted-by":"crossref","unstructured":"Rubens N. Kaplan D. and Sugiyama M. 2011. Active learning in recommender systems. In Recommender Systems Handbook F. Ricci L. Rokach B. Shapira and P. Kantor Eds. Springer 735--767.  Rubens N. Kaplan D. and Sugiyama M. 2011. Active learning in recommender systems. In Recommender Systems Handbook F. Ricci L. Rokach B. Shapira and P. Kantor Eds. Springer 735--767.","DOI":"10.1007\/978-0-387-85820-3_23"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/564376.564421"},{"key":"e_1_2_1_41_1","doi-asserted-by":"crossref","unstructured":"Shani G. and Gunawardana A. 2010. Evaluating recommendation systems. In Recommender Systems Handbook F. Ricci L. Rokach and B. Shapira Eds. Springer 257--298.  Shani G. and Gunawardana A. 2010. Evaluating recommendation systems. In Recommender Systems Handbook F. Ricci L. Rokach and B. Shapira Eds. Springer 257--298.","DOI":"10.1007\/978-0-387-85820-3_8"},{"key":"e_1_2_1_42_1","unstructured":"Timely Development L. 2008. Netflix prize. http:\/\/www.timelydevelopment.com\/demos\/NetflixPrize.aspx.  Timely Development L. 2008. Netflix prize. http:\/\/www.timelydevelopment.com\/demos\/NetflixPrize.aspx."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1454008.1454050"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2009961"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2542182.2542195","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2542182.2542195","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T08:09:56Z","timestamp":1750234196000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2542182.2542195"}},"subtitle":["A system-wide perspective"],"short-title":[],"issued":{"date-parts":[[2013,12]]},"references-count":44,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2013,12]]}},"alternative-id":["10.1145\/2542182.2542195"],"URL":"https:\/\/doi.org\/10.1145\/2542182.2542195","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,12]]},"assertion":[{"value":"2012-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2012-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2014-01-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}