{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T12:01:43Z","timestamp":1772798503086,"version":"3.50.1"},"reference-count":79,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2021,5,25]],"date-time":"2021-05-25T00:00:00Z","timestamp":1621900800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Australian Research Council Discovery","award":["DP190101485"],"award-info":[{"award-number":["DP190101485"]}]},{"name":"Australian Technology Network"},{"name":"Spanish Government","award":["PID2019-108965GB-I00"],"award-info":[{"award-number":["PID2019-108965GB-I00"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2021,7,26]]},"abstract":"<jats:p>\n            We investigate the impact of popularity bias in false-positive metrics in the offline evaluation of recommender systems. Unlike their true-positive complements, false-positive metrics reward systems that minimize recommendations disliked by users. Our analysis is, to the best of our knowledge, the first to show that false-positive metrics tend to penalise popular items, the opposite behavior of true-positive metrics\u2014causing a disagreement trend between both types of metrics in the presence of popularity biases. We present a theoretical analysis of the metrics that identifies the reason that the metrics disagree and determines rare situations where the metrics might agree\u2014the key to the situation lies in the relationship between popularity and relevance distributions, in terms of their\n            <jats:italic>agreement<\/jats:italic>\n            and\n            <jats:italic>steepness<\/jats:italic>\n            \u2014two fundamental concepts we formalize. We then examine three well-known datasets using multiple popular true- and false-positive metrics on 16 recommendation algorithms. Specific datasets are chosen to allow us to estimate both biased and unbiased metric values. The results of the empirical study confirm and illustrate our analytical findings. With the conditions of the disagreement of the two types of metrics established, we then determine under which circumstances true-positive or false-positive metrics should be used by researchers of offline evaluation in recommender systems.\n            <jats:sup>1<\/jats:sup>\n          <\/jats:p>","DOI":"10.1145\/3452740","type":"journal-article","created":{"date-parts":[[2021,5,25]],"date-time":"2021-05-25T22:18:01Z","timestamp":1621981081000},"page":"1-43","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":17,"title":["Popularity Bias in False-positive Metrics for Recommender Systems Evaluation"],"prefix":"10.1145","volume":"39","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1723-1916","authenticated-orcid":false,"given":"Elisa","family":"Mena-Maldonado","sequence":"first","affiliation":[{"name":"RMIT University, Melbourne, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2278-0445","authenticated-orcid":false,"given":"Roc\u00edo","family":"Ca\u00f1amares","sequence":"additional","affiliation":[{"name":"Universidad Aut\u00f3noma de Madrid, Madrid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0668-6317","authenticated-orcid":false,"given":"Pablo","family":"Castells","sequence":"additional","affiliation":[{"name":"Universidad Aut\u00f3noma de Madrid, Madrid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3137-9653","authenticated-orcid":false,"given":"Yongli","family":"Ren","sequence":"additional","affiliation":[{"name":"RMIT University, Melbourne, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0487-9609","authenticated-orcid":false,"given":"Mark","family":"Sanderson","sequence":"additional","affiliation":[{"name":"RMIT University, Melbourne, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,5,25]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2005.99"},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the 15th National Conference on Artificial Intelligence (AAAI\u201998)","author":"Basu C.","unstructured":"C. Basu , H. Hirsh , and W. W. Cohen . 1998. Recommendation as classification: Using social and content-based information in recommendation . In Proceedings of the 15th National Conference on Artificial Intelligence (AAAI\u201998) . AAAI Press, Menlo Park, CA, 714\u2013720. C. Basu, H. Hirsh, and W. W. Cohen. 1998. Recommendation as classification: Using social and content-based information in recommendation. In Proceedings of the 15th National Conference on Artificial Intelligence (AAAI\u201998). AAAI Press, Menlo Park, CA, 714\u2013720."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1037\/1089-2680.5.4.323"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 5th ACM Conference on Recommender Systems (RecSys\u201911)","author":"Bellog\u00edn A.","unstructured":"A. Bellog\u00edn , P. Castells , and I. Cantador . 2011. Precision-oriented evaluation of recommender systems: An algorithmic comparison . In Proceedings of the 5th ACM Conference on Recommender Systems (RecSys\u201911) . ACM, New York, NY, 333\u2013336. A. Bellog\u00edn, P. Castells, and I. Cantador. 2011. Precision-oriented evaluation of recommender systems: An algorithmic comparison. In Proceedings of the 5th ACM Conference on Recommender Systems (RecSys\u201911). ACM, New York, NY, 333\u2013336."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-017-9312-z"},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the 17th ACM Conference on Information and Knowledge Management (CIKM\u201913)","author":"Broder A.","unstructured":"A. Broder , M. Ciaramita , M. Fontoura , E. Gabrilovich , V. Josifovski , D. Metzler , V. Murdock , and V. Plachouras . 2008. To swing or not to swing: Learning when (not) to advertise . In Proceedings of the 17th ACM Conference on Information and Knowledge Management (CIKM\u201913) . ACM, New York, NY, 1003\u20131012. A. Broder, M. Ciaramita, M. Fontoura, E. Gabrilovich, V. Josifovski, D. Metzler, V. Murdock, and V. Plachouras. 2008. To swing or not to swing: Learning when (not) to advertise. In Proceedings of the 17th ACM Conference on Information and Knowledge Management (CIKM\u201913). ACM, New York, NY, 1003\u20131012."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308560.3317304"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the World Wide Web Conference (WWW \u201919)","author":"Brost B.","unstructured":"B. Brost , R. Mehrotra , and T. Jehan . 2019. The music streaming sessions dataset . In Proceedings of the World Wide Web Conference (WWW \u201919) . ACM, New York, NY, 2594\u20132600. B. Brost, R. Mehrotra, and T. Jehan. 2019. The music streaming sessions dataset. In Proceedings of the World Wide Web Conference (WWW \u201919). ACM, New York, NY, 2594\u20132600."},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201904)","author":"Buckley C.","unstructured":"C. Buckley and E. M. Voorhees . 2004. Retrieval evaluation with incomplete information . In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201904) . ACM, New York, NY, 25\u201332. C. Buckley and E. M. Voorhees. 2004. Retrieval evaluation with incomplete information. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201904). ACM, New York, NY, 25\u201332."},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the 40th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201917)","author":"Ca\u00f1amares R.","unstructured":"R. Ca\u00f1amares and P. Castells . 2017. A probabilistic reformulation of memory-based collaborative filtering: Implications on popularity biases . In Proceedings of the 40th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201917) . ACM, New York, NY, 215\u2013224. R. Ca\u00f1amares and P. Castells. 2017. A probabilistic reformulation of memory-based collaborative filtering: Implications on popularity biases. In Proceedings of the 40th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201917). ACM, New York, NY, 215\u2013224."},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the 41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201918)","author":"Ca\u00f1amares R.","unstructured":"R. Ca\u00f1amares and P. Castells . 2018. Should I follow the crowd? A probabilistic analysis of the effectiveness of popularity in recommender systems . In Proceedings of the 41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201918) . ACM, New York, NY, 415\u2013424. R. Ca\u00f1amares and P. Castells. 2018. Should I follow the crowd? A probabilistic analysis of the effectiveness of popularity in recommender systems. In Proceedings of the 41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201918). ACM, New York, NY, 415\u2013424."},{"key":"e_1_2_1_12_1","volume-title":"14th ACM Conference on Recommender Systems (RecSys\u201920)","author":"Ca\u00f1amares R.","unstructured":"R. Ca\u00f1amares and P. Castells . 2020. On target item sampling in offline recommender system evaluation . In 14th ACM Conference on Recommender Systems (RecSys\u201920) . ACM, New York, NY, 259\u2013268. R. Ca\u00f1amares and P. Castells. 2020. On target item sampling in offline recommender system evaluation. In 14th ACM Conference on Recommender Systems (RecSys\u201920). ACM, New York, NY, 259\u2013268."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010037"},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the Workshop on Offline Evaluation for Recommender Systems (REVEAL 2018) at the 12th ACM Conference on Recommender Systems (RecSys\u201918)","author":"Castells P.","unstructured":"P. Castells and R. Ca\u00f1amares . 2018. Characterization of fair experiments for recommender system evaluation: A formal analysis . In Proceedings of the Workshop on Offline Evaluation for Recommender Systems (REVEAL 2018) at the 12th ACM Conference on Recommender Systems (RecSys\u201918) . P. Castells and R. Ca\u00f1amares. 2018. Characterization of fair experiments for recommender system evaluation: A formal analysis. In Proceedings of the Workshop on Offline Evaluation for Recommender Systems (REVEAL 2018) at the 12th ACM Conference on Recommender Systems (RecSys\u201918)."},{"key":"e_1_2_1_15_1","doi-asserted-by":"crossref","unstructured":"P. Castells N. J. Hurley and S. Vargas. 2015. Novelty and diversity in recommender systems. In Recommender Systems Handbook (2nd ed.) F. Ricci L. Rokach and B. Shapira (Eds.). Springer New York NY 881\u2013918.  P. Castells N. J. Hurley and S. Vargas. 2015. Novelty and diversity in recommender systems. In Recommender Systems Handbook (2nd ed.) F. Ricci L. Rokach and B. Shapira (Eds.). Springer New York NY 881\u2013918.","DOI":"10.1007\/978-1-4899-7637-6_26"},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM\u201909)","author":"Chapelle O.","unstructured":"O. Chapelle , D. Metlzer , Y. Zhang , and P. Grinspan . 2009. Expected reciprocal rank for graded relevance . In Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM\u201909) . ACM, New York, NY, 621\u2013630. O. Chapelle, D. Metlzer, Y. Zhang, and P. Grinspan. 2009. Expected reciprocal rank for graded relevance. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM\u201909). ACM, New York, NY, 621\u2013630."},{"key":"e_1_2_1_17_1","doi-asserted-by":"crossref","unstructured":"P. Y. K. Chau S. Y. Ho K. K. W. Ho and Y. Yao. 2013. Examining the effects of malfunctioning personalized services on online users\u2019 distrust and behaviors. Decis. Support Syst. 56 C (Dec. 2013) 180\u2013191.  P. Y. K. Chau S. Y. Ho K. K. W. Ho and Y. Yao. 2013. Examining the effects of malfunctioning personalized services on online users\u2019 distrust and behaviors. Decis. Support Syst. 56 C (Dec. 2013) 180\u2013191.","DOI":"10.1016\/j.dss.2013.05.023"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 4th ACM International Conference on Web Search and Data Mining (WSDM\u201911)","author":"Clarke C. L. A.","unstructured":"C. L. A. Clarke , N. Craswell , I. Soboroff , and A. Ashkan . 2011. A comparative analysis of cascade measures for novelty and diversity . In Proceedings of the 4th ACM International Conference on Web Search and Data Mining (WSDM\u201911) . ACM, New York, NY, 75\u201384. C. L. A. Clarke, N. Craswell, I. Soboroff, and A. Ashkan. 2011. A comparative analysis of cascade measures for novelty and diversity. In Proceedings of the 4th ACM International Conference on Web Search and Data Mining (WSDM\u201911). ACM, New York, NY, 75\u201384."},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 13th International Conference on Human-Computer Interaction (Interact\u201911)","author":"Cremonesi P.","unstructured":"P. Cremonesi , F. Garzotto , S. Negro , A. V. Papadopoulos , and R. Turrin . 2011. Looking for \u201cGood\u201d recommendations: A comparative evaluation of recommender systems . In Proceedings of the 13th International Conference on Human-Computer Interaction (Interact\u201911) . Springer, New York, NY, 152\u2013168. P. Cremonesi, F. Garzotto, S. Negro, A. V. Papadopoulos, and R. Turrin. 2011. Looking for \u201cGood\u201d recommendations: A comparative evaluation of recommender systems. In Proceedings of the 13th International Conference on Human-Computer Interaction (Interact\u201911). Springer, New York, NY, 152\u2013168."},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the 14th International Conference on Human-Computer Interaction (Interact\u201913)","author":"Cremonesi P.","unstructured":"P. Cremonesi , F. Garzotto , and R. Turrin . 2013. User-centric vs. system-centric evaluation of recommender systems . In Proceedings of the 14th International Conference on Human-Computer Interaction (Interact\u201913) . Springer, New York, NY, 334\u2013351. P. Cremonesi, F. Garzotto, and R. Turrin. 2013. User-centric vs. system-centric evaluation of recommender systems. In Proceedings of the 14th International Conference on Human-Computer Interaction (Interact\u201913). Springer, New York, NY, 334\u2013351."},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the 17th International Joint Conference of Artificial Intelligence (IJCAI\u201901)","author":"Elkan C.","year":"2001","unstructured":"C. Elkan . 2001 . The foundations of cost-sensitive learning . In Proceedings of the 17th International Joint Conference of Artificial Intelligence (IJCAI\u201901) . Morgan Kaufmann, Burlington, MA, 973\u2013978. C. Elkan. 2001. The foundations of cost-sensitive learning. In Proceedings of the 17th International Joint Conference of Artificial Intelligence (IJCAI\u201901). Morgan Kaufmann, Burlington, MA, 973\u2013978."},{"key":"e_1_2_1_23_1","first-page":"5","article-title":"Blockbuster culture\u2019s next rise or fall: The impact of recommender systems on sales diversity. Manage","volume":"55","author":"Fleder D.","year":"2009","unstructured":"D. Fleder and K. Hosanagar . 2009 . Blockbuster culture\u2019s next rise or fall: The impact of recommender systems on sales diversity. Manage . Sci. 55 , 5 (May 2009), 697\u2013712. D. Fleder and K. Hosanagar. 2009. Blockbuster culture\u2019s next rise or fall: The impact of recommender systems on sales diversity. Manage. Sci. 55, 5 (May 2009), 697\u2013712.","journal-title":"Sci."},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 10th ACM Conference on Recommender Systems (RecSys\u201916)","author":"Frolov E.","unstructured":"E. Frolov and I. Oseledets . 2016. Fifty shades of ratings: How to benefit from a negative feedback in top-N recommendations tasks . In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys\u201916) . ACM, New York, NY, 91\u201398. E. Frolov and I. Oseledets. 2016. Fifty shades of ratings: How to benefit from a negative feedback in top-N recommendations tasks. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys\u201916). ACM, New York, NY, 91\u201398."},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the International Conference on KDD Cup 2011 (KDDCUP\u201911)","author":"Gantner Z.","unstructured":"Z. Gantner , L. Drumond , C. Freudenthaler , and L. Schmidt-Thieme . 2011. Personalized ranking for non-uniformly sampled items . In Proceedings of the International Conference on KDD Cup 2011 (KDDCUP\u201911) . JLMR.org, 231\u2013247. Z. Gantner, L. Drumond, C. Freudenthaler, and L. Schmidt-Thieme. 2011. Personalized ranking for non-uniformly sampled items. In Proceedings of the International Conference on KDD Cup 2011 (KDDCUP\u201911). JLMR.org, 231\u2013247."},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the 41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201918)","author":"Garcia-Gathright J.","unstructured":"J. Garcia-Gathright , B. St. Thomas , C. Hosey , Z. Nazari , and F. Diaz . 2018. Understanding and evaluating user satisfaction with music discovery . In Proceedings of the 41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201918) . ACM, New York, NY, 55\u201364. J. Garcia-Gathright, B. St. Thomas, C. Hosey, Z. Nazari, and F. Diaz. 2018. Understanding and evaluating user satisfaction with music discovery. In Proceedings of the 41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201918). ACM, New York, NY, 55\u201364."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the IEEE 15th International Workshop on Multimedia Signal Processing (MMSP\u201913)","author":"Germain A.","unstructured":"A. Germain and J. Chakareski . 2013. Spotify me: Facebook-assisted automatic playlist generation . In Proceedings of the IEEE 15th International Workshop on Multimedia Signal Processing (MMSP\u201913) . IEEE Press, Los Alamitos, CA, 25\u201328. A. Germain and J. Chakareski. 2013. Spotify me: Facebook-assisted automatic playlist generation. In Proceedings of the IEEE 15th International Workshop on Multimedia Signal Processing (MMSP\u201913). IEEE Press, Los Alamitos, CA, 25\u201328."},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the 11th ACM International Conference on Web Search and Data Mining (WSDM\u201918)","author":"Gilotte A.","unstructured":"A. Gilotte , C. Calauz\u00e8nes , T. Nedelec , A. Abraham , and S. Doll\u00e9 . 2018. Offline A\/B testing for recommender systems . In Proceedings of the 11th ACM International Conference on Web Search and Data Mining (WSDM\u201918) . ACM, New York, NY, 198\u2013206. A. Gilotte, C. Calauz\u00e8nes, T. Nedelec, A. Abraham, and S. Doll\u00e9. 2018. Offline A\/B testing for recommender systems. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining (WSDM\u201918). ACM, New York, NY, 198\u2013206."},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the 22nd International Conference on World Wide Web (WWW\u201913)","author":"Goldstein D. G.","unstructured":"D. G. Goldstein , R. P. McAfee , and S. Suri . 2013. The cost of annoying ads . In Proceedings of the 22nd International Conference on World Wide Web (WWW\u201913) . ACM, New York, NY, 459\u2013470. D. G. Goldstein, R. P. McAfee, and S. Suri. 2013. The cost of annoying ads. In Proceedings of the 22nd International Conference on World Wide Web (WWW\u201913). ACM, New York, NY, 459\u2013470."},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence (UAI\u201915)","author":"Gopalan P.","unstructured":"P. Gopalan , J. M. Hofman , and D. M. Blei . 2015. Scalable recommendation with poisson factorization . In Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence (UAI\u201915) . AUAI Press, Arlington, VA, 326\u2013335. P. Gopalan, J. M. Hofman, and D. M. Blei. 2015. Scalable recommendation with poisson factorization. In Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence (UAI\u201915). AUAI Press, Arlington, VA, 326\u2013335."},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 12th ACM International Conference on Web Search and Data Mining (WSDM\u201919)","author":"Gruson A.","unstructured":"A. Gruson , P. Chandar , C. Charbuillet , J. McInerney , S. Hansen , D. Tardieu , and B. Carterette . 2019. Offline evaluation to make decisions about playlist recommendation . In Proceedings of the 12th ACM International Conference on Web Search and Data Mining (WSDM\u201919) . ACM, New York, NY, 420\u2013428. A. Gruson, P. Chandar, C. Charbuillet, J. McInerney, S. Hansen, D. Tardieu, and B. Carterette. 2019. Offline evaluation to make decisions about playlist recommendation. In Proceedings of the 12th ACM International Conference on Web Search and Data Mining (WSDM\u201919). ACM, New York, NY, 420\u2013428."},{"key":"e_1_2_1_32_1","doi-asserted-by":"crossref","unstructured":"A. Gunawardana and G. Shani. 2015. Evaluating recommender systems. In Recommender Systems Handbook (2nd ed.) F. Ricci L. Rokach and B. Shapira (Eds.). Springer New York NY 265\u2013308.  A. Gunawardana and G. Shani. 2015. Evaluating recommender systems. In Recommender Systems Handbook (2nd ed.) F. Ricci L. Rokach and B. Shapira (Eds.). Springer New York NY 265\u2013308.","DOI":"10.1007\/978-1-4899-7637-6_8"},{"key":"e_1_2_1_33_1","volume-title":"Late-breaking Results and Workshop Proceedings of the 23rd Conference on User Modelling, Adaptation and Personalization (UMAP\u201915)","author":"Guo G.","unstructured":"G. Guo , J. Zhang , Z. Sun , and N. Yorke-Smith . 2015. LibRec: A Java library for recommender systems. In Posters, Demos , Late-breaking Results and Workshop Proceedings of the 23rd Conference on User Modelling, Adaptation and Personalization (UMAP\u201915) . G. Guo, J. Zhang, Z. Sun, and N. Yorke-Smith. 2015. LibRec: A Java library for recommender systems. In Posters, Demos, Late-breaking Results and Workshop Proceedings of the 23rd Conference on User Modelling, Adaptation and Personalization (UMAP\u201915)."},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201916)","author":"He X.","unstructured":"X. He , H. Zhang , M. Y. Kan , and T. S. Chua . 2016. Fast matrix factorization for online recommendation with implicit feedback . In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201916) . ACM, New York, NY, 549\u2013558. X. He, H. Zhang, M. Y. Kan, and T. S. Chua. 2016. Fast matrix factorization for online recommendation with implicit feedback. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201916). ACM, New York, NY, 549\u2013558."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/963770.963772"},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the 31st International Conference on Machine Learning (ICML\u201914)","author":"Hern\u00e1ndez-Lobato J. M.","unstructured":"J. M. Hern\u00e1ndez-Lobato , N. Houlsby , and Z. Ghahramani . 2014. Probabilistic matrix factorization with non-random missing data . In Proceedings of the 31st International Conference on Machine Learning (ICML\u201914) . 1512\u20131520. J. M. Hern\u00e1ndez-Lobato, N. Houlsby, and Z. Ghahramani. 2014. Probabilistic matrix factorization with non-random missing data. In Proceedings of the 31st International Conference on Machine Learning (ICML\u201914). 1512\u20131520."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/963770.963774"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the 8th IEEE International Conference on Data Mining (ICDM\u201908)","author":"Hu Y.","unstructured":"Y. Hu , Y. Koren , and C. Volinsky . 2008. Collaborative filtering for implicit feedback datasets . In Proceedings of the 8th IEEE International Conference on Data Mining (ICDM\u201908) . IEEE Computer Society, Los Alamitos, CA, 15\u201319. Y. Hu, Y. Koren, and C. Volinsky. 2008. Collaborative filtering for implicit feedback datasets. In Proceedings of the 8th IEEE International Conference on Data Mining (ICDM\u201908). IEEE Computer Society, Los Alamitos, CA, 15\u201319."},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the International Conference on KDD Cup 2011 (KDDCUP\u201911)","author":"Jahrer M.","unstructured":"M. Jahrer and A. T\u00f6scher . 2011. Collaborative filtering ensemble for ranking . In Proceedings of the International Conference on KDD Cup 2011 (KDDCUP\u201911) . JLMR.org, 153\u2013167. M. Jahrer and A. T\u00f6scher. 2011. Collaborative filtering ensemble for ranking. In Proceedings of the International Conference on KDD Cup 2011 (KDDCUP\u201911). JLMR.org, 153\u2013167."},{"key":"e_1_2_1_40_1","doi-asserted-by":"crossref","unstructured":"D. Jannach L. Lerche I. Kamehkhosh and M. Jugovac. 2015. What recommenders recommend: An analysis of recommendation biases and possible countermeasures. User Model\u2019 User-Adapt\u2019 Interact\u2019 25 5 (Dec. 2015) 427\u2013491.  D. Jannach L. Lerche I. Kamehkhosh and M. Jugovac. 2015. What recommenders recommend: An analysis of recommendation biases and possible countermeasures. User Model\u2019 User-Adapt\u2019 Interact\u2019 25 5 (Dec. 2015) 427\u2013491.","DOI":"10.1007\/s11257-015-9165-3"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401944"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1557019.1557072"},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201920)","author":"Krichene W.","unstructured":"W. Krichene and S. Rendle . 2020. On sampled metrics for item recommendation . In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201920) . ACM, New York, NY, 1748\u20131757. W. Krichene and S. Rendle. 2020. On sampled metrics for item recommendation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201920). ACM, New York, NY, 1748\u20131757."},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201915)","author":"Lipani A.","unstructured":"A. Lipani , M. Lupu , and A. Hanbury . 2015. Splitting water: Precision and anti-precision to reduce pool bias . In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201915) . ACM, New York, NY, 103\u2013112. A. Lipani, M. Lupu, and A. Hanbury. 2015. Splitting water: Precision and anti-precision to reduce pool bias. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201915). ACM, New York, NY, 103\u2013112."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401083"},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of the 42nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201919)","author":"Lu H.","unstructured":"H. Lu , M. Zhang , W. Ma , C. Wang , F. xia, Y. Liu , L. Lin , and S. Ma . 2019. Effects of user negative experience in mobile news streaming . In Proceedings of the 42nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201919) . 705\u2013714. H. Lu, M. Zhang, W. Ma, C. Wang, F. xia, Y. Liu, L. Lin, and S. Ma. 2019. Effects of user negative experience in mobile news streaming. In Proceedings of the 42nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201919). 705\u2013714."},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the 3rd ACM Conference on Recommender Systems (RecSys\u201909)","author":"Marlin B. M.","unstructured":"B. M. Marlin and R. S. Zemel . 2009. Collaborative prediction and ranking with non-random missing data . In Proceedings of the 3rd ACM Conference on Recommender Systems (RecSys\u201909) . ACM, New York, NY, 5\u201312. B. M. Marlin and R. S. Zemel. 2009. Collaborative prediction and ranking with non-random missing data. In Proceedings of the 3rd ACM Conference on Recommender Systems (RecSys\u201909). ACM, New York, NY, 5\u201312."},{"key":"e_1_2_1_48_1","volume-title":"Proceedings of the 23rd Conference on Uncertainty in Artificial Intelligence (UAI\u201907)","author":"Marlin B. M.","unstructured":"B. M. Marlin , R. S. Zemel , S. T. Roweis , and M. Slaney . 2007. Collaborative filtering and the missing at random assumption . In Proceedings of the 23rd Conference on Uncertainty in Artificial Intelligence (UAI\u201907) . 267\u2013275. B. M. Marlin, R. S. Zemel, S. T. Roweis, and M. Slaney. 2007. Collaborative filtering and the missing at random assumption. In Proceedings of the 23rd Conference on Uncertainty in Artificial Intelligence (UAI\u201907). 267\u2013275."},{"key":"e_1_2_1_49_1","first-page":"4","article-title":"The movielens datasets: History and context","volume":"5","author":"Maxwell F. M.","year":"2015","unstructured":"F. M. Maxwell and J. A. Konstan . 2015 . The movielens datasets: History and context . ACM Transactions on Interactive Intelligent Systems (TIIS) 5 , 4 (Dec. 2015), 1--19. F. M. Maxwell and J. A. Konstan. 2015. The movielens datasets: History and context. ACM Transactions on Interactive Intelligent Systems (TIIS) 5, 4 (Dec. 2015), 1--19.","journal-title":"ACM Transactions on Interactive Intelligent Systems (TIIS)"},{"key":"e_1_2_1_50_1","volume-title":"Proceedings of ACM CHI 2006 Conference on Human Factors in Computing Systems (CHI\u201906)","author":"McNee S. M.","unstructured":"S. M. McNee , J. Riedl , and J. A. Konstan . 2006. Being accurate is not enough: How accuracy metrics have hurt recommender systems . In Proceedings of ACM CHI 2006 Conference on Human Factors in Computing Systems (CHI\u201906) . ACM, New York, NY, 1097\u20131101. S. M. McNee, J. Riedl, and J. A. Konstan. 2006. Being accurate is not enough: How accuracy metrics have hurt recommender systems. In Proceedings of ACM CHI 2006 Conference on Human Factors in Computing Systems (CHI\u201906). ACM, New York, NY, 1097\u20131101."},{"key":"e_1_2_1_51_1","volume-title":"Proceedings of the 43rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201920)","author":"Mena-Maldonado E.","unstructured":"E. Mena-Maldonado , R. Ca\u00f1amares , P. Castells , Y. Ren , and M. Sanderson . 2020. Agreement and disagreement between true and false-positive metrics in recommender systems evaluation . In Proceedings of the 43rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201920) . ACM, New York, NY, 841\u2013850. E. Mena-Maldonado, R. Ca\u00f1amares, P. Castells, Y. Ren, and M. Sanderson. 2020. Agreement and disagreement between true and false-positive metrics in recommender systems evaluation. In Proceedings of the 43rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201920). ACM, New York, NY, 841\u2013850."},{"key":"e_1_2_1_52_1","doi-asserted-by":"crossref","unstructured":"A. Moffat F. Scholer and Z. Yang. 2018. Estimating measurement uncertainty for information retrieval effectiveness metrics. J\u2019 Data Inf\u2019 Qual\u2019 10 3 (2018) 1--22.  A. Moffat F. Scholer and Z. Yang. 2018. Estimating measurement uncertainty for information retrieval effectiveness metrics. J\u2019 Data Inf\u2019 Qual\u2019 10 3 (2018) 1--22.","DOI":"10.1145\/3239572"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/1416950.1416952"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/336597.336662"},{"key":"e_1_2_1_55_1","doi-asserted-by":"crossref","unstructured":"X. Ning C. Desrosiers and G. Karypis. 2015. A comprehensive survey of neighborhood-based recommender systems. In Recommender Systems Handbook (2nd ed.) F. Ricci L. Rokach and B. Shapira (Eds.). Springer New York NY 37\u201376.  X. Ning C. Desrosiers and G. Karypis. 2015. A comprehensive survey of neighborhood-based recommender systems. In Recommender Systems Handbook (2nd ed.) F. Ricci L. Rokach and B. Shapira (Eds.). Springer New York NY 37\u201376.","DOI":"10.1007\/978-1-4899-7637-6_2"},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the IEEE 11th International Conference on Data Mining (ICDM\u201911)","author":"Ning X.","unstructured":"X. Ning and G. Karypis . 2011. SLIM: Sparse linear methods for top-N recommender systems . In Proceedings of the IEEE 11th International Conference on Data Mining (ICDM\u201911) . IEEE Computer Society, Los Alamitos, CA, 497\u2013506. X. Ning and G. Karypis. 2011. SLIM: Sparse linear methods for top-N recommender systems. In Proceedings of the IEEE 11th International Conference on Data Mining (ICDM\u201911). IEEE Computer Society, Los Alamitos, CA, 497\u2013506."},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of the 6th International Conference on Music Information Retrieval (ISMIR\u201905)","author":"Pampalk E.","unstructured":"E. Pampalk , T. Pohle , and G. Widmer . 2005. Dynamic playlist generation based on skip-ping behavior . In Proceedings of the 6th International Conference on Music Information Retrieval (ISMIR\u201905) . 634\u2013637. E. Pampalk, T. Pohle, and G. Widmer. 2005. Dynamic playlist generation based on skip-ping behavior. In Proceedings of the 6th International Conference on Music Information Retrieval (ISMIR\u201905). 634\u2013637."},{"key":"e_1_2_1_58_1","volume-title":"Proceedings of the 23rd International Joint Conference of Artificial Intelligence (IJCAI\u201913)","author":"Pan W.","unstructured":"W. Pan and L. Chen . 2013. GBPR: Group preference based Bayesian personalized ranking for one-class collaborative filtering . In Proceedings of the 23rd International Joint Conference of Artificial Intelligence (IJCAI\u201913) . AAAI Press, 2691\u20132697. W. Pan and L. Chen. 2013. GBPR: Group preference based Bayesian personalized ranking for one-class collaborative filtering. In Proceedings of the 23rd International Joint Conference of Artificial Intelligence (IJCAI\u201913). AAAI Press, 2691\u20132697."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-012-9125-0"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007601015854"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458092"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026647"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277756"},{"key":"e_1_2_1_64_1","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1007\/s10791-008-9059-7","article-title":"On information retrieval metrics designed for evaluation with incomplete relevance assessments","volume":"11","author":"Sakai T.","year":"2008","unstructured":"T. Sakai and N. Kando . 2008 . On information retrieval metrics designed for evaluation with incomplete relevance assessments . Inf. Retriev. 11 , 5 (Mar. 2008), 447\u2013470. T. Sakai and N. Kando. 2008. On information retrieval metrics designed for evaluation with incomplete relevance assessments. Inf. Retriev. 11, 5 (Mar. 2008), 447\u2013470.","journal-title":"Inf. Retriev."},{"key":"e_1_2_1_65_1","volume-title":"Proceedings of the 12th ACM Conference on Recommender Systems (RecSys\u201918)","author":"S\u00e1nchez P.","unstructured":"P. S\u00e1nchez and A. Bellog\u00edn . 2018. Measuring anti-relevance: A study on when recommendation algorithms produce bad suggestions . In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys\u201918) . ACM, New York, NY, 367\u2013371. P. S\u00e1nchez and A. Bellog\u00edn. 2018. Measuring anti-relevance: A study on when recommendation algorithms produce bad suggestions. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys\u201918). ACM, New York, NY, 367\u2013371."},{"key":"e_1_2_1_66_1","volume-title":"Proceedings of the 3rd International Conference on Machine Learning (ICML\u201916)","author":"Schnabel T.","unstructured":"T. Schnabel , A. Swaminathan , A. Singh , N. Chandak , and T. Joachims . 2016. Recommendations as treatments: Debiasing learning and evaluation . In Proceedings of the 3rd International Conference on Machine Learning (ICML\u201916) . 1670\u20131679. T. Schnabel, A. Swaminathan, A. Singh, N. Chandak, and T. Joachims. 2016. Recommendations as treatments: Debiasing learning and evaluation. In Proceedings of the 3rd International Conference on Machine Learning (ICML\u201916). 1670\u20131679."},{"key":"e_1_2_1_67_1","volume-title":"Proceedings of the 4th ACM Conference on Recommender Systems (RecSys\u201910)","author":"Shi Y.","unstructured":"Y. Shi , M. Larson , and A. Hanjalic . 2010. List-wise learning to rank with matrix factorization for collaborative filtering . In Proceedings of the 4th ACM Conference on Recommender Systems (RecSys\u201910) . ACM, New York, NY, 269\u2013272. Y. Shi, M. Larson, and A. Hanjalic. 2010. List-wise learning to rank with matrix factorization for collaborative filtering. In Proceedings of the 4th ACM Conference on Recommender Systems (RecSys\u201910). ACM, New York, NY, 269\u2013272."},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835804.1835895"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043957"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/2507157.2507160"},{"key":"e_1_2_1_71_1","volume-title":"Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS\u201917)","author":"Swaminathan A.","unstructured":"A. Swaminathan , A. Krishnamurthy , A. Agarwal , M. Dud\u00edk , J. Langford , D. Jose , and I. Zitouni . 2017. Off-policy evaluation for slate recommendation . In Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS\u201917) . Curran Associates, Inc., Red Hook, NY, 3635\u20133645. A. Swaminathan, A. Krishnamurthy, A. Agarwal, M. Dud\u00edk, J. Langford, D. Jose, and I. Zitouni. 2017. Off-policy evaluation for slate recommendation. In Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS\u201917). Curran Associates, Inc., Red Hook, NY, 3635\u20133645."},{"key":"e_1_2_1_72_1","first-page":"1","article-title":"How should relative changes be measured","volume":"39","author":"T\u00f6rnqvist L.","year":"1985","unstructured":"L. T\u00f6rnqvist , P. Vartia , and Y. O. Vartia . 1985 . How should relative changes be measured . Am. Stat. 39 , 1 (Feb. 1985), 43\u201346. L. T\u00f6rnqvist, P. Vartia, and Y. O. Vartia. 1985. How should relative changes be measured. Am. Stat. 39, 1 (Feb. 1985), 43\u201346.","journal-title":"Am. Stat."},{"key":"e_1_2_1_73_1","volume-title":"Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201919)","author":"Wang K.","unstructured":"K. Wang , T. Walker , and Z. Zheng . 2019. PSkip: Estimating relevance ranking quality from web search clickthrough data . In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201919) . ACM, New York, NY, 1355\u20131364. K. Wang, T. Walker, and Z. Zheng. 2019. PSkip: Estimating relevance ranking quality from web search clickthrough data. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201919). ACM, New York, NY, 1355\u20131364."},{"key":"e_1_2_1_74_1","volume-title":"Proceedings of the 13th ACM International Conference on Web Search and Data Mining (WSDM\u201920)","author":"Wicaksono A. F.","unstructured":"A. F. Wicaksono and A. Moffat . 2020. Metrics, user models, and satisfaction . In Proceedings of the 13th ACM International Conference on Web Search and Data Mining (WSDM\u201920) . ACM, New York, NY, 654\u2013662. A. F. Wicaksono and A. Moffat. 2020. Metrics, user models, and satisfaction. In Proceedings of the 13th ACM International Conference on Web Search and Data Mining (WSDM\u201920). ACM, New York, NY, 654\u2013662."},{"key":"e_1_2_1_75_1","volume-title":"Proceedings of the 12th ACM Conference on Recommender Systems (RecSys\u201918)","author":"Yang L.","unstructured":"L. Yang , Y. Cui , Y. Xuan , C. Wang , S. Belongie , and D. Estrin . 2018. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback . In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys\u201918) . ACM, New York, NY, 279\u2013287. L. Yang, Y. Cui, Y. Xuan, C. Wang, S. Belongie, and D. Estrin. 2018. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys\u201918). ACM, New York, NY, 279\u2013287."},{"key":"e_1_2_1_76_1","volume-title":"Proceedings of the 15th ACM International Conference on Information and Knowledge Management (CIKM\u201906)","author":"Yilmaz E.","unstructured":"E. Yilmaz and J. A. Aslam . 2006. Estimating average precision with incomplete and imperfect judgments . In Proceedings of the 15th ACM International Conference on Information and Knowledge Management (CIKM\u201906) . ACM, New York, NY, 102\u2013111. E. Yilmaz and J. A. Aslam. 2006. Estimating average precision with incomplete and imperfect judgments. In Proceedings of the 15th ACM International Conference on Information and Knowledge Management (CIKM\u201906). ACM, New York, NY, 102\u2013111."},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-007-0101-7"},{"key":"e_1_2_1_78_1","volume-title":"Proceedings of the 31st International Conference on Information Systems (ICIS\u201910)","author":"Yin D.","unstructured":"D. Yin , S. D. Bond , and H. Zhang . 2010. Are bad reviews always stronger than good? Asymmetric negativity bias in the formation of online consumer trust . In Proceedings of the 31st International Conference on Information Systems (ICIS\u201910) . Association for Information Systems, 1\u201318. D. Yin, S. D. Bond, and H. Zhang. 2010. Are bad reviews always stronger than good? Asymmetric negativity bias in the formation of online consumer trust. In Proceedings of the 31st International Conference on Information Systems (ICIS\u201910). Association for Information Systems, 1\u201318."},{"key":"e_1_2_1_79_1","volume-title":"Proceedings of the 14th Scandinavian Conference on Image Analysis (SCIA\u201905)","author":"Yuan Z.","unstructured":"Z. Yuan and E. Oja . 2005. Projective nonnegative matrix factorization for image compression and feature extraction . In Proceedings of the 14th Scandinavian Conference on Image Analysis (SCIA\u201905) . Springer-Verlag, Berlin, 333\u2013342. Z. Yuan and E. Oja. 2005. Projective nonnegative matrix factorization for image compression and feature extraction. In Proceedings of the 14th Scandinavian Conference on Image Analysis (SCIA\u201905). Springer-Verlag, Berlin, 333\u2013342."},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2004.11.003"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3452740","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3452740","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:47:27Z","timestamp":1750193247000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3452740"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,25]]},"references-count":79,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,7,26]]}},"alternative-id":["10.1145\/3452740"],"URL":"https:\/\/doi.org\/10.1145\/3452740","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,25]]},"assertion":[{"value":"2020-08-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-02-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-05-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}