{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,11]],"date-time":"2026-05-11T10:56:37Z","timestamp":1778496997146,"version":"3.51.4"},"reference-count":53,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2019,5,10]],"date-time":"2019-05-10T00:00:00Z","timestamp":1557446400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100013296","name":"EIT Digital","doi-asserted-by":"publisher","award":["17164"],"award-info":[{"award-number":["17164"]}],"id":[{"id":"10.13039\/100013296","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Recommender systems have gained a lot of popularity due to their large adoption in various industries such as entertainment and tourism. Numerous research efforts have focused on formulating and advancing state-of-the-art of systems that recommend the right set of items to the right person. However, these recommender systems are hard to compare since the published evaluation results are computed on diverse datasets and obtained using different methodologies. In this paper, we researched and prototyped an offline evaluation framework called Sequeval that is designed to evaluate recommender systems capable of suggesting sequences of items. We provide a mathematical definition of such sequence-based recommenders, a methodology for performing their evaluation, and the implementation details of eight metrics. We report the lessons learned using this framework for assessing the performance of four baselines and two recommender systems based on Conditional Random Fields (CRF) and Recurrent Neural Networks (RNN), considering two different datasets. Sequeval is publicly available and it aims to become a focal point for researchers and practitioners when experimenting with sequence-based recommender systems, providing comparable and objective evaluation results.<\/jats:p>","DOI":"10.3390\/info10050174","type":"journal-article","created":{"date-parts":[[2019,5,13]],"date-time":"2019-05-13T03:57:07Z","timestamp":1557719827000},"page":"174","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Sequeval: An Offline Evaluation Framework for Sequence-Based Recommender Systems"],"prefix":"10.3390","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3821-5379","authenticated-orcid":false,"given":"Diego","family":"Monti","sequence":"first","affiliation":[{"name":"Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129 Turin, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3898-7480","authenticated-orcid":false,"given":"Enrico","family":"Palumbo","sequence":"additional","affiliation":[{"name":"Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129 Turin, Italy"},{"name":"Istituto Superiore Mario Boella, Via Pier Carlo Boggio 61, 10138 Turin, Italy"},{"name":"EURECOM, Campus SophiaTech, 450 Route des Chappes, 06410 Biot, France"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0083-813X","authenticated-orcid":false,"given":"Giuseppe","family":"Rizzo","sequence":"additional","affiliation":[{"name":"LINKS Foundation, Via Pier Carlo Boggio 61, 10138 Turin, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7362-906X","authenticated-orcid":false,"given":"Maurizio","family":"Morisio","sequence":"additional","affiliation":[{"name":"Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129 Turin, Italy"}]}],"member":"1968","published-online":{"date-parts":[[2019,5,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Ricci, F., Rokach, L., and Shapira, B. (2015). Recommender Systems: Introduction and Challenges. Recommender Systems Handbook, Springer. [2nd ed.]. Chapter 1.","DOI":"10.1007\/978-1-4899-7637-6_1"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Su, X., and Khoshgoftaar, T.M. (2009). A Survey of Collaborative Filtering Techniques. Adv. Artif. Intell., 1\u201319.","DOI":"10.1155\/2009\/421425"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1145\/245108.245124","article-title":"Fab: Content-based, collaborative recommendation","volume":"40","author":"Shoham","year":"1997","journal-title":"ACM Commun."},{"key":"ref_4","unstructured":"Basu, C., Hirsh, H., and Cohen, W. (1998, January 26\u201330). Recommendation As Classification: Using Social and Content-based Information in Recommendation. Proceedings of the Fifteenth National Conference on Artificial Intelligence, Madison, WI, USA."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1007\/s11042-016-4209-1","article-title":"A holistic approach for personalization, relevance feedback & recommendation in enriched multimedia content","volume":"77","author":"Stai","year":"2016","journal-title":"Multimed. Tools Appl."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1007\/s11257-012-9136-x","article-title":"Time-aware recommender systems: A comprehensive survey and analysis of existing evaluation protocols","volume":"24","author":"Campos","year":"2013","journal-title":"User Model. User-Adapt. Interact."},{"key":"ref_7","unstructured":"Ding, Y., and Li, X. (November, January 31). Time weight collaborative filtering. Proceedings of the 14th ACM International Conference on Information and Knowledge Management, Bremen, Germany."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Rendle, S., Freudenthaler, C., and Schmidt-Thieme, L. (2010, January 26\u201330). Factorizing personalized Markov chains for next-basket recommendation. Proceedings of the 19th International Conference on World Wide Web, Raleigh, NC, USA.","DOI":"10.1145\/1772690.1772773"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"He, R., Kang, W.C., and McAuley, J. (2017, January 27\u201331). Translation-based Recommendation. Proceedings of the Eleventh ACM Conference on Recommender Systems, Como, Italy.","DOI":"10.1145\/3109859.3109882"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Wang, P., Guo, J., Lan, Y., Xu, J., Wan, S., and Cheng, X. (2015, January 9\u201313). Learning Hierarchical Representation Model for Next Basket Recommendation. Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, Santiago, Chile.","DOI":"10.1145\/2766462.2767694"},{"key":"ref_11","unstructured":"Zhou, B., Hui, S., and Chang, K. (2004, January 1\u20133). An intelligent recommender system using sequential Web access patterns. Proceedings of the IEEE Conference on Cybernetics and Intelligent Systems, Singapore."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3190616","article-title":"Sequence-Aware Recommender Systems","volume":"51","author":"Quadrana","year":"2018","journal-title":"ACM Comput. Surv."},{"key":"ref_13","unstructured":"Jurafsky, D., and Martin, J.H. (2008). Speech and Language Processing, Prentice Hall."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1145\/963770.963772","article-title":"Evaluating collaborative filtering recommender systems","volume":"22","author":"Herlocker","year":"2004","journal-title":"ACM Trans. Inf. Syst."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Chen, S., Moore, J.L., Turnbull, D., and Joachims, T. (2012, January 12\u201316). Playlist prediction via metric embedding. Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Beijing, China.","DOI":"10.1145\/2339530.2339643"},{"key":"ref_16","unstructured":"Feng, S., Li, X., Zeng, Y., Cong, G., Chee, Y.M., and Yuan, Q. (2015, January 25\u201331). Personalized Ranking Metric Embedding for Next New POI Recommendation. Proceedings of the 24th International Conference on Artificial Intelligence, Buenos Aires, Argentina."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1007\/s11257-018-9209-6","article-title":"Evaluation of session-based recommendation algorithms","volume":"28","author":"Ludewig","year":"2018","journal-title":"User Model. User-Adapt. Interact."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Gunawardana, A., and Shani, G. (2015). Evaluating Recommender Systems. Recommender Systems Handbook, Springer. [2nd ed.]. Chapter 8.","DOI":"10.1007\/978-1-4899-7637-6_8"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1007\/s11257-015-9165-3","article-title":"What recommenders recommend: An analysis of recommendation biases and possible countermeasures","volume":"25","author":"Jannach","year":"2015","journal-title":"User Model. User-Adapt. Interact."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Costa, A., and Roda, F. (2011, January 25\u201327). Recommender systems by means of information retrieval. Proceedings of the International Conference on Web Intelligence, Mining and Semantics, Sogndal, Norway.","DOI":"10.1145\/1988688.1988755"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Pouli, V., Kafetzoglou, S., Tsiropoulou, E.E., Dimitriou, A., and Papavassiliou, S. (2015, January 13\u201315). Personalized multimedia content retrieval through relevance feedback techniques for enhanced user experience. Proceedings of the IEEE 13th International Conference on Telecommunications, Graz, Austria.","DOI":"10.1109\/ConTEL.2015.7231205"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Albanese, M., d\u2019Acierno, A., Moscato, V., Persia, F., and Picariello, A. (2011, January 18\u201321). A Multimedia Semantic Recommender System for Cultural Heritage Applications. Proceedings of the IEEE Fifth International Conference on Semantic Computing, Palo Alto, CA, USA.","DOI":"10.1109\/ICSC.2011.47"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Aggarwal, C.C. (2015). Mining Discrete Sequences. Data Mining, Springer International Publishing. Chapter 15.","DOI":"10.1007\/978-3-319-14142-8_15"},{"key":"ref_24","first-page":"432","article-title":"Collaborative filtering based on subsequence matching: A new approach","volume":"418\u2013419","year":"2017","journal-title":"Inf. Sci."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1145\/360825.360861","article-title":"A linear space algorithm for computing maximal common subsequences","volume":"18","author":"Hirschberg","year":"1975","journal-title":"ACM Commun."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1007\/s11042-010-0480-8","article-title":"A multimedia recommender integrating object features and user behavior","volume":"50","author":"Albanese","year":"2010","journal-title":"Multimed. Tools Appl."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2532640","article-title":"A Multimedia Recommender System","volume":"13","author":"Albanese","year":"2013","journal-title":"ACM Trans. Internet Technol."},{"key":"ref_28","unstructured":"Amato, F., Moscato, V., Picariello, A., and Sperli, G. (February, January 30). KIRA: A System for Knowledge-Based Access to Multimedia Art Collections. Proceedings of the IEEE 11th International Conference on Semantic Computing, San Diego, CA, USA."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Su, X., Sperli, G., Moscato, V., Picariello, A., Esposito, C., and Choi, C. (2019). An Edge Intelligence Empowered Recommender System Enabling Cultural Heritage Applications. IEEE Trans. Ind. Inform., 1.","DOI":"10.1109\/TII.2019.2908056"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"606","DOI":"10.1007\/s10791-017-9312-z","article-title":"Statistical biases in Information Retrieval metrics for recommender systems","volume":"20","author":"Castells","year":"2017","journal-title":"Inf. Retr. J."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Said, A., and Bellog\u00edn, A. (2014, January 6\u201310). Comparative recommender system evaluation. Proceedings of the 8th ACM Conference on Recommender Systems, Foster City, CA, USA.","DOI":"10.1145\/2645710.2645746"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Turpin, A.H., and Hersh, W. (2001, January 9\u201312). Why batch and user evaluations do not give the same results. Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New Orleans, LA, USA.","DOI":"10.1145\/383952.383992"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Ge, M., Delgado-Battenfeld, C., and Jannach, D. (2010, January 26\u201330). Beyond Accuracy: Evaluating Recommender Systems by Coverage and Serendipity. Proceedings of the Fourth ACM Conference on Recommender Systems, Barcelona, Spain.","DOI":"10.1145\/1864708.1864761"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Paraschakis, D., Nilsson, B.J., and Hollander, J. (2015, January 9\u201311). Comparative Evaluation of Top-N Recommenders in e-Commerce: An Industrial Perspective. Proceedings of the IEEE 14th International Conference on Machine Learning and Applications, Miami, FL, USA.","DOI":"10.1109\/ICMLA.2015.183"},{"key":"ref_35","first-page":"1137","article-title":"A Neural Probabilistic Language Model","volume":"3","author":"Bengio","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_36","unstructured":"Rijsbergen, C.J.V. (1979). Information Retrieval, Butterworth-Heinemann."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Sarwar, B., Karypis, G., Konstan, J., and Riedl, J. (2000, January 17\u201320). Analysis of recommendation algorithms for e-commerce. Proceedings of the 2nd ACM Conference on Electronic Commerce, Minneapolis, MN, USA.","DOI":"10.1145\/352871.352887"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1002\/(SICI)1097-4571(199503)46:2<133::AID-ASI6>3.0.CO;2-Z","article-title":"Measuring retrieval effectiveness based on user preference of documents","volume":"46","author":"Yao","year":"1995","journal-title":"J. Am. Soc. Inf. Sci."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"422","DOI":"10.1145\/582415.582418","article-title":"Cumulated gain-based evaluation of IR techniques","volume":"20","year":"2002","journal-title":"ACM Trans. Inf. Syst."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Ziegler, C.N., McNee, S.M., Konstan, J.A., and Lausen, G. (2005, January 10\u201314). Improving recommendation lists through topic diversification. Proceedings of the 14th International Conference on World Wide Web, Chiba, Japan.","DOI":"10.1145\/1060745.1060754"},{"key":"ref_41","unstructured":"Noia, T.D., Ostuni, V.C., Rosati, J., Tomeo, P., and Sciascio, E.D. (2014, January 6\u201310). An analysis of users\u2019 propensity toward diversity in recommendations. Proceedings of the 8th ACM Conference on Recommender Systems, Foster City, CA, USA."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Vargas, S., and Castells, P. (2011, January 23\u201327). Rank and relevance in novelty and diversity metrics for recommender systems. Proceedings of the Fifth ACM Conference on Recommender Systems, Chicago, IL, USA.","DOI":"10.1145\/2043932.2043955"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"695","DOI":"10.1016\/j.ipm.2015.06.008","article-title":"An investigation on the serendipity problem in recommender systems","volume":"51","author":"Lops","year":"2015","journal-title":"Inf. Process. Manag."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Herlocker, J.L., Konstan, J.A., and Riedl, J. (2000, January 2\u20136). Explaining collaborative filtering recommendations. Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work, Philadelphia, PA, USA.","DOI":"10.1145\/358916.358995"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Monti, D., Palumbo, E., Rizzo, G., and Morisio, M. (2018, January 2). Sequeval: A framework to assess and benchmark sequence-based recommender systems. Proceedings of the Workshop on Offline Evaluation for Recommender Systems at the 12th ACM Conference on Recommender Systems, Vancouver, BC, Canada.","DOI":"10.3390\/info10050174"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1006\/csla.1999.0128","article-title":"An empirical study of smoothing techniques for language modeling","volume":"13","author":"Chen","year":"1999","journal-title":"Comput. Speech Lang."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1561\/2200000013","article-title":"An Introduction to Conditional Random Fields","volume":"4","author":"Sutton","year":"2011","journal-title":"Found. Trends Mach. Learn."},{"key":"ref_48","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press. Available online: http:\/\/www.deeplearningbook.org."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"773","DOI":"10.1090\/S0025-5718-1980-0572855-7","article-title":"Updating quasi-Newton matrices with limited storage","volume":"35","author":"Nocedal","year":"1980","journal-title":"Math. Comput."},{"key":"ref_50","unstructured":"Palumbo, E., Rizzo, G., Troncy, R., and Baralis, E. (2017, January 27). Predicting Your Next Stop-over from Location-based Social Network Data with Recurrent Neural Networks. Proceedings of the 2nd Workshop on Recommenders in Tourism Co-Located with 11th ACM Conference on Recommender Systems, Como, Italy."},{"key":"ref_51","unstructured":"Sutskever, I., Martens, J., and Hinton, G. (July, January 28). Generating text with recurrent neural networks. Proceedings of the 28th International Conference on Machine Learning, Bellevue, WA, USA."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2827872","article-title":"The MovieLens Datasets: History and Context","volume":"5","author":"Harper","year":"2015","journal-title":"ACM Trans. Interact. Intell. Syst."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","article-title":"A Mathematical Theory of Communication","volume":"27","author":"Shannon","year":"1948","journal-title":"Bell Syst. Tech. J."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/10\/5\/174\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T12:50:44Z","timestamp":1760187044000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/10\/5\/174"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,5,10]]},"references-count":53,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2019,5]]}},"alternative-id":["info10050174"],"URL":"https:\/\/doi.org\/10.3390\/info10050174","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,5,10]]}}}