{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T02:39:36Z","timestamp":1772159976000,"version":"3.50.1"},"reference-count":59,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2022,10,31]],"date-time":"2022-10-31T00:00:00Z","timestamp":1667174400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Estonian Centre of Excellence in IT (EXCITE)","award":["VLTTI19503"],"award-info":[{"award-number":["VLTTI19503"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Boosting the sales of e-commerce services is guaranteed once users find more items matching their interests in a short amount of time. Consequently, recommendation systems have become a crucial part of any successful e-commerce service. Although various recommendation techniques could be used in e-commerce, a considerable amount of attention has been drawn to session-based recommendation systems in recent years. This growing interest is due to security concerns over collecting personalized user behavior data, especially due to recent general data protection regulations. In this work, we present a comprehensive evaluation of the state-of-the-art deep learning approaches used in the session-based recommendation. In session-based recommendation, a recommendation system counts on the sequence of events made by a user within the same session to predict and endorse other items that are more likely to correlate with their preferences. Our extensive experiments investigate baseline techniques (e.g., nearest neighbors and pattern mining algorithms) and deep learning approaches (e.g., recurrent neural networks, graph neural networks, and attention-based networks). Our evaluations show that advanced neural-based models and session-based nearest neighbor algorithms outperform the baseline techniques in most scenarios. However, we found that these models suffer more in the case of long sessions when there exists drift in user interests, and when there are not enough data to correctly model different items during training. Our study suggests that using the hybrid models of different approaches combined with baseline algorithms could lead to substantial results in session-based recommendations based on dataset characteristics. We also discuss the drawbacks of current session-based recommendation algorithms and further open research directions in this field.<\/jats:p>","DOI":"10.3390\/e24111575","type":"journal-article","created":{"date-parts":[[2022,10,31]],"date-time":"2022-10-31T23:26:32Z","timestamp":1667258792000},"page":"1575","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Comprehensive Empirical Evaluation of Deep Learning Approaches for Session-Based Recommendation in E-Commerce"],"prefix":"10.3390","volume":"24","author":[{"given":"Mohamed","family":"Maher","sequence":"first","affiliation":[{"name":"iCV Lab, Institute of Technology, University of Tartu, 51009 Tartu, Estonia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Perseverance Munga","family":"Ngoy","sequence":"additional","affiliation":[{"name":"iCV Lab, Institute of Technology, University of Tartu, 51009 Tartu, Estonia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Aleksandrs","family":"Rebriks","sequence":"additional","affiliation":[{"name":"iCV Lab, Institute of Technology, University of Tartu, 51009 Tartu, Estonia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Cagri","family":"Ozcinar","sequence":"additional","affiliation":[{"name":"iCV Lab, Institute of Technology, University of Tartu, 51009 Tartu, Estonia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Josue","family":"Cuevas","sequence":"additional","affiliation":[{"name":"Machine Learning Group, Big Data Department, Rakuten Inc., Tokyo 158-0094, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rajasekhar","family":"Sanagavarapu","sequence":"additional","affiliation":[{"name":"Machine Learning Group, Big Data Department, Rakuten Inc., Tokyo 158-0094, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8460-5717","authenticated-orcid":false,"given":"Gholamreza","family":"Anbarjafari","sequence":"additional","affiliation":[{"name":"iCV Lab, Institute of Technology, University of Tartu, 51009 Tartu, Estonia"},{"name":"PwC Advisory, 00180 Helsinki, Finland"},{"name":"Institute of Higher Education, Yildiz Technical University, Yildiz, Be\u015fikta\u015f District, Istanbul 34349, Turkey"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,10,31]]},"reference":[{"key":"ref_1","unstructured":"Lee, D., and Hosanagar, K. (2022, August 01). Impact of Recommender Systems on Sales Volume and Diversity 2014. Available online: https:\/\/www.semanticscholar.org\/paper\/Impact-of-Recommender-Systems-on-Sales-Volume-and-Lee-Hosanagar\/8109f0a47f433b32979d9b3f2da9facee5eba4ad."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Liu, D.C., Rogers, S., Shiau, R., Kislyuk, D., Ma, K.C., Zhong, Z., Liu, J., and Jing, Y. (2017, January 3\u20137). Related pins at pinterest: The evolution of a real-world recommender system. Proceedings of the 26th International Conference on World Wide Web Companion, Perth, Australia.","DOI":"10.1145\/3041021.3054202"},{"key":"ref_3","unstructured":"Van den Oord, A., Dieleman, S., and Schrauwen, B. (2013, January 5\u201310). Deep content-based music recommendation. Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1007\/s13740-016-0060-9","article-title":"Content-based video recommendation system based on stylistic visual features","volume":"5","author":"Deldjoo","year":"2016","journal-title":"J. Data Semant."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Wang, H., Zhang, F., Xie, X., and Guo, M. (2018, January 23\u201327). DKN: Deep knowledge-aware network for news recommendation. Proceedings of the 2018 World Wide Web Conference, Lyon, France.","DOI":"10.1145\/3178876.3186175"},{"key":"ref_6","unstructured":"Koren, Y. (July, January 28). Collaborative filtering with temporal dynamics. Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Paris, France."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Song, Y., Elkahky, A.M., and He, X. (2016, January 17\u201321). Multi-rate deep learning for temporal recommendation. Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, Pisa, Italy.","DOI":"10.1145\/2911451.2914726"},{"key":"ref_8","unstructured":"Fan, W., Ma, Y., Li, Q., He, Y., Zhao, Y.E., Tang, J., and Yin, D. (2022, August 01). Graph Neural Networks for Social Recommendation. Available online: https:\/\/ieeexplore.ieee.org\/document\/9139346."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1164","DOI":"10.1109\/TNNLS.2016.2514368","article-title":"On deep learning for trust-aware recommendations in social networks","volume":"28","author":"Deng","year":"2016","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Mohallick, I., De Moor, K., \u00d6zg\u00f6bek, \u00d6., and Gulla, J.A. (2018, January 11\u201313). Towards New Privacy Regulations in Europe: Users\u2019 Privacy Perception in Recommender Systems. Proceedings of the International Conference on Security, Privacy and Anonymity in Computation, Communication and Storage, Melbourne, NSW, Australia.","DOI":"10.1007\/978-3-030-05345-1_27"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Wang, H., Wang, N., and Yeung, D.Y. (2015, January 10\u201313). Collaborative deep learning for recommender systems. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Sydney, Australia.","DOI":"10.1145\/2783258.2783273"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Kim, D., Park, C., Oh, J., Lee, S., and Yu, H. (2016, January 15\u201319). Convolutional matrix factorization for document context-aware recommendation. Proceedings of the 10th ACM Conference on Recommender Systems, Boston, MA, USA.","DOI":"10.1145\/2959100.2959165"},{"key":"ref_14","unstructured":"Hidasi, B., Karatzoglou, A., Baltrunas, L., and Tikk, D. (2015). Session-based recommendations with recurrent neural networks. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Kang, W.C., and McAuley, J. (2018, January 17\u201320). Self-attentive sequential recommendation. Proceedings of the 2018 IEEE International Conference on Data Mining (ICDM), Singapore.","DOI":"10.1109\/ICDM.2018.00035"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Liu, Q., Zeng, Y., Mokhosi, R., and Zhang, H. (2018, January 19\u201323). STAMP: Short-term attention\/memory priority model for session-based recommendation. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.","DOI":"10.1145\/3219819.3219950"},{"key":"ref_17","unstructured":"Wu, S., Tang, Y., Zhu, Y., Wang, L., Xie, X., and Tan, T. (February, January 27). Session-based recommendation with graph neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1007\/s11257-018-9209-6","article-title":"Evaluation of session-based recommendation algorithms","volume":"28","author":"Ludewig","year":"2018","journal-title":"User Model. User-Adapt. Interact."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Jannach, D., and Ludewig, M. (2017, January 27\u201331). When recurrent neural networks meet the neighborhood for session-based recommendation. Proceedings of the Eleventh ACM Conference on Recommender Systems, Como, Italy.","DOI":"10.1145\/3109859.3109872"},{"key":"ref_20","unstructured":"Ludewig, M., Mauro, N., Latifi, S., and Jannach, D. (2019). Empirical Analysis of Session-Based Recommendation Algorithms. arXiv."},{"key":"ref_21","unstructured":"Kamehkhosh, I., Jannach, D., and Ludewig, M. (2017, January 27\u201331). A Comparison of Frequent Pattern Techniques and a Deep Learning Method for Session-Based Recommendation. Proceedings of the RecTemp@ RecSys, Como, Italy. Available online: https:\/\/www.semanticscholar.org\/paper\/A-Comparison-of-Frequent-Pattern-Techniques-and-a-Kamehkhosh-Jannach\/5f6ff6990a1afac7e7b5616283b1d4e67a9d034f."},{"key":"ref_22","first-page":"66","article-title":"Sequence-aware recommender systems","volume":"51","author":"Quadrana","year":"2018","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Jannach, D., Lerche, L., and Jugovac, M. (2015, January 16\u201320). Adaptation and evaluation of recommendations for short-term shopping goals. Proceedings of the 9th ACM Conference on Recommender Systems, Vienna, Austria.","DOI":"10.1145\/2792838.2800176"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1145\/2652481","article-title":"Automated generation of music playlists: Survey and experiments","volume":"47","author":"Bonnin","year":"2015","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Garcin, F., Dimitrakakis, C., and Faltings, B. (2013). Personalized news recommendation with context trees. arXiv.","DOI":"10.1145\/2507157.2507166"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Hosseinzadeh, A., Hariri, N., Mobasher, B., and Burke, R. (2015, January 16\u201320). Adapting recommendations to contextual changes using hierarchical hidden markov models. Proceedings of the 9th ACM Conference on Recommender Systems, Vienna, Austria.","DOI":"10.1145\/2792838.2799684"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Tavakol, M., and Brefeld, U. (2014, January 6\u201310). Factored MDPs for detecting topics of user sessions. Proceedings of the 8th ACM Conference on Recommender Systems, Foster City, CA, USA.","DOI":"10.1145\/2645710.2645739"},{"key":"ref_28","unstructured":"Cheng, C., Yang, H., Lyu, M.R., and King, I. (2013, January 3\u20139). Where you like to go next: Successive point-of-interest recommendation. Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, Beijing, China."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"He, R., and McAuley, J. (2016, January 12\u201315). Fusing similarity models with markov chains for sparse sequential recommendation. Proceedings of the 2016 IEEE 16th International Conference on Data Mining (ICDM), Barcelona, Spain.","DOI":"10.1109\/ICDM.2016.0030"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Hidasi, B., and Karatzoglou, A. (2018, January 22\u201326). Recurrent neural networks with top-k gains for session-based recommendations. Proceedings of the 27th ACM International Conference on Information and Knowledge Management, Torino, Italy.","DOI":"10.1145\/3269206.3271761"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Hidasi, B., Quadrana, M., Karatzoglou, A., and Tikk, D. (2016, January 15\u201319). Parallel recurrent neural network architectures for feature-rich session-based recommendations. Proceedings of the 10th ACM Conference on Recommender Systems, Boston, MA, USA.","DOI":"10.1145\/2959100.2959167"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Quadrana, M., Karatzoglou, A., Hidasi, B., and Cremonesi, P. (2017, January 27\u201331). Personalizing session-based recommendations with hierarchical recurrent neural networks. Proceedings of the Eleventh ACM Conference on Recommender Systems, Como, Italy.","DOI":"10.1145\/3109859.3109896"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Ruocco, M., Skrede, O.S.L., and Langseth, H. (2017, January 27). Inter-session modeling for session-based recommendation. Proceedings of the 2nd Workshop on Deep Learning for Recommender Systems, Como, Italy.","DOI":"10.1145\/3125486.3125491"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Tuan, T.X., and Phuong, T.M. (2017, January 27\u201331). 3D convolutional networks for session-based recommendation with content features. Proceedings of the Eleventh ACM Conference on Recommender Systems, Como, Italy.","DOI":"10.1145\/3109859.3109900"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Yuan, F., Karatzoglou, A., Arapakis, I., Jose, J.M., and He, X. (2019, January 11\u201315). A Simple Convolutional Generative Network for Next Item Recommendation. Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, Melbourne, VIC, Australia.","DOI":"10.1145\/3289600.3290975"},{"key":"ref_36","unstructured":"Wang, M., Ren, P., Mei, L., Chen, Z., Ma, J., and de Rijke, M. (2022, August 01). A Collaborative Session-based Recommendation Approach with Parallel Memory Modules 2019. Available online: https:\/\/dl.acm.org\/doi\/10.1145\/3331184.3331210#:~:text=We%20propose%20a%20Collaborative%20Session-based%20Recommendation%20Machine%20%28CSRM%29%2C,Encoder%20%28IME%29%20and%20an%20Outer%20Memory%20Encoder%20%28OME%29."},{"key":"ref_37","unstructured":"Graves, A., Wayne, G., and Danihelka, I. (2014). Neural turing machines. arXiv."},{"key":"ref_38","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017, January 4\u20139). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Li, J., Ren, P., Chen, Z., Ren, Z., Lian, T., and Ma, J. (2017, January 6\u201310). Neural attentive session-based recommendation. Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, Singapore.","DOI":"10.1145\/3132847.3132926"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Chen, W., Cai, F., Chen, H., and de Rijke, M. (2019, January 3\u20137). A Dynamic Co-attention Network for Session-based Recommendation. Proceedings of the 28th ACM International Conference on Information and Knowledge Management, Beijing, China.","DOI":"10.1145\/3357384.3357964"},{"key":"ref_41","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W., and Jiang, P. (2019). BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. arXiv.","DOI":"10.1145\/3357384.3357895"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Barkan, O., and Koenigstein, N. (2016). Item2Vec: Neural Item Embedding for Collaborative Filtering. arXiv.","DOI":"10.1109\/MLSP.2016.7738886"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Tang, J., and Wang, K. (2018, January 5\u20139). Personalized top-n sequential recommendation via convolutional sequence embedding. Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, Marina Del Rey, CA, USA.","DOI":"10.1145\/3159652.3159656"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"734","DOI":"10.1109\/TKDE.2005.99","article-title":"Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions","volume":"17","author":"Adomavicius","year":"2005","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1145\/170036.170072","article-title":"Mining association rules between sets of items in large databases","volume":"22","author":"Agrawal","year":"1993","journal-title":"ACM Sigmod Rec."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Steck, H. (2011, January 23\u201327). Item popularity and recommendation accuracy. Proceedings of the Fifth ACM conference on Recommender Systems, Chicago, IL, USA.","DOI":"10.1145\/2043932.2043957"},{"key":"ref_48","unstructured":"Wen, Z. (2022, August 01). Recommendation System Based on Collaborative Filtering. CS229 Lecture Notes. Available online: https:\/\/www.dominodatalab.com\/blog\/recommender-systems-collaborative-filtering."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Kabbur, S., Ning, X., and Karypis, G. (2013, January 11\u201314). Fism: Factored item similarity models for top-n recommender systems. Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Chicago, IL, USA.","DOI":"10.1145\/2487575.2487589"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Hidasi, B., and Karatzoglou, A. (2018). Recurrent Neural Networks with Top-k Gains for Session-based Recommendations. arXiv.","DOI":"10.1145\/3269206.3271761"},{"key":"ref_51","unstructured":"Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Shani, G., and Gunawardana, A. (2011). Evaluating recommendation systems. Recommender Systems Handbook, Springer.","DOI":"10.1007\/978-0-387-85820-3_8"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1016\/j.neucom.2017.05.046","article-title":"Bag-of-concepts: Comprehending document representation through clustering words in distributed representation","volume":"266","author":"Kim","year":"2017","journal-title":"Neurocomputing"},{"key":"ref_54","first-page":"12","article-title":"An evaluation of training size impact on validation accuracy for optimized convolutional neural networks","volume":"1","author":"Tschannen","year":"2018","journal-title":"SMU Data Sci. Rev."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"351","DOI":"10.1007\/s11257-017-9194-1","article-title":"Session-based item recommendation in e-commerce: On short-term intents, reminders, trends and discounts","volume":"27","author":"Jannach","year":"2017","journal-title":"User Model. User-Adapt. Interact."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Kumar, S., Zhang, X., and Leskovec, J. (2019, January 4\u20138). Predicting dynamic embedding trajectory in temporal interaction networks. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA.","DOI":"10.1145\/3292500.3330895"},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Hutter, F., Babic, D., Hoos, H.H., and Hu, A.J. (2007, January 11\u201314). Boosting verification by automatic tuning of decision procedures. Proceedings of the Formal Methods in Computer Aided Design (FMCAD\u201907), Austin, TX, USA.","DOI":"10.1109\/FAMCAD.2007.9"},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Hutter, F., Kotthoff, L., and Vanschoren, J. (2019). Automated Machine Learning, Springer.","DOI":"10.1007\/978-3-030-05318-5"},{"key":"ref_59","unstructured":"Elshawi, R., Maher, M., and Sakr, S. (2019). Automated Machine Learning: State-of-The-Art and Open Challenges. arXiv."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/11\/1575\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:06:54Z","timestamp":1760144814000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/11\/1575"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,31]]},"references-count":59,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2022,11]]}},"alternative-id":["e24111575"],"URL":"https:\/\/doi.org\/10.3390\/e24111575","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,31]]}}}