{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T13:02:19Z","timestamp":1783429339219,"version":"3.54.6"},"reference-count":54,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2019,9,18]],"date-time":"2019-09-18T00:00:00Z","timestamp":1568764800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Center for Intelligent Information Retrieval"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2019,9,30]]},"abstract":"<jats:p>The ACM Recommender Systems Challenge 2018 focused on the task of automatic music playlist continuation, which is a form of the more general task of sequential recommendation. Given a playlist of arbitrary length with some additional meta-data, the task was to recommend up to 500 tracks that fit the target characteristics of the original playlist. For the RecSys Challenge, Spotify released a dataset of one million user-generated playlists. Participants could compete in two tracks, i.e., main and creative tracks. Participants in the main track were only allowed to use the provided training set, however, in the creative track, the use of external public sources was permitted. In total, 113 teams submitted 1,228 runs to the main track; 33 teams submitted 239 runs to the creative track. The highest performing team in the main track achieved an R-precision of 0.2241, an NDCG of 0.3946, and an average number of recommended songs clicks of 1.784. In the creative track, an R-precision of 0.2233, an NDCG of 0.3939, and a click rate of 1.785 was obtained by the best team. This article provides an overview of the challenge, including motivation, task definition, dataset description, and evaluation. We further report and analyze the results obtained by the top-performing teams in each track and explore the approaches taken by the winners. We finally summarize our key findings, discuss generalizability of approaches and results to domains other than music, and list the open avenues and possible future directions in the area of automatic playlist continuation.<\/jats:p>","DOI":"10.1145\/3344257","type":"journal-article","created":{"date-parts":[[2019,9,18]],"date-time":"2019-09-18T18:35:06Z","timestamp":1568831706000},"page":"1-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":48,"title":["An Analysis of Approaches Taken in the ACM RecSys Challenge 2018 for Automatic Music Playlist Continuation"],"prefix":"10.1145","volume":"10","author":[{"given":"Hamed","family":"Zamani","sequence":"first","affiliation":[{"name":"University of Massachusetts Amherst, Amherst, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1706-3406","authenticated-orcid":false,"given":"Markus","family":"Schedl","sequence":"additional","affiliation":[{"name":"Johannes Kepler University Linz, Linz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paul","family":"Lamere","sequence":"additional","affiliation":[{"name":"Spotify, New York"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ching-Wei","family":"Chen","sequence":"additional","affiliation":[{"name":"Spotify, New York"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,9,18]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267471.3267475"},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the 17th International Society for Music Information Retrieval Conference (ISMIR\u201916)","author":"Bogdanov Dmitry","year":"2016"},{"key":"e_1_2_1_3_1","doi-asserted-by":"crossref","unstructured":"Piotr Bojanowski Edouard Grave Armand Joulin and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606.  Piotr Bojanowski Edouard Grave Armand Joulin and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606.","DOI":"10.1162\/tacl_a_00051"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2652481"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1021240730564"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3240323.3240342"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080819"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2339530.2339643"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939785"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-36973-5_36"},{"key":"e_1_2_1_12_1","unstructured":"Jean-Charles de Borda. 1781. M\u00e9moire sur les \u00e9lections au scrutin. Histoire de l\u2019Acad\u00e9mie Royale des Sciences.  Jean-Charles de Borda. 1781. M\u00e9moire sur les \u00e9lections au scrutin. Histoire de l\u2019Acad\u00e9mie Royale des Sciences."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645737"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267471.3267486"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267471.3267473"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1013203451"},{"key":"e_1_2_1_17_1","volume-title":"Deep learning","author":"Goodfellow Ian"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052569"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2008.22"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026526"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267471.3267478"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 23rd ACM Conference on Intelligent User Interfaces Workshops: Intelligent Music Interfaces for Listening and Creation (MILC\u201918)","author":"Kamehkhosh Iman","year":"2018"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267471.3267472"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267471.3267477"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267471.3267485"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-011-9118-4"},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the 2nd Workshop on New Trends on Content-based Recommender Systems co-located with 9th ACM Conference on Recommender Systems. 14--21","author":"Kula Maciej","year":"2015"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11042-011-0742-0"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.383972"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2009937"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1348\/000712610X506831"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267471.3267474"},{"key":"e_1_2_1_35_1","unstructured":"Alireza Makhzani Jonathon Shlens Navdeep Jaitly Ian Goodfellow and Brendan Frey. 2015. Adversarial autoencoders. arXiv preprint arXiv:1511.05644.  Alireza Makhzani Jonathon Shlens Navdeep Jaitly Ian Goodfellow and Brendan Frey. 2015. Adversarial autoencoders. arXiv preprint arXiv:1511.05644."},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the 12th International Society for Music Information Retrieval Conference (ISMIR\u201911)","author":"McFee Brian","year":"2011"},{"key":"e_1_2_1_37_1","volume-title":"Proceedings of the Web Audio Conference on Collaborative Audio","author":"Meseguer-Brocal Gabriel","year":"2017"},{"key":"e_1_2_1_38_1","volume-title":"Advances in Neural Information Processing Systems 26","author":"Mikolov Tomas"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267471.3267484"},{"key":"e_1_2_1_40_1","volume-title":"Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI\u201909)","author":"Rendle Steffen","year":"2009"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201994)","author":"Robertson S. E.","year":"1884"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267471.3267488"},{"key":"e_1_2_1_43_1","volume-title":"A short history of the RecSys challenge. 37 (12","author":"Said Alan","year":"2016"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(88)90021-0"},{"key":"e_1_2_1_45_1","volume-title":"The psychological functions of music listening. Front. Psychol. 4","author":"Sch\u00e4fer Thomas","year":"2013"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13735-018-0154-2"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3079628.3079633"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267471.3267476"},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the 11th ACM Conference on Recommender Systems (RecSys\u201917)","author":"Vall Andreu","year":"2017"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267471.3267483"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267471.3267480"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2009934"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267471.3267482"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267471.3267479"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267471.3267481"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3344257","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3344257","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:44:24Z","timestamp":1750203864000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3344257"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9,18]]},"references-count":54,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2019,9,30]]}},"alternative-id":["10.1145\/3344257"],"URL":"https:\/\/doi.org\/10.1145\/3344257","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,9,18]]},"assertion":[{"value":"2018-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-09-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}