{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,8]],"date-time":"2026-02-08T03:36:20Z","timestamp":1770521780762,"version":"3.49.0"},"reference-count":91,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2023,7,14]],"date-time":"2023-07-14T00:00:00Z","timestamp":1689292800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"Open Access Publishing Fund of the Free University of Bozen-Bolzano"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Recomm. Syst."],"published-print":{"date-parts":[[2023,9,30]]},"abstract":"<jats:p>Reproducibility is a main principle in science and fundamental to ensure scientific progress. However, many recent works point out that there are widespread deficiencies for this aspect in the AI field, making the reproducibility of results impractical or even impossible. We therefore studied the state of reproducibility support on the topic of Reinforcement Learning &amp; Recommender Systems to analyse the situation in this context. We collected a total of 60 papers and analysed them by defining a set of variables to inspect the most important aspects that enable reproducibility, such as dataset, pre-processing code, hardware specifications, software dependencies, algorithm implementation, algorithm hyperparameters, and experiment code. Furthermore, we used the ACM Badges definitions assigning them to the selected papers. We discovered that, like in many other AI domains, the Reinforcement Learning &amp; Recommender Systems field is grappling with a reproducibility crisis, as none of the selected papers were reproducible when strictly applying the ACM Badges definitions according to our analysis.<\/jats:p>","DOI":"10.1145\/3596519","type":"journal-article","created":{"date-parts":[[2023,5,17]],"date-time":"2023-05-17T12:03:20Z","timestamp":1684325000000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["A Systematic Study on Reproducibility of Reinforcement Learning in Recommendation Systems"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0235-0421","authenticated-orcid":false,"given":"Emanuele","family":"Cavenaghi","sequence":"first","affiliation":[{"name":"Free University of Bozen-Bolzano, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9983-2330","authenticated-orcid":false,"given":"Gabriele","family":"Sottocornola","sequence":"additional","affiliation":[{"name":"Free University of Bozen-Bolzano, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1394-0507","authenticated-orcid":false,"given":"Fabio","family":"Stella","sequence":"additional","affiliation":[{"name":"University of Milano-Bicocca, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4805-5516","authenticated-orcid":false,"given":"Markus","family":"Zanker","sequence":"additional","affiliation":[{"name":"Free University of Bozen-Bolzano, Italy and University of Klagenfurt, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,7,14]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"crossref","unstructured":"Xiangyu Zhao Liang Zhang Long Xia Zhuoye Ding Dawei Yin and Jiliang Tang. 2019. Deep Reinforcement Learning for List-wise Recommendations. arXiv:1801.00209 [cs.LG].","DOI":"10.1145\/3320496.3320500"},{"key":"e_1_3_2_3_2","doi-asserted-by":"crossref","unstructured":"M. Mehdi Afsar Trafford Crump and Behrouz Far. 2021. Reinforcement learning based recommender systems: A survey. arXiv preprint arXiv:2101.06286 1 (2021).","DOI":"10.1145\/3543846"},{"issue":"7","key":"e_1_3_2_4_2","first-page":"145","article-title":"Reinforcement learning based recommender systems: A survey","volume":"55","author":"Afsar M. Mehdi","year":"2022","unstructured":"M. Mehdi Afsar, Trafford Crump, and Behrouz Far. 2022. Reinforcement learning based recommender systems: A survey. ACM Computing Surveys 55, 7, Article 145 (Dec.2022), 38 pages.","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_2_5_2","volume-title":"Advances in Neural Information Processing Systems","author":"Bai Xueying","year":"2019","unstructured":"Xueying Bai, Jian Guan, and Hongning Wang. 2019. A model-based reinforcement learning with adversarial training for online recommendation. In Advances in Neural Information Processing Systems. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch\u00e9-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32, Curran Associates, Inc., Vancouver."},{"key":"e_1_3_2_6_2","article-title":"A Markovian decision process","author":"Bellman Richard","year":"1957","unstructured":"Richard Bellman. 1957. A Markovian decision process. Journal of Mathematics and Mechanics 6, 5 (1957), 679\u2013684.","journal-title":"Journal of Mathematics and Mechanics"},{"key":"e_1_3_2_7_2","first-page":"69","volume-title":"Proceedings of the Workshop on AI and Electronic Commerce","author":"Burke Robin","year":"1999","unstructured":"Robin Burke. 1999. Integrating knowledge-based and collaborative-filtering recommender systems. In Proceedings of the Workshop on AI and Electronic Commerce. AAAI, 69\u201372."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2012.03.025"},{"key":"e_1_3_2_9_2","article-title":"Music recommender using deep embedding-based features and behavior-based reinforcement learning","author":"Chang Jia-Wei","year":"2019","unstructured":"Jia-Wei Chang, Ching-Yi Chiou, Jia-Yi Liao, Ying-Kai Hung, Chien-Che Huang, Kuan-Cheng Lin, and Ying-Hung Pu. 2019. Music recommender using deep embedding-based features and behavior-based reinforcement learning. Multimedia Tools and Applications 80, 26 (2019), 1\u201328.","journal-title":"Multimedia Tools and Applications"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33013312"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3289600.3290999"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3437963.3441764"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220122"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN48605.2020.9207010"},{"key":"e_1_3_2_15_2","first-page":"1052","volume-title":"Proceedings of the 36th International Conference on Machine Learning","volume":"97","author":"Chen Xinshi","year":"2019","unstructured":"Xinshi Chen, Shuang Li, Hui Li, Shaohua Jiang, Yuan Qi, and Le Song. 2019. Generative adversarial user model for reinforcement learning based recommendation system. In Proceedings of the 36th International Conference on Machine Learning. Vol. 97, PMLR, 1052\u20131061."},{"key":"e_1_3_2_16_2","unstructured":"Xiaocong Chen Lina Yao Julian McAuley Guanglin Zhou and Xianzhi Wang. 2021. A survey of deep reinforcement learning in recommender systems: A systematic review and future directions. arXiv preprint arXiv:2109.03540 1 (2021)."},{"key":"e_1_3_2_17_2","article-title":"Deep reinforcement learning in recommender systems: A survey and new perspectives","author":"Chen Xiaocong","year":"2023","unstructured":"Xiaocong Chen, Lina Yao, Julian McAuley, Guanglin Zhou, and Xianzhi Wang. 2023. Deep reinforcement learning in recommender systems: A survey and new perspectives. Knowledge-Based Systems 264, 1 (2023), 110335.","journal-title":"Knowledge-Based Systems"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3482347"},{"key":"e_1_3_2_19_2","unstructured":"Sungwoon Choi Heonseok Ha Uiwon Hwang Chanju Kim Jung-Woo Ha and Sungroh Yoon. 2018. Reinforcement learning based recommender system using biclustering technique. (2018). arXiv:1801.05532 [cs.IR]."},{"key":"e_1_3_2_20_2","unstructured":"Gabriel Dulac-Arnold Richard Evans Hado van Hasselt Peter Sunehag Timothy Lillicrap Jonathan Hunt Timothy Mann Theophane Weber Thomas Degris and Ben Coppin. 2016. Deep reinforcement learning in large discrete action spaces. (2016). arXiv:1512.07679 [cs.AI]."},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186165"},{"key":"e_1_3_2_22_2","volume-title":"Proceedings of the 13th ACM Conference on Recommender Systems (RecSys\u201919 Copenhagen, Denmark)","author":"Dacrema Maurizio Ferrari","year":"2019","unstructured":"Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. 2019. Are we really making much progress? A worrying analysis of recent neural recommendation approaches. In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys\u201919 Copenhagen, Denmark), Association for Computing Machinery, New York, NY, 101\u2013109."},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2019.00122"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/103162.103163"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330832"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10710-017-9314-z"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-70169-1_28"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11503"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/AIAM48774.2019.00011"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2021.102766"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052569"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-021-01256-7"},{"key":"e_1_3_2_33_2","unstructured":"Bal\u00e1zs Hidasi Alexandros Karatzoglou Linas Baltrunas and Domonkos Tikk. 2016. Session-based recommendations with recurrent neural networks. (2016). arXiv:1511.06939 [cs.LG]."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401225"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-68121-4_18"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.359.6377.725"},{"key":"e_1_3_2_37_2","unstructured":"Eugene Ie Vihan Jain Jing Wang Sanmit Narvekar Ritesh Agarwal Rui Wu Heng-Tze Cheng Morgane Lustman Vince Gatto Paul Covington Jim McFadden Tushar Chandra and Craig Boutilier. 2019. Reinforcement learning for slate-based recommender systems: A tractable decomposition and practical methodology. (2019). arXiv:1905.12767 [cs.LG]."},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/GWS.2018.8686513"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/CCAA.2015.7148534"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.5555\/1941904"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3359554"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401237"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2020.2998695"},{"key":"e_1_3_2_44_2","first-page":"591","volume-title":"Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems (AAMAS\u201915 Istanbul, Turkey)","author":"Liebman Elad","year":"2015","unstructured":"Elad Liebman, Maytal Saar-Tsechansky, and Peter Stone. 2015. DJ-MC: A reinforcement-learning agent for music playlist recommendation. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems (AAMAS\u201915 Istanbul, Turkey). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 591\u2013599."},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3371858"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2020.07.057"},{"key":"e_1_3_2_47_2","unstructured":"Feng Liu Ruiming Tang Xutao Li Weinan Zhang Yunming Ye Haokun Chen Huifeng Guo and Yuzhou Zhang. 2018. Deep reinforcement learning based recommendation with explicit user-item interactions modeling. arXiv preprint arXiv:1810.12027 (2018)."},{"key":"e_1_3_2_48_2","article-title":"State representation modeling for deep reinforcement learning based recommendation","author":"Liu Feng","year":"2020","unstructured":"Feng Liu, Ruiming Tang, Xutao Li, Weinan Zhang, Yunming Ye, Haokun Chen, Huifeng Guo, Yuzhou Zhang, and Xiuqiang He. 2020. State representation modeling for deep reinforcement learning based recommendation. Knowledge-Based Systems 205, 1 (2020), 106170.","journal-title":"Knowledge-Based Systems"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/TALE.2018.8615217"},{"key":"e_1_3_2_50_2","unstructured":"Zhongqi Lu and Qiang Yang. 2016. Partially observable Markov decision process for recommender systems. arXiv preprint arXiv:1608.07793 (2016)."},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380130"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICOIACT.2018.8350761"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.5555\/1768197.1768209"},{"key":"e_1_3_2_54_2","first-page":"771","volume-title":"Proceedings of the 35th IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201920 Virtual Event, Australia)","author":"Pham Hung Viet","year":"2021","unstructured":"Hung Viet Pham, Shangshu Qian, Jiannan Wang, Thibaud Lutellier, Jonathan Rosenthal, Lin Tan, Yaoliang Yu, and Nachiappan Nagappan. 2021. Problems and opportunities in training deep learning software systems: An analysis of variance. In Proceedings of the 35th IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201920 Virtual Event, Australia). ACM, New York, NY, 771\u2013783."},{"key":"e_1_3_2_55_2","article-title":"Improving reproducibility in machine learning research: A report from the NeurIPS 2019 reproducibility program","volume":"22","author":"Pineau Joelle","year":"2021","unstructured":"Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivi\u00e8re, Alina Beygelzimer, Florence d\u2019Alch\u00e9 Buc, Emily Fox, and Hugo Larochelle. 2021. Improving reproducibility in machine learning research: A report from the NeurIPS 2019 reproducibility program. Journal of Machine Learning Research 22, 1 (2021), 7459\u20137478.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13638-019-1619-6"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3412488"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/192844.192905"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-85820-3_1"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1090\/S0002-9904-1952-09620-8"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1561\/2200000068"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3488560.3498471"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210002"},{"key":"e_1_3_2_64_2","unstructured":"Peter Sunehag Richard Evans Gabriel Dulac-Arnold Yori Zwols Daniel Visentin and Ben Coppin. 2015. Deep reinforcement learning with attention for slate Markov decision processes with high-dimensional states and actions. arXiv preprint arXiv:1512.01124 (2015)."},{"key":"e_1_3_2_65_2","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton Richard S.","year":"2018","unstructured":"Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. MIT Press."},{"key":"e_1_3_2_66_2","first-page":"20","volume-title":"Proceedings of the 24th International Joint Conference on Artificial Intelligence","author":"Theocharous Georgios","year":"2015","unstructured":"Georgios Theocharous, Philip S. Thomas, and Mohammad Ghavamzadeh. 2015. Personalized ad recommendation systems for life-time value optimization with guarantees. In Proceedings of the 24th International Joint Conference on Artificial Intelligence. AAAI, Buenos Aires, Argentina, 20."},{"key":"e_1_3_2_67_2","first-page":"732","volume-title":"Proceedings of the 6th Hellenic European Conference on Computer Mathematics & Its Applications","author":"Vozalis Emmanouil","year":"2003","unstructured":"Emmanouil Vozalis and Konstantinos G. Margaritis. 2003. Analysis of recommender systems algorithms. In Proceedings of the 6th Hellenic European Conference on Computer Mathematics & Its Applications. 732\u2013745."},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i5.16569"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219961"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401134"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2018.00074"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380098"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-47426-3_8"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/JIOT.2020.2974848"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331203"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1109\/JIOT.2020.3027586"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i5.16580"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401147"},{"key":"e_1_3_2_79_2","first-page":"1186","volume-title":"Proceedings of the 15th ACM International Conference on Web Search and Data Mining (WSDM\u201922 Virtual Event, AZ, USA)","author":"Xin Xin","year":"2022","unstructured":"Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, and Joemon M. Jose. 2022. Supervised advantage actor-critic for recommender systems. In Proceedings of the 15th ACM International Conference on Web Search and Data Mining (WSDM\u201922 Virtual Event, AZ, USA). ACM, New York, NY, 1186\u20131196."},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3343031.3350935"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCT46805.2019.8947012"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.3301435"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-29894-4_59"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401170"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1145\/3320496.3320500"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1145\/3240323.3240374"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3412044"},{"key":"e_1_3_2_88_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219886"},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3185994"},{"key":"e_1_3_2_90_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401174"},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-18579-4_7"},{"key":"e_1_3_2_92_2","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3371801"}],"container-title":["ACM Transactions on Recommender Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3596519","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3596519","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:48:00Z","timestamp":1750178880000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3596519"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,14]]},"references-count":91,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,9,30]]}},"alternative-id":["10.1145\/3596519"],"URL":"https:\/\/doi.org\/10.1145\/3596519","relation":{},"ISSN":["2770-6699"],"issn-type":[{"value":"2770-6699","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,14]]},"assertion":[{"value":"2022-05-18","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-04-05","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-07-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}