{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T04:34:04Z","timestamp":1777696444840,"version":"3.51.4"},"reference-count":32,"publisher":"SAGE Publications","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IDA"],"published-print":{"date-parts":[[2023,7,20]]},"abstract":"<jats:p>Keyphrase generation (KG) aims at condensing the content from the source text to the target concise phrases. Though many KG algorithms have been proposed, most of them are tailored into deep learning settings with various specially designed strategies and may fail in solving the bias exposure problem. Reinforcement Learning (RL), a class of control optimization techniques, are well suited to compensate for some of the limitations of deep learning methods. Nevertheless, RL methods typically suffer from four core difficulties in keyphrase generation: environment interaction and effective exploration, complex action control, reward design, and task-specific obstacle. To tackle this difficult but significant task, we present RegRL-KG, including actor-critic based-reinforcement learning control and L1 policy regularization under the first principle of minimizing the maximum likelihood estimation (MLE) criterion by a sequence-to-sequence (Seq2Seq) deep learnining model, for efficient keyphrase generation. The agent utilizes an actor-critic network to control the generated probability distribution and employs L1 policy regularization to solve the bias exposure problem. Extensive experiments show that our method brings improvement in terms of the evaluation metrics on five scientific article benchmark datasets.<\/jats:p>","DOI":"10.3233\/ida-226561","type":"journal-article","created":{"date-parts":[[2023,6,2]],"date-time":"2023-06-02T13:06:28Z","timestamp":1685711188000},"page":"1003-1021","source":"Crossref","is-referenced-by-count":0,"title":["RegRL-KG: Learning an L1 regularized reinforcement agent for keyphrase generation"],"prefix":"10.1177","volume":"27","author":[{"given":"Yu","family":"Yao","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Peng","family":"Yang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guangzhen","family":"Zhao","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Juncheng","family":"Leng","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","reference":[{"issue":"5","key":"10.3233\/IDA-226561_ref1","doi-asserted-by":"crossref","first-page":"834","DOI":"10.1109\/TSMC.1983.6313077","article-title":"Neuron like adaptive elements that can solve difficult learning control problems","volume":"SMC-13","author":"Barto","year":"1983","journal-title":"IEEE Transactions on Systems, Man, and Cybernetics"},{"key":"10.3233\/IDA-226561_ref2","unstructured":"F. Boudin, Y. Gallina and A. Aizawa, Keyphrase generation for scientific document retrieval, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 20\u2019, Online, 2016, pp. 1118\u20131126."},{"key":"10.3233\/IDA-226561_ref3","unstructured":"K. Bousmalis, G. Trigeorgis, N. Silberman, D. Krishnan and D. Erhan, Domain Separation Networks, in: Proceedings of the Advances in Neural Information Processing Systems, NIPS 16\u2019, Barcelona, Spain, 2016, pp. 343\u2013351."},{"key":"10.3233\/IDA-226561_ref4","doi-asserted-by":"crossref","unstructured":"H.P. Chan, W. Chen, L. Wang and I. King, Neural keyphrase generation via reinforcement learning with adaptive rewards, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 19\u2019, Florence, Italy, 2019, pp. 2163\u20132174.","DOI":"10.18653\/v1\/P19-1208"},{"key":"10.3233\/IDA-226561_ref5","doi-asserted-by":"crossref","unstructured":"W. Chen, H.P. Chan, P.J. Li, L.D. Bing and I. King, An integrated approach for keyphrase generation via exploring the power of retrieval and extraction, in: Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 19\u2019, Minneapolis, Minnesota, 2019, pp. 2846\u20132856.","DOI":"10.18653\/v1\/N19-1292"},{"key":"10.3233\/IDA-226561_ref6","doi-asserted-by":"crossref","unstructured":"W. Chen, H.P. Chan, P.J. Li and I. King, Exclusive hierarchical decoding for deep keyphrase generation, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 20\u2019, Online, 2020, pp. 1095\u20131105.","DOI":"10.18653\/v1\/2020.acl-main.103"},{"key":"10.3233\/IDA-226561_ref7","doi-asserted-by":"crossref","unstructured":"W. Chen, Y.F. Gao, J.N. Zhang, I. King and M.R. Lyu, Title-guided encoding for keyphrase generation, in: Proceedings of the AAAI Conference on Artificial Intelligence, AAAI 19\u2019, Honolulu, Hawaii, USA, 2019, pp. 6268\u20136275.","DOI":"10.1609\/aaai.v33i01.33016268"},{"key":"10.3233\/IDA-226561_ref8","doi-asserted-by":"crossref","unstructured":"B. Chiu, S.K. Sahu, D. Thomas, N. Sengupta and M. Mahdy, Autoencoding keyword correlation graph for document clustering, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 20\u2019, Online, 2020, pp. 3974\u20133981.","DOI":"10.18653\/v1\/2020.acl-main.366"},{"key":"10.3233\/IDA-226561_ref9","doi-asserted-by":"crossref","unstructured":"K. Cho, B.V. Merri\u00ebnboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk and Y. Bengio, Learning phrase representations using RNN encoder\u00e2\u0080\u0093decoder for statistical machine translation, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP 14\u2019, Doha, Qatar, 2014, pp. 1724\u20131734.","DOI":"10.3115\/v1\/D14-1179"},{"key":"10.3233\/IDA-226561_ref10","doi-asserted-by":"crossref","unstructured":"S.D. Gollapalli, X.L. Li and P. Yang, Incorporating expert knowledge into keyphrase extraction, in: Proceedings of the AAAI Conference on Artificial Intelligence, AAAI 2017\u2019, San Francisco, California, USA, 2017, 31(1).","DOI":"10.1609\/aaai.v31i1.10986"},{"key":"10.3233\/IDA-226561_ref11","doi-asserted-by":"crossref","unstructured":"A. Hulth, Improved automatic keyword extraction given more linguistic knowledge, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP 03\u2019, 2003, pp. 216\u2013223.","DOI":"10.3115\/1119355.1119383"},{"key":"10.3233\/IDA-226561_ref12","doi-asserted-by":"crossref","unstructured":"A. Hulth and B.B. Megyesi, A study on automatically extracted keywords in text categorization, in: Proceedings of the International Conference on Computational Linguistics and Annual Meeting of the Association for Computational Linguistics, ACL 06\u2019, Sydney, Australia, 2006, pp. 537\u2013544.","DOI":"10.3115\/1220175.1220243"},{"key":"10.3233\/IDA-226561_ref13","doi-asserted-by":"crossref","unstructured":"N. Jiang, S. Jin, Z.Y. Duan and C.S. Zhang, RL-duet: Online music accompaniment generation using deep reinforcement learning, in: Proceedings of the AAAI Conference on Artificial Intelligence, AAAI 20\u2019, 2020, pp. 710\u2013718.","DOI":"10.1609\/aaai.v34i01.5413"},{"key":"10.3233\/IDA-226561_ref14","unstructured":"S.N. Kim, O. Medelyan, M.Y. Kan and T. Baldwin, SemEval-2010 task 5: Automatic keyphrase extraction from scientific articles, in: Proceedings of the 5th International Workshop on Semantic Evaluation, SemEval 10\u2019, Uppsala, Sweden, 2010, pp. 21\u201326."},{"key":"10.3233\/IDA-226561_ref15","unstructured":"D.P. Kingma and J. Ba, Adam: A method for stochastic optimization, in: Proceedings of the International Conference on Learning Representations, ICLR 15\u2019, San Diego, USA, 2015."},{"key":"10.3233\/IDA-226561_ref17","doi-asserted-by":"crossref","unstructured":"Y. Luan, M. Ostendorf and H. Hajishirzi, Scientific information extraction with semi-supervised neural tagging, in: Proceedings of the Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 17\u2019, Copenhagen, Denmark, 2017, pp. 2641\u20132651.","DOI":"10.18653\/v1\/D17-1279"},{"key":"10.3233\/IDA-226561_ref18","doi-asserted-by":"crossref","unstructured":"R. Meng, S.Q. Zhao, S.G. Han, D.Q. He, P. Brusilovsky and Y. Chi, Deep keyphrase generation, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 17\u2019, Vancouver, Canada, 2017, pp. 582\u2013592.","DOI":"10.18653\/v1\/P17-1054"},{"key":"10.3233\/IDA-226561_ref19","unstructured":"B. Samy, V. Oriol, J. Navdeep and S. Noam, Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks, in: Proceedings of the 28th International Conference on Neural Information Processing Systems \u2013 Volume 1, Montreal, Canada, 2015, pp. 1171\u20131179."},{"key":"10.3233\/IDA-226561_ref21","doi-asserted-by":"crossref","unstructured":"M. Sarmad, H.J. Lee and Y.M. Kim, Rl-gan-net: A reinforcement learning agent controlled gan network for real-time point cloud shape completion, in: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 19\u2019, Long Beach, Canada, 2019, pp. 5898\u20135907.","DOI":"10.1109\/CVPR.2019.00605"},{"key":"10.3233\/IDA-226561_ref22","unstructured":"J. Schulman, P. Moritz, S. Levine, M. Jordan and P. Abbeel, High-dimensional continuous control using generalized advantage estimation, in: Proceedings of the International Conference on Learning Representations, ICLR 16\u2019, San Juan, Puerto Rico, 2016."},{"issue":"5","key":"10.3233\/IDA-226561_ref23","doi-asserted-by":"crossref","first-page":"1054","DOI":"10.1109\/TNN.1998.712192","article-title":"Reinforcement learning: An introduction","volume":"9","author":"Sutton","year":"1998","journal-title":"IEEE Transactions on Neural Networks"},{"key":"10.3233\/IDA-226561_ref24","doi-asserted-by":"crossref","unstructured":"A. Swaminathan, H.M. Zhang, D. Mahata, R. Gosangi, R.R. Shah and A. Stent, A preliminary exploration of GANs for keyphrase generation, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP 20\u2019, Online, 2020, pp. 8021\u20138030.","DOI":"10.18653\/v1\/2020.emnlp-main.645"},{"key":"10.3233\/IDA-226561_ref25","unstructured":"T.P. Lillicrap, J.J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver and D. Wierstra, Continuous control with deep reinforcement learning, in: Proceedings of the International Conference on Learning Representations, ICLR 16\u2019, San Juan, Puerto Rico, 2016."},{"key":"10.3233\/IDA-226561_ref26","unstructured":"Z.P. Tu, Z.D. Lu, Y. Liu, X.H. Liu and H. Li, Modeling coverage for neural machine translation, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 16\u2019, Berlin, Germany, 2016, pp. 76\u201385."},{"key":"10.3233\/IDA-226561_ref27","doi-asserted-by":"crossref","unstructured":"A. Ushiku, S. Mori, H. Kameko and Y. Tsuruoka, Game state retrieval with keyword queries, in: Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 17\u2019, Shinjuku, Tokyo, Japan, 2017, pp. 877\u2013880.","DOI":"10.1145\/3077136.3080668"},{"key":"10.3233\/IDA-226561_ref28","unstructured":"L. Wang and C. Cardie, Domain-independent abstract generation for focused meeting summarization, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 13\u2019, Sofia, Bulgaria, 2013, pp. 1395\u20131405."},{"key":"10.3233\/IDA-226561_ref29","doi-asserted-by":"crossref","unstructured":"Y. Wang, J. Li, H.P. Chan, I. King, M.R. Lyu and S.M. Shi, Topic-Aware Neural Keyphrase Generation for Social Media Language, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 19\u2019, Florence, Italy, 2019, pp. 2516\u20132526.","DOI":"10.18653\/v1\/P19-1240"},{"issue":"2","key":"10.3233\/IDA-226561_ref30","doi-asserted-by":"crossref","first-page":"270","DOI":"10.1162\/neco.1989.1.2.270","article-title":"A learning algorithm for continually running fully recurrent neural networks","volume":"1","author":"Williams","year":"1989","journal-title":"Neural Computation"},{"key":"10.3233\/IDA-226561_ref31","unstructured":"J.C. Ye, T. Gui, Y.C. Luo, Y.G. Xu and Q. Zhang, One2Set: Generating diverse keyphrases as a set, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 21\u2019, Online, 2021, pp. 4598\u20134608."},{"key":"10.3233\/IDA-226561_ref32","doi-asserted-by":"crossref","unstructured":"X.D. Yuan, T. Wang, R. Meng, K. Thaker, P. Brusilovsky, D.Q. He and Adam Trischler, One size does not fit all: Generating and evaluating variable number of keyphrases, in Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 20\u2019, Online, 2020, pp. 7961\u20137975.","DOI":"10.18653\/v1\/2020.acl-main.710"},{"key":"10.3233\/IDA-226561_ref33","doi-asserted-by":"crossref","unstructured":"Q. Zhang, Y. Wang, Y.Y. Gong and X.J. Huang, Keyphrase extraction using deep recurrent neural networks on Twitter, in: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 16\u2019, Austin, Texas, 2016, pp. 836\u2013845.","DOI":"10.18653\/v1\/D16-1080"},{"key":"10.3233\/IDA-226561_ref34","unstructured":"J.W. Li, G. Michel, B. Chris, J.F. Gao and D. Bill, A Diversity-Promoting Objective Function for Neural Conversation Models, in: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, California, 2016, pp. 110\u2013119."}],"container-title":["Intelligent Data Analysis"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/IDA-226561","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T09:20:05Z","timestamp":1777454405000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/full\/10.3233\/IDA-226561"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,20]]},"references-count":32,"journal-issue":{"issue":"4"},"URL":"https:\/\/doi.org\/10.3233\/ida-226561","relation":{},"ISSN":["1088-467X","1571-4128"],"issn-type":[{"value":"1088-467X","type":"print"},{"value":"1571-4128","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,20]]}}}