{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,11]],"date-time":"2026-05-11T04:08:58Z","timestamp":1778472538336,"version":"3.51.4"},"reference-count":38,"publisher":"Cambridge University Press (CUP)","issue":"2","license":[{"start":{"date-parts":[[2026,3,30]],"date-time":"2026-03-30T00:00:00Z","timestamp":1774828800000},"content-version":"unspecified","delay-in-days":57,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotica"],"published-print":{"date-parts":[[2026,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Preference-based reinforcement learning (PbRL) significantly simplifies the design of reward functions in reinforcement learning (RL) tasks. However, because of the tasks\u2019 complexity, intransitive preferences, and sensitivity to preference errors, PbRL requires substantial feedback to achieve the desired performance. This extensive reliance on expert input notably increases the burden on participants. We have developed a novel framework: Self-teacher-learning preference-based reinforcement learning (STL-PbRL). In the teacher-led (TL) module, agents learn more reliable reward prediction models (RM) through TL PbRL. In the self-learning (SL) module, agents utilizing the preference comparison approach for trajectory segments integrate sparse but critical, easily designed task-oriented information into the feedback process. The STL-PbRL framework incorporates the SL module to refine the RM initially generated by the TL module. We have demonstrated that this integration significantly enhances RM by enabling RM to converge toward an optimal reward model that effectively supports achieving a training policy that meets task objectives. This streamlined and efficient STL-PbRL framework enables a more accurate and efficient training process. Our experimental results confirm that the SL module seamlessly integrates with existing PbRL algorithms, significantly reducing the need for feedback and alleviating the impact of errors in preference indications. This efficiency and effectiveness highlight that STL-PbRL innovates, simplifies, and enhances the RL training process across various applications.<\/jats:p>","DOI":"10.1017\/s0263574726103300","type":"journal-article","created":{"date-parts":[[2026,3,30]],"date-time":"2026-03-30T10:31:29Z","timestamp":1774866689000},"page":"739-763","source":"Crossref","is-referenced-by-count":0,"title":["Integrating task objectives for improved performance of preference-based reinforcement learning in different tasks"],"prefix":"10.1017","volume":"44","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-8728-7115","authenticated-orcid":false,"given":"Jiabin","family":"Hu","sequence":"first","affiliation":[{"id":[{"id":"https:\/\/ror.org\/013q1eq08","id-type":"ROR","asserted-by":"publisher"}],"name":"College of Intelligent Robotics and Advanced Manufacturing, Fudan University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Peng","family":"Zhai","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/013q1eq08","id-type":"ROR","asserted-by":"publisher"}],"name":"College of Intelligent Robotics and Advanced Manufacturing, Fudan University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaoyi","family":"Wei","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/013q1eq08","id-type":"ROR","asserted-by":"publisher"}],"name":"College of Intelligent Robotics and Advanced Manufacturing, Fudan University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Cunwang","family":"Liu","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/013q1eq08","id-type":"ROR","asserted-by":"publisher"}],"name":"College of Intelligent Robotics and Advanced Manufacturing, Fudan University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lihua","family":"Zhang","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/013q1eq08","id-type":"ROR","asserted-by":"publisher"}],"name":"College of Intelligent Robotics and Advanced Manufacturing, Fudan University"},{"name":"Jilin Provincial Key Laboratory of Intelligence Science and Engineering"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"56","published-online":{"date-parts":[[2026,3,30]]},"reference":[{"key":"S0263574726103300_ref34","unstructured":"[34] Ye, D. , Chen, G. , Zhang, W. , Chen, S. , Yuan, B. , Liu, B. , Chen, J. , Liu, Z. , Qiu, F. , Yu, H. , Yin, Y. , Shi, B. , Wang, L. , Shi, T. , Fu, Q. , Yang, W. , Huang, L. and Liu, W. , \u201cTowards Playing Full Moba Games with Deep Reinforcement Learning,\u201d In: Advances in Neural Information Processing Systems (H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, eds.), vol. 33 (Curran Associates, Inc., 2020) pp. 621\u2013632."},{"key":"S0263574726103300_ref8","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-05732-2"},{"key":"S0263574726103300_ref4","article-title":"Open problems and fundamental limitations of reinforcement learning from human feedback","author":"Casper","year":"2023","journal-title":"Trans. Mach. Learn. Res."},{"key":"S0263574726103300_ref7","doi-asserted-by":"crossref","unstructured":"[7] de Heuvel, J. , Seiler, F. and Bennewitz, M. , \u201cEnQuery: Ensemble Policies for Diverse Query-Generation in Preference Alignment of Robot Navigation (2024). arXiv preprint arXiv:2404.04852.","DOI":"10.1109\/RO-MAN60168.2024.10731470"},{"key":"S0263574726103300_ref13","unstructured":"[13] Hu, X. , Li, J. , Zhan, X. , Jia, Q.-S. and Zhang, Y.-Q. , \u201cQuery-policy misalignment in preference-based reinforcement learning,\u201d (2023). arXiv preprint arXiv:2305.17400."},{"key":"S0263574726103300_ref32","unstructured":"[32] Verma, M. and Metcalf, K. , \u201cHindsight PRIORs for Reward Learning from Human Preferences,\u201d In: The Twelfth International Conference on Learning Representations 2024, OpenReview.net (2024)."},{"key":"S0263574726103300_ref24","unstructured":"[24] Park, J. , Seo, Y. , Shin, J. , Lee, H. , Abbeel, P. and Lee, K. , \u201cSURF: Semi-Supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-Based Reinforcement Learning,\u201d In: International Conference on Learning Representations 2022, OpenReview.net (2022)."},{"key":"S0263574726103300_ref37","unstructured":"[37] Zheng, R. , Dou, S. , Gao, S. , Hua, Y. , Shen, W. , Wang, B. , Liu, Y. , Jin, S. , Liu, Q. , Zhou, Y. , Xiong, L. , Chen, L. , Xi, Z. , Xu, N. , Lai, W. , Zhu, M. , Chang, C. , Yin, Z. , Weng, R. , Cheng, W. , Huang, H. , Sun, T. , Yan, H. , Gui, T. , Zhang, Q. , Qiu, X. and Huang, X. \u201cSecrets of RLHF in large language models part I: PPO,\u201d (2023). arXiv preprint arXiv:2307.04964."},{"key":"S0263574726103300_ref5","unstructured":"[5] Cheng, J. , Xiong, G. , Dai, X. , Miao, Q. , Lv, Y. and Wang, F.-Y. , \u201cRIME: Robust Preference-Based Reinforcement Learning with Noisy Preferences,\u201d In: Forty-first International Conference on Machine Learning (PMLR, 2024)."},{"key":"S0263574726103300_ref10","doi-asserted-by":"publisher","DOI":"10.3390\/s23073762"},{"key":"S0263574726103300_ref2","first-page":"324","article-title":"Rank analysis of incomplete block designs: I. the method of paired comparisons","volume":"39","author":"Bradley","year":"1952","journal-title":"Biometrika"},{"key":"S0263574726103300_ref21","unstructured":"[21] Liang, X. , Shu, K. , Lee, K. and Abbeel, P. , \u201cReward Uncertainty for Exploration in Preference-Based Reinforcement Learning,\u201d In: International Conference on Learning Representations 2022, OpenReview.net (2022)."},{"key":"S0263574726103300_ref12","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA57147.2024.10610151"},{"key":"S0263574726103300_ref22","doi-asserted-by":"publisher","DOI":"10.52202\/068431-1618"},{"key":"S0263574726103300_ref1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v37i5.25733"},{"key":"S0263574726103300_ref23","unstructured":"[23] Ma, Y. J. , Liang, W. , Wang, G. , Huang, D.-A. , Bastani, O. , Jayaraman, D. , Zhu, Y. , Fan, L. and Anandkumar, A. , \u201cEureka: Human-Level Reward Design Via Coding Large Language Models,\u201d In: The Twelfth International Conference on Learning Representations 2024, OpenReview.net (2024)."},{"key":"S0263574726103300_ref6","unstructured":"[6] Christiano, P. F. , Leike, J. , Brown, T. , Martic, M. , Legg, S. and Amodei, D. , \u201cDeep Reinforcement Learning From Human Preferences,\u201d In: Advances in Neural Information Processing Systems (I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, eds.), vol. 30 (Curran Associates, Inc., 2017)."},{"key":"S0263574726103300_ref28","unstructured":"[28] Stiennon, N. , Ouyang, L. , Wu, J. , Ziegler, D. , Lowe, R. , Voss, C. , Radford, A. , Amodei, D. and Christiano, P. F. , \u201cLearning to Summarize with Human Feedback,\u201d In: Advances in Neural Information Processing Systems (H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, eds.), vol. 33 (Curran Associates, Inc., 2020) pp. 3008\u20133021."},{"key":"S0263574726103300_ref31","doi-asserted-by":"publisher","DOI":"10.1016\/j.simpa.2020.100022"},{"key":"S0263574726103300_ref9","unstructured":"[9] Haarnoja, T. , Zhou, A. , Abbeel, P. and Levine, S. , \u201cSoft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor,\u201d In: International Conference on Machine Learning (PMLR, 2018) pp. 1861\u20131870."},{"key":"S0263574726103300_ref33","doi-asserted-by":"publisher","DOI":"10.1109\/TSMC.2024.3384992"},{"key":"S0263574726103300_ref27","unstructured":"[27] Singh, S. , Lewis, R. L. and Barto, A. G. , \u201dWhere Do Rewards Come From,\u201c In: Proceedings of the Annual Conference of the Cognitive Science Society (Cognitive Science Society, 2009) pp. 2601\u20132606."},{"key":"S0263574726103300_ref3","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA57147.2024.10610878"},{"key":"S0263574726103300_ref36","unstructured":"[36] Yuan, Z. , Yuan, H. , Li, C. , Dong, G. , Tan, C. and Zhou, C. , \u201cScaling relationship on learning mathematical reasoning with large language models,\u201d (2023). arXiv preprint arXiv:2308.01825."},{"key":"S0263574726103300_ref30","unstructured":"[30] Swamy, G. , Dann, C. , Kidambi, R. , Wu, S. and Agarwal, A. , \u201cA Minimaximalist Approach to Reinforcement Learning from Human Feedback,\u201d In: Forty-first International Conference on Machine Learning 2024 (PMLR, 2024)."},{"key":"S0263574726103300_ref38","unstructured":"[38] Ziegler, D. M. , Stiennon, N. , Wu, J. , Brown, T. B. , Radford, A. , Amodei, D. , Christiano, P. and Irving, G. , \u201cFine-tuning language models from human preferences,\u201d (2019). arXiv preprint arXiv:1909.08593."},{"key":"S0263574726103300_ref19","unstructured":"[19] Lee, K. , Smith, L. , Dragan, A. and Abbeel, P. , \u201cB-Pref: Benchmarking preference-based reinforcement learning.\u201d (2021a). arXiv preprint arXiv:2111.03026."},{"key":"S0263574726103300_ref18","doi-asserted-by":"publisher","DOI":"10.52202\/068431-0804"},{"key":"S0263574726103300_ref20","unstructured":"[20] Lee, K. , Smith, L. M. and Abbeel, P. , \u201cPEBBLE: Feedback-Efficient Interactive Reinforcement Learning Via Relabeling Experience and Unsupervised Pre-Training,\u201d In: Proceedings of the 38th International Conference on Machine Learning (M. Meila and T. Zhang, eds.) vol. 139 of Proceedings of Machine Learning Research (PMLR, 2021b) pp. 6152\u20136163."},{"key":"S0263574726103300_ref15","doi-asserted-by":"publisher","DOI":"10.1177\/0278364920987859"},{"key":"S0263574726103300_ref14","doi-asserted-by":"publisher","DOI":"10.52202\/075280-2133"},{"key":"S0263574726103300_ref29","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton","year":"2018"},{"key":"S0263574726103300_ref25","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-03051-4"},{"key":"S0263574726103300_ref35","unstructured":"[35] Yu, T. , Quillen, D. , He, Z. , Julian, R. , Hausman, K. , Finn, C. and Levine, S. , \u201cMeta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning,\u201d In: Conference on Robot Learning (PMLR, 2020) pp. 1094\u20131100."},{"key":"S0263574726103300_ref11","unstructured":"[11] Havrilla, A. , Du, Y. , Raparthy, S. C. , Nalmpantis, C. , Dwivedi-Yu, J. , Hambro, E. , Sukhbaatar, S. and Raileanu, R. , \u201cTeaching Large Language Models to Reason with Reinforcement Learning,\u201d In: AI for Math Workshop @ ICML 2024, OpenReview.net (2024)."},{"key":"S0263574726103300_ref16","doi-asserted-by":"crossref","unstructured":"[16] Kadokawa, Y. , Frey, J. , Miki, T. , Matsubara, T. and Hutter, M. , \u201cDAPPER: Discriminability-aware policy-to-policy preference-based reinforcement learning for query-efficient robot skill acquisition,\u201d (2025). arXiv preprint arXiv:2505.06357.","DOI":"10.1109\/MRA.2026.3650847"},{"key":"S0263574726103300_ref26","doi-asserted-by":"publisher","DOI":"10.1038\/nature24270"},{"key":"S0263574726103300_ref17","doi-asserted-by":"publisher","DOI":"10.1109\/IROS55552.2023.10342081"}],"container-title":["Robotica"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S0263574726103300","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,11]],"date-time":"2026-05-11T03:31:32Z","timestamp":1778470292000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S0263574726103300\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2]]},"references-count":38,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,2]]}},"alternative-id":["S0263574726103300"],"URL":"https:\/\/doi.org\/10.1017\/s0263574726103300","relation":{},"ISSN":["0263-5747","1469-8668"],"issn-type":[{"value":"0263-5747","type":"print"},{"value":"1469-8668","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2]]}}}