{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T18:15:46Z","timestamp":1779387346811,"version":"3.53.1"},"reference-count":45,"publisher":"Institution of Engineering and Technology (IET)","issue":"2","license":[{"start":{"date-parts":[[2026,1,28]],"date-time":"2026-01-28T00:00:00Z","timestamp":1769558400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2026,1,28]],"date-time":"2026-01-28T00:00:00Z","timestamp":1769558400000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62473052"],"award-info":[{"award-number":["62473052"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["ietresearch.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["CAAI Trans on Intel Tech"],"published-print":{"date-parts":[[2026,4]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:p>\n                    Efficient exploration is critical in handling sparse rewards and partial observability in deep reinforcement learning. However, most existing intrinsic reward methods based on novelty rely on single\u2010step observations or Euclidean distances. These approaches struggle to capture trajectory\u2010level novelty and often perform poorly in partially observable settings. Moreover, they typically ignore the role of actions in driving observation changes, as not all actions lead to meaningful state transitions. To overcome these limitations, we propose a trajectory\u2010level novelty measure that estimates the novelty of a state by comparing current observations with past ones along the trajectory. To focus on meaningful exploration, we incorporate the mutual information between actions and trajectory novelty to filter out random fluctuations and retain only novelty caused by the agent's actions. Additionally, we introduce a first\u2010visit constraint on observation\u2013action pairs, rewarding only interactions that result in state transitions to enhance exploration efficiency. We conducted experiments in the MiniGrid\u2010ObstructedMaze environment characterised by complex object interactions and sparse rewards. Results demonstrate that our method achieves state\u2010of\u2010the\u2010art performance in convergence speed and average returns. Furthermore, it shows strong generalisation on high\u2010dimensional Atari benchmarks and demonstrates robust performance in more challenging MiniGrid variants. Implementation code is available at:\n                    <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/github.com\/MurrayMa0816\/TNCOA\">https:\/\/github.com\/MurrayMa0816\/TNCOA<\/jats:ext-link>\n                    .\n                  <\/jats:p>","DOI":"10.1049\/cit2.70100","type":"journal-article","created":{"date-parts":[[2026,1,29]],"date-time":"2026-01-29T03:41:10Z","timestamp":1769658070000},"page":"411-427","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["TNCOA: Efficient Exploration via Observation\u2010Action Constraint on Trajectory\u2010Based Intrinsic Reward"],"prefix":"10.1049","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-2139-4102","authenticated-orcid":false,"given":"Jingxiang","family":"Ma","sequence":"first","affiliation":[{"name":"School of Automation Beijing Institute of Technology  Beijing China"},{"name":"State Key Laboratory of Autonomous Intelligent Unmanned System  Beijing China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5734-3157","authenticated-orcid":false,"given":"Hongbin","family":"Ma","sequence":"additional","affiliation":[{"name":"School of Automation Beijing Institute of Technology  Beijing China"},{"name":"State Key Laboratory of Autonomous Intelligent Unmanned System  Beijing China"},{"name":"Beijing Institute of Technology  Zhuhai China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2984-734X","authenticated-orcid":false,"given":"Youzhi","family":"Zhang","sequence":"additional","affiliation":[{"name":"Centre for Artificial Intelligence and Robotics Hong Kong Institute of Science &amp; Innovation Chinese Academy of Sciences  Hong Kong China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"265","published-online":{"date-parts":[[2026,1,28]]},"reference":[{"key":"e_1_2_14_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS45743.2020.9341714"},{"key":"e_1_2_14_3_1","doi-asserted-by":"publisher","DOI":"10.1049\/trit.2020.0024"},{"key":"e_1_2_14_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA40945.2020.9196958"},{"key":"e_1_2_14_5_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41586\u2010023\u201005732\u20102"},{"key":"e_1_2_14_6_1","doi-asserted-by":"publisher","DOI":"10.1049\/cit2.12394"},{"key":"e_1_2_14_7_1","first-page":"2469","volume-title":"Proceedings of the 35th International Conference on Machine Learning (ICML)","author":"Kang B.","year":"2018"},{"key":"e_1_2_14_8_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2003.04960"},{"key":"e_1_2_14_9_1","doi-asserted-by":"publisher","DOI":"10.1049\/cit2.12025"},{"key":"e_1_2_14_10_1","article-title":"Concrete Problems in AI Safety","author":"Amodei D.","year":"2016","journal-title":"arXiv preprint arXiv:1606.06565"},{"key":"e_1_2_14_11_1","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton R. S.","year":"2018"},{"key":"e_1_2_14_12_1","doi-asserted-by":"publisher","DOI":"10.1561\/2200000070"},{"key":"e_1_2_14_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/tnnls.2023.3236361"},{"key":"e_1_2_14_14_1","first-page":"11834","volume-title":"Proceedings of the 34th Conference on Neural Information Processing Systems","author":"Guo Y.","year":"2020"},{"key":"e_1_2_14_15_1","first-page":"1","volume-title":"Proceedings of the 7th Conference on Robot Learning (CoRL)","author":"Gieselmann R.","year":"2023"},{"key":"e_1_2_14_16_1","volume-title":"Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS)","author":"Parisi S.","year":"2024"},{"key":"e_1_2_14_17_1","first-page":"60414","volume-title":"Proceedings of the 41st International Conference on Machine Learning (ICML)","author":"Zhang J.","year":"2024"},{"key":"e_1_2_14_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2017.70"},{"key":"e_1_2_14_19_1","volume-title":"Proceedings of the 7th International Conference on Learning Representations (ICLR)","author":"Burda Y.","year":"2019"},{"key":"e_1_2_14_20_1","volume-title":"Proceedings of the 8th International Conference on Learning Representations (ICLR)","author":"Raileanu R.","year":"2020"},{"key":"e_1_2_14_21_1","volume-title":"Proceedings of the 9th International Conference on Learning Representations (ICLR)","author":"Flet\u2010Berliac Y.","year":"2021"},{"key":"e_1_2_14_22_1","volume-title":"Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS)","author":"Yuan M.","year":"2022"},{"key":"e_1_2_14_23_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2023\/477"},{"key":"e_1_2_14_24_1","first-page":"4582","volume-title":"Proceedings of the 42nd International Conference on Machine Learning (ICML)","author":"Ma H.","year":"2025"},{"key":"e_1_2_14_25_1","first-page":"34219","volume-title":"Proceedings of the 41st International Conference on Machine Learning (ICML)","author":"Mahankali S.","year":"2024"},{"key":"e_1_2_14_26_1","volume-title":"Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS)","author":"Wang Y.","year":"2024"},{"key":"e_1_2_14_27_1","first-page":"25217","volume-title":"Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS)","author":"Zhang T.","year":"2021"},{"key":"e_1_2_14_28_1","first-page":"1081","volume-title":"Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS)","author":"Tang H.","year":"2017"},{"key":"e_1_2_14_29_1","first-page":"1070","volume-title":"Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS)","author":"Fu J.","year":"2017"},{"key":"e_1_2_14_30_1","volume-title":"Proceedings of the 6th International Conference on Learning Representations (ICLR)","author":"Choi J.","year":"2018"},{"key":"e_1_2_14_31_1","volume-title":"Proceedings of the 7th International Conference on Learning Representations (ICLR)","author":"Badia A. P.","year":"2019"},{"key":"e_1_2_14_32_1","volume-title":"Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS)","author":"Trott A.","year":"2019"},{"key":"e_1_2_14_33_1","unstructured":"M.Chevalier\u2010Boisvert L.Willems andS.Pal Minimalistic Grid World Environment for Openai Gym(2018) https:\/\/github.com\/maximecb\/gym\u2010minigrid."},{"key":"e_1_2_14_34_1","first-page":"10405","volume-title":"Proceedings of the 40th International Conference on Machine Learning (ICML)","author":"Fu Y.","year":"2023"},{"key":"e_1_2_14_35_1","volume-title":"Proceedings of the 12th International Conference on Learning Representations (ICLR)","author":"Wu Z.","year":"2024"},{"key":"e_1_2_14_36_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v38i15.29660"},{"key":"e_1_2_14_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcss.2007.08.009"},{"key":"e_1_2_14_38_1","first-page":"2845","volume-title":"Proceedings of the 30th Conference on Neural Information Processing Systems (NeurIPS)","author":"Bellemare M.","year":"2016"},{"key":"e_1_2_14_39_1","first-page":"2721","volume-title":"Proceedings of the 34th International Conference on Machine Learning (ICML)","author":"Ostrovski G.","year":"2017"},{"key":"e_1_2_14_40_1","first-page":"2950","volume-title":"Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI)","author":"Seurin M.","year":"2021"},{"key":"e_1_2_14_41_1","volume-title":"Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS)","author":"Wang Y.","year":"2023"},{"key":"e_1_2_14_42_1","doi-asserted-by":"publisher","DOI":"10.52202\/068431-2206"},{"key":"e_1_2_14_43_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.2567"},{"key":"e_1_2_14_44_1","article-title":"Proximal Policy Optimization Algorithms","author":"Schulman J.","year":"2017","journal-title":"arXiv preprint arXiv:1707.06347"},{"key":"e_1_2_14_45_1","volume-title":"Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS)","author":"Zhang T.","year":"2021"},{"key":"e_1_2_14_46_1","first-page":"40531","volume-title":"Proceedings of the 40th International Conference on Machine Learning (ICML)","author":"Yuan M.","year":"2023"}],"container-title":["CAAI Transactions on Intelligence Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ietresearch.onlinelibrary.wiley.com\/doi\/pdf\/10.1049\/cit2.70100","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/ietresearch.onlinelibrary.wiley.com\/doi\/full-xml\/10.1049\/cit2.70100","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/ietresearch.onlinelibrary.wiley.com\/doi\/pdf\/10.1049\/cit2.70100","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T17:56:50Z","timestamp":1779386210000},"score":1,"resource":{"primary":{"URL":"https:\/\/ietresearch.onlinelibrary.wiley.com\/doi\/10.1049\/cit2.70100"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,28]]},"references-count":45,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,4]]}},"alternative-id":["10.1049\/cit2.70100"],"URL":"https:\/\/doi.org\/10.1049\/cit2.70100","archive":["Portico"],"relation":{},"ISSN":["2468-6557","2468-2322"],"issn-type":[{"value":"2468-6557","type":"print"},{"value":"2468-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,28]]},"assertion":[{"value":"2025-06-14","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-29","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-01-28","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}