{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T16:05:04Z","timestamp":1781366704512,"version":"3.54.1"},"reference-count":59,"publisher":"Association for Computing Machinery (ACM)","issue":"8","funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62372021"],"award-info":[{"award-number":["62372021"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100012166","name":"National Key R&D Program of China","doi-asserted-by":"crossref","award":["2024YFB3311503"],"award-info":[{"award-number":["2024YFB3311503"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,11,30]]},"abstract":"<jats:p>\n            Deep reinforcement learning systems have been increasingly applied in various domains. Testing them, however, remains a major open research problem. Mutation testing is a popular test suite evaluation technique that analyzes the extent to which test suites detect injected faults. It has been widely researched in both traditional software and the field of deep learning. However, due to the fundamental differences between deep reinforcement learning systems and traditional software, as well as deep learning systems, in aspects such as environment interaction, network decision-making, and data efficiency, previous mutation testing techniques cannot be directly applied to deep reinforcement learning systems. In this article, we proposed a comprehensive mutation testing framework specifically designed for deep reinforcement learning systems,\n            <jats:monospace>DRLMutation<\/jats:monospace>\n            , to further fill this gap. We first considered the characteristics of deep reinforcement learning, and based on both the training process and the model of trained agent, examined combinations from three dimensions: objects, operation methods, and injection methods. This approach led to a more comprehensive design methodology for deep reinforcement learning mutation operators. After filtering, we identified a total of 107 applicable deep reinforcement learning mutation operators. Then, in the realm of evaluation, we formulated a set of metrics tailored to assess test suites. Finally, we validated the stealthiness and effectiveness of the proposed mutation operators in the\n            <jats:italic toggle=\"yes\">Cart Pole<\/jats:italic>\n            ,\n            <jats:italic toggle=\"yes\">Mountain Car Continuous<\/jats:italic>\n            ,\n            <jats:italic toggle=\"yes\">Lunar Lander<\/jats:italic>\n            ,\n            <jats:italic toggle=\"yes\">Breakout<\/jats:italic>\n            , and\n            <jats:italic toggle=\"yes\">CARLA<\/jats:italic>\n            environments. We show inspiring findings that the majority of these designed deep reinforcement learning mutation operators potentially undermine the decision-making capabilities of the agent without affecting normal training. The varying degrees of disruption achieved by these mutation operators can be used to assess the quality of different test suites.\n          <\/jats:p>","DOI":"10.1145\/3721978","type":"journal-article","created":{"date-parts":[[2025,3,7]],"date-time":"2025-03-07T15:57:21Z","timestamp":1741363041000},"page":"1-32","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["DRLMutation: A Comprehensive Framework for Mutation Testing in Deep Reinforcement Learning Systems"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6499-3852","authenticated-orcid":false,"given":"Jiapeng","family":"Li","sequence":"first","affiliation":[{"name":"School of Automation Science and Electrical Engineering, Beihang University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7922-9067","authenticated-orcid":false,"given":"Zheng","family":"Zheng","sequence":"additional","affiliation":[{"name":"School of Automation Science and Electrical Engineering, Beihang University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1609-0480","authenticated-orcid":false,"given":"Xiaoting","family":"Du","sequence":"additional","affiliation":[{"name":"School of Computer, Beijing University of Posts and Telecommunications, Beijing, China  and State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-1648-1556","authenticated-orcid":false,"given":"Haoyu","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Automation Science and Electrical Engineering, Beihang University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-6646-6305","authenticated-orcid":false,"given":"Yanwen","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Automation Science and Electrical Engineering, Beihang University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,10,6]]},"reference":[{"key":"e_1_3_2_2_2","volume-title":"On Mutation","author":"Acree Jr Allen Troy","year":"1980","unstructured":"Allen Troy Acree Jr. 1980. On Mutation. Georgia Institute of Technology."},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2023.110289"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSMC.1983.6313077"},{"key":"e_1_3_2_5_2","unstructured":"Vahid Behzadan and Arslan Munir. 2018. The faults in our pi stars: Security issues and open challenges in deep reinforcement learning. arXiv:1810.10369. Retrieved from https:\/\/arxiv.org\/abs\/1810.10369"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.5555\/909408"},{"key":"e_1_3_2_7_2","first-page":"187","volume-title":"Proceedings of the 2005 IEEE 5th International Conference on Quality Software","author":"Chan Wing-Kwong","year":"2005","unstructured":"Wing-Kwong Chan, Shing-Chi Cheung, and T. H. Tse. 2005. Fault-based testing of database application programs with conceptual data model. In Proceedings of the 2005 IEEE 5th International Conference on Quality Software. IEEE, 187\u2013196."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2023.3285442"},{"key":"e_1_3_2_9_2","first-page":"1052","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Chen Xinshi","year":"2019","unstructured":"Xinshi Chen, Shuang Li, Hui Li, Shaohua Jiang, Yuan Qi, and Le Song. 2019. Generative adversarial user model for reinforcement learning based recommendation system. In Proceedings of the International Conference on Machine Learning. PMLR, 1052\u20131061."},{"key":"e_1_3_2_10_2","first-page":"313","volume-title":"Proceedings of the 2018 IEEE International Conference on Software Quality, Reliability and Security","author":"Cheng Dawei","year":"2018","unstructured":"Dawei Cheng, Chun Cao, Chang Xu, and Xiaoxing Ma. 2018. Manifesting bugs in machine learning code: An explorative study with mutation testing. In Proceedings of the 2018 IEEE International Conference on Software Quality, Reliability and Security. IEEE, 313\u2013324."},{"key":"e_1_3_2_11_2","first-page":"79","article-title":"Proteum-A tool for the assessment of test adequacy for C programs user\u2019s guide","volume":"96","author":"Delamaro M\u00e1rcio E.","year":"1996","unstructured":"M\u00e1rcio E. Delamaro, Jos\u00e9 C. Maldonado, and Aditya Mathur. 1996. Proteum-A tool for the assessment of test adequacy for C programs user\u2019s guide. In Proceedings of the Conference on Performability in Computing Systems (PCS), Vol. 96, 79\u201395.","journal-title":"Proceedings of the Conference on Performability in Computing Systems (PCS)"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1728"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/C-M.1978.218136"},{"key":"e_1_3_2_14_2","first-page":"227","volume-title":"Proceedings of the 2006 IEEE 6th International Conference on Quality Software","author":"Derezinska Anna","year":"2006","unstructured":"Anna Derezinska. 2006. Quality assessment of mutation operators dedicated for C# programs. In Proceedings of the 2006 IEEE 6th International Conference on Quality Software. IEEE, 227\u2013234."},{"key":"e_1_3_2_15_2","first-page":"283","volume-title":"Software Engineering Techniques: Design for Quality","author":"Derezi\u0144ska Anna","year":"2007","unstructured":"Anna Derezi\u0144ska. 2007. Advanced mutation operators applicable in C# programs. In Software Engineering Techniques: Design for Quality. Springer, 283\u2013288."},{"key":"e_1_3_2_16_2","unstructured":"Prafulla Dhariwal Christopher Hesse Oleg Klimov Alex Nichol Matthias Plappert Alec Radford John Schulman Szymon Sidor Yuhuai Wu and Peter Zhokhov. 2017. OpenAI baselines. Retrieved from https:\/\/github.com\/openai\/baselines"},{"key":"e_1_3_2_17_2","first-page":"1","volume-title":"Proceedings of the Conference on Robot Learning","author":"Dosovitskiy Alexey","year":"2017","unstructured":"Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An open urban driving simulator. In Proceedings of the Conference on Robot Learning. PMLR, 1\u201316."},{"key":"e_1_3_2_18_2","first-page":"3145","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Filos Angelos","year":"2020","unstructured":"Angelos Filos, Panagiotis Tigkas, Rowan McAllister, Nicholas Rhinehart, Sergey Levine, and Yarin Gal. 2020. Can autonomous vehicles identify, recover from, and adapt to distribution shifts? In Proceedings of the International Conference on Machine Learning. PMLR, 3145\u20133153."},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.asr.2019.12.030"},{"key":"e_1_3_2_20_2","first-page":"1792","volume-title":"Proceedings of the International Conference on Machine Learning. PMLR","author":"Greydanus Samuel","year":"2018","unstructured":"Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. 2018. Visualizing and understanding Atari agents. In Proceedings of the International Conference on Machine Learning. PMLR, 1792\u20131801."},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1977.231145"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2020.3008612"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2023.3251376"},{"key":"e_1_3_2_24_2","first-page":"1158","volume-title":"Proceedings of the 2019 IEEE 34th International Conference on Automated Software Engineering","author":"Hu Qiang","year":"2019","unstructured":"Qiang Hu, Lei Ma, Xiaofei Xie, Bing Yu, Yang Liu, and Jianjun Zhao. 2019. DeepMutation++: A mutation testing framework for deep learning systems. In Proceedings of the 2019 IEEE 34th International Conference on Automated Software Engineering. IEEE, 1158\u20131161."},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2023.3254579"},{"key":"e_1_3_2_26_2","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1145\/3460319.3464825","volume-title":"Proceedings of the 2021 ACM 30th SIGSOFT International Symposium on Software Testing and Analysis","author":"Humbatova Nargiz","year":"2021","unstructured":"Nargiz Humbatova, Gunel Jahangirova, and Paolo Tonella. 2021. DeepCrime: Mutation testing of deep learning systems based on real faults. In Proceedings of the 2021 ACM 30th SIGSOFT International Symposium on Software Testing and Analysis, 67\u201378."},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.62"},{"key":"e_1_3_2_28_2","article-title":"The rigorous generation of Java mutation operators using HAZOP","author":"Kim Sunwoo","year":"1999","unstructured":"Sunwoo Kim, John A. Clark, and John McDermid. 1999. The rigorous generation of Java mutation operators using HAZOP. Informe T\u00e9cnico, The University of York.","journal-title":"Informe T\u00e9cnico, The University of York."},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1002\/spe.4380210704"},{"key":"e_1_3_2_30_2","first-page":"36","volume-title":"Proceedings of the 2011 AAAI Spring Symposium Series","author":"Knox W. B.","year":"2011","unstructured":"W. B. Knox, Adam B. Setapen, and Peter Stone. 2011. Reinforcement learning with human feedback in mountain car. In Proceedings of the 2011 AAAI Spring Symposium Series, 36\u201341."},{"key":"e_1_3_2_31_2","first-page":"200","volume-title":"Proceedings of the 2001 IEEE 12th International Symposium on Software Reliability Engineering","author":"Lee Suet C.","year":"2001","unstructured":"Suet C. Lee and Jeff Offutt. 2001. Generating test cases for XML-based Web component interactions using mutation analysis. In Proceedings of the 2001 IEEE 12th International Symposium on Software Reliability Engineering. IEEE, 200\u2013209."},{"key":"e_1_3_2_32_2","unstructured":"Chengjie Lu Tao Yue Man Zhang and Shaukat Ali. 2023. DeepQTest: Testing autonomous driving systems with reinforcement learning and real-world weather data. arXiv:2310.05170. Retrieved from https:\/\/arxiv.org\/abs\/2310.05170"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.sysarc.2022.102701"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2018.00021"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.308"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2946563"},{"key":"e_1_3_2_37_2","first-page":"153","volume-title":"Proceedings of the 2011 IEEE 4th International Conference on Software Testing, Verification, and Validation Workshops","author":"McMinn Phil","year":"2011","unstructured":"Phil McMinn. 2011. Search-based software testing: Past, present and future. In Proceedings of the 2011 IEEE 4th International Conference on Software Testing, Verification, and Validation Workshops. IEEE, 153\u2013163."},{"key":"e_1_3_2_38_2","unstructured":"Volodymyr Mnih Koray Kavukcuoglu David Silver Alex Graves Ioannis Antonoglou Daan Wierstra and Martin Riedmiller. 2013. Playing Atari with deep reinforcement learning. arXiv:1312.5602. Retrieved from https:\/\/arxiv.org\/abs\/1312.5602"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature14236"},{"key":"e_1_3_2_40_2","unstructured":"Andrew W. Moore. 1990. Efficient Memory-Based Learning for Robot Control. Technical Report. University of Cambridge Computer Laboratory."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.5555\/1308173.1308248"},{"key":"e_1_3_2_42_2","first-page":"26","volume-title":"Proceedings of the 2017 IEEE International Conference on Advances in Computing, Communications and Applied Informatics","author":"Nagendra Savinay","year":"2017","unstructured":"Savinay Nagendra, Nikhil Podila, Rashmi Ugarakhod, and Koshy George. 2017. Comparison of reinforcement learning algorithms applied to the cart-pole problem. In Proceedings of the 2017 IEEE International Conference on Advances in Computing, Communications and Applied Informatics. IEEE, 26\u201332."},{"issue":"1","key":"e_1_3_2_43_2","first-page":"896","article-title":"Digital twin for transportation big data: A reinforcement learning-based network traffic prediction approach","volume":"25","author":"Nie Laisen","year":"2023","unstructured":"Laisen Nie, Xiaojie Wang, Qinglin Zhao, Zhigang Shang, Li Feng, and Guojun Li. 2023. Digital twin for transportation big data: A reinforcement learning-based network traffic prediction approach. IEEE Transactions on Intelligent Transportation Systems 25, 1 (2023), 896\u2013906.","journal-title":"IEEE Transactions on Intelligent Transportation Systems"},{"issue":"8","key":"e_1_3_2_44_2","first-page":"1","article-title":"Faults in deep reinforcement learning programs: A taxonomy and a detection approach","volume":"29","author":"Nikanjam Amin","year":"2022","unstructured":"Amin Nikanjam, Mohammad M. Morovati, Foutse Khomh, and Ben B. Houssem. 2022. Faults in deep reinforcement learning programs: A taxonomy and a detection approach. Automated Software Engineering 29, 8 (2022), 1\u201332.","journal-title":"Automated Software Engineering"},{"key":"e_1_3_2_45_2","unstructured":"B\u0142a\u017cej Osi\u0144ski Piotr Mi\u0142o\u015b Adam Jakubowski Pawe\u0142 Zi\u0119cina Micha\u0142 Martyniak Christopher Galias Antonia Breuer Silviu Homoceanu and Henryk Michalewski. 2020. CARLA real traffic scenarios\u2013novel training ground and benchmark for autonomous driving. arXiv:2012.11329. Retrieved from https:\/\/arxiv.org\/abs\/2012.11329"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534388"},{"key":"e_1_3_2_47_2","first-page":"66","volume-title":"Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering: New Ideas and Emerging Results","author":"Panichella Annibale","year":"2021","unstructured":"Annibale Panichella and Cynthia C. S. Liem. 2021. What are we really testing in mutation testing for machine learning? A critical reflection. In Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering: New Ideas and Emerging Results. IEEE, 66\u201370."},{"key":"e_1_3_2_48_2","first-page":"275","volume-title":"Advances in Computers","author":"Papadakis Mike","year":"2019","unstructured":"Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Mutation testing advances: An analysis and survey. In Advances in Computers, Vol. 112. Elsevier, 275\u2013378."},{"key":"e_1_3_2_49_2","doi-asserted-by":"crossref","first-page":"338","DOI":"10.1109\/ICST.2008.44","volume-title":"Proceedings of the 2008 1st International Conference on Software Testing, Verification, and Validation","author":"Pretschner Alexander","year":"2008","unstructured":"Alexander Pretschner, Tejeddine Mouelhi, and Yves Le Traon. 2008. Model-based tests for access control policies. In Proceedings of the 2008 1st International Conference on Software Testing, Verification, and Validation. IEEE, 338\u2013347."},{"key":"e_1_3_2_50_2","first-page":"297","volume-title":"Proceedings of the 2009 7th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Schuler David","year":"2009","unstructured":"David Schuler and Andreas Zeller. 2009. Javalanche: Efficient mutation testing for Java. In Proceedings of the 2009 7th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 297\u2013298."},{"key":"e_1_3_2_51_2","first-page":"108","volume-title":"Proceedings of the 2018 IEEE International Conference on Software Quality, Reliability and Security Companion","author":"Shen Weijun","year":"2018","unstructured":"Weijun Shen, Jun Wan, and Zhenyu Chen. 2018. MuNN: Mutation analysis of neural networks. In Proceedings of the 2018 IEEE International Conference on Software Quality, Reliability and Security Companion. IEEE, 108\u2013115."},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2024.112016"},{"key":"e_1_3_2_53_2","first-page":"80","volume-title":"Proceedings of the 7th Annual Joint Conference of the IEEE Computer and Communications Societies. Networks: Evolution or Revolution? (IEEE INFOCOM \u201988)","author":"Sidhu Deepinder","year":"1988","unstructured":"Deepinder Sidhu and T-K Leung. 1988. Fault coverage of protocol test methods. In Proceedings of the 7th Annual Joint Conference of the IEEE Computer and Communications Societies. Networks: Evolution or Revolution? (IEEE INFOCOM \u201988) IEEE, 80\u201385."},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-008-9083-7"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2024.111963"},{"key":"e_1_3_2_56_2","first-page":"706","volume-title":"Proceedings of the 2023 IEEE 34th International Symposium on Software Reliability Engineering","author":"Wang Kairui","year":"2023","unstructured":"Kairui Wang, Yawen Wang, Junjie Wang, and Qing Wang. 2023. Fuzzing with sequence diversity inference for sequential decision-making model testing. In Proceedings of the 2023 IEEE 34th International Symposium on Software Reliability Engineering. IEEE, 706\u2013717."},{"key":"e_1_3_2_57_2","unstructured":"Tianpei Yang Hongyao Tang Chenjia Bai Jinyi Liu Jianye Hao Zhaopeng Meng and Peng Liu. 2021. Exploration in deep reinforcement learning: A comprehensive survey. arXiv:2109.06668. Retrieved from https:\/\/arxiv.org\/abs\/2109.06668"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2962027"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3512345"},{"issue":"7","key":"e_1_3_2_60_2","first-page":"6030","article-title":"A search-based testing approach for deep reinforcement learning agents","volume":"49","author":"Zolfagharian Amirhossein","year":"2023","unstructured":"Amirhossein Zolfagharian, Manel Abdellatif, Lionel C. Briand, Mojtaba Bagherzadeh, and S. Ramesh. 2023. A search-based testing approach for deep reinforcement learning agents. IEEE Transactions on Software Engineering 49, 7 (2023), 6030\u20136041.","journal-title":"IEEE Transactions on Software Engineering"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3721978","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,6]],"date-time":"2025-10-06T13:43:18Z","timestamp":1759758198000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3721978"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,6]]},"references-count":59,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2025,11,30]]}},"alternative-id":["10.1145\/3721978"],"URL":"https:\/\/doi.org\/10.1145\/3721978","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,6]]},"assertion":[{"value":"2023-12-23","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-22","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-06","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}