{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T20:29:56Z","timestamp":1792787396064,"version":"4.5.0"},"reference-count":70,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2022,12,3]],"date-time":"2022-12-03T00:00:00Z","timestamp":1670025600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"TAPAS","award":["H2020-SESAR-2019-2, 892358"],"award-info":[{"award-number":["H2020-SESAR-2019-2, 892358"]}]},{"name":"Towards an Automated and exPlainable Air traffic management (ATM) System"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2023,5,31]]},"abstract":"<jats:p>Interpretability, explainability, and transparency are key issues to introducing artificial intelligence methods in many critical domains. This is important due to ethical concerns and trust issues strongly connected to reliability, robustness, auditability, and fairness, and has important consequences toward keeping the human in the loop in high levels of automation, especially in critical cases for decision making, where both (human and the machine) play important roles. Although the research community has given much attention to explainability of closed (or black) prediction boxes, there are tremendous needs for explainability of closed-box methods that support agents to act autonomously in the real world. Reinforcement learning methods, and especially their deep versions, are such closed-box methods. In this article, we aim to provide a review of state-of-the-art methods for explainable deep reinforcement learning methods, taking also into account the needs of human operators\u2014that is, of those who make the actual and critical decisions in solving real-world problems. We provide a formal specification of the deep reinforcement learning explainability problems, and we identify the necessary components of a general explainable reinforcement learning framework. Based on these, we provide a comprehensive review of state-of-the-art methods, categorizing them into classes according to the paradigm they follow, the interpretable models they use, and the surface representation of explanations provided. The article concludes by identifying open questions and important challenges.<\/jats:p>","DOI":"10.1145\/3527448","type":"journal-article","created":{"date-parts":[[2022,3,31]],"date-time":"2022-03-31T08:01:16Z","timestamp":1648713676000},"page":"1-39","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":164,"title":["Explainable Deep Reinforcement Learning: State of the Art and Challenges"],"prefix":"10.1145","volume":"55","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5451-622X","authenticated-orcid":false,"given":"George A.","family":"Vouros","sequence":"first","affiliation":[{"name":"University of Piraeus, Piraeus, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,12,3]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"crossref","unstructured":"E. Puiutta and E. M. S. P. Veith. 2020. Explainable Reinforcement Learning: A Survey . arXiv:2005.06247 (2020).","DOI":"10.1007\/978-3-030-57321-8_5"},{"key":"e_1_3_2_3_2","doi-asserted-by":"crossref","unstructured":"M. T. Ribeiro S. Singh and C. Guestrin. 2016. Why should I trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD KDD . ACM New York NY 1135\u20131144.","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_3_2_4_2","doi-asserted-by":"crossref","unstructured":"R. Iyer Y. Li H. Li M. Lewis R. Sundar and K. Sycara. 2018. Transparency and explanation in deep reinforcement learning neural networks. In Proceedings of the 2018 AAAI\/ACM Conference on AI Ethics and Society.","DOI":"10.1145\/3278721.3278776"},{"key":"e_1_3_2_5_2","doi-asserted-by":"crossref","unstructured":"R. Pocius L. Neal and A. Fern. 2019. Strategic tasks for explainable reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence 33 1 (2019) AAAI-19 IAAI-19 EAAI-20.","DOI":"10.1609\/aaai.v33i01.330110007"},{"key":"e_1_3_2_6_2","unstructured":"W. Shi S. Song Z. Wang and G. Huang. 2020. Self-supervised discovering of causal features: Towards interpretable reinforcement learning. arXiv:2003.07069v2 (2020)."},{"key":"e_1_3_2_7_2","unstructured":"Z. Juozapaitis A. Koul A. Fern M. Erwig and F. Doshi-Velez. 2019. Explainable reinforcement learning via reward decomposition. In Proceedings of the IJCAI\/ECAI Workshop on Explainable Artificial Intelligence ."},{"key":"e_1_3_2_8_2","doi-asserted-by":"crossref","unstructured":"N. Topin and M. Veloso. 2019. Generation of policy-level explanations for reinforcement learning. arXiv:1905.12044 (2019).","DOI":"10.1609\/aaai.v33i01.33012514"},{"key":"e_1_3_2_9_2","unstructured":"J. Goldberger G. E. Hinton S. T. Roweis and R. R. Salakhutdinov. 2004. Neighbourhood components analysis. In Advances in Neural Information Processing Systems 17 (NIPS\u201904) . 513\u2013520."},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","unstructured":"F. Cruz R. Dazeley and P. Vamplew. 2019. Memory-based explainable reinforcement learning. In AI 2019: Advances in Artificial Intelligence . Lecture Notes in Computer Science Vol. 11919. Springer 66\u201377.","DOI":"10.1007\/978-3-030-35288-2_6"},{"key":"e_1_3_2_11_2","unstructured":"T. Leech. 2019. Explainable machine learning for task planning in robotics. Master\u2019s Thesis. Massachusetts Institute of Technology Cambridge MA."},{"key":"e_1_3_2_12_2","doi-asserted-by":"crossref","unstructured":"D. Dancey Z. A. Bandar and D. McLean. 2007. Logistic model tree extraction from artificial neural networks. IEEE Transactions on Systems Man and Cybernetics: Part B 37 4 (2007) 794\u2013802.","DOI":"10.1109\/TSMCB.2007.895334"},{"key":"e_1_3_2_13_2","doi-asserted-by":"crossref","unstructured":"G. Liu O. Schulte W. Zhu and Q. Li. 2018. Toward interpretable deep reinforcement learning with linear model U-trees. arXiv:1807.05887 (2018).","DOI":"10.1007\/978-3-030-10928-8_25"},{"key":"e_1_3_2_14_2","unstructured":"Z. Che S. Purushotham R. Khemani and Y. Liu. 2016. Interpretable deep models for ICU outcome prediction. In Proceedings of the AMIA Annual Symposium ."},{"key":"e_1_3_2_15_2","unstructured":"N. Wang D. V. Pynadath and S. G. Hill. 2016. The impact of POMDP-generated explanations on trust and performance in human-robot teams. In Proceedings of the 2016 International Conference on Autonomous Agents and Multiagent Systems (AAMAS\u201916) . 997\u20131005."},{"key":"e_1_3_2_16_2","unstructured":"D. V. Pynadath and S. C. Marsella. 2004. Fitting and compilation of multiagent models through piecewise linear functions. In Proceedings of the 3rd International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS\u201904) . 1197\u20131204."},{"key":"e_1_3_2_17_2","unstructured":"P. Madumal T. Miller L. Sonenberg and F. Vetere. 2019. Explainable reinforcement learning through a causal lens. arXiv:1905.10958 (2019)."},{"key":"e_1_3_2_18_2","doi-asserted-by":"crossref","unstructured":"J. Y. Halpern and J. Pearl. 2005. Causes and explanations: A structural-model approach. Part I: Causes. British Journal for the Philosophy of Science 56 4 (2005) 843\u2013887.","DOI":"10.1093\/bjps\/axi147"},{"key":"e_1_3_2_19_2","unstructured":"Y. Coppens K. Efthymiadis T. Lenaerts A. Now\u00e9 T. Miller R. Weber and D. Magazzeni. 2019. Distilling deep reinforcement learning policies in soft decision trees. In Proceedings of the 2019 IJCAI Workshop on Explainable Artificial Intelligence ."},{"key":"e_1_3_2_20_2","unstructured":"N. Frosst and G. Hinton. 2017. Distilling a neural network into a soft decision tree. In Proceedings of the 1st International Workshop on Comprehensibility and Explanation in AI and ML Vol. 2071 of the AI*IA Series at CEUR Workshop Proceedings ."},{"key":"e_1_3_2_21_2","unstructured":"P. Madumal T. Miller L. Sonenberg and F. Vetere. 2020. Distal explanations for explainable RL agents. arXiv: 2001.10284 (2020)."},{"key":"e_1_3_2_22_2","unstructured":"A. M. Roth N. Topin P. Jamshidi and M. Veloso. 2019. Conservative Q-improvement: Reinforcement learning for an interpretable decision-tree policy. arXiv:1907.01180 (2019)."},{"key":"e_1_3_2_23_2","doi-asserted-by":"crossref","unstructured":"Y. Fukuchi M. Osawa H. Yamakawa and M. Imai. 2017. Autonomous self-explanation of behavior for interactive reinforcement learning agents. In Proceedings of the 5th International Conference on Human-Agent Interaction .","DOI":"10.1145\/3125739.3125746"},{"key":"e_1_3_2_24_2","doi-asserted-by":"crossref","unstructured":"B. Hayes and J. A. Shah. 2017. Improving robot controller transparency through autonomous policy explanation. In Proceedings of the 12th ACM\/IEEE International Conference on Human-Robot Interaction .","DOI":"10.1145\/2909824.3020233"},{"key":"e_1_3_2_25_2","unstructured":"S. Mohseni N. Zarei and E. D. Ragan. 2018. A multidisciplinary survey and framework for design and evaluation of explainable AI systems. arXiv:1811.11839 (2018)."},{"key":"e_1_3_2_26_2","doi-asserted-by":"crossref","unstructured":"R. Guidotti A. Monreale S. Ruggieri F. Turini F. Giannotti and D. Pedreschi. 2018. A survey of methods for explaining black box models. ACM Computing Surveys 51 5 (2018) 1\u201342.","DOI":"10.1145\/3236009"},{"key":"e_1_3_2_27_2","unstructured":"S. M. Lundberg and S. Lee. 2017. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS\u201917) . 4768\u20134777."},{"key":"e_1_3_2_28_2","unstructured":"A. Verma V. Murali R. Singh P. Kohli and S. Chaudhuri. 2018. Programmatically interpretable reinforcement learning. arXiv:1804.02477 (2018)."},{"key":"e_1_3_2_29_2","unstructured":"T. Shu C. Xiong and R. Socher. 2018. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. In Proceedings of the 6th International Conference on Learning Representation (ICLR\u201918) ."},{"key":"e_1_3_2_30_2","doi-asserted-by":"crossref","unstructured":"O. Boz. 2002. Extracting decision trees from trained neural networks. In Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201902) . 456\u2013461.","DOI":"10.1145\/775047.775113"},{"key":"e_1_3_2_31_2","doi-asserted-by":"crossref","unstructured":"S. H. Huang D. Held P. Abbeel and A. D. Dragan. 2019. Enabling robots to communicate their objectives. Autonomous Robots 43 2 (2019) 309\u2013326.","DOI":"10.1007\/s10514-018-9771-0"},{"key":"e_1_3_2_32_2","doi-asserted-by":"crossref","unstructured":"M. T. Ribeiro S. Singh and C. Guestrin. 2018. Anchors: High-precision model-agnostic explanations. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence .","DOI":"10.1609\/aaai.v32i1.11491"},{"key":"e_1_3_2_33_2","unstructured":"R. Guidotti A. Monreale S. Ruggieri D. Pedreschi F. Turini and F. Giannotti. 2018. Local rule-based explanations of black box decision systems. arXiv:1805.10820 (2018)."},{"key":"e_1_3_2_34_2","unstructured":"A. Shrikumar P. Greenside A. Shcherbina and A. Kundaje. 2016. Not just a black box: Learning important features through propagating activation differences. arXiv:1605.01713 (2016)."},{"key":"e_1_3_2_35_2","doi-asserted-by":"crossref","unstructured":"S. Bach A. Binder G. Montavon F. Klauschen K.-R. Muller and W. Samek. 2015. On pixel-wise explanations for non-linear classifier decisions by layerwise relevance propagation. PLoS One 10 7 (2015) e0130140.","DOI":"10.1371\/journal.pone.0130140"},{"key":"e_1_3_2_36_2","unstructured":"L. D. Pyeatt and A. E. Howe. 2001. Decision tree function approximation in reinforcement learning. In Proceedings of the 3rd International Symposium on Adaptive Systems: Evolutionary Computation and Probabilistic Graphical Models."},{"key":"e_1_3_2_37_2","doi-asserted-by":"crossref","unstructured":"A. K. McCallum. 1996. Learning to use selective attention and short-term memory in sequential tasks. In Proceedings of the 4th International Conference on Simulation of Adaptive Behavior (SAB\u201996). 315\u2013325.","DOI":"10.7551\/mitpress\/3118.003.0039"},{"key":"e_1_3_2_38_2","unstructured":"W. T. Uther and M. M. Veloso. 1998. Tree based discretization for continuous state space reinforcement learning. In Proceedings of the 15th National\/10th Conference on Artificial Intelligence\/Innovative Applications on Artificial Intelligence (AAAI\u201998\/IAAI\u201998) . 769\u2013774."},{"key":"e_1_3_2_39_2","doi-asserted-by":"crossref","unstructured":"S. G. Rizzo G. Vantini and S. Chawla. 2019. Reinforcement learning with explainability for traffic signal control. In Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC\u201919) . 3567\u20133572.","DOI":"10.1109\/ITSC.2019.8917519"},{"key":"e_1_3_2_40_2","doi-asserted-by":"crossref","unstructured":"Z. C. Lipton. 2018. The mythos of model interpretability: In machine learning the concept of interpretability is both important and slippery. Queue 16 3 (May\u2013June 2018) 31\u201357.","DOI":"10.1145\/3236386.3241340"},{"key":"e_1_3_2_41_2","unstructured":"O. Biran and C. V. Cotton. 2017. Explanation and justification in machine learning: A survey. In Proceedings of the 2017 IJCAI Workshop on Explainable Artificial Intelligence ."},{"key":"e_1_3_2_42_2","doi-asserted-by":"crossref","unstructured":"M. A. de Graaf B. F. Malle A. Dragan and T. Ziemke. 2018. Explainable robotic systems. In Companion of the ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201918) . 387\u2013388.","DOI":"10.1145\/3173386.3173568"},{"key":"e_1_3_2_43_2","doi-asserted-by":"crossref","unstructured":"T. Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence 267 (2019) 1\u201338.","DOI":"10.1016\/j.artint.2018.07.007"},{"key":"e_1_3_2_44_2","unstructured":"D. Amir and O. Amir. 2018. HIGHLIGHTS: Summarizing agent behavior to people. In Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems (AAMAS\u201918) ."},{"key":"e_1_3_2_45_2","doi-asserted-by":"crossref","unstructured":"P. Sequeira and M. Gervasio. 2020. Interestingness elements for explainable reinforcement learning: Understanding agents\u2019 capabilities and limitations. arXiv:1912.09007v2 (2020).","DOI":"10.1016\/j.artint.2020.103367"},{"key":"e_1_3_2_46_2","unstructured":"A. A. Rusu S. G. Colmenarejo C. Gulcehre G. Desjardins J. Kirkpatrick R. Pascanu V. Mnih K. Kavukcuoglu and R. Hadsell. 2015. Policy distillation. arXiv:1511.06295v2 (2015)."},{"key":"e_1_3_2_47_2","unstructured":"G. Hinton O. Vinyals and J. Dean. 2014. Distilling the knowledge in a neural network. arXiv:1503.02531 (2014)."},{"key":"e_1_3_2_48_2","doi-asserted-by":"crossref","unstructured":"S. H. Huang K. Bhatia. P. Abbeel and A. D. Dragan. 2018. Establishing appropriate trust via critical states. arXiv:1810.08174 (2018).","DOI":"10.1109\/IROS.2018.8593649"},{"key":"e_1_3_2_49_2","unstructured":"J. van der Waa J. van Diggelen K. van den Bosch and M. Neerincx. 2018. Contrastive explanations for reinforcement learning in terms of expected consequences. arXiv:1807.08706 (2018)."},{"key":"e_1_3_2_50_2","unstructured":"National Security Commission on Artificial Intelligence. 2019. Interim Report . National Security Commission on Artificial Intelligence."},{"key":"e_1_3_2_51_2","unstructured":"EU High-Level Expert Group on Artificial Intelligence. 2019. Ethics Guidelines for Trustworthy Artificial Intelligence. Retrieved April 7 2022 from https:\/\/ec.europa.eu\/futurium\/en\/ai-alliance-consultation.1.html."},{"key":"e_1_3_2_52_2","doi-asserted-by":"crossref","unstructured":"R. M. Annasamy and K. Sycara. 2019. Towards better interpretability in deep Q-networks. arXiv:1809.05630 (2019).","DOI":"10.1609\/aaai.v33i01.33014561"},{"key":"e_1_3_2_53_2","unstructured":"A. Mott D. Zoran M. Chrzanowski D. Wierstra and D. J. Rezende. 2019. Towards interpretable reinforcement learning using attention augmented agents. arXiv:1906.02500 (2019)."},{"key":"e_1_3_2_54_2","unstructured":"T. Zahavy N. Ben Zrihem and S. Mannor. 2017. Graying the black box: Understanding DQNs. arXiv:1602.02658 (2017)."},{"key":"e_1_3_2_55_2","unstructured":"O. Bastani Y. Pu and A. Solar-Lezama. 2019. Verifiable reinforcement learning via policy extraction. arXiv:1805.08328 (2019)."},{"key":"e_1_3_2_56_2","doi-asserted-by":"crossref","unstructured":"W. J. Murdoch C. Singh K. Kumbier R. Abbasi-Asl and B. Yu. 2019. Definitions methods and applications in interpretable machine learning. Proceedings of the National Academy of Sciences 116 44 (2019) 22071\u201322080.","DOI":"10.1073\/pnas.1900654116"},{"key":"e_1_3_2_57_2","doi-asserted-by":"crossref","unstructured":"A. Holzinger A. Carrington and H. Meuller. 2020. Measuring the quality of explanations: The system causability scale (SCS). K\u00fcnstliche Intelligenz 34 (2020) 193\u2013198.","DOI":"10.1007\/s13218-020-00636-z"},{"key":"e_1_3_2_58_2","doi-asserted-by":"crossref","unstructured":"C. Rudin C. Chen Z. Chen H. Huang L. Semenova and C. Zhong. 2021. Interpretable machine learning: Fundamental principles and 10 grand challenges. arXiv:2103.11251v2 (2021).","DOI":"10.1214\/21-SS133"},{"key":"e_1_3_2_59_2","doi-asserted-by":"crossref","unstructured":"V. Belle and I. Papantonis. 2020. Principles and practice of explainable machine learning arXiv:2009.11698v1 (2020).","DOI":"10.3389\/fdata.2021.688969"},{"key":"e_1_3_2_60_2","unstructured":"H. Yau C. Russell and S. Hadfield. 2020. What did you think would happen? Explaining agent behaviour through intended outcomes. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS\u201920) ."},{"key":"e_1_3_2_61_2","unstructured":"A. Atrey K. Clary and D. Jensen. 2019. Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning. arXiv:1912.05743 (2019)."},{"key":"e_1_3_2_62_2","unstructured":"A. H\u00fcy\u00fck D. Jarrett C. Tekin and M. van der Schaar. 2021. Explaining by imitating: Understanding decisions by interpretable policy learning. In Proceedings of the International Conference on Learning Representations (ICLR\u201921) ."},{"key":"e_1_3_2_63_2","unstructured":"Omer Gottesman Joseph Futoma Yao Liu Sonali Parbhoo Leo Celi Emma Brunskill and Finale Doshi-Velez. 2020. Interpretable off-policy evaluation in reinforcement learning by highlighting influential transitions. arXiv:2002.03478 (2020)."},{"key":"e_1_3_2_64_2","unstructured":"N. Puri S. Verma P. Gupta D. Kayastha S. Deshmukh B. Krishnamurthy and S. Singh. 2020. Explain your move: Understanding agent actions using specific and relevant feature attribution. arXiv:1912.12191 (2020)."},{"key":"e_1_3_2_65_2","doi-asserted-by":"crossref","unstructured":"T. Huber K. Weitz E. Andr\u00e9 and O. Amir. 2021. Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps. arXiv:2005.08874 (2021).","DOI":"10.24963\/ijcai.2022\/803"},{"key":"e_1_3_2_66_2","doi-asserted-by":"crossref","unstructured":"J. Skirzy\u0144ski F. Becker and F. Lieder. 2021. Automatic discovery of interpretable planning strategies. Machine Learning 110 (2021) 2641\u20132683.","DOI":"10.1007\/s10994-021-05963-2"},{"key":"e_1_3_2_67_2","unstructured":"S. Greydanus A. Koul J. Dodge and A. Fern. 2018. Visualizing and inderstanding Atari agents. In Proceedings of the 35th International Conference on Machine Learning ."},{"key":"e_1_3_2_68_2","unstructured":"T. Silver R. A. Kelsey A. K. Lew L. P. Kaelbling and J. Tenenbaum. 2019. Few-shot Bayesian imitation learning with logical program policies. arXiv:1904.06317 (2019)."},{"key":"e_1_3_2_69_2","doi-asserted-by":"crossref","unstructured":"A. Alqaraawi M. Schuessler P. Weiss E. Costanza and N. Berthouze. 2020. Evaluating saliency map explanations for convolutional neural networks: A user study. arXiv:2002.00772 (2020).","DOI":"10.1145\/3377325.3377519"},{"key":"e_1_3_2_70_2","unstructured":"M. Erwig A. Fern M. Murali and A. Koul. 2018. Explaining deep adaptive programs via reward decomposition. In Proceedings of the IJCAI\/ECAI Workshop on Explainable Artificial Intelligence ."},{"key":"e_1_3_2_71_2","doi-asserted-by":"crossref","unstructured":"T. Huber D. Schiller and E. Andr\u00e9. 2019. Enhancing explainability of deep reinforcement learning through selective layer-wise relevance propagation. In Proceedings of KI 2019: Advances in Artificial Intelligence . 188\u2013202.","DOI":"10.1007\/978-3-030-30179-8_16"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3527448","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3527448","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:18:53Z","timestamp":1750177133000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3527448"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,3]]},"references-count":70,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2023,5,31]]}},"alternative-id":["10.1145\/3527448"],"URL":"https:\/\/doi.org\/10.1145\/3527448","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,3]]},"assertion":[{"value":"2020-10-02","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-03-16","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-12-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}