{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,27]],"date-time":"2026-06-27T03:50:51Z","timestamp":1782532251381,"version":"3.54.5"},"reference-count":77,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,12,16]],"date-time":"2024-12-16T00:00:00Z","timestamp":1734307200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2024,12,31]]},"abstract":"<jats:p>\n            Reinforcement Learning (RL) is crucial in decision optimization, but its inherent complexity often presents challenges in interpretation and communication. Building upon AutoDOViz\u2014an interface that pushed the boundaries of Automated RL for Decision Optimization\u2014this article unveils an open-source expansion with a web-based platform for RL. Our work introduces a taxonomy of RL visualizations and launches a dynamic web platform, leveraging backend flexibility for AutoRL frameworks like ARLO and Svelte.js for a smooth interactive user experience in the front end. Since AutoDOViz is not open-source, we present AutoRL X, a new interface designed to visualize RL processes. AutoRL X is shaped by the extensive user feedback and expert interviews from AutoDOViz studies, and it brings forth an intelligent interface with real-time, intuitive visualization capabilities that enhance understanding, collaborative efforts, and personalization of RL agents. Addressing the gap in accurately representing complex real-world challenges within standard RL environments, we demonstrate our tool\u2019s application in healthcare, explicitly optimizing brain stimulation trajectories. A user study contrasts the performance of human users optimizing electric fields via a 2D interface with RL agents\u2019 behavior that we visually analyze in AutoRL X, assessing the practicality of automated RL. All our data and code is openly available at:\n            <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/lorifranke\/autorlx\">https:\/\/github.com\/lorifranke\/autorlx<\/jats:ext-link>\n            .\n          <\/jats:p>","DOI":"10.1145\/3670692","type":"journal-article","created":{"date-parts":[[2024,6,3]],"date-time":"2024-06-03T15:05:54Z","timestamp":1717427154000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["AutoRL X: Automated Reinforcement Learning on the Web"],"prefix":"10.1145","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8560-2729","authenticated-orcid":false,"given":"Loraine","family":"Franke","sequence":"first","affiliation":[{"name":"University of Massachusetts Boston, Boston, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5253-0511","authenticated-orcid":false,"given":"Daniel Karl I.","family":"Weidele","sequence":"additional","affiliation":[{"name":"IBM Research, Cambridge, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1617-5502","authenticated-orcid":false,"given":"Nima","family":"Dehmamy","sequence":"additional","affiliation":[{"name":"IBM Research, Cambridge, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4992-459X","authenticated-orcid":false,"given":"Lipeng","family":"Ning","sequence":"additional","affiliation":[{"name":"Harvard Medical School, Boston, MA, USA and Brigham Womens\u2019 Hospital, Boston, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9144-3461","authenticated-orcid":false,"given":"Daniel","family":"Haehn","sequence":"additional","affiliation":[{"name":"University of Massachusetts Boston, Boston, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,12,16]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"crossref","first-page":"101805","DOI":"10.1016\/j.inffus.2023.101805","article-title":"Explainable artificial intelligence (XAI): What we know and what is left to attain trustworthy artificial intelligence","volume":"99","author":"Ali Sajid","year":"2023","unstructured":"Sajid Ali, Tamer Abuhmed, Shaker El-Sappagh, Khan Muhammad, Jose M. Alonso-Moral, Roberto Confalonieri, Riccardo Guidotti, Javier Del Ser, Natalia D\u00edaz-Rodr\u00edguez, and Francisco Herrera. 2023. Explainable artificial intelligence (XAI): What we know and what is left to attain trustworthy artificial intelligence. Information Fusion 99 (2023), 101805.","journal-title":"Information Fusion"},{"issue":"8437","key":"e_1_3_2_3_2","doi-asserted-by":"crossref","first-page":"1106","DOI":"10.1016\/S0140-6736(85)92413-4","article-title":"Non-invasive magnetic stimulation of human motor cortex","volume":"325","author":"Barker Anthony T.","year":"1985","unstructured":"Anthony T. Barker, Reza Jalinous, and Ian L. Freeston. 1985. Non-invasive magnetic stimulation of human motor cortex. The Lancet 325, 8437 (1985), 1106\u20131107.","journal-title":"The Lancet"},{"key":"e_1_3_2_4_2","first-page":"112","volume-title":"Proceedings of the 3rd International Conference on Development and Learning","author":"Barto Andrew G.","year":"2004","unstructured":"Andrew G. Barto, Satinder Singh, and Nuttapong Chentanez. 2004. Intrinsically motivated learning of hierarchical collections of skills. In Proceedings of the 3rd International Conference on Development and Learning. ACM, Piscataway, NJ, 112\u201319."},{"key":"e_1_3_2_5_2","first-page":"1","volume-title":"Proceedings of the 2nd International Conference for Innovation in Technology (INOCON \u201923)","author":"Bhardwaz Saumyamani","year":"2023","unstructured":"Saumyamani Bhardwaz and Rohan Godha. 2023. Svelte. js: The most loved framework today. In Proceedings of the 2nd International Conference for Innovation in Technology (INOCON \u201923). IEEE, New Delhi, India, 1\u20137."},{"key":"e_1_3_2_6_2","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1007\/978-1-4842-4470-8_42","volume-title":"Building Machine Learning and Deep Learning Models on Google Cloud Platform","author":"Bisong Ekaba","year":"2019","unstructured":"Ekaba Bisong. 2019. Google AutoML: Cloud vision. In Building Machine Learning and Deep Learning Models on Google Cloud Platform. Springer, 581\u2013598."},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","unstructured":"Greg Brockman Vicki Cheung Ludwig Pettersson Jonas Schneider John Schulman Jie Tang and Wojciech Zaremba. 2016. Openai gym. arXiv:1606.01540. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.1606.01540","DOI":"10.48550\/arXiv.1606.01540"},{"key":"e_1_3_2_8_2","unstructured":"Ricardo Cabello. 2010. Three.js. Retrieved from https:\/\/threejs.org\/"},{"key":"e_1_3_2_9_2","first-page":"713","volume-title":"Computer Graphics Forum","author":"Chatzimparmpas Angelos","year":"2020","unstructured":"Angelos Chatzimparmpas, Rafael Messias Martins, Ilir Jusufi, Kostiantyn Kucher, Fabrice Rossi, and Andreas Kerren. 2020. The state of the art in enhancing trust in machine learning models with the use of visualizations. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 713\u2013756."},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1007\/s11257-010-9093-1","article-title":"Empirically evaluating the application of reinforcement learning to the induction of effective and adaptive pedagogical strategies","volume":"21","author":"Chi Min","year":"2011","unstructured":"Min Chi, Kurt VanLehn, Diane Litman, and Pamela Jordan. 2011. Empirically evaluating the application of reinforcement learning to the induction of effective and adaptive pedagogical strategies. User Modeling and User-Adapted Interaction 21 (2011), 137\u2013180.","journal-title":"User Modeling and User-Adapted Interaction"},{"issue":"2","key":"e_1_3_2_11_2","doi-asserted-by":"crossref","first-page":"2007","DOI":"10.1109\/LRA.2019.2899918","article-title":"Learning navigation behaviors end-to-end with autoRL","volume":"4","author":"Chiang Hao-Tien L.","year":"2019","unstructured":"Hao-Tien L. Chiang, Aleksandra Faust, Marek Fiser, and Anthony Francis. 2019. Learning navigation behaviors end-to-end with autoRL. IEEE Robotics and Automation Letters 4, 2 (2019), 2007\u20132014.","journal-title":"IEEE Robotics and Automation Letters"},{"key":"e_1_3_2_12_2","first-page":"907","volume-title":"Proceedings of the 46th AIAA Aerospace Sciences Meeting and Exhibit","author":"Chiu Po-Wen","year":"2008","unstructured":"Po-Wen Chiu and Christina Bloebaum. 2008. Hyper-radial visualization (HRV) for decision-making in multi-objective optimization. In Proceedings of the 46th AIAA Aerospace Sciences Meeting and Exhibit. AIAA, 907."},{"key":"e_1_3_2_13_2","first-page":"1","volume-title":"Proceedings of the 4th International Workshop on Data Management for End-to-End Machine Learning","author":"Das Piali","year":"2020","unstructured":"Piali Das, Nikita Ivkin, Tanya Bansal, Laurence Rouesnel, Philip Gautier, Zohar Karnin, Leo Dirac, Lakshmi Ramakrishnan, Andre Perunicic, Iaroslav Shcherbatyi, Wilton Wu, Aida Zolic, Huibin Shen, Amr Ahmed, Fela Winkelmolen, Miroslav Miladinovic, Cedric Archembeau, Alex Tang, Bhaskar Dutt, Patricia Grao, and Kumar Venkateswar. 2020. Amazon SageMaker Autopilot: A white box AutoML solution at scale. In Proceedings of the 4th International Workshop on Data Management for End-to-End Machine Learning. 1\u20137."},{"issue":"131","key":"e_1_3_2_14_2","first-page":"1","article-title":"MushroomRL: Simplifying reinforcement learning research","volume":"22","author":"D\u2019Eramo Carlo","year":"2021","unstructured":"Carlo D\u2019Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. 2021. MushroomRL: Simplifying reinforcement learning research. Journal of Machine Learning Research 22, 131 (2021), 1\u20135. Retrieved from http:\/\/jmlr.org\/papers\/v22\/18-056.html","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","unstructured":"Shuby Deshpande Benjamin Eysenbach and Jeff Schneider. 2020. Interactive visualization for debugging RL. arXiv:2008.07331. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2008.07331","DOI":"10.48550\/arXiv.2008.07331"},{"issue":"1","key":"e_1_3_2_16_2","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1145\/3359786","article-title":"Techniques for interpretable machine learning","volume":"63","author":"Du Mengnan","year":"2019","unstructured":"Mengnan Du, Ninghao Liu, and Xia Hu. 2019. Techniques for interpretable machine learning. Communications of the ACM 63, 1 (2019), 68\u201377.","journal-title":"Communications of the ACM"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","unstructured":"Aleksandra Faust Anthony Francis and Dar Mehta. 2019. Evolving rewards to automate reinforcement learning. arXiv:1905.07628. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.1905.07628","DOI":"10.48550\/arXiv.1905.07628"},{"key":"e_1_3_2_18_2","first-page":"28","article-title":"Efficient and robust automated machine learning","author":"Feurer Matthias","year":"2015","unstructured":"Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and robust automated machine learning. Advances in Neural Information Processing Systems 28 (2015), 2755\u20132763.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_19_2","unstructured":"J\u00f6rg K. H. Franke Gregor K\u00f6hler Andr\u00e9 Biedenkapp and Frank Hutter. 2020. Sample-efficient automated deep reinforcement learning. arXiv:2009.01555. Retrieved from https:\/\/arxiv.org\/abs\/2009.01555"},{"issue":"4","key":"e_1_3_2_20_2","doi-asserted-by":"crossref","first-page":"37","DOI":"10.3390\/informatics7040037","article-title":"Modern scientific visualizations on the web","volume":"7","author":"Franke Loraine","year":"2020","unstructured":"Loraine Franke and Daniel Haehn. 2020. Modern scientific visualizations on the web. Informatics 7, 4 (2020), 37.","journal-title":"Informatics"},{"issue":"3","key":"e_1_3_2_21_2","doi-asserted-by":"crossref","first-page":"807","DOI":"10.1016\/j.ejor.2020.08.045","article-title":"Optimization problems for machine learning: A survey","volume":"290","author":"Gambella Claudio","year":"2021","unstructured":"Claudio Gambella, Bissan Ghaddar, and Joe Naoum-Sawaya. 2021. Optimization problems for machine learning: A survey. European Journal of Operational Research 290, 3 (2021), 807\u2013828.","journal-title":"European Journal of Operational Research"},{"key":"e_1_3_2_22_2","first-page":"1792","volume-title":"Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research)","volume":"80","author":"Greydanus Samuel","year":"2018","unstructured":"Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. 2018. Visualizing and understanding atari agents. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research), Vol. 80. Jennifer Dy and Andreas Krause (Eds.), PMLR, Stockholm, Sweden, 1792\u20131801. Retrieved from https:\/\/proceedings.mlr.press\/v80\/greydanus18a.html"},{"key":"e_1_3_2_23_2","unstructured":"Khronos WebGL Working Group. 2011. WebGL. Retrieved from https:\/\/www.khronos.org\/webgl\/"},{"key":"e_1_3_2_24_2","first-page":"1861","article-title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","author":"Haarnoja Tuomas","year":"2018","unstructured":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference on Machine Learning. PMLR, 1861\u20131870.","journal-title":"Proceedings of the International Conference on Machine Learning"},{"key":"e_1_3_2_25_2","first-page":"36","volume-title":"Proceedings of the IEEE Pacific Visualization Symposium (PacificVis \u201920)","author":"He Wenbin","year":"2020","unstructured":"Wenbin He, Teng-Yok Lee, Jeroen van Baar, Kent Wittenburg, and Han-Wei Shen. 2020. DynamicsExplorer: Visual analytics for robot control tasks involving dynamics and LSTM-based control policies. In Proceedings of the IEEE Pacific Visualization Symposium (PacificVis \u201920). IEEE, Tianjin, China, 36\u201345."},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2020.106622"},{"key":"e_1_3_2_27_2","doi-asserted-by":"crossref","first-page":"4794","DOI":"10.1145\/3534678.3542630","volume-title":"Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","author":"Hirzel Martin","year":"2022","unstructured":"Martin Hirzel, Kiran Kate, Parikshit Ram, Avraham Shinnar, and Jason Tsay. 2022. Gradual AutoML using Lale. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 4794\u20134795."},{"issue":"8","key":"e_1_3_2_28_2","doi-asserted-by":"crossref","first-page":"2674","DOI":"10.1109\/TVCG.2018.2843369","article-title":"Visual analytics in deep learning: An interrogative survey for the next frontiers","volume":"25","author":"Hohman Fred","year":"2018","unstructured":"Fred Hohman, Minsuk Kahng, Robert Pienta, and Duen Horng Chau. 2018. Visual analytics in deep learning: An interrogative survey for the next frontiers. IEEE Transactions on Visualization and Computer Graphics 25, 8 (2018), 2674\u20132693.","journal-title":"IEEE Transactions on Visualization and Computer Graphics"},{"issue":"2","key":"e_1_3_2_29_2","doi-asserted-by":"crossref","first-page":"e13898","DOI":"10.1002\/acm2.13898","article-title":"Reinforcement learning in medical image analysis: Concepts, applications, challenges, and future directions","volume":"24","author":"Hu Mingzhe","year":"2023","unstructured":"Mingzhe Hu, Jiahan Zhang, Luke Matkovic, Tian Liu, and Xiaofeng Yang. 2023. Reinforcement learning in medical image analysis: Concepts, applications, challenges, and future directions. Journal of Applied Clinical Medical Physics 24, 2 (2023), e13898.","journal-title":"Journal of Applied Clinical Medical Physics"},{"key":"e_1_3_2_30_2","first-page":"16041","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","volume":"35","author":"Hulbert Neil","year":"2021","unstructured":"Neil Hulbert, Sam Spillers, Brandon Francis, James Haines-Temons, Ken G. Romero, Benjamin De Jager, Sam Wong, Kevin Flora, Bowei Huang, and Athirai A. Irissappane. 2021. EasyRL: A simple and extensible reinforcement learning framework. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 16041\u201316043."},{"issue":"3","key":"e_1_3_2_31_2","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1111\/cgf.13962","article-title":"DRLViz: Understanding decisions and memory in deep reinforcement learning","volume":"39","author":"Jaunet Theo","year":"2020","unstructured":"Theo Jaunet, Romain Vuillemot, and Christian Wolf. 2020. DRLViz: Understanding decisions and memory in deep reinforcement learning. Computer Graphics Forum 39, 3 (2020), 49\u201361.","journal-title":"Computer Graphics Forum"},{"key":"e_1_3_2_32_2","first-page":"1946","volume-title":"Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","author":"Jin Haifeng","year":"2019","unstructured":"Haifeng Jin, Qingquan Song, and Xia Hu. 2019. Auto-keras: An efficient neural architecture search system. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1946\u20131956."},{"key":"e_1_3_2_33_2","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1613\/jair.301","article-title":"Reinforcement learning: A survey","volume":"4","author":"Kaelbling Leslie P.","year":"1996","unstructured":"Leslie P. Kaelbling, Michael L. Littman, and Andrew W. Moore. 1996. Reinforcement learning: A survey. Journal of Artificial Intelligence Research 4 (1996), 237\u2013285.","journal-title":"Journal of Artificial Intelligence Research"},{"key":"e_1_3_2_34_2","first-page":"1","volume-title":"Proceedings of the IEEE International Conference on Data Science and Advanced Analytics (DSAA \u201915)","author":"Kanter James M.","year":"2015","unstructured":"James M. Kanter and Kalyan Veeramachaneni. 2015. Deep feature synthesis: Towards automating data science endeavors. In Proceedings of the IEEE International Conference on Data Science and Advanced Analytics (DSAA \u201915). IEEE, 1\u201310."},{"key":"e_1_3_2_35_2","first-page":"1304","volume-title":"Proceedings of the IEEE 16th International Conference on Data Mining Workshops (ICDMW \u201916)","author":"Khurana Udayan","year":"2016","unstructured":"Udayan Khurana, Deepak Turaga, Horst Samulowitz, and Srinivasan Parthasrathy. 2016. Cognito: Automated feature engineering for supervised learning. In Proceedings of the IEEE 16th International Conference on Data Mining Workshops (ICDMW \u201916). IEEE, 1304\u20131307."},{"issue":"22","key":"e_1_3_2_36_2","doi-asserted-by":"crossref","first-page":"2751","DOI":"10.3390\/electronics10222751","article-title":"Marsexplorer: Exploration of unknown terrains via deep reinforcement learning and procedurally generated environments","volume":"10","author":"Koutras Dimitrios I.","year":"2021","unstructured":"Dimitrios I. Koutras, Athanasios C. Kapoutsis, Angelos A. Amanatiadis, and Elias B. Kosmatopoulos. 2021. Marsexplorer: Exploration of unknown terrains via deep reinforcement learning and procedurally generated environments. Electronics 10, 22 (2021), 2751.","journal-title":"Electronics"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","unstructured":"Hoang T. Lam Johann-Michael Thiebaut Mathieu Sinn Bei Chen Tiep Mai and Oznur Alkan. 2017. One button machine for automating feature engineering in relational databases. arXiv:1706.00327. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.1706.00327","DOI":"10.48550\/arXiv.1706.00327"},{"key":"e_1_3_2_38_2","volume-title":"Proceedings of the AutoML Workshop at ICML","volume":"2020","author":"LeDell Erin","year":"2020","unstructured":"Erin LeDell and Sebastien Poirier. 2020. H2O autoML: Scalable automatic machine learning. In Proceedings of the AutoML Workshop at ICML, Vol. 2020."},{"key":"e_1_3_2_39_2","unstructured":"Yuxi Li. 2018. Deep reinforcement learning. arXiv:1810.06339. Retrieved from http:\/\/arxiv.org\/abs\/1810.06339"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","unstructured":"Timothy P. Lillicrap Jonathan J. Hunt Alexander Pritzel Nicolas Heess Tom Erez Yuval Tassa David Silver and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv:1509.02971. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.1509.02971","DOI":"10.48550\/arXiv.1509.02971"},{"key":"e_1_3_2_41_2","first-page":"561","volume-title":"Proceedings of the Conference on Robot Learning","author":"Mahmood A. Rupam","year":"2018","unstructured":"A. Rupam Mahmood, Dmytro Korenkevych, Gautham Vasan, William Ma, and James Bergstra. 2018. Benchmarking reinforcement learning algorithms on real-world robots. In Proceedings of the Conference on Robot Learning. PMLR, 561\u2013591."},{"key":"e_1_3_2_42_2","unstructured":"Radu Marinescu Tejaswini Pedapati Long Vu Paulito Palmes Todd Mummert Peter Kirchner Dharmashankar Subramanian Parikshit Ram and Djallel Bouneffouf. 2022. Automated decision optimization with reinforcement learning. (2022)."},{"key":"e_1_3_2_43_2","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1109\/PacificVis53943.2022.00020","volume-title":"Proceedings of the IEEE 15th Pacific Visualization Symposium (PacificVis \u201922)","author":"Mishra Aditi","year":"2022","unstructured":"Aditi Mishra, Utkarsh Soni, Jinbin Huang, and Chris Bryan. 2022. Why? Why not? When? Visual explanations of agent behaviour in reinforcement learning. In Proceedings of the IEEE 15th Pacific Visualization Symposium (PacificVis \u201922). IEEE, Tsukuba, Japan, 111\u2013120."},{"key":"e_1_3_2_44_2","first-page":"1928","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Mnih Volodymyr","year":"2016","unstructured":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In Proceedings of the International Conference on Machine Learning. PMLR, 1928\u20131937."},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","unstructured":"Volodymyr Mnih Koray Kavukcuoglu David Silver Alex Graves Ioannis Antonoglou Daan Wierstra and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv:1312.5602. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.1312.5602","DOI":"10.48550\/arXiv.1312.5602"},{"issue":"7540","key":"e_1_3_2_46_2","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih Volodymyr","year":"2015","unstructured":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529\u2013533.","journal-title":"Nature"},{"key":"e_1_3_2_47_2","volume-title":"Practical Automated Machine Learning on Azure: Using Azure Machine Learning to Quickly Build AI Solutions","author":"Mukunthu Deepak","year":"2019","unstructured":"Deepak Mukunthu, Parashar Shah, and Wee Hyong Tok. 2019. Practical Automated Machine Learning on Azure: Using Azure Machine Learning to Quickly Build AI Solutions. O\u2019Reilly Media."},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","unstructured":"Marco Mussi Davide Lombarda Alberto Maria Metelli Francesco Trovo and Marcello Restelli. 2022. ARLO: A framework for automated reinforcement learning. DOI: arXiv:2205.10416. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2205.10416","DOI":"10.48550\/arXiv.2205.10416"},{"key":"e_1_3_2_49_2","first-page":"590","volume-title":"Proceedings of the 3rd IEEE International Conference on Robotic Computing (IRC \u201919)","author":"Nguyen Hai","year":"2019","unstructured":"Hai Nguyen and Hung La. 2019. Review of deep reinforcement learning for robot manipulation. In Proceedings of the 3rd IEEE International Conference on Robotic Computing (IRC \u201919). 590\u2013595. DOI: 10.1109\/IRC.2019.00120"},{"key":"e_1_3_2_50_2","doi-asserted-by":"crossref","first-page":"485","DOI":"10.1145\/2908812.2908918","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference 2016","author":"Olson Randal S.","year":"2016","unstructured":"Randal S. Olson, Nathan Bartley, Ryan J. Urbanowicz, and Jason H. Moore. 2016. Evaluation of a tree-based pipeline optimization tool for automating data science. In Proceedings of the Genetic and Evolutionary Computation Conference 2016. 485\u2013492."},{"key":"e_1_3_2_51_2","doi-asserted-by":"crossref","first-page":"517","DOI":"10.1613\/jair.1.13596","article-title":"Automated reinforcement learning (autoRL): A survey and open problems","volume":"74","author":"Parker-Holder Jack","year":"2022","unstructured":"Jack Parker-Holder, Raghu Rajan, Xingyou Song, Andr\u00e9 Biedenkapp, Yingjie Miao, Theresa Eimer, Baohe Zhang, Vu Nguyen, Roberto Calandra, Aleksandra Faust, Frank Hutter, and Marius Lindauer. 2022. Automated reinforcement learning (autoRL): A survey and open problems. Journal of Artificial Intelligence Research 74 (2022), 517\u2013568.","journal-title":"Journal of Artificial Intelligence Research"},{"issue":"2","key":"e_1_3_2_52_2","doi-asserted-by":"crossref","first-page":"70","DOI":"10.3905\/jfds.2021.1.060","article-title":"Building cross-sectional systematic strategies by learning to rank","volume":"3","author":"Poh Daniel","year":"2021","unstructured":"Daniel Poh, Bryan Lim, Stefan Zohren, and Stephen Roberts. 2021. Building cross-sectional systematic strategies by learning to rank. The Journal of Financial Data Science 3, 2 (2021), 70\u201386.","journal-title":"The Journal of Financial Data Science"},{"key":"e_1_3_2_53_2","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1007\/978-3-030-57321-8_5","volume-title":"Proceedings of the International Cross-Domain Conference for Machine Learning and Knowledge Extraction","author":"Puiutta Erika","year":"2020","unstructured":"Erika Puiutta and Eric Veith. 2020. Explainable reinforcement learning: A survey. In Proceedings of the International Cross-Domain Conference for Machine Learning and Knowledge Extraction. Springer, 77\u201395."},{"key":"e_1_3_2_54_2","unstructured":"Rich Harris and Svelte Core team. 2020. [n. d.]. Svelte.js\u2014Cybernetically enhanced web apps."},{"key":"e_1_3_2_55_2","first-page":"43","volume-title":"Proceedings of the EuroVis (Short Papers)","author":"Saldanha Emily","year":"2019","unstructured":"Emily Saldanha, Brenda Praggastis, Todd Billow, and Dustin Lockhart Arendt. 2019. ReLVis: Visual analytics for situational awareness during reinforcement learning experimentation.. In Proceedings of the EuroVis (Short Papers). IEEE, Porto, Portugal, 43\u201347."},{"key":"e_1_3_2_56_2","unstructured":"John Schulman Filip Wolski Prafulla Dhariwal Alec Radford and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv:1707.06347."},{"issue":"5306","key":"e_1_3_2_57_2","doi-asserted-by":"crossref","first-page":"1593","DOI":"10.1126\/science.275.5306.1593","article-title":"A neural substrate of prediction and reward","volume":"275","author":"Schultz Wolfram","year":"1997","unstructured":"Wolfram Schultz, Peter Dayan, and P Read Montague. 1997. A neural substrate of prediction and reward. Science 275, 5306 (1997), 1593\u20131599.","journal-title":"Science"},{"key":"e_1_3_2_58_2","doi-asserted-by":"crossref","first-page":"2584","DOI":"10.1145\/3448016.3457557","volume-title":"Proceedings of the 2021 International Conference on Management of Data","author":"Shah Syed Y.","year":"2021","unstructured":"Syed Y. Shah, Dhaval Patel, Long Vu, Xuan-Hong Dang, Bei Chen, Peter Kirchner, Horst Samulowitz, David Wood, Gregory Bramble, Wesley M. Gifford, Giridhar Ganapavarapu, Roman Vaculin, and Petros Zerfos. 2021. AutoAI-TS: AutoAI for time series forecasting. In Proceedings of the 2021 International Conference on Management of Data. 2584\u20132596."},{"key":"e_1_3_2_59_2","volume-title":"Proceedings of the 6th Workshop on Meta-Learning at the Conference on Neural Information Processing Systems","author":"Shala Gresa","year":"2022","unstructured":"Gresa Shala, Sebastian P. Arango, Andr\u00e9 Biedenkapp, Frank Hutter, and Josif Grabocka. 2022. AutoRL-Bench 1.0. In Proceedings of the 6th Workshop on Meta-Learning at the Conference on Neural Information Processing Systems. NeurIPS, New Orleans USA, Poster."},{"key":"e_1_3_2_60_2","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton Richard S.","year":"2018","unstructured":"Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA."},{"key":"e_1_3_2_61_2","first-page":"5026","volume-title":"Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems","author":"Todorov Emanuel","year":"2012","unstructured":"Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012. Mujoco: A physics engine for model-based control. In Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems. IEEE, IEEE, Vilamoura Portugal, 5026\u20135033."},{"key":"e_1_3_2_62_2","first-page":"1380","volume-title":"Proceedings of the 29th IEEE International Conference on Robot And Human Interactive Communication (RO-MAN \u201920)","author":"Varier Vignesh Manoj","year":"2020","unstructured":"Vignesh Manoj Varier, Dhruv Kool Rajamani, Nathaniel Goldfarb, Farid Tavakkolmoghaddam, Adnan Munawar, and Gregory S. Fischer. 2020. Collaborative suturing: A reinforcement learning approach to automate hand-off task in suturing for surgical robots. In Proceedings of the 29th IEEE International Conference on Robot And Human Interactive Communication (RO-MAN \u201920). IEEE, 1380\u20131386."},{"key":"e_1_3_2_63_2","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"3","author":"Wang Dakuo","year":"2019","unstructured":"Dakuo Wang, Justin D. Weisz, Michael Muller, Parikshit Ram, Werner Geyer, Casey Dugan, Yla Tausczik, Horst Samulowitz, and Alexander Gray. 2019. Human-AI collaboration in data science: Exploring data scientists\u2019 perceptions of automated AI. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1\u201324."},{"issue":"1","key":"e_1_3_2_64_2","doi-asserted-by":"crossref","first-page":"288","DOI":"10.1109\/TVCG.2018.2864504","article-title":"DQNViz: A visual analytics approach to understand deep Q-networks","volume":"25","author":"Wang Junpeng","year":"2018","unstructured":"Junpeng Wang, Liang Gou, Han-Wei Shen, and Hao Yang. 2018. DQNViz: A visual analytics approach to understand deep Q-networks. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2018), 288\u2013298.","journal-title":"IEEE Transactions on Visualization and Computer Graphics"},{"issue":"12","key":"e_1_3_2_65_2","doi-asserted-by":"crossref","first-page":"4141","DOI":"10.1109\/TVCG.2021.3076749","article-title":"Visual analytics for RNN-based deep reinforcement learning","volume":"28","author":"Wang Junpeng","year":"2021","unstructured":"Junpeng Wang, Wei Zhang, Hao Yang, Chin-Chia Michael Yeh, and Liang Wang. 2021. Visual analytics for RNN-based deep reinforcement learning. IEEE Transactions on Visualization and Computer Graphics 28, 12 (2021), 4141\u20134155.","journal-title":"IEEE Transactions on Visualization and Computer Graphics"},{"issue":"12","key":"e_1_3_2_66_2","doi-asserted-by":"crossref","first-page":"5134","DOI":"10.1109\/TVCG.2021.3106142","article-title":"A survey on ML4VIS: Applying machinelearning advances to data visualization","volume":"28","author":"Wang Qianwen","year":"2021","unstructured":"Qianwen Wang, Zhutian Chen, Yong Wang, and Huamin Qu. 2021. A survey on ML4VIS: Applying machinelearning advances to data visualization. IEEE Transactions on Visualization and Computer Graphics 28, 12 (2021), 5134\u20135153.","journal-title":"IEEE Transactions on Visualization and Computer Graphics"},{"key":"e_1_3_2_67_2","first-page":"1","volume-title":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems","author":"Wang Qianwen","year":"2019","unstructured":"Qianwen Wang, Yao Ming, Zhihua Jin, Qiaomu Shen, Dongyu Liu, Micah J. Smith, Kalyan Veeramachaneni, and Huamin Qu. 2019. Atmseer: Increasing transparency and controllability in automated machine learning. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1\u201312."},{"key":"e_1_3_2_68_2","unstructured":"Sen Wang Daoyuan Jia and Xinshuo Weng. 2018. Deep reinforcement learning for autonomous driving. arXiv:1811.11329."},{"key":"e_1_3_2_69_2","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1109\/VISUAL.2019.8933632","volume-title":"Proceedings of the IEEE Visualization Conference (VIS \u201919)","author":"Weidele Daniel Karl I.","year":"2019","unstructured":"Daniel Karl I. Weidele. 2019. Conditional parallel coordinates. In Proceedings of the IEEE Visualization Conference (VIS \u201919). IEEE, Vancouver, Canada, 221\u2013225."},{"key":"e_1_3_2_70_2","doi-asserted-by":"crossref","first-page":"664","DOI":"10.1145\/3581641.3584094","volume-title":"Proceedings of the 28th International Conference on Intelligent User Interfaces","author":"Weidele Daniel Karl I.","year":"2023","unstructured":"Daniel Karl I. Weidele, Shazia Afzal, Abel N. Valente, Cole Makuch, Owen Cornec, Long Vu, Dharmashankar Subramanian, Werner Geyer, Rahul Nair, Inge Vejsbjerg, Radu Marinescu, Paulito Palmes, Elizabeth M. Daly, Loraine Franke, and Daniel Haehn. 2023. AutoDOViz: Human-centered automation for decision optimization. In Proceedings of the 28th International Conference on Intelligent User Interfaces. ACM, Sydney, 664\u2013680."},{"key":"e_1_3_2_71_2","doi-asserted-by":"crossref","first-page":"308","DOI":"10.1145\/3377325.3377538","volume-title":"Proceedings of the 25th International Conference on Intelligent User Interfaces","author":"Weidele Daniel Karl I.","year":"2020","unstructured":"Daniel Karl I. Weidele, Justin D. Weisz, Erick Oduor, Michael Muller, Josh Andres, Alexander Gray, and Dakuo Wang. 2020. AutoAIViz: opening the blackbox of automated artificial intelligence with conditional parallel coordinates. In Proceedings of the 25th International Conference on Intelligent User Interfaces. ACM, Cagliari, Italy, 308\u2013312."},{"key":"e_1_3_2_72_2","doi-asserted-by":"crossref","first-page":"550030","DOI":"10.3389\/frai.2021.550030","article-title":"Explainable AI and reinforcement learning\u2014A systematic review of current approaches and trends","volume":"4","author":"Wells Lindsay","year":"2021","unstructured":"Lindsay Wells and Tomasz Bednarz. 2021. Explainable AI and reinforcement learning\u2014A systematic review of current approaches and trends. Frontiers in Artificial Intelligence 4 (2021), 550030.","journal-title":"Frontiers in Artificial Intelligence"},{"key":"e_1_3_2_73_2","first-page":"8","volume-title":"Proceedings of the 1st ACM International Conference on AI in Finance (ICAIF \u201920)","author":"Yang Hongyang","year":"2021","unstructured":"Hongyang Yang, Xiao-Yang Liu, Shan Zhong, and Anwar Walid. 2021. Deep reinforcement learning for automated stock trading: An ensemble strategy. In Proceedings of the 1st ACM International Conference on AI in Finance (ICAIF \u201920). Association for Computing Machinery, New York, NY, Article 31, 8 pages. DOI: 10.1145\/3383455.3422540"},{"issue":"1","key":"e_1_3_2_74_2","first-page":"1","article-title":"Reinforcement learning in healthcare: A survey","volume":"55","author":"Yu Chao","year":"2021","unstructured":"Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. 2021. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR \u201921) 55, 1 (2021), 1\u201336.","journal-title":"ACM Computing Surveys (CSUR \u201921)"},{"key":"e_1_3_2_75_2","first-page":"1899","volume-title":"Proceedings of The 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research","author":"Zahavy Tom","year":"2016","unstructured":"Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. 2016. Graying the black box: Understanding DQNs. In Proceedings of The 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research ), Vol. 48. Maria Florina Balcan and Kilian Q. Weinberger (Eds.), PMLR, New York, New York, 1899\u20131908. Retrieved from https:\/\/proceedings.mlr.press\/v48\/zahavy16.html"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.17775\/CSEEJPES.2018.00520"},{"key":"e_1_3_2_77_2","doi-asserted-by":"crossref","first-page":"409","DOI":"10.1613\/jair.1.11854","article-title":"Benchmark and survey of automated machine learning frameworks","volume":"70","author":"Z\u00f6ller Marc-Andr\u00e9","year":"2021","unstructured":"Marc-Andr\u00e9 Z\u00f6ller and Marco F. Huber. 2021. Benchmark and survey of automated machine learning frameworks. Journal of Artificial Intelligence Research 70 (2021), 409\u2013472.","journal-title":"Journal of Artificial Intelligence Research"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3625240"}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3670692","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3670692","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:58:21Z","timestamp":1750294701000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3670692"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,16]]},"references-count":77,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,12,31]]}},"alternative-id":["10.1145\/3670692"],"URL":"https:\/\/doi.org\/10.1145\/3670692","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"value":"2160-6455","type":"print"},{"value":"2160-6463","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,16]]},"assertion":[{"value":"2023-11-07","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-03-26","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}