{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,9]],"date-time":"2026-05-09T05:14:43Z","timestamp":1778303683086,"version":"3.51.4"},"reference-count":96,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2022,9,2]],"date-time":"2022-09-02T00:00:00Z","timestamp":1662076800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Hum.-Robot Interact."],"published-print":{"date-parts":[[2022,9,30]]},"abstract":"<jats:p>The growth of scale and complexity of interactions between humans and robots highlights the need for new computational methods to automatically evaluate novel algorithms and applications. Exploring diverse scenarios of humans and robots interacting in simulation can improve understanding of the robotic system and avoid potentially costly failures in real-world settings. We formulate this problem as a quality diversity (QD) problem, of which the goal is to discover diverse failure scenarios by simultaneously exploring both environments and human actions. We focus on the shared autonomy domain, in which the robot attempts to infer the goal of a human operator, and adopt the QD algorithms CMA-ME and MAP-Elites to generate scenarios for two published algorithms in this domain: shared autonomy via hindsight optimization and linear policy blending. Some of the generated scenarios confirm previous theoretical findings, while others are surprising and bring about a new understanding of state-of-the-art implementations. Our experiments show that the QD algorithms CMA-ME and MAP-Elites outperform Monte-Carlo simulation and optimization-based methods in effectively searching the scenario space, highlighting their promise for automatic evaluation of algorithms in human\u2013robot interaction.<\/jats:p>","DOI":"10.1145\/3476412","type":"journal-article","created":{"date-parts":[[2022,4,18]],"date-time":"2022-04-18T11:39:03Z","timestamp":1650281943000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["Evaluating Human\u2013Robot Interaction Algorithms in Shared Autonomy via Quality Diversity Scenario Generation"],"prefix":"10.1145","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9354-196X","authenticated-orcid":false,"given":"Matthew C.","family":"Fontaine","sequence":"first","affiliation":[{"name":"University of Southern California, Los Angeles, CA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8617-3871","authenticated-orcid":false,"given":"Stefanos","family":"Nikolaidis","sequence":"additional","affiliation":[{"name":"University of Southern California, Los Angeles, CA"}]}],"member":"320","published-online":{"date-parts":[[2022,9,2]]},"reference":[{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.2005.1570269"},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2019.8793740"},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3022099.3022101"},{"key":"e_1_3_4_5_2","first-page":"69","volume-title":"Haifa Verification Conference","author":"Araiza-Illan Dejanira","year":"2015","unstructured":"Dejanira Araiza-Illan, David Western, Anthony Pipe, and Kerstin Eder. 2015. Coverage-Driven Verification - An approach to verify code for robots that directly interact with humans. In Haifa Verification Conference. Springer, 69\u201384."},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-40379-3_3"},{"key":"e_1_3_4_7_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-40793-2_4"},{"key":"e_1_3_4_8_2","doi-asserted-by":"crossref","first-page":"1769","DOI":"10.1109\/CEC.2005.1554902","volume-title":"2005 IEEE Congress on Evolutionary Computation","author":"Auger Anne","year":"2005","unstructured":"Anne Auger and Nikolaus Hansen. 2005. A restart CMA evolution strategy with increasing population size. In 2005 IEEE Congress on Evolutionary Computation, Vol. 2. IEEE, 1769\u20131776."},{"key":"e_1_3_4_9_2","volume-title":"Proceedings of the Annual Meeting of the Cognitive Science Society","volume":"29","author":"Baker Chris L.","year":"2007","unstructured":"Chris L. Baker, Joshua B. Tenenbaum, and Rebecca R. Saxe. 2007. Goal inference as inverse planning. In Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 29."},{"key":"e_1_3_4_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.swevo.2019.100627"},{"key":"e_1_3_4_11_2","doi-asserted-by":"publisher","DOI":"10.1023\/B:AURO.0000016864.12513.77"},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377929.3389938"},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCB.2011.2181833"},{"key":"e_1_3_4_14_2","unstructured":"Jiyoung Choi. 2013. Model checking for decision making behaviour of heterogeneous multi-agent autonomous system. PhD Dissertation. Cranfield University Cranfield UK."},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/IRDS.2002.1043932"},{"key":"e_1_3_4_16_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature14422"},{"key":"e_1_3_4_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3126521"},{"key":"e_1_3_4_18_2","doi-asserted-by":"crossref","first-page":"500","DOI":"10.1007\/978-3-319-24953-7_35","volume-title":"International Symposium on Automated Technology for Verification and Analysis","author":"Deshmukh Jyotirmoy","year":"2015","unstructured":"Jyotirmoy Deshmukh, Xiaoqing Jin, James Kapinski, and Oded Maler. 2015. Stochastic local search for falsification of hybrid systems. In International Symposium on Automated Technology for Verification and Analysis. Springer, 500\u2013517."},{"key":"e_1_3_4_19_2","article-title":"Openrave: A planning architecture for autonomous robotics","author":"Diankov Rosen","year":"2008","unstructured":"Rosen Diankov and James Kuffner. 2008. Openrave: A planning architecture for autonomous robotics. Technical Report CMU-RI-TR-08-34. Robotics Institute. Pittsburgh, PA.","journal-title":"Technical Report CMU-RI-TR-08-34. Robotics Institute. Pittsburgh, PA."},{"key":"e_1_3_4_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/1228716.1228726"},{"key":"e_1_3_4_21_2","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2012.VIII.010"},{"key":"e_1_3_4_22_2","unstructured":"Sixense Entertainment. [n.d.]. Razer hydra. 2011. ([n. d.])."},{"key":"e_1_3_4_23_2","doi-asserted-by":"publisher","DOI":"10.2514\/6.2004-6309"},{"key":"e_1_3_4_24_2","article-title":"Probabilistically safe robot planning with confidence-based human predictions","author":"Fisac Jaime F.","year":"2018","unstructured":"Jaime F. Fisac, Andrea Bajcsy, Sylvia L. Herbert, David Fridovich-Keil, Steven Wang, Claire J. Tomlin, and Anca D. Dragan. 2018. Probabilistically safe robot planning with confidence-based human predictions. arXiv preprint arXiv:1806.00109 (2018).","journal-title":"arXiv preprint arXiv:1806.00109"},{"key":"e_1_3_4_25_2","doi-asserted-by":"publisher","DOI":"10.1162\/isal_a_00316"},{"key":"e_1_3_4_26_2","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2021.XVII.038"},{"key":"e_1_3_4_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3321707.3321794"},{"key":"e_1_3_4_28_2","article-title":"Illuminating Mario scenes in the latent space of a generative adversarial network","author":"Fontaine Matthew C.","year":"2020","unstructured":"Matthew C. Fontaine, Ruilin Liu, Julian Togelius, Amy K. Hoover, and Stefanos Nikolaidis. 2020. Illuminating Mario scenes in the latent space of a generative adversarial network. arXiv preprint arXiv:2007.05674 (2020).","journal-title":"arXiv preprint arXiv:2007.05674"},{"key":"e_1_3_4_29_2","first-page":"arXiv\u20132106","article-title":"Differentiable quality diversity","author":"Fontaine Matthew C.","year":"2021","unstructured":"Matthew C. Fontaine and Stefanos Nikolaidis. 2021. Differentiable quality diversity. arXiv e-prints (2021), arXiv\u20132106.","journal-title":"arXiv e-prints"},{"key":"e_1_3_4_30_2","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2021.XVII.036"},{"key":"e_1_3_4_31_2","article-title":"Covariance matrix adaptation for the rapid illumination of behavior space","author":"Fontaine Matthew C.","year":"2020","unstructured":"Matthew C. Fontaine, Julian Togelius, Stefanos Nikolaidis, and Amy K. Hoover. 2020. Covariance matrix adaptation for the rapid illumination of behavior space. In Proceedings of the Genetic and Evolutionary Computation Conference 94\u2013102.","journal-title":"Proceedings of the Genetic and Evolutionary Computation Conference"},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3071178.3071282"},{"key":"e_1_3_4_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330566"},{"key":"e_1_3_4_34_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364910387173"},{"key":"e_1_3_4_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2016.2593928"},{"key":"e_1_3_4_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/CIG.2019.8848053"},{"key":"e_1_3_4_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/1570256.1570333"},{"key":"e_1_3_4_38_2","article-title":"The CMA evolution strategy: A tutorial","author":"Hansen Nikolaus","year":"2016","unstructured":"Nikolaus Hansen. 2016. The CMA evolution strategy: A tutorial. arXiv preprint arXiv:1604.00772 (2016).","journal-title":"arXiv preprint arXiv:1604.00772"},{"key":"e_1_3_4_39_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10514-013-9350-3"},{"key":"e_1_3_4_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/2422956.2422957"},{"key":"e_1_3_4_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/HRI.2016.7451731"},{"key":"e_1_3_4_42_2","unstructured":"Ronald C. Hofer Edward Ramirez and Scott H. Smith. 1998. Automated Scenario Generation Environment. Institute for Simulation and Training. 31."},{"key":"e_1_3_4_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2018.8593766"},{"key":"e_1_3_4_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3359614"},{"key":"e_1_3_4_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICORR.2015.7281253"},{"key":"e_1_3_4_46_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364918776060"},{"key":"e_1_3_4_47_2","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2015.XI.032"},{"key":"e_1_3_4_48_2","article-title":"Shared autonomy with learned latent actions","author":"Jeon Hong Jun","year":"2020","unstructured":"Hong Jun Jeon, Dylan P. Losey, and Dorsa Sadigh. 2020. Shared autonomy with learned latent actions. arXiv preprint arXiv:2005.03210 (2020).","journal-title":"arXiv preprint arXiv:2005.03210"},{"key":"e_1_3_4_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2005.846356"},{"key":"e_1_3_4_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3319619.3321904"},{"key":"e_1_3_4_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/MCS.2016.2602089"},{"key":"e_1_3_4_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3321707.3321849"},{"key":"e_1_3_4_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIE.2005.855696"},{"key":"e_1_3_4_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2015.2430335"},{"key":"e_1_3_4_55_2","unstructured":"Hadas Kress-Gazit Kerstin Eder Guy Hoffman Henny Admoni Brenna Argall Ruediger Ehlers Christoffer Heckman Nils Jansen Ross Knepper Jan K\u0159et\u00ednsk\u1ef3 et\u00a0al. 2020. Formalizing and Guaranteeing* Human-Robot Interaction. Retrieved September 01 2020 from https:\/\/github.com."},{"key":"e_1_3_4_56_2","unstructured":"Personal Robotics Lab. 2017. https:\/\/github.com\/personalrobotics\/ada\/."},{"key":"e_1_3_4_57_2","unstructured":"Personal Robotics Lab. 2017. Retrieved September 01 2020 from https:\/\/github.com."},{"key":"e_1_3_4_58_2","unstructured":"Personal Robotics Lab. 2017. Retrieved September 01 2020 from https:\/\/github.com."},{"key":"e_1_3_4_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCA.2009.2028239"},{"key":"e_1_3_4_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS45743.2020.9340736"},{"key":"e_1_3_4_61_2","first-page":"125","volume-title":"Proceedings of the 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS\u201903).","author":"Li Ming","year":"2003","unstructured":"Ming Li and Allison M. Okamura. 2003. Recognition of operator motions for real-time assistance using virtual fixtures. In Proceedings of the 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS\u201903). IEEE, 125\u2013131."},{"key":"e_1_3_4_62_2","doi-asserted-by":"crossref","first-page":"4278","DOI":"10.1109\/ICRA.2011.5980055","volume-title":"2011 IEEE International Conference on Robotics and Automation","author":"Li Qinan","year":"2011","unstructured":"Qinan Li, Weidong Chen, and Jingchuan Wang. 2011. Dynamic shared control for human-wheelchair cooperation. In 2011 IEEE International Conference on Robotics and Automation. IEEE, 4278\u20134283."},{"key":"e_1_3_4_63_2","doi-asserted-by":"publisher","DOI":"10.1016\/B978-1-55860-377-6.50052-9"},{"key":"e_1_3_4_64_2","article-title":"Controlling assistive robots with learned latent actions","author":"Losey Dylan P.","year":"2019","unstructured":"Dylan P. Losey, Krishnan Srinivasan, Ajay Mandlekar, Animesh Garg, and Dorsa Sadigh. 2019. Controlling assistive robots with learned latent actions. arXiv preprint arXiv:1909.09674 (2019).","journal-title":"arXiv preprint arXiv:1909.09674"},{"key":"e_1_3_4_65_2","volume-title":"Automatic Scenario Generation using Procedural Modeling Techniques","author":"Martin Glenn A.","year":"2012","unstructured":"Glenn A. Martin. 2012. Automatic Scenario Generation using Procedural Modeling Techniques. Ph.D. Dissertation. University of Central Florida. Orlando, FL."},{"key":"e_1_3_4_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/1814256.1814262"},{"key":"e_1_3_4_67_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00371-005-0289-z"},{"key":"e_1_3_4_68_2","first-page":"3","volume-title":"SEFM 2015 Collocated Workshops","author":"Meinke Karl","year":"2015","unstructured":"Karl Meinke and Peter Nycander. 2015. Learning-based testing of distributed microservice architectures: Correctness and fault injection. In SEFM 2015 Collocated Workshops. Springer, 3\u201310."},{"key":"e_1_3_4_69_2","doi-asserted-by":"crossref","unstructured":"Douglas Morrison Peter Corke and Jurgen Leitner. 2020. Egad! an evolved grasping analysis dataset for diversity and reproducibility in robotic manipulation. IEEE Robotics and Automation Letters 5 3 (2020) 4368\u20134375.","DOI":"10.1109\/LRA.2020.2992195"},{"key":"e_1_3_4_70_2","article-title":"Illuminating search spaces by mapping elites","author":"Mouret Jean-Baptiste","year":"2015","unstructured":"Jean-Baptiste Mouret and Jeff Clune. 2015. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909 (2015).","journal-title":"arXiv preprint arXiv:1504.04909"},{"key":"e_1_3_4_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377930.3390203"},{"key":"e_1_3_4_72_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10514-017-9622-4"},{"key":"e_1_3_4_73_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2017.10.031"},{"key":"e_1_3_4_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298640"},{"key":"e_1_3_4_75_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364917690593"},{"key":"e_1_3_4_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/2909824.3020252"},{"key":"e_1_3_4_77_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-11900-7_39"},{"key":"e_1_3_4_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3242587.3242634"},{"key":"e_1_3_4_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/2739480.2754664"},{"key":"e_1_3_4_80_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.swevo.2016.09.002"},{"key":"e_1_3_4_81_2","article-title":"Shared autonomy via deep reinforcement learning","author":"Reddy Siddharth","year":"2018","unstructured":"Siddharth Reddy, Sergey Levine, and Anca Dragan. 2018. Shared autonomy via deep reinforcement learning. arXiv preprint arXiv:1802.01744 (2018).","journal-title":"arXiv preprint arXiv:1802.01744"},{"key":"e_1_3_4_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2008.917246"},{"key":"e_1_3_4_83_2","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC.2017.8317919"},{"key":"e_1_3_4_84_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-42716-4"},{"key":"e_1_3_4_85_2","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.12276"},{"key":"e_1_3_4_86_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-45823-6_85"},{"key":"e_1_3_4_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/1514095.1514115"},{"key":"e_1_3_4_88_2","doi-asserted-by":"publisher","DOI":"10.1109\/TG.2018.2846639"},{"key":"e_1_3_4_89_2","doi-asserted-by":"publisher","DOI":"10.1561\/2300000049"},{"key":"e_1_3_4_90_2","doi-asserted-by":"publisher","DOI":"10.1109\/SMC.2015.534"},{"key":"e_1_3_4_91_2","doi-asserted-by":"publisher","DOI":"10.1145\/3205455.3205602"},{"key":"e_1_3_4_92_2","first-page":"1","volume-title":"6th International Symposium on Mechatronics and Its Applications (ISMA\u201909).","author":"Veras Eduardo","year":"2009","unstructured":"Eduardo Veras, Karan Khokar, Redwan Alqasemi, and Rajiv Dubey. 2009. Scaled telerobotic control of a manipulator in real time with laser assistance for ADL tasks. In 6th International Symposium on Mechatronics and Its Applications (ISMA\u201909). IEEE, 1\u20136."},{"key":"e_1_3_4_93_2","doi-asserted-by":"publisher","DOI":"10.1017\/9781108231596.006"},{"key":"e_1_3_4_94_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364913478447"},{"key":"e_1_3_4_95_2","doi-asserted-by":"publisher","DOI":"10.1109\/MCS.2003.1213603"},{"key":"e_1_3_4_96_2","first-page":"1433","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Ziebart Brian D.","year":"2008","unstructured":"Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. 2008. Maximum entropy inverse reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence. 1433\u20131438."},{"key":"e_1_3_4_97_2","doi-asserted-by":"publisher","DOI":"10.1145\/2282338.2282371"}],"container-title":["ACM Transactions on Human-Robot Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3476412","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3476412","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:45Z","timestamp":1750188645000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3476412"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,2]]},"references-count":96,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,9,30]]}},"alternative-id":["10.1145\/3476412"],"URL":"https:\/\/doi.org\/10.1145\/3476412","relation":{},"ISSN":["2573-9522","2573-9522"],"issn-type":[{"value":"2573-9522","type":"print"},{"value":"2573-9522","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,2]]},"assertion":[{"value":"2020-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-09-02","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}