{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T19:45:31Z","timestamp":1782848731486,"version":"3.54.5"},"reference-count":48,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2025,1,22]],"date-time":"2025-01-22T00:00:00Z","timestamp":1737504000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000181","name":"Air Force Office of Scientific Research","doi-asserted-by":"crossref","award":["FA9550-23-1-0283"],"award-info":[{"award-number":["FA9550-23-1-0283"]}],"id":[{"id":"10.13039\/100000181","id-type":"DOI","asserted-by":"crossref"}]},{"name":"National Science Foundation","award":["1828010"],"award-info":[{"award-number":["1828010"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Hum.-Robot Interact."],"published-print":{"date-parts":[[2025,6,30]]},"abstract":"<jats:p>\n            To achieve effective coordination in human\u2013robot teams, robots must have an accurate model of human decision-making (i.e., theory of mind) to predict human actions and plan appropriate supplemental strategies. However, humans are often assumed to be approximately rational decision-makers which is not always the case, especially when faced with risky or uncertain decisions. Recent works in human\u2013robot interaction have begun to address this by implementing risk-sensitive models of human behavior to characterize an individual\u2019s risk-sensitivity. However, little attention is given to the following question: what happens when the robot makes the incorrect inference about human risk-sensitivity? Failure to consider this may lead to ineffective coordination and degradation of perceived trustworthiness in the robot. In this article, we adopt a popular risk-sensitive model based on Cumulative Prospect Theory, where model accuracy is varied in the robot\u2019s theory of mind when interacting with either a risk-averse (pessimistic) or risk-seeking (optimistic) human. We designed a joint-pursuit game where the human and robot are conditioned with different (assumptions of) human risk-sensitivity in a 2\n            <jats:inline-formula content-type=\"math\/tex\">\n              <jats:tex-math notation=\"LaTeX\" version=\"MathJax\">\\( \\times \\)<\/jats:tex-math>\n            <\/jats:inline-formula>\n            2, between-subject study. Results from both simulated and human experiments showed that team performance was decreased and perceived trustworthiness in the robot was negatively impacted when the robot made the incorrect assumption of human risk-sensitivity. Overall, this work shows that risk-sensitive models can be used to great effect but only if we remain diligent to the fact that misspecification can lead to negative outcomes.\n          <\/jats:p>","DOI":"10.1145\/3706068","type":"journal-article","created":{"date-parts":[[2024,12,4]],"date-time":"2024-12-04T06:55:01Z","timestamp":1733295301000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["What If I\u2019m Wrong? Team Performance and Trustworthiness When Modeling Risk-Sensitivity in Human\u2013Robot Collaboration"],"prefix":"10.1145","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3772-143X","authenticated-orcid":false,"given":"Mason O.","family":"Smith","sequence":"first","affiliation":[{"name":"School of Manufacturing and Systems Networks, Arizona State University Ira A Fulton Schools of Engineering, Mesa, Arizona, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4046-2213","authenticated-orcid":false,"given":"Wenlong","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Manufacturing and Systems Networks, Arizona State University Ira A Fulton Schools of Engineering, Mesa, Arizona, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,1,22]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-398532-3.00007-5"},{"key":"e_1_3_3_3_2","volume-title":"Proceedings of the Stanford Encyclopedia of Philosophy","author":"Briggs Rachael A.","year":"2023","unstructured":"Rachael A. Briggs. 2023. Normative theories of rational choice: Expected utility. In Proceedings of the Stanford Encyclopedia of Philosophy (Winter 2023 ed.). Edward N. Zalta and Uri Nodelman (Eds.), Metaphysics Research Lab, Stanford University. Retrieved from https:\/\/plato.stanford.edu\/archives\/win2023\/entries\/rationality-normative-utility\/"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2020.2964824"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2010.05.002"},{"key":"e_1_3_3_6_2","volume-title":"Thinking, Fast and Slow","author":"Daniel Kahneman","year":"2017","unstructured":"Kahneman Daniel. 2017. Thinking, Fast and Slow. Farrar, Straus and Giroux."},{"key":"e_1_3_3_7_2","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1137\/1.9781611977745.2","volume-title":"2023 Proceedings of the Conference on Control and Its Applications (CT \u201923)","author":"Danis Dominic","year":"2023","unstructured":"Dominic Danis, Parker Parmacek, David Dunajsky, and Bhaskar Ramasubramanian. 2023. Multi-agent reinforcement learning with prospect theory. In 2023 Proceedings of the Conference on Control and Its Applications (CT \u201923). SIAM, 9\u201316. DOI: 10.1137\/1.9781611977745.2"},{"key":"e_1_3_3_8_2","unstructured":"Ruth A. David and Paul Nielsen. 2016. Defense science board summer study on autonomy. Defense Science Board Washington United States. Technical Report. Office of the Under Secretary of Defense for Acquisition Technology and Logistics Washington DC 14\u201323. Retrieved from https:\/\/apps.dtic.mil\/sti\/citations\/AD1017790"},{"key":"e_1_3_3_9_2","first-page":"319","volume-title":"Proceedings of the 2016 11th ACM\/IEEE International Conference on Human-Robot Interaction (HRI \u201916)","author":"Devin Sandra","year":"2016","unstructured":"Sandra Devin and Rachid Alami. 2016. An implemented theory of mind to improve human-robot shared plans execution. In Proceedings of the 2016 11th ACM\/IEEE International Conference on Human-Robot Interaction (HRI \u201916), 319\u2013326. DOI: 10.1109\/HRI.2016.7451768"},{"key":"e_1_3_3_10_2","first-page":"1279","volume-title":"Proceedings of the 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN \u201923)","author":"Favier Anthony","year":"2023","unstructured":"Anthony Favier, Shashank Shekhar, and Rachid Alami. 2023. Models and algorithms for human-aware task planning with integrated theory of mind. In Proceedings of the 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN \u201923), 1279\u20131286. DOI: 10.1109\/RO-MAN57019.2023.10309437"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1009167"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-52581-1_4"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10111-011-0194-3"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1075\/is.8.3.10gro"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2019.2929980"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1177\/0018720814547570"},{"key":"e_1_3_3_17_2","first-page":"1984","volume-title":"Proceedings of the 2020 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS \u201920)","author":"Hsu Ya-Chuan","year":"2020","unstructured":"Ya-Chuan Hsu, Swaminathan Gopalswamy, Srikanth Saripalli, and Dylan A. Shell. 2020. A POMDP treatment of vehicle-pedestrian interaction: Implicit coordination via uncertainty-aware planning. In Proceedings of the 2020 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS \u201920), 1984\u20131991. DOI: 10.1109\/IROS45743.2020.9341320"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/TAC.2018.2822658"},{"key":"e_1_3_3_19_2","unstructured":"Gregory Kahn Adam Villaflor Vitchyr Pong Pieter Abbeel and Sergey Levine. 2017. Uncertainty-aware reinforcement learning for collision avoidance. arXiv:1702.01182. Retrieved from https:\/\/arxiv.org\/abs\/1702.01182"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.185.4157.1124"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1142\/9789814417358_0006"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-020-18353-4"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.trc.2021.103428"},{"key":"e_1_3_3_24_2","first-page":"43","volume-title":"Proceedings of the 2020 ACM\/IEEE International Conference on Human-Robot Interaction (HRI \u201920)","author":"Kwon Minae","year":"2020","unstructured":"Minae Kwon, Erdem Biyik, Aditi Talati, Karan Bhasin, Dylan P. Losey, and Dorsa Sadigh. 2020. When humans aren\u2019t optimal: Robots that collaborate with risk-aware humans. In Proceedings of the 2020 ACM\/IEEE International Conference on Human-Robot Interaction (HRI \u201920). ACM, New York, NY, 43\u201352. DOI: 10.1145\/3319502.3374832"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.2514\/6.2013-4806"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1518\/hfes.46.1.50_30392"},{"key":"e_1_3_3_27_2","first-page":"177","volume-title":"Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI \u201919)","author":"Li Mengyao","year":"2019","unstructured":"Mengyao Li, Brittany E. Holthausen, Rachel E. Stuck, and Bruce N. Walker. 2019. No risk no trust: Investigating perceived risk in highly automated driving. In Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI \u201919). ACM, New York, NY, 177\u2013185. DOI: 10.1145\/3342197.3344525"},{"key":"e_1_3_3_28_2","unstructured":"Ruisen Liu Manisha Natarajan and Matthew Gombolay. 2020. Human-robot team coordination with dynamic and latent human task proficiencies: Scheduling with learning curves. arXiv:2007.01921. Retrieved from https:\/\/arxiv.org\/abs\/2007.01921"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11238-006-9014-2"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.5465\/amr.1995.9508080335"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1006\/game.1995.1023"},{"key":"e_1_3_3_32_2","first-page":"956","volume-title":"Proceedings of the 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN \u201920)","author":"Mou Wenxuan","year":"2020","unstructured":"Wenxuan Mou, Martina Ruocco, Debora Zanatto, and Angelo Cangelosi. 2020. When would you trust a robot? A study on trust and theory of mind in human-robot interactions. In Proceedings of the 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN \u201920), 956\u2013962. DOI: 10.1109\/RO-MAN47096.2020.9223551"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0022-2496(02)00028-7"},{"key":"e_1_3_3_34_2","doi-asserted-by":"crossref","first-page":"3299","DOI":"10.1109\/ICRA48506.2021.9560871","volume-title":"Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA \u201921)","author":"Novin Roya Sabbagh","year":"2021","unstructured":"Roya Sabbagh Novin, Amir Yazdani, Andrew Merryweather, and Tucker Hermans. 2021. Risk-aware decision making for service robots to minimize risk of patient falls in hospitals. In Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA \u201921), 3299\u20133305. DOI: 10.1109\/ICRA48506.2021.9560871"},{"key":"e_1_3_3_35_2","first-page":"11","article-title":"The neural basis of always looking on the bright side","volume":"8","author":"O\u2019Sullivan Owen","year":"2015","unstructured":"Owen O\u2019Sullivan. 2015. The neural basis of always looking on the bright side. Dialogues in Philosophy, Mental and Neuro Sciences 8 (Nov. 2015), 11\u201315.","journal-title":"Dialogues in Philosophy, Mental and Neuro Sciences"},{"key":"e_1_3_3_36_2","first-page":"1406","volume-title":"Proceedings of the 33rd International Conference on Machine Learning","volume":"48","author":"Prashanth L. A.","year":"2016","unstructured":"L. A. Prashanth, Cheng Jie, Michael Fu, Steve Marcus, and Csaba Szepesvari. 2016. Cumulative prospect theory meets reinforcement learning: Prediction and control. In Proceedings of the 33rd International Conference on Machine Learning. Maria Florina Balcan and Kilian Q. Weinberger (Eds.), Proceedings of Machine Learning Research, Vol. 48, PMLR, New York, NY, 1406\u20131415. Retrieved from https:\/\/proceedings.mlr.press\/v48\/la16.html"},{"key":"e_1_3_3_37_2","doi-asserted-by":"crossref","first-page":"1528","DOI":"10.1109\/CDC45484.2021.9683261","volume-title":"Proceedings of the 2021 60th IEEE Conference on Decision and Control (CDC \u201921)","author":"Ramasubramanian Bhaskar","year":"2021","unstructured":"Bhaskar Ramasubramanian, Luyao Niu, Andrew Clark, and Radha Poovendran. 2021. Reinforcement learning beyond expectation. In Proceedings of the 2021 60th IEEE Conference on Decision and Control (CDC \u201921), 1528\u20131535. DOI: 10.1109\/CDC45484.2021.9683261"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.49.1.95"},{"key":"e_1_3_3_39_2","first-page":"227","volume-title":"Proceedings of the 9th International Conference on Human-Agent Interaction (HAI \u201921)","author":"Ruocco Martina","year":"2021","unstructured":"Martina Ruocco, Wenxuan Mou, Angelo Cangelosi, Caroline Jay, and Debora Zanatto. 2021. Theory of mind improves human\u2019s trust in an iterative human-robot game. In Proceedings of the 9th International Conference on Human-Agent Interaction (HAI \u201921). ACM, New York, NY, 227\u2013234. DOI: 10.1145\/3472307.3484176"},{"key":"e_1_3_3_40_2","first-page":"324","volume-title":"Proceedings of the Human Factors and Ergonomics Society Annual Meeting","volume":"61","author":"Satterfield Kelly","year":"2017","unstructured":"Kelly Satterfield, Carryl Baldwin, Ewart de Visser, and Tyler Shaw. 2017. The influence of risky conditions in trust in autonomous systems. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 61, 1 (2017), 324\u2013328. DOI: 10.1177\/1541931213601562"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-7668-0_10"},{"key":"e_1_3_3_42_2","first-page":"9","volume-title":"Proceedings of the 6th IEEE International Workshop on Robot and Human Communication (RO-MAN \u201997 SENDAI)","author":"Sheridan T. B.","year":"1997","unstructured":"T. B. Sheridan. 1997. Eight ultimate challenges of human-robot communication. In Proceedings of the 6th IEEE International Workshop on Robot and Human Communication (RO-MAN \u201997 SENDAI), 9\u201314. DOI: 10.1109\/ROMAN.1997.646944"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2023.119475"},{"key":"e_1_3_3_44_2","first-page":"4329","volume-title":"Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC \u201919)","author":"Sun Liting","year":"2019","unstructured":"Liting Sun, Wei Zhan, Yeping Hu, and Masayoshi Tomizuka. 2019. Interpretable modelling of driving behaviors in interactive driving scenarios based on cumulative prospect theory. In Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC \u201919), 4329\u20134335. DOI: 10.1109\/ITSC.2019.8916944"},{"key":"e_1_3_3_45_2","volume-title":"Introduction to Reinforcement Learning","author":"Sutton Richard S.","year":"1998","unstructured":"Richard S. Sutton and Andrew G. Barto. 1998. Introduction to Reinforcement Learning. Vol. 135. MIT press Cambridge."},{"key":"e_1_3_3_46_2","unstructured":"Vincent Y. F. Tan Prashanth L. A. and Krishna Jagannathan. 2022. A survey of risk-aware multi-armed bandits. arXiv:2205.05843. Retrieved from https:\/\/arxiv.org\/abs\/2205.05843"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.7455683"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00122574"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1037\/dec0000141"}],"container-title":["ACM Transactions on Human-Robot Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3706068","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3706068","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:18:14Z","timestamp":1750295894000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3706068"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,22]]},"references-count":48,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,6,30]]}},"alternative-id":["10.1145\/3706068"],"URL":"https:\/\/doi.org\/10.1145\/3706068","relation":{},"ISSN":["2573-9522"],"issn-type":[{"value":"2573-9522","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,22]]},"assertion":[{"value":"2024-02-23","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-11-02","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-01-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}