{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,23]],"date-time":"2026-04-23T07:35:29Z","timestamp":1776929729225,"version":"3.51.2"},"reference-count":47,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2020,1,30]],"date-time":"2020-01-30T00:00:00Z","timestamp":1580342400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001459","name":"Singapore Ministry of Education","doi-asserted-by":"crossref","award":["MOE2016-T2-2-068"],"award-info":[{"award-number":["MOE2016-T2-2-068"]}],"id":[{"id":"10.13039\/501100001459","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000006","name":"Office of Naval Research","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100000006","id-type":"DOI","asserted-by":"crossref"}]},{"name":"U.S. National Institutes of Health R01","award":["R01EB019335"],"award-info":[{"award-number":["R01EB019335"]}]},{"name":"U.S. National Science Foundation NRI","award":["1637748"],"award-info":[{"award-number":["1637748"]}]},{"name":"U.S. National Science Foundation CPS","award":["1544797"],"award-info":[{"award-number":["1544797"]}]},{"DOI":"10.13039\/501100001352","name":"National University of Singapore","doi-asserted-by":"crossref","award":["R-252-000-587-112"],"award-info":[{"award-number":["R-252-000-587-112"]}],"id":[{"id":"10.13039\/501100001352","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Hum.-Robot Interact."],"published-print":{"date-parts":[[2020,6,30]]},"abstract":"<jats:p>Trust in autonomy is essential for effective human-robot collaboration and user adoption of autonomous systems such as robot assistants. This article introduces a computational model that integrates trust into robot decision making. Specifically, we learn from data a partially observable Markov decision process (POMDP) with human trust as a latent variable. The trust-POMDP model provides a principled approach for the robot to (i) infer the trust of a human teammate through interaction, (ii) reason about the effect of its own actions on human trust, and (iii) choose actions that maximize team performance over the long term. We validated the model through human subject experiments on a table clearing task in simulation (201 participants) and with a real robot (20 participants). In our studies, the robot builds human trust by manipulating low-risk objects first. Interestingly, the robot sometimes fails intentionally to modulate human trust and achieve the best team performance. These results show that the trust-POMDP calibrates trust to improve human-robot team performance over the long term. Further, they highlight that maximizing trust alone does not always lead to the best performance.<\/jats:p>","DOI":"10.1145\/3359616","type":"journal-article","created":{"date-parts":[[2020,4,4]],"date-time":"2020-04-04T00:54:52Z","timestamp":1585961692000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":130,"title":["Trust-Aware Decision Making for Human-Robot Collaboration"],"prefix":"10.1145","volume":"9","author":[{"given":"Min","family":"Chen","sequence":"first","affiliation":[{"name":"National University of Singapore, Singapore"}]},{"given":"Stefanos","family":"Nikolaidis","sequence":"additional","affiliation":[{"name":"University of Southern California, Los Angeles, CA"}]},{"given":"Harold","family":"Soh","sequence":"additional","affiliation":[{"name":"National University of Singapore, Singapore"}]},{"given":"David","family":"Hsu","sequence":"additional","affiliation":[{"name":"National University of Singapore, Singapore"}]},{"given":"Siddhartha","family":"Srinivasa","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA"}]}],"member":"320","published-online":{"date-parts":[[2020,1,30]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the IEEE International Conference on Robotics and Automation.","author":"Bai H. Y.","unstructured":"H. Y. Bai , S. J. Cai , D. Hsu , and W. S. Lee . 2015. Intention-aware online POMDP planning for autonomous driving in a crowd . In Proceedings of the IEEE International Conference on Robotics and Automation. H. Y. Bai, S. J. Cai, D. Hsu, and W. S. Lee. 2015. Intention-aware online POMDP planning for autonomous driving in a crowd. In Proceedings of the IEEE International Conference on Robotics and Automation."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2015.7139219"},{"key":"e_1_2_1_3_1","volume-title":"Emilio Frazzoli, David Hsu, Wee Sun Lee, and Daniela Rus.","author":"Bandyopadhyay Tirthankar","year":"2013","unstructured":"Tirthankar Bandyopadhyay , Kok Sung Won , Emilio Frazzoli, David Hsu, Wee Sun Lee, and Daniela Rus. 2013 . Intention-aware motion planning. In Algorithmic Foundations of Robotics X. Springer , 475--491. Tirthankar Bandyopadhyay, Kok Sung Won, Emilio Frazzoli, David Hsu, Wee Sun Lee, and Daniela Rus. 2013. Intention-aware motion planning. In Algorithmic Foundations of Robotics X. Springer, 475--491."},{"key":"e_1_2_1_4_1","volume-title":"The Logic and Limits of Trust\/Bernard Barber","author":"Barber Bernard","unstructured":"Bernard Barber . 1983. The Logic and Limits of Trust\/Bernard Barber . Rutgers University Press , New Brunswick, NJ . Bernard Barber. 1983. The Logic and Limits of Trust\/Bernard Barber. Rutgers University Press, New Brunswick, NJ."},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the 21st European Conference on Artificial Intelligence. 45--50","author":"Barrett Samuel","year":"2014","unstructured":"Samuel Barrett , Noa Agmon , Noam Hazon , Sarit Kraus , and Peter Stone . 2014 . Communicating with unknown teammates . In Proceedings of the 21st European Conference on Artificial Intelligence. 45--50 . Samuel Barrett, Noa Agmon, Noam Hazon, Sarit Kraus, and Peter Stone. 2014. Communicating with unknown teammates. In Proceedings of the 21st European Conference on Artificial Intelligence. 45--50."},{"key":"e_1_2_1_6_1","first-page":"1","article-title":"Stan: A probabilistic programming language","volume":"20","author":"Carpenter Bob","year":"2016","unstructured":"Bob Carpenter , Andrew Gelman , Matt Hoffman , Daniel Lee , Ben Goodrich , Michael Betancourt , Michael A. Brubaker , Jiqiang Guo , Peter Li , and Allen Riddell . 2016 . Stan: A probabilistic programming language . Journal of Statistical Software 20 (2016), 1 -- 37 . Bob Carpenter, Andrew Gelman, Matt Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Michael A. Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. 2016. Stan: A probabilistic programming language. Journal of Statistical Software 20 (2016), 1--37.","journal-title":"Journal of Statistical Software"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3171221.3171264"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature04766"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.5555\/2447556.2447663"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2157689.2157702"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the Robotics: Science and Systems Conference. 10","author":"Dragan Anca D.","unstructured":"Anca D. Dragan , Rachel M. Holladay , and Siddhartha S. Srinivasa . 2014. An analysis of deceptive robot motion . In Proceedings of the Robotics: Science and Systems Conference. 10 . Anca D. Dragan, Rachel M. Holladay, and Siddhartha S. Srinivasa. 2014. An analysis of deceptive robot motion. In Proceedings of the Robotics: Science and Systems Conference. 10."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 2013 8th ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201913)","author":"Dragan Anca D.","unstructured":"Anca D. Dragan , Kenton C. T. Lee , and Siddhartha S. Srinivasa . 2013. Legibility and predictability of robot motion . In Proceedings of the 2013 8th ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201913) . IEEE, Los Alamitos, CA, 301--308. Anca D. Dragan, Kenton C. T. Lee, and Siddhartha S. Srinivasa. 2013. Legibility and predictability of robot motion. In Proceedings of the 2013 8th ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201913). IEEE, Los Alamitos, CA, 301--308."},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the 24th International Joint Conference on Artificial Intelligence. 4261--4267","author":"Floyd Michael W.","unstructured":"Michael W. Floyd , Michael Drinkwater , and David W. Aha . 2015. Trust-guided behavior adaptation using case-based reasoning . In Proceedings of the 24th International Joint Conference on Artificial Intelligence. 4261--4267 . Michael W. Floyd, Michael Drinkwater, and David W. Aha. 2015. Trust-guided behavior adaptation using case-based reasoning. In Proceedings of the 24th International Joint Conference on Artificial Intelligence. 4261--4267."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2015.XI.043"},{"key":"e_1_2_1_16_1","first-page":"185","article-title":"The centrality of interpersonal trust in group processes","volume":"131","author":"Golembiewski Robert T.","year":"1975","unstructured":"Robert T. Golembiewski and Mark McConkie . 1975 . The centrality of interpersonal trust in group processes . Theories of Group Processes 131 (1975), 185 . Robert T. Golembiewski and Mark McConkie. 1975. The centrality of interpersonal trust in group processes. Theories of Group Processes 131 (1975), 185.","journal-title":"Theories of Group Processes"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/KBSE.1996.552822"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the Robotics: Science and Systems Workshop on Human Robot Collaboration","volume":"381","author":"Hoffman Guy","year":"2013","unstructured":"Guy Hoffman . 2013 . Evaluating fluency in human-robot collaboration . In Proceedings of the Robotics: Science and Systems Workshop on Human Robot Collaboration , Vol. 381 . 1--8. Guy Hoffman. 2013. Evaluating fluency in human-robot collaboration. In Proceedings of the Robotics: Science and Systems Workshop on Human Robot Collaboration, Vol. 381. 1--8."},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","unstructured":"Shervin Javdani Siddhartha S. Srinivasa and J. Andrew Bagnell. 2015. Shared autonomy via hindsight optimization. arXIv:1503.07619.  Shervin Javdani Siddhartha S. Srinivasa and J. Andrew Bagnell. 2015. Shared autonomy via hindsight optimization. arXIv:1503.07619.","DOI":"10.15607\/RSS.2015.XI.032"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327566IJCE0401_04"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0026818"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/1643275.1643301"},{"key":"e_1_2_1_23_1","volume-title":"Tyler","author":"Kramer Roderick M.","year":"1995","unstructured":"Roderick M. Kramer and Tom R . Tyler . 1995 . Trust in Organizations : Frontiers of Theory and Research. Sage Publications . Roderick M. Kramer and Tom R. Tyler. 1995. Trust in Organizations: Frontiers of Theory and Research. Sage Publications."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2008.IV.009"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1080\/00140139208967392"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1518\/hfes.46.1.50.30392"},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the 8th Artificial Intelligence and Interactive Digital Entertainment Conference.","author":"Macindoe Owen","year":"2012","unstructured":"Owen Macindoe , Leslie Pack Kaelbling , and Tom\u00e1s Lozano-P\u00e9rez . 2012 . POMCoP: Belief space planning for sidekicks in cooperative games . In Proceedings of the 8th Artificial Intelligence and Interactive Digital Entertainment Conference. Owen Macindoe, Leslie Pack Kaelbling, and Tom\u00e1s Lozano-P\u00e9rez. 2012. POMCoP: Belief space planning for sidekicks in cooperative games. In Proceedings of the 8th Artificial Intelligence and Interactive Digital Entertainment Conference."},{"key":"e_1_2_1_28_1","volume-title":"Cannon-Bowers","author":"Mathieu John E.","year":"2000","unstructured":"John E. Mathieu , Tonia S. Heffner , Gerald F. Goodwin , Eduardo Salas , and Janis A . Cannon-Bowers . 2000 . The influence of shared mental models on team process and performance.Journal of Applied Psychology 85, 2 (2000), 273. John E. Mathieu, Tonia S. Heffner, Gerald F. Goodwin, Eduardo Salas, and Janis A. Cannon-Bowers. 2000. The influence of shared mental models on team process and performance.Journal of Applied Psychology 85, 2 (2000), 273."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.5465\/amr.1995.9508080335"},{"key":"e_1_2_1_30_1","volume-title":"Trust in and Use of Automatic Controllers in a Supervisory Process Control Task","author":"Muir Bonnie Marlene","unstructured":"Bonnie Marlene Muir . 1990. Operators \u2019 Trust in and Use of Automatic Controllers in a Supervisory Process Control Task . University of Toronto . Bonnie Marlene Muir. 1990. Operators\u2019 Trust in and Use of Automatic Controllers in a Supervisory Process Control Task. University of Toronto."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1177\/0278364917690593"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/HRI.2016.7451736"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2696454.2696455"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1177\/0278364910369861"},{"key":"e_1_2_1_35_1","volume-title":"Robotics Research","author":"Pierson Alyssa","unstructured":"Alyssa Pierson and Mac Schwager . 2016. Adaptive inter-robot trust for robust multi-robot sensor coverage . In Robotics Research . Springer , 167--183. Alyssa Pierson and Mac Schwager. 2016. Adaptive inter-robot trust for robust multi-robot sensor coverage. In Robotics Research. Springer, 167--183."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2014.6906590"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.49.1.95"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.5555\/1622673.1622690"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the Robotics: Science and Systems Conference (RSS\u201916)","author":"Sadigh Dorsa","unstructured":"Dorsa Sadigh , Shankar Sastry , Sanjit A. Seshia , and Anca D. Dragan . 2016. Planning for autonomous cars that leverages effects on human actions . In Proceedings of the Robotics: Science and Systems Conference (RSS\u201916) . Dorsa Sadigh, Shankar Sastry, Sanjit A. Seshia, and Anca D. Dragan. 2016. Planning for autonomous cars that leverages effects on human actions. In Proceedings of the Robotics: Science and Systems Conference (RSS\u201916)."},{"key":"e_1_2_1_40_1","unstructured":"D. Silver and J. Veness. 2010. Monte-Carlo planning in large POMDPs. In Advances in Neural Information Processing Systems.  D. Silver and J. Veness. 2010. Monte-Carlo planning in large POMDPs. In Advances in Neural Information Processing Systems."},{"key":"e_1_2_1_41_1","unstructured":"Harold Soh Pan Shu Min Chen and David Hsu. 2018. The transfer of human trust in robot capabilities across tasks. arXiv:1807.01866.  Harold Soh Pan Shu Min Chen and David Hsu. 2018. The transfer of human trust in robot capabilities across tasks. arXiv:1807.01866."},{"key":"e_1_2_1_42_1","volume-title":"DESPOT: Online POMDP planning with regularization. In Advances in Neural Information Processing Systems.","author":"Somani A.","year":"2013","unstructured":"A. Somani , N. Ye , D. Hsu , and W. S. Lee . 2013 . DESPOT: Online POMDP planning with regularization. In Advances in Neural Information Processing Systems. A. Somani, N. Ye, D. Hsu, and W. S. Lee. 2013. DESPOT: Online POMDP planning with regularization. In Advances in Neural Information Processing Systems."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-014-0231-5"},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the 2016 11th ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201916)","author":"Wang Ning","unstructured":"Ning Wang , David V. Pynadath , and Susan G. Hill . 2016. Trust calibration within a human-robot team: Comparing automatically generated explanations . In Proceedings of the 2016 11th ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201916) . IEEE, Los Alamitos, CA, 109--116. Ning Wang, David V. Pynadath, and Susan G. Hill. 2016. Trust calibration within a human-robot team: Comparing automatically generated explanations. In Proceedings of the 2016 11th ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201916). IEEE, Los Alamitos, CA, 109--116."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/HRI.2019.8673084"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2696454.2696492"},{"key":"e_1_2_1_47_1","volume-title":"Robotics Research","author":"Xu Anqi","unstructured":"Anqi Xu and Gregory Dudek . 2016. Towards modeling real-time trust in asymmetric human--robot collaborations . In Robotics Research . Springer , 113--129. Anqi Xu and Gregory Dudek. 2016. Towards modeling real-time trust in asymmetric human--robot collaborations. In Robotics Research. Springer, 113--129."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2909824.3020230"}],"container-title":["ACM Transactions on Human-Robot Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3359616","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3359616","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3359616","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:23:06Z","timestamp":1750202586000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3359616"}},"subtitle":["Model Learning and Planning"],"short-title":[],"issued":{"date-parts":[[2020,1,30]]},"references-count":47,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,6,30]]}},"alternative-id":["10.1145\/3359616"],"URL":"https:\/\/doi.org\/10.1145\/3359616","relation":{},"ISSN":["2573-9522"],"issn-type":[{"value":"2573-9522","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1,30]]},"assertion":[{"value":"2018-08-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-01-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}