{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,19]],"date-time":"2026-02-19T14:06:08Z","timestamp":1771509968924,"version":"3.50.1"},"reference-count":46,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2025,3,17]],"date-time":"2025-03-17T00:00:00Z","timestamp":1742169600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Hum.-Robot Interact."],"published-print":{"date-parts":[[2025,6,30]]},"abstract":"<jats:p>\n            Multi-task missions for unmanned aerial vehicles (UAVs) involving inspection and landing tasks are challenging for novice pilots due to the difficulties associated with depth perception and the control interface. We propose a shared autonomy system, alongside supplementary information displays, to assist pilots to successfully complete multi-task missions without any pilot training. Our approach comprises of three modules: (1) a perception module that encodes visual information onto a latent representation, (2) a policy module that augments pilot\u2019s actions, and (3) an information augmentation module that provides additional information to the pilot. The policy module is trained in simulation with simulated users and transferred to the real world without modification in a user study (\n            <jats:inline-formula content-type=\"math\/tex\">\n              <jats:tex-math notation=\"LaTeX\" version=\"MathJax\">\\(n=29\\)<\/jats:tex-math>\n            <\/jats:inline-formula>\n            ), alongside alternative supplementary information schemes including learnt red\/green light feedback cues and an augmented reality display. The pilot\u2019s intent is unknown to the policy module and is inferred from the pilot\u2019s input and UAV\u2019s states. The assistant increased task success rate for the landing and inspection tasks from 16.67% and 54.29%, respectively, to 95.59% and 96.22%. With the assistant, inexperienced pilots achieved similar performance to experienced pilots. Red\/green light feedback cues reduced the required time by 19.53% and trajectory length by 17.86% for the inspection task, where participants rated it as their preferred condition due to the intuitive interface and providing reassurance. This work demonstrates that simple user models can train shared autonomy systems in simulation, and transfer to physical tasks to estimate user intent and provide effective assistance and information to the pilot.\n          <\/jats:p>","DOI":"10.1145\/3712712","type":"journal-article","created":{"date-parts":[[2025,1,20]],"date-time":"2025-01-20T13:12:21Z","timestamp":1737378741000},"page":"1-37","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["From Novice to Skilled: RL-Based Shared Autonomy Communicating with Pilots in UAV Multi-Task Missions"],"prefix":"10.1145","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6898-4945","authenticated-orcid":false,"given":"Kal","family":"Backman","sequence":"first","affiliation":[{"name":"Monash University, Clayton, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4169-2141","authenticated-orcid":false,"given":"Dana","family":"Kuli\u0107","sequence":"additional","affiliation":[{"name":"Monash University, Clayton, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7044-3729","authenticated-orcid":false,"given":"Hoam","family":"Chung","sequence":"additional","affiliation":[{"name":"Monash University, Clayton, Australia"}]}],"member":"320","published-online":{"date-parts":[[2025,3,17]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMC.2021.3051273"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.3390\/s17102234"},{"key":"e_1_3_2_4_2","doi-asserted-by":"crossref","first-page":"83","DOI":"10.3389\/fnbot.2018.00083","article-title":"System transparency in shared autonomy:","volume":"12","author":"Alonso Victoria","year":"2018","unstructured":"Victoria Alonso and Paloma de la Puente. 2018. System transparency in shared autonomy: A mini review. Frontiers in Neurorobotics 12 (2018), 83.","journal-title":"A mini review. Frontiers in Neurorobotics"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10514-023-10143-3"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2021.3062572"},{"key":"e_1_3_2_7_2","first-page":"3392","volume-title":"Proceedings of the 2022 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)","author":"Bartolomei Luca","year":"2022","unstructured":"Luca Bartolomei, Yves Kompis, Lucas Teixeira, and Margarita Chli. 2022. Autonomous emergency landing for multicopters using deep reinforcement learning. In Proceedings of the 2022 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), 3392\u20133399."},{"key":"e_1_3_2_8_2","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1145\/3434073.3444645","volume-title":"Proceedings of the 2021 ACM\/IEEE International Conference on Human-Robot Interaction","author":"Bevins Alisha","year":"2021","unstructured":"Alisha Bevins and Brittany A. Duncan. 2021. Aerial flight paths for communication: How participants perceive and intend to respond to drone movements. In Proceedings of the 2021 ACM\/IEEE International Conference on Human-Robot Interaction, 16\u201323."},{"key":"e_1_3_2_9_2","first-page":"1","volume-title":"Proceedings of the 7th ACM International Symposium on Pervasive Displays","author":"Brock Anke M.","year":"2018","unstructured":"Anke M. Brock, Julia Chatain, Michelle Park, Tommy Fang, Martin Hachet, James A. Landay, and Jessica R. Cauchard. 2018. Flymap: Interacting with maps projected from a drone. In Proceedings of the 7th ACM International Symposium on Pervasive Displays, 1\u20139."},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS45743.2020.9340964"},{"key":"e_1_3_2_11_2","first-page":"1559","article-title":"Neural networks for incremental dimensionality reduced reinforcement learning","author":"Curran W.","year":"2017","unstructured":"W. Curran, R. Pocius, and W. D. Smart. 2017. Neural networks for incremental dimensionality reduced reinforcement learning. In Proceedings of the 2017 IEEE\/RSJ IROS, 1559\u20131565.","journal-title":"Proceedings of the 2017 IEEE\/RSJ IROS"},{"issue":"4","key":"e_1_3_2_12_2","doi-asserted-by":"crossref","first-page":"1385","DOI":"10.3390\/s21041385","article-title":"Learning-based autonomous UAV system for electrical and mechanical & device inspection","volume":"21","author":"Feng Yurong","year":"2021","unstructured":"Yurong Feng, Kwaiwa Tse, Shengyang Chen, Chih-Yung Wen, and Boyang Li. 2021. Learning-based autonomous UAV system for electrical and mechanical & device inspection. Sensors 21, 4 (2021), 1385.","journal-title":"Sensors"},{"issue":"3","key":"e_1_3_2_13_2","first-page":"109","article-title":"Survey of sUAS unintended flight termination as depicted in Internet video","volume":"5","author":"Fernando Anthony V.","year":"2017","unstructured":"Anthony V. Fernando. 2017. Survey of sUAS unintended flight termination as depicted in Internet video. Journal of Unmanned Vehicle Systems 5, 3 (2017), 109\u2013114.","journal-title":"Journal of Unmanned Vehicle Systems"},{"key":"e_1_3_2_14_2","first-page":"1587","volume-title":"Proceedings of the 35th International Conference on Machine Learning","volume":"80","author":"Fujimoto Scott","year":"2018","unstructured":"Scott Fujimoto, Herke van Hoof, and David Meger. 2018. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, Vol. 80, 1587\u20131596."},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0166-4115(08)62386-9"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3171221.3171251"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445495"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364918776060"},{"key":"e_1_3_2_19_2","volume-title":"Proceedings of the Robotics: Science and Systems Conference","author":"Jeon Hong Jun","year":"2020","unstructured":"Hong Jun Jeon, Dylan P. Losey, and Dorsa Sadigh. 2020. Shared autonomy with learned latent actions. In Proceedings of the Robotics: Science and Systems Conference."},{"key":"e_1_3_2_20_2","doi-asserted-by":"crossref","first-page":"287","DOI":"10.1016\/j.wem.2019.03.004","article-title":"Using an unmanned aircraft system (drone) to conduct a complex high altitude search and rescue operation: A case study","volume":"30","author":"J. N McRae","year":"2019","unstructured":"McRae J. N, Gay C. J, Nielsen B. M, and Hunt A. P. 2019. Using an unmanned aircraft system (drone) to conduct a complex high altitude search and rescue operation: A case study. Wilderness & Environmental Medicine 30 (2019), 287\u2013290.","journal-title":"Wilderness & Environmental Medicine"},{"issue":"1","key":"e_1_3_2_21_2","doi-asserted-by":"crossref","first-page":"2","DOI":"10.7771\/2159-6670.1117","article-title":"Synthesis of unmanned aircraft systems safety reports","volume":"5","author":"Joslin Robert","year":"2015","unstructured":"Robert Joslin. 2015. Synthesis of unmanned aircraft systems safety reports. Journal of Aviation Technology and Engineering 5, 1 (2015), 2.","journal-title":"Journal of Aviation Technology and Engineering"},{"issue":"11","key":"e_1_3_2_22_2","doi-asserted-by":"crossref","first-page":"0355","DOI":"10.3390\/drones6110355","article-title":"A comparative study of bridge inspection and condition assessment between manpower and a UAS","volume":"6","author":"Kim In-Ho","year":"2022","unstructured":"In-Ho Kim, Sungsik Yoon, Jin Hwan Lee, Sungwook Jung, Soojin Cho, and Hyung-Jo Jung. 2022. A comparative study of bridge inspection and condition assessment between manpower and a UAS. Drones 6, 11 (2022) 0355.","journal-title":"Drones"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.3390\/drones6020043"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/THMS.2017.2647882"},{"key":"e_1_3_2_25_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Lillicrap Timothy P.","year":"2016","unstructured":"Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2016. Continuous control with deep reinforcement learning. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2018.2795643"},{"key":"e_1_3_2_27_2","doi-asserted-by":"crossref","first-page":"7475","DOI":"10.1109\/TSMC.2023.3299237","article-title":"Vision-based plane estimation and following for building inspection with autonomous UAV","volume":"53","author":"Lyu Yang","year":"2023","unstructured":"Yang Lyu, Muqing Cao, Shenghai Yuan, and Lihua Xie. 2023. Vision-based plane estimation and following for building inspection with autonomous UAV. IEEE Transactions on Systems, Man, and Cybernetics: Systems 53 (2023), 7475\u20137488.","journal-title":"IEEE Transactions on Systems, Man, and Cybernetics: Systems"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.3390\/drones6110347"},{"key":"e_1_3_2_29_2","first-page":"8894055","article-title":"Research on determining the inspection point of multirotor UAV power tower","volume":"2021","author":"Neagu Adrian","year":"2021","unstructured":"Adrian Neagu, Hongxing Wang, Hang Zhou, Haoran Liu, Zheng Huang, and Mingduan Feng. 2021. Research on determining the inspection point of multirotor UAV power tower. Mathematical Problems in Engineering 2021 (2021), 8894055.","journal-title":"Mathematical Problems in Engineering"},{"key":"e_1_3_2_30_2","doi-asserted-by":"crossref","first-page":"618\u2013634","DOI":"10.1177\/0278364917690593","article-title":"Human-robot mutual adaptation in collaborative tasks: Models and experiments","volume":"36","author":"Nikolaidis S.","year":"2017","unstructured":"S. Nikolaidis, D. Hsu, and S. Srinivasa. 2017. Human-robot mutual adaptation in collaborative tasks: Models and experiments. The International Journal of Robotics Research 36 (2017), 618\u2013634.","journal-title":"The International Journal of Robotics Research"},{"key":"e_1_3_2_31_2","first-page":"1","volume-title":"Proceedings of the 2013 IEEE Aerospace Conference","author":"Nikolic Janosch","year":"2013","unstructured":"Janosch Nikolic, Michael Burri, Joern Rehder, Stefan Leutenegger, Christoph Huerzeler, and Roland Siegwart. 2013. A UAV system for inspection of industrial facilities. In Proceedings of the 2013 IEEE Aerospace Conference, 1\u20138."},{"key":"e_1_3_2_32_2","doi-asserted-by":"crossref","first-page":"1066","DOI":"10.1109\/ICUAS.2017.7991349","volume-title":"Proceedings of the 2017 International Conference on Unmanned Aircraft Systems (ICUAS)","author":"Perez-Grau F. J.","year":"2017","unstructured":"F. J. Perez-Grau, R. Ragel, F. Caballero, A. Viguria, and A. Ollero. 2017. Semi-autonomous teleoperation of UAVs in search and rescue scenarios. In Proceedings of the 2017 International Conference on Unmanned Aircraft Systems (ICUAS), 1066\u20131074."},{"issue":"1","key":"e_1_3_2_33_2","doi-asserted-by":"crossref","first-page":"0008","DOI":"10.3390\/robotics9010008","article-title":"Sim-to-real quadrotor landing via sequential deep Q-networks and domain randomization","volume":"9","author":"Polvara Riccardo","year":"2020","unstructured":"Riccardo Polvara, Massimiliano Patacchiola, Marc Hanheide, and Gerhard Neumann. 2020. Sim-to-real quadrotor landing via sequential deep Q-networks and domain randomization. Robotics 9, 1 (2020) 0008.","journal-title":"Robotics"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2018.XIV.005"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10846-018-0891-8"},{"issue":"9","key":"e_1_3_2_36_2","article-title":"Inspection of pole-like structures using a visual-inertial aided VTOL platform with shared autonomy","volume":"15","author":"Sa Inkyu","year":"2015","unstructured":"Inkyu Sa, Stefan Hrabar, and Peter Corke. 2015. Inspection of pole-like structures using a visual-inertial aided VTOL platform with shared autonomy. Sensors 15, 9 (2015), 22003\u201322048.","journal-title":"Sensors"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-020-00261-3"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-67361-5_40"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.3389\/frobt.2017.00011"},{"issue":"4","key":"e_1_3_2_40_2","doi-asserted-by":"crossref","first-page":"0087","DOI":"10.3390\/drones6040087","article-title":"Automating aircraft scanning for inspection or 3D model creation with a UAV and optimal path planning","volume":"6","author":"Sun Yufeng","year":"2022","unstructured":"Yufeng Sun and Ou Ma. 2022. Automating aircraft scanning for inspection or 3D model creation with a UAV and optimal path planning. Drones 6, 4 (2022), 0087.","journal-title":"Drones"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/2696454.2696475"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3171221.3171253"},{"issue":"1","key":"e_1_3_2_43_2","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1080\/21642583.2018.1477634","article-title":"Autonomous robots for harsh environments: A holistic overview of current solutions and ongoing challenges","volume":"6","author":"Wong Cuebong","year":"2018","unstructured":"Cuebong Wong, Erfu Yang, Xiu-Tian Yan, and Dongbing Gu. 2018. Autonomous robots for harsh environments: A holistic overview of current solutions and ongoing challenges. Systems Science & Control Engineering 6, 1 (2018), 213\u2013219.","journal-title":"Systems Science & Control Engineering"},{"issue":"8","key":"e_1_3_2_44_2","doi-asserted-by":"crossref","first-page":"04022070","DOI":"10.1061\/(ASCE)CO.1943-7862.0002322","article-title":"Benchmarking human versus robot performance in emergency structural inspection","volume":"148","author":"Xia Pengxiang","year":"2022","unstructured":"Pengxiang Xia, Fang Xu, Tianyu Zhou, and Jing Du. 2022. Benchmarking human versus robot performance in emergency structural inspection. Journal of Construction Engineering and Management 148, 8 (2022), 04022070.","journal-title":"Journal of Construction Engineering and Management"},{"issue":"19","key":"e_1_3_2_45_2","first-page":"95630","article-title":"UAV autonomous tracking and landing based on deep reinforcement learning strategy","volume":"20","author":"Xie Jingyi","year":"2020","unstructured":"Jingyi Xie, Xiaodong Peng, Haijiao Wang, Wenlong Niu, and Xiao Zheng. 2020. UAV autonomous tracking and landing based on deep reinforcement learning strategy. Sensors 20, 19 (2020), 95630.","journal-title":"Sensors"},{"key":"e_1_3_2_46_2","first-page":"332","article-title":"Shared autonomy of a robotic manipulator for grasping under human intent uncertainty using POMDPs","volume":"40","author":"Yow J.-Anne","year":"2023","unstructured":"J.-Anne Yow, Neha Priyadarshini Garg, and Wei Tech Ang. 2023. Shared autonomy of a robotic manipulator for grasping under human intent uncertainty using POMDPs. IEEE Transactions on Robotics 40 (2023), 332\u2013350.","journal-title":"IEEE Transactions on Robotics"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48506.2021.9560991"}],"container-title":["ACM Transactions on Human-Robot Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3712712","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3712712","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:18:09Z","timestamp":1750295889000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3712712"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,17]]},"references-count":46,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,6,30]]}},"alternative-id":["10.1145\/3712712"],"URL":"https:\/\/doi.org\/10.1145\/3712712","relation":{},"ISSN":["2573-9522"],"issn-type":[{"value":"2573-9522","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,17]]},"assertion":[{"value":"2023-10-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-12","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}