{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T15:40:13Z","timestamp":1781797213301,"version":"3.54.5"},"reference-count":61,"publisher":"Association for Computing Machinery (ACM)","issue":"2","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Hum.-Robot Interact."],"published-print":{"date-parts":[[2026,3,31]]},"abstract":"<jats:p>\n                    Learning from Demonstration (LfD) empowers robots to acquire new skills through human demonstrations, making it feasible for everyday users to teach robots. However, the success of learning and generalization heavily depends on the quality of these demonstrations. Consistency is often used to indicate quality in LfD, yet the factors that define this consistency remain underexplored. In this article, we evaluate a comprehensive set of motion data characteristics to determine which consistency measures best predict learning performance. By ensuring demonstration consistency prior to training, we enhance models\u2019 predictive accuracy and generalization to novel scenarios. We validate our approach with two user studies involving participants with diverse levels of robotics expertise. In the first study (\n                    <jats:italic toggle=\"yes\">N\u2009<\/jats:italic>\n                    =\n                    <jats:italic toggle=\"yes\">\u2009<\/jats:italic>\n                    24), users taught a PR2 robot to perform a button-pressing task in a constrained environment, while in the second study (\n                    <jats:italic toggle=\"yes\">N\u2009<\/jats:italic>\n                    =\n                    <jats:italic toggle=\"yes\">\u2009<\/jats:italic>\n                    30), participants trained an UR5 robot on a pick-and-place task. Results show that demonstration consistency significantly impacts success rates in both learning and generalization, with 70% and 89% of task success rates in the two studies predicted using our consistency metrics. Moreover, our metrics estimate generalized performance success rates with 76% and 91% accuracy. These findings suggest that our proposed measures provide an intuitive, practical way to assess demonstration data quality before training, without requiring expert data or algorithm-specific modifications. Our approach offers a systematic way to evaluate demonstration quality, addressing a critical gap in LfD by formalizing consistency metrics that enhance the reliability of robot learning from human demonstrations.\n                  <\/jats:p>","DOI":"10.1145\/3773904","type":"journal-article","created":{"date-parts":[[2025,10,28]],"date-time":"2025-10-28T16:11:44Z","timestamp":1761667904000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Consistency Matters: Defining Demonstration Data Quality Metrics in Robot Learning from Demonstration"],"prefix":"10.1145","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4110-2359","authenticated-orcid":false,"given":"Maram","family":"Sakr","sequence":"first","affiliation":[{"name":"University of British Columbia, Vancouver, British Columbia, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-6680-4270","authenticated-orcid":false,"given":"Juyan","family":"Zhang","sequence":"additional","affiliation":[{"name":"Monash University, Melbourne, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1355-980X","authenticated-orcid":false,"given":"H. F. Machiel Van der","family":"Loos","sequence":"additional","affiliation":[{"name":"Mechanical Engineering, University of British Columbia, Vancouver, British Columbia, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4169-2141","authenticated-orcid":false,"given":"Dana","family":"Kuli\u0107","sequence":"additional","affiliation":[{"name":"Monash University, Clayton, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9639-5291","authenticated-orcid":false,"given":"Elizabeth","family":"Croft","sequence":"additional","affiliation":[{"name":"University of Victoria, Victoria, British Columbia, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,12,22]]},"reference":[{"issue":"6","key":"e_1_3_2_2_2","doi-asserted-by":"crossref","first-page":"3146","DOI":"10.3390\/s23063146","article-title":"Motion smoothness-based assessment of surgical expertise: The importance of selecting proper metrics","volume":"23","author":"Aghazadeh Farzad","year":"2023","unstructured":"Farzad Aghazadeh, Bin Zheng, Mahdi Tavakoli, and Hossein Rouhani. 2023. Motion smoothness-based assessment of surgical expertise: The importance of selecting proper metrics. Sensors 23, 6 (2023), 3146.","journal-title":"Sensors"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0294786"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-024-01164-8"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v35i4.2513"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2008.10.024"},{"key":"e_1_3_2_7_2","first-page":"1732","volume-title":"International Conference on Machine Learning","author":"Beliaev Mark","year":"2022","unstructured":"Mark Beliaev, Andy Shih, Stefano Ermon, Dorsa Sadigh, and Ramtin Pedarsani. 2022. Imitation learning by estimating expertise of demonstrators. In International Conference on Machine Learning. PMLR, 1732\u20131748."},{"key":"e_1_3_2_8_2","first-page":"80375","volume-title":"Advances in Neural Information Processing Systems","volume":"36","author":"Belkhale Suneel","year":"2024","unstructured":"Suneel Belkhale, Yuchen Cui, and Dorsa Sadigh. 2024. Data quality in imitation learning. Advances in Neural Information Processing Systems 36 (2024), 80375\u201380395."},{"key":"e_1_3_2_9_2","doi-asserted-by":"crossref","first-page":"5120","DOI":"10.1109\/IROS58592.2024.10802187","volume-title":"2024 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)","author":"Bilal Muhammad","year":"2024","unstructured":"Muhammad Bilal, Nir Lipovetzky, Denny Oetomo, and Wafa Johal. 2024. Beyond success: Quantifying demonstration quality in learning from demonstration. In 2024 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 5120\u20135127."},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-30301-5_60"},{"key":"e_1_3_2_11_2","first-page":"330","volume-title":"Conference on Robot Learning","author":"Brown Daniel S.","year":"2020","unstructured":"Daniel S. Brown, Wonjoon Goo, and Scott Niekum. 2020. Better-than-demonstrator imitation learning via automatically-ranked demonstrations. In Conference on Robot Learning. PMLR, 330\u2013359."},{"key":"e_1_3_2_12_2","doi-asserted-by":"crossref","first-page":"431","DOI":"10.1145\/2559636.2559675","volume-title":"Proceedings of the 2014 ACM\/IEEE International Conference on Human-Robot Interaction","author":"Cakmak Maya","year":"2014","unstructured":"Maya Cakmak and Leila Takayama. 2014. Teaching people how to teach robots: The effect of instructional materials and dialog design. In Proceedings of the 2014 ACM\/IEEE International Conference on Human-Robot Interaction, 431\u2013438."},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11370-015-0187-9"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1075\/is.8.3.08cal"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364903022005002"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.5555\/2621996"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1177\/02783649241273668"},{"key":"e_1_3_2_18_2","doi-asserted-by":"crossref","first-page":"24634","DOI":"10.1109\/ACCESS.2019.2899751","article-title":"An overview of data quality frameworks","volume":"7","author":"Cichy Corinna","year":"2019","unstructured":"Corinna Cichy and Stefan Rass. 2019. An overview of data quality frameworks. IEEE Access 7 (2019), 24634\u201324648.","journal-title":"IEEE Access"},{"issue":"2","key":"e_1_3_2_19_2","doi-asserted-by":"crossref","first-page":"944","DOI":"10.1109\/TMECH.2014.2326304","article-title":"Closed-loop inverse kinematics for redundant robots: Comparative assessment and two enhancements","volume":"20","author":"Colom\u00e9 Adria","year":"2014","unstructured":"Adria Colom\u00e9 and Carme Torras. 2014. Closed-loop inverse kinematics for redundant robots: Comparative assessment and two enhancements. IEEE\/ASME Transactions on Mechatronics 20, 2 (2014), 944\u2013955.","journal-title":"IEEE\/ASME Transactions on Mechatronics"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.5555\/2447556.2447672"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/HRI.2016.7451754"},{"key":"e_1_3_2_22_2","doi-asserted-by":"crossref","first-page":"3970","DOI":"10.1109\/IROS.2012.6385619","volume-title":"2012 IEEE\/RSJ International Conference on Intelligent Robots and Systems","author":"Flacco Fabrizio","year":"2012","unstructured":"Fabrizio Flacco, Alessandro De Luca, and Oussama Khatib. 2012. Prioritized multi-task motion control of redundant robots under hard joint constraints. In 2012 IEEE\/RSJ International Conference on Intelligent Robots and Systems. IEEE, 3970\u20133977."},{"key":"e_1_3_2_23_2","first-page":"1981","volume-title":"Conference on Robot Learning","author":"Gandhi Kanishk","year":"2023","unstructured":"Kanishk Gandhi, Siddharth Karamcheti, Madeline Liao, and Dorsa Sadigh. 2023. Eliciting compatible demonstrations for multi-human imitation learning. In Conference on Robot Learning. PMLR, 1981\u20131991."},{"issue":"1","key":"e_1_3_2_24_2","first-page":"1","article-title":"Data quality considerations for big data and machine learning: Going beyond data cleaning and transformations","volume":"10","author":"Gudivada Venkat","year":"2017","unstructured":"Venkat Gudivada, Amy Apon, and Junhua Ding. 2017. Data quality considerations for big data and machine learning: Going beyond data cleaning and transformations. International Journal on Advances in Software 10, 1 (2017), 1\u201320.","journal-title":"International Journal on Advances in Software"},{"key":"e_1_3_2_25_2","volume-title":"Data Mining: Concepts and Techniques","author":"Han Jiawei","year":"2012","unstructured":"Jiawei Han, Micheline Kamber, and Jian Pei. 2012. Data Mining: Concepts and Techniques. Morgan Kaufmann Publishers, Waltham."},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.2307\/2346830"},{"key":"e_1_3_2_27_2","first-page":"6840","article-title":"Denoising diffusion probabilistic models","volume":"33","author":"Ho Jonathan","year":"2020","unstructured":"Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33 (2020), 6840\u20136851.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_28_2","unstructured":"ISO\/IEC. 2008. Software engineering\u2014Software product Quality Requirements and Evaluation (SQuaRE)\u2014Data quality model. Technical Report. International Standard under systematic review ISO\/IEC 25012."},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364920946815"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364920946815"},{"key":"e_1_3_2_31_2","volume-title":"Programming by Demonstration vs. Learning from Examples Workshop at ML","author":"Kaiser Michael","year":"1995","unstructured":"Michael Kaiser, Holger Friedrich, and Rudiger Dillmann. 1995. Obtaining good performance from a bad teacher. In Programming by Demonstration vs. Learning from Examples Workshop at ML."},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/PRDC.2015.41"},{"key":"e_1_3_2_33_2","doi-asserted-by":"crossref","unstructured":"Itay Levy Ben Bogin and Jonathan Berant. 2022. Diverse demonstrations improve in-context compositional generalization. arXiv:2212.06800. Retrieved from https:\/\/arxiv.org\/abs\/2212.06800","DOI":"10.18653\/v1\/2023.acl-long.78"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSMC.1977.4309644"},{"key":"e_1_3_2_35_2","doi-asserted-by":"crossref","unstructured":"Ajay Mandlekar Danfei Xu Roberto Mart\u00edn-Mart\u00edn Silvio Savarese and Li Fei-Fei. 2020. Learning to generalize across long-horizon tasks from human demonstrations. arXiv:2003.06085. Retrieved from https:\/\/arxiv.org\/abs\/2003.06085","DOI":"10.15607\/RSS.2020.XVI.061"},{"key":"e_1_3_2_36_2","unstructured":"Ajay Mandlekar Danfei Xu Josiah Wong Soroush Nasiriany Chen Wang Rohun Kulkarni Li Fei-Fei Silvio Savarese Yuke Zhu and Roberto Mart\u00edn-Mart\u00edn. 2021. What matters in learning from offline human demonstrations for robot manipulation. arXiv:2108.03298. Retrieved from https:\/\/arxiv.org\/abs\/2108.03298"},{"key":"e_1_3_2_37_2","doi-asserted-by":"crossref","first-page":"13015","DOI":"10.1109\/ICRA57147.2024.10611024","volume-title":"2024 IEEE International Conference on Robotics and Automation (ICRA)","author":"Meixner Andre","year":"2024","unstructured":"Andre Meixner, Mischa Carl, Franziska Krebs, No\u00e9mie Jaquier, and Tamim Asfour. 2024. Towards unifying human likeness: Evaluating metrics for human-like motion retargeting on bimanual manipulation tasks. In 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 13015\u201313022."},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1115\/1.3143764"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.5555\/3235188"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/2701973.2702017"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11370-017-0235-8"},{"key":"e_1_3_2_42_2","first-page":"305","article-title":"Alvinn: An autonomous land vehicle in a neural network","volume":"1","author":"Pomerleau Dean A.","year":"1988","unstructured":"Dean A. Pomerleau. 1988. Alvinn: An autonomous land vehicle in a neural network. Advances in Neural Information Processing Systems 1 (1988), 305\u2013313.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-control-100819-063206"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"e_1_3_2_45_2","unstructured":"Boem-Sahng Ryuh. 1989. Robot Trajectory Planning Using the Curvature Theory of Ruled Surfaces. Ph.D. Dissertation. Purdue University."},{"key":"e_1_3_2_46_2","doi-asserted-by":"crossref","first-page":"800","DOI":"10.1109\/RO-MAN47096.2020.9223430","volume-title":"2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)","author":"Sakr Maram","year":"2020","unstructured":"Maram Sakr, Martin Freeman, H. F. Machiel Van der Loos, and Elizabeth Croft. 2020. Training human teacher to improve robot learning from demonstration: A pilot study on kinesthetic teaching. In 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE, 800\u2013806."},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2022.3191950"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3737892"},{"key":"e_1_3_2_49_2","volume-title":"Motor Learning and Performance: A Situation-Based Learning Approach","author":"Schmidt Richard A.","year":"2008","unstructured":"Richard A. Schmidt and Craig A. Wrisberg. 2008. Motor Learning and Performance: A Situation-Based Learning Approach. Human Kinetics."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364919884623"},{"key":"e_1_3_2_51_2","unstructured":"Voot Tangkaratt Bo Han Mohammad Emtiyaz Khan and Masashi Sugiyama. 2019. Vild: Variational imitation learning with diverse-quality demonstrations. arXiv:1909.06769. Retrieved from https:\/\/arxiv.org\/abs\/1909.06769"},{"key":"e_1_3_2_52_2","first-page":"9407","volume-title":"International Conference on Machine Learning","author":"Tangkaratt Voot","year":"2020","unstructured":"Voot Tangkaratt, Bo Han, Mohammad Emtiyaz Khan, and Masashi Sugiyama. 2020. Variational imitation learning with diverse-quality demonstrations. In International Conference on Machine Learning. PMLR, 9407\u20139417."},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10514-014-9394-z"},{"issue":"4","key":"e_1_3_2_54_2","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1101\/lm.58503","article-title":"Sleep and the time course of motor skill learning","volume":"10","author":"Walker Matthew P.","year":"2003","unstructured":"Matthew P. Walker, Tiffany Brakefield, Joshua Seidman, Alexandra Morgan, J. Allan Hobson, and Robert Stickgold. 2003. Sleep and the time course of motor skill learning. Learning and Memory 10, 4 (2003), 275\u2013284.","journal-title":"Learning and Memory"},{"key":"e_1_3_2_55_2","unstructured":"Sebastian Wallkotter Mohamed Chetouani and Ginevra Castellano. 2022. A new approach to evaluating legibility: Comparing legibility frameworks using framework-independent robot motion trajectories. arXiv:2201.05765. Retrieved from https:\/\/arxiv.org\/abs\/2201.05765"},{"key":"e_1_3_2_56_2","first-page":"5326","article-title":"Robust imitation of diverse behaviors","volume":"30","author":"Wang Ziyu","year":"2017","unstructured":"Ziyu Wang, Josh S. Merel, Scott E. Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. 2017. Robust imitation of diverse behaviors. Advances in Neural Information Processing Systems 30 (2017), 5326\u20135335.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364920910802"},{"key":"e_1_3_2_58_2","first-page":"6818","volume-title":"International Conference on Machine Learning","author":"Wu Yueh-Hua","year":"2019","unstructured":"Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. 2019. Imitation learning from imperfect demonstration. In International Conference on Machine Learning. PMLR, 6818\u20136827."},{"key":"e_1_3_2_59_2","volume-title":"IEEE Transactions on Neural Networks and Learning Systems","author":"Xu Shuqi","year":"2024","unstructured":"Shuqi Xu, Hao Zhang, and Zhuping Wang. 2024. Learning to perform trajectory generation from low-quality demonstrations. IEEE Transactions on Neural Networks and Learning Systems (2024)."},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1177\/027836498500400201"},{"key":"e_1_3_2_61_2","volume-title":"Proceedings of Robotics: Science and Systems (RSS)","author":"Ze Yanjie","year":"2024","unstructured":"Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 2024. 3D diffusion policy: Generalizable visuomotor policy learning via simple 3D representations. In Proceedings of Robotics: Science and Systems (RSS)."},{"key":"e_1_3_2_62_2","first-page":"12340","article-title":"Confidence-aware imitation learning from demonstrations with varying optimality","volume":"34","author":"Zhang Songyuan","year":"2021","unstructured":"Songyuan Zhang, Zhangjie Cao, Dorsa Sadigh, and Yanan Sui. 2021. Confidence-aware imitation learning from demonstrations with varying optimality. Advances in Neural Information Processing Systems 34 (2021), 12340\u201312350.","journal-title":"Advances in Neural Information Processing Systems"}],"container-title":["ACM Transactions on Human-Robot Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3773904","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,22]],"date-time":"2025-12-22T14:46:38Z","timestamp":1766414798000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3773904"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,22]]},"references-count":61,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,3,31]]}},"alternative-id":["10.1145\/3773904"],"URL":"https:\/\/doi.org\/10.1145\/3773904","relation":{},"ISSN":["2573-9522"],"issn-type":[{"value":"2573-9522","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,22]]},"assertion":[{"value":"2024-12-11","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-14","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-12-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}