{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,18]],"date-time":"2025-12-18T14:17:47Z","timestamp":1766067467483,"version":"3.38.0"},"reference-count":47,"publisher":"SAGE Publications","issue":"1","license":[{"start":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T00:00:00Z","timestamp":1670371200000},"content-version":"vor","delay-in-days":365,"URL":"http:\/\/www.sagepub.com\/licence-information-for-chorus"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["DGE 1745016"],"award-info":[{"award-number":["DGE 1745016"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["IIS-1755823"],"award-info":[{"award-number":["IIS-1755823"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of Robotics Research"],"published-print":{"date-parts":[[2022,1]]},"abstract":"<jats:p> We present the Human And Robot Multimodal Observations of Natural Interactive Collaboration (HARMONIC) dataset. This is a large multimodal dataset of human interactions with a robotic arm in a shared autonomy setting designed to imitate assistive eating. The dataset provides human, robot, and environmental data views of 24 different people engaged in an assistive eating task with a 6-degree-of-freedom (6-DOF) robot arm. From each participant, we recorded video of both eyes, egocentric video from a head-mounted camera, joystick commands, electromyography from the forearm used to operate the joystick, third-person stereo video, and the joint positions of the 6-DOF robot arm. Also included are several features that come as a direct result of these recordings, such as eye gaze projected onto the egocentric video, body pose, hand pose, and facial keypoints. These data streams were collected specifically because they have been shown to be closely related to human mental states and intention. This dataset could be of interest to researchers studying intention prediction, human mental state modeling, and shared autonomy. Data streams are provided in a variety of formats such as video and human-readable CSV and YAML files. <\/jats:p>","DOI":"10.1177\/02783649211050677","type":"journal-article","created":{"date-parts":[[2021,12,8]],"date-time":"2021-12-08T05:00:30Z","timestamp":1638939630000},"page":"3-11","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":25,"title":["HARMONIC: A multimodal dataset of assistive human\u2013robot collaboration"],"prefix":"10.1177","volume":"41","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9651-3470","authenticated-orcid":false,"given":"Benjamin A.","family":"Newman","sequence":"first","affiliation":[{"name":"Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8615-3715","authenticated-orcid":false,"given":"Reuben M.","family":"Aronson","sequence":"additional","affiliation":[{"name":"Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"given":"Siddhartha S.","family":"Srinivasa","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}]},{"given":"Kris","family":"Kitani","sequence":"additional","affiliation":[{"name":"Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"given":"Henny","family":"Admoni","sequence":"additional","affiliation":[{"name":"Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA"}]}],"member":"179","published-online":{"date-parts":[[2021,12,7]]},"reference":[{"key":"bibr1-02783649211050677","first-page":"104","author":"Admoni H","year":"2014","journal-title":"Annual Conference of the Cognitive Science Society (CogSci)"},{"journal-title":"RSS Workshop: Towards a Framework for Joint Action","year":"2018","author":"Aronson RM","key":"bibr2-02783649211050677"},{"journal-title":"ACM\/IEEE International Conference on Human-Robot Interaction (HRI)","year":"2018","author":"Aronson RM","key":"bibr3-02783649211050677"},{"key":"bibr4-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2009.2039378"},{"journal-title":"NIPS 2016, workshop Future of Interactive Learning Machines","year":"2016","author":"Azagra P","key":"bibr5-02783649211050677"},{"key":"bibr6-02783649211050677","unstructured":"Bastianelli E, Castellucci G, Croce D, Iocchi L, Basili R, Nardi D (2014) Huric: A human robot interaction corpus. In: Choukri K, Declerck T, Loftsson H, et al. (eds.) Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC\u201914). Reykjavik, Iceland: European Language Resources Association (ELRA)."},{"key":"bibr7-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.91.2.276"},{"key":"bibr8-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1145\/3204493.3204577"},{"journal-title":"CoRR","year":"2012","author":"Bellemare MG","key":"bibr9-02783649211050677"},{"key":"bibr10-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1145\/3136755.3136814"},{"key":"bibr11-02783649211050677","doi-asserted-by":"publisher","DOI":"10.3389\/fnbot.2012.00003"},{"key":"bibr12-02783649211050677","first-page":"1652","author":"Braunagel C","year":"2015","journal-title":"2015 IEEE 18th International Conference on Intelligent Transportation Systems"},{"journal-title":"Proceedings of CVPR","year":"2017","author":"Cao Z","key":"bibr13-02783649211050677"},{"journal-title":"CoRR","year":"2018","author":"Damen D","key":"bibr14-02783649211050677"},{"key":"bibr15-02783649211050677","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2018.XIV.063"},{"key":"bibr16-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33718-5_23"},{"key":"bibr17-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2005.02.009"},{"key":"bibr18-02783649211050677","first-page":"83","author":"Huang CM","year":"2016","journal-title":"ACM\/IEEE International Conference on Human\u2013Robot Interaction (HRI)"},{"key":"bibr19-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1177\/0278364918776060"},{"journal-title":"Robotics: Science and Systems (RSS)","year":"2015","author":"Javdani S","key":"bibr20-02783649211050677"},{"key":"bibr21-02783649211050677","first-page":"149","author":"Jayagopi DB","year":"2013","journal-title":"2013 8th ACM\/IEEE International Conference on Human-Robot Interaction (HRI)"},{"key":"bibr22-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.21-17-06917.2001"},{"key":"bibr23-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(98)00023-X"},{"key":"bibr24-02783649211050677","first-page":"1151","author":"Kassner M","year":"2014","journal-title":"Adjunct Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp \u201914) Adjunct"},{"key":"bibr25-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0203629"},{"key":"bibr26-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1016\/S0042-6989(01)00102-X"},{"key":"bibr27-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1016\/B978-1-55860-377-6.50052-9"},{"key":"bibr28-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2015.7139282"},{"key":"bibr29-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2012.6248010"},{"key":"bibr30-02783649211050677","unstructured":"Pupil Labs, Inc. (2017) Pupil labs - Pupil. https:\/\/pupil-labs.com\/pupil\/."},{"journal-title":"CoRR","year":"2018","author":"Reddy S","key":"bibr31-02783649211050677"},{"journal-title":"CoRR","year":"2018","author":"Reddy S","key":"bibr32-02783649211050677"},{"key":"bibr33-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1002\/9780470989418"},{"key":"bibr34-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-34014-7_9"},{"journal-title":"International Joint Conference on Artificial Intelligence (IJCAI)","year":"2016","author":"Shu T","key":"bibr35-02783649211050677"},{"journal-title":"CoRR","year":"2018","author":"Sigurdsson GA","key":"bibr36-02783649211050677"},{"journal-title":"CoRR","year":"2018","author":"Sigurdsson GA","key":"bibr37-02783649211050677"},{"journal-title":"Proceedings of CVPR","year":"2017","author":"Simon T","key":"bibr38-02783649211050677"},{"key":"bibr39-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1287\/opre.26.2.282"},{"journal-title":"Proceedings of LREC","year":"2016","author":"Stefanov K","key":"bibr40-02783649211050677"},{"key":"bibr41-02783649211050677","unstructured":"Stereolabs, Inc. (2018) Stereolabs. https:\/\/www.stereolabs.com\/."},{"key":"bibr42-02783649211050677","unstructured":"Thalmic Labs, Inc. (2018) Myo Gesture Control Armband. https:\/\/www.myo.com\/."},{"key":"bibr43-02783649211050677","unstructured":"Tobii AB (2017) Tobii pro. https:\/\/www.tobiipro.com\/"},{"journal-title":"Proceedings of CVPR","year":"2016","author":"Wei SE","key":"bibr44-02783649211050677"},{"key":"bibr45-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1075\/is.14.3.05xu"},{"key":"bibr46-02783649211050677","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01252-6_41"},{"key":"bibr47-02783649211050677","first-page":"1433","author":"Ziebart BD","year":"2008","journal-title":"Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence"}],"container-title":["The International Journal of Robotics Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/02783649211050677","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/02783649211050677","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/02783649211050677","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/02783649211050677","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,2]],"date-time":"2025-03-02T15:14:34Z","timestamp":1740928474000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/02783649211050677"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,7]]},"references-count":47,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,1]]}},"alternative-id":["10.1177\/02783649211050677"],"URL":"https:\/\/doi.org\/10.1177\/02783649211050677","relation":{},"ISSN":["0278-3649","1741-3176"],"issn-type":[{"type":"print","value":"0278-3649"},{"type":"electronic","value":"1741-3176"}],"subject":[],"published":{"date-parts":[[2021,12,7]]}}}