{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,30]],"date-time":"2026-05-30T00:23:35Z","timestamp":1780100615180,"version":"3.54.0"},"publisher-location":"California","reference-count":0,"publisher":"International Joint Conferences on Artificial Intelligence Organization","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,8]]},"abstract":"<jats:p>Human-in-the-loop Machine Learning (HIL-ML) is a widely adopted paradigm for instilling human knowledge in autonomous agents. Many design choices influence the efficiency and effectiveness of such interactive learning processes, particularly the interaction type through which the human teacher may provide feedback. While different interaction types (demonstrations, preferences, etc.) have been proposed and evaluated in the HIL-ML literature, there has been little discussion of how these compare or how they should be selected to best address a particular learning problem. In this survey, we propose an organizing principle for HIL-ML that provides a way to analyze the effects of interaction types on human performance and training data. We also identify open problems in understanding the effects of interaction types.<\/jats:p>","DOI":"10.24963\/ijcai.2021\/599","type":"proceedings-article","created":{"date-parts":[[2021,8,11]],"date-time":"2021-08-11T11:00:49Z","timestamp":1628679649000},"page":"4382-4391","source":"Crossref","is-referenced-by-count":23,"title":["Understanding the Relationship between Interactions and Outcomes in Human-in-the-Loop Machine Learning"],"prefix":"10.24963","author":[{"given":"Yuchen","family":"Cui","sequence":"first","affiliation":[{"name":"Department of Computer Science, The University of Texas at Austin"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pallavi","family":"Koppol","sequence":"additional","affiliation":[{"name":"School of Computer Science, Carnegie Mellon University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Henny","family":"Admoni","sequence":"additional","affiliation":[{"name":"School of Computer Science, Carnegie Mellon University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Scott","family":"Niekum","sequence":"additional","affiliation":[{"name":"Department of Computer Science, The University of Texas at Austin"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Reid","family":"Simmons","sequence":"additional","affiliation":[{"name":"School of Computer Science, Carnegie Mellon University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aaron","family":"Steinfeld","sequence":"additional","affiliation":[{"name":"School of Computer Science, Carnegie Mellon University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tesca","family":"Fitzgerald","sequence":"additional","affiliation":[{"name":"School of Computer Science, Carnegie Mellon University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10584","event":{"name":"Thirtieth International Joint Conference on Artificial Intelligence {IJCAI-21}","theme":"Artificial Intelligence","location":"Montreal, Canada","acronym":"IJCAI-2021","number":"30","sponsor":["International Joint Conferences on Artificial Intelligence Organization (IJCAI)"],"start":{"date-parts":[[2021,8,19]]},"end":{"date-parts":[[2021,8,27]]}},"container-title":["Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence"],"original-title":[],"deposited":{"date-parts":[[2021,8,11]],"date-time":"2021-08-11T11:04:15Z","timestamp":1628679855000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.ijcai.org\/proceedings\/2021\/599"}},"subtitle":[],"proceedings-subject":"Artificial Intelligence Research Articles","short-title":[],"issued":{"date-parts":[[2021,8]]},"references-count":0,"URL":"https:\/\/doi.org\/10.24963\/ijcai.2021\/599","relation":{},"subject":[],"published":{"date-parts":[[2021,8]]}}}