{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T23:12:20Z","timestamp":1776121940328,"version":"3.50.1"},"publisher-location":"California","reference-count":0,"publisher":"International Joint Conferences on Artificial Intelligence Organization","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,7]]},"abstract":"<jats:p>Machine learning (ML) models are increasingly being used in application domains that often involve working together with human experts. In this context, it can be advantageous to defer certain instances to a single human expert when they are difficult to predict for the ML model. While previous work has focused on scenarios with one distinct human expert, in many real-world situations several human experts with varying capabilities may be available. In this work, we propose an approach that trains a classification model to complement the capabilities of multiple human experts. By jointly training the classifier together with an allocation system, the classifier learns to accurately predict those instances that are difficult for the human experts, while the allocation system learns to pass each instance to the most suitable team member\u2014either the classifier or one of the human experts. We evaluate our proposed approach in multiple experiments on public datasets with \u201csynthetic\u201d experts and a real-world medical dataset annotated by multiple radiologists. Our approach outperforms prior work and is more accurate than the best human expert or a classifier. Furthermore, it is flexibly adaptable to teams of varying sizes and different levels of expert diversity.<\/jats:p>","DOI":"10.24963\/ijcai.2022\/344","type":"proceedings-article","created":{"date-parts":[[2022,7,16]],"date-time":"2022-07-16T02:55:56Z","timestamp":1657940156000},"page":"2478-2484","source":"Crossref","is-referenced-by-count":18,"title":["Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts"],"prefix":"10.24963","author":[{"given":"Patrick","family":"Hemmer","sequence":"first","affiliation":[{"name":"Karlsruhe Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sebastian","family":"Schellhammer","sequence":"additional","affiliation":[{"name":"Karlsruhe Institute of Technology"},{"name":"GESIS - Leibniz Institute for the Social Sciences"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"V\u00f6ssing","sequence":"additional","affiliation":[{"name":"Karlsruhe Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Johannes","family":"Jakubik","sequence":"additional","affiliation":[{"name":"Karlsruhe Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gerhard","family":"Satzger","sequence":"additional","affiliation":[{"name":"Karlsruhe Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"10584","event":{"name":"Thirty-First International Joint Conference on Artificial Intelligence {IJCAI-22}","theme":"Artificial Intelligence","location":"Vienna, Austria","acronym":"IJCAI-2022","number":"31","sponsor":["International Joint Conferences on Artificial Intelligence Organization (IJCAI)"],"start":{"date-parts":[[2022,7,23]]},"end":{"date-parts":[[2022,7,29]]}},"container-title":["Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence"],"original-title":[],"deposited":{"date-parts":[[2022,7,18]],"date-time":"2022-07-18T11:09:11Z","timestamp":1658142551000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.ijcai.org\/proceedings\/2022\/344"}},"subtitle":[],"proceedings-subject":"Artificial Intelligence Research Articles","short-title":[],"issued":{"date-parts":[[2022,7]]},"references-count":0,"URL":"https:\/\/doi.org\/10.24963\/ijcai.2022\/344","relation":{},"subject":[],"published":{"date-parts":[[2022,7]]}}}