{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T05:23:25Z","timestamp":1780637005119,"version":"3.54.1"},"reference-count":0,"publisher":"IOS Press","isbn-type":[{"value":"9781643684369","type":"print"},{"value":"9781643684376","type":"electronic"}],"license":[{"start":{"date-parts":[[2023,9,28]],"date-time":"2023-09-28T00:00:00Z","timestamp":1695859200000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,9,28]]},"abstract":"<jats:p>Training the machine learning (ML) models require a large amount of data, still the capacity of these models is limited. To enhance model performance, recent literature focuses on combining ML models\u2019 predictions with that of human experts, a setting popularly known as the human-in-the-loop or human-AI teams. Human experts can complement the ML models as they are well-equipped with vast real-world experience and sometimes have access to private information that may not be accessible while training the ML model. Existing approaches for combining an expert and ML model either require end-to-end training of the combined model or require expert annotations for every task. End-to-end training further needs a custom loss function and human annotations, which is cumbersome, results in slower convergence, and may adversely impact the ML model\u2019s accuracy. On the other hand, using expert annotations for every task is also cost-ineffective. We propose a novel technique that optimizes the cost of seeking the expert\u2019s advice while utilizing the ML model\u2019s predictions to improve accuracy. Our model considers two intrinsic parameters: the expert\u2019s cost for each prediction and the misclassification cost of the combined human-AI model. Further, we present the impact of group-wise calibration on the combined model that improves the overall model\u2019s performance. Experimental results on our combined model with group-wise calibration show a significant increase in accuracy with limited expert advice against different established ML models for the image classification task. In addition, the combined model\u2019s accuracy is always greater than that of the ML model, irrespective of the expert\u2019s accuracy, the expert\u2019s cost, and the misclassification cost.<\/jats:p>","DOI":"10.3233\/faia230366","type":"book-chapter","created":{"date-parts":[[2023,9,29]],"date-time":"2023-09-29T09:10:11Z","timestamp":1695978611000},"source":"Crossref","is-referenced-by-count":4,"title":["Take Expert Advice Judiciously: Combining Groupwise Calibrated Model Probabilities with Expert Predictions"],"prefix":"10.3233","author":[{"given":"Sumeet","family":"Gupta","sequence":"first","affiliation":[{"name":"Indian Institute of Technology Ropar"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2666-9058","authenticated-orcid":false,"given":"Shweta","family":"Jain","sequence":"additional","affiliation":[{"name":"Indian Institute of Technology Ropar"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1375-2266","authenticated-orcid":false,"given":"Shashi Shekhar","family":"Jha","sequence":"additional","affiliation":[{"name":"Indian Institute of Technology Ropar"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3639-1467","authenticated-orcid":false,"given":"Pao-Ann","family":"Hsiung","sequence":"additional","affiliation":[{"name":"National Chung Cheng University, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5680-4003","authenticated-orcid":false,"given":"Ming-Hung","family":"Wang","sequence":"additional","affiliation":[{"name":"National Chung Cheng University, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"7437","container-title":["Frontiers in Artificial Intelligence and Applications","ECAI 2023"],"original-title":[],"link":[{"URL":"https:\/\/ebooks.iospress.nl\/pdf\/doi\/10.3233\/FAIA230366","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,29]],"date-time":"2023-09-29T09:10:13Z","timestamp":1695978613000},"score":1,"resource":{"primary":{"URL":"https:\/\/ebooks.iospress.nl\/doi\/10.3233\/FAIA230366"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,28]]},"ISBN":["9781643684369","9781643684376"],"references-count":0,"URL":"https:\/\/doi.org\/10.3233\/faia230366","relation":{},"ISSN":["0922-6389","1879-8314"],"issn-type":[{"value":"0922-6389","type":"print"},{"value":"1879-8314","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,28]]}}}