{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T16:08:07Z","timestamp":1778256487016,"version":"3.51.4"},"publisher-location":"California","reference-count":0,"publisher":"International Joint Conferences on Artificial Intelligence Organization","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,8]]},"abstract":"<jats:p>As model finetuning is central to the modern NLP, we set to maximize its efficiency. Motivated by redundancy in training examples and the sheer sizes of pretrained models, we exploit a key opportunity: training only on important data. To this end, we set to filter training examples in a streaming fashion, in tandem with training the target model. Our key techniques are two: (1) automatically determine a training loss threshold for skipping backward training passes; (2) run a meta predictor for further skipping forward training passes. We integrate the above techniques in a holistic, three-stage training pro- cess. On a diverse set of benchmarks, our method reduces the required training examples by up to 5.3\u00d7 and training time by up to 6.8\u00d7, while only seeing minor accuracy degradation. Our method is effective even for training one epoch, where each training example is encountered only once. It is simple to implement and is compatible with the existing finetuning techniques. Code is available at: https:\/\/github.com\/xo28\/efficient-NLP-multistage-training<\/jats:p>","DOI":"10.24963\/ijcai.2023\/455","type":"proceedings-article","created":{"date-parts":[[2023,8,11]],"date-time":"2023-08-11T08:31:30Z","timestamp":1691742690000},"page":"4091-4099","source":"Crossref","is-referenced-by-count":2,"title":["Efficient NLP Model Finetuning via Multistage Data Filtering"],"prefix":"10.24963","author":[{"given":"Xu","family":"Ouyang","sequence":"first","affiliation":[{"name":"University of Virginia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shahina Mohd Azam","family":"Ansari","sequence":"additional","affiliation":[{"name":"University of Virginia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Felix Xiaozhu","family":"Lin","sequence":"additional","affiliation":[{"name":"University of Virginia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yangfeng","family":"Ji","sequence":"additional","affiliation":[{"name":"University of Virginia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"10584","event":{"name":"Thirty-Second International Joint Conference on Artificial Intelligence {IJCAI-23}","theme":"Artificial Intelligence","location":"Macau, SAR China","acronym":"IJCAI-2023","number":"32","sponsor":["International Joint Conferences on Artificial Intelligence Organization (IJCAI)"],"start":{"date-parts":[[2023,8,19]]},"end":{"date-parts":[[2023,8,25]]}},"container-title":["Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence"],"original-title":[],"deposited":{"date-parts":[[2023,8,11]],"date-time":"2023-08-11T08:48:56Z","timestamp":1691743736000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.ijcai.org\/proceedings\/2023\/455"}},"subtitle":[],"proceedings-subject":"Artificial Intelligence Research Articles","short-title":[],"issued":{"date-parts":[[2023,8]]},"references-count":0,"URL":"https:\/\/doi.org\/10.24963\/ijcai.2023\/455","relation":{},"subject":[],"published":{"date-parts":[[2023,8]]}}}