{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T16:15:50Z","timestamp":1775578550288,"version":"3.50.1"},"publisher-location":"California","reference-count":0,"publisher":"International Joint Conferences on Artificial Intelligence Organization","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,7]]},"abstract":"<jats:p>Recently, self-supervised vision transformers have attracted unprecedented attention for their impressive representation learning ability. \n\nHowever, the dominant method, contrastive learning, mainly relies on an instance discrimination pretext task, which learns a global understanding of the image. \n\nThis paper incorporates local feature learning into self-supervised vision transformers via Reconstructive Pre-training (RePre).\n\nOur RePre extends contrastive frameworks by adding a branch for reconstructing raw image pixels in parallel with the existing contrastive objective. \n\nRePre equips with a lightweight convolution-based decoder that fuses the multi-hierarchy features from the transformer encoder. \n\nThe multi-hierarchy features provide rich supervisions from low to high semantic information, crucial for our RePre.\n\nOur RePre brings decent improvements on various contrastive frameworks with different vision transformer architectures. \n\nTransfer performance in downstream tasks outperforms supervised pre-training and state-of-the-art (SOTA) self-supervised counterparts.<\/jats:p>","DOI":"10.24963\/ijcai.2022\/200","type":"proceedings-article","created":{"date-parts":[[2022,7,16]],"date-time":"2022-07-16T02:55:56Z","timestamp":1657940156000},"page":"1437-1443","source":"Crossref","is-referenced-by-count":11,"title":["RePre: Improving Self-Supervised Vision Transformer with Reconstructive Pre-training"],"prefix":"10.24963","author":[{"given":"Luya","family":"Wang","sequence":"first","affiliation":[{"name":"Beijing University of Posts and Telecommunications"}]},{"given":"Feng","family":"Liang","sequence":"additional","affiliation":[{"name":"The University of Texas at Austin"}]},{"given":"Yangguang","family":"Li","sequence":"additional","affiliation":[{"name":"SenseTime Group Limited"}]},{"given":"Honggang","family":"Zhang","sequence":"additional","affiliation":[{"name":"Beijing University of Posts and Telecommunications"}]},{"given":"Wanli","family":"Ouyang","sequence":"additional","affiliation":[{"name":"The University of Sydney"}]},{"given":"Jing","family":"Shao","sequence":"additional","affiliation":[{"name":"SenseTime Group Limited"}]}],"member":"10584","event":{"name":"Thirty-First International Joint Conference on Artificial Intelligence {IJCAI-22}","theme":"Artificial Intelligence","location":"Vienna, Austria","acronym":"IJCAI-2022","number":"31","sponsor":["International Joint Conferences on Artificial Intelligence Organization (IJCAI)"],"start":{"date-parts":[[2022,7,23]]},"end":{"date-parts":[[2022,7,29]]}},"container-title":["Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence"],"original-title":[],"deposited":{"date-parts":[[2022,7,18]],"date-time":"2022-07-18T11:08:18Z","timestamp":1658142498000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.ijcai.org\/proceedings\/2022\/200"}},"subtitle":[],"proceedings-subject":"Artificial Intelligence Research Articles","short-title":[],"issued":{"date-parts":[[2022,7]]},"references-count":0,"URL":"https:\/\/doi.org\/10.24963\/ijcai.2022\/200","relation":{},"subject":[],"published":{"date-parts":[[2022,7]]}}}