{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T01:41:42Z","timestamp":1787017302392,"version":"build-2736575974"},"publisher-location":"New York, NY, USA","reference-count":50,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,8,14]],"date-time":"2021-08-14T00:00:00Z","timestamp":1628899200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,8,14]]},"DOI":"10.1145\/3447548.3467206","type":"proceedings-article","created":{"date-parts":[[2021,8,13]],"date-time":"2021-08-13T14:21:39Z","timestamp":1628864499000},"page":"3251-3261","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":26,"title":["M6: Multi-Modality-to-Multi-Modality Multitask Mega-transformer for Unified Pretraining"],"prefix":"10.1145","author":[{"given":"Junyang","family":"Lin","sequence":"first","affiliation":[{"name":"Alibaba Group, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rui","family":"Men","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"An","family":"Yang","sequence":"additional","affiliation":[{"name":"Alibaba Group, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chang","family":"Zhou","sequence":"additional","affiliation":[{"name":"Alibaba Group, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yichang","family":"Zhang","sequence":"additional","affiliation":[{"name":"Alibaba Group, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peng","family":"Wang","sequence":"additional","affiliation":[{"name":"Alibaba Group, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jingren","family":"Zhou","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jie","family":"Tang","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hongxia","family":"Yang","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,8,14]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"International Conference on Machine Learning. PMLR, 642--652","author":"Bao Hangbo","year":"2020","unstructured":"Hangbo Bao , Li Dong , Furu Wei , Wenhui Wang , Nan Yang , Xiaodong Liu , Yu Wang , Jianfeng Gao , Songhao Piao , Ming Zhou , 2020 . Unilmv2: Pseudo-masked language models for unified language model pre-training . In International Conference on Machine Learning. PMLR, 642--652 . Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, et al. 2020. Unilmv2: Pseudo-masked language models for unified language model pre-training. In International Conference on Machine Learning. PMLR, 642--652."},{"key":"e_1_3_2_1_2_1","unstructured":"Tom B Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell etal 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020).  Tom B Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020)."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58452-8_13"},{"key":"e_1_3_2_1_4_1","first-page":"104","article-title":"UNITER","volume":"2020","author":"Chen Yen-Chun","year":"2020","unstructured":"Yen-Chun Chen , Linjie Li , Licheng Yu , Ahmed El Kholy , Faisal Ahmed , Zhe Gan , Yu Cheng , and Jingjing Liu . 2020 . UNITER : UNiversal Image-TExt Representation Learning. In ECCV 2020. 104 -- 120 . Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. UNITER: UNiversal Image-TExt Representation Learning. In ECCV 2020. 104--120.","journal-title":"UNiversal Image-TExt Representation Learning. In ECCV"},{"key":"e_1_3_2_1_5_1","volume-title":"Revisiting pre-trained models for chinese natural language processing. arXiv preprint arXiv:2004.13922","author":"Cui Yiming","year":"2020","unstructured":"Yiming Cui , Wanxiang Che , Ting Liu , Bing Qin , Shijin Wang , and Guoping Hu. 2020. Revisiting pre-trained models for chinese natural language processing. arXiv preprint arXiv:2004.13922 ( 2020 ). Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pre-trained models for chinese natural language processing. arXiv preprint arXiv:2004.13922 (2020)."},{"key":"e_1_3_2_1_6_1","volume-title":"A span-extraction dataset for chinese machine reading comprehension. arXiv preprint arXiv:1810.07366","author":"Cui Yiming","year":"2018","unstructured":"Yiming Cui , Ting Liu , Wanxiang Che , Li Xiao , Zhipeng Chen , Wentao Ma , Shijin Wang , and Guoping Hu. 2018. A span-extraction dataset for chinese machine reading comprehension. arXiv preprint arXiv:1810.07366 ( 2018 ). Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. 2018. A span-extraction dataset for chinese machine reading comprehension. arXiv preprint arXiv:1810.07366 (2018)."},{"key":"e_1_3_2_1_7_1","first-page":"4171","article-title":"BERT","volume":"2019","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2019 . BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT 2019. 4171 -- 4186 . Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT 2019. 4171--4186.","journal-title":"Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT"},{"key":"e_1_3_2_1_8_1","volume-title":"NeurIPS","author":"Dong Li","year":"2019","unstructured":"Li Dong , Nan Yang , Wenhui Wang , Furu Wei , Xiaodong Liu , Yu Wang , Jianfeng Gao , Ming Zhou , and Hsiao-Wuen Hon . 2019. Unified Language Model Pre-training for Natural Language Understanding and Generation . In NeurIPS 2019 . 13042--13054. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified Language Model Pre-training for Natural Language Understanding and Generation. In NeurIPS 2019. 13042--13054."},{"key":"e_1_3_2_1_9_1","unstructured":"Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai Thomas Unterthiner Mostafa Dehghani Matthias Minderer Georg Heigold Sylvain Gelly etal 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).  Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai Thomas Unterthiner Mostafa Dehghani Matthias Minderer Georg Heigold Sylvain Gelly et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)."},{"key":"e_1_3_2_1_10_1","unstructured":"Zhe Gan Yen-Chun Chen Linjie Li Chen Zhu Yu Cheng and Jingjing Liu. 2020. Large-Scale Adversarial Training for Vision-and-Language Representation Learning. In NeurIPS 2020 .  Zhe Gan Yen-Chun Chen Linjie Li Chen Zhu Yu Cheng and Jingjing Liu. 2020. Large-Scale Adversarial Training for Vision-and-Language Representation Learning. In NeurIPS 2020 ."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401430"},{"key":"e_1_3_2_1_12_1","volume-title":"Are you talking to a machine? dataset and methods for multilingual image question answering. arXiv preprint arXiv:1505.05612","author":"Gao Haoyuan","year":"2015","unstructured":"Haoyuan Gao , Junhua Mao , Jie Zhou , Zhiheng Huang , Lei Wang , and Wei Xu. 2015. Are you talking to a machine? dataset and methods for multilingual image question answering. arXiv preprint arXiv:1505.05612 ( 2015 ). Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. 2015. Are you talking to a machine? dataset and methods for multilingual image question answering. arXiv preprint arXiv:1505.05612 (2015)."},{"key":"e_1_3_2_1_13_1","volume-title":"2020 a. Making Pre-trained Language Models Better Few-shot Learners. arXiv preprint arXiv:2012.15723","author":"Gao Tianyu","year":"2020","unstructured":"Tianyu Gao , Adam Fisch , and Danqi Chen . 2020 a. Making Pre-trained Language Models Better Few-shot Learners. arXiv preprint arXiv:2012.15723 ( 2020 ). Tianyu Gao, Adam Fisch, and Danqi Chen. 2020 a. Making Pre-trained Language Models Better Few-shot Learners. arXiv preprint arXiv:2012.15723 (2020)."},{"key":"e_1_3_2_1_14_1","volume-title":"Deep Residual Learning for Image Recognition. In CVPR","author":"He Kaiming","year":"2016","unstructured":"Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . 2016 . Deep Residual Learning for Image Recognition. In CVPR 2016. 770--778. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In CVPR 2016. 770--778."},{"key":"e_1_3_2_1_15_1","volume-title":"Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654","author":"He Pengcheng","year":"2020","unstructured":"Pengcheng He , Xiaodong Liu , Jianfeng Gao , and Weizhu Chen . 2020 . Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654 (2020). Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654 (2020)."},{"key":"e_1_3_2_1_16_1","volume-title":"Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849","author":"Huang Zhicheng","year":"2020","unstructured":"Zhicheng Huang , Zhaoyang Zeng , Bei Liu , Dongmei Fu , and Jianlong Fu . 2020 . Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849 (2020). Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. 2020. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849 (2020)."},{"key":"e_1_3_2_1_17_1","volume-title":"Convbert: Improving bert with span-based dynamic convolution. arXiv preprint arXiv:2008.02496","author":"Jiang Zihang","year":"2020","unstructured":"Zihang Jiang , Weihao Yu , Daquan Zhou , Yunpeng Chen , Jiashi Feng , and Shuicheng Yan . 2020 . Convbert: Improving bert with span-based dynamic convolution. arXiv preprint arXiv:2008.02496 (2020). Zihang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. 2020. Convbert: Improving bert with span-based dynamic convolution. arXiv preprint arXiv:2008.02496 (2020)."},{"key":"e_1_3_2_1_18_1","volume-title":"ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. CoRR","author":"Lan Zhenzhong","year":"2019","unstructured":"Zhenzhong Lan , Mingda Chen , Sebastian Goodman , Kevin Gimpel , Piyush Sharma , and Radu Soricut . 2019 . ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. CoRR , Vol. abs\/ 1909 .11942 (2019). Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. CoRR, Vol. abs\/1909.11942 (2019)."},{"key":"e_1_3_2_1_19_1","volume-title":"2019 a. Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training. CoRR","author":"Li Gen","year":"2019","unstructured":"Gen Li , Nan Duan , Yuejian Fang , Daxin Jiang , and Ming Zhou . 2019 a. Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training. CoRR , Vol. abs\/ 1908 .06066 ( 2019 ). Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. 2019 a. Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training. CoRR, Vol. abs\/1908.06066 (2019)."},{"key":"e_1_3_2_1_20_1","volume-title":"2019 b. VisualBERT: A Simple and Performant Baseline for Vision and Language. CoRR","author":"Li Liunian Harold","year":"2019","unstructured":"Liunian Harold Li , Mark Yatskar , Da Yin , Cho-Jui Hsieh , and Kai-Wei Chang . 2019 b. VisualBERT: A Simple and Performant Baseline for Vision and Language. CoRR , Vol. abs\/ 1908 .03557 ( 2019 ). Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019 b. VisualBERT: A Simple and Performant Baseline for Vision and Language. CoRR, Vol. abs\/1908.03557 (2019)."},{"key":"e_1_3_2_1_21_1","volume-title":"Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. CoRR","author":"Li Xiujun","year":"2020","unstructured":"Xiujun Li , Xi Yin , Chunyuan Li , Pengchuan Zhang , Xiaowei Hu , Lei Zhang , Lijuan Wang , Houdong Hu , Li Dong , Furu Wei , Yejin Choi , and Jianfeng Gao . 2020 . Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. CoRR , Vol. abs\/ 2004 .06165 (2020). Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020. Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. CoRR, Vol. abs\/2004.06165 (2020)."},{"key":"e_1_3_2_1_22_1","volume-title":"Interbert: Vision-and-language interaction for multi-modal pretraining. arXiv preprint arXiv:2003.13198","author":"Lin Junyang","year":"2020","unstructured":"Junyang Lin , An Yang , Yichang Zhang , Jie Liu , Jingren Zhou , and Hongxia Yang . 2020 . Interbert: Vision-and-language interaction for multi-modal pretraining. arXiv preprint arXiv:2003.13198 (2020). Junyang Lin, An Yang, Yichang Zhang, Jie Liu, Jingren Zhou, and Hongxia Yang. 2020. Interbert: Vision-and-language interaction for multi-modal pretraining. arXiv preprint arXiv:2003.13198 (2020)."},{"key":"e_1_3_2_1_23_1","first-page":"740","article-title":"Microsoft COCO","volume":"2014","author":"Lin Tsung-Yi","year":"2014","unstructured":"Tsung-Yi Lin , Michael Maire , Serge J. Belongie , James Hays , Pietro Perona , Deva Ramanan , Piotr Doll\u00e1 r, and C. Lawrence Zitnick . 2014 . Microsoft COCO : Common Objects in Context. In ECCV 2014. 740 -- 755 . Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll\u00e1 r, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In ECCV 2014. 740--755.","journal-title":"Common Objects in Context. In ECCV"},{"key":"e_1_3_2_1_24_1","volume-title":"RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR","author":"Liu Yinhan","year":"2019","unstructured":"Yinhan Liu , Myle Ott , Naman Goyal , Jingfei Du , Mandar Joshi , Danqi Chen , Omer Levy , Mike Lewis , Luke Zettlemoyer , and Veselin Stoyanov . 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR , Vol. abs\/ 1907 .11692 ( 2019 ). Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR, Vol. abs\/1907.11692 (2019)."},{"key":"e_1_3_2_1_25_1","volume-title":"Decoupled Weight Decay Regularization. In ICLR 2019 .","author":"Loshchilov Ilya","year":"2019","unstructured":"Ilya Loshchilov and Frank Hutter . 2019 . Decoupled Weight Decay Regularization. In ICLR 2019 . Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In ICLR 2019 ."},{"key":"e_1_3_2_1_26_1","first-page":"13","article-title":"a","volume":"2019","author":"Lu Jiasen","year":"2019","unstructured":"Jiasen Lu , Dhruv Batra , Devi Parikh , and Stefan Lee . 2019 a . ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. In NeurIPS 2019. 13 -- 23 . Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019 a. ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. In NeurIPS 2019. 13--23.","journal-title":"ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. In NeurIPS"},{"key":"e_1_3_2_1_27_1","volume-title":"2019 b. 12-in-1: Multi-Task Vision and Language Representation Learning. CoRR","author":"Lu Jiasen","year":"2019","unstructured":"Jiasen Lu , Vedanuj Goswami , Marcus Rohrbach , Devi Parikh , and Stefan Lee . 2019 b. 12-in-1: Multi-Task Vision and Language Representation Learning. CoRR , Vol. abs\/ 1912 .02315 ( 2019 ). Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2019 b. 12-in-1: Multi-Task Vision and Language Representation Learning. CoRR, Vol. abs\/1912.02315 (2019)."},{"key":"e_1_3_2_1_28_1","volume-title":"Improving language understanding by generative pre-training. URL https:\/\/s3-us-west-2.amazonaws.com\/openai-assets\/researchcovers\/ languageunsupervised\/language understanding paper. pdf","author":"Radford Alec","year":"2018","unstructured":"Alec Radford , Karthik Narasimhan , Tim Salimans , and Ilya Sutskever . 2018. Improving language understanding by generative pre-training. URL https:\/\/s3-us-west-2.amazonaws.com\/openai-assets\/researchcovers\/ languageunsupervised\/language understanding paper. pdf ( 2018 ). Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https:\/\/s3-us-west-2.amazonaws.com\/openai-assets\/researchcovers\/ languageunsupervised\/language understanding paper. pdf (2018)."},{"key":"e_1_3_2_1_29_1","unstructured":"Alec Radford Jeffrey Wu Rewon Child David Luan Dario Amodei and Ilya Sutskever. [n.d.]. Language models are unsupervised multitask learners. ( [n. d.]).  Alec Radford Jeffrey Wu Rewon Child David Luan Dario Amodei and Ilya Sutskever. [n.d.]. Language models are unsupervised multitask learners. ( [n. d.])."},{"key":"e_1_3_2_1_30_1","volume-title":"Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054","author":"Rajbhandari Samyam","year":"2019","unstructured":"Samyam Rajbhandari , Jeff Rasley , Olatunji Ruwase , and Yuxiong He . 2019 . Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054 (2019). Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2019. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054 (2019)."},{"key":"e_1_3_2_1_31_1","volume-title":"Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He.","author":"Ren Jie","year":"2021","unstructured":"Jie Ren , Samyam Rajbhandari , Reza Yazdani Aminabadi , Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021 . ZeRO-Offload: Democratizing Billion-Scale Model Training . arXiv preprint arXiv:2101.06840 (2021). Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021. ZeRO-Offload: Democratizing Billion-Scale Model Training. arXiv preprint arXiv:2101.06840 (2021)."},{"key":"e_1_3_2_1_32_1","first-page":"91","article-title":"Faster R-CNN","volume":"2015","author":"Ren Shaoqing","year":"2015","unstructured":"Shaoqing Ren , Kaiming He , Ross B. Girshick , and Jian Sun . 2015 . Faster R-CNN : Towards Real-Time Object Detection with Region Proposal Networks. In NeurIPS 2015. 91 -- 99 . Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In NeurIPS 2015. 91--99.","journal-title":"Towards Real-Time Object Detection with Region Proposal Networks. In NeurIPS"},{"key":"e_1_3_2_1_33_1","volume-title":"It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. arXiv preprint arXiv:2009.07118","author":"Schick Timo","year":"2020","unstructured":"Timo Schick and Hinrich Sch\u00fctze . 2020. It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. arXiv preprint arXiv:2009.07118 ( 2020 ). Timo Schick and Hinrich Sch\u00fctze. 2020. It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. arXiv preprint arXiv:2009.07118 (2020)."},{"key":"e_1_3_2_1_34_1","volume-title":"Neural Machine Translation of Rare Words with Subword Units. In ACL 2016 .","author":"Sennrich Rico","year":"2016","unstructured":"Rico Sennrich , Barry Haddow , and Alexandra Birch . 2016 . Neural Machine Translation of Rare Words with Subword Units. In ACL 2016 . Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In ACL 2016 ."},{"key":"e_1_3_2_1_35_1","volume-title":"ICML","author":"Song Kaitao","year":"2019","unstructured":"Kaitao Song , Xu Tan , Tao Qin , Jianfeng Lu , and Tie-Yan Liu . 2019 . MASS: Masked Sequence to Sequence Pre-training for Language Generation . In ICML 2019. 5926--5936. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. MASS: Masked Sequence to Sequence Pre-training for Language Generation. In ICML 2019. 5926--5936."},{"key":"e_1_3_2_1_36_1","volume-title":"VL-BERT: Pre-training of Generic Visual-Linguistic Representations. In ICLR 2020 .","author":"Su Weijie","year":"2020","unstructured":"Weijie Su , Xizhou Zhu , Yue Cao , Bin Li , Lewei Lu , Furu Wei , and Jifeng Dai . 2020 . VL-BERT: Pre-training of Generic Visual-Linguistic Representations. In ICLR 2020 . Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020. VL-BERT: Pre-training of Generic Visual-Linguistic Representations. In ICLR 2020 ."},{"key":"e_1_3_2_1_37_1","volume-title":"Thuctc: an efficient chinese text classifier. GitHub Repository","author":"Sun Maosong","year":"2016","unstructured":"Maosong Sun , Jingyang Li , Zhipeng Guo , Z Yu , Y Zheng , X Si , and Z Liu . 2016. Thuctc: an efficient chinese text classifier. GitHub Repository ( 2016 ). Maosong Sun, Jingyang Li, Zhipeng Guo, Z Yu, Y Zheng, X Si, and Z Liu. 2016. Thuctc: an efficient chinese text classifier. GitHub Repository (2016)."},{"key":"e_1_3_2_1_38_1","first-page":"5099","article-title":"LXMERT: Learning Cross-Modality Encoder Representations from Transformers","volume":"2019","author":"Tan Hao","year":"2019","unstructured":"Hao Tan and Mohit Bansal . 2019 . LXMERT: Learning Cross-Modality Encoder Representations from Transformers . In EMNLP-IJCNLP 2019. 5099 -- 5110 . Hao Tan and Mohit Bansal. 2019. LXMERT: Learning Cross-Modality Encoder Representations from Transformers. In EMNLP-IJCNLP 2019. 5099--5110.","journal-title":"EMNLP-IJCNLP"},{"key":"e_1_3_2_1_39_1","volume-title":"NeurIPS","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N. Gomez , Lukasz Kaiser , and Illia Polosukhin . 2017. Attention is All you Need . In NeurIPS 2017 . 5998--6008. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In NeurIPS 2017. 5998--6008."},{"key":"e_1_3_2_1_40_1","volume-title":"Whale: A Unified Distributed Training Framework. arXiv preprint arXiv:2011.09208","author":"Wang Ang","year":"2020","unstructured":"Ang Wang , Xianyan Jia , Le Jiang , Jie Zhang , Yong Li , and Wei Lin . 2020 . Whale: A Unified Distributed Training Framework. arXiv preprint arXiv:2011.09208 (2020). Ang Wang, Xianyan Jia, Le Jiang, Jie Zhang, Yong Li, and Wei Lin. 2020. Whale: A Unified Distributed Training Framework. arXiv preprint arXiv:2011.09208 (2020)."},{"key":"e_1_3_2_1_41_1","volume-title":"Structbert: Incorporating language structures into pre-training for deep language understanding. arXiv preprint arXiv:1908.04577","author":"Wang Wei","year":"2019","unstructured":"Wei Wang , Bin Bi , Ming Yan , Chen Wu , Zuyi Bao , Jiangnan Xia , Liwei Peng , and Luo Si . 2019 . Structbert: Incorporating language structures into pre-training for deep language understanding. arXiv preprint arXiv:1908.04577 (2019). Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Jiangnan Xia, Liwei Peng, and Luo Si. 2019. Structbert: Incorporating language structures into pre-training for deep language understanding. arXiv preprint arXiv:1908.04577 (2019)."},{"key":"e_1_3_2_1_42_1","unstructured":"Yonghui Wu Mike Schuster Zhifeng Chen Quoc V Le Mohammad Norouzi Wolfgang Macherey Maxim Krikun Yuan Cao Qin Gao Klaus Macherey etal 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016).  Yonghui Wu Mike Schuster Zhifeng Chen Quoc V Le Mohammad Norouzi Wolfgang Macherey Maxim Krikun Yuan Cao Qin Gao Klaus Macherey et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)."},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.634"},{"key":"e_1_3_2_1_44_1","first-page":"4762","article-title":"a","volume":"2020","author":"Xu Liang","year":"2020","unstructured":"Liang Xu , Hai Hu , Xuanwei Zhang , Lu Li , Chenjie Cao , Yudong Li , Yechen Xu , Kai Sun , Dian Yu , Cong Yu , 2020 a . CLUE: A Chinese Language Understanding Evaluation Benchmark. In COLING 2020. 4762 -- 4772 . Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. 2020 a. CLUE: A Chinese Language Understanding Evaluation Benchmark. In COLING 2020. 4762--4772.","journal-title":"CLUE: A Chinese Language Understanding Evaluation Benchmark. In COLING"},{"key":"e_1_3_2_1_45_1","volume-title":"2020 b. CLUECorpus2020: A Large-scale Chinese Corpus for Pre-trainingLanguage Model. arXiv preprint arXiv:2003.01355","author":"Xu Liang","year":"2020","unstructured":"Liang Xu , Xuanwei Zhang , and Qianqian Dong . 2020 b. CLUECorpus2020: A Large-scale Chinese Corpus for Pre-trainingLanguage Model. arXiv preprint arXiv:2003.01355 ( 2020 ). Liang Xu, Xuanwei Zhang, and Qianqian Dong. 2020 b. CLUECorpus2020: A Large-scale Chinese Corpus for Pre-trainingLanguage Model. arXiv preprint arXiv:2003.01355 (2020)."},{"key":"e_1_3_2_1_46_1","first-page":"5754","article-title":"XLNet","volume":"2019","author":"Yang Zhilin","year":"2019","unstructured":"Zhilin Yang , Zihang Dai , Yiming Yang , Jaime G. Carbonell , Ruslan Salakhutdinov , and Quoc V. Le . 2019 . XLNet : Generalized Autoregressive Pretraining for Language Understanding. In NeurIPS 2019. 5754 -- 5764 . Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In NeurIPS 2019. 5754--5764.","journal-title":"Generalized Autoregressive Pretraining for Language Understanding. In NeurIPS"},{"key":"e_1_3_2_1_47_1","volume-title":"Ernie-vil: Knowledge enhanced vision-language representations through scene graph. arXiv preprint arXiv:2006.16934","author":"Yu Fei","year":"2020","unstructured":"Fei Yu , Jiji Tang , Weichong Yin , Yu Sun , Hao Tian , Hua Wu , and Haifeng Wang . 2020 . Ernie-vil: Knowledge enhanced vision-language representations through scene graph. arXiv preprint arXiv:2006.16934 (2020). Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie-vil: Knowledge enhanced vision-language representations through scene graph. arXiv preprint arXiv:2006.16934 (2020)."},{"key":"e_1_3_2_1_48_1","volume-title":"CPM: A Large-scale Generative Chinese Pre-trained Language Model. arXiv preprint arXiv:2012.00413","author":"Zhang Zhengyan","year":"2020","unstructured":"Zhengyan Zhang , Xu Han , Hao Zhou , Pei Ke , Yuxian Gu , Deming Ye , Yujia Qin , Yusheng Su , Haozhe Ji , Jian Guan , 2020 . CPM: A Large-scale Generative Chinese Pre-trained Language Model. arXiv preprint arXiv:2012.00413 (2020). Zhengyan Zhang, Xu Han, Hao Zhou, Pei Ke, Yuxian Gu, Deming Ye, Yujia Qin, Yusheng Su, Haozhe Ji, Jian Guan, et al. 2020. CPM: A Large-scale Generative Chinese Pre-trained Language Model. arXiv preprint arXiv:2012.00413 (2020)."},{"key":"e_1_3_2_1_49_1","first-page":"778","article-title":"ChID","volume":"2019","author":"Zheng Chujie","year":"2019","unstructured":"Chujie Zheng , Minlie Huang , and Aixin Sun . 2019 . ChID : A Large-scale Chinese IDiom Dataset for Cloze Test. In ACL 2019. 778 -- 787 . Chujie Zheng, Minlie Huang, and Aixin Sun. 2019. ChID: A Large-scale Chinese IDiom Dataset for Cloze Test. In ACL 2019. 778--787.","journal-title":"A Large-scale Chinese IDiom Dataset for Cloze Test. In ACL"},{"key":"e_1_3_2_1_50_1","volume-title":"Unified Vision-Language Pre-Training for Image Captioning and VQA. In AAAI","author":"Zhou Luowei","year":"2020","unstructured":"Luowei Zhou , Hamid Palangi , Lei Zhang , Houdong Hu , Jason J. Corso , and Jianfeng Gao . 2020 . Unified Vision-Language Pre-Training for Image Captioning and VQA. In AAAI 2020. 13041--13049. Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. 2020. Unified Vision-Language Pre-Training for Image Captioning and VQA. In AAAI 2020. 13041--13049."}],"event":{"name":"KDD '21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","location":"Virtual Event Singapore","acronym":"KDD '21","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data"]},"container-title":["Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3447548.3467206","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3447548.3467206","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:18:27Z","timestamp":1750177107000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3447548.3467206"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,14]]},"references-count":50,"alternative-id":["10.1145\/3447548.3467206","10.1145\/3447548"],"URL":"https:\/\/doi.org\/10.1145\/3447548.3467206","relation":{},"subject":[],"published":{"date-parts":[[2021,8,14]]},"assertion":[{"value":"2021-08-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}