{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,6]],"date-time":"2026-11-06T17:46:03Z","timestamp":1793987163257,"version":"4.8.0"},"reference-count":70,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2022,6,21]],"date-time":"2022-06-21T00:00:00Z","timestamp":1655769600000},"content-version":"vor","delay-in-days":171,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,6,17]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Prompt-based approaches excel at few-shot learning. However, Perez et al. (2021) recently cast doubt on their performance as they had difficulty getting good results in a \u201ctrue\u201d few-shot setting in which prompts and hyperparameters cannot be tuned on a dev set. In view of this, we conduct an extensive study of Pet, a method that combines textual instructions with example-based finetuning. We show that, if correctly configured, Pet performs strongly in true few-shot settings without a dev set. Crucial for this strong performance is a number of design choices, including Pet\u2019s ability to intelligently handle multiple prompts. We put our findings to a real-world test by running Pet on RAFT, a benchmark of tasks taken from realistic NLP applications for which no labeled dev or test sets are available. Pet achieves a new state of the art on RAFT and performs close to non-expert humans for 7 out of 11 tasks. These results demonstrate that prompt-based learners can successfully be applied in true few-shot settings and underpin our belief that learning from instructions will play an important role on the path towards human-like few-shot learning capabilities.<\/jats:p>","DOI":"10.1162\/tacl_a_00485","type":"journal-article","created":{"date-parts":[[2022,6,21]],"date-time":"2022-06-21T14:09:20Z","timestamp":1655820560000},"page":"716-731","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":37,"title":["True Few-Shot Learning with Prompts\u2014A Real-World Perspective"],"prefix":"10.1162","volume":"10","author":[{"given":"Timo","family":"Schick","sequence":"first","affiliation":[{"name":"Center for Information and Language Processing (CIS), LMU Munich, Germany. schickt@cis.lmu.de"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hinrich","family":"Sch\u00fctze","sequence":"additional","affiliation":[{"name":"Center for Information and Language Processing (CIS), LMU Munich, Germany. inquiries@cislmu.org"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2022,6,17]]},"reference":[{"key":"2022062118084934300_bib1","article-title":"RAFT: A real-world few- shot text classification benchmark","volume-title":"Thirty- fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)","author":"Alex","year":"2021"},{"issue":"05","key":"2022062118084934300_bib2","doi-asserted-by":"publisher","first-page":"7383","DOI":"10.1609\/aaai.v34i05.6233","article-title":"Do not have enough data? Deep learning to the rescue!","volume":"34","author":"Anaby-Tavor","year":"2020","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"2022062118084934300_bib3","article-title":"FLEX: Unifying evaluation for few-shot NLP","volume-title":"Thirty-Fifth Conference on Neural Information Processing Systems","author":"Bragg","year":"2021"},{"key":"2022062118084934300_bib4","first-page":"1877","article-title":"Language models are few- shot learners","volume-title":"Advances in Neural Information Processing Systems","author":"Brown","year":"2020"},{"key":"2022062118084934300_bib5","first-page":"830","article-title":"Importance of semantic representation: Dataless classification","volume-title":"Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 2","author":"Chang","year":"2008"},{"key":"2022062118084934300_bib6","doi-asserted-by":"publisher","first-page":"2147","DOI":"10.18653\/v1\/2020.acl-main.194","article-title":"MixText: Linguistically-informed interpolation of hidden space for semi-supervised text classification","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Chen","year":"2020"},{"key":"2022062118084934300_bib7","doi-asserted-by":"publisher","first-page":"2089","DOI":"10.18653\/v1\/N19-1213","article-title":"An embarrassingly simple approach for transfer learning from pretrained language models","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Chronopoulou","year":"2019"},{"key":"2022062118084934300_bib8","doi-asserted-by":"publisher","first-page":"4165","DOI":"10.18653\/v1\/2021.findings-acl.365","article-title":"Unsupervised label refinement improves dataless text classification","volume-title":"Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021","author":"Chu","year":"2021"},{"key":"2022062118084934300_bib9","doi-asserted-by":"publisher","first-page":"1173","DOI":"10.18653\/v1\/D19-1109","article-title":"Commonsense knowledge mining from pretrained models","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Davison","year":"2019"},{"key":"2022062118084934300_bib10","doi-asserted-by":"publisher","first-page":"295","DOI":"10.18653\/v1\/2020.emnlp-main.21","article-title":"Calibration of pre-trained transformers","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Desai","year":"2020"},{"key":"2022062118084934300_bib11","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2022062118084934300_bib12","article-title":"Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping","volume":"arXiv:2002.06305","author":"Dodge","year":"2020","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib13","article-title":"The turking test: Can language models understand instructions?","volume":"arXiv: 2010.11982","author":"Efrat","year":"2020","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib14","doi-asserted-by":"publisher","first-page":"1012","DOI":"10.1162\/tacl_a_00410","article-title":"Measuring and improving consistency in pretrained language models","volume":"9","author":"Elazar","year":"2021","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2022062118084934300_bib15","article-title":"Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity","volume":"arXiv:2101.03961","author":"Fedus","year":"2021","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib16","first-page":"3816","article-title":"Making pre-trained language models better few-shot learners","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Gao","year":"2021"},{"key":"2022062118084934300_bib17","doi-asserted-by":"publisher","first-page":"8342","DOI":"10.18653\/v1\/2020.acl-main.740","article-title":"Don\u2019t stop pretraining: Adapt language models to domains and tasks","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Gururangan","year":"2020"},{"key":"2022062118084934300_bib18","doi-asserted-by":"publisher","first-page":"4921","DOI":"10.18653\/v1\/2021.acl-long.381","article-title":"WARP: Word-level Adversarial ReProgramming","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Hambardzumyan","year":"2021"},{"key":"2022062118084934300_bib19","article-title":"Distilling the knowledge in a neural network","volume":"arXiv:1503.02531","author":"Hinton","year":"2015","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib20","article-title":"The curious case of neural text degeneration","volume-title":"International Conference on Learning Representations","author":"Holtzman","year":"2020"},{"key":"2022062118084934300_bib21","doi-asserted-by":"publisher","first-page":"962","DOI":"10.1162\/tacl_a_00407","article-title":"How can we know when language models know? On the calibration of language models for question answering","volume":"9","author":"Jiang","year":"2021","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2022062118084934300_bib22","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1162\/tacl_a_00324","article-title":"How can we know what language models know?","volume":"8","author":"Jiang","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2022062118084934300_bib23","doi-asserted-by":"publisher","first-page":"4507","DOI":"10.18653\/v1\/2021.findings-acl.395","article-title":"Reordering examples helps during priming-based few-shot learning","volume-title":"Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021","author":"Kumar","year":"2021"},{"key":"2022062118084934300_bib24","first-page":"18","article-title":"Data augmentation using pre-trained transformer models","volume-title":"Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems","author":"Kumar","year":"2020"},{"key":"2022062118084934300_bib25","article-title":"ALBERT: A lite BERT for self-supervised learning of language representations","volume-title":"International Conference on Learning Representations","author":"Lan","year":"2020"},{"key":"2022062118084934300_bib26","doi-asserted-by":"publisher","first-page":"3045","DOI":"10.18653\/v1\/2021.emnlp-main.243","article-title":"The power of scale for parameter- efficient prompt tuning","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Lester","year":"2021"},{"key":"2022062118084934300_bib27","doi-asserted-by":"publisher","first-page":"333","DOI":"10.18653\/v1\/K17-1034","article-title":"Zero-shot relation extraction via reading comprehension","volume-title":"Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)","author":"Levy","year":"2017"},{"key":"2022062118084934300_bib28","first-page":"4582","article-title":"Prefix- tuning: Optimizing continuous prompts for generation","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Li","year":"2021"},{"key":"2022062118084934300_bib29","article-title":"RoBERTa: A robustly optimized BERT pretraining approach","volume":"arXiv:1907.11692","author":"Liu","year":"2019","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib30","article-title":"Cutting down on prompts and parameters: Simple few-shot learning with language models","volume":"arXiv:2106.13353","author":"Logan","year":"2021","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib31","article-title":"Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity","volume":"arXiv:2104.08786","author":"Yao","year":"2021","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib32","article-title":"The natural language decathlon: Multitask learning as question answering","volume":"arXiv:1806.08730","author":"McCann","year":"2018","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib33","article-title":"Noisy channel language model prompting for few-shot text classification","volume":"arXiv:2108.04106","author":"Min","year":"2021","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib34","article-title":"Cross-task generalization via natural language crowdsourcing instructions","volume":"arXiv:2104.08773","author":"Mishra","year":"2021","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib35","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-emnlp.103","article-title":"Simulated chats for task-oriented dialog: Learning to generate conversations from instructions","volume":"arXiv:2010. 10216","author":"Mohapatra","year":"2020","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib36","unstructured":"Mohammad\n              Norouzi\n            , TomasMikolov, SamyBengio, YoramSinger, JonathonShlens, AndreaFrome, Greg S.Corrado, and JeffreyDean. 2014. Zero-shot learning by convex combination of semantic embeddings."},{"key":"2022062118084934300_bib37","first-page":"193","article-title":"Argumentative relation classification as plausibility ranking","volume-title":"Preliminary Proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019): Long Papers","author":"Opitz","year":"2019"},{"key":"2022062118084934300_bib38","article-title":"DARE: Data augmented relation extraction with GPT-2","volume":"arXiv:2004.13845","author":"Papanikolaou","year":"2020","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib39","article-title":"True few-shot learning with language models","volume-title":"Thirty-Fifth Conference on Neural Information Processing Systems","author":"Perez","year":"2021"},{"key":"2022062118084934300_bib40","article-title":"Zero-shot text classification with generative language models","volume":"arXiv: 1912.10165","author":"Puri","year":"2019","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib41","unstructured":"Alec\n              Radford\n            , JeffWu, RewonChild, DavidLuan, DarioAmodei, and IlyaSutskever. 2019. Language models are unsupervised multitask learners. Technical report, Open AI."},{"issue":"140","key":"2022062118084934300_bib42","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"Journal of Machine Learning Research"},{"key":"2022062118084934300_bib43","doi-asserted-by":"publisher","first-page":"4902","DOI":"10.18653\/v1\/2020.acl-main.442","article-title":"Beyond accuracy: Behavioral testing of NLP models with CheckList","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Ribeiro","year":"2020"},{"key":"2022062118084934300_bib44","first-page":"2152","article-title":"An embarrassingly simple approach to zero-shot learning","volume-title":"International Conference on Machine Learning","author":"Romera-Paredes","year":"2015"},{"key":"2022062118084934300_bib45","doi-asserted-by":"publisher","first-page":"1199","DOI":"10.18653\/v1\/2021.emnlp-main.92","article-title":"Label verbalization and entailment for effective zero and few-shot relation extraction","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Sainz","year":"2021"},{"key":"2022062118084934300_bib46","article-title":"Multitask prompted training enables zero-shot task generalization","volume-title":"International Conference on Learning Representations","author":"Sanh","year":"2022"},{"key":"2022062118084934300_bib47","doi-asserted-by":"publisher","first-page":"5569","DOI":"10.18653\/v1\/2020.coling-main.488","article-title":"Automatically identifying words that can serve as labels for few-shot text classification","volume-title":"Proceedings of the 28th International Conference on Computational Linguistics","author":"Schick","year":"2020"},{"key":"2022062118084934300_bib48","doi-asserted-by":"publisher","first-page":"2339","DOI":"10.18653\/v1\/2021.naacl-main.185","article-title":"It\u2019s not just size that matters: Small language models are also few-shot learners","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Schick","year":"2021"},{"key":"2022062118084934300_bib49","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.eacl-main.20","article-title":"Exploiting cloze questions for few shot text classification and natural language inference","volume":"arXiv:2001.07676","author":"Schick","year":"2020","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib50","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.555","article-title":"Generating datasets with pretrained language models","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Schick","year":"2021"},{"key":"2022062118084934300_bib51","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00434","article-title":"Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP","author":"Schick","year":"2021","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2022062118084934300_bib52","doi-asserted-by":"publisher","first-page":"4222","DOI":"10.18653\/v1\/2020.emnlp-main.346","article-title":"AutoPrompt: Eliciting knowledge from language models with automatically generated prompts","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Shin","year":"2020"},{"key":"2022062118084934300_bib53","doi-asserted-by":"publisher","first-page":"3645","DOI":"10.18653\/v1\/P19-1355","article-title":"Energy and policy considerations for deep learning in NLP","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Strubell","year":"2019"},{"key":"2022062118084934300_bib54","article-title":"NSP-BERT: A prompt-based zero- shot learner through an original pre-training task\u2013next sentence prediction","volume":"arXiv:2109.03564","author":"Yi","year":"2021","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib55","doi-asserted-by":"publisher","first-page":"4980","DOI":"10.18653\/v1\/2021.emnlp-main.407","article-title":"Improving and simplifying pattern exploiting training","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Tam","year":"2021"},{"key":"2022062118084934300_bib56","first-page":"423","article-title":"Using semantic similarity for multi-label zero- shot classification of text documents","volume-title":"Proceeding of European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. Bruges, Belgium: Elsevier","author":"Veeranna","year":"2016"},{"key":"2022062118084934300_bib57","article-title":"Superglue: A stickier benchmark for general- purpose language understanding systems","volume-title":"Advances in Neural Information Processing Systems","author":"Wang","year":"2019"},{"key":"2022062118084934300_bib58","doi-asserted-by":"crossref","first-page":"353","DOI":"10.18653\/v1\/W18-5446","article-title":"GLUE: A multi-task benchmark and analysis platform for natural language understanding","volume-title":"Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Wang","year":"2018"},{"key":"2022062118084934300_bib59","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-5446","article-title":"Do prompt-based models really understand the meaning of their prompts?","volume":"arXiv:2109.01247","author":"Webson","year":"2021","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib60","article-title":"Finetuned language models are zero-shot learners","volume-title":"International Conference on Learning Representations","author":"Wei","year":"2022"},{"key":"2022062118084934300_bib61","doi-asserted-by":"publisher","first-page":"1361","DOI":"10.18653\/v1\/2020.emnlp-main.105","article-title":"Learning from task descriptions","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Weller","year":"2020"},{"key":"2022062118084934300_bib62","article-title":"Unsupervised data augmentation for consistency training","volume":"arXiv:1904.12848","author":"Xie","year":"2019","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib63","article-title":"FewCLUE: A Chinese few-shot learning evaluation benchmark","volume":"arXiv:2107.07498","author":"Liang","year":"2021","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib64","doi-asserted-by":"publisher","first-page":"1008","DOI":"10.18653\/v1\/2020.findings-emnlp.90","article-title":"Generative data augmentation for commonsense reasoning","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2020","author":"Yang","year":"2020"},{"key":"2022062118084934300_bib65","article-title":"CrossFit: A few-shot learning challenge for cross-task generalization in NLP","volume":"arXiv:2104.08835","author":"Ye","year":"2021","journal-title":"Computing Research Repository"},{"key":"2022062118084934300_bib66","first-page":"3914","article-title":"Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP)","author":"Yin","year":"2019"},{"key":"2022062118084934300_bib67","first-page":"11328","article-title":"PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization","volume-title":"Proceedings of the 37th International Conference on Machine Learning","author":"Zhang","year":"2020"},{"key":"2022062118084934300_bib68","first-page":"649","article-title":"Character-level convolutional networks for text classification","volume-title":"Advances in Neural Information Processing Systems 28","author":"Zhang","year":"2015"},{"key":"2022062118084934300_bib69","first-page":"12697","article-title":"Calibrate before use: Improving few-shot performance of language models","volume-title":"Proceedings of the 38th International Conference on Machine Learning","author":"Zhao","year":"2021"},{"key":"2022062118084934300_bib70","doi-asserted-by":"publisher","first-page":"2065","DOI":"10.18653\/v1\/D18-1231","article-title":"Zero-shot open entity typing as type-compatible grounding","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Zhou","year":"2018"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00485\/2030692\/tacl_a_00485.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00485\/2030692\/tacl_a_00485.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,6,21]],"date-time":"2022-06-21T14:09:55Z","timestamp":1655820595000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00485\/111728\/True-Few-Shot-Learning-with-Prompts-A-Real-World"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":70,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00485","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022]]},"published":{"date-parts":[[2022]]}}}