{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T02:23:10Z","timestamp":1781749390245,"version":"3.54.5"},"reference-count":56,"publisher":"Cambridge University Press (CUP)","issue":"5","license":[{"start":{"date-parts":[[2023,8,25]],"date-time":"2023-08-25T00:00:00Z","timestamp":1692921600000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["cambridge.org"],"crossmark-restriction":true},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2024,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>GPT-3 is a large-scale natural language model developed by OpenAI that can perform many different tasks, including topic classification. Although researchers claim that it requires only a small number of in-context examples to learn a task, in practice GPT-3 requires these training examples to be either of exceptional quality or a higher quantity than easily created by hand. To address this issue, this study teaches GPT-3 to classify whether a question is related to data science by augmenting a small training set with additional examples generated by GPT-3 itself. This study compares two augmented classifiers: the Classification Endpoint with an increased training set size and the Completion Endpoint with an augmented prompt optimized using a genetic algorithm. We find that data augmentation significantly increases the accuracy of both classifiers, and that the embedding-based Classification Endpoint achieves the best accuracy of about 76%, compared to human accuracy of 85%. In this way, giving large language models like GPT-3 the ability to propose their own training examples can improve short text classification performance.<\/jats:p>","DOI":"10.1017\/s1351324923000438","type":"journal-article","created":{"date-parts":[[2023,8,25]],"date-time":"2023-08-25T05:46:10Z","timestamp":1692942370000},"page":"943-972","update-policy":"https:\/\/doi.org\/10.1017\/policypage","source":"Crossref","is-referenced-by-count":47,"title":["Improving short text classification with augmented data using GPT-3"],"prefix":"10.1017","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4695-833X","authenticated-orcid":false,"given":"Salvador V.","family":"Balkus","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Donghui","family":"Yan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"56","published-online":{"date-parts":[[2023,8,25]]},"reference":[{"key":"S1351324923000438_ref53","unstructured":"Zhang, H. , Cisse, M. , Dauphin, Y. N. and Lopez-Paz, D. (2018). mixup: Beyond empirical risk minimization. In International Conference on Learning Representations."},{"key":"S1351324923000438_ref49","unstructured":"Vaswani, A. , Shazeer, N. M. , Parmar, N. , Uszkoreit, J. , Jones, L. , Gomez, A. N. , Kaiser, L. and Polosukhin, I. (2017). Attention is all you need. arXiv:1706.03762."},{"key":"S1351324923000438_ref36","unstructured":"OpenAI (2022). Embeddings. OpenAI Documentation. Available at https:\/\/platform.openai.com\/docs\/guides\/embeddings\/what-are-embeddings"},{"key":"S1351324923000438_ref27","doi-asserted-by":"publisher","DOI":"10.1145\/3439726"},{"key":"S1351324923000438_ref37","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2009.191"},{"key":"S1351324923000438_ref15","unstructured":"GPT-3. (2020). A robot wrote this entire article. Are you scared yet, human? The Guardian. Available at https:\/\/www.theguardian.com\/commentisfree\/2020\/sep\/08\/robot-wrote-this-article-gpt-3."},{"key":"S1351324923000438_ref47","unstructured":"Sun, Y. , Wang, S. , Feng, S. , Ding, S. , Pang, C. , Shang, J. , Liu, J. , Chen, X. , Zhao, Y. , Lu, Y. , Liu, W. , Wu, Z. , Gong, W. , Liang, J. , Shang, Z. , Sun, P. , Liu, W. , Ouyang, X. , Yu, D. , Tian, H. , Wu, H. and Wang, H. (2021). Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv:2107.02137."},{"key":"S1351324923000438_ref55","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPSW55747.2022.00171"},{"key":"S1351324923000438_ref11","unstructured":"Devlin, J. , Chang, M.-W. , Lee, K. and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805."},{"key":"S1351324923000438_ref29","doi-asserted-by":"publisher","DOI":"10.1155\/2019\/5901087"},{"key":"S1351324923000438_ref9","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324920000601"},{"key":"S1351324923000438_ref22","unstructured":"Kumar, V. , Choudhary, A. and Cho, E. (2020). Data augmentation using pre-trained transformer models. In Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems. Suzhou, China: Association for Computational Linguistics, pp. 18\u201326."},{"key":"S1351324923000438_ref45","doi-asserted-by":"publisher","DOI":"10.4304\/jmm.9.5.635-643"},{"key":"S1351324923000438_ref23","first-page":"1","article-title":"A survey on text classification: From shallow to deep learning","volume":"13","author":"Li","year":"2020","journal-title":"ACM Transactions on Intelligent Systems and Technology"},{"key":"S1351324923000438_ref41","unstructured":"Radford, A. , Narasimhan, K. , Salimans, T. and Sutskever, I. (2018). Improving language understanding by generative pre-training. OpenAI. Available at https:\/\/openai.com\/research\/language-unsupervised."},{"key":"S1351324923000438_ref8","unstructured":"Dai, A. M. and Le, Q. V. (2015). Semi-supervised sequence learning. arXiv:1511.01432."},{"key":"S1351324923000438_ref21","doi-asserted-by":"publisher","DOI":"10.3390\/info10040150"},{"key":"S1351324923000438_ref12","volume-title":"North American Chapter of the Association for Computational Linguistics","author":"Du","year":"2020"},{"key":"S1351324923000438_ref4","unstructured":"Bajaj, P. , Xiong, C. , Ke, G. , Liu, X. , He, D. , Tiwary, S. , Liu, T.-Y. , Bennett, P. , Song, X. and Gao, J. (2022). Metro: Efficient denoising pretraining of large scale autoencoding language models with model generated signals. arXiv:2204.06644."},{"key":"S1351324923000438_ref28","doi-asserted-by":"publisher","DOI":"10.1155\/2018\/2497471"},{"key":"S1351324923000438_ref43","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1009"},{"key":"S1351324923000438_ref30","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2945911"},{"key":"S1351324923000438_ref33","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2017.02.008"},{"key":"S1351324923000438_ref7","unstructured":"Chen, W. , Ramos, K. , Mullaguri, K. N. and Wu, A. S. (2021). Genetic algorithms for extractive summarization. arXiv:2105.02365."},{"key":"S1351324923000438_ref26","unstructured":"Liu, Y. , Ott, M. , Goyal, N. , Du, J. , Joshi, M. , Chen, D. , Levy, O. , Lewis, M. , Zettlemoyer, L. and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692."},{"key":"S1351324923000438_ref2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6233"},{"key":"S1351324923000438_ref51","doi-asserted-by":"publisher","DOI":"10.3233\/IDA-2004-8405"},{"key":"S1351324923000438_ref54","unstructured":"Zhao, T. , Wallace, E. , Feng, S. , Klein, D. and Singh, S. (2021). Calibrate before use: Improving few-shot performance of language models. arXiv:2102.09690."},{"key":"S1351324923000438_ref18","unstructured":"Jiang, Z. , Araki, J. , Ding, H. and Neubig, G. (2022). Understanding and improving zero-shot multi-hop reasoning in generative question answering. In International Conference on Computational Linguistics."},{"key":"S1351324923000438_ref56","doi-asserted-by":"publisher","DOI":"10.11591\/ijeecs.v19.i1.pp325-335"},{"key":"S1351324923000438_ref35","unstructured":"OpenAI (2021b). Completion. OpenAI Documentation. Available at https:\/\/beta.openai.com\/docs\/guides\/completion"},{"key":"S1351324923000438_ref5","doi-asserted-by":"publisher","DOI":"10.1002\/widm.1484"},{"key":"S1351324923000438_ref3","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2017.2743240"},{"key":"S1351324923000438_ref32","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2016.03.045"},{"key":"S1351324923000438_ref38","unstructured":"Pilipiszyn, A. (2021). GPT-3 Powers the Next Generation. OpenAI. Available at https:\/\/openai.com\/blog\/gpt-3-apps\/"},{"key":"S1351324923000438_ref50","unstructured":"Wang, A. , Pruksachatkun, Y. , Nangia, N. , Singh, A. , Michael, J. , Hill, F. , Levy, O. and Bowman, S. R. (2019). SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Proceedings of the 33rd International Conference on Neural Information Processing Systems.Curran Associates Inc, pp. 3266\u20133280."},{"key":"S1351324923000438_ref44","doi-asserted-by":"publisher","DOI":"10.1186\/s40537-019-0197-0"},{"key":"S1351324923000438_ref6","unstructured":"Brown, T. B. , Mann, B. , Ryder, N. , Subbiah, M. , Kaplan, J. , Dhariwal, P. , Neelakantan, A. , Shyam, P. , Sastry, G. , Askell, A. , Agarwal, S. , Herbert-Voss, A. , Krueger, G. , Henighan, T. J. , Child, R. , Ramesh, A. , Ziegler, D. M. , Wu, J. , Winter, C. , Hesse, C. , Chen, M. , Sigler, E. , Litwin, M. , Gray, S. , Chess, B. , Clark, J. , Berner, C. , McCandlish, S. , Radford, A. , Sutskever, I. and Amodei, D. (2020). Language models are few-shot learners. arXiv:2005.14165."},{"key":"S1351324923000438_ref13","volume-title":"Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021","author":"Feng","year":"2021"},{"key":"S1351324923000438_ref14","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00032"},{"key":"S1351324923000438_ref31","doi-asserted-by":"publisher","DOI":"10.1177\/0165551515613226"},{"key":"S1351324923000438_ref48","unstructured":"Thiergart, J. , Huber, S. and Ubellacker, T. (2021). Understanding emails and drafting responses \u2013 an approach using GPT-3. arXiv:2102.03062."},{"key":"S1351324923000438_ref25","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.726"},{"key":"S1351324923000438_ref39","unstructured":"Qu, Y. , Shen, D. , Shen, Y. , Sajeev, S. , Han, J. and Chen, W. (2020). CoDA: Contrast-enhanced and diversity-promoting data augmentation for natural language understanding. arXiv:2010.08670."},{"key":"S1351324923000438_ref42","first-page":"5","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2022","journal-title":"Journal of Machine Learning Research"},{"key":"S1351324923000438_ref34","unstructured":"OpenAI (2021a). Classification. OpenAI Documentation. Available at https:\/\/beta.openai.com\/docs\/guides\/classifications"},{"key":"S1351324923000438_ref17","unstructured":"Hestness, J. , Narang, S. , Ardalani, N. , Diamos, G. , Jun, H. , Kianinejad, H. , Patwary, M. M. A. , Yang, Y. and Zhou, Y. (2017). Deep learning scaling is predictable, empirically. arXiv:1712.00409."},{"key":"S1351324923000438_ref20","first-page":"452","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)","author":"Kobayashi","year":"2018"},{"key":"S1351324923000438_ref46","doi-asserted-by":"publisher","DOI":"10.1109\/2.294849"},{"key":"S1351324923000438_ref10","doi-asserted-by":"publisher","DOI":"10.1007\/s11042-018-6083-5"},{"key":"S1351324923000438_ref52","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2021","author":"Yoo","year":"2021"},{"key":"S1351324923000438_ref16","unstructured":"Guo, X. and Yu, H. (2022). On the domain adaptation and generalization of pretrained language models: A survey. arXiv:2211.03154."},{"key":"S1351324923000438_ref24","first-page":"100","volume-title":"Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures","author":"Liu","year":"2022"},{"key":"S1351324923000438_ref19","doi-asserted-by":"publisher","DOI":"10.1007\/s11042-020-10139-6"},{"key":"S1351324923000438_ref1","unstructured":"Adhikari, A. , Ram, A. , Tang, R. and Lin, J. (2019). Docbert: Bert for document classification. arXiv:1904.08398."},{"key":"S1351324923000438_ref40","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.600"}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324923000438","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,8]],"date-time":"2024-11-08T01:04:53Z","timestamp":1731027893000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324923000438\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,25]]},"references-count":56,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,9]]}},"alternative-id":["S1351324923000438"],"URL":"https:\/\/doi.org\/10.1017\/s1351324923000438","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,25]]},"assertion":[{"value":"\u00a9 The Author(s), 2023. Published by Cambridge University Press","name":"copyright","label":"Copyright","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http:\/\/creativecommons.org\/licenses\/by\/4.0\/), which permits unrestricted re-use, distribution and reproduction, provided the original article is properly cited.","name":"license","label":"License","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This content has been made available to all.","name":"free","label":"Free to read"}]}}