{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T03:28:58Z","timestamp":1778556538496,"version":"3.51.4"},"reference-count":64,"publisher":"Cambridge University Press (CUP)","issue":"2","license":[{"start":{"date-parts":[[2022,2,8]],"date-time":"2022-02-08T00:00:00Z","timestamp":1644278400000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["cambridge.org"],"crossmark-restriction":true},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2022,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Many papers are chasing state-of-the-art (SOTA) numbers, and more will do so in the future. SOTA-chasing comes with many costs. SOTA-chasing squeezes out more promising opportunities such as coopetition and interdisciplinary collaboration. In addition, there is a risk that too much SOTA-chasing could lead to claims of superhuman performance, unrealistic expectations, and the next AI winter. Two root causes for SOTA-chasing will be discussed: (1) lack of leadership and (2) iffy reviewing processes. SOTA-chasing may be similar to the replication crisis in the scientific literature. The replication crisis is yet another example, like evaluation, of over-confidence in accepted practices and the scientific method, even when such practices lead to absurd consequences.<\/jats:p>","DOI":"10.1017\/s1351324922000043","type":"journal-article","created":{"date-parts":[[2022,2,8]],"date-time":"2022-02-08T03:24:10Z","timestamp":1644290650000},"page":"249-269","update-policy":"https:\/\/doi.org\/10.1017\/policypage","source":"Crossref","is-referenced-by-count":20,"title":["Emerging Trends: SOTA-Chasing"],"prefix":"10.1017","volume":"28","author":[{"given":"Kenneth Ward","family":"Church","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Valia","family":"Kordoni","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"56","published-online":{"date-parts":[[2022,2,8]]},"reference":[{"key":"S1351324922000043_ref5","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-5301"},{"key":"S1351324922000043_ref35","doi-asserted-by":"publisher","DOI":"10.1093\/biostatistics\/kxt036"},{"key":"S1351324922000043_ref63","doi-asserted-by":"publisher","DOI":"10.1177\/0956797618761661"},{"key":"S1351324922000043_ref12","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0149144"},{"key":"S1351324922000043_ref19","first-page":"1","article-title":"Introduction to the special issue on computational linguistics using large corpora","volume":"19","author":"Church","year":"1993","journal-title":"Computational Linguistics"},{"key":"S1351324922000043_ref8","unstructured":"Bengio, Y. , Deleu, T. , Rahaman, N. , Ke, R. , Lachapelle, S. , Bilaniuk, O. , Goyal, A. and Pal, C. (2019). A meta-transfer objective for learning to disentangle causal mechanisms."},{"key":"S1351324922000043_ref24","volume-title":"Introduction to Natural Language Processing","author":"Eisenstein","year":"2019"},{"key":"S1351324922000043_ref47","volume-title":"Foundations of Statistical Natural Language Processing","author":"Manning","year":"1999"},{"key":"S1351324922000043_ref50","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1449"},{"key":"S1351324922000043_ref39","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-08-051584-7.50045-0"},{"key":"S1351324922000043_ref23","doi-asserted-by":"publisher","DOI":"10.1201\/9780824746346"},{"key":"S1351324922000043_ref33","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.0020124"},{"key":"S1351324922000043_ref22","unstructured":"Craswell, N. , Mitra, B. , Yilmaz, E. , Campos, D. and Voorhees, E.M. (2020). Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820."},{"key":"S1351324922000043_ref42","unstructured":"Koch, B. , Denton, E. , Hanna, A. and Foster, J.G. (2021). Reduced, reused and recycled: The life of a dataset in machine learning research. NeurIPS."},{"key":"S1351324922000043_ref7","doi-asserted-by":"crossref","unstructured":"Bender, E.M. , Gebru, T. , McMillan-Major, A. and Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 610\u2013623.","DOI":"10.1145\/3442188.3445922"},{"key":"S1351324922000043_ref53","doi-asserted-by":"publisher","DOI":"10.1121\/1.1911801"},{"key":"S1351324922000043_ref54","unstructured":"Pierce, J.R. and Carroll, J.B. (1966). Language and machines: Computers in translation and linguistics."},{"key":"S1351324922000043_ref14","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324917000067"},{"key":"S1351324922000043_ref21","volume-title":"The Handbook of Computational Linguistics and Natural Language Processing","author":"Clark","year":"2013"},{"key":"S1351324922000043_ref26","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v31i3.2303"},{"key":"S1351324922000043_ref57","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1951.tb01366.x"},{"key":"S1351324922000043_ref44","doi-asserted-by":"publisher","DOI":"10.2307\/249270"},{"key":"S1351324922000043_ref15","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324917000389"},{"key":"S1351324922000043_ref52","volume-title":"Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy","author":"O\u2019Neil","year":"2016"},{"key":"S1351324922000043_ref58","unstructured":"Toral, A. (2020). Reassessing claims of human parity and super-human performance in machine translation at WMT 2019. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, Lisboa, Portugal: European Association for Machine Translation, pp. 185\u2013194."},{"key":"S1351324922000043_ref49","volume-title":"The Oxford Handbook of Computational Linguistics","author":"Mitkov","year":"2003"},{"key":"S1351324922000043_ref41","volume-title":"Speech and Language Processing","author":"Jurafsky","year":"2000"},{"key":"S1351324922000043_ref60","doi-asserted-by":"crossref","unstructured":"Von Foerster, H. (1960). On self-organizing systems and their environments. In Idem, Understanding Understanding: Essays of Cybernetics and Cognition, New York: Springer, pp. 1\u201320.","DOI":"10.1007\/0-387-21722-3_1"},{"key":"S1351324922000043_ref62","doi-asserted-by":"crossref","unstructured":"Voorhees, E. (2021). Coopetition in IR research. In ACM SIGIR Forum, vol. 54, pp. 1\u20133. New York, NY, USA: ACM.","DOI":"10.1145\/3483382.3483384"},{"key":"S1351324922000043_ref40","volume-title":"Statistical Methods for Speech Recognition","author":"Jelinek","year":"1997"},{"key":"S1351324922000043_ref34","doi-asserted-by":"publisher","DOI":"10.1177\/1745691612464056"},{"key":"S1351324922000043_ref3","doi-asserted-by":"publisher","DOI":"10.1038\/nature.2016.19269"},{"key":"S1351324922000043_ref37","doi-asserted-by":"publisher","DOI":"10.1093\/biostatistics\/kxt038"},{"key":"S1351324922000043_ref2","doi-asserted-by":"publisher","DOI":"10.1038\/533452a"},{"key":"S1351324922000043_ref11","first-page":"263","article-title":"The mathematics of statistical machine translation: Parameter estimation","volume":"19","author":"Brown","year":"1993","journal-title":"Computational Linguistics"},{"key":"S1351324922000043_ref51","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2021-1114"},{"key":"S1351324922000043_ref59","unstructured":"Vogel, A. and Jurafsky, D. (2012). He said, she said: Gender in the ACL Anthology. In Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, Jeju Island, Korea: Association for Computational Linguistics, pp. 33\u201341."},{"key":"S1351324922000043_ref1","unstructured":"Anderson, A. , Jurafsky, D. and McFarland, D.A. (2012). Towards a computational history of the ACL: 1980\u20132008. In Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, Jeju Island, Korea: Association for Computational Linguistics, pp. 13\u201321."},{"key":"S1351324922000043_ref13","doi-asserted-by":"publisher","DOI":"10.33011\/lilt.v6i.1245"},{"key":"S1351324922000043_ref18","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.bppf-1.1"},{"key":"S1351324922000043_ref30","doi-asserted-by":"publisher","DOI":"10.4324\/9781315270456"},{"key":"S1351324922000043_ref31","volume-title":"Deep Learning","author":"Goodfellow","year":"2016"},{"key":"S1351324922000043_ref32","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-2605"},{"key":"S1351324922000043_ref64","unstructured":"Yarowsky, D. and Florian, R. (1999). Taking the load off the conference chairs-towards a digital paper-routing assistant. In 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora."},{"key":"S1351324922000043_ref17","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324920000030"},{"key":"S1351324922000043_ref55","unstructured":"Raji, I.D. , Bender, E.M. , Paullada, A. , Denton, E. and Hanna, A. (2021). Ai and the everything in the whole wide world benchmark. NeurIPS."},{"key":"S1351324922000043_ref46","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v41i2.5297"},{"key":"S1351324922000043_ref38","doi-asserted-by":"publisher","DOI":"10.1109\/PROC.1976.10159"},{"key":"S1351324922000043_ref6","doi-asserted-by":"publisher","DOI":"10.1038\/483531a"},{"key":"S1351324922000043_ref43","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00276"},{"key":"S1351324922000043_ref36","doi-asserted-by":"publisher","DOI":"10.1093\/biostatistics\/kxt007"},{"key":"S1351324922000043_ref20","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324921000231"},{"key":"S1351324922000043_ref25","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1954.1057468"},{"key":"S1351324922000043_ref29","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-5801"},{"key":"S1351324922000043_ref48","first-page":"313","article-title":"Building a large annotated corpus of English: The Penn Treebank","volume":"19","author":"Marcus","year":"1993","journal-title":"Computational Linguistics"},{"key":"S1351324922000043_ref61","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324901002789"},{"key":"S1351324922000043_ref27","doi-asserted-by":"publisher","DOI":"10.1147\/JRD.2012.2184356"},{"key":"S1351324922000043_ref10","first-page":"79","article-title":"A statistical approach to machine translation","volume":"16","author":"Brown","year":"1990","journal-title":"Computational Linguistics"},{"key":"S1351324922000043_ref56","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2021.3058954"},{"key":"S1351324922000043_ref16","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324918000335"},{"key":"S1351324922000043_ref4","unstructured":"Banchs, R.E. (ed) (2012). Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, Jeju Island, Korea: Association for Computational Linguistics."},{"key":"S1351324922000043_ref9","volume-title":"Information Science and Statistics","author":"Bishop","year":"2016"},{"key":"S1351324922000043_ref45","doi-asserted-by":"publisher","DOI":"10.1162\/coli_a_00032"},{"key":"S1351324922000043_ref28","unstructured":"Fillmore, C.J. (1968). In Bach, Emmon & Harms, R. (eds.), Universals in Linguistic Theory. Holt, Rinehart, and Winston."}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324922000043","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,2,8]],"date-time":"2022-02-08T03:26:12Z","timestamp":1644290772000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324922000043\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,8]]},"references-count":64,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,3]]}},"alternative-id":["S1351324922000043"],"URL":"https:\/\/doi.org\/10.1017\/s1351324922000043","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,8]]},"assertion":[{"value":"\u00a9 The Author(s), 2022. Published by Cambridge University Press","name":"copyright","label":"Copyright","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http:\/\/creativecommons.org\/licenses\/by\/4.0\/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.","name":"license","label":"License","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This content has been made available to all.","name":"free","label":"Free to read"}]}}