{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T07:43:18Z","timestamp":1782805398679,"version":"3.54.5"},"reference-count":35,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2020,5,21]],"date-time":"2020-05-21T00:00:00Z","timestamp":1590019200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100007515","name":"National Science Foundation","doi-asserted-by":"publisher","award":["IIS-1562364"],"award-info":[{"award-number":["IIS-1562364"]}],"id":[{"id":"10.13039\/100007515","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Commun. ACM"],"published-print":{"date-parts":[[2020,5,21]]},"abstract":"<jats:p>Advances in how programs treat natural language words have a big impact in AI.<\/jats:p>","DOI":"10.1145\/3347145","type":"journal-article","created":{"date-parts":[[2020,5,24]],"date-time":"2020-05-24T05:11:30Z","timestamp":1590297090000},"page":"66-74","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":28,"title":["Contextual word representations"],"prefix":"10.1145","volume":"63","author":[{"given":"Noah A.","family":"Smith","sequence":"first","affiliation":[{"name":"University of Washington and Allen Institute for Artificial Intelligence, Seattle, WA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,5,22]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Morgan & Claypool","author":"Bender E.M.","year":"2013","unstructured":"Bender , E.M. Linguistic Fundamentais for Natural Language Processing: 100 Essentials from Morphology and Syntax . Morgan & Claypool , 2013 . Bender, E.M. Linguistic Fundamentais for Natural Language Processing: 100 Essentials from Morphology and Syntax. Morgan & Claypool, 2013."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.2200\/S00935ED1V02Y201907HLT043"},{"key":"e_1_2_1_3_1","first-page":"4357","article-title":"Man is to computer programmer as woman is to homemaker? Debiasing word embeddings","volume":"4349","author":"Bolukbasi T.","year":"2016","unstructured":"Bolukbasi , T. , Chang , K. , Zou , J.Y. , Saligrama , V. , and Kalai , A.T . Man is to computer programmer as woman is to homemaker? Debiasing word embeddings . In Proceedings of Advances in Neural Information Processing Systems , 2016 , 4349 -- 4357 . Bolukbasi, T., Chang, K., Zou, J.Y., Saligrama, V., and Kalai, A.T. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Proceedings of Advances in Neural Information Processing Systems, 2016, 4349--4357.","journal-title":"Proceedings of Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_4_1","first-page":"4","article-title":"Class-based n-gram models of natural language","volume":"18","author":"Brown P.F.","year":"1992","unstructured":"Brown , P.F. , Desouza , P.V. , Mercer , R.L. , Della Pietra , V.J. , and Lai , J.C . Class-based n-gram models of natural language . Computational Linguistics 18 , 4 ( 1992 ), 467--479. Brown, P.F., Desouza, P.V., Mercer, R.L., Della Pietra, V.J., and Lai, J.C. Class-based n-gram models of natural language. Computational Linguistics 18, 4 (1992), 467--479.","journal-title":"Computational Linguistics"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.aal4230"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078186"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of Advances in Neural Information Processing Systems","author":"Dai A.D.","year":"2015","unstructured":"Dai , A.D. and Le , Q.V . Semisupervised sequence learning . In Proceedings of Advances in Neural Information Processing Systems , 2015 . Dai, A.D. and Le, Q.V. Semisupervised sequence learning. In Proceedings of Advances in Neural Information Processing Systems, 2015."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of 2019 NAACL.","author":"Devlin J.","unstructured":"Devlin , J. , Chang , M.W. , Lee , K. , and Toutanova , K . BERT: Pre-training of deep bidirectional transformers for language understanding . In Proceedings of 2019 NAACL. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of 2019 NAACL."},{"key":"e_1_2_1_10_1","volume-title":"Introduction to Natural Language Processing","author":"Eisenstein J.","year":"2019","unstructured":"Eisenstein , J. Introduction to Natural Language Processing . MIT Press , 2019 . Eisenstein, J. Introduction to Natural Language Processing. MIT Press, 2019."},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of 2016 NAACL.","author":"Faruqui M.","unstructured":"Faruqui , M. , Dodge , J. , Jauhar , S.K. , Dyer , C. , Hovy , E. , and Smith , N.A . Retrofitting word vectors to semantic lexicons . In Proceedings of 2016 NAACL. Faruqui, M., Dodge, J., Jauhar, S.K., Dyer, C., Hovy, E., and Smith, N.A. Retrofitting word vectors to semantic lexicons. In Proceedings of 2016 NAACL."},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of 2014 EACL.","author":"Faruqui M.","unstructured":"Faruqui , M. and Dyer , C . Improving vector space word representations using multilingual correlation . In Proceedings of 2014 EACL. Faruqui, M. and Dyer, C. Improving vector space word representations using multilingual correlation. In Proceedings of 2014 EACL."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/7287.001.0001"},{"key":"e_1_2_1_14_1","first-page":"32","article-title":"A synopsis of linguistic theory 1930--1955","volume":"1","author":"Firth J.R","year":"1957","unstructured":"Firth , J.R . A synopsis of linguistic theory 1930--1955 . Studies in Linguistic Analysis. Blackwell , 1957 , 1 -- 32 . Firth, J.R. A synopsis of linguistic theory 1930--1955. Studies in Linguistic Analysis. Blackwell, 1957, 1--32.","journal-title":"Studies in Linguistic Analysis. Blackwell"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.2200\/S00762ED1V01Y201703HLT037"},{"key":"e_1_2_1_16_1","volume-title":"Assessing BERT's syntactic abilities. 2019","author":"Goldberg Y.","year":"1901","unstructured":"Goldberg , Y. Assessing BERT's syntactic abilities. 2019 ; arXiv: 1901 .05287. Goldberg, Y. Assessing BERT's syntactic abilities. 2019; arXiv:1901.05287."},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of 2019 ACL.","author":"Gururangan S.","unstructured":"Gururangan , S. , Dang , T. , Card , D. , and Smith , N.A . Variational pretraining for semi-supervised text classification . In Proceedings of 2019 ACL. Gururangan, S., Dang, T., Card, D., and Smith, N.A. Variational pretraining for semi-supervised text classification. In Proceedings of 2019 ACL."},{"key":"e_1_2_1_18_1","volume-title":"Universal language model finetuning for text classification. 2018","author":"Howard J.","year":"1801","unstructured":"Howard , J. and Ruder , S . Universal language model finetuning for text classification. 2018 ; arXiv: 1801 .06146. Howard, J. and Ruder, S. Universal language model finetuning for text classification. 2018; arXiv:1801.06146."},{"key":"e_1_2_1_19_1","volume-title":"Computational Linguistics, and Speech Recognition","author":"Jurafsky D.","unstructured":"Jurafsky , D. and Martin , J.H . Speech and Language Processing: An Introduction to Natural Language Processing , Computational Linguistics, and Speech Recognition ( 3 rd Ed). Prentice Hall , forthcoming. Jurafsky, D. and Martin, J.H. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition (3rd Ed). Prentice Hall, forthcoming.","edition":"3"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of 2019 NeurIPS.","author":"Lample G.","unstructured":"Lample , G. and Conneau , A . Cross-lingual language model pretraining . In Proceedings of 2019 NeurIPS. Lample, G. and Conneau, A. Cross-lingual language model pretraining. In Proceedings of 2019 NeurIPS."},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of 2015 EMNLP.","author":"Ling W.","unstructured":"Ling , W. et al. Finding function in form: Compositional character models for open vocabulary word representation . In Proceedings of 2015 EMNLP. Ling, W. et al. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of 2015 EMNLP."},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of 2019 NAACL.","author":"Liu N.F.","unstructured":"Liu , N.F. , Gardner , M. , Belinkov , Y. , Peters , M.E. and Smith , N.A . Linguistic knowledge and transferability of contextual representations . In Proceedings of 2019 NAACL. Liu, N.F., Gardner, M., Belinkov, Y., Peters, M.E. and Smith, N.A. Linguistic knowledge and transferability of contextual representations. In Proceedings of 2019 NAACL."},{"key":"e_1_2_1_23_1","volume-title":"et al. RoBERTa: A robustly optimized BERT pretraining approach. 2019","author":"Liu Y.","year":"1907","unstructured":"Liu , Y. et al. RoBERTa: A robustly optimized BERT pretraining approach. 2019 ; arXiv: 1907 .11692. Liu, Y. et al. RoBERTa: A robustly optimized BERT pretraining approach. 2019; arXiv:1907.11692."},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of 2013 ICLR.","author":"Mikolov T.","unstructured":"Mikolov , T. , Chen , K. , Corrado , G. , and Dean , J . Efficient estimation of word representations in vector space . In Proceedings of 2013 ICLR. Mikolov, T., Chen, K., Corrado, G., and Dean, J. Efficient estimation of word representations in vector space. In Proceedings of 2013 ICLR."},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of 2019 NAACL.","author":"Mulcaire P.","unstructured":"Mulcaire , P. , Kasai , J. , and Smith , N.A . Polyglot contextual representations improve crosslingual transfer . In Proceedings of 2019 NAACL. Mulcaire, P., Kasai, J., and Smith, N.A. Polyglot contextual representations improve crosslingual transfer. In Proceedings of 2019 NAACL."},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of 2013 NAACL.","author":"Owoputi O.","unstructured":"Owoputi , O. , O'Connor , B. , Dyer , C. , Gimpel , K. , Schneider , N. , and Smith , N.A . Improved part-of-speech tagging for online conversational text with word clusters . In Proceedings of 2013 NAACL. Owoputi, O., O'Connor, B., Dyer, C., Gimpel, K., Schneider, N., and Smith, N.A. Improved part-of-speech tagging for online conversational text with word clusters. In Proceedings of 2013 NAACL."},{"key":"e_1_2_1_27_1","volume-title":"Opinion Mining and Sentiment Analysis. Foundations and Trends in Information Retrieval","author":"Pang B.","year":"2008","unstructured":"Pang , B. and Lee , L . Opinion Mining and Sentiment Analysis. Foundations and Trends in Information Retrieval , Vol. 2 . Now Publishers, Inc. 2008 . Pang, B. and Lee, L. Opinion Mining and Sentiment Analysis. Foundations and Trends in Information Retrieval, Vol. 2. Now Publishers, Inc. 2008."},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of 2018 NAACL.","author":"Peters M.E.","unstructured":"Peters , M.E. , Neumann , M. , Iyyer , M. , Gardner , M. , Clark , C. , Lee , K. , and Zettlemoyer , L . Deep contextualized word representations . In Proceedings of 2018 NAACL. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. Deep contextualized word representations. In Proceedings of 2018 NAACL."},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of 2018 EMNLP.","author":"Peters M.E.","unstructured":"Peters , M.E. , Neumann , M. , Zettlemoyer , L. , and Yih , W . Dissecting contextual word embeddings: Architecture and representation . In Proceedings of 2018 EMNLP. Peters, M.E., Neumann, M., Zettlemoyer, L., and Yih, W. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of 2018 EMNLP."},{"key":"e_1_2_1_30_1","volume-title":"Language models are unsupervised multitask learners","author":"Radford A.","year":"2019","unstructured":"Radford , A. , Wu , J. , Child , R. , Luan , D. , Amodei , D. and Sutskever , I . Language models are unsupervised multitask learners . 2019 . Radford, A., Wu, J., Child, R., Luan, D., Amodei, D. and Sutskever, I. Language models are unsupervised multitask learners. 2019."},{"key":"e_1_2_1_31_1","volume-title":"et al. Exploring the limits of transfer learning with a unified text-to-text transformer. 2019","author":"Raffel C.","year":"1910","unstructured":"Raffel , C. et al. Exploring the limits of transfer learning with a unified text-to-text transformer. 2019 ; arXiv: 1910 .10983. Raffel, C. et al. Exploring the limits of transfer learning with a unified text-to-text transformer. 2019; arXiv:1910.10983."},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of 2012 Interspeech.","author":"Sundermeyer M.","unstructured":"Sundermeyer , M. , Schl\u00fcter , R. , and Ney , H . LSTM neural networks for language modeling . In Proceedings of 2012 Interspeech. Sundermeyer, M., Schl\u00fcter, R., and Ney, H. LSTM neural networks for language modeling. In Proceedings of 2012 Interspeech."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.2934"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of Advances in Neural Information Processing Systems","author":"Yang Z.","year":"2019","unstructured":"Yang , Z. , Dai , Y , Yang , Y. , Carbonell , J. , Salakhutdinov , R. and Le , Q.V . XLNet: Generalized autoregressive pretraining for language understanding . In Proceedings of Advances in Neural Information Processing Systems , 2019 .. Yang, Z., Dai, Y, Yang, Y., Carbonell, J., Salakhutdinov, R. and Le, Q.V. XLNet: Generalized autoregressive pretraining for language understanding. In Proceedings of Advances in Neural Information Processing Systems, 2019.."},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of 2018 EMNLP.","author":"Zellers R.","unstructured":"Zellers , R. , Bisk , Y. , Schwartz , R. , and Choi , Y . SWAG: A large-scale adversarial dataset for grounded commonsense inference . In Proceedings of 2018 EMNLP. Zellers, R., Bisk, Y., Schwartz, R., and Choi, Y. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of 2018 EMNLP."}],"container-title":["Communications of the ACM"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3347145","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3347145","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:23:59Z","timestamp":1750202639000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3347145"}},"subtitle":["putting words into computers"],"short-title":[],"issued":{"date-parts":[[2020,5,21]]},"references-count":35,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2020,5,21]]}},"alternative-id":["10.1145\/3347145"],"URL":"https:\/\/doi.org\/10.1145\/3347145","relation":{},"ISSN":["0001-0782","1557-7317"],"issn-type":[{"value":"0001-0782","type":"print"},{"value":"1557-7317","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,5,21]]},"assertion":[{"value":"2020-05-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}