{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T10:15:21Z","timestamp":1784888121430,"version":"3.55.0"},"reference-count":98,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2024,4,17]],"date-time":"2024-04-17T00:00:00Z","timestamp":1713312000000},"content-version":"vor","delay-in-days":107,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,4,16]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>While pretrained language models (PLMs) have been shown to possess a plethora of linguistic knowledge, the existing body of research has largely neglected extralinguistic knowledge, which is generally difficult to obtain by pretraining on text alone. Here, we contribute to closing this gap by examining geolinguistic knowledge, i.e., knowledge about geographic variation in language. We introduce geoadaptation, an intermediate training step that couples language modeling with geolocation prediction in a multi-task learning setup. We geoadapt four PLMs, covering language groups from three geographic areas, and evaluate them on five different tasks: fine-tuned (i.e., supervised) geolocation prediction, zero-shot (i.e., unsupervised) geolocation prediction, fine-tuned language identification, zero-shot language identification, and zero-shot prediction of dialect features. Geoadaptation is very successful at injecting geolinguistic knowledge into the PLMs: The geoadapted PLMs consistently outperform PLMs adapted using only language modeling (by especially wide margins on zero-shot prediction tasks), and we obtain new state-of-the-art results on two benchmarks for geolocation prediction and language identification. Furthermore, we show that the effectiveness of geoadaptation stems from its ability to geographically retrofit the representation space of the PLMs.<\/jats:p>","DOI":"10.1162\/tacl_a_00652","type":"journal-article","created":{"date-parts":[[2024,4,17]],"date-time":"2024-04-17T20:20:51Z","timestamp":1713385251000},"page":"411-431","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":9,"title":["Geographic Adaptation of Pretrained Language Models"],"prefix":"10.1162","volume":"12","author":[{"given":"Valentin","family":"Hofmann","sequence":"first","affiliation":[{"name":"Allen Institute for AI, US. valentinh@allenai.org"},{"name":"University of Oxford, UK"},{"name":"LMU Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Goran","family":"Glava\u0161","sequence":"additional","affiliation":[{"name":"CAIDAS, University of W\u00fcrzburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nikola","family":"Ljube\u0161i\u0107","sequence":"additional","affiliation":[{"name":"Jo\u017eef Stefan Institute, Slovenia"},{"name":"University of Ljubljana, Slovenia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Janet B.","family":"Pierrehumbert","sequence":"additional","affiliation":[{"name":"University of Oxford, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hinrich","family":"Sch\u00fctze","sequence":"additional","affiliation":[{"name":"LMU Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2024,4,16]]},"reference":[{"key":"2024041720204244600_bib1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.468","article-title":"Muppet: Massive multi-task representations with pre-finetuning","volume-title":"arXiv preprint arXiv 2101.11038","author":"Aghajanyan","year":"2021"},{"key":"2024041720204244600_bib2","volume-title":"Bosnian, Croatian, Serbian: A Grammar with Sociolinguistic Commentary","author":"Alexander","year":"2006"},{"key":"2024041720204244600_bib3","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-2134","article-title":"Distributed representations of geographically situated language","volume-title":"Annual Meeting of the Association for Computational Linguistics (ACL) 52","author":"Bamman","year":"2014"},{"key":"2024041720204244600_bib4","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1120","article-title":"Demographic dialectal variation in social media: A case study of African-American English","volume-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP) 2016","author":"Su","year":"2016"},{"key":"2024041720204244600_bib5","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1162\/tacl_a_00051","article-title":"Enriching word vectors with subword information","volume":"5","author":"Bojanowski","year":"2017","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2024041720204244600_bib6","article-title":"Language models are few-shot learners","volume-title":"Advances in Neural Information Processing Systems (NeurIPS) 33","author":"Brown","year":"2020"},{"key":"2024041720204244600_bib7","article-title":"Iglue: A benchmark for transfer learning across modalities, tasks, and languages","author":"Bugliarello","year":"2022","journal-title":"arXiv preprint arXiv:2201.11732"},{"issue":"6334","key":"2024041720204244600_bib8","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1126\/science.aal4230","article-title":"Semantics derived automatically from language corpora contain human-like biases","volume":"356","author":"Caliskan","year":"2017","journal-title":"Science"},{"key":"2024041720204244600_bib9","article-title":"Findings of the VarDial evaluation campaign 2021","volume-title":"Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial) 8","author":"Chakravarthi","year":"2021"},{"key":"2024041720204244600_bib10","article-title":"PaLM: Scaling language modeling with pathways","volume-title":"arXiv preprint arXiv 2204.02311","author":"Chowdhery","year":"2022"},{"key":"2024041720204244600_bib11","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.naacl-main.96","article-title":"Efficient hierarchical domain adaptation for pretrained language models","author":"Chronopoulou","year":"2021","journal-title":"arXiv preprint arXiv:2112.08786"},{"key":"2024041720204244600_bib12","article-title":"ELECTRA: Pre-training text encoders as discriminators rather than generators","volume-title":"International Conference on Learning Representations (ICLR) 8","author":"Clark","year":"2020"},{"issue":"1","key":"2024041720204244600_bib13","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1016\/S0095-4470(03)00009-3","article-title":"Some acoustic cues for the perceptual categorization of American English regional dialects","volume":"32","author":"Clopper","year":"2004","journal-title":"Journal of Phonetics"},{"key":"2024041720204244600_bib14","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.747","article-title":"Unsupervised cross-lingual representation learning at scale","volume-title":"Annual Meeting of the Association for Computational Linguistics (ACL) 58","author":"Conneau","year":"2020"},{"key":"2024041720204244600_bib15","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.184","article-title":"Learning to recognize dialect features","volume-title":"Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HTL) 2021","author":"Demszky","year":"2021"},{"key":"2024041720204244600_bib16","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HTL) 2019","author":"Devlin","year":"2019"},{"key":"2024041720204244600_bib17","article-title":"Time-aware language models as temporal knowledge bases","volume-title":"arXiv preprint arXiv 2106.15110","author":"Dhingra","year":"2021"},{"key":"2024041720204244600_bib18","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/E14-1011","article-title":"Mapping dialectal variation by querying social media","volume-title":"Conference of the European Chapter of the Association for Computational Linguistics (EACL) 14","author":"Doyle","year":"2014"},{"key":"2024041720204244600_bib19","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1128","article-title":"The hitchhiker\u2019s guide to testing statistical significance in natural language processing","volume-title":"Annual Meeting of the Association for Computational Linguistics (ACL) 56","author":"Dror","year":"2018"},{"key":"2024041720204244600_bib20","article-title":"Modeling global syntactic variation in English using dialect classification","volume-title":"Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial) 6","author":"Dunn","year":"2019"},{"key":"2024041720204244600_bib21","article-title":"A systematic analysis of morphological content in BERT models for multiple languages","volume-title":"arXiv preprint arXiv 2004.03032","author":"Edmiston","year":"2020"},{"key":"2024041720204244600_bib22","article-title":"A latent variable model for geographic lexical variation","volume-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP) 2010","author":"Eisenstein","year":"2010"},{"issue":"11","key":"2024041720204244600_bib23","doi-asserted-by":"publisher","first-page":"e113114","DOI":"10.1371\/journal.pone.0113114","article-title":"Diffusion of lexical change in social media","volume":"9","author":"Eisenstein","year":"2014","journal-title":"PloS ONE"},{"key":"2024041720204244600_bib24","article-title":"Discovering sociolinguistic associations with structured sparsity","volume-title":"Annual Meeting of the Association for Computational Linguistics (ACL) 49","author":"Eisenstein","year":"2011"},{"key":"2024041720204244600_bib25","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1006","article-title":"How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings","volume-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP) 2019","author":"Ethayarajh","year":"2019"},{"key":"2024041720204244600_bib26","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1162\/tacl_a_00298","article-title":"What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models","volume":"8","author":"Ettinger","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"issue":"2","key":"2024041720204244600_bib27","doi-asserted-by":"publisher","first-page":"399","DOI":"10.1017\/S004740451900099X","article-title":"Simultaneity and the refusal to choose: The semiotics of Serbian youth identity on Facebook","volume":"49","author":"George","year":"2019","journal-title":"Language in Society"},{"key":"2024041720204244600_bib28","doi-asserted-by":"publisher","first-page":"6350","DOI":"10.18653\/v1\/2020.coling-main.559","article-title":"Xhate-999: Analyzing and detecting abusive language across domains and languages","volume-title":"Proceedings of the 28th International Conference on Computational Linguistics","author":"Glava\u0161","year":"2020"},{"key":"2024041720204244600_bib29","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.eacl-main.270","article-title":"Is supervised syntactic parsing beneficial for language understanding tasks? An empirical investigation","volume-title":"Conference of the European Chapter of the Association for Computational Linguistics (EACL) 16","author":"Glava\u0161","year":"2021"},{"key":"2024041720204244600_bib30","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.740","article-title":"Don\u2019t stop pretraining: Adapt language models to domains and tasks","volume-title":"Annual Meeting of the Association for Computational Linguistics (ACL) 58","author":"Gururangan","year":"2020"},{"key":"2024041720204244600_bib31","article-title":"Discriminating between similar Nordic languages","volume-title":"Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial) 8","author":"Haas","year":"2021"},{"key":"2024041720204244600_bib32","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1141","article-title":"Diachronic word embeddings reveal statistical laws of semantic change","volume-title":"Annual Meeting of the Association for Computational Linguistics (ACL) 54","author":"Hamilton","year":"2016"},{"key":"2024041720204244600_bib33","article-title":"A structural probe for finding syntax in word representations","volume-title":"Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HTL) 2019","author":"Hewitt","year":"2019"},{"key":"2024041720204244600_bib34","article-title":"Training compute-optimal large language models","volume-title":"arXiv preprint arXiv 2203.15556","author":"Hoffmann","year":"2022"},{"key":"2024041720204244600_bib35","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.316","article-title":"DagoBERT: Generating derivational morphology with a pretrained language model","volume-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP) 2020","author":"Hofmann","year":"2020"},{"key":"2024041720204244600_bib36","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.542","article-title":"Dynamic contextualized word embeddings","volume-title":"Annual Meeting of the Association for Computational Linguistics (ACL) 59","author":"Hofmann","year":"2021"},{"issue":"2","key":"2024041720204244600_bib37","first-page":"65","article-title":"A simple sequentially rejective multiple test procedure","volume":"6","author":"Holm","year":"1979","journal-title":"Scandinavian Journal of Statistics"},{"key":"2024041720204244600_bib38","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.564","article-title":"Surface form competition: Why the highest probability answer isn\u2019t always right","volume-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP) 2021","author":"Holtzman","year":"2021"},{"key":"2024041720204244600_bib39","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1469","article-title":"Capturing regional variation with distributed place representations and geographic retrofitting","volume-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP) 2018","author":"Hovy","year":"2018"},{"key":"2024041720204244600_bib40","doi-asserted-by":"publisher","first-page":"3719","DOI":"10.1007\/978-3-030-02438-3_175","article-title":"Visualizing regional language variation across Europe on Twitter","volume-title":"Handbook of the Changing World Language Map","author":"Hovy","year":"2020"},{"key":"2024041720204244600_bib41","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.49","article-title":"The importance of modeling social factors of language: Theory and practice","volume-title":"Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HTL) 2021","author":"Hovy","year":"2021"},{"issue":"9","key":"2024041720204244600_bib42","doi-asserted-by":"publisher","first-page":"244","DOI":"10.1016\/j.compenvurbsys.2015.12.003","article-title":"Understanding U.S. regional linguistic variation with Twitter data analysis","volume":"59","author":"Huang","year":"2016","journal-title":"Computers, Environment and Urban Systems"},{"key":"2024041720204244600_bib43","article-title":"Learning word-level dialectal variation as phonological replacement rules using a limited parallel corpus","volume-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP) 2011","author":"Hulden","year":"2011"},{"key":"2024041720204244600_bib44","doi-asserted-by":"crossref","first-page":"891","DOI":"10.18653\/v1\/2022.findings-acl.72","article-title":"Ds-tod: Efficient domain specialization for task-oriented dialog","volume-title":"Findings of the Association for Computational Linguistics: ACL 2022","author":"Hung","year":"2022"},{"key":"2024041720204244600_bib45","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1356","article-title":"What does BERT learn about the structure of language?","volume-title":"Annual Meeting of the Association for Computational Linguistics (ACL) 57","author":"Jawahar","year":"2019"},{"key":"2024041720204244600_bib46","doi-asserted-by":"publisher","first-page":"6282","DOI":"10.18653\/v1\/2020.acl-main.560","article-title":"The state and fate of linguistic diversity and inclusion in the NLP world","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Joshi","year":"2020"},{"key":"2024041720204244600_bib47","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/E17-2068","article-title":"Bag of tricks for efficient text classification","volume-title":"Conference of the European Chapter of the Association for Computational Linguistics (EACL) 15","author":"Joulin","year":"2017"},{"key":"2024041720204244600_bib48","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-emnlp.410","article-title":"GlotLID: Language identification for low-resource languages","volume-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP) 2023","author":"Kargaran","year":"2023"},{"key":"2024041720204244600_bib49","article-title":"SocialBERT: Transformers for online social network language modelling","volume-title":"arXiv preprint arXiv 2111.07148","author":"Karpov","year":"2021"},{"key":"2024041720204244600_bib50","article-title":"What uncertainties do we need in Bayesian deep learning for computer vision?","volume-title":"Advances in Neural Information Processing Systems (NIPS) 31","author":"Kendall","year":"2017"},{"key":"2024041720204244600_bib51","article-title":"Multi-task learning using uncertainty to weigh losses for scene geometry and semantics","volume-title":"Conference on Computer Vision and Pattern Recognition (CVPR) 31","author":"Kendall","year":"2018"},{"key":"2024041720204244600_bib52","article-title":"Adam: A method for stochastic optimization","volume-title":"International Conference on Learning Representations (ICLR) 3","author":"Kingma","year":"2015"},{"key":"2024041720204244600_bib53","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v10i1.14798","article-title":"Freshman or fresher? Quantifying the geographic variation of language in online social media","volume-title":"International AAAI Conference on Weblogs and Social Media (ICWSM) 10","author":"Kulkarni","year":"2016"},{"key":"2024041720204244600_bib54","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-emnlp.254","article-title":"LMSOC: An approach for socially sensitive pretraining","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2021","author":"Kulkarni","year":"2021"},{"key":"2024041720204244600_bib55","doi-asserted-by":"crossref","DOI":"10.5617\/dhnbpub.11050","article-title":"The Nordic Tweet Stream: A dynamic real-time monitor corpus of big and rich language data","volume-title":"Digital Humanities in the Nordic Countries (DHN) 3","author":"Laitinen","year":"2018"},{"issue":"2","key":"2024041720204244600_bib56","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1016\/S0095-4470(19)31044-7","article-title":"The effect of phonetic complexity on speaker race and sex identifications","volume":"7","author":"Lass","year":"1979","journal-title":"Journal of Phonetics"},{"key":"2024041720204244600_bib57","doi-asserted-by":"publisher","first-page":"43","DOI":"10.18653\/v1\/2020.deelio-1.5","article-title":"Common sense or world knowledge? Investigating adapter-based knowledge injection into pretrained transformers","volume-title":"Proceedings of Deep Learning Inside Out (DeeLIO): The First Workshop on Knowledge Extraction and Integration for Deep Learning Architectures","author":"Lauscher","year":"2020"},{"issue":"2","key":"2024041720204244600_bib58","doi-asserted-by":"publisher","first-page":"149","DOI":"10.1007\/s10791-022-09406-x","article-title":"On cross-lingual retrieval with multilingual text encoders","volume":"25","author":"Litschko","year":"2022","journal-title":"Information Retrieval Journal"},{"key":"2024041720204244600_bib59","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/P19-1441","article-title":"Multi-task deep neural networks for natural language understanding","volume-title":"Annual Meeting of the Association for Computational Linguistics (ACL) 57","author":"Liu","year":"2019"},{"key":"2024041720204244600_bib60","article-title":"RoBERTa: A robustly optimized BERT pretraining approach","volume-title":"arXiv preprint arXiv 1907.11692","author":"Liu","year":"2019"},{"key":"2024041720204244600_bib61","article-title":"Berti\u0107: The transformer language model for Bosnian, Croatian, Montenegrin and Serbian","volume-title":"Workshop on Balto-Slavic Natural Language Processing (BSNLP) 8","author":"Ljube\u0161i\u0107","year":"2021"},{"issue":"2","key":"2024041720204244600_bib62","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1017\/jlg.2018.9","article-title":"Borders and boundaries in Bosnian, Croatian, Montenegrin and Serbian: Twitter data to the rescue","volume":"6","author":"Ljube\u0161i\u0107","year":"2018","journal-title":"Journal of Linguistic Geography"},{"key":"2024041720204244600_bib63","article-title":"Time waits for no one! Analysis and challenges of temporal misalignment","volume-title":"arXiv preprint arXiv 2111.07408","author":"Luu","year":"2021"},{"key":"2024041720204244600_bib64","article-title":"Dissociating language and thought in large language models: A cognitive perspective","volume-title":"arXiv preprint arXiv 2301.06627","author":"Mahowald","year":"2023"},{"issue":"2","key":"2024041720204244600_bib65","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1007\/BF02295996","article-title":"Note on the sampling error of the difference between correlated proportions or percentages","volume":"12","author":"McNemar","year":"1947","journal-title":"Psychometrika"},{"key":"2024041720204244600_bib66","doi-asserted-by":"publisher","first-page":"1791","DOI":"10.18653\/v1\/2022.naacl-main.130","article-title":"Bad-x: Bilingual adapters improve zero-shot cross-lingual transfer","volume-title":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Parovi\u0107","year":"2022"},{"key":"2024041720204244600_bib67","doi-asserted-by":"publisher","first-page":"7654","DOI":"10.18653\/v1\/2020.emnlp-main.617","article-title":"Mad-x: An adapter-based framework for multi-task cross-lingual transfer","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Pfeiffer","year":"2020"},{"key":"2024041720204244600_bib68","article-title":"Sentence encoders on STILTs: Supplementary training on intermediate labeled-data tasks","author":"Phang","year":"2018","journal-title":"arXiv 1811.01088"},{"key":"2024041720204244600_bib69","volume-title":"Language models are unsupervised multitask learners","author":"Radford","year":"2019"},{"key":"2024041720204244600_bib70","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"Journal of Machine Learning Research"},{"key":"2024041720204244600_bib71","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P15-2104","article-title":"Twitter user geolocation using a unified text and network prediction model","volume-title":"Annual Meeting of the Association for Computational Linguistics (ACL) 53","author":"Rahimi","year":"2015"},{"key":"2024041720204244600_bib72","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-2033","article-title":"A neural model for user geolocation and lexical dialectology","volume-title":"Annual Meeting of the Association for Computational Linguistics (ACL) 55","author":"Rahimi","year":"2017"},{"key":"2024041720204244600_bib73","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1187","article-title":"Semi-supervised user geolocation via graph convolutional networks","volume-title":"Annual Meeting of the Association for Computational Linguistics (ACL) 56","author":"Rahimi","year":"2018"},{"key":"2024041720204244600_bib74","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/N15-1153","article-title":"Exploiting text and network context for geolocation of social media users","volume-title":"Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT) 2015","author":"Rahimi","year":"2015"},{"key":"2024041720204244600_bib75","doi-asserted-by":"publisher","first-page":"1351","DOI":"10.1613\/jair.1.13083","article-title":"Crossing the conversational chasm: A primer on natural language processing for multilingual task-oriented dialogue systems","volume":"74","author":"Razumovskaia","year":"2022","journal-title":"Journal of Artificial Intelligence Research"},{"key":"2024041720204244600_bib76","doi-asserted-by":"publisher","first-page":"842","DOI":"10.1162\/tacl_a_00349","article-title":"A primer in bertology: What we know about how BERT works","volume":"8","author":"Rogers","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2024041720204244600_bib77","doi-asserted-by":"publisher","DOI":"10.1145\/3488560.3498529","article-title":"Time masking for temporal language models","volume-title":"International Conference on Web Search and Data Mining (WSDM) 15","author":"Rosin","year":"2022"},{"key":"2024041720204244600_bib78","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-emnlp.206","article-title":"Temporal adaptation of BERT and performance on downstream document classification: Insights from social media","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2021","author":"R\u00f6ttger","year":"2021"},{"key":"2024041720204244600_bib79","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.vardial-1.11","article-title":"BENCHi\u0107-lang: A benchmark for discriminating between Bosnian, Croatian, Montenegrin and Serbian","volume-title":"Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial) 10","author":"Rupnik","year":"2023"},{"key":"2024041720204244600_bib80","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-4415","article-title":"Huntsville, hospitals, and hockey teams: Names can reveal your location","volume-title":"Workshop on Noisy User-generated Text (WNUT) 3","author":"Salehi","year":"2017"},{"key":"2024041720204244600_bib81","article-title":"HeLju@VarDial 2020: Social media variety geolocation with BERT models","volume-title":"Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial) 7","author":"Scherrer","year":"2020"},{"key":"2024041720204244600_bib82","article-title":"Social media variety geolocation with geoBERT","volume-title":"Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial) 8","author":"Scherrer","year":"2021"},{"issue":"1","key":"2024041720204244600_bib83","doi-asserted-by":"publisher","DOI":"10.1007\/BF02289451","article-title":"A generalized solution of the orthogonal Procrustes problem","volume":"36","author":"Sch\u00f6nemann","year":"1966","journal-title":"Psychometrika"},{"key":"2024041720204244600_bib84","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1355","article-title":"Energy and policy considerations for deep learning in NLP","volume-title":"Annual Meeting of the Association for Computational Linguistics (ACL) 57","author":"Strubell","year":"2019"},{"key":"2024041720204244600_bib85","article-title":"LLaMA: Open and efficient foundation language models","volume-title":"arXiv 2302.13971","author":"Touvron","year":"2023"},{"issue":"5","key":"2024041720204244600_bib86","doi-asserted-by":"publisher","first-page":"2936","DOI":"10.1121\/1.414099","article-title":"Voice quality: Listener identification of African\u2013American versus Caucasian speakers","volume":"98","author":"Trent","year":"1995","journal-title":"The Journal of the Acoustical Society of America"},{"key":"2024041720204244600_bib87","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.635","article-title":"Exploring and predicting transferability across NLP tasks","volume-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP) 2020","author":"Tu","year":"2020"},{"key":"2024041720204244600_bib88","doi-asserted-by":"publisher","first-page":"7222","DOI":"10.18653\/v1\/2020.emnlp-main.586","article-title":"Probing pretrained language models for lexical semantics","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Vuli\u0107","year":"2020"},{"key":"2024041720204244600_bib89","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.72","article-title":"Frequency effects on syntactic rule learning in transformers","volume-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP) 2021","author":"Wei","year":"2021"},{"key":"2024041720204244600_bib90","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.401","article-title":"Counting the bugs in ChatGPT\u2019s wugs: A multilingual investigation into the morphological capabilities of a large language model","volume-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP) 2023","author":"Weissweiler","year":"2023"},{"key":"2024041720204244600_bib91","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.746","article-title":"The better your syntax, the better your semantics? Probing pretrained language models for the English comparative correlative","volume-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP) 2022","author":"Weissweiler","year":"2022"},{"key":"2024041720204244600_bib92","article-title":"Does BERT make any sense? Interpretable word sense disambiguation with contextualized embeddings","volume-title":"arXiv preprint arXiv 1909.10430","author":"Wiedemann","year":"2019"},{"issue":"1","key":"2024041720204244600_bib93","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1146\/annurev-linguist-030514-124930","article-title":"Advances in dialectometry","volume":"1","author":"Wieling","year":"2015","journal-title":"Annual Review of Linguistics"},{"key":"2024041720204244600_bib94","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-demos.6","article-title":"Transformers: State-of-the-art natural language processing","volume-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP) 2020","author":"Wolf","year":"2020"},{"key":"2024041720204244600_bib95","article-title":"Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model","volume-title":"International Conference on Learning Representations (ICLR) 8","author":"Xiong","year":"2020"},{"key":"2024041720204244600_bib96","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-1201","article-title":"Findings of the VarDial evaluation campaign 2017","volume-title":"Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial) 4","author":"Zampieri","year":"2017"},{"key":"2024041720204244600_bib97","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-5307","article-title":"A report on the DSL shared task 2014","volume-title":"Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects (VarDial) 1","author":"Zampieri","year":"2014"},{"key":"2024041720204244600_bib98","article-title":"Calibrate before use: Improving few-shot performance of language models","volume-title":"International Conference on Machine Learning (ICML) 38","author":"Zhao","year":"2021"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00652\/2364111\/tacl_a_00652.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00652\/2364111\/tacl_a_00652.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,16]],"date-time":"2024-11-16T13:15:00Z","timestamp":1731762900000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00652\/120648\/Geographic-Adaptation-of-Pretrained-Language"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":98,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00652","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}