{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,10]],"date-time":"2026-01-10T01:24:35Z","timestamp":1768008275775,"version":"3.49.0"},"reference-count":36,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2023,3,28]],"date-time":"2023-03-28T00:00:00Z","timestamp":1679961600000},"content-version":"vor","delay-in-days":1,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,3,27]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Complementary to finding good general word embeddings, an important question for representation learning is to find dynamic word embeddings, for example, across time or domain. Current methods do not offer a way to use or predict information on structure between sub-corpora, time or domain and dynamic embeddings can only be compared after post-alignment. We propose novel word embedding methods that provide general word representations for the whole corpus, domain- specific representations for each sub-corpus, sub-corpus structure, and embedding alignment simultaneously. We present an empirical evaluation on New York Times articles and two English Wikipedia datasets with articles on science and philosophy. Our method, called Word2Vec with Structure Prediction (W2VPred), provides better performance than baselines in terms of the general analogy tests, domain-specific analogy tests, and multiple specific word embedding evaluations as well as structure prediction performance when no structure is given a priori. As a use case in the field of Digital Humanities we demonstrate how to raise novel research questions for high literature from the German Text Archive.<\/jats:p>","DOI":"10.1162\/tacl_a_00538","type":"journal-article","created":{"date-parts":[[2023,3,28]],"date-time":"2023-03-28T16:07:31Z","timestamp":1680019651000},"page":"320-335","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":3,"title":["Domain-Specific Word Embeddings with Structure Prediction"],"prefix":"10.1162","volume":"11","author":[{"given":"David","family":"Lassner","sequence":"first","affiliation":[{"name":"TU Berlin, Germany. lassner@tu-berlin.de"},{"name":"BIFOLD, Germany"}]},{"given":"Stephanie","family":"Brandl","sequence":"additional","affiliation":[{"name":"TU Berlin, Germany"},{"name":"BIFOLD, Germany"},{"name":"University of Copenhagen, Denmark. brandl@di.ku.dk"}]},{"given":"Anne","family":"Baillot","sequence":"additional","affiliation":[{"name":"Le Mans Universit\u00e9, France"}]},{"given":"Shinichi","family":"Nakajima","sequence":"additional","affiliation":[{"name":"TU Berlin, Germany"},{"name":"BIFOLD, Germany"},{"name":"RIKEN Center for AIP, Japan"}]}],"member":"281","published-online":{"date-parts":[[2023,3,27]]},"reference":[{"key":"2023032816071446700_","doi-asserted-by":"publisher","first-page":"1509","DOI":"10.1145\/3132847.3132878","article-title":"Words are malleable: Computing semantic shifts in political and media discourse","volume-title":"Proceedings of the 2017 ACM on Conference on Information and Knowledge Management","author":"Azarbonyad","year":"2017"},{"key":"2023032816071446700_","doi-asserted-by":"publisher","first-page":"355","DOI":"10.1515\/9783110523300-016","article-title":"Die Krux mit dem Netz Verkn\u00fcpfung und Visualisierung bei digitalen Briefeditionen","volume-title":"Quantitative Ans\u00e4tze in den Literatur- und Geisteswissenschaften. Systematische und historische Perspektiven","author":"Baillot","year":"2018"},{"key":"2023032816071446700_","article-title":"Dynamic word embeddings","author":"Bamler","year":"2017","journal-title":"arXiv preprint arXiv:1702.08359"},{"issue":"7","key":"2023032816071446700_","doi-asserted-by":"publisher","first-page":"1109","DOI":"10.1080\/01419870.2015.1103886","article-title":"The effect of terrorist events on media portrayals of Islam and Muslims: Evidence from New York Times headlines, 1985\u20132013","volume":"39","author":"Bleich","year":"2016","journal-title":"Ethnic and Racial Studies"},{"issue":"3","key":"2023032816071446700_","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1093\/llc\/17.3.267","article-title":"\u2018Delta\u2019: A measure of stylistic difference and a guide to likely authorship","volume":"17","author":"Burrows","year":"2002","journal-title":"Literary and Linguistic Computing"},{"key":"2023032816071446700_","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2023032816071446700_","doi-asserted-by":"publisher","first-page":"19","DOI":"10.3115\/v1\/P14-5004","article-title":"Community evaluation and exchange of word vectors at wordvectors.org","volume-title":"Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations","author":"Faruqui","year":"2014"},{"key":"2023032816071446700_","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/7287.001.0001","article-title":"Wordnet: An electronic lexical database and some of its applications","author":"Fellbaum","year":"1998"},{"key":"2023032816071446700_","doi-asserted-by":"publisher","first-page":"538","DOI":"10.18653\/v1\/2020.acl-main.51","article-title":"Simple, interpretable and stable method for detecting words with usage change across corpora","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Gonen","year":"2020"},{"key":"2023032816071446700_","first-page":"1880","article-title":"Unsupervised alignment of embeddings with Wasserstein Procrustes","volume-title":"The 22nd International Conference on Artificial Intelligence and Statistics","author":"Grave","year":"2019"},{"key":"2023032816071446700_","doi-asserted-by":"publisher","first-page":"1489","DOI":"10.18653\/v1\/P16-1141","article-title":"Diachronic word embeddings reveal statistical laws of semantic change","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Hamilton","year":"2016"},{"key":"2023032816071446700_","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.542","article-title":"Dynamic contextualized word embeddings","author":"Hofmann","year":"2020","journal-title":"arXiv preprint arXiv:2010.12684"},{"key":"2023032816071446700_","doi-asserted-by":"publisher","first-page":"35","DOI":"10.18653\/v1\/W19-4705","article-title":"Contextualized diachronic word representations","volume-title":"Proceedings of the 1st International Workshop on Computational Approaches to Historical Language Change","author":"Jawahar","year":"2019"},{"key":"2023032816071446700_","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/E17-2068","article-title":"Bag of tricks for efficient text classification","author":"Joulin","year":"2016","journal-title":"arXiv preprint arXiv:1607.01759"},{"key":"2023032816071446700_","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-2517","article-title":"Temporal analysis of language through neural language models","author":"Kim","year":"2014","journal-title":"arXiv preprint arXiv:1405.3515"},{"key":"2023032816071446700_","article-title":"Adam: A method for stochastic optimization","author":"Kingma","year":"2014","journal-title":"arXiv preprint arXiv:1412.6980"},{"key":"2023032816071446700_","doi-asserted-by":"publisher","first-page":"625","DOI":"10.1145\/2736277.2741627","article-title":"Statistically significant detection of linguistic change","volume-title":"Proceedings of the 24th International Conference on World Wide Web","author":"Kulkarni","year":"2015"},{"key":"2023032816071446700_","first-page":"1384","article-title":"Diachronic word embeddings and semantic shifts: A survey","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics","author":"Kutuzov","year":"2018"},{"issue":"4","key":"2023032816071446700_","doi-asserted-by":"publisher","first-page":"E457","DOI":"10.1073\/pnas.1606380114","article-title":"Content analysis of 150 years of british periodicals","volume":"114","author":"Lansdall-Welfare","year":"2017","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2023032816071446700_","first-page":"2177","article-title":"Neural word embedding as implicit matrix factorization","volume-title":"Advances in neural information processing systems","author":"Levy","year":"2014"},{"key":"2023032816071446700_","article-title":"Clustering ideological terms in historical newspaper data with diachronic word embeddings","volume-title":"5th International Workshop on Computational History, HistoInformatics 2019","author":"Marjanen","year":"2019"},{"key":"2023032816071446700_","article-title":"Exploiting similarities among languages for machine translation","author":"Mikolov","year":"2013","journal-title":"CoRR"},{"key":"2023032816071446700_","first-page":"3111","article-title":"Distributed representations of words and phrases and their compositionality","volume-title":"Advances in Neural Information Processing Systems","author":"Mikolov","year":"2013"},{"key":"2023032816071446700_","volume-title":"Graphs, Maps, Trees: Abstract Models for a Literary History","author":"Moretti","year":"2005"},{"key":"2023032816071446700_","doi-asserted-by":"publisher","first-page":"1532","DOI":"10.3115\/v1\/D14-1162","article-title":"GloVe: Global vectors for word representation","volume-title":"Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)","author":"Pennington","year":"2014"},{"issue":"6","key":"2023032816071446700_","doi-asserted-by":"publisher","first-page":"777","DOI":"10.1177\/1464884909344480","article-title":"Framing the war on terror: The internalization of policy in the US press","volume":"10","author":"Reese","year":"2009","journal-title":"Journalism"},{"key":"2023032816071446700_","first-page":"45","article-title":"Software framework for topic modelling with large corpora","volume-title":"Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks","author":"\u0158eh\u016f\u0159ek","year":"2010"},{"key":"2023032816071446700_","doi-asserted-by":"publisher","first-page":"1003","DOI":"10.1145\/3178876.3185999","article-title":"Dynamic embeddings for language evolution","volume-title":"Proceedings of the 2018 World Wide Web Conference on World Wide Web","author":"Rudolph","year":"2018"},{"key":"2023032816071446700_","first-page":"478","article-title":"Exponential family embeddings","volume-title":"Advances in Neural Information Processing Systems","author":"Rudolph","year":"2016"},{"key":"2023032816071446700_","doi-asserted-by":"publisher","first-page":"66","DOI":"10.18653\/v1\/D19-1007","article-title":"Room to glo: A systematic comparison of semantic change detection approaches with word embeddings","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Shoemark","year":"2019"},{"key":"2023032816071446700_","doi-asserted-by":"publisher","DOI":"10.1109\/MLSP.2007.4414315","article-title":"Non-negative CCA for audio-visual source separation","volume-title":"Proceedings of the IEEE Workshop on Machine Learning for Signal Processing","author":"Sigg","year":"2007"},{"key":"2023032816071446700_","article-title":"Survey of computational approaches to lexical semantic change","author":"Tahmasebi","year":"2018","journal-title":"arXiv preprint arXiv:1811.06278"},{"key":"2023032816071446700_","doi-asserted-by":"publisher","first-page":"2049","DOI":"10.18653\/v1\/D15-1243","article-title":"Evaluation of word vector representations by subspace alignment","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","author":"Tsvetkov","year":"2015"},{"key":"2023032816071446700_","doi-asserted-by":"publisher","first-page":"673","DOI":"10.1145\/3159652.3159703","article-title":"Dynamic word embeddings for evolving semantic discovery","volume-title":"Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining","author":"Yao","year":"2018"},{"key":"2023032816071446700_","doi-asserted-by":"publisher","first-page":"3915","DOI":"10.24963\/ijcai.2017\/547","article-title":"Socialized word embeddings.","volume-title":"IJCAI","author":"Zeng","year":"2017"},{"issue":"10","key":"2023032816071446700_","doi-asserted-by":"publisher","first-page":"2793","DOI":"10.1109\/TKDE.2016.2591008","article-title":"The past is not a foreign country: Detecting semantically similar terms across time","volume":"28","author":"Zhang","year":"2016","journal-title":"IEEE Transactions on Knowledge and Data Engineering"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00538\/2075946\/tacl_a_00538.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00538\/2075946\/tacl_a_00538.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,3,28]],"date-time":"2023-03-28T16:07:31Z","timestamp":1680019651000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00538\/115372\/Domain-Specific-Word-Embeddings-with-Structure"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,27]]},"references-count":36,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00538","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,27]]}}}