{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T12:38:22Z","timestamp":1721911102965},"reference-count":43,"publisher":"Cambridge University Press (CUP)","issue":"5","license":[{"start":{"date-parts":[[2020,8,10]],"date-time":"2020-08-10T00:00:00Z","timestamp":1597017600000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["cambridge.org"],"crossmark-restriction":true},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2020,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Benchmarks can be a useful step toward the goals of the field (when the benchmark is on the critical path), as demonstrated by the GLUE benchmark, and deep nets such as BERT and ERNIE. The case for other benchmarks such as MUSE and WN18RR is less well established. Hopefully, these benchmarks are on a critical path toward progress on bilingual lexicon induction (BLI) and knowledge graph completion (KGC). Many KGC algorithms have been proposed such as Trans[DEHRM], but it remains to be seen how this work improves WordNet coverage. Given how much work is based on these benchmarks, the literature should have more to say than it does about the connection between benchmarks and goals. Is optimizing P@10 on WN18RR likely to produce more complete knowledge graphs? Is MUSE likely to improve Machine Translation?<\/jats:p>","DOI":"10.1017\/s1351324920000418","type":"journal-article","created":{"date-parts":[[2020,8,10]],"date-time":"2020-08-10T13:29:47Z","timestamp":1597066187000},"page":"579-592","update-policy":"http:\/\/dx.doi.org\/10.1017\/policypage","source":"Crossref","is-referenced-by-count":2,"title":["Benchmarks and goals"],"prefix":"10.1017","volume":"26","author":[{"given":"Kenneth Ward","family":"Church","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"56","published-online":{"date-parts":[[2020,8,10]]},"reference":[{"key":"S1351324920000418_ref40","doi-asserted-by":"crossref","unstructured":"Wang, Z. , Zhang, J. , Feng, J. and Chen, Z. (2014). Knowledge graph embedding by translating on hyperplanes. In AAAI.","DOI":"10.1609\/aaai.v28i1.8870"},{"key":"S1351324920000418_ref37","unstructured":"Sun, Z. , Deng, Z.-H. , Nie, J.-Y. and Tang, J. (2019). Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197."},{"key":"S1351324920000418_ref36","unstructured":"Sun, Y. , Wang, S. , Li, Y. , Feng, S. , Tian, H. , Wu, H. and Wang, H. (2020). Ernie 2.0: A continual pre-training framework for language understanding. In AAAI."},{"key":"S1351324920000418_ref35","unstructured":"Smith, S. , Turban, D. , Hamblin, S. and Hammerla, N. (2017). Offline bilingual word vectors, orthogonal transformations and the inverted softmax. arXiv preprint arXiv:1702.03859."},{"key":"S1351324920000418_ref33","unstructured":"Ruder, S. , Vuli\u0107, I. and S\u00f8gaard, A. (2017). A survey of cross-lingual word embedding models. arXiv preprint arXiv:1706.04902."},{"key":"S1351324920000418_ref31","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"S1351324920000418_ref30","unstructured":"Nguyen, D.Q. , Nguyen, T.D. , Nguyen, D.Q. and Phung, D. (2017). Novel embedding model for knowledge base completion based on convolutional neural network. arXiv preprint arXiv:1712.02121."},{"key":"S1351324920000418_ref29","unstructured":"Nguyen, D.Q. (2017). An overview of embedding models of entities and relationships for knowledge base completion. arXiv preprint arXiv:1703.08098."},{"key":"S1351324920000418_ref27","volume-title":"WordNet: An Electronic Lexical Database","author":"Miller","year":"1998"},{"key":"S1351324920000418_ref25","unstructured":"Mikolov, T. , Grave, E. , Bojanowski, P. , Puhrsch, C. and Joulin, A. (2017). Advances in pre-training distributed word representations. arXiv preprint arXiv:1712.09405."},{"key":"S1351324920000418_ref23","doi-asserted-by":"publisher","DOI":"10.3115\/1557769.1557821"},{"key":"S1351324920000418_ref21","first-page":"105","article-title":"The sketch engine","author":"Kilgarriff","year":"2004","journal-title":"Information Technology"},{"key":"S1351324920000418_ref19","first-page":"1","article-title":"Technical terminology: Some linguistic properties and an algorithm for identification in text","volume":"22","author":"Justeson","year":"1995","journal-title":"Computational Linguistics"},{"key":"S1351324920000418_ref16","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1399"},{"key":"S1351324920000418_ref13","doi-asserted-by":"publisher","DOI":"10.1007\/BF00136984"},{"key":"S1351324920000418_ref20","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1328"},{"key":"S1351324920000418_ref11","first-page":"53","volume-title":"HowNet and the Computation of Meaning","author":"Dong","year":"2010"},{"key":"S1351324920000418_ref10","unstructured":"Devlin, J. , Chang, M.-W. , Lee, K. and Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pp. 4171\u20134186."},{"key":"S1351324920000418_ref4","first-page":"263","article-title":"The mathematics of statistical machine translation: Parameter estimation","volume":"19","author":"Brown","year":"1993","journal-title":"Computational Linguistics"},{"key":"S1351324920000418_ref3","unstructured":"Bordes, A. , Usunier, N. , Garcia-Duran, A. , Weston, J. and Yakhnenko, O. (2013). Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, pp. 2787\u20132795."},{"key":"S1351324920000418_ref12","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-49478-2_1"},{"key":"S1351324920000418_ref9","doi-asserted-by":"crossref","unstructured":"Dettmers, T. , Minervini, P. , Stenetorp, P. and Riedel, S. (2018). Convolutional 2D knowledge graph embeddings. In AAAI.","DOI":"10.1609\/aaai.v32i1.11573"},{"key":"S1351324920000418_ref34","first-page":"1","article-title":"Translating collocations for bilingual lexicons: A statistical approach","volume":"22","author":"Smadja","year":"1996","journal-title":"Computational Linguistics"},{"key":"S1351324920000418_ref17","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00284"},{"key":"S1351324920000418_ref6","first-page":"1","article-title":"Introduction to the special issue on computational linguistics using large corpora","volume":"19","author":"Church","year":"1993","journal-title":"Computational Linguistics"},{"key":"S1351324920000418_ref28","unstructured":"Nickel, M. , Rosasco, L. and Poggio, T. (2019). Holographic embeddings of knowledge graphs. In AAAI."},{"key":"S1351324920000418_ref1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1073"},{"key":"S1351324920000418_ref24","doi-asserted-by":"crossref","unstructured":"Lin, Y. , Liu, Z. , Sun, M. , Liu, Y. and Zhu, X. (2015). Learning entity and relation embeddings for knowledge graph completion. In AAAI.","DOI":"10.1609\/aaai.v29i1.9491"},{"key":"S1351324920000418_ref22","unstructured":"Koehn, P. (2005). Europarl: A parallel corpus for statistical machine translation. In MT Summit, vol. 5, pp. 79\u201386."},{"key":"S1351324920000418_ref32","doi-asserted-by":"publisher","DOI":"10.3115\/981658.981709"},{"key":"S1351324920000418_ref38","unstructured":"Trouillon, T. , Welbl, J. , Riedel, S. , Gaussier, \u00c9. and Bouchard, G. (2016). Complex embeddings for simple link prediction. International Conference on Machine Learning (ICML)."},{"key":"S1351324920000418_ref7","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324920000030"},{"key":"S1351324920000418_ref18","unstructured":"Irvine, A. and Callison-Burch, C. (2013). Supervised bilingual lexicon induction with multiple monolingual signals. In NAACL."},{"key":"S1351324920000418_ref39","unstructured":"Wang, A. , Singh, A. , Michael, J. , Hill, F. , Levy, O. and Bowman, S.R. (2018). Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461."},{"key":"S1351324920000418_ref2","doi-asserted-by":"publisher","DOI":"10.1016\/S0065-2458(08)60607-5"},{"key":"S1351324920000418_ref43","unstructured":"Yu, S.Y. , Rokka Chhetri, S. , Canedo, A. , Goyal, P. , Faruque, M.A.A. (2019). Pykg2vec: A python library for knowledge graph embedding. arXiv preprint arXiv:1906.04239."},{"key":"S1351324920000418_ref41","unstructured":"Wu, Y. , Schuster, M. , Chen, Z. , Le, Q.V. , Norouzi, M. , Macherey, W. , Krikun, M. , Cao, Y. , Gao, Q. , Macherey, K. , Klingner, J. , Shah, A. , Johnson, M. , Liu, X. , Kaiser, \u0141. , Gouws, S. , Kato, Y. , Kudo, T. , Kazawa, H. , Stevens, K. , Kurian, G. , Patil, N. , Wang, W. , Young, C. , Smith, J. , Riesa, J. , Rudnick, A. , Vinyals, O. , Corrado, G. , Hughes, M. and Dean, J. (2016). Google\u2019s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144."},{"key":"S1351324920000418_ref5","unstructured":"Conneau, A. , Lample, G. , Ranzato, M. , Denoyer, L. and J\u00e9gou, H. (2017). Word translation without parallel data. arXiv preprint arXiv:1710.04087."},{"key":"S1351324920000418_ref26","unstructured":"Mikolov, T. , Le, Q.V. and Sutskever, I. (2013). Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168."},{"key":"S1351324920000418_ref14","volume-title":"The Goal: A Process of Ongoing Improvement","author":"Goldratt","year":"1984"},{"key":"S1351324920000418_ref42","unstructured":"Yang, B. , Yih, W.-T. , He, X. , Gao, J. and Deng, L. (2015). Embedding entities and relations for learning and inference in knowledge bases. In ICLR."},{"key":"S1351324920000418_ref8","doi-asserted-by":"publisher","DOI":"10.3115\/974358.974367"},{"key":"S1351324920000418_ref15","unstructured":"Hamp, B. and Feldweg, H. (1997). Germanet \u2013 A lexical-semantic net for german. In Automatic Information Extraction and Building of Lexical Semantic Resources for NLP Applications, ACL Workshop."}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324920000418","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,6]],"date-time":"2022-11-06T11:07:02Z","timestamp":1667732822000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324920000418\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,10]]},"references-count":43,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2020,9]]}},"alternative-id":["S1351324920000418"],"URL":"https:\/\/doi.org\/10.1017\/s1351324920000418","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,10]]},"assertion":[{"value":"\u00a9 The Author(s), 2020. Published by Cambridge University Press","name":"copyright","label":"Copyright","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http:\/\/creativecommons.org\/licenses\/by\/4.0\/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.","name":"license","label":"License","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This content has been made available to all.","name":"free","label":"Free to read"}]}}