{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,13]],"date-time":"2026-08-13T09:02:04Z","timestamp":1786611724095,"version":"build-2736575974"},"reference-count":86,"publisher":"MIT Press","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2013,9]]},"abstract":"<jats:p>Word sense disambiguation (WSD) is an old and important task in computational linguistics that still remains challenging, to machines as well as to human annotators. Recently there have been several proposals for representing word meaning in context that diverge from the traditional use of a single best sense for each occurrence. They represent word meaning in context through multiple paraphrases, as points in vector space, or as distributions over latent senses. New methods of evaluating and comparing these different representations are needed.<\/jats:p><jats:p>In this paper we propose two novel annotation schemes that characterize word meaning in context in a graded fashion. In WSsim annotation, the applicability of each dictionary sense is rated on an ordinal scale. Usim annotation directly rates the similarity of pairs of usages of the same lemma, again on a scale. We find that the novel annotation schemes show good inter-annotator agreement, as well as a strong correlation with traditional single-sense annotation and with annotation of multiple lexical paraphrases. Annotators make use of the whole ordinal scale, and give very fine-grained judgments that \u201cmix and match\u201d senses for each individual usage. We also find that the Usim ratings obey the triangle inequality, justifying models that treat usage similarity as metric.<\/jats:p><jats:p>There has recently been much work on grouping senses into coarse-grained groups. We demonstrate that graded WSsim and Usim ratings can be used to analyze existing coarse-grained sense groupings to identify sense groups that may not match intuitions of untrained native speakers. In the course of the comparison, we also show that the WSsim ratings are not subsumed by any static sense grouping.<\/jats:p>","DOI":"10.1162\/coli_a_00142","type":"journal-article","created":{"date-parts":[[2012,11,16]],"date-time":"2012-11-16T14:50:51Z","timestamp":1353077451000},"page":"511-554","source":"Crossref","is-referenced-by-count":23,"title":["Measuring Word Meaning in Context"],"prefix":"10.1162","volume":"39","author":[{"given":"Katrin","family":"Erk","sequence":"first","affiliation":[{"name":"University of Texas at Austin"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Diana","family":"McCarthy","sequence":"additional","affiliation":[{"name":"University of Cambridge"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicholas","family":"Gaylord","sequence":"additional","affiliation":[{"name":"University of Texas at Austin"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4020-4809-8"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.3115\/1621474"},{"key":"R3","first-page":"1,183","volume-title":"Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing","author":"Baroni Marco","year":"2010"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.3115\/1557690.1557762"},{"key":"R5","unstructured":"Brown, Susan. 2010. Finding Meaning: Sense Inventories for Improved Word Sense Disambiguation. Ph.D. thesis, University of Colorado at Boulder."},{"key":"R6","first-page":"969","volume-title":"Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC 2006)","author":"Burchardt Aljoscha","year":"2006"},{"key":"R7","first-page":"43","volume-title":"Proceedings of the 11th Conference on Theoretical and Methodological Issues in Machine Translation (TMI 2007)","author":"Carpuat Marine","year":"2007"},{"key":"R8","first-page":"61","volume-title":"Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL 2007)","author":"Carpuat Marine","year":"2007"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-009-9085-0"},{"key":"R10","first-page":"345","volume":"36","author":"Coecke Bob","year":"2010","journal-title":"Lambek Festschrift, Linguistic Analysis"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.2307\/414285"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1093\/jos\/12.1.15"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511527227.004"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.1145\/1273496.1273523"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.3115\/1699510.1699514"},{"key":"R16","first-page":"1,162","volume-title":"Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing","author":"Dinu Georgiana","year":"2010"},{"key":"R17","volume-title":"Proceedings of the SensEval-2 Workshop.","author":"Edmonds Philip","year":"2001"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.3115\/1699510.1699568"},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.3115\/1687878.1687882"},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.3115\/1613715.1613831"},{"key":"R21","first-page":"92","volume-title":"Proceedings of the ACL 2010 Conference Short Papers","author":"Erk Katrin","year":"2010"},{"key":"R22","volume-title":"Proceedings of the 5th International Workshop on Semantic Evaluation(SemEval).","author":"Erk Katrin","year":"2010"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1016\/0749-596X(90)90071-7"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.3758\/BF03200458"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066X.52.1.45"},{"key":"R26","first-page":"1,394","volume-title":"Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing","author":"Grefenstette Edward","year":"2011"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.1016\/S0022-5371(79)90246-9"},{"key":"R28","doi-asserted-by":"publisher","DOI":"10.1080\/15326900701326402"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.1023\/A:1002471322828"},{"key":"R30","doi-asserted-by":"publisher","DOI":"10.3115\/1614049.1614064"},{"key":"R31","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4020-4809-8_3"},{"key":"R32","unstructured":"Kilgarriff, Adam. 1992. Polysemy. Ph.D. thesis, University of Sussex."},{"key":"R33","doi-asserted-by":"publisher","DOI":"10.1023\/A:1000583911091"},{"key":"R34","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4020-4809-8_2"},{"issue":"1","key":"R35","volume-title":"Senseval: Special Issue of the Journal Computers and the Humanities","volume":"34","author":"Kilgarriff Adam","year":"2000"},{"key":"R36","doi-asserted-by":"publisher","DOI":"10.1023\/A:1002693207386"},{"key":"R37","first-page":"89","volume-title":"Handbook of Latent Semantic Analysis.","author":"Kintsch Walter","year":"2007"},{"key":"R38","doi-asserted-by":"publisher","DOI":"10.1006\/jmla.2001.2779"},{"key":"R39","doi-asserted-by":"publisher","DOI":"10.1016\/S0749-596X(02)00020-7"},{"key":"R40","doi-asserted-by":"publisher","DOI":"10.1006\/brln.2001.2518"},{"key":"R41","doi-asserted-by":"publisher","DOI":"10.1037\/a0013012"},{"key":"R42","doi-asserted-by":"publisher","DOI":"10.1023\/A:1002407003264"},{"key":"R43","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.104.2.211"},{"key":"R44","volume-title":"WordNet: An Electronic Lexical Database.","author":"Landes Shari","year":"1998"},{"key":"R45","first-page":"15","volume-title":"Proceedings of the 5th International Workshop on Semantic Evaluation","author":"Lefever Els","year":"2010"},{"key":"R46","doi-asserted-by":"publisher","DOI":"10.3115\/1118675.1118691"},{"key":"R47","doi-asserted-by":"publisher","DOI":"10.3115\/1621474.1621483"},{"issue":"2","key":"R48","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1007\/s10579-009-9084-1","volume":"43","author":"McCarthy Diana","year":"2009","journal-title":"Language Resources and Evaluation Special Issue on Computational Semantic Analysis of Language: SemEval-2007 and Beyond"},{"key":"R49","doi-asserted-by":"publisher","DOI":"10.4324\/9780203338001"},{"key":"R50","first-page":"53","volume-title":"Proceedings of the EACL 2003 Workshop on Linguistically Annotated Corpora (LINC 2003)","author":"Mihalcea Rada","year":"2003"},{"key":"R51","first-page":"25","volume-title":"Proceedings SENSEVAL-3 Second International Workshop on Evaluating Word Sense Disambiguation Systems","author":"Mihalcea Rada","year":"2004"},{"key":"R52","volume-title":"Proceedings SENSEVAL-3 Second International Workshop on Evaluating Word Sense Disambiguation Systems","author":"Mihalcea Rada","year":"2004"},{"key":"R53","first-page":"9","volume-title":"Proceedings of the 5th International Workshop on Semantic Evaluation","author":"Mihalcea Rada","year":"2010"},{"key":"R54","doi-asserted-by":"publisher","DOI":"10.3115\/1075671.1075742"},{"key":"R55","first-page":"236","volume-title":"Proceedings of ACL-08: HLT","author":"Mitchell Jeff","year":"2008"},{"key":"R56","doi-asserted-by":"publisher","DOI":"10.1111\/j.1551-6709.2010.01106.x"},{"key":"R57","author":"Moon Taesun","journal-title":"ACM Transactions on Intelligent Systems and Technology"},{"key":"R58","first-page":"11","volume-title":"The Psychology of Word Meanings.","author":"Murphy Gregory L.","year":"1991"},{"key":"R59","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/1602.001.0001"},{"key":"R60","doi-asserted-by":"publisher","DOI":"10.1145\/1459352.1459355"},{"key":"R61","doi-asserted-by":"publisher","DOI":"10.3115\/1621474.1621480"},{"key":"R62","doi-asserted-by":"publisher","DOI":"10.1017\/S135132490500402X"},{"key":"R63","first-page":"3,244","volume-title":"Proceedings of LREC-7","author":"Passonneau Rebecca","year":"2010"},{"key":"R64","doi-asserted-by":"publisher","DOI":"10.1037\/0278-7393.27.2.556"},{"key":"R65","doi-asserted-by":"publisher","DOI":"10.3115\/1621474.1621490"},{"key":"R66","volume-title":"Proceedings of Senseval-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems","author":"Preiss Judita","year":"2001"},{"issue":"4","key":"R67","first-page":"409","volume":"17","author":"Pustejovsky James","year":"1991","journal-title":"Computational Linguistics"},{"key":"R68","first-page":"210","volume-title":"Proceedings of The 5th International Joint Conference on Natural Language Processing 2011 (IJCNLP 2011)","author":"Reddy Siva","year":"2011"},{"key":"R69","first-page":"109","volume-title":"Proceedings of Human Language Technologies: The 11th Annual Conference of the North American Chapter of the Association for Computational Linguistics","author":"Reisinger Joseph","year":"2010"},{"issue":"3","key":"R70","first-page":"113","volume":"5","author":"Resnik Philip","year":"2000","journal-title":"Natural Language Engineering"},{"key":"R71","doi-asserted-by":"publisher","DOI":"10.1037\/0096-3445.104.3.192"},{"key":"R72","doi-asserted-by":"publisher","DOI":"10.1016\/0010-0285(75)90024-9"},{"issue":"1","key":"R73","first-page":"97","volume":"24","author":"Sch\u00fctze Hinrich","year":"1998","journal-title":"Computational Linguistics"},{"key":"R75","first-page":"41","volume-title":"3rd International Workshop on Semantic Evaluations (SensEval-3) at ACL-2004","author":"Snyder Benjamin","year":"2004"},{"key":"R76","first-page":"801","volume-title":"Advances in Neural Information Processing Systems 24","author":"Socher Richard","year":"2011"},{"key":"R77","doi-asserted-by":"publisher","DOI":"10.3115\/1220575.1220626"},{"key":"R78","doi-asserted-by":"crossref","DOI":"10.1093\/oso\/9780199266647.001.0001","volume-title":"Linguistic Categorization.","author":"Taylor John R.","year":"2003"},{"key":"R79","first-page":"948","volume-title":"Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics","author":"Thater Stefan","year":"2010"},{"key":"R80","doi-asserted-by":"publisher","DOI":"10.1515\/cogl.1993.4.3.273"},{"key":"R81","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.84.4.327"},{"key":"R82","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.89.2.123"},{"key":"R83","first-page":"1,012","volume-title":"Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing","author":"Van de Cruys Tim","year":"2011"},{"key":"R84","first-page":"45","volume-title":"Proceedings of the 2010 Workshop on GEometrical Models of Natural Language Semantics","author":"Washtell Justin","year":"2010"},{"key":"R85","doi-asserted-by":"publisher","DOI":"10.1007\/BF01068072"},{"key":"R86","first-page":"78","volume-title":"Proceedings of the ACL 2010 System Demonstrations","author":"Zhong Zhi","year":"2010"},{"key":"R87","doi-asserted-by":"publisher","DOI":"10.3115\/1613715.1613845"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/COLI_a_00142","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,1]],"date-time":"2024-05-01T12:21:19Z","timestamp":1714566079000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/39\/3\/511-554\/1437"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,9]]},"references-count":86,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2013,9]]}},"alternative-id":["10.1162\/COLI_a_00142"],"URL":"https:\/\/doi.org\/10.1162\/coli_a_00142","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,9]]}}}