{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T15:13:08Z","timestamp":1714403588739},"reference-count":27,"publisher":"MIT Press - Journals","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2014,9]]},"abstract":"<jats:p> Information-theoretic measures are among the most standard techniques for evaluation of clustering methods including word sense induction (WSI) systems. Such measures rely on sample-based estimates of the entropy. However, the standard maximum likelihood estimates of the entropy are heavily biased with the bias dependent on, among other things, the number of clusters and the sample size. This makes the measures unreliable and unfair when the number of clusters produced by different systems vary and the sample size is not exceedingly large. This corresponds exactly to the setting of WSI evaluation where a ground-truth cluster sense number arguably does not exist and the standard evaluation scenarios use a small number of instances of each word to compute the score. We describe more accurate entropy estimators and analyze their performance both in simulations and on evaluation of WSI systems. <\/jats:p>","DOI":"10.1162\/coli_a_00196","type":"journal-article","created":{"date-parts":[[2014,3,28]],"date-time":"2014-03-28T23:10:37Z","timestamp":1396048237000},"page":"671-685","source":"Crossref","is-referenced-by-count":1,"title":["Improved Estimation of Entropy for Evaluation of Word Sense Induction"],"prefix":"10.1162","volume":"40","author":[{"given":"Linlin","family":"Li","sequence":"first","affiliation":[{"name":"Microsoft Development Center Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ivan","family":"Titov","sequence":"additional","affiliation":[{"name":"University of Amsterdam"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Caroline","family":"Sporleder","sequence":"additional","affiliation":[{"name":"Trier University"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.3115\/1654758.1654776"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.3115\/1621474.1621476"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-008-9066-8"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.1002\/rsa.10019"},{"key":"R5","first-page":"563","volume-title":"The First International Conference on Language Resources and Evaluation Workshop on Linguistics Coreference","author":"Bagga Amit","year":"1998"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1109\/CCC.2002.1004329"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.1063\/1.166191"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-30120-2_14"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-012-9205-0"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.3115\/1220575.1220579"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.3115\/1621969.1621990"},{"key":"R13","first-page":"63","volume-title":"Proceedings of the 5th InternationalWorkshop on Semantic Evaluation","author":"Manandhar Suresh","year":"2010"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.1016\/j.jmva.2006.11.013"},{"key":"R15","first-page":"95","author":"Miller G.","year":"1955","journal-title":"Information Theory in Psychology II-B"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.1162\/089976603321780272"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.2004.833360"},{"key":"R18","first-page":"41","volume-title":"Proceedings of the CoNLL","author":"Purandare Amruta","year":"2004"},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/43.3-4.353"},{"key":"R20","first-page":"410","volume-title":"Proceedings of the 2007 EMNLP-CoNll Joint Conference","author":"Rosenberg Andrew","year":"2007"},{"issue":"1","key":"R21","first-page":"97","volume":"24","author":"Sch\u00fctze Hinrich","year":"1998","journal-title":"Computational Linguistics"},{"key":"R22","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176349952"},{"key":"R23","first-page":"583","volume":"3","author":"Strehl Alexander","year":"2002","journal-title":"Journal of Machine Learning Research"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.80.197"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177706647"},{"key":"R26","first-page":"295","volume":"2","author":"Wang Y. H.","year":"1993","journal-title":"Statistica Sinica 3"},{"key":"R27","volume-title":"Data Mining: Practical Machine Learning Tools and Techniques.","author":"Witte Ian","year":"2005"},{"key":"R28","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-005-0361-3"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/COLI_a_00196","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:27:37Z","timestamp":1615584457000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/40\/3\/671-685\/1477"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,9]]},"references-count":27,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2014,9]]}},"alternative-id":["10.1162\/COLI_a_00196"],"URL":"https:\/\/doi.org\/10.1162\/coli_a_00196","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2014,9]]}}}