{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T15:30:49Z","timestamp":1781191849639,"version":"3.54.1"},"reference-count":32,"publisher":"Cambridge University Press (CUP)","issue":"4","license":[{"start":{"date-parts":[[2010,12,6]],"date-time":"2010-12-06T00:00:00Z","timestamp":1291593600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2011,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This paper addresses the current state of coreference resolution evaluation, in which different measures (notably, MUC, B<jats:sup>3<\/jats:sup>, CEAF, and ACE-value) are applied in different studies. None of them is fully adequate, and their measures are not commensurate. We enumerate the desiderata for a coreference scoring measure, discuss the strong and weak points of the existing measures, and propose the BiLateral Assessment of Noun-Phrase Coreference, a variation of the Rand index created to suit the coreference task. The BiLateral Assessment of Noun-Phrase Coreference rewards both coreference and non-coreference links by averaging the F-scores of the two types, does not ignore singletons \u2013 the main problem with the MUC score \u2013 and does not inflate the score in their presence \u2013 a problem with the B<jats:sup>3<\/jats:sup> and CEAF scores. In addition, its fine granularity is consistent over the whole range of scores and affords better discrimination between systems.<\/jats:p>","DOI":"10.1017\/s135132491000029x","type":"journal-article","created":{"date-parts":[[2010,12,6]],"date-time":"2010-12-06T05:58:26Z","timestamp":1291615106000},"page":"485-510","source":"Crossref","is-referenced-by-count":78,"title":["BLANC: Implementing the Rand index for coreference evaluation"],"prefix":"10.1017","volume":"17","author":[{"given":"M.","family":"RECASENS","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"E.","family":"HOVY","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"56","published-online":{"date-parts":[[2010,12,6]]},"reference":[{"key":"S135132491000029X_ref20","doi-asserted-by":"publisher","DOI":"10.3115\/1613715.1613796"},{"key":"S135132491000029X_ref32","first-page":"843","volume-title":"Proceedings of ACL-HLT","author":"Yang","year":"2008"},{"key":"S135132491000029X_ref13","volume-title":"Proceedings of MUC-7","author":"Hirschman","year":"1997"},{"key":"S135132491000029X_ref1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-008-9066-8"},{"key":"S135132491000029X_ref29","doi-asserted-by":"publisher","DOI":"10.3115\/1690219.1690238"},{"key":"S135132491000029X_ref14","doi-asserted-by":"publisher","DOI":"10.1007\/BF01908075"},{"key":"S135132491000029X_ref6","first-page":"81","volume-title":"Proceedings of HLT-NAACL","author":"Culotta","year":"2007"},{"key":"S135132491000029X_ref24","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1971.10482356"},{"key":"S135132491000029X_ref22","first-page":"1507","volume-title":"Proceedings of LREC","author":"Popescu-Belis","year":"2004"},{"key":"S135132491000029X_ref30","first-page":"45","volume-title":"Proceedings of MUC-6","author":"Vilain","year":"1995"},{"key":"S135132491000029X_ref2","first-page":"563","volume-title":"Proceedings of the LREC 1998 Workshop on Linguistic Coreference","author":"Bagga","year":"1998"},{"key":"S135132491000029X_ref16","first-page":"21","volume-title":"Proceedings of ACL","author":"Luo","year":"2004"},{"key":"S135132491000029X_ref7","first-page":"97","volume-title":"Proceedings of HLT-EMNLP","author":"Daum\u00e9","year":"2005"},{"key":"S135132491000029X_ref3","doi-asserted-by":"publisher","DOI":"10.3115\/1613715.1613756"},{"key":"S135132491000029X_ref4","first-page":"28","volume-title":"Proceedings of SIGDIAL","author":"Cai","year":"2010"},{"key":"S135132491000029X_ref5","first-page":"65","volume-title":"Proceedings of HLT-NAACL","author":"Choi","year":"2007"},{"key":"S135132491000029X_ref8","first-page":"87","article-title":"Global joint models for coreference resolution and named entity classification","volume":"42","author":"Denis","year":"2009","journal-title":"Procesamiento del Lenguaje Natural"},{"key":"S135132491000029X_ref9","first-page":"837","volume-title":"Proceedings of LREC","author":"Doddington","year":"2004"},{"key":"S135132491000029X_ref10","first-page":"45","volume-title":"Proceedings of ACL-HLT","author":"Finkel","year":"2008"},{"key":"S135132491000029X_ref11","first-page":"848","volume-title":"Proceedings of ACL","author":"Haghighi","year":"2007"},{"key":"S135132491000029X_ref15","first-page":"25","volume-title":"Proceedings of HLT-EMNLP","author":"Luo","year":"2005"},{"key":"S135132491000029X_ref17","first-page":"660","volume-title":"Proceedings of HLT-EMNLP","author":"Luo","year":"2005"},{"key":"S135132491000029X_ref18","first-page":"575","volume-title":"Proceedings of NAACL-HLT","author":"Ng","year":"2009"},{"key":"S135132491000029X_ref19","first-page":"104","volume-title":"Proceedings of ACL","author":"Ng","year":"2002"},{"key":"S135132491000029X_ref21","first-page":"117","article-title":"\u00c9valuation num\u00e9rique de la r\u00e9solution de la r\u00e9f\u00e9rence: critiques et propositions","volume":"40","author":"Popescu-Belis","year":"2000","journal-title":"T.A.L.: Traitement automatique de la langue"},{"key":"S135132491000029X_ref23","doi-asserted-by":"publisher","DOI":"10.3115\/1699571.1699639"},{"key":"S135132491000029X_ref25","first-page":"1423","volume-title":"Proceedings of ACL","author":"Recasens","year":"2010"},{"key":"S135132491000029X_ref26","first-page":"1","volume-title":"Proceedings of the Fifth International Workshop on Semantic Evaluation (SemEval 2010)","author":"Recasens","year":"2010"},{"key":"S135132491000029X_ref27","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-009-9108-x"},{"key":"S135132491000029X_ref28","doi-asserted-by":"publisher","DOI":"10.1162\/089120101753342653"},{"key":"S135132491000029X_ref31","doi-asserted-by":"crossref","unstructured":"Wick M. , and McCallum A. 2009. Advances in learning and inference for partition-wise models of coreference resolution. Technical Report UM-CS-2009-028, Department of Computer Science, University of Massachusetts.","DOI":"10.1137\/1.9781611972795.32"},{"key":"S135132491000029X_ref12","doi-asserted-by":"publisher","DOI":"10.3115\/1699648.1699661"}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S135132491000029X","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,4,27]],"date-time":"2019-04-27T00:36:29Z","timestamp":1556325389000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S135132491000029X\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2010,12,6]]},"references-count":32,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2011,10]]}},"alternative-id":["S135132491000029X"],"URL":"https:\/\/doi.org\/10.1017\/s135132491000029x","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2010,12,6]]}}}