{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,28]],"date-time":"2026-01-28T15:16:38Z","timestamp":1769613398073,"version":"3.49.0"},"reference-count":40,"publisher":"MIT Press - Journals","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2015,9]]},"abstract":"<jats:p> Agreement measures have been widely used in computational linguistics for more than 15 years to check the reliability of annotation processes. Although considerable effort has been made concerning categorization, fewer studies address unitizing, and when both paradigms are combined even fewer methods are available and discussed. The aim of this article is threefold. First, we advocate that to deal with unitizing, alignment and agreement measures should be considered as a unified process, because a relevant measure should rely on an alignment of the units from different annotators, and this alignment should be computed according to the principles of the measure. Second, we propose the new versatile measure \u03b3, which fulfills this requirement and copes with both paradigms, and we introduce its implementation. Third, we show that this new method performs as well as, or even better than, other more specialized methods devoted to categorization or segmentation, while combining the two paradigms at the same time. <\/jats:p>","DOI":"10.1162\/coli_a_00227","type":"journal-article","created":{"date-parts":[[2015,7,20]],"date-time":"2015-07-20T18:19:23Z","timestamp":1437416363000},"page":"437-479","source":"Crossref","is-referenced-by-count":44,"title":["The Unified and Holistic Method Gamma (\u03b3) for Inter-Annotator Agreement Measure and Alignment"],"prefix":"10.1162","volume":"41","author":[{"given":"Yann","family":"Mathet","sequence":"first","affiliation":[{"name":"Universit\u00e9 de Caen Basse-Normandie, GREYC, CNRS, UMR 6072"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Antoine","family":"Widl\u00f6cher","sequence":"additional","affiliation":[{"name":"Universit\u00e9 de Caen Basse-Normandie, GREYC, CNRS, UMR 6072"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jean-Philippe","family":"M\u00e9tivier","sequence":"additional","affiliation":[{"name":"Universit\u00e9 de Caen Basse-Normandie, GREYC, CNRS, UMR 6072"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","reference":[{"key":"R1","unstructured":"Afantenos, Stergos, Nicholas Asher, Farah Benamara, Myriam Bras, C\u00e9cile Fabre, Mai Ho-Dac, Anne Le Draoulec, Philippe Muller, Marie-Paule Pery-Woodley, Laurent Pr\u00e9vot, Josette Rebeyrolles, Ludovic Tanguy, Marianne Vergez-Couret, and Laure Vieu. 2012. An empirical resource for discovering cognitive principles of discourse organisation: The annodis corpus. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 2727\u20132734, Istanbul."},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.1162\/coli.07-034-R2"},{"key":"R3","unstructured":"Beeferman, Douglas, Adam Berger, and John Lafferty. 1997. Text segmentation using exponential models. In Proceedings of the 2nd Conference on Empirical Methods in Natural Language Processing, pages 35\u201346, Providence, RI."},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.1086\/266520"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.1001\/jama.268.18.2513"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1162\/coli.2006.32.1.5"},{"key":"R7","unstructured":"Bestgen, Yves. 2009. Quel indice pour mesurer l'efficacit\u00e9 en segmentation de textes ? In Actes de TALN 2009 (Traitement automatique des langues naturelles), Senlis."},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.1023\/A:1020499411651"},{"key":"R9","unstructured":"Bunt, Harry, Jan Alexandersson, Jean Carletta, Jae-Woong Choe, Alex C. Fang, Koiti Hasida, Kiyong Lee, Volha Petukhova, Andrei Popescu-Belis, Laurent Romary, Claudia Soria, and David Traum. 2010. Towards an ISO standard for dialogue act annotation. In Proceedings of the Seventh Conference on International Language Resources and Evaluation (LREC'10), pages 2548\u20132555, Valletta."},{"key":"R10","unstructured":"Carletta, J. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249\u2013254."},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-007-9040-x"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1017\/S0959269505002036"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.1037\/h0026256"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1162\/089120104773633402"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.3115\/1620754.1620806"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.1037\/h0031619"},{"key":"R18","unstructured":"Fort, Kar\u00ebn, Claire Fran\u00e7ois, Olivier Galibert, and Maha Ghribi. 2012. Analyzing the impact of prevalence on the evaluation of a manual annotation campaign. In Eighth International Conference on Language Resources and Evaluation (LREC 2012), pages 1474\u20131480, Istanbul."},{"key":"R19","unstructured":"Galibert, Olivier, Ludovic Quintard, Sophie Rosset, Pierre Zweigenbaum, Claire N\u00e9dellec, Sophie Aubin, Laurent Gillard, Jean-Pierre Raysz, Delphine Pois, Xavier Tannier, Louise Del\u00e9ger, and Dominique Laurent. 2010. Named and specific entity detection in varied data: The quaero named entity baseline evaluation. In Seventh International Conference on Language Resources and Evaluation (LREC 2010), pages 3453\u20133458, Valetta."},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.1001\/jama.268.18.2513"},{"key":"R23","unstructured":"Hearst, Marti. 1997. Texttiling: Segmenting text into multi-paragraph subtopic passages. Computational Linguistics, 23(1):33\u201364."},{"key":"R25","unstructured":"Kazantseva, Anna and Stan Szpakowicz. 2012. Topical segmentation: A study of human performance and a new measure of quality. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT '12, pages 211\u2013220, Stroudsburg, PA."},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.2307\/271061"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.1080\/19312458.2011.568376"},{"key":"R31","unstructured":"Kuper, Jan, Horacio Saggion, Hamish Cunningham, Thierry Declerck, Franciska de Jong, Dennis Reidsma, Yorick Wilks, and Peter Wittenburg. 2003. Intelligent multimedia indexing and retrieval through multi-source information extraction and merging. In IJCAI, pages 409\u2013414. Acapulco."},{"key":"R32","unstructured":"Labadi\u00e9, Alexandre, Patrice Enjalbert, Yann Mathet, and Antoine Widl\u00f6cher. 2010. Discourse structure annotation: Creating reference corpora. In Workshop on Language Resource and Language Technology Standards - State of the Art, Emerging Needs, and Future Developments, La Valetta."},{"key":"R34","unstructured":"Makhoul, John, Francis Kubala, Richard Schwartz, and Ralph Weischedel. 1999. Performance measures for information extraction. In Proceedings of DARPA Broadcast News Workshop, pages 249\u2013252, Herndon, VA."},{"key":"R35","unstructured":"Mathet, Yann and Antoine Widl\u00f6cher. 2011. Une approche holiste et unifi\u00e9e de l'alignement et de la mesure d'accord inter-annotateurs. In Traitement Automatique des Langues Naturelles 2011 (TALN 2011), Montpellier."},{"key":"R36","unstructured":"Mathet, Yann, Antoine Widl\u00f6cher, Kar\u00ebn Fort, Claire Francois, Olivier Galibert, Cyril Grouin, Juliette Kahn, Sophie Rosset, and Pierre Zweigenbaum. 2012. Manual corpus annotation: Giving meaning to the evaluation metrics. In COLING 2012, pages 809\u2013817, Mumbai."},{"key":"R37","doi-asserted-by":"publisher","DOI":"10.1075\/li.30.1.03nad"},{"key":"R39","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-012-9188-x"},{"key":"R40","doi-asserted-by":"publisher","DOI":"10.1162\/089120102317341756"},{"key":"R41","unstructured":"Reidsma, D. 2008. Annotations and Subjective Machines of Annotators, Embodied Agents, Users, and Other Humans. Ph.D. thesis, University of Twente."},{"key":"R43","doi-asserted-by":"publisher","DOI":"10.1162\/coli.2008.34.3.319"},{"key":"R44","doi-asserted-by":"publisher","DOI":"10.1086\/266577"},{"key":"R46","unstructured":"Teufel, Simone. 1999. Argumentative Zoning: Information Extraction from Scientific Articles. Ph.D. thesis, University of Edinburgh."},{"key":"R47","doi-asserted-by":"crossref","unstructured":"Teufel, Simone, Jean Carletta, and Marc Moens. 1999. An annotation scheme for discourse-level argumentation in research articles. In Proceedings of the 9th Conference of the European Chapter of the Association for Computational Linguistics, pages 110\u2013117, Bergen.","DOI":"10.3115\/977035.977051"},{"key":"R48","doi-asserted-by":"publisher","DOI":"10.1162\/089120102762671936"},{"key":"R49","doi-asserted-by":"crossref","unstructured":"Widl\u00f6cher, Antoine and Yann Mathet. 2012. The glozz platform: A corpus annotation and mining tool. In ACM Symposium on Document Engineering (DocEng'12), pages 171\u2013180, Paris.","DOI":"10.1145\/2361354.2361394"},{"key":"R50","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.103.3.374"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/COLI_a_00227","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:27:45Z","timestamp":1615584465000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/41\/3\/437-479\/1524"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,9]]},"references-count":40,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2015,9]]}},"alternative-id":["10.1162\/COLI_a_00227"],"URL":"https:\/\/doi.org\/10.1162\/coli_a_00227","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,9]]}}}