{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,28]],"date-time":"2025-09-28T04:16:45Z","timestamp":1759033005218},"reference-count":5,"publisher":"MIT Press - Journals","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2005,9]]},"abstract":"<jats:p> Agreement statistics play an important role in the evaluation of coding schemes for discourse and dialogue. Unfortunately there is a lack of understanding regarding appropriate agreement measures and how their results should be interpreted. In this article we describe the role of agreement measures and argue that only chance-corrected measures that assume a common distribution of labels for all coders are suitable for measuring agreement in reliability studies. We then provide recommendations for how reliability should be inferred from the results of agreement statistics. <\/jats:p>","DOI":"10.1162\/089120105774321109","type":"journal-article","created":{"date-parts":[[2005,9,28]],"date-time":"2005-09-28T20:53:50Z","timestamp":1127940830000},"page":"289-296","source":"Crossref","is-referenced-by-count":32,"title":["Evaluating Discourse and Dialogue Coding Schemes"],"prefix":"10.1162","volume":"31","author":[{"given":"Richard","family":"Craggs","sequence":"first","affiliation":[{"name":"University of Manchester"}]},{"given":"Mary McGee","family":"Wood","sequence":"additional","affiliation":[{"name":"University of Manchester"}]}],"member":"281","reference":[{"issue":"2","key":"p_1","first-page":"249","volume":"22","author":"Carletta Jean","year":"1996","journal-title":"Computational Linguistics"},{"key":"p_2","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"p_3","doi-asserted-by":"publisher","DOI":"10.1162\/089120104773633402"},{"issue":"3","key":"p_6","first-page":"411","volume":"30","author":"Krippendorff Klaus","year":"2004","journal-title":"Human Communication Research"},{"key":"p_8","doi-asserted-by":"publisher","DOI":"10.1086\/266577"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/089120105774321109","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:42:28Z","timestamp":1615585348000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/31\/3\/289-296\/1882"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2005,9]]},"references-count":5,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2005,9]]}},"alternative-id":["10.1162\/089120105774321109"],"URL":"https:\/\/doi.org\/10.1162\/089120105774321109","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2005,9]]}}}