{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2022,4,1]],"date-time":"2022-04-01T22:09:36Z","timestamp":1648850976927},"reference-count":1,"publisher":"MIT Press - Journals","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2001,6]]},"abstract":"<jats:p> As evaluations of computational linguistics technology progress toward higher-level interpretation tasks, the problem of determining alignments between system responses and answer key entries may become less straightforward. We present an extensive analysis of the alignment procedure used in the MUC-6 evaluation of information extraction technology, which reveals effects that interfere with the stated goals of the evaluation. These effects are shown to be pervasive enough that they have the potential to adversely impact the technology development process. These results argue strongly for the use of accurate alignment criteria in natural language evaluations, and for maintaining the independence of alignment criteria and mechanisms used to calculate scores. <\/jats:p>","DOI":"10.1162\/089120101750300517","type":"journal-article","created":{"date-parts":[[2002,7,27]],"date-time":"2002-07-27T11:55:01Z","timestamp":1027770901000},"page":"231-248","source":"Crossref","is-referenced-by-count":0,"title":["The Need for Accurate Alignment in Natural Language System Evaluation"],"prefix":"10.1162","volume":"27","author":[{"given":"Andrew","family":"Kehler","sequence":"first","affiliation":[{"name":"UC San Diego, Department of Linguistics #0108, University of California, San Diego, 9500 Gilman Drive, La Jolla, CA 92093-0108."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"John","family":"Bear","sequence":"additional","affiliation":[{"name":"SRI International, Artificial Intelligence Center, 333 Ravenswood Avenue, Menlo Park, CA 94025."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Douglas","family":"Appelt","sequence":"additional","affiliation":[{"name":"SRI International, Artificial Intelligence Center, 333 Ravenswood Avenue, Menlo Park, CA 94025."}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","reference":[{"key":"p_6","doi-asserted-by":"publisher","DOI":"10.1006\/csla.1998.0102"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/089120101750300517","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:41:52Z","timestamp":1615585312000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/27\/2\/231-248\/1703"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2001,6]]},"references-count":1,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2001,6]]}},"alternative-id":["10.1162\/089120101750300517"],"URL":"https:\/\/doi.org\/10.1162\/089120101750300517","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2001,6]]}}}