{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,19]],"date-time":"2026-06-19T17:35:55Z","timestamp":1781890555861,"version":"3.54.5"},"reference-count":19,"publisher":"MIT Press","issue":"3","license":[{"start":{"date-parts":[[2024,3,4]],"date-time":"2024-03-04T00:00:00Z","timestamp":1709510400000},"content-version":"vor","delay-in-days":428,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>We propose a novel method for calculating PARSEVAL measures to evaluate constituent parsing results. Previous constituent parsing evaluation techniques were constrained by the requirement for consistent sentence boundaries and tokenization results, proving to be stringent and inconvenient. Our new approach handles constituent parsing results obtained from raw text, even when sentence boundaries and tokenization differ from the preprocessed gold sentence. Implementing this measure is our evaluation by alignment approach. The algorithm enables the alignment of tokens and sentences in the gold and system parse trees. Our proposed algorithm draws on the analogy of sentence and word alignment commonly used in machine translation (MT). To demonstrate the intricacy of calculations and clarify any integration of configurations, we explain the implementations in detailed pseudo-code and provide empirical proof for how sentence and word alignment can improve evaluation reliability.<\/jats:p>","DOI":"10.1162\/coli_a_00512","type":"journal-article","created":{"date-parts":[[2024,3,4]],"date-time":"2024-03-04T19:05:40Z","timestamp":1709579140000},"page":"1181-1190","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":4,"title":["A Novel Alignment-based Approach for PARSEVAL Measuress"],"prefix":"10.1162","volume":"50","author":[{"given":"Eunkyul Leah","family":"Jo","sequence":"first","affiliation":[{"name":"The University of British Columbia, Department of Computer Science. eunkyul@student.ubc.ca"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Angela Yoonseo","family":"Park","sequence":"additional","affiliation":[{"name":"The University of British Columbia, Department of Linguistics. apark03@student.ubc.ca"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jungyeul","family":"Park","sequence":"additional","affiliation":[{"name":"The University of British Columbia, Department of Linguistics. jungyeul@mail.ubc.ca"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2023,9,1]]},"reference":[{"issue":"1","key":"2024092014245088200_bib1","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1016\/j.tcs.2004.12.030","article-title":"A survey on tree edit distance and related problems","volume":"337","author":"Bille","year":"2005","journal-title":"Theoretical Computer Science"},{"key":"2024092014245088200_bib2","first-page":"306","article-title":"A procedure for quantitatively comparing the syntactic coverage of English grammars","volume-title":"Speech and Natural Language: Proceedings","author":"Black","year":"1991"},{"key":"2024092014245088200_bib3","first-page":"75","article-title":"On aligning trees","volume-title":"Second Conference on Empirical Methods in Natural Language Processing","author":"Calder","year":"1997"},{"key":"2024092014245088200_bib4","unstructured":"Collins, Michael\n          . 1999. Head-Driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania."},{"key":"2024092014245088200_bib5","doi-asserted-by":"publisher","first-page":"4092","DOI":"10.18653\/v1\/2020.acl-main.376","article-title":"Enriched in-order linearization for faster sequence-to-sequence constituent parsing","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Fern\u00e1ndez-Gonz\u00e1lez","year":"2020"},{"key":"2024092014245088200_bib6","doi-asserted-by":"publisher","first-page":"1712","DOI":"10.18653\/v1\/D17-1180","article-title":"TAG parsing with neural networks and vector representations of supertags","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","author":"Kasai","year":"2017"},{"key":"2024092014245088200_bib7","doi-asserted-by":"publisher","first-page":"1181","DOI":"10.18653\/v1\/N18-1107","article-title":"End-to-end graph-based TAG parsing with neural networks","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Kasai","year":"2018"},{"key":"2024092014245088200_bib8","first-page":"1048","article-title":"Parser showdown at the Wall Street corral: An empirical investigation of error types in parser output","volume-title":"Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning","author":"Kummerfeld","year":"2012"},{"key":"2024092014245088200_bib9","doi-asserted-by":"publisher","first-page":"2366","DOI":"10.18653\/v1\/D16-1262","article-title":"Global neural CCG parsing with optimality guarantees","volume-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","author":"Lee","year":"2016"},{"key":"2024092014245088200_bib10","doi-asserted-by":"publisher","first-page":"221","DOI":"10.18653\/v1\/N16-1026","article-title":"LSTM CCG parsing","volume-title":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Lewis","year":"2016"},{"key":"2024092014245088200_bib11","first-page":"4034","article-title":"Universal Dependencies v2: An evergrowing multilingual treebank collection","volume-title":"Proceedings of the 12th Language Resources and Evaluation Conference","author":"Nivre","year":"2020"},{"key":"2024092014245088200_bib12","first-page":"333","article-title":"SParseval: Evaluation metrics for parsing speech","volume-title":"Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC\u201906)","author":"Roark","year":"2006"},{"key":"2024092014245088200_bib13","first-page":"103","article-title":"Introducing the SPMRL 2014 shared task on parsing morphologically-rich languages","volume-title":"Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of Non-Canonical Languages","author":"Seddah","year":"2014"},{"key":"2024092014245088200_bib14","first-page":"146","article-title":"Overview of the SPMRL 2013 shared task: A cross-framework evaluation of parsing morphologically rich languages","volume-title":"Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages","author":"Seddah","year":"2013"},{"key":"2024092014245088200_bib15","doi-asserted-by":"publisher","first-page":"4111","DOI":"10.18653\/v1\/2020.acl-main.378","article-title":"Max-margin incremental CCG parsing","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Stanojevi\u0107","year":"2020"},{"key":"2024092014245088200_bib16","first-page":"6","article-title":"Joint evaluation of morphological segmentation and syntactic parsing","volume-title":"Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Tsarfaty","year":"2012"},{"key":"2024092014245088200_bib17","first-page":"2773","article-title":"Grammar as a foreign language","volume-title":"Advances in Neural Information Processing Systems 28","author":"Vinyals","year":"2015"},{"key":"2024092014245088200_bib18","doi-asserted-by":"publisher","first-page":"3267","DOI":"10.18653\/v1\/2020.acl-main.299","article-title":"A span-based linearization for constituent trees","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Wei","year":"2020"},{"key":"2024092014245088200_bib19","doi-asserted-by":"publisher","first-page":"262","DOI":"10.18653\/v1\/2023.acl-long.15","article-title":"Holographic CCG parsing","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Yamaki","year":"2023"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/50\/3\/1181\/2470892\/coli_a_00512.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/50\/3\/1181\/2470892\/coli_a_00512.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,20]],"date-time":"2024-09-20T14:25:04Z","timestamp":1726842304000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/50\/3\/1181\/119825\/A-Novel-Alignment-based-Approach-for-PARSEVAL"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":19,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2023,9,1]]},"published-print":{"date-parts":[[2023,9,1]]}},"URL":"https:\/\/doi.org\/10.1162\/coli_a_00512","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}