{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T08:23:29Z","timestamp":1768983809324,"version":"3.49.0"},"reference-count":51,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2013,1,1]],"date-time":"2013-01-01T00:00:00Z","timestamp":1356998400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100004965","name":"Sixth Framework Programme","doi-asserted-by":"publisher","award":["IST-FP6-026978"],"award-info":[{"award-number":["IST-FP6-026978"]}],"id":[{"id":"10.13039\/501100004965","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Speech Lang. Process."],"published-print":{"date-parts":[[2013,1]]},"abstract":"<jats:p>With the growing interest in statistical parsing, special attention has recently been devoted to the problem of comparing different treebanks to assess which languages or domains are more difficult to parse relative to a given model. A common methodology for comparing parsing difficulty across treebanks is based on the use of the standard labeled precision and recall measures. As an alternative, in this article we propose an information-theoretic measure, called the expected conditional cross-entropy (ECC). One important advantage with respect to standard performance measures is that ECC can be directly expressed as a function of the parameters of the model. We evaluate ECC across several treebanks for English, French, German, and Italian, and show that ECC is an effective measure of parsing difficulty, with an increase in ECC always accompanied by a degradation in parsing accuracy.<\/jats:p>","DOI":"10.1145\/2407736.2407737","type":"journal-article","created":{"date-parts":[[2013,1,29]],"date-time":"2013-01-29T16:20:55Z","timestamp":1359476455000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["An information-theoretic measure to evaluate parsing difficulty across treebanks"],"prefix":"10.1145","volume":"9","author":[{"given":"Anna","family":"Corazza","sequence":"first","affiliation":[{"name":"Universit\u00e0 di Napoli \u201cFederico II\u201d, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alberto","family":"Lavelli","sequence":"additional","affiliation":[{"name":"FBK-irst, Trento, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Giorgio","family":"Satta","sequence":"additional","affiliation":[{"name":"Universit\u00e0 di Padova, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2013,1,30]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the 2nd International Conference on Language Resources and Evaluation (LREC'00)","author":"Abeill\u00e9 A."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.3115\/1075671.1075725"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.3115\/1219840.1219878"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.5555\/234285.234289"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1162\/0891201042544929"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.3115\/1117769.1117771"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'08)","author":"Birch A."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.3115\/112405.112467"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1214\/ss\/1009213286"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.5555\/146680.146685"},{"key":"e_1_2_1_12_1","volume-title":"Statistical Language Learning","author":"Charniak E."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 13th National Conference on Artificial Intelligence (AAAI-96)","author":"Charniak E.","year":"1996"},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the Ist NAACL. Association for Computational Linguistics.","author":"Charniak E.","year":"2000"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.3115\/1073012.1073029"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/TC.1983.1676313"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.5555\/972732.972738"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1162\/089120103322753356"},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","unstructured":"Cover T.M. and Thomas J.A. 1991. Elements of Information Theory. Wiley Hoboken NJ.   Cover T.M. and Thomas J.A. 1991. Elements of Information Theory. Wiley Hoboken NJ.","DOI":"10.1002\/0471200611"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.588021"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.3115\/992628.992688"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'01)","author":"Gildea D.","year":"2001"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog0000_64"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.3115\/1218955.1218968"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1162\/0891201041850894"},{"key":"e_1_2_1_26_1","volume-title":"Statistical Methods for Speech Recognition","author":"Jelinek F."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.5555\/972764.972768"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.3115\/1073012.1073054"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.3115\/1034678.1034758"},{"key":"e_1_2_1_30_1","unstructured":"Jurafsky D. and Martin J. 2000. Speech and Language Processing. Prentice-Hall Upper Saddle River NJ.   Jurafsky D. and Martin J. 2000. Speech and Language Processing. Prentice-Hall Upper Saddle River NJ."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.3115\/1118693.1118695"},{"key":"e_1_2_1_32_1","unstructured":"Klein D. and Manning C. 2002b. Fast exact inference with a factored model for natural language parsing. In Advances in Neural Information Processing Systems 15 (NIPS'02). MIT Press Cambridge MA.  Klein D. and Manning C. 2002b. Fast exact inference with a factored model for natural language parsing. In Advances in Neural Information Processing Systems 15 (NIPS'02). MIT Press Cambridge MA."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.3115\/1075096.1075150"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the 12th Machine Translation Summit. Association for Machine Translation in the Americas, 65--72","author":"Koehn P."},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of RANL. John Benjamins Borovets.","author":"Kubler S.","year":"2005"},{"key":"e_1_2_1_36_1","doi-asserted-by":"crossref","unstructured":"Kubler S. Hinrichs E. and Maier W. 2006. Is it really that difficult to parse German&quest; In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'06). Association for Computational Linguistics 111--119.   Kubler S. Hinrichs E. and Maier W. 2006. Is it really that difficult to parse German&quest; In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'06). Association for Computational Linguistics 111--119.","DOI":"10.3115\/1610075.1610093"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.5555\/1557856.1557861"},{"key":"e_1_2_1_38_1","unstructured":"Manning C. and Schutze H. 1999. Foundations of Statistical Natural Language Processing. MIT Press Cambridge MA.   Manning C. and Schutze H. 1999. Foundations of Statistical Natural Language Processing. MIT Press Cambridge MA."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.5555\/972470.972475"},{"key":"e_1_2_1_40_1","doi-asserted-by":"crossref","unstructured":"Montemagni S. Barsotti F. Battista M. Calzolari N. Corazzari O. Lenci A. Zampolli A. Fanciulli F. Massetani M. Raffaelli R. Basili R. Pazienza M.T. Saracino D. Zanzotto F. Mana N. Pianesi F. and Delmonte R. 2003. Building the Italian syntactic-semantics treebank. In Building and Using Syntactically Annotated Corpora A. Abeille Ed. Kluwer Dordrecht 189--210.  Montemagni S. Barsotti F. Battista M. Calzolari N. Corazzari O. Lenci A. Zampolli A. Fanciulli F. Massetani M. Raffaelli R. Basili R. Pazienza M.T. Saracino D. Zanzotto F. Mana N. Pianesi F. and Delmonte R. 2003. Building the Italian syntactic-semantics treebank. In Building and Using Syntactically Annotated Corpora A. Abeille Ed. Kluwer Dordrecht 189--210.","DOI":"10.1007\/978-94-010-0201-1_11"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the LREC Workshop Beyond PARSEVAL. Towards Improved Evaluation Measures for Parsing Systems. European Language Resources Association (ELRA).","author":"Musillo G."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.3115\/1220175.1220230"},{"key":"e_1_2_1_43_1","unstructured":"Petrov S. and Klein D. 2007. Improved inference for unlexicalized parsing. In Proceedings of the Main Conference on Human Language Technologies: The Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics 404--411.  Petrov S. and Klein D. 2007. Improved inference for unlexicalized parsing. In Proceedings of the Main Conference on Human Language Technologies: The Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics 404--411."},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL). Association for Computational Linguistics, 630--639","author":"Rehbein I."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.3115\/992133.992135"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.3115\/992133.992136"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1951.tb01366.x"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.3115\/974557.974571"},{"key":"e_1_2_1_49_1","volume-title":"Linguistic Structure Prediction. Synthesis Lectures on Human Language Technologies. Morgan and Claypool","author":"Smith N.A."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0019-9958(74)90799-2"},{"key":"e_1_2_1_51_1","unstructured":"Telljohann H. Hinrichs E. Kubler S. and Zinsmeister H. 2006. Stylebook for the Tubingen treebank of written German (TuBa-D\/Z). Tech. rep. Universitat Tubingen Seminar fur Sprachwissenschaft July.  Telljohann H. Hinrichs E. Kubler S. and Zinsmeister H. 2006. Stylebook for the Tubingen treebank of written German (TuBa-D\/Z). Tech. rep. Universitat Tubingen Seminar fur Sprachwissenschaft July."},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the 2nd Workshop on Treebanks and Linguistic Theories (TLT'03)","author":"Ule T.","year":"2003"}],"container-title":["ACM Transactions on Speech and Language Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2407736.2407737","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2407736.2407737","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T08:18:28Z","timestamp":1750234708000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2407736.2407737"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,1]]},"references-count":51,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2013,1]]}},"alternative-id":["10.1145\/2407736.2407737"],"URL":"https:\/\/doi.org\/10.1145\/2407736.2407737","relation":{},"ISSN":["1550-4875","1550-4883"],"issn-type":[{"value":"1550-4875","type":"print"},{"value":"1550-4883","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,1]]},"assertion":[{"value":"2011-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2012-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2013-01-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}