{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T13:13:04Z","timestamp":1740143584758,"version":"3.37.3"},"reference-count":35,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2022,5,17]],"date-time":"2022-05-17T00:00:00Z","timestamp":1652745600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,5,17]],"date-time":"2022-05-17T00:00:00Z","timestamp":1652745600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100005366","name":"University of Oslo","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005366","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Lang Resources &amp; Evaluation"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We discuss methodological choices in diagnostic evaluation and error analysis in meaning representation parsing (MRP), i.e. mapping from natural language utterances to graph-based encodings of semantic structure. We expand on a pilot quantitative study in contrastive diagnostic evaluation, inspired by earlier work in syntactic dependency parsing, and propose a novel methodology for qualitative error analysis. This two-pronged study is performed using a selection of submissions, data, and evaluation tools featured in the 2019 shared task on MRP. Our aim is to devise methods for identifying strengths and weaknesses in different broad families of parsing techniques, as well as investigating the relations between specific parsing approaches, different meaning representation frameworks, and individual linguistic phenomena\u2014by identifying and comparing common error patterns. Our preliminary empirical results suggest that the proposed methodologies can be meaningfully applied to parsing into graph-structured target representations, as a side-effect uncovering hitherto unknown properties of the different systems that can inform future development and cross-fertilization across approaches.<\/jats:p>","DOI":"10.1007\/s10579-022-09591-7","type":"journal-article","created":{"date-parts":[[2022,5,17]],"date-time":"2022-05-17T17:17:53Z","timestamp":1652807873000},"page":"1075-1102","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["A tale of four parsers: methodological reflections on diagnostic evaluation and in-depth error analysis for meaning representation parsing"],"prefix":"10.1007","volume":"56","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2033-1408","authenticated-orcid":false,"given":"Maja","family":"Buljan","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Joakim","family":"Nivre","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stephan","family":"Oepen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lilja","family":"\u00d8vrelid","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,5,17]]},"reference":[{"key":"9591_CR1","unstructured":"Banarescu, L., Bonial, C., Cai, S., Georgescu, M., Griffitt, K., Hermjakob, U., Knight, K., Koehn, P., Palmer, M., & Schneider, N. (2013). Abstract Meaning Representation for sembanking. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, Sofia (pp. 178\u2013186). http:\/\/www.aclweb.org\/anthology\/W13-2322"},{"key":"9591_CR2","unstructured":"Bender, E. M., Flickinger, D., Oepen, S., Packard, W., & Copestake, A. (2015). Layers of interpretation: On grammar and compositionality. In Proceedings of the 11th international conference on computational semantics (pp. 239\u2013249)."},{"key":"9591_CR3","doi-asserted-by":"crossref","unstructured":"Buchholz, S., & Marsi, E. (2006). CoNLL-X shared task on multilingual dependency parsing. In Proceedings of the 10th conference on natural language learning, New York, NY (pp. 149\u2013164). http:\/\/www.aclweb.org\/anthology\/W\/W06\/W06-2920","DOI":"10.3115\/1596276.1596305"},{"key":"9591_CR4","unstructured":"Buljan, M., Nivre, J., Oepen, S., & \u00d8vrelid, L. (2020). A tale of three parsers: Towards diagnostic evaluation for meaning representation parsing. In Proceedings of the 12th language resources and evaluation conference (pp. 1902\u20131909)."},{"key":"9591_CR5","unstructured":"Cai, S., & Knight, K. (2013). Smatch: An evaluation metric for semantic feature structures. In Proceedings of the 51th meeting of the Association for Computational Linguistics, Sofia (pp. 748\u2013752). http:\/\/www.aclweb.org\/anthology\/P13-2131"},{"key":"9591_CR6","doi-asserted-by":"crossref","unstructured":"Cao, J., Zhang, Y., Youssef, A., & Srikumar, V. (2019). Amazon at MRP\u00a02019: Parsing meaning representations with lexical and phrasal anchoring. In Proceedings of the shared task on cross-framework meaning representation parsing at the 2019 conference on computational natural language learning, Hong Kong (pp. 138\u2013148).","DOI":"10.18653\/v1\/K19-2013"},{"key":"9591_CR7","doi-asserted-by":"crossref","unstructured":"Che, W., Dou, L., Xu, Y., Wang, Y., Liu, Y., & Liu, T. (2019). HIT-SCIR at MRP\u00a02019: A unified pipeline for meaning representation parsing via efficient training and effective encoding. In Proceedings of the shared task on cross-framework meaning representation parsing at the 2019 conference on computational natural language learning, Hong Kong (pp. 76\u201385).","DOI":"10.18653\/v1\/K19-2007"},{"key":"9591_CR8","doi-asserted-by":"crossref","unstructured":"Damonte, M., Cohen, S. B., & Satta, G. (2017). An incremental parser for Abstract Meaning Representation. In Proceedings of the 15th conference of the European Chapter of the Association for Computational Linguistics: Volume 1, long papers, Association for Computational linguistics, Valencia (pp. 536\u2013546). https:\/\/aclanthology.org\/E17-1051","DOI":"10.18653\/v1\/E17-1051"},{"key":"9591_CR9","unstructured":"Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN"},{"key":"9591_CR10","doi-asserted-by":"crossref","unstructured":"Donatelli, L., Fowlie, M., Groschwitz, J., Koller, A., Lindemann, M., Mina, M., & Wei\u00dfenhorn, P. (2019). Saarland at MRP\u00a02019: Compositional parsing across all graphbanks. In Proceedings of the shared task on cross-framework meaning representation parsing at the 2019 conference on computational natural language learning, Hong Kong (pp. 66\u201375).","DOI":"10.18653\/v1\/K19-2006"},{"key":"9591_CR11","unstructured":"Dozat, T., & Manning, C. D. (2016). Deep biaffine attention for neural dependency parsing. arXiv preprint http:\/\/arxiv.org\/abs\/161101734"},{"key":"9591_CR12","unstructured":"Dridan, R., & Oepen, S. (2011). Parser evaluation using elementary dependency matching. In Proceedings of the 12th international conference on parsing technologies, Dublin (pp. 225\u2013230)."},{"key":"9591_CR13","doi-asserted-by":"crossref","unstructured":"Flanigan, J., Thomson, S., Carbonell, J., Dyer, C., & Smith, N. A. (2014). A discriminative graph-based parser for the Abstract Meaning Representation. In Proceedings of the 52nd meeting of the Association for Computational Linguistics, Baltimore, MD (pp. 1426\u20131436). http:\/\/www.aclweb.org\/anthology\/P14-1134","DOI":"10.3115\/v1\/P14-1134"},{"key":"9591_CR14","unstructured":"Groschwitz, J., Fowlie, M., Johnson, M., & Koller, A. (2017). A constrained graph algebra for semantic parsing with AMRs. In Proceedings of the 12th international conference on computational semantics, Montpellier."},{"key":"9591_CR15","unstructured":"Haji\u010d, J., Haji\u010dov\u00e1, E., Panevov\u00e1, J., Sgall, P., Bojar, O., Cinkov\u00e1, S., Fu\u010d\u00edkov\u00e1, E., Mikulov\u00e1, M., Pajas, P., Popelka, J., Semeck\u00fd, J., \u0160indlerov\u00e1, J., \u0160t\u011bp\u00e1nek, J., Toman, J., Ure\u0161ov\u00e1, Z., & \u017dabokrtsk\u00fd, Z. (2012). Announcing Prague Czech-English Dependency Treebank 2.0. In Proceedings of the 8th international conference on language resources and evaluation, Istanbul (pp. 3153\u20133160). http:\/\/www.lrec-conf.org\/proceedings\/lrec2012\/pdf\/510_Paper.pdf"},{"key":"9591_CR16","unstructured":"Ivanova, A., Oepen, S., \u00d8vrelid, L., & Flickinger, D. (2012). Who did what to whom? A contrastive study of syntacto-semantic dependencies. In Proceedings of the 6th linguistic annotation workshop, Jeju (pp. 2\u201311)."},{"key":"9591_CR17","unstructured":"Kate, R. J., & Wong, Y. W. (2010). Semantic parsing. The task, the state of the art and the future. In Tutorial abstracts of the 20th meeting of the Association for Computational Linguistics, Uppsala (p.\u00a06). http:\/\/www.aclweb.org\/anthology\/P10-5006"},{"key":"9591_CR18","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1162\/tacl_a_00101","volume":"4","author":"E Kiperwasser","year":"2016","unstructured":"Kiperwasser, E., & Goldberg, Y. (2016). Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics, 4, 313\u2013327.","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"9591_CR19","doi-asserted-by":"publisher","unstructured":"Koller, A., Oepen, S., & Sun, W. (2019). Graph-based meaning representations. Design and processing. In Proceedings of the 57th meeting of the Association for Computational Linguistics: Tutorial Abstracts, Florence (pp. 6\u201311). https:\/\/doi.org\/10.18653\/v1\/P19-4002","DOI":"10.18653\/v1\/P19-4002"},{"issue":"4","key":"9591_CR20","doi-asserted-by":"publisher","first-page":"819","DOI":"10.1162\/COLI_a_00268","volume":"42","author":"M Kuhlmann","year":"2016","unstructured":"Kuhlmann, M., & Oepen, S. (2016). Towards a catalogue of linguistic graph banks. Computational Linguistics, 42(4), 819\u2013827.","journal-title":"Computational Linguistics"},{"key":"9591_CR21","doi-asserted-by":"crossref","unstructured":"Kulmizev, A., de\u00a0Lhoneux, M., Gontrum, J., Fano, E., & Nivre, J. (2019). Deep contextualized word embeddings in transition-based and graph-based dependency parsing: A tale of two parsers revisited. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing, Hong Kong (pp. 2755\u20132768).","DOI":"10.18653\/v1\/D19-1277"},{"key":"9591_CR22","doi-asserted-by":"crossref","unstructured":"Li, Z., Zhao, H., Zhang, Z., Wang, R., Utiyama, M., & Sumita, E. (2019). SJTU\u2013NICT at MRP\u00a02019: Multi-task learning for end-to-end uniform semantic graph parsing. In Proceedings of the shared task on cross-framework meaning representation parsing at the 2019 conference on computational natural language learning, Hong Kong (pp 45\u201354).","DOI":"10.18653\/v1\/K19-2004"},{"key":"9591_CR23","doi-asserted-by":"publisher","unstructured":"Lin, Z., & Xue, N. (2019). Parsing meaning representations: Is easier always better? In Proceedings of the first international workshop on designing meaning representations, Association for Computational Linguistics, Florence (pp. 34\u201343). https:\/\/doi.org\/10.18653\/v1\/W19-3304, https:\/\/www.aclweb.org\/anthology\/W19-3304","DOI":"10.18653\/v1\/W19-3304"},{"key":"9591_CR24","doi-asserted-by":"crossref","unstructured":"Lindemann, M., Groschwitz, J., & Koller, A. (2019). Compositional semantic parsing across graphbanks. In Proceedings of the 57th annual meeting of the Association for Computational Linguistics, Florence (pp. 4576\u20134585).","DOI":"10.18653\/v1\/P19-1450"},{"key":"9591_CR25","doi-asserted-by":"crossref","unstructured":"Lyu, C., & Titov, I. (2018). AMR parsing as graph prediction with latent alignment. arXiv preprint retrieved from http:\/\/arxiv.org\/abs\/180505286","DOI":"10.18653\/v1\/P18-1037"},{"key":"9591_CR26","unstructured":"McDonald, R., & Nivre, J. (2007). Characterizing the errors of data-driven dependency parsing models. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and conference on natural language learning, Prague."},{"issue":"1","key":"9591_CR27","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1162\/coli_a_00039","volume":"37","author":"R McDonald","year":"2011","unstructured":"McDonald, R., & Nivre, J. (2011). Analyzing and integrating dependency parsers. Computational Linguistics, 37(1), 197\u2013230.","journal-title":"Computational Linguistics"},{"key":"9591_CR28","doi-asserted-by":"crossref","unstructured":"Miyao, Y., Oepen, S., & Zeman, D. (2014). In-house: An ensemble of pre-existing off-the-shelf parsers. In Proceedings of the 8th international workshop on semantic evaluation, Dublin (pp. 63\u201372).","DOI":"10.3115\/v1\/S14-2056"},{"key":"9591_CR33","doi-asserted-by":"crossref","unstructured":"Oepen, S., Abend, O., Abzianidze, L., Bos, J., Haji\u010d, J., Hershcovich, D., Li, B., O\u2019Gorman, T., Xue, N., & Zeman, D. (2020). MRP\u00a02020: The second shared task on cross-framework and cross-lingual meaning representation parsing. In Proceedings of the CoNLL 2020 shared task: Cross-framework and cross-lingual meaning representation parsing, online (pp. 1\u201322).","DOI":"10.18653\/v1\/2020.conll-shared.1"},{"key":"9591_CR32","doi-asserted-by":"crossref","unstructured":"Oepen, S., Abend, O., Haji\u010d, J., Hershcovich, D., Kuhlmann, M., O\u2019Gorman, T., Xue, N., Chun, J., Straka, M., & Ure\u0161ov\u00e1, Z. (2019). MRP\u00a02019: Cross-framework Meaning Representation Parsing. In Proceedings of the shared task on cross-framework meaning representation parsing at the 2019 conference on computational natural language learning, Hong Kong (pp. 1\u201327).","DOI":"10.18653\/v1\/K19-2001"},{"key":"9591_CR31","doi-asserted-by":"crossref","unstructured":"Oepen, S., Kuhlmann, M., Miyao, Y., Zeman, D., Cinkov\u00e1, S., Flickinger, D., Haji\u010d, J., & Ure\u0161ov\u00e1, Z. (2015). SemEval 2015 Task 18. Broad-coverage semantic dependency parsing. In Proceedings of the 9th international workshop on semantic evaluation, Denver, CO (pp. 915\u2013926).","DOI":"10.18653\/v1\/S15-2153"},{"key":"9591_CR30","doi-asserted-by":"crossref","unstructured":"Oepen, S., Kuhlmann, M., Miyao, Y., Zeman, D., Flickinger, D., Haji\u010d, J., Ivanova, A., & Zhang, Y. (2014). SemEval 2014 Task 8. Broad-coverage semantic dependency parsing. In Proceedings of the 8th international workshop on semantic evaluation, Dublin (pp. 63\u201372). http:\/\/www.aclweb.org\/anthology\/S15-2153","DOI":"10.3115\/v1\/S14-2008"},{"key":"9591_CR29","unstructured":"Oepen, S., & L\u00f8nning, J. T. (2006). Discriminant-based MRS banking. In Proceedings of the 5th international conference on language resources and evaluation, Genoa (pp. 1250\u20131255)."},{"key":"9591_CR34","doi-asserted-by":"crossref","unstructured":"Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of the 2018 conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, volume 1 (long papers), New Orleans, LA (pp. 2227\u20132237).","DOI":"10.18653\/v1\/N18-1202"},{"key":"9591_CR35","doi-asserted-by":"crossref","unstructured":"Wang, Y., Che, W., Guo, J., & Liu, T. (2018). A neural transition-based approach for semantic dependency graph parsing. In Thirty-second AAAI conference on artificial intelligence.","DOI":"10.1609\/aaai.v32i1.11968"}],"container-title":["Language Resources and Evaluation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-022-09591-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10579-022-09591-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-022-09591-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,5]],"date-time":"2023-02-05T11:43:30Z","timestamp":1675597410000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10579-022-09591-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,17]]},"references-count":35,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["9591"],"URL":"https:\/\/doi.org\/10.1007\/s10579-022-09591-7","relation":{},"ISSN":["1574-020X","1574-0218"],"issn-type":[{"type":"print","value":"1574-020X"},{"type":"electronic","value":"1574-0218"}],"subject":[],"published":{"date-parts":[[2022,5,17]]},"assertion":[{"value":"5 April 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 May 2022","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}