{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T13:34:04Z","timestamp":1774618444673,"version":"3.50.1"},"reference-count":95,"publisher":"MIT Press","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2015,12]]},"abstract":"<jats:p>System design and evaluation methodologies receive significant attention in natural language processing (NLP), with the systems typically being evaluated on a common task and against shared data sets. This enables direct system comparison and facilitates progress in the field. However, computational work on metaphor is considerably more fragmented than similar research efforts in other areas of NLP and semantics. Recent years have seen a growing interest in computational modeling of metaphor, with many new statistical techniques opening routes for improving system accuracy and robustness. However, the lack of a common task definition, shared data set, and evaluation strategy makes the methods hard to compare, and thus hampers our progress as a community in this area. The goal of this article is to review the system features and evaluation strategies that have been proposed for the metaphor processing task, and to analyze their benefits and downsides, with the aim of identifying the desired properties of metaphor processing systems and a set of requirements for their evaluation.<\/jats:p>","DOI":"10.1162\/coli_a_00233","type":"journal-article","created":{"date-parts":[[2015,12,10]],"date-time":"2015-12-10T19:06:29Z","timestamp":1449774389000},"page":"579-623","source":"Crossref","is-referenced-by-count":47,"title":["Design and Evaluation of Metaphor Processing Systems"],"prefix":"10.1162","volume":"41","author":[{"given":"Ekaterina","family":"Shutova","sequence":"first","affiliation":[{"name":"University of California, Berkeley"}]}],"member":"281","reference":[{"key":"R1","unstructured":"Agerri, Rodrigo. 2008. Metaphor in textual entailment. In Proceedings of COLING 2008, pages 3\u20136, Manchester, UK."},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.3115\/1621474.1621476"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.3115\/1626481.1626508"},{"key":"R5","unstructured":"Badryzlova, Yulia, Natalia Shekhtman, Yekaterina Isaeva, and Ruslan Kerimov. 2013. Annotating a Russian corpus of conceptual metaphor: A bottom\u2013up approach. In Proceedings of the First Workshop on Metaphor in NLP, pages 77\u201386, Atlanta, GA."},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.12775\/ths.2002.017"},{"key":"R7","unstructured":"Baumer, Eric, Bill Tomlinson, and Lindsey Richland. 2009. Computational metaphor identification: A method for identifying conceptual metaphors in written text. In Proceedings of Analogy '09, pages 20\u201329, Sofia."},{"key":"R8","unstructured":"Beigman Klebanov, Beata and Eyal Beigman. 2010. A game-theoretic model of metaphorical bargaining. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 698\u2013709, Uppsala."},{"key":"R9","doi-asserted-by":"crossref","unstructured":"Beigman Klebanov, Beata and Michael Flor. 2013. Argumentation-relevant metaphors in test-taker essays. In Proceedings of the First Workshop on Metaphor in NLP, pages 11\u201320, Atlanta, GA.","DOI":"10.3115\/v1\/W14-2302"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1162\/jmlr.2003.3.4-5.993"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0074304"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324911000386"},{"key":"R15","unstructured":"Burnard, Lou. 2007. Reference Guide for the British National Corpus (XML Edition). Available from www.natcorp.ox.ac.uk\/corpus\/babyinfo.html."},{"key":"R16","unstructured":"Burstein, Jill, John Sabatini, Jane Shore, Brad Moulder, and Jennifer Lentini. 2013. A user study: Technology to increase teachers' linguistic awareness to improve instructional language support for English language learners. In Proceedings of the Workshop on Natural Language Processing for Improving Textual Accessibility, pages 1\u201310, Atlanta, GA."},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.1016\/S0889-4906(98)00025-8"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.1016\/S0889-4906(00)00009-0"},{"key":"R22","unstructured":"Chung, Siaw-Fong, Kathleen Ahrens, and Chu-Ren Huang. 2005. Source domains as concept domains in metaphorical expressions. International Journal of Computational Linguistics and Chinese Language Processing, 10(4):553\u2013570."},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2007.48"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.1007\/BF00994018"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.1075\/ijcl.14.2.02dav"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.1016\/j.pragma.2003.10.010"},{"key":"R27","unstructured":"Desalle, Yann, Bruno Gaume, and Karine Duvignau. 2009. Slam: Solutions lexicales automatique pour m\u00e9taphores. Traitement Automatique des Langues, 50(1):145\u2013175."},{"key":"R28","doi-asserted-by":"publisher","DOI":"10.1515\/ip-2013-0012"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.1080\/10926488.2011.535416"},{"key":"R30","doi-asserted-by":"crossref","unstructured":"Dunn, Jonathan. 2013a. Evaluating the premises and results of four metaphor identification systems. In Proceedings of CICLing'13, pages 471\u2013486, Samos.","DOI":"10.1007\/978-3-642-37247-6_38"},{"key":"R31","unstructured":"Dunn, Jonathan. 2013b. What metaphor identification systems can tell us about metaphor-in-language. In Proceedings of the First Workshop on Metaphor in NLP, pages 1\u201310, Atlanta, GA."},{"key":"R32","unstructured":"Fass, Dan. 1991. met*: A method for discriminating metonymy and metaphor by computer. Computational Linguistics, 17(1):49\u201390."},{"key":"R34","doi-asserted-by":"publisher","DOI":"10.1016\/S0093-934X(03)00355-9"},{"key":"R35","doi-asserted-by":"publisher","DOI":"10.1093\/ijl\/16.3.235"},{"key":"R37","doi-asserted-by":"crossref","unstructured":"Gandy, Lisa, Nadji Allan, Mark Atallah, Ophir Frieder, Newton Howard, Sergey Kanareykin, Moshe Koppel, Mark Last, Yair Neuman, and Shlomo Argamon. 2013. Automatic identification of conceptual metaphors with limited knowledge. In Proceedings of AAAI 2013, pages 328\u2013334, Bellevue, WA.","DOI":"10.1609\/aaai.v27i1.8648"},{"key":"R38","doi-asserted-by":"publisher","DOI":"10.3115\/1621459.1621467"},{"key":"R39","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog0702_3"},{"key":"R40","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog0803_4"},{"key":"R41","doi-asserted-by":"publisher","DOI":"10.1142\/S1793840608001755"},{"key":"R42","unstructured":"Grady, Joe. 1997. Foundations of Meaning: Primary Metaphors and Primary Scenes. Ph.D. thesis, University of California at Berkeley."},{"key":"R43","unstructured":"Hardie, Andrew, Veronika Koller, Paul Rayson, and Elena Semino. 2007. Exploiting a semantic annotation tool for metaphor analysis. In Proceedings of the Corpus Linguistics Conference, pages 1\u201312, Birmingham, UK."},{"key":"R44","doi-asserted-by":"publisher","DOI":"10.3115\/992133.992154"},{"key":"R45","unstructured":"Heintz, Ilana, Ryan Gabbard, Mahesh Srivastava, Dave Barner, Donald Black, Majorie Friedman, and Ralph Weischedel. 2013. Automatic extraction of linguistic metaphors with lda topic modeling. In Proceedings of the First Workshop on Metaphor in NLP, pages 58\u201366, Atlanta, GA."},{"key":"R47","unstructured":"Hobbs, Jerry R. 1981. Metaphor interpretation as selective inferencing. In Proceedings of the 7th International Joint Conference on Artificial Intelligence Volume 1, IJCAI'81, pages 85\u201391, Vancouver."},{"key":"R50","unstructured":"Hovy, Dirk, Shashank Shrivastava, Sujay Kumar Jauhar, Mrinmaya Sachan, Kartik Goyal, Huying Li, Whitney Sanders, and Eduard Hovy. 2013. Identifying metaphorical word use with tree kernels. In Proceedings of the First Workshop on Metaphor in NLP, pages 52\u201357, Atlanta, GA."},{"key":"R51","unstructured":"Izwaini, Sattar. 2003. Corpus-based study of metaphor in information technology. In Proceedings of the Workshop on Corpus-based Approaches to Figurative Language, Corpus Linguistics 2003, pages 1\u20138, Lancaster."},{"key":"R52","unstructured":"Jamrozik, Anja, Eyal Sagi, Micah Goldwater, and Dedre Gentner. 2013. Relational words have high metaphoric potential. In Proceedings of the First Workshop on Metaphor in NLP, pages 21\u201326, Atlanta, GA."},{"key":"R53","unstructured":"Karov, Yael and Shimon Edelman. 1998. Similarity-based word sense disambiguation. Computational Linguistics, 24(1):41\u201359."},{"key":"R54","unstructured":"Kingsbury, Paul and Martha Palmer. 2002. From TreeBank to PropBank. In Proceedings of LREC-2002, pages 1989\u20131993, Gran Canaria."},{"key":"R56","unstructured":"Korkontzelos, Ioannis, Torsten Zesch, Fabio Massimo Zanzotto, and Chris Biemann. 2013. Semeval-2013 task 5: Evaluating phrasal semantics. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 39\u201347, Atlanta, GA."},{"key":"R58","doi-asserted-by":"publisher","DOI":"10.3115\/1611528.1611531"},{"key":"R63","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-9280.2009.02462.x"},{"key":"R64","doi-asserted-by":"crossref","unstructured":"Lefever, Els and V\u00e9ronique Hoste. 2010. Semeval-2010 task 3: Cross-lingual word sense disambiguation. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 15\u201320, Uppsala.","DOI":"10.3115\/1621969.1621984"},{"key":"R65","unstructured":"Lewis, David D., Yiming Yang, Tony G. Rose, and Fan Li. 2004. Rcv1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5:361\u2013397."},{"key":"R66","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00235"},{"key":"R67","doi-asserted-by":"publisher","DOI":"10.3115\/1699510.1699552"},{"key":"R68","unstructured":"Li, Linlin and Caroline Sporleder. 2010. Using Gaussian mixture models to detect figurative language in context. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 297\u2013300, Singapore."},{"key":"R69","unstructured":"L\u00f6nneker, Birte. 2004. Lexical databases as resources for linguistic creativity: Focus on metaphor. In Proceedings of the LREC 2004 Workshop on Language Resources for Linguistic Creativity, pages 9\u201316, Lisbon."},{"key":"R70","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-008-9073-9"},{"key":"R72","doi-asserted-by":"publisher","DOI":"10.1177\/0957926508088966"},{"key":"R73","doi-asserted-by":"crossref","unstructured":"Manandhar, Suresh, Ioannis Klapaftis, Dmitriy Dligach, and Sameer Pradhan. 2010. Semeval-2010 task 14: Word sense induction & disambiguation. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 63\u201368, Uppsala.","DOI":"10.3115\/1621969.1621990"},{"key":"R76","doi-asserted-by":"publisher","DOI":"10.1162\/089120104773633376"},{"key":"R77","doi-asserted-by":"publisher","DOI":"10.3115\/1621474.1621483"},{"key":"R78","unstructured":"Mihalcea, Rada, Ravi Sinha, and Diana McCarthy. 2010. Semeval-2010 task 2: Cross-lingual lexical substitution. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 9\u201314, Uppsala."},{"key":"R79","unstructured":"Mohler, Michael, David Bracewell, Marc Tomlinson, and David Hinote. 2013. Semantic signatures for example-based linguistic metaphor detection. In Proceedings of the First Workshop on Metaphor in NLP, pages 27\u201335, Atlanta, GA."},{"key":"R80","unstructured":"Moschitti, Ro, Daniele Pighin, and Roberto Basili. 2006. Tree kernel engineering for proposition re-ranking. In Proceedings of Mining and Learning with Graphs (MLG), pages 165\u2013172, Berlin."},{"key":"R82","unstructured":"Nakov, Preslav, Sara Rosenthal, Zornitsa Kozareva, Veselin Stoyanov, Alan Ritter, and Theresa Wilson. 2013. Semeval-2013 task 2: Sentiment analysis in twitter. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 312\u2013320, Atlanta, GA."},{"key":"R83","unstructured":"Narayanan, Srini. 1997. Knowledge-Based Action Representations for Metaphor and Aspect (KARMA). Ph.D. thesis, University of California at Berkeley."},{"key":"R84","unstructured":"Narayanan, Srini. 1999. Moving right along: A computational model of metaphoric reasoning about events. In Proceedings of AAAI 99), pages 121\u2013128, Orlando, FL."},{"key":"R85","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0062343"},{"key":"R87","doi-asserted-by":"crossref","unstructured":"Niles, Ian and Adam Pease. 2001. Towards a standard upper ontology. In Proceedings of the International Conference on Formal Ontology in Information Systems -Volume 2001, FOIS '01, pages 2\u20139, New York, NY.","DOI":"10.1145\/505168.505170"},{"key":"R88","unstructured":"Niles, Ian and Adam Pease. 2003. Linking lexicons and ontologies: Mapping WordNet to the suggested upper merged ontology. In Proceedings of the International Conference on Information and Knowledge Engineering, pages 412\u2013416, Las Vegas, NV."},{"key":"R89","doi-asserted-by":"publisher","DOI":"10.3115\/980304.980349"},{"key":"R90","unstructured":"Peters, Wim and Ivonne Peters. 2000. Lexicalised systematic polysemy in WordNet. In Proceedings of LREC 2000, pages 1\u20137, Athens."},{"key":"R91","doi-asserted-by":"publisher","DOI":"10.1080\/10926480709336752"},{"key":"R92","doi-asserted-by":"crossref","unstructured":"Recasens, Marta, Llu\u00eds M\u00e0rquez, Emili Sapena, M. Ant\u00f2nia Mart\u00ed, Mariona Taul\u00e9, V\u00e9ronique Hoste, Massimo Poesio, and Yannick Versley. 2010. Semeval-2010 task 1: Coreference resolution in multiple languages. In Proceedings of the 5th International Workshop on Semantic Evaluation, SemEval '10, pages 1\u20138, Uppsala.","DOI":"10.3115\/1621969.1621982"},{"key":"R93","doi-asserted-by":"publisher","DOI":"10.3115\/1611528.1611530"},{"key":"R94","doi-asserted-by":"crossref","unstructured":"Resnik, Philip. 1993. Selection and Information: A Class-Based Approach to Lexical Relationships. Ph.D. thesis, University of Pennsylvania.","DOI":"10.3115\/981967.982021"},{"key":"R96","unstructured":"Sandhaus, Evan. 2008. The New York Times Annotated Corpus. Available from https:\/\/catalog\/lde.upenn.edu\/LDC2008T19."},{"key":"R97","doi-asserted-by":"publisher","DOI":"10.1177\/0957926599010002004"},{"key":"R98","unstructured":"Shutova, Ekaterina. 2010. Automatic metaphor interpretation as a paraphrasing task. In Proceedings of NAACL 2010, pages 1029\u20131037, Los Angeles, CA."},{"key":"R99","unstructured":"Shutova, Ekaterina. 2013. Metaphor identification as interpretation. In Proceedings of *SEM 2013, pages 276\u2013285, Atlanta, GA."},{"key":"R100","unstructured":"Shutova, Ekaterina and Lin Sun. 2013. Unsupervised metaphor identification using hierarchical graph factorization clustering. In Proceedings of NAACL 2013, pages 978\u2013988, Atlanta, GA."},{"key":"R102","unstructured":"Shutova, Ekaterina and Simone Teufel. 2010. Metaphor corpus annotated for source\u2013 target domain mappings. In Proceedings of LREC 2010, pages 3255\u20133261, Malta."},{"key":"R103","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00124"},{"key":"R104","unstructured":"Shutova, Ekaterina, Tim Van de Cruys, and Anna Korhonen. 2012. Unsupervised metaphor paraphrasing using a vector space model. In Proceedings of COLING 2012, pages 1121\u20131130, Mumbai."},{"key":"R105","unstructured":"Skorczynska Sznajder, Hanna and Jordi Pique-Angordans. 2004. A corpus-based description of metaphorical marking patterns in scientific and popular business discourse. In Proceedings of European Research Conference on Mind, Language and Metaphor (Euresco Conference), pages 112\u2013129, Granada."},{"key":"R108","unstructured":"Strzalkowski, Tomek, George Aaron Broadwell, Sarah Taylor, Laurie Feldman, Samira Shaikh, Ting Liu, Boris Yamrom, Kit Cho, Umit Boz, Ignacio Cases, and Kyle Elliot. 2013. Robust extraction of metaphor from novel data. In Proceedings of the First Workshop on Metaphor in NLP, pages 67\u201376, Atlanta, GA."},{"key":"R109","unstructured":"Sullivan, Karen. 2007. Grammar in metaphor: A construction grammar account of metaphoric language. Ph.D. thesis, University of California, Berkeley."},{"key":"R111","doi-asserted-by":"crossref","unstructured":"Sun, Lin and Anna Korhonen. 2009. Improving verb clustering with automatically acquired selectional preferences. In Proceedings of EMNLP 2009, pages 638\u2013647, Singapore.","DOI":"10.3115\/1699571.1699596"},{"key":"R112","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0016782"},{"key":"R113","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-1024"},{"key":"R114","unstructured":"Tsvetkov, Yulia, Elena Mukomel, and Anatole Gershman. 2013. Cross-lingual metaphor detection using common semantic features. In Proceedings of the First Workshop on Metaphor in NLP, pages 45\u201351, Atlanta, GA."},{"key":"R116","unstructured":"Turney, Peter D., Yair Neuman, Dan Assaf, and Yohai Cohen. 2011. Literal and metaphorical sense identification through concrete and abstract context. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP '11, pages 680\u2013690, Stroudsburg, PA."},{"key":"R117","unstructured":"Veale, Tony. 2011. Creative language retrieval: A robust hybrid of information retrieval and linguistic creativity. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 278\u2013287, Portland, OR."},{"key":"R118","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-2307"},{"key":"R119","doi-asserted-by":"crossref","unstructured":"Veale, Tony and Yanfen Hao. 2008. A fluid knowledge representation for understanding and generating creative metaphors. In Proceedings of COLING 2008, pages 945\u2013952, Manchester, UK.","DOI":"10.3115\/1599081.1599200"},{"key":"R121","unstructured":"Wikberg, Kay. 2006. The role of corpus studies in metaphor research. In 2006 Stockholm Metaphor Festival, pages 33\u201348, Stockholm."},{"key":"R122","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(75)90016-8"},{"key":"R123","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(78)90001-2"},{"key":"R124","unstructured":"Wilks, Yorick, Adam Dalton, James Allen, and Lucian Galescu. 2013. Automatic metaphor detection using large-scale lexical resources and conventional metaphor extraction. In Proceedings of the First Workshop on Metaphor in NLP, pages 36\u201344, Atlanta, GA."}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/COLI_a_00233","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,5,31]],"date-time":"2025-05-31T18:53:50Z","timestamp":1748717630000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/41\/4\/579-623\/1515"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,12]]},"references-count":95,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2015,12]]}},"alternative-id":["10.1162\/COLI_a_00233"],"URL":"https:\/\/doi.org\/10.1162\/coli_a_00233","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,12]]}}}