{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,19]],"date-time":"2026-03-19T12:20:02Z","timestamp":1773922802118,"version":"3.50.1"},"reference-count":43,"publisher":"Cambridge University Press (CUP)","issue":"4","license":[{"start":{"date-parts":[[2017,4,9]],"date-time":"2017-04-09T00:00:00Z","timestamp":1491696000000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2017,7]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Current methods for assessing the impact of authors and scientific media employ tools such as H-Index, Co-Citation and PageRank. These tools are primarily based on citation counting, which considers all citations to be equal. This type of methods can produce perverse incentives to publish controversial or incomplete papers, as mixed or negative reviews often generate larger citation counts and better indexes, regardless of whether the citations were critical or exerted minimal influence on the citing document. Passing citations that are employed to establish background, which do not have a real impact on the citing paper, are common in scientific literature. However, these citations have equal weight in impact evaluations. Notable researchers have emphasized the need to correct this situation by developing estimation methods that consider the different roles of quotations in citing papers. To accomplish this type of evaluation, a context citation analysis should be applied to determine the nature of the citations. We propose that citations should be categorized using four dimensions \u2013 FUNCTION, POLARITY, ASPECTS and INFLUENCE \u2013 as these dimensions provide adequate information that can be employed toward the generation of a qualitative method to measure the impact of a given publication in a citing paper. In this paper, we used interchangeably the words influence and impact. We present a method for obtaining this information using our proposed classification scheme and manually annotated corpus, which is marked with meaningful keywords and labels to help identify the characteristics or properties that constitute what we call ASPECTS. We develop a classification scheme which considers purpose definition shared by previous works. Our contribution is to abstract purpose classes from several other schemes and divide a complex structure in more manageable parts, to attain a simple system that combines low granularity dimensions but nevertheless produces a fine-grained classification. For annotators, the classification process is simple because in a first step, the coders distinguish only four primary classes, and in a second pass, they add the information contained in ASPECTS keyword and labels to obtain the more specific functions. This way, we gain a high granularity labeling that gives enough information about the citations to characterize and classify them, and we achieve this detailed coding with a straightforward process where the level of human error could be minimized.<\/jats:p>","DOI":"10.1017\/s1351324916000346","type":"journal-article","created":{"date-parts":[[2017,4,9]],"date-time":"2017-04-09T18:11:14Z","timestamp":1491761474000},"page":"561-588","source":"Crossref","is-referenced-by-count":43,"title":["Citation function, polarity and influence classification"],"prefix":"10.1017","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4718-0400","authenticated-orcid":false,"given":"MYRIAM","family":"HERN\u00c1NDEZ-ALVAREZ","sequence":"first","affiliation":[]},{"given":"JOS\u00c9 M.","family":"GOMEZ SORIANO","sequence":"additional","affiliation":[]},{"given":"PATRICIO","family":"MART\u00cdNEZ-BARCO","sequence":"additional","affiliation":[]}],"member":"56","published-online":{"date-parts":[[2017,4,9]]},"reference":[{"key":"S1351324916000346_ref018","doi-asserted-by":"crossref","unstructured":"Hyland K. 1998. Hedging in Scientific Research Articles, Vol. 54. Amsterdam: John Benjamins Publishing.","DOI":"10.1075\/pbns.54"},{"key":"S1351324916000346_ref030","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-24840-8_6"},{"key":"S1351324916000346_ref029","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23612"},{"key":"S1351324916000346_ref038","first-page":"364","article-title":"The introduction, methods, results, and discussion (IMRAD) structure: a fifty-year survey","volume":"92","author":"Sollaci","year":"2004","journal-title":"Journal of the Medical Library Association"},{"key":"S1351324916000346_ref035","doi-asserted-by":"publisher","DOI":"10.1038\/srep00815"},{"key":"S1351324916000346_ref034","doi-asserted-by":"crossref","unstructured":"Prabha C. G. 1983. Some aspects of citation behavior: a pilot study in business administration. Journal of the American Society for Information Science, 34 (3): 202\u20136.","DOI":"10.1002\/asi.4630340305"},{"key":"S1351324916000346_ref037","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630240406"},{"key":"S1351324916000346_ref031","first-page":"460","volume-title":"Proceedings of the International Conference Recent Advances in Natural Language Processing RANLP 2013","author":"Meyers","year":"2013"},{"key":"S1351324916000346_ref002","doi-asserted-by":"publisher","DOI":"10.1162\/coli.07-034-R2"},{"key":"S1351324916000346_ref016","first-page":"45","article-title":"Esquema de anotaci\u00f3n para categorizaci\u00f3n de citas en bibliograf\u00eda cient\u00edfica","volume":"54","author":"Hern\u00e1ndez\u2013Alvarez","year":"2015","journal-title":"Procesamiento del Lenguaje Natural"},{"key":"S1351324916000346_ref004","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(198907)40:4<284::AID-ASI10>3.0.CO;2-Z"},{"key":"S1351324916000346_ref010","doi-asserted-by":"publisher","DOI":"10.1037\/h0031619"},{"key":"S1351324916000346_ref021","first-page":"1343","volume-title":"Proceedings of COLING\u201912","author":"Jochim","year":"2012"},{"key":"S1351324916000346_ref008","first-page":"66","volume-title":"Extended Semantic Web Conference","author":"Di Iorio","year":"2013"},{"key":"S1351324916000346_ref003","unstructured":"Athar A. 2014. Sentiment analysis of scientific citations. Technical Report (UCAM-CL-TR-856), University of Cambridge, Computer Laboratoy."},{"key":"S1351324916000346_ref036","unstructured":"Sample I. 2013. Nobel winner declares boycott of top science journals. The Guardian. http:\/\/www.theguardian.com\/science\/2013\/dec\/09\/nobel-winner-boycott-science-journals."},{"key":"S1351324916000346_ref014","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630290608"},{"key":"S1351324916000346_ref022","first-page":"1340","volume-title":"AAAI Conference on Artificial Intelligence","author":"Kataria","year":"2010"},{"key":"S1351324916000346_ref033","unstructured":"Page L. , Brin S. , Motwani R. , and Winograd T. 1999. The PageRank citation ranking: bringing order to the web. Technical Report (SIDL-WP-1999-0120), Stanford InfoLab, Stanford University."},{"key":"S1351324916000346_ref040","unstructured":"Teufel S. 1999. Argumentative zoning: information extraction from scientific text. Doctoral dissertation, School of Cognitive Science, University of Edinburgh, UK. http:\/\/www.cl.cam.ac.uk\/~sht25\/thesis\/t1.pdf."},{"key":"S1351324916000346_ref025","first-page":"402","volume-title":"Proceedings of Recent Advances in Natural Language Processing","author":"Li","year":"2013"},{"key":"S1351324916000346_ref007","doi-asserted-by":"publisher","DOI":"10.1007\/BF00994018"},{"key":"S1351324916000346_ref026","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/bts071"},{"key":"S1351324916000346_ref009","first-page":"623","volume-title":"Proceedings of 5th International Joint Conference on Natural Language Processing, Asian Federation of Natural Language Processing","author":"Dong","year":"2011"},{"key":"S1351324916000346_ref039","volume-title":"Genre analysis: English in academic and research settings","author":"Swales","year":"1990"},{"key":"S1351324916000346_ref024","doi-asserted-by":"crossref","unstructured":"Landis J. R. , and Koch G. G. (1977). The measurement of observer agreement for categorical data. Biometrics 33: 159\u201374.","DOI":"10.2307\/2529310"},{"key":"S1351324916000346_ref027","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1016516107"},{"key":"S1351324916000346_ref028","doi-asserted-by":"publisher","DOI":"10.1007\/BF02017729"},{"key":"S1351324916000346_ref015","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324915000388"},{"key":"S1351324916000346_ref006","doi-asserted-by":"crossref","unstructured":"Ciancarini P. , Di Iorio A. , Nuzzolese A. G. , Peroni S. , and Vitali F. 2013. Semantic annotation of scholarly documents and citations. In M. Baldoni , C. Baroglio , G. Boella , and R. Micalizio (eds.), AI*IA 2013: Advances in Artificial Intelligence, 8249: pp. 336\u201347. Berlin: Springer.","DOI":"10.1007\/978-3-319-03524-6_29"},{"key":"S1351324916000346_ref005","doi-asserted-by":"crossref","unstructured":"Ciancarini P. , DiAAAAIorio A. , Nuzzolese A. G. , Peroni S. , and Vitali F. 2014. Evaluating citation functions in CiTO: Cognitive issues. In V. Presutti , M. Stankovic , E. Cambria , I. Cantador , A. DiAAAAIorio , T. DiAAAANoia , C. Lange , D. R. Recupero , and A. Tordai (eds.), Semantic Web: Trends and Challenges, pp. 580\u201394. Berlin: Springer International Publishing.","DOI":"10.1007\/978-3-319-07443-6_39"},{"key":"S1351324916000346_ref011","unstructured":"Garzone M. A. 1997. Automated classification of citations using linguistic semantic grammars. Master\u2019s Thesis. The University of Western Ontario. Available at http:\/\/www.collectionscanada.gc.ca\/obj\/s4\/f2\/dsk2\/ftp04\/mq28570.pdf"},{"key":"S1351324916000346_ref043","unstructured":"Verlic M. , Stiglic G. , Kocbek S. , and Kokol P. 2008. Sentiment in Science - a case study of CBMS contributions in years 2003 to 2007. In 2008 21st IEEE International Symposium on Computer-Based Medical Systems, Finland: University of Jyv\u00e4skyl\u00e4, pp. 138\u201343."},{"key":"S1351324916000346_ref032","doi-asserted-by":"publisher","DOI":"10.1177\/030631277500500106"},{"key":"S1351324916000346_ref042","first-page":"80","volume-title":"Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue","author":"Teufel","year":"2009"},{"key":"S1351324916000346_ref023","first-page":"411","article-title":"Reliability in content analysis: some common misconceptions and recommendations","volume":"30","author":"Krippendorff","year":"2004","journal-title":"Human Communication Research"},{"key":"S1351324916000346_ref001","first-page":"596","volume-title":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Abu-Jbara","year":"2013"},{"key":"S1351324916000346_ref012","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-45486-1_28"},{"key":"S1351324916000346_ref019","doi-asserted-by":"publisher","DOI":"10.1093\/applin\/17.4.433"},{"key":"S1351324916000346_ref041","first-page":"103","volume-title":"Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing","author":"Teufel","year":"2006"},{"key":"S1351324916000346_ref020","unstructured":"Di Iorio A. , Nuzzolese A. G. , and Peroni S. 2013. Towards the automatic identification of the nature of citations. In A. Garc\u00eda , C. Lange , P. Lord and R. Stevens (eds.), SePublica, pp. 63\u201374. Montpellier, France: SePublica."},{"key":"S1351324916000346_ref013","unstructured":"Geertzen J. 2012. Inter-Rater Agreement with multiple raters and variables. Retrieved October 8, 2014, from https:\/\/nlp-ml.io\/jg\/software\/ira\/."},{"key":"S1351324916000346_ref017","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.0507655102"}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324916000346","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,9,20]],"date-time":"2019-09-20T16:56:46Z","timestamp":1568998606000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324916000346\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,4,9]]},"references-count":43,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2017,7]]}},"alternative-id":["S1351324916000346"],"URL":"https:\/\/doi.org\/10.1017\/s1351324916000346","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,4,9]]}}}