{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,10]],"date-time":"2026-11-10T22:51:59Z","timestamp":1794351119270,"version":"4.9.0"},"reference-count":57,"publisher":"MIT Press","issue":"3","license":[{"start":{"date-parts":[[2021,9,6]],"date-time":"2021-09-06T00:00:00Z","timestamp":1630886400000},"content-version":"vor","delay-in-days":248,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000026","name":"National Institute on Drug Abuse","doi-asserted-by":"publisher","award":["4R44DA050155-02"],"award-info":[{"award-number":["4R44DA050155-02"]}],"id":[{"id":"10.13039\/100000026","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,11,5]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Citation indices are tools used by the academic community for research and research evaluation that aggregate scientific literature output and measure impact by collating citation counts. Citation indices help measure the interconnections between scientific papers but fall short because they fail to communicate contextual information about a citation. The use of citations in research evaluation without consideration of context can be problematic because a citation that presents contrasting evidence to a paper is treated the same as a citation that presents supporting evidence. To solve this problem, we have used machine learning, traditional document ingestion methods, and a network of researchers to develop a \u201csmart citation index\u201d called scite, which categorizes citations based on context. Scite shows how a citation was used by displaying the surrounding textual context from the citing paper and a classification from our deep learning model that indicates whether the statement provides supporting or contrasting evidence for a referenced work, or simply mentions it. Scite has been developed by analyzing over 25 million full-text scientific articles and currently has a database of more than 880 million classified citation statements. Here we describe how scite works and how it can be used to further research and research evaluation.<\/jats:p>","DOI":"10.1162\/qss_a_00146","type":"journal-article","created":{"date-parts":[[2021,7,17]],"date-time":"2021-07-17T10:55:20Z","timestamp":1626519320000},"page":"882-898","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":146,"title":["scite: A smart citation index that displays the context of citations and classifies their intent using deep learning"],"prefix":"10.1162","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1111-1828","authenticated-orcid":true,"given":"Josh M.","family":"Nicholson","sequence":"first","affiliation":[{"name":"scite, Brooklyn, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5395-4252","authenticated-orcid":true,"given":"Milo","family":"Mordaunt","sequence":"additional","affiliation":[{"name":"scite, Brooklyn, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9959-9441","authenticated-orcid":true,"given":"Patrice","family":"Lopez","sequence":"additional","affiliation":[{"name":"science-miner, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8748-1465","authenticated-orcid":true,"given":"Ashish","family":"Uppala","sequence":"additional","affiliation":[{"name":"scite, Brooklyn, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2666-7615","authenticated-orcid":true,"given":"Domenic","family":"Rosati","sequence":"additional","affiliation":[{"name":"scite, Brooklyn, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6950-2135","authenticated-orcid":true,"given":"Neves P.","family":"Rodrigues","sequence":"additional","affiliation":[{"name":"scite, Brooklyn, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5658-2482","authenticated-orcid":true,"given":"Peter","family":"Grabitz","sequence":"additional","affiliation":[{"name":"scite, Brooklyn, NY, USA"},{"name":"Charite Universitaetsmedizin Berlin, Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6748-0841","authenticated-orcid":true,"given":"Sean C.","family":"Rife","sequence":"additional","affiliation":[{"name":"scite, Brooklyn, NY, USA"},{"name":"Murray State University, Murray, KY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2021,11,5]]},"reference":[{"key":"2021110817022675400_bib1","first-page":"81","article-title":"Sentiment analysis of citations using sentence structure-based features","volume-title":"Proceedings of the ACL 2011 Student Session","author":"Athar","year":"2011"},{"key":"2021110817022675400_bib2","volume-title":"Sentiment analysis of scientific citations","author":"Athar","year":"2014"},{"key":"2021110817022675400_bib3","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1371","article-title":"SciBERT: A pretrained language model for scientific text","author":"Beltagy","year":"2019","journal-title":"ArXiv:1903.10676 [Cs]"},{"key":"2021110817022675400_bib4","volume-title":"Natural language processing with Python","author":"Bird","year":"2009","edition":"1st ed"},{"issue":"2","key":"2021110817022675400_bib5","doi-asserted-by":"publisher","first-page":"1225","DOI":"10.1007\/s11192-020-03536-z","article-title":"Self-correction of science: A comparative study of negative citations and post-publication peer review","volume":"124","author":"Bordignon","year":"2020","journal-title":"Scientometrics"},{"key":"2021110817022675400_bib6","doi-asserted-by":"publisher","first-page":"580","DOI":"10.1007\/978-3-319-07443-6_39","article-title":"Evaluating citation functions in CiTO: Cognitive issues","volume-title":"The Semantic Web: Trends and Challenges","author":"Ciancarini","year":"2014"},{"key":"2021110817022675400_bib7","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1361","article-title":"Structural scaffolds for citation intent classification in scientific publications","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics","author":"Cohan","year":"2019"},{"key":"2021110817022675400_bib8","volume-title":"Automatic structure and keyphrase analysis of scientific publications","author":"Constantin","year":"2014"},{"key":"2021110817022675400_bib9","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1423","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics","author":"Devlin","year":"2019"},{"issue":"7718","key":"2021110817022675400_bib10","doi-asserted-by":"publisher","first-page":"290","DOI":"10.1038\/d41586-018-05968-3","article-title":"How Unpaywall is transforming open science","volume":"560","author":"Else","year":"2018","journal-title":"Nature"},{"issue":"1","key":"2021110817022675400_bib11","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1186\/s13326-015-0037-x","article-title":"Colil: A database and search service for citation contexts in the life sciences domain","volume":"6","author":"Fujiwara","year":"2015","journal-title":"Journal of Biomedical Semantics"},{"issue":"3159","key":"2021110817022675400_bib12","doi-asserted-by":"publisher","first-page":"108","DOI":"10.1126\/science.122.3159.108","article-title":"Citation indexes for science: A new dimension in documentation through association of ideas","volume":"122","author":"Garfield","year":"1955","journal-title":"Science"},{"key":"2021110817022675400_bib13","article-title":"Letter to Dr. Joshua Lederberg, Stanford University","author":"Garfield","year":"1959"},{"key":"2021110817022675400_bib14","first-page":"189","article-title":"Can Citation Indexing be Automated?","volume-title":"Statistical association methods for mechanized documentation, symposium proceedings, Washington 1964","author":"Garfield","year":"1964"},{"issue":"4060","key":"2021110817022675400_bib15","doi-asserted-by":"publisher","first-page":"471","DOI":"10.1126\/science.178.4060.471","article-title":"Citation analysis as a tool in journal evaluation: Journals can be ranked by frequency and impact of citations for science policy studies","volume":"178","author":"Garfield","year":"1972","journal-title":"Science"},{"key":"2021110817022675400_bib16","doi-asserted-by":"publisher","first-page":"b2680","DOI":"10.1136\/bmj.b2680","article-title":"How citation distortions create unfounded authority: Analysis of a citation network","volume":"339","author":"Greenberg","year":"2009","journal-title":"BMJ"},{"issue":"1","key":"2021110817022675400_bib17","doi-asserted-by":"publisher","first-page":"2","DOI":"10.29024\/joa.1","article-title":"An initiative to track sentiments in altmetrics","volume":"1","author":"Halevi","year":"2018","journal-title":"Journal of Altmetrics"},{"issue":"3","key":"2021110817022675400_bib18","doi-asserted-by":"publisher","first-page":"1645","DOI":"10.1007\/s11192-018-2944-y","article-title":"Deep context of citations using machine-learning models in scholarly full-text articles","volume":"117","author":"Hassan","year":"2018","journal-title":"Scientometrics"},{"issue":"46","key":"2021110817022675400_bib19","doi-asserted-by":"publisher","first-page":"16569","DOI":"10.1073\/pnas.0507655102","article-title":"An index to quantify an individual\u2019s scientific research output","volume":"102","author":"Hirsch","year":"2005","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2021110817022675400_bib20","doi-asserted-by":"publisher","DOI":"10.5281\/ZENODO.1212304","volume-title":"Explosion\/paCy: V2.0.11: Alpha Vietnamese support, fixes to vectors, improved errors and more","author":"Honnibal","year":"2018"},{"key":"2021110817022675400_bib21","author":"Initiative for Open Citations","year":"2017"},{"issue":"1","key":"2021110817022675400_bib22","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1186\/s40537-019-0192-5","article-title":"Survey on deep learning with class imbalance","volume":"6","author":"Johnson","year":"2019","journal-title":"Journal of Big Data"},{"issue":"7612","key":"2021110817022675400_bib23","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1038\/nj7612-457a","article-title":"Scientific literature: Information overload","volume":"535","author":"Landhuis","year":"2016","journal-title":"Nature"},{"key":"2021110817022675400_bib24","first-page":"162","article-title":"Shepardizing","volume-title":"West\u2019s encyclopedia of American law","author":"Lehman","year":"2005","edition":"2nd ed."},{"issue":"9","key":"2021110817022675400_bib25","doi-asserted-by":"publisher","first-page":"e0222213","DOI":"10.1371\/journal.pone.0222213","article-title":"Affirmative citation bias in scientific myth debunking: A three-in-one case study","volume":"14","author":"Letrud","year":"2019","journal-title":"PLOS ONE"},{"issue":"22","key":"2021110817022675400_bib26","doi-asserted-by":"publisher","first-page":"2194","DOI":"10.1056\/NEJMc1700150","article-title":"A 1980 letter on the risk of opioid addiction","volume":"376","author":"Leung","year":"2017","journal-title":"New England Journal of Medicine"},{"key":"2021110817022675400_bib27","article-title":"S2ORC: The Semantic Scholar Open Research Corpus","author":"Lo","year":"2020","journal-title":"ArXiv:1911.02782 [Cs]"},{"key":"2021110817022675400_bib28","volume-title":"GROBID","author":"Lopez","year":"2020"},{"key":"2021110817022675400_bib29","volume-title":"Pub2TEI","author":"Lopez","year":"2020"},{"key":"2021110817022675400_bib30","volume-title":"biblio-glutton","author":"Lopez","year":"2020"},{"key":"2021110817022675400_bib31","volume-title":"delft","author":"Lopez","year":"2020"},{"key":"2021110817022675400_bib32","first-page":"2370","article-title":"Measuring disagreement in science","volume-title":"17th International Conference on Scientometrics & Informetrics","author":"Murray","year":"2019"},{"key":"2021110817022675400_bib33","article-title":"doccano: Text annotation tool for humans","author":"Nakayama","year":"2018"},{"key":"2021110817022675400_bib34","doi-asserted-by":"publisher","first-page":"e05068","DOI":"10.7554\/eLife.05068","article-title":"Chromosome mis-segregation and cytokinesis failure in trisomic human cells","volume":"4","author":"Nicholson","year":"2015","journal-title":"eLife"},{"issue":"14","key":"2021110817022675400_bib35","doi-asserted-by":"publisher","first-page":"4242","DOI":"10.1111\/febs.15608","article-title":"Measuring the quality of scientific references in Wikipedia: An analysis of more than 115M citations to over 800 000 scientific articles","volume":"288","author":"Nicholson","year":"2020","journal-title":"FEBS Journal"},{"key":"2021110817022675400_bib36","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1145\/3292522.3326050","article-title":"Extremist propaganda tweet classification with deep learning in realistic scenarios","volume-title":"Proceedings of the 10th ACM Conference on Web Science - WebSci \u201919","author":"Nizzoli","year":"2019"},{"key":"2021110817022675400_bib37","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1016\/j.websem.2012.08.001","article-title":"FaBiO and CiTO: Ontologies for describing bibliographic resources and citations","volume":"17","author":"Peroni","year":"2012","journal-title":"Journal of Web Semantics"},{"issue":"1","key":"2021110817022675400_bib38","doi-asserted-by":"publisher","first-page":"428","DOI":"10.1162\/qss_a_00023","article-title":"OpenCitations, an infrastructure organization for open scholarship","volume":"1","author":"Peroni","year":"2020","journal-title":"Quantitative Science Studies"},{"key":"2021110817022675400_bib39","doi-asserted-by":"publisher","first-page":"2227","DOI":"10.18653\/v1\/N18-1202","article-title":"Deep contextualized word representations","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Peters","year":"2018"},{"issue":"5","key":"2021110817022675400_bib40","doi-asserted-by":"publisher","first-page":"424","DOI":"10.1016\/j.biopsych.2005.08.012","article-title":"Association between amygdala hyperactivity to harsh faces and severity of social anxiety in generalized social phobia","volume":"59","author":"Phan","year":"2006","journal-title":"Biological Psychiatry"},{"key":"2021110817022675400_bib41","doi-asserted-by":"publisher","DOI":"10.1101\/795310","article-title":"The future of OA: A large-scale analysis projecting Open Access publication and readership [Preprint]","author":"Piwowar","year":"2019","journal-title":"Scientific Communication and Education"},{"key":"2021110817022675400_bib42","volume-title":"Rich_citations","author":"PLOS","year":"2015"},{"issue":"2","key":"2021110817022675400_bib43","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1056\/NEJM198001103020221","article-title":"Addiction rare in patients treated with narcotics","volume":"302","author":"Porter","year":"1980","journal-title":"New England Journal of Medicine"},{"key":"2021110817022675400_bib44","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1109\/JCDL.2019.00055","article-title":"ACT: An annotation platform for citation typing at scale","volume-title":"2019 ACM\/IEEE Joint Conference on Digital Libraries (JCDL)","author":"Pride","year":"2019"},{"issue":"3","key":"2021110817022675400_bib45","doi-asserted-by":"publisher","first-page":"2877","DOI":"10.1007\/s11192-020-03631-1","article-title":"Continued post-retraction citation of a fraudulent clinical trial report, 11 years after it was retracted for falsifying data","volume":"125","author":"Schneider","year":"2020","journal-title":"Scientometrics"},{"key":"2021110817022675400_bib46","doi-asserted-by":"publisher","first-page":"1027","DOI":"10.1001\/archpsyc.59.11.1027","article-title":"Increased amygdala activation to angry and contemptuous faces in generalized social phobia","volume":"59","author":"Stein","year":"2002","journal-title":"Archives of General Psychiatry"},{"issue":"2","key":"2021110817022675400_bib47","doi-asserted-by":"publisher","first-page":"318","DOI":"10.1176\/ajp.2007.164.2.318","article-title":"Increased amygdala and insula activation during emotion processing in anxiety-prone subjects","volume":"164","author":"Stein","year":"2007","journal-title":"American Journal of Psychiatry"},{"issue":"11","key":"2021110817022675400_bib48","doi-asserted-by":"publisher","first-page":"e1915552","DOI":"10.1001\/jamanetworkopen.2019.15552","article-title":"Assessment of citations of the retracted article by Wakefield et al with fraudulent claims of an association between vaccination and autism","volume":"2","author":"Suelzer","year":"2019","journal-title":"JAMA Network Open"},{"key":"2021110817022675400_bib49","doi-asserted-by":"publisher","first-page":"103","DOI":"10.3115\/1610075.1610091","article-title":"Automatic classification of citation function","volume-title":"Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing","author":"Teufel","year":"2006"},{"key":"2021110817022675400_bib50","doi-asserted-by":"publisher","DOI":"10.1145\/3197026.3197048","article-title":"Machine learning vs. rules and out-of-the-box vs. retrained: An evaluation of open-source bibliographic reference and citation parsers","author":"Tkaczyk","year":"2018","journal-title":"ArXiv:1802.01168 [Cs]"},{"issue":"9","key":"2021110817022675400_bib51","doi-asserted-by":"publisher","first-page":"1031","DOI":"10.1091\/mbc.E17-10-0577","article-title":"Quantitative proteomic and phosphoproteomic comparison of human colon cancer DLD-1 cells differing in ploidy and chromosome stability","volume":"29","author":"Vigan\u00f3","year":"2018","journal-title":"Molecular Biology of the Cell"},{"issue":"1","key":"2021110817022675400_bib52","doi-asserted-by":"publisher","first-page":"6193","DOI":"10.1038\/s41598-018-24571-0","article-title":"SciRide Finder: A citation-based paradigm in biomedical literature search","volume":"8","author":"Volanakis","year":"2018","journal-title":"Scientific Reports"},{"issue":"9103","key":"2021110817022675400_bib53","doi-asserted-by":"publisher","first-page":"637","DOI":"10.1016\/S0140-6736(97)11096-0","article-title":"RETRACTED: Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children","volume":"351","author":"Wakefield","year":"1998","journal-title":"The Lancet"},{"key":"2021110817022675400_bib54","article-title":"CORD-19: The COVID-19 Open Research Dataset","author":"Wang","year":"2020","journal-title":"ArXiv:2004.10706 [Cs]"},{"key":"2021110817022675400_bib55","first-page":"181","volume-title":"The STM Report: An overview of scientific and scholarly journal publishing","author":"Ware","year":"2015"},{"issue":"2","key":"2021110817022675400_bib56","doi-asserted-by":"publisher","first-page":"664","DOI":"10.1162\/qss_a_00040","article-title":"The relationship between journal citation impact and citation sentiment: A study of 32 million citances in PubMed Central","volume":"1","author":"Yan","year":"2020","journal-title":"Quantitative Science Studies"},{"issue":"3","key":"2021110817022675400_bib57","doi-asserted-by":"publisher","first-page":"1805","DOI":"10.1007\/s10462-017-9597-8","article-title":"A survey on sentiment analysis of scientific citations","volume":"52","author":"Yousif","year":"2019","journal-title":"Artificial Intelligence Review"}],"container-title":["Quantitative Science Studies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/2\/3\/882\/1970740\/qss_a_00146.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/2\/3\/882\/1970740\/qss_a_00146.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,11,8]],"date-time":"2021-11-08T12:02:48Z","timestamp":1636372968000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/qss\/article\/2\/3\/882\/102990\/scite-A-smart-citation-index-that-displays-the"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":57,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,11,5]]}},"URL":"https:\/\/doi.org\/10.1162\/qss_a_00146","relation":{"has-review":[{"id-type":"doi","id":"10.1162\/QSS_A_00146\/v1\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00146\/v1\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00146\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00146\/v2\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00146\/v3\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00146\/v2\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00146\/v2\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00146\/v1\/review3","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00146\/v3\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00146\/v1\/decision1","asserted-by":"object"}],"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.03.15.435418","asserted-by":"object"}]},"ISSN":["2641-3337"],"issn-type":[{"value":"2641-3337","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}