{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T04:00:52Z","timestamp":1772164852934,"version":"3.50.1"},"reference-count":50,"publisher":"MIT Press","issue":"1","license":[{"start":{"date-parts":[[2021,12,13]],"date-time":"2021-12-13T00:00:00Z","timestamp":1639353600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["314727790"],"award-info":[{"award-number":["314727790"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,4,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>With this work, we present a publicly available data set of the history of all the references (more than 55 million) ever used in the English Wikipedia until June 2019. We have applied a new method for identifying and monitoring references in Wikipedia, so that for each reference we can provide data about associated actions: creation, modifications, deletions, and reinsertions. The high accuracy of this method and the resulting data set was confirmed via a comprehensive crowdworker labeling campaign. We use the data set to study the temporal evolution of Wikipedia references as well as users\u2019 editing behavior. We find evidence of a mostly productive and continuous effort to improve the quality of references: There is a persistent increase of reference and document identifiers (DOI, PubMedID, PMC, ISBN, ISSN, ArXiv ID) and most of the reference curation work is done by registered humans (not bots or anonymous editors). We conclude that the evolution of Wikipedia references, including the dynamics of the community processes that tend to them, should be leveraged in the design of relevance indexes for altmetrics, and our data set can be pivotal for such an effort.<\/jats:p>","DOI":"10.1162\/qss_a_00171","type":"journal-article","created":{"date-parts":[[2021,12,13]],"date-time":"2021-12-13T10:36:38Z","timestamp":1639391798000},"page":"147-173","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":5,"title":["\u201cI updated the &amp;lt;ref&amp;gt;\u201d: The evolution of references in the English Wikipedia and the implications for altmetrics"],"prefix":"10.1162","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4693-9668","authenticated-orcid":true,"given":"Olga","family":"Zagovora","sequence":"first","affiliation":[{"name":"GESIS-Leibniz Institute for the Social Sciences"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9870-5505","authenticated-orcid":true,"given":"Roberto","family":"Ulloa","sequence":"additional","affiliation":[{"name":"GESIS-Leibniz Institute for the Social Sciences"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3799-1146","authenticated-orcid":true,"given":"Katrin","family":"Weller","sequence":"additional","affiliation":[{"name":"GESIS-Leibniz Institute for the Social Sciences"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0727-1319","authenticated-orcid":true,"given":"Fabian","family":"Fl\u00f6ck","sequence":"additional","affiliation":[{"name":"GESIS-Leibniz Institute for the Social Sciences"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2022,4,12]]},"reference":[{"issue":"1","key":"2023111415174826700_bib1","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1080\/13614533.2012.740439","article-title":"Exploring the cautionary attitude toward Wikipedia in higher education: Implications for higher education institutions","volume":"19","author":"Bayliss","year":"2013","journal-title":"New Review of Academic Librarianship"},{"key":"2023111415174826700_bib2","doi-asserted-by":"publisher","first-page":"g1585","DOI":"10.1136\/bmj.g1585","article-title":"References that anyone can edit: Review of Wikipedia citations in peer reviewed health science literature","volume":"348","author":"Bould","year":"2014","journal-title":"British Medical Journal"},{"key":"2023111415174826700_bib3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2462932.2462943","article-title":"{{Citation needed}}: The dynamics of referencing in Wikipedia","volume-title":"Proceedings of the Eighth Annual International Symposium on Wikis and Open Collaboration","author":"Chen","year":"2012"},{"issue":"12","key":"2023111415174826700_bib4","doi-asserted-by":"publisher","first-page":"152","DOI":"10.1145\/1101779.1101804","article-title":"Wikipedia risks","volume":"48","author":"Denning","year":"2005","journal-title":"Communications of the ACM"},{"issue":"3","key":"2023111415174826700_bib5","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1108\/10650741011054474","article-title":"Academics and Wikipedia: Reframing Web 2.0+ as a disruptor of traditional academic power-knowledge arrangements","volume":"27","author":"Eijkman","year":"2010","journal-title":"Campus-Wide Information Systems"},{"key":"2023111415174826700_bib6","doi-asserted-by":"publisher","first-page":"843","DOI":"10.1145\/2566486.2568026","article-title":"WikiWho: Precise and efficient attribution of authorship of revisioned content","volume-title":"Proceedings of the 23rd International Conference on World Wide Web","author":"Fl\u00f6ck","year":"2014"},{"key":"2023111415174826700_bib7","doi-asserted-by":"crossref","DOI":"10.1609\/icwsm.v11i1.14860","article-title":"TokTrack: A complete token provenance and change tracking dataset for the English Wikipedia","volume-title":"Eleventh International AAAI Conference on Web and Social Media","author":"Fl\u00f6ck","year":"2017"},{"key":"2023111415174826700_bib8","article-title":"Wikipedia comes of age","volume":"57","author":"Grathwohl","year":"2011","journal-title":"Chronicle of Higher Education"},{"key":"2023111415174826700_bib9","doi-asserted-by":"publisher","DOI":"10.6084\/m9.figshare.1299540","article-title":"Citations with identifiers in Wikipedia","author":"Halfaker","year":"2019","journal-title":"figshare"},{"issue":"1","key":"2023111415174826700_bib10","doi-asserted-by":"publisher","first-page":"413","DOI":"10.1007\/s11192-016-1910-9","article-title":"Grand challenges in altmetrics: Heterogeneity, data quality and dependencies","volume":"108","author":"Haustein","year":"2016","journal-title":"Scientometrics"},{"issue":"1","key":"2023111415174826700_bib11","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1108\/00907320810851998","article-title":"Comparison of Wikipedia and other encyclopedias for accuracy, breadth, and depth in historical articles","volume":"36","author":"Holman Rector","year":"2008","journal-title":"Reference Services Review"},{"key":"2023111415174826700_bib12","volume-title":"Altmetrics for information professionals: Past, present and future","author":"Holmberg","year":"2015"},{"issue":"3","key":"2023111415174826700_bib13","article-title":"Where does the information come from? Information source use patterns in Wikipedia","volume":"15","author":"Huvila","year":"2010","journal-title":"Information Research"},{"key":"2023111415174826700_bib14","first-page":"1339","article-title":"Exploiting social networks of Twitter in altmetrics big data","volume-title":"STI 2018 Conference Proceedings","author":"Imran","year":"2018"},{"key":"2023111415174826700_bib15","doi-asserted-by":"publisher","DOI":"10.1145\/3442442.3452337","article-title":"References in Wikipedia: The editors\u2019 perspective","volume-title":"8th Wiki Workshop at The Web Conference","author":"Kaffee","year":"2021"},{"key":"2023111415174826700_bib17","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1145\/1240624.1240698","article-title":"He says, she says: Conflict and coordination in Wikipedia","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Kittur","year":"2007"},{"issue":"3","key":"2023111415174826700_bib18","doi-asserted-by":"publisher","first-page":"762","DOI":"10.1002\/asi.23694","article-title":"Are Wikipedia citations important evidence of the impact of scholarly articles and books?","volume":"68","author":"Kousha","year":"2017","journal-title":"Journal of the Association for Information Science and Technology"},{"key":"2023111415174826700_bib19","doi-asserted-by":"publisher","first-page":"561","DOI":"10.1007\/978-3-319-67642-5_47","article-title":"Analysis of references across Wikipedia languages","volume-title":"Information and Software Technologies","author":"Lewoniewski","year":"2017"},{"issue":"5","key":"2023111415174826700_bib20","doi-asserted-by":"publisher","first-page":"263","DOI":"10.3390\/info11050263","article-title":"Modeling popularity and reliability of sources in multilingual Wikipedia","volume":"11","author":"Lewoniewski","year":"2020","journal-title":"Information"},{"issue":"2","key":"2023111415174826700_bib21","doi-asserted-by":"publisher","first-page":"20","DOI":"10.3789\/isqv25no2.2013.04","article-title":"Altmetrics in evolution: Defining and redefining the ontology of article-level metrics","volume":"25","author":"Lin","year":"2013","journal-title":"Information Standards Quarterly"},{"key":"2023111415174826700_bib22","doi-asserted-by":"publisher","first-page":"23","DOI":"10.6084\/m9.figshare.1048991.v3","article-title":"An analysis of Wikipedia references across PLOS publications","volume-title":"Expanding Impacts and Metrics, An ACM Web Science Conference 2014 Workshop","author":"Lin","year":"2014"},{"issue":"4","key":"2023111415174826700_bib23","doi-asserted-by":"publisher","first-page":"715","DOI":"10.1002\/asi.21304","article-title":"Improving Wikipedia\u2019s credibility: References and citations in a sample of history articles","volume":"61","author":"Luyt","year":"2010","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"2","key":"2023111415174826700_bib24","doi-asserted-by":"publisher","first-page":"219","DOI":"10.1002\/asi.23172","article-title":"\u201cThe sum of all human knowledge\u201d: A systematic review of scholarly research on the content of Wikipedia","volume":"66","author":"Mesgari","year":"2015","journal-title":"Journal of the Association for Information Science and Technology"},{"key":"2023111415174826700_bib25","doi-asserted-by":"publisher","first-page":"74:1","DOI":"10.1145\/3359176","article-title":"Collaboration drives individual productivity","volume-title":"Proceedings of the ACM on Human-Computer Interaction","author":"Muri\u0107","year":"2019"},{"issue":"8","key":"2023111415174826700_bib26","doi-asserted-by":"publisher","DOI":"10.5210\/fm.v12i8.1997","article-title":"Scientific citations in Wikipedia","volume":"12","author":"Nielsen","year":"2007","journal-title":"First Monday"},{"key":"2023111415174826700_bib27","article-title":"Clustering of scientific citations in Wikipedia","author":"Nielsen","year":"2008","journal-title":"Wikimania 2008"},{"issue":"12","key":"2023111415174826700_bib28","doi-asserted-by":"publisher","first-page":"2381","DOI":"10.1002\/asi.23162","article-title":"Wikipedia in the eyes of its beholders: A systematic review of scholarly research on Wikipedia readers and readership","volume":"65","author":"Okoli","year":"2014","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"10","key":"2023111415174826700_bib29","doi-asserted-by":"publisher","first-page":"2550","DOI":"10.1002\/asi.23590","article-title":"Evaluation of the citation matching algorithms of CWTS and iFQ in comparison to the Web of science","volume":"67","author":"Olensky","year":"2016","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"3","key":"2023111415174826700_bib30","doi-asserted-by":"publisher","first-page":"2123","DOI":"10.1007\/s11192-018-2838-z","article-title":"Reliability and accuracy of altmetric providers: A comparison among Altmetric.com, PlumX, and Crossref Event Data","volume":"116","author":"Ortega","year":"2018","journal-title":"Scientometrics"},{"key":"2023111415174826700_bib31","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1145\/1531674.1531682","article-title":"Wikipedians are born, not made: A study of power editors on Wikipedia","volume-title":"Proceedings of the ACM 2009 International Conference on Supporting Group Work \u2013 GROUP \u201909","author":"Panciera","year":"2009"},{"key":"2023111415174826700_bib32","doi-asserted-by":"publisher","first-page":"2365","DOI":"10.1145\/3366423.3380300","article-title":"Quantifying engagement with citations on Wikipedia","volume-title":"Proceedings of The Web Conference 2020 (WWW \u201920)","author":"Piccardi","year":"2020"},{"key":"2023111415174826700_bib33","doi-asserted-by":"publisher","first-page":"455","DOI":"10.1007\/s11192-017-2474-z","article-title":"Methodological issues in measuring citations in Wikipedia: A case study in library and information science","volume":"113","author":"Pooladian","year":"2017","journal-title":"Scientometrics"},{"key":"2023111415174826700_bib34","volume-title":"Altmetrics: A manifesto","author":"Priem","year":"2010"},{"key":"2023111415174826700_bib35","volume-title":"Research: Characterizing Wikipedia citation usage","author":"Redi","year":"2018"},{"key":"2023111415174826700_bib36","doi-asserted-by":"publisher","DOI":"10.6084\/m9.figshare.6819710.v1","volume-title":"Accessibility and topics of citations with identifiers in Wikipedia","author":"Redi","year":"2018"},{"issue":"8","key":"2023111415174826700_bib37","doi-asserted-by":"publisher","first-page":"e0183551","DOI":"10.1371\/journal.pone.0183551","article-title":"The unbearable emptiness of tweeting\u2014About journal articles","volume":"12","author":"Robinson-Garcia","year":"2017","journal-title":"PLOS ONE"},{"key":"2023111415174826700_bib38","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1016\/0377-0427(87)90125-7","article-title":"Silhouettes: A graphical aid to the interpretation and validation of cluster analysis","volume":"20","author":"Rousseeuw","year":"1987","journal-title":"Journal of Computational and Applied Mathematics"},{"issue":"S1","key":"2023111415174826700_bib39","doi-asserted-by":"publisher","first-page":"399","DOI":"10.1093\/poq\/nfab018","article-title":"A total error framework for digital traces of human behavior on online platforms","volume":"85","author":"Sen","year":"2021","journal-title":"Public Opinion Quarterly"},{"issue":"2\u20133","key":"2023111415174826700_bib40","doi-asserted-by":"publisher","first-page":"98","DOI":"10.1515\/iwp-2019-2006","article-title":"Retractions from altmetric and bibliometric perspectives","volume":"70","author":"Shema","year":"2019","journal-title":"Information \u2013 Wissenschaft & Praxis"},{"key":"2023111415174826700_bib41","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1145\/2467696.2467746","article-title":"A comparative study of academic and Wikipedia ranking","volume-title":"Proceedings of the 13th ACM\/IEEE-CS joint conference on Digital libraries","author":"Shuai","year":"2013"},{"issue":"1","key":"2023111415174826700_bib42","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1162\/qss_a_00105","article-title":"Wikipedia citations: A comprehensive dataset of citations with identifiers extracted from English Wikipedia","volume":"2","author":"Singh","year":"2021","journal-title":"Quantitative Science Studies"},{"issue":"9","key":"2023111415174826700_bib43","doi-asserted-by":"publisher","first-page":"2037","DOI":"10.1002\/asi.23833","article-title":"Scholarly use of social media and altmetrics: A review of the literature","volume":"68","author":"Sugimoto","year":"2017","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"2","key":"2023111415174826700_bib44","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1109\/TSMC.1981.4308636","article-title":"Methods for visual understanding of hierarchical system structures","volume":"11","author":"Sugiyama","year":"1981","journal-title":"IEEE Transactions on Systems, Man, and Cybernetics"},{"issue":"9","key":"2023111415174826700_bib45","doi-asserted-by":"publisher","first-page":"2116","DOI":"10.1002\/asi.23687","article-title":"Amplifying the impact of open access: Wikipedia and the diffusion of science","volume":"68","author":"Teplitskiy","year":"2017","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"6","key":"2023111415174826700_bib46","doi-asserted-by":"publisher","first-page":"893","DOI":"10.3145\/epi.2016.nov.06","article-title":"Does astronomy research become too dated for the public? Wikipedia citations to astronomy and astrophysics journal articles 1996\u20132014","volume":"25","author":"Thelwall","year":"2016","journal-title":"El Profesional de La Informaci\u00f3n"},{"issue":"4","key":"2023111415174826700_bib47","doi-asserted-by":"publisher","first-page":"20:1","DOI":"10.1145\/1852102.1852106","article-title":"A similarity measure for indefinite rankings","volume":"28","author":"Webber","year":"2010","journal-title":"ACM Transactions on Information Systems (TOIS)"},{"key":"2023111415174826700_bib48","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.3964990","volume-title":"Individual edit histories of all references in the English Wikipedia","author":"Zagovora","year":"2020"},{"issue":"5","key":"2023111415174826700_bib49","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0197326","article-title":"General discussion of data quality challenges in social media metrics: Extensive comparison of four major altmetric data aggregators","volume":"13","author":"Zahedi","year":"2018","journal-title":"PLOS ONE"},{"issue":"3","key":"2023111415174826700_bib50","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1145\/3180492","article-title":"Responsible research with crowds: Pay crowdworkers at least minimum wage","volume":"61","author":"Zaldivar","year":"2018","journal-title":"Communications of the ACM"},{"issue":"7","key":"2023111415174826700_bib51","doi-asserted-by":"publisher","DOI":"10.1093\/gigascience\/giy083","article-title":"Clustering trees: A visualization for evaluating clusterings at multiple resolutions","volume":"7","author":"Zappia","year":"2018","journal-title":"GigaScience"}],"container-title":["Quantitative Science Studies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/3\/1\/147\/2175824\/qss_a_00171.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/3\/1\/147\/2175824\/qss_a_00171.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,14]],"date-time":"2023-11-14T10:18:03Z","timestamp":1699957083000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/qss\/article\/3\/1\/147\/108660\/I-updated-the-lt-ref-gt-The-evolution-of"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":50,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,4,12]]}},"URL":"https:\/\/doi.org\/10.1162\/qss_a_00171","relation":{"has-review":[{"id-type":"doi","id":"10.1162\/QSS_A_00171\/v3\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00171\/v2\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00171\/v1\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00171\/v3\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00171\/v1\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00171\/v1\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00171\/v2\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00171\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00171\/v3\/decision1","asserted-by":"object"}]},"ISSN":["2641-3337"],"issn-type":[{"value":"2641-3337","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022]]},"published":{"date-parts":[[2022]]}}}