{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,19]],"date-time":"2026-04-19T07:06:09Z","timestamp":1776582369355,"version":"3.51.2"},"reference-count":58,"publisher":"MIT Press","issue":"3","content-domain":{"domain":["www.mitpressjournals.org"],"crossmark-restriction":true},"short-container-title":["Quantitative Science Studies"],"published-print":{"date-parts":[[2020,8]]},"abstract":"<jats:p>Recently, Wu, Wang, and Evans (2019) proposed a new family of indicators, which measure whether a scientific publication is disruptive to a field or tradition of research. Such disruptive influences are characterized by citations to a focal paper, but not its cited references. In this study, we are interested in the question of convergent validity. We used external criteria of newness to examine convergent validity: In the postpublication peer review system of F1000Prime, experts assess papers whether the reported research fulfills these criteria (e.g., reports new findings). This study is based on 120,179 papers from F1000Prime published between 2000 and 2016. In the first part of the study we discuss the indicators. Based on the insights from the discussion, we propose alternate variants of disruption indicators. In the second part, we investigate the convergent validity of the indicators and the (possibly) improved variants. Although the results of a factor analysis show that the different variants measure similar dimensions, the results of regression analyses reveal that one variant ( DI<jats:sub>5<\/jats:sub>) performs slightly better than the others.<\/jats:p>","DOI":"10.1162\/qss_a_00068","type":"journal-article","created":{"date-parts":[[2020,5,28]],"date-time":"2020-05-28T19:05:22Z","timestamp":1590692722000},"page":"1242-1259","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":85,"title":["Are disruption index indicators convergently valid? The comparison of several indicator variants with assessments by peers"],"prefix":"10.1162","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0810-7091","authenticated-orcid":true,"given":"Lutz","family":"Bornmann","sequence":"first","affiliation":[{"name":"Division for Science and Innovation Studies, Administrative Headquarters of the Max Planck Society, Hofgartenstr. 8, 80539 Munich, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7722-756X","authenticated-orcid":true,"given":"Sitaram","family":"Devarakonda","sequence":"additional","affiliation":[{"name":"NET ESolutions Corporation, 8180 Greensboro Dr, McLean, VA 22102, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8765-9331","authenticated-orcid":true,"given":"Alexander","family":"Tekles","sequence":"additional","affiliation":[{"name":"Division for Science and Innovation Studies, Administrative Headquarters of the Max Planck Society, Hofgartenstr. 8, 80539 Munich, Germany"},{"name":"Ludwig-Maximilians-Universit\u00e4t Munich, Department of Sociology, Konradstr. 6, 80801 Munich, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2127-1892","authenticated-orcid":true,"given":"George","family":"Chacko","sequence":"additional","affiliation":[{"name":"NET ESolutions Corporation, 8180 Greensboro Dr, McLean, VA 22102, USA"}]}],"member":"281","reference":[{"key":"bib1","volume-title":"Practical multivariate analysis","author":"Afifi A.","year":"2012","edition":"5"},{"issue":"4","key":"bib2","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1038\/nn0405-397","volume":"8","author":"Anon","year":"2005","journal-title":"Nature Neuroscience"},{"key":"bib3","doi-asserted-by":"crossref","first-page":"330","DOI":"10.1038\/d41586-019-00350-3","volume":"566","author":"Azoulay P.","year":"2019","journal-title":"Nature"},{"key":"bib4","volume-title":"Psychological statistics and psychometrics using stata","author":"Baldwin S.","year":"2019"},{"key":"bib5","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1002\/aris.2011.1440450112","volume":"45","author":"Bornmann L.","year":"2011","journal-title":"Annual Review of Information Science and Technology"},{"issue":"12","key":"bib6","doi-asserted-by":"crossref","first-page":"2415","DOI":"10.1002\/asi.23334","volume":"66","author":"Bornmann L.","year":"2015","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"1","key":"bib7","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1108\/00220410810844150","volume":"64","author":"Bornmann L.","year":"2008","journal-title":"Journal of Documentation"},{"issue":"2","key":"bib8","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1016\/j.joi.2012.12.003","volume":"7","author":"Bornmann L.","year":"2013","journal-title":"Journal of Informetrics"},{"issue":"1","key":"bib9","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1016\/j.joi.2012.10.001","volume":"7","author":"Bornmann L.","year":"2013","journal-title":"Journal of Informetrics"},{"issue":"2","key":"bib10","doi-asserted-by":"crossref","first-page":"e280207","DOI":"10.3145\/epi.2019.mar.07","volume":"28","author":"Bornmann L.","year":"2019","journal-title":"El profesional de la informaci\u00f3n"},{"issue":"4","key":"bib11","doi-asserted-by":"crossref","first-page":"100979","DOI":"10.1016\/j.joi.2019.100979","volume":"13","author":"Bornmann L.","year":"2019","journal-title":"Journal of Informetrics"},{"issue":"10","key":"bib12","doi-asserted-by":"crossref","first-page":"2765","DOI":"10.1287\/mnsc.2015.2285","volume":"62","author":"Boudreau K. J.","year":"2016","journal-title":"Management Science"},{"key":"bib13","first-page":"64","volume-title":"Proceedings of the Science and Technology Indicators Conference 2014 Leiden: \u201cContext Counts: Pathways to Master Big and Little Data\u201d","author":"Boyack K.","year":"2014"},{"key":"bib14","doi-asserted-by":"publisher","DOI":"10.1162\/qss_a_00007"},{"key":"bib16","volume-title":"Novelty in science Proceedings of the Science, Technology, & Innovation Indicators Conference Open indicators: Innovation, participation and actor-based STI indicators","author":"Carayol N.","year":"2017"},{"issue":"2","key":"bib17","first-page":"e00158","volume":"7","author":"Casadevall A.","year":"2016","journal-title":"mBio"},{"key":"bib18","volume-title":"Statistical power analysis for the behavioral sciences","author":"Cohen J.","year":"1988","edition":"2"},{"key":"bib19","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1007\/978-3-319-10377-8_9","volume-title":"Measuring scholarly impact","author":"Deschacht N.","year":"2014"},{"issue":"12","key":"bib20","doi-asserted-by":"crossref","first-page":"3008","DOI":"10.1002\/asi.23548","volume":"67","author":"Du J.","year":"2016","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"2","key":"bib21","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1207\/S15326934CRJ1402_2","volume":"14","author":"Estes Z.","year":"2002","journal-title":"Creativity Research Journal"},{"issue":"5","key":"bib22","doi-asserted-by":"crossref","first-page":"875","DOI":"10.1177\/0003122415601618","volume":"80","author":"Foster J. G.","year":"2015","journal-title":"American Sociological Review"},{"issue":"8","key":"bib23","doi-asserted-by":"crossref","first-page":"1563","DOI":"10.1111\/j.1420-9101.2009.01775.x","volume":"22","author":"Frank S. A.","year":"2009","journal-title":"Journal of Evolutionary Biology"},{"issue":"3","key":"bib24","doi-asserted-by":"crossref","first-page":"791","DOI":"10.1287\/mnsc.2015.2366","volume":"63","author":"Funk R. J.","year":"2017","journal-title":"Management Science"},{"issue":"3","key":"bib25","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1016\/j.ijnurstu.2013.10.005","volume":"51","author":"Gaskin C. J.","year":"2014","journal-title":"International Journal of Nursing Studies"},{"issue":"5","key":"bib26","doi-asserted-by":"crossref","first-page":"646","DOI":"10.1016\/j.cell.2011.02.013","volume":"144","author":"Hanahan D.","year":"2011","journal-title":"Cell"},{"key":"bib27","doi-asserted-by":"crossref","first-page":"1539","DOI":"10.1061\/taceat.0002563","volume":"77","author":"Hazen A.","year":"1914","journal-title":"Transactions of American Society of Civil Engineers"},{"key":"bib28","first-page":"1","volume-title":"Creativity and leadership in science, technology, and innovation","author":"Hemlin S.","year":"2013"},{"issue":"7548","key":"bib29","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1038\/520429a","volume":"520","author":"Hicks D.","year":"2015","journal-title":"Nature"},{"key":"bib30","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9781139236065","volume-title":"Modelling count data","author":"Hilbe J. M.","year":"2014"},{"issue":"5","key":"bib31","first-page":"277","volume":"60","author":"Jemal A.","year":"2010","journal-title":"CA: A Cancer Journal for Clinicians"},{"issue":"8","key":"bib33","doi-asserted-by":"crossref","first-page":"938","DOI":"10.1038\/ni.3480","volume":"17","author":"Kincaid E. Z.","year":"2016","journal-title":"Nature Immunology"},{"issue":"7419","key":"bib35","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1038\/nature11417","volume":"490","author":"Kourtis N.","year":"2012","journal-title":"Nature"},{"issue":"12","key":"bib36","doi-asserted-by":"crossref","first-page":"1524","DOI":"10.1097\/00004583-200312000-00022","volume":"42","author":"Kraemer H. C.","year":"2003","journal-title":"Journal of the American Academy of Child and Adolescent Psychiatry"},{"key":"bib37","volume-title":"The structure of scientific revolutions","author":"Kuhn T. S.","year":"1962"},{"issue":"3","key":"bib38","doi-asserted-by":"crossref","first-page":"555","DOI":"10.1016\/j.shpsa.2007.06.011","volume":"38","author":"Kuukkanen J.-M.","year":"2007","journal-title":"Studies In History and Philosophy of Science Part A"},{"issue":"3","key":"bib39","doi-asserted-by":"crossref","first-page":"684","DOI":"10.1016\/j.respol.2014.10.007","volume":"44","author":"Lee Y.-N.","year":"2015","journal-title":"Research Policy"},{"issue":"7032","key":"bib40","doi-asserted-by":"crossref","first-page":"505","DOI":"10.1038\/nature03380","volume":"434","author":"Lolle S. J.","year":"2005","journal-title":"Nature"},{"key":"bib41","volume-title":"Regression models for categorical dependent variables using Stata","author":"Long J. S.","year":"2014","edition":"3"},{"key":"bib42","first-page":"212","volume-title":"Proceedings of the Science and Technology Indicators Conference 2018 Leiden, \u201cScience, technology and innovation indicators in transition\u201d","author":"Mairesse J.","year":"2018"},{"issue":"2","key":"bib43","doi-asserted-by":"crossref","first-page":"1397","DOI":"10.1007\/s11192-016-2111-2","volume":"109","author":"Marx W.","year":"2016","journal-title":"Scientometrics"},{"issue":"9","key":"bib44","first-page":"1753","volume":"46","author":"McEniery C. M.","year":"2005","journal-title":"The Anglo-Cardiff Collaborative Trial (ACCT)"},{"issue":"2","key":"bib45","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1007\/s11192-013-0993-9","volume":"97","author":"Mohammadi E.","year":"2013","journal-title":"Scientometrics"},{"issue":"43","key":"bib46","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1016\/S0968-8080(14)43766-2","volume":"22","author":"Mohan V.","year":"2014","journal-title":"Reproductive Health Matters"},{"issue":"2","key":"bib47","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1007\/s11192-018-2796-5","volume":"116","author":"Petrovich E.","year":"2018","journal-title":"Scientometrics"},{"key":"bib48","first-page":"287","volume-title":"Creativity and leadership in science, technology, and innovation","author":"Puccio G. J.","year":"2013"},{"issue":"8","key":"bib49","doi-asserted-by":"crossref","DOI":"10.1093\/femsle\/fny059","volume":"365","author":"Rowlands I.","year":"2018","journal-title":"FEMS Microbiology Letters"},{"issue":"1","key":"bib50","first-page":"11","volume":"63","author":"Siegel R.","year":"2013","journal-title":"CA: A Cancer Journal for Clinicians"},{"key":"bib51","volume-title":"Stata statistical software: release 15","author":"StataCorp","year":"2017"},{"issue":"7651","key":"bib52","doi-asserted-by":"crossref","first-page":"411","DOI":"10.1038\/544411a","volume":"544","author":"Stephan P.","year":"2017","journal-title":"Nature"},{"issue":"3","key":"bib53","doi-asserted-by":"crossref","first-page":"906","DOI":"10.1016\/j.joi.2018.07.005","volume":"12","author":"Tahamtan I.","year":"2018","journal-title":"Journal of Informetrics"},{"issue":"3","key":"bib54","doi-asserted-by":"crossref","first-page":"1635","DOI":"10.1007\/s11192-019-03243-4","volume":"121","author":"Tahamtan I.","year":"2019","journal-title":"Scientometrics"},{"issue":"6157","key":"bib55","doi-asserted-by":"crossref","first-page":"468","DOI":"10.1126\/science.1240474","volume":"342","author":"Uzzi B.","year":"2013","journal-title":"Science"},{"issue":"2","key":"bib56","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1177\/1536867X1201200208","volume":"12","author":"Verardi V.","year":"2012","journal-title":"Stata Journal"},{"issue":"5","key":"bib57","doi-asserted-by":"crossref","first-page":"1260","DOI":"10.1016\/j.respol.2019.01.002","volume":"48","author":"Wagner C. S.","year":"2019","journal-title":"Research Policy"},{"issue":"3","key":"bib58","doi-asserted-by":"crossref","first-page":"433","DOI":"10.1002\/asi.23040","volume":"65","author":"Waltman L.","year":"2014","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"6","key":"bib59","doi-asserted-by":"crossref","first-page":"1070","DOI":"10.1016\/j.respol.2018.03.014","volume":"47","author":"Wang J.","year":"2018","journal-title":"Research Policy"},{"issue":"8","key":"bib60","doi-asserted-by":"crossref","first-page":"1416","DOI":"10.1016\/j.respol.2017.06.006","volume":"46","author":"Wang J.","year":"2017","journal-title":"Research Policy"},{"key":"bib61","doi-asserted-by":"crossref","first-page":"378","DOI":"10.1038\/s41586-019-0941-9","volume":"566","author":"Wu L.","year":"2019","journal-title":"Nature"}],"container-title":["Quantitative Science Studies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/qss_a_00068","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,1]],"date-time":"2023-10-01T16:51:32Z","timestamp":1696179092000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/qss\/article\/1\/3\/1242-1259\/96102"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8]]},"references-count":58,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2020,8]]}},"alternative-id":["10.1162\/qss_a_00068"],"URL":"https:\/\/doi.org\/10.1162\/qss_a_00068","relation":{},"ISSN":["2641-3337"],"issn-type":[{"value":"2641-3337","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8]]},"assertion":[{"value":"2019-11-20","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-05-01","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-09-04","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}