{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T17:50:36Z","timestamp":1789667436832,"version":"4.0.1"},"reference-count":68,"publisher":"MIT Press","issue":"2","license":[{"start":{"date-parts":[[2021,2,22]],"date-time":"2021-02-22T00:00:00Z","timestamp":1613952000000},"content-version":"vor","delay-in-days":52,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,7,15]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Open Science is an umbrella term that encompasses many recommendations for possible changes in research practices, management, and publishing with the objective to increase transparency and accessibility. This has become an important science policy issue that all disciplines should consider. Many Open Science recommendations may be valuable for the further development of research and publishing, but not all are relevant to all fields. This opinion paper considers the aspects of Open Science that are most relevant for scientometricians, discussing how they can be usefully applied.<\/jats:p>","DOI":"10.1162\/qss_e_00121","type":"journal-article","created":{"date-parts":[[2021,2,10]],"date-time":"2021-02-10T16:27:55Z","timestamp":1612974475000},"page":"438-453","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":11,"title":["Which aspects of the Open Science agenda are most relevant to scientometric research and publishing? An opinion paper"],"prefix":"10.1162","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0810-7091","authenticated-orcid":true,"given":"Lutz","family":"Bornmann","sequence":"first","affiliation":[{"name":"Science Policy and Strategy Department, Administrative Headquarters of the Max Planck Society, Hofgartenstr. 8, 80539 Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3129-0330","authenticated-orcid":true,"given":"Raf","family":"Guns","sequence":"additional","affiliation":[{"name":"Centre for R&D Monitoring, Faculty of Social Sciences, University of Antwerp, Middelheimlaan 1, 2020 Antwerpen, Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6065-205X","authenticated-orcid":true,"given":"Michael","family":"Thelwall","sequence":"additional","affiliation":[{"name":"Statistical Cybermetrics Research Group, School of Mathematics and Computer Science, University of Wolverhampton Wulfruna Street, Wolverhampton WV1 1LY, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4991-276X","authenticated-orcid":true,"given":"Dietmar","family":"Wolfram","sequence":"additional","affiliation":[{"name":"School of Information Studies, University of Wisconsin\u2014Milwaukee, P.O. Box 413, Milwaukee, WI, USA, 53201"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2021,7,15]]},"reference":[{"key":"2021072016155108400_bib1","doi-asserted-by":"crossref","DOI":"10.1038\/nature.2016.20504","article-title":"Why scientists must share their research code","author":"Baker","year":"2016","journal-title":"Nature"},{"key":"2021072016155108400_bib2","article-title":"Writing a reproducible paper in R markdown (SSRN scholarly paper no. ID 3175518)","author":"Bauer","year":"2018"},{"issue":"3","key":"2021072016155108400_bib3","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1007\/s11192-005-0281-4","article-title":"Does the h-index for ranking of scientists really work?","volume":"65","author":"Bornmann","year":"2005","journal-title":"Scientometrics"},{"issue":"3","key":"2021072016155108400_bib4","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1016\/j.joi.2009.10.004","article-title":"A meta-evaluation of scientific research proposals: Different ways of comparing rejected to awarded applications","volume":"4","author":"Bornmann","year":"2010","journal-title":"Journal of Informetrics"},{"issue":"2","key":"2021072016155108400_bib5","doi-asserted-by":"crossref","first-page":"105","DOI":"10.3152\/095820210X510089","article-title":"From black box to white box at open access journals: Predictive validity of manuscript reviewing and editorial decisions at Atmospheric Chemistry and Physics","volume":"19","author":"Bornmann","year":"2010","journal-title":"Research Evaluation"},{"issue":"11","key":"2021072016155108400_bib6","doi-asserted-by":"crossref","first-page":"2215","DOI":"10.1002\/asi.23329","article-title":"Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references","volume":"66","author":"Bornmann","year":"2015","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"12","key":"2021072016155108400_bib7","doi-asserted-by":"crossref","first-page":"2711","DOI":"10.1002\/asi.23451","article-title":"How well does a university perform in comparison with its peers? The use of odds, and odds ratios, for the comparison of institutional citation impact using the Leiden Rankings","volume":"66","author":"Bornmann","year":"2015","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"3","key":"2021072016155108400_bib8","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1016\/j.joi.2011.01.006","article-title":"A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants","volume":"5","author":"Bornmann","year":"2011","journal-title":"Journal of Informetrics"},{"key":"2021072016155108400_bib9","first-page":"61","article-title":"Is interactive open access publishing able to identify high-impact submissions? A study on the predictive validity of Atmospheric Chemistry and Physics by using percentile rank classes","volume":"62","author":"Bornmann","year":"2011","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"3","key":"2021072016155108400_bib10","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1016\/j.joi.2014.04.008","article-title":"What is the effect of country-specific characteristics on the research performance of scientific institutions? Using multi-level statistical models to rank and map universities and research-focused institutions worldwide","volume":"8","author":"Bornmann","year":"2014","journal-title":"Journal of Informetrics"},{"key":"2021072016155108400_bib11","article-title":"Improving the future of research communications and e-scholarship (Dagstuhl Perspectives Workshop 11331)","author":"Bourne","year":"2012"},{"issue":"12","key":"2021072016155108400_bib12","doi-asserted-by":"crossref","first-page":"997","DOI":"10.1037\/0003-066X.49.12.997","article-title":"The earth is round (p &lt; .05)","volume":"49","author":"Cohen","year":"1994","journal-title":"American Psychologist"},{"issue":"4","key":"2021072016155108400_bib13","doi-asserted-by":"crossref","first-page":"e0230416","DOI":"10.1371\/journal.pone.0230416","article-title":"The citation advantage of linking publications to research data","volume":"15","author":"Colavizza","year":"2020","journal-title":"PLOS ONE"},{"key":"2021072016155108400_bib14","doi-asserted-by":"crossref","DOI":"10.4324\/9781315708607","volume-title":"Introduction to the new statistics: Estimation, open science, and beyond","author":"Cumming","year":"2016"},{"issue":"1","key":"2021072016155108400_bib15","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1007\/s11192-006-0143-8","article-title":"An informetric model for the Hirsch-index","volume":"69","author":"Egghe","year":"2006","journal-title":"Scientometrics"},{"issue":"4","key":"2021072016155108400_bib17","doi-asserted-by":"crossref","first-page":"e123","DOI":"10.2196\/jmir.2012","article-title":"Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact","volume":"13","author":"Eysenbach","year":"2011","journal-title":"Journal of Medical Internet Research"},{"key":"2021072016155108400_bib18","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1007\/978-3-319-00026-8_2","article-title":"Open science: One term, five schools of thought","volume-title":"Opening science: The evolving guide on how the internet is changing research, collaboration and scholarly publishing","author":"Fecher","year":"2014"},{"issue":"3","key":"2021072016155108400_bib19","doi-asserted-by":"crossref","first-page":"859","DOI":"10.1016\/j.joi.2017.06.006","article-title":"What drives university research performance? An analysis using the CWTS Leiden Ranking data","volume":"11","author":"Frenken","year":"2017","journal-title":"Journal of Informetrics"},{"issue":"6379","key":"2021072016155108400_bib20","doi-asserted-by":"crossref","DOI":"10.1126\/science.aao0185","article-title":"Science of science","volume":"359","author":"Fortunato","year":"2018","journal-title":"Science"},{"issue":"4","key":"2021072016155108400_bib21","doi-asserted-by":"crossref","first-page":"520","DOI":"10.1108\/14684521011072963","article-title":"Open knowledge: Challenges and facts","volume":"34","author":"Garc\u00eda-Pe\u00f1alvo Francisco","year":"2010","journal-title":"Online Information Review"},{"key":"2021072016155108400_bib22","doi-asserted-by":"crossref","first-page":"3","DOI":"10.3102\/0013189X005010003","article-title":"Primary, secondary, and meta-analysis","volume":"5","author":"Glass","year":"1976","journal-title":"Educational Researcher"},{"key":"2021072016155108400_bib23","doi-asserted-by":"crossref","first-page":"1442","DOI":"10.12688\/gatesopenres.12958.2","article-title":"An open toolkit for tracking open science partnership implementation and impact","volume":"3","author":"Gold","year":"2019","journal-title":"Gates Open Research"},{"key":"2021072016155108400_bib24","volume-title":"Future of scholarly publishing and scholarly communication: Report of the expert group to the European Commission","author":"Gu\u00e9don","year":"2019"},{"key":"2021072016155108400_bib25","volume-title":"The improbability principle: Why coincidences, miracles, and rare events happen every day","author":"Hand","year":"2014"},{"issue":"3","key":"2021072016155108400_bib26","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1093\/reseval\/rvz015","article-title":"Can topic models be used in research evaluations? Reproducibility, validity, and reliability when compared with semantic maps","volume":"28","author":"Hecking","year":"2019","journal-title":"Research Evaluation"},{"key":"2021072016155108400_bib27","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1086\/286983","article-title":"Studies in the logic of explanation","volume":"15","author":"Hempel","year":"1948","journal-title":"Philosophy of Science"},{"key":"2021072016155108400_bib29","volume-title":"Foresight in science: Picking the winners","author":"Irvine","year":"1984"},{"key":"2021072016155108400_bib30","volume-title":"Does data sharing influence data reuse in biodiversity? A citation analysis","author":"Khan","year":"2019"},{"key":"2021072016155108400_bib31","volume-title":"More and more scientists are preregistering their studies. Should you?","author":"Kupferschmidt","year":"2018"},{"key":"2021072016155108400_bib32","volume-title":"Science in action","author":"Latour","year":"1987"},{"key":"2021072016155108400_bib33","volume-title":"Laboratory life: The social construction of scientific facts","author":"Latour","year":"1979"},{"issue":"5","key":"2021072016155108400_bib34","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1002\/asi.24130","article-title":"Statistical significance and effect sizes of differences among research universities at the level of nations and worldwide based on the Leiden rankings","volume":"70","author":"Leydesdorff","year":"2019","journal-title":"Journal of the Association for Information Science and Technology"},{"key":"2021072016155108400_bib36","volume-title":"Opium in science and society: Numbers","author":"Marewski","year":"2018"},{"issue":"5\u20136","key":"2021072016155108400_bib37","doi-asserted-by":"crossref","first-page":"A41","DOI":"10.1002\/andp.201400806","article-title":"The Shockley-Queisser paper\u2014A notable example of a scientific sleeping beauty","volume":"526","author":"Marx","year":"2014","journal-title":"Annalen der Physik"},{"key":"2021072016155108400_bib38","first-page":"115","article-title":"Science and technology in a democratic order","volume":"1","author":"Merton","year":"1942","journal-title":"Journal of Legal and Political Sociology"},{"key":"2021072016155108400_bib39","doi-asserted-by":"crossref","DOI":"10.1186\/2050-6511-15-55","article-title":"Open, single-blind, double-blind: Which peer review process do you prefer?","volume":"15","author":"Moylan","year":"2014","journal-title":"BMC Pharmacology & Toxicology"},{"issue":"1","key":"2021072016155108400_bib40","doi-asserted-by":"crossref","first-page":"132","DOI":"10.1002\/asi.22798","article-title":"Peer review in a changing world: An international study measuring the attitudes of researchers","volume":"64","author":"Mulligan","year":"2013","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"11","key":"2021072016155108400_bib41","doi-asserted-by":"crossref","first-page":"2600","DOI":"10.1073\/pnas.1708274114","article-title":"The preregistration revolution","volume":"115","author":"Nosek","year":"2018","journal-title":"Proceedings of the National Academy of Sciences of the United States of America"},{"issue":"6242","key":"2021072016155108400_bib42","doi-asserted-by":"crossref","first-page":"1422","DOI":"10.1126\/science.aab2374","article-title":"Promoting an open research culture","volume":"348","author":"Nosek","year":"2015","journal-title":"Science"},{"issue":"11","key":"2021072016155108400_bib43","doi-asserted-by":"crossref","first-page":"1346","DOI":"10.1002\/asi.24049","article-title":"Informal data citation for data sharing and reuse is more common than formal data citation in biomedical fields","volume":"69","author":"Park","year":"2018","journal-title":"Journal of the Association for Information Science and Technology"},{"key":"2021072016155108400_bib44","doi-asserted-by":"crossref","first-page":"e175","DOI":"10.7717\/peerj.175","article-title":"Data reuse and the open data citation advantage","volume":"1","author":"Piwowar","year":"2013","journal-title":"PeerJ"},{"key":"2021072016155108400_bib45","volume-title":"PLOS journals now open for published peer review","author":"PLOS","year":"2019"},{"key":"2021072016155108400_bib46","volume-title":"The poverty of historicism","author":"Popper","year":"1967"},{"issue":"12","key":"2021072016155108400_bib47","doi-asserted-by":"crossref","first-page":"2964","DOI":"10.1002\/asi.23529","article-title":"Analyzing data citation practices using the data citation index","volume":"67","author":"Robinson-Garc\u00eda","year":"2016","journal-title":"Journal of the Association for Information Science and Technology"},{"key":"2021072016155108400_bib80","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1002\/leap.1111","article-title":"Peer review: The experience and views of early career researchers","volume":"30","author":"Rodriguez-Bravo","year":"2017","journal-title":"Learned Publishing"},{"issue":"2","key":"2021072016155108400_bib48","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1007\/s11024-015-9274-5","article-title":"Accounting for impact? The journal impact factor and the making of biomedical research in the Netherlands","volume":"53","author":"Rushforth","year":"2015","journal-title":"Minerva"},{"issue":"969","key":"2021072016155108400_bib49","article-title":"Ten considerations for open peer review [version 1; peer review: 2 approved]","volume":"7","author":"Schmidt","year":"2018","journal-title":"F1000Research"},{"key":"2021072016155108400_bib50","doi-asserted-by":"crossref","DOI":"10.1002\/9781119170174.epcn519","article-title":"Open science","volume-title":"Stevens\u2019 handbook of experimental psychology and cognitive neuroscience","author":"Spellman","year":"2018"},{"key":"2021072016155108400_bib51","first-page":"249","article-title":"Literature dynamics: Studies on growth, diffusion, and epidemics","volume":"34","author":"Tabah","year":"1999","journal-title":"Annual Review of Information Science and Technology"},{"key":"2021072016155108400_bib52","doi-asserted-by":"crossref","unstructured":"Tennant, J., Agarwal, R., Ba\u017edari\u0107, K., Brassard, D., Crick, T., \u2026 Yarkoni, T. (2020). A tale of two \u201copens\u201d: Intersections between free and open source software and open scholarship. DOI: https:\/\/doi.org\/10.31235\/osf.io\/2kxq8 (accessed July 30, 2020).","DOI":"10.31235\/osf.io\/2kxq8"},{"issue":"8","key":"2021072016155108400_bib53","doi-asserted-by":"crossref","first-page":"e0134826","DOI":"10.1371\/journal.pone.0134826","article-title":"Changes in data sharing and data reuse practices and perceptions among scientists worldwide","volume":"10","author":"Tenopir","year":"2015","journal-title":"PLOS ONE"},{"issue":"1","key":"2021072016155108400_bib54","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1108\/AJIM-09-2016-0159","article-title":"Do journal data sharing mandates work? Life sciences evidence from Dryad","volume":"69","author":"Thelwall","year":"2017","journal-title":"Aslib Journal of Information Management"},{"issue":"2","key":"2021072016155108400_bib55","first-page":"730","article-title":"How common are explicit research questions in journal articles?","volume":"1","author":"Thelwall","year":"2020","journal-title":"Quantitative Science Studies"},{"issue":"2","key":"2021072016155108400_bib56","doi-asserted-by":"crossref","first-page":"e0229578","DOI":"10.1371\/journal.pone.0229578","article-title":"Is useful research data usually shared? An investigation of genome-wide association study summary statistics","volume":"15","author":"Thelwall","year":"2020","journal-title":"PLOS ONE"},{"issue":"2","key":"2021072016155108400_bib57","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1108\/14684521111128050","article-title":"The calculation of the single publication h index and related performance measures: A web application based on Google Scholar data","volume":"35","author":"Thor","year":"2011","journal-title":"Online Information Review"},{"key":"2021072016155108400_bib58","volume-title":"For fun and profit: A history of the free and open source software revolution","author":"Tozzi","year":"2017"},{"issue":"2","key":"2021072016155108400_bib59","doi-asserted-by":"crossref","first-page":"146","DOI":"10.1515\/zfsoz-1975-0204","article-title":"Resistenz und Rezeptivit\u00e4t der Wissenschaft zu den Entstehungsbedingungen neuer Disziplinen durch wissenschaftliche und politische Steuerung","volume":"4","author":"van den Daele","year":"1975","journal-title":"Zeitschrift f\u00fcr Soziologie"},{"issue":"7442","key":"2021072016155108400_bib60","doi-asserted-by":"crossref","first-page":"426","DOI":"10.1038\/495426a","article-title":"Open access: The true cost of science publishing","volume":"495","author":"Van Noorden","year":"2013","journal-title":"Nature"},{"key":"2021072016155108400_bib61","first-page":"612","article-title":"Exploration of reproducibility issues in scientometric research","volume-title":"Proceedings of the 23rd International Conference on Science and Technology Indicators","author":"Velden","year":"2018"},{"key":"2021072016155108400_bib62","volume-title":"Large-scale comparison of bibliographic data sources: Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic","author":"Visser","year":"2020"},{"key":"2021072016155108400_bib63","volume-title":"From Journal of Informetrics to Quantitative Science Studies","author":"Waltman","year":"2019"},{"issue":"1","key":"2021072016155108400_bib64","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1186\/s13059-015-0669-2","article-title":"When will \u201copen science\u201d become simply \u201cscience\u201d?","volume":"16","author":"Watson","year":"2015","journal-title":"Genome Biology"},{"key":"2021072016155108400_bib65","volume-title":"The intellectual and social organization of the sciences","author":"Whitley","year":"1984"},{"key":"2021072016155108400_bib66","doi-asserted-by":"crossref","DOI":"10.1038\/sdata.2016.18","article-title":"Comment: The FAIR guiding principles for scientific data management and stewardship","volume":"3","author":"Wilkinson","year":"2016","journal-title":"Scientific Data"},{"issue":"4","key":"2021072016155108400_bib67","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1108\/07378830510636300","article-title":"Open journal systems: An example of open source software for journal management and publishing","volume":"23","author":"Willinsky","year":"2005","journal-title":"Library Hi Tech"},{"issue":"10","key":"2021072016155108400_bib68","doi-asserted-by":"crossref","DOI":"10.5210\/fm.v9i10.1177","article-title":"Internet time and the reliability of search engines","volume":"9","author":"Wouters","year":"2004","journal-title":"First Monday"},{"issue":"1","key":"2021072016155108400_bib69","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1002\/asi.23919","article-title":"Data set mentions and citations: A content analysis of full-text publications","volume":"69","author":"Zhao","year":"2018","journal-title":"Journal of the Association for Information Science and Technology"},{"key":"2021072016155108400_bib70","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511541391","volume-title":"Real science: What it is and what it means","author":"Ziman","year":"2000"}],"container-title":["Quantitative Science Studies"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/qss\/article-pdf\/2\/2\/438\/1930758\/qss_e_00121.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/qss\/article-pdf\/2\/2\/438\/1930758\/qss_e_00121.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,16]],"date-time":"2022-12-16T11:54:17Z","timestamp":1671191657000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/qss\/article\/2\/2\/438\/96160\/Which-aspects-of-the-Open-Science-agenda-are-most"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":68,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,7,15]]}},"URL":"https:\/\/doi.org\/10.1162\/qss_e_00121","relation":{},"ISSN":["2641-3337"],"issn-type":[{"value":"2641-3337","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}