{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,2]],"date-time":"2026-11-02T21:37:17Z","timestamp":1793655437018,"version":"4.5.1"},"reference-count":60,"publisher":"MIT Press","issue":"2","license":[{"start":{"date-parts":[[2021,5,12]],"date-time":"2021-05-12T00:00:00Z","timestamp":1620777600000},"content-version":"vor","delay-in-days":131,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Commission Directorate-General for Research and Innovation"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,7,15]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Cross-disciplinary research (multi-\/interdisciplinarity) is incentivized by funding agencies to foster research outcomes addressing complex societal challenges. This study focuses on the link between cross-disciplinary research and its uptake in a broad set of policy-related documents. Using the new policy-oriented database Overton, matched to Scopus, logistic regression was used in assessing this relationship in publications from FP7- and H2020-supported projects. Cross-disciplinary research was captured through two lenses at the paper level, namely from the disciplinary diversity of contributing authors (DDA) and of cited references (DDR). DDA increased the likelihood that publications were cited in policy documents, with DDR possibly making a contribution, but only when publications result from the work of few authors. Citations to publications captured by Overton were found to originate in scientific advice documents, rather than in legislative or executive records. Our approach enables testing in a general way the assumption underlying many funding programs, namely that cross-disciplinary research will increase the policy relevance of research outcomes. Findings suggest that research assessments could benefit from measuring uptake in policy-related literature, following additional characterization of the Overton database; of the science-policy interactions it captures; and of the contribution of these interactions within the larger policymaking process.<\/jats:p>","DOI":"10.1162\/qss_a_00137","type":"journal-article","created":{"date-parts":[[2021,5,12]],"date-time":"2021-05-12T11:40:12Z","timestamp":1620819612000},"page":"616-642","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":47,"title":["A large-scale validation of the relationship between cross-disciplinary research and its uptake in policy-related documents, using the novel Overton altmetrics database"],"prefix":"10.1162","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2175-7518","authenticated-orcid":true,"given":"Henrique","family":"Pinheiro","sequence":"first","affiliation":[{"name":"Science-Metrix, an Elsevier company, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4948-4363","authenticated-orcid":true,"given":"Etienne","family":"Vignola-Gagn\u00e9","sequence":"additional","affiliation":[{"name":"Science-Metrix, an Elsevier company, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3806-3237","authenticated-orcid":true,"given":"David","family":"Campbell","sequence":"additional","affiliation":[{"name":"Science-Metrix, an Elsevier company, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2021,7,15]]},"reference":[{"issue":"4","key":"2025073014032346300_bib1","doi-asserted-by":"publisher","first-page":"1182","DOI":"10.1016\/j.joi.2018.09.001","article-title":"A comparison of two approaches for measuring interdisciplinary research output: The disciplinary diversity of authors vs the disciplinary diversity of the reference list","volume":"12","author":"Abramo","year":"2018","journal-title":"Journal of Informetrics"},{"key":"2025073014032346300_bib2","volume-title":"Interdisciplinary research: Methodologies for identification and assessment","author":"Adams","year":"2016"},{"key":"2025073014032346300_bib3","article-title":"16.1 Delta method","volume-title":"Categorical Data Analysis","author":"Agresti","year":"2012","edition":"3rd ed."},{"key":"2025073014032346300_bib4","volume-title":"Quests for interdisciplinarity: A challenge for the ERA and HORIZON 2020. Policy Brief by the Research, Innovation, and Science Policy Experts (RISE)","author":"Allmendinger","year":"2015"},{"key":"2025073014032346300_bib5","first-page":"66","article-title":"Towards a multilingual, comprehensive and open scientific journal ontology","volume-title":"Proceedings of the 13th International Conference of the International Society for Scientometrics and Informetrics","author":"Archambault","year":"2011"},{"key":"2025073014032346300_bib6","doi-asserted-by":"publisher","first-page":"rvaa024","DOI":"10.1093\/reseval\/rvaa024","article-title":"Understanding and evaluating the impact of integrated problem-oriented research programmes: Concepts and considerations","author":"Belcher","year":"2020","journal-title":"Research Evaluation"},{"key":"2025073014032346300_bib7","doi-asserted-by":"publisher","first-page":"1477","DOI":"10.1007\/s11192-016-2115-y","article-title":"Policy documents as sources for measuring societal impact: How often is climate change research mentioned in policy-related documents?","volume":"109","author":"Bornmann","year":"2016","journal-title":"Scientometrics"},{"issue":"7609","key":"2025073014032346300_bib8","doi-asserted-by":"publisher","first-page":"684","DOI":"10.1038\/nature18315","article-title":"Interdisciplinary research has consistently lower funding success","volume":"534","author":"Bromham","year":"2016","journal-title":"Nature"},{"key":"2025073014032346300_bib9","doi-asserted-by":"publisher","first-page":"1576","DOI":"10.2105\/AJPH.2008.156224","article-title":"Understanding evidence-based public health policy","volume":"99","author":"Brownson","year":"2009","journal-title":"American Journal of Public Health"},{"issue":"3","key":"2025073014032346300_bib10","doi-asserted-by":"publisher","first-page":"103924","DOI":"10.1016\/j.respol.2020.103924","article-title":"Interrelated funding streams in a multi-funder university system: Evidence from the German Exzellenzinitiative","volume":"49","author":"Buenstorf","year":"2020","journal-title":"Research Policy"},{"key":"2025073014032346300_bib11","article-title":"Systems biology, interdisciplinarity and disciplinary identity","volume-title":"Collaboration in the new life sciences","author":"Calvert","year":"2010"},{"key":"2025073014032346300_bib12","article-title":"Application of an \u201cinterdisciplinarity\u201d metric at the paper level and its use in a comparative analysis of the most publishing ERA and non-ERA universities","author":"Campbell","year":"2015","journal-title":"20th International Conference on Science and Technology Indicators"},{"key":"2025073014032346300_bib13","first-page":"1276","article-title":"Reliability of Scopus author identifiers (AUIDs) for research evaluation purposes at different scales","volume-title":"Proceedings of the 17th International Conference of the International Society for Scientometrics and Informetrics (ISSI), 2\u20135 September 2019","author":"Campbell","year":"2019"},{"key":"2025073014032346300_bib14","article-title":"Impact of multidisciplinary research on innovation","volume-title":"16th International Conference of the International Society for Scientometrics and Informetrics (ISSI), 16\u201320 October 2017","author":"Campbell","year":"2017"},{"issue":"3","key":"2025073014032346300_bib15","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1093\/reseval\/rvu012","article-title":"Interdisciplinarity and research on local issues: evidence from a developing country","volume":"23","author":"Chavarro","year":"2014","journal-title":"Research Evaluation"},{"issue":"5","key":"2025073014032346300_bib16","doi-asserted-by":"publisher","first-page":"293","DOI":"10.1016\/j.technovation.2011.12.005","article-title":"Inventors and entrepreneurs in academia: What types of skills and experience matter?","volume":"32","author":"Deste","year":"2012","journal-title":"Technovation"},{"issue":"4","key":"2025073014032346300_bib17","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0216146","article-title":"The emergent integrated network structure of scientific research","volume":"14","author":"Dworkin","year":"2019","journal-title":"PLOS ONE"},{"key":"2025073014032346300_bib18","volume-title":"Next-generation metrics: Responsible metrics and evaluation for open science","author":"European Commission Expert Group on Altmetrics","year":"2017"},{"key":"2025073014032346300_bib19","volume-title":"Qualitative evaluation of completed projects funded by the European Research Council","author":"European Research Council","year":"2018"},{"key":"2025073014032346300_bib20","doi-asserted-by":"publisher","first-page":"rvaa017","DOI":"10.1093\/reseval\/rvaa017","article-title":"The program and policy change framework: A new tool to measure research use in low- and middle-income countries","author":"Fowle","year":"2020","journal-title":"Research Evaluation"},{"issue":"2","key":"2025073014032346300_bib21","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1093\/reseval\/rvs004","article-title":"Assessing research network and disciplinary engagement changes induced by an NSF program","volume":"21","author":"Garner","year":"2012","journal-title":"Research Evaluation"},{"key":"2025073014032346300_bib22","volume-title":"Interdisciplinarity survey report for the Global Research Council 2016 annual meeting","author":"Gleed","year":"2016"},{"issue":"1","key":"2025073014032346300_bib23","doi-asserted-by":"publisher","DOI":"10.1016\/j.respol.2020.104069","article-title":"Do synthesis centers synthesize? A semantic analysis of topical diversity in research","volume":"50","author":"Hackett","year":"2021","journal-title":"Research Policy"},{"issue":"3","key":"2025073014032346300_bib24","doi-asserted-by":"publisher","first-page":"1209","DOI":"10.1007\/s11192-016-2237-2","article-title":"How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data","volume":"110","author":"Haunschild","year":"2017","journal-title":"Scientometrics"},{"issue":"1","key":"2025073014032346300_bib25","doi-asserted-by":"publisher","first-page":"413","DOI":"10.1007\/s11192-016-1910-9","article-title":"Grand challenges in altmetrics: Heterogeneity, data quality and dependencies","volume":"108","author":"Haustein","year":"2016","journal-title":"Scientometrics"},{"issue":"3","key":"2025073014032346300_bib26","doi-asserted-by":"publisher","first-page":"557","DOI":"10.1016\/j.respol.2016.10.008","article-title":"How complex international partnerships shape domestic research clusters: Difference-in-difference network formation and research re-orientation in the MIT Portugal Program","volume":"46","author":"Hird","year":"2017","journal-title":"Research Policy"},{"issue":"1","key":"2025073014032346300_bib27","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1093\/reseval\/rvu027","article-title":"How accurately does output reflect the nature and design of transdisciplinary research programmes?","volume":"24","author":"Koier","year":"2015","journal-title":"Research Evaluation"},{"issue":"4","key":"2025073014032346300_bib28","doi-asserted-by":"publisher","first-page":"389","DOI":"10.3233\/ISU-170857","article-title":"Influence seekers: The production of grey literature for policy and practice","volume":"37","author":"Lawrence","year":"2018","journal-title":"Information Services and Use"},{"key":"2025073014032346300_bib29","volume-title":"Where is the evidence? Realising the value of grey literature for public policy & practice","author":"Lawrence","year":"2014"},{"key":"2025073014032346300_bib30","volume-title":"Interdisciplinarity and the 21st century research-intensive university","author":"LERU","year":"2016"},{"issue":"1","key":"2025073014032346300_bib31","doi-asserted-by":"publisher","first-page":"255","DOI":"10.1016\/j.joi.2018.12.006","article-title":"Interdisciplinarity as diversity in citation patterns among journals: Rao-Stirling diversity, relative variety, and the Gini coefficient","volume":"13","author":"Leydesdorff","year":"2019","journal-title":"Journal of Informetrics"},{"issue":"1","key":"2025073014032346300_bib32","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1093\/scipol\/scs121","article-title":"The role of funding agencies in creating interdisciplinary knowledge","volume":"40","author":"Lyall","year":"2013","journal-title":"Science and Public Policy"},{"key":"2025073014032346300_bib33","doi-asserted-by":"publisher","first-page":"150","DOI":"10.1016\/j.futures.2014.08.009","article-title":"A rose by any other name? Transdisciplinarity in the context of UK research policy","volume":"65","author":"Lyall","year":"2015","journal-title":"Futures"},{"issue":"September","key":"2025073014032346300_bib34","first-page":"69","article-title":"On the relationship between interdisciplinarity and impact: Different modalities of interdisciplinarity lead to different types of impact","volume":"29","author":"Molas-Gallart","year":"2014","journal-title":"Journal of Science Policy and Research Management"},{"issue":"2","key":"2025073014032346300_bib35","doi-asserted-by":"publisher","first-page":"1145","DOI":"10.1007\/s11192-020-03481-x","article-title":"A comparison of three multidisciplinarity indices based on the diversity of Scopus subject areas of authors\u2019 documents, their bibliography and their citing papers","volume":"125","author":"Moschini","year":"2020","journal-title":"Scientometrics"},{"issue":"1","key":"2025073014032346300_bib36","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1186\/s12961-018-0326-9","article-title":"Does citation matter? Research citation in policy documents as an indicator of research impact\u2014An Australian obesity policy case-study","volume":"16","author":"Newson","year":"2018","journal-title":"Health Research Policy and Systems"},{"issue":"1","key":"2025073014032346300_bib37","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1057\/s41599-019-0352-4","article-title":"Interdisciplinarity revisited: evidence for research impact and dynamism","volume":"5","author":"Okamura","year":"2019","journal-title":"Palgrave Communications"},{"issue":"3","key":"2025073014032346300_bib38","doi-asserted-by":"publisher","first-page":"282","DOI":"10.1525\/bio.2012.62.3.9","article-title":"Research coordination networks: Evidence of the relationship between funded interdisciplinary networking and scholarly impact","volume":"62","author":"Porter","year":"2012","journal-title":"Bioscience"},{"issue":"3","key":"2025073014032346300_bib39","doi-asserted-by":"publisher","first-page":"719","DOI":"10.1007\/s11192-008-2197-2","article-title":"Is science becoming more interdisciplinary? Measuring and mapping six research fields over time","volume":"81","author":"Porter","year":"2009","journal-title":"Scientometrics"},{"issue":"8","key":"2025073014032346300_bib40","doi-asserted-by":"publisher","first-page":"e0203117","DOI":"10.1371\/journal.pone.0203117","article-title":"Social impact in social media: A new method to evaluate the social impact of research","volume":"13","author":"Pulido","year":"2018","journal-title":"PLOS ONE"},{"issue":"2","key":"2025073014032346300_bib41","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1007\/s11192-009-0041-y","article-title":"Diversity and network coherence as indicators of interdisciplinarity: case studies in bionanoscience","volume":"82","author":"Rafols","year":"2010","journal-title":"Scientometrics"},{"key":"2025073014032346300_bib42","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0251493","article-title":"Article-level classification of scientific publications: a comparison of deep learning, direct citation and bibliographic coupling","author":"Rivest","year":"2021","journal-title":"PLoS ONE"},{"key":"2025073014032346300_bib43","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1038\/525313a","article-title":"Grant giving: Global funders to focus on interdisciplinarity","volume":"525","author":"Rylance","year":"2015","journal-title":"Nature"},{"issue":"3","key":"2025073014032346300_bib44","doi-asserted-by":"publisher","first-page":"463","DOI":"10.1093\/scipol\/scy074","article-title":"Research funding programmes aiming for societal transformations: Ten key stages","volume":"46","author":"Schneider","year":"2019","journal-title":"Science and Public Policy"},{"key":"2025073014032346300_bib45","volume-title":"Review of the Human Frontier Science Program 2018","author":"Science-Metrix","year":"2018"},{"issue":"15","key":"2025073014032346300_bib46","doi-asserted-by":"publisher","first-page":"707","DOI":"10.1098\/rsif.2007.0213","article-title":"A general framework for analysing diversity in science, technology and society","volume":"4","author":"Stirling","year":"2007","journal-title":"Journal of the Royal Society Interface"},{"issue":"1","key":"2025073014032346300_bib47","doi-asserted-by":"publisher","first-page":"e290102","DOI":"10.3145\/epi.2020.ene.02","article-title":"Altmetrics and societal impact measurements: Match or mismatch? A literature review","volume":"29","author":"Tahamtan","year":"2020","journal-title":"El Profesional de La Informaci\u00f3n"},{"issue":"9","key":"2025073014032346300_bib48","doi-asserted-by":"publisher","DOI":"10.3389\/frma.2017.00009","article-title":"What can Altmetric.com tell us about policy citations of research? An analysis of Altmetric.com data for research articles from the University of Sheffield","volume":"2","author":"Tattersall","year":"2018","journal-title":"Frontiers in Research Metrics and Analytics"},{"key":"2025073014032346300_bib49","volume-title":"Evaluation of the Belmont Forum: Final report","author":"Technopolis Group & Science-Metrix","year":"2020"},{"key":"2025073014032346300_bib50","doi-asserted-by":"publisher","DOI":"10.2200\/s00733ed1v01y201609icr052","article-title":"Web indicators for research evaluation: A practical guide","volume-title":"Synthesis Lectures on Information Concepts, Retrieval, and Services","author":"Thelwall","year":"2016"},{"issue":"5","key":"2025073014032346300_bib51","doi-asserted-by":"publisher","first-page":"587","DOI":"10.3145\/epi.2015.sep.08","article-title":"Web indicators for research evaluation. Part 1: Citations and links to academic articles from the Web","volume":"24","author":"Thelwall","year":"2015","journal-title":"El Profesional de La Informaci\u00f3n"},{"key":"2025073014032346300_bib52","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1016\/j.envsci.2016.03.012","article-title":"New science for global sustainability? The institutionalisation of knowledge co-production in Future Earth","volume":"61","author":"van der Hel","year":"2016","journal-title":"Environmental Science and Policy"},{"key":"2025073014032346300_bib53","doi-asserted-by":"publisher","first-page":"e0127298","DOI":"10.1371\/journal.pone.0127298","article-title":"Interdisciplinarity and impact: Distinct effects of variety, balance, and disparity","volume":"10","author":"Wang","year":"2015","journal-title":"PLOS ONE"},{"key":"2025073014032346300_bib54","doi-asserted-by":"crossref","first-page":"576","DOI":"10.1145\/3190645","article-title":"Effects of knowledge flows from public science to industrial technologies","volume-title":"Proceedings of the 2018 STI Conference","author":"Wang","year":"2018"},{"issue":"2","key":"2025073014032346300_bib55","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1136\/tobaccocontrol-2011-050396","article-title":"The impact of tobacco control research on policy: 20 years of progress","volume":"21","author":"Warner","year":"2012","journal-title":"Tobacco Control"},{"issue":"5","key":"2025073014032346300_bib56","doi-asserted-by":"publisher","first-page":"426","DOI":"10.2307\/3109916","article-title":"The many meanings of research utilization","volume":"39","author":"Weiss","year":"1979","journal-title":"Public Administration Review"},{"issue":"2","key":"2025073014032346300_bib57","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1093\/reseval\/rvx010","article-title":"Evaluating the impact of applied prevention research centres: results from a modified Delphi approach","volume":"26","author":"Willis","year":"2017","journal-title":"Research Evaluation"},{"key":"2025073014032346300_bib58","doi-asserted-by":"publisher","DOI":"10.13140\/RG.2.1.4929.1363","volume-title":"The metric tide: Report of the independent review of the role of metrics in research assessment and management","author":"Wilsdon","year":"2015"},{"issue":"1","key":"2025073014032346300_bib59","doi-asserted-by":"publisher","first-page":"271","DOI":"10.1007\/s11192-018-2853-0","article-title":"Interdisciplinarity and collaboration: on the relationship between disciplinary diversity in departmental affiliations and reference lists","volume":"117","author":"Zhang","year":"2018","journal-title":"Scientometrics"},{"issue":"3","key":"2025073014032346300_bib60","doi-asserted-by":"publisher","first-page":"736","DOI":"10.1016\/j.joi.2018.06.006","article-title":"The more multidisciplinary the better? The prevalence and interdisciplinarity of research collaborations in multidisciplinary institutions","volume":"12","author":"Zuo","year":"2018","journal-title":"Journal of Informetrics"}],"container-title":["Quantitative Science Studies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/2\/2\/616\/1930734\/qss_a_00137.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/2\/2\/616\/1930734\/qss_a_00137.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T18:03:41Z","timestamp":1753898621000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/qss\/article\/2\/2\/616\/101030\/A-large-scale-validation-of-the-relationship"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":60,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,7,15]]}},"URL":"https:\/\/doi.org\/10.1162\/qss_a_00137","relation":{"has-review":[{"id-type":"doi","id":"10.1162\/QSS_A_00137\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00137\/v1\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00137\/v2\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00137\/v1\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00137\/v1\/review1","asserted-by":"object"}]},"ISSN":["2641-3337"],"issn-type":[{"value":"2641-3337","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}