{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,23]],"date-time":"2026-04-23T20:06:50Z","timestamp":1776974810171,"version":"3.51.4"},"reference-count":54,"publisher":"MIT Press","issue":"4","license":[{"start":{"date-parts":[[2024,8,6]],"date-time":"2024-08-06T00:00:00Z","timestamp":1722902400000},"content-version":"vor","delay-in-days":218,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Commission H2020 projects OpenAIRE Nexus","award":["101017452"],"award-info":[{"award-number":["101017452"]}]},{"name":"EOSC-Future","award":["101017536"],"award-info":[{"award-number":["101017536"]}]}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,11,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Open science has revolutionized scholarly communication and research assessment by introducing research data and software as first-class citizens. Scholarly knowledge graphs (SKGs) are expected to play a crucial role in generating research assessment indicators being able to aggregate bibliographic metadata records and semantic relationships describing all research products and their links (e.g., citations, affiliations, funding). However, the rapid advance of open science has led to publication workflows that do not adequately support and guarantee the authenticity of products and metadata quality required for research assessment. Additionally, the heterogeneity of research communities and the multitude of data sources and exchange formats complicate the provision of consistent and stable SKGs. This work builds upon the experience gained from pioneering and addressing these challenges in the OpenAIRE Graph SKG. The aim is twofold and broader. First, we identify obstacles to the creation of SKGs for research assessment caused by the state-of-the-art publishing workflows for publications, software, and data. Second, we describe repurposing SKGs as tools to monitor such workflows to identify and heal their shortcomings, taking advantage of tools, techniques, and practices that support the actors involved, namely research communities, scientists, organizations, data source providers, and SKG providers, to improve the Open Science scholarly publishing ecosystem.<\/jats:p>","DOI":"10.1162\/qss_a_00322","type":"journal-article","created":{"date-parts":[[2024,8,6]],"date-time":"2024-08-06T12:43:09Z","timestamp":1722948189000},"page":"991-1021","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":7,"title":["Challenges in building scholarly knowledge graphs for research assessment in open science"],"prefix":"10.1162","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7291-3210","authenticated-orcid":true,"given":"Paolo","family":"Manghi","sequence":"first","affiliation":[{"name":"National Research Council Institute of Information Science and Technologies Alessandro Faedo, Pisa, Italy"},{"name":"OpenAIRE A. M. K. E., Marousi Athens, Greece"}]}],"member":"281","published-online":{"date-parts":[[2024,11,1]]},"reference":[{"key":"2024122514555384500_bib1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2302.02231","article-title":"Pubgraph: A large-scale scholarly knowledge graph","author":"Ahrabian","year":"2023","journal-title":"arXiv"},{"key":"2024122514555384500_bib2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33247-0_11","article-title":"DIRECTions: Design and specification of an IR evaluation infrastructure","volume-title":"Information access evaluation. Multilinguality, multimodality, and visual analytics","author":"Agosti","year":"2012"},{"issue":"1","key":"2024122514555384500_bib3","doi-asserted-by":"publisher","first-page":"2158244019829575","DOI":"10.1177\/2158244019829575","article-title":"Citations, citation indicators, and research quality: An overview of basic concepts and theories","volume":"9","author":"Aksnes","year":"2019","journal-title":"SAGE Open"},{"issue":"8","key":"2024122514555384500_bib4","doi-asserted-by":"publisher","first-page":"4447","DOI":"10.1007\/s11192-023-04746-x","article-title":"A novel methodology to disambiguate organization names: An application to EU framework programmes data","volume":"128","author":"Ancona","year":"2023","journal-title":"Scientometrics"},{"key":"2024122514555384500_bib5","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1007\/978-3-030-55814-7_16","article-title":"Open science graphs must interoperate!","volume-title":"ADBIS, TPDL and EDA 2020 Common Workshops and Doctoral Consortium","author":"Aryani","year":"2020"},{"key":"2024122514555384500_bib6","doi-asserted-by":"publisher","first-page":"38","DOI":"10.5334\/dsj-2021-038","article-title":"We can make a better use of ORCID: Five observed misapplications","volume":"20","author":"Baglioni","year":"2021","journal-title":"Data Science Journal"},{"key":"2024122514555384500_bib7","article-title":"(Semi)automated disambiguation of scholarly repositories","volume-title":"Proceedings of the 19th IRCDL (The Conference on Information and Research Science Connecting to Digital and Library Science)","author":"Baglioni","year":"2023"},{"key":"2024122514555384500_bib8","doi-asserted-by":"publisher","first-page":"arXiv:2307.02647","DOI":"10.48550\/arxiv.2307.02647","article-title":"Introducing the fair principles for research software","author":"Barker","year":"2022","journal-title":"arXiv"},{"key":"2024122514555384500_bib9","doi-asserted-by":"publisher","first-page":"arXiv:2310.02192","DOI":"10.48550\/arxiv.2310.02192","article-title":"Sneaked references: Cooked reference metadata inflate citation counts","author":"Besan\u00e7on","year":"2023","journal-title":"arXiv"},{"key":"2024122514555384500_bib10","doi-asserted-by":"publisher","first-page":"arXiv:2105.08599","DOI":"10.48550\/arxiv.2105.08599","article-title":"Can we assess research using open scholarly knowledge graphs? A case study within the Italian national scientific qualification","author":"Bologna","year":"2021","journal-title":"arXiv"},{"key":"2024122514555384500_bib11","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/978-3-030-54956-5_1","article-title":"Requirements analysis for an open research knowledge graph","volume-title":"Digital libraries for open knowledge","author":"Brack","year":"2020"},{"issue":"5","key":"2024122514555384500_bib12","doi-asserted-by":"publisher","first-page":"954","DOI":"10.1162\/rest_a_00926","article-title":"The impact of open access mandates on invention","volume":"103","author":"Bryan","year":"2021","journal-title":"Review of Economics and Statistics"},{"key":"2024122514555384500_bib13","doi-asserted-by":"publisher","first-page":"39","DOI":"10.5334\/dsj-2019-039","article-title":"Research data publication: Moving beyond the metaphor","volume":"18","author":"Callaghan","year":"2019","journal-title":"Data Science Journal"},{"issue":"1","key":"2024122514555384500_bib14","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1177\/09610006211058908","article-title":"Preprint paper platforms in the academic scholarly communication environment","volume":"55","author":"Chaleplioglou","year":"2023","journal-title":"Journal of Librarianship and Information Science"},{"issue":"2","key":"2024122514555384500_bib15","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1016\/j.lisr.2019.04.004","article-title":"Global perspectives of research data sharing: A systematic literature review","volume":"41","author":"Chawinga","year":"2019","journal-title":"Library and Information Science Research"},{"issue":"4","key":"2024122514555384500_bib16","doi-asserted-by":"publisher","first-page":"e0230416","DOI":"10.1371\/journal.pone.0230416","article-title":"The citation advantage of linking publications to research data","volume":"15","author":"Colavizza","year":"2020","journal-title":"PLOS ONE"},{"key":"2024122514555384500_bib17","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.4420095","article-title":"EOSC interoperability framework reference architecture","author":"Corcho","year":"2021","journal-title":"Zenodo"},{"issue":"3","key":"2024122514555384500_bib18","doi-asserted-by":"publisher","first-page":"446","DOI":"10.1002\/asi.23056","article-title":"The google scholar experiment: How to index false papers and manipulate bibliometric indicators","volume":"65","author":"Delgado L\u00f3pez-C\u00f3zar","year":"2014","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"5","key":"2024122514555384500_bib19","doi-asserted-by":"publisher","first-page":"e157","DOI":"10.1371\/journal.pbio.0040157","article-title":"Citation advantage of open access articles","volume":"4","author":"Eysenbach","year":"2006","journal-title":"PLoS Biology"},{"issue":"12","key":"2024122514555384500_bib20","doi-asserted-by":"publisher","first-page":"e0187394","DOI":"10.1371\/journal.pone.0187394","article-title":"Authorship and citation manipulation in academic research","volume":"12","author":"Fong","year":"2017","journal-title":"PLOS ONE"},{"key":"2024122514555384500_bib21","doi-asserted-by":"publisher","first-page":"118","DOI":"10.12688\/f1000research.78195.1","article-title":"Research software vs. research data I: Towards a research data definition in the open science context","volume":"11","author":"Gomez-Diaz","year":"2022","journal-title":"F1000Research"},{"key":"2024122514555384500_bib22","doi-asserted-by":"publisher","first-page":"117","DOI":"10.12688\/f1000research.78459.1","article-title":"Research software vs. research data II: Protocols for research data dissemination and evaluation in the open science context","volume":"11","author":"Gomez-Diaz","year":"2022","journal-title":"F1000Research"},{"key":"2024122514555384500_bib23","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.5504015","article-title":"Defining research software: A controversial discussion","author":"Gruenpeter","year":"2021","journal-title":"Zenodo"},{"key":"2024122514555384500_bib24","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1007\/978-3-031-21756-2_24","article-title":"Scholarly knowledge extraction from published software packages","volume-title":"From born-physical to born-virtual: Augmenting intelligence in digital libraries","author":"Haris","year":"2022"},{"key":"2024122514555384500_bib25","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.3126669","article-title":"Publication performance vs. influence: On the questionable value of quality weighted publication rankings","author":"Haucap","year":"2018","journal-title":"SSRN Electronic Journal"},{"key":"2024122514555384500_bib26","doi-asserted-by":"publisher","first-page":"1213","DOI":"10.1007\/s11192-019-03217-6","article-title":"Software review: COCI, the OpenCitations index of Crossref open DOI-to-DOI citations","volume":"121","author":"Heibi","year":"2019","journal-title":"Scientometrics"},{"key":"2024122514555384500_bib27","doi-asserted-by":"publisher","DOI":"10.59350\/9qn73-phk11","volume-title":"FAIR principles for research software released","author":"Hong","year":"2022"},{"issue":"1\u20132","key":"2024122514555384500_bib28","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1162\/dint_a_00025","article-title":"Unique, persistent, resolvable: Identifiers as the foundation of FAIR","volume":"2","author":"Juty","year":"2020","journal-title":"Data Intelligence"},{"issue":"1","key":"2024122514555384500_bib29","doi-asserted-by":"publisher","first-page":"228","DOI":"10.3998\/3336451.0008.101","article-title":"Locally controlled scholarly publishing via the internet: The guild model","volume":"39","author":"Kling","year":"2002","journal-title":"Proceedings of the American Society for Information Science and Technology"},{"key":"2024122514555384500_bib30","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.10037121","article-title":"OpenAIRE Graph dataset","author":"Manghi","year":"2023","journal-title":"Zenodo"},{"key":"2024122514555384500_bib31","doi-asserted-by":"publisher","first-page":"409","DOI":"10.1108\/dta-09-2019-0163","article-title":"Entity deduplication in big data graphs for scholarly communication","volume":"54","author":"Manghi","year":"2020","journal-title":"Data Technologies and Applications"},{"issue":"4","key":"2024122514555384500_bib32","doi-asserted-by":"publisher","first-page":"1296","DOI":"10.1162\/qss_e_00160","article-title":"New trends in scientific knowledge graphs and research impact assessment","volume":"2","author":"Manghi","year":"2021","journal-title":"Quantitative Science Studies"},{"key":"2024122514555384500_bib33","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2207.03121","article-title":"Will open science change authorship for good?","volume-title":"Proceedings of the 18th Italian Research Conference on Digital Libraries","author":"Mannocci","year":"2022"},{"key":"2024122514555384500_bib34","doi-asserted-by":"publisher","first-page":"WDS32","DOI":"10.2481\/dsj.WDS-042","article-title":"Is data publication the right metaphor?","volume":"12","author":"Parsons","year":"2013","journal-title":"Data Science Journal"},{"key":"2024122514555384500_bib35","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.5101096","article-title":"OpenOrgs: Bridging registries of research organisations. Supporting disambiguation and improving the quality of data","author":"Pavone","year":"2021","journal-title":"Zenodo"},{"issue":"11","key":"2024122514555384500_bib36","doi-asserted-by":"publisher","first-page":"13071","DOI":"10.1007\/s10462-023-10465-9","article-title":"Knowledge graphs: Opportunities and challenges","volume":"56","author":"Peng","year":"2023","journal-title":"Artificial Intelligence Review"},{"issue":"3","key":"2024122514555384500_bib37","doi-asserted-by":"publisher","first-page":"e308","DOI":"10.1371\/journal.pone.0000308","article-title":"Sharing detailed research data is associated with increased citation rate","volume":"2","author":"Piwowar","year":"2007","journal-title":"PLOS ONE"},{"key":"2024122514555384500_bib38","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1109\/bigdata.2013.6691588","article-title":"Scalable data citation in dynamic, large databases: Model and reference implementation","volume-title":"2013 IEEE International Conference on Big Data","author":"Pr\u00f6ll","year":"2013"},{"issue":"4","key":"2024122514555384500_bib39","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1177\/0165551520961048","article-title":"Do researchers use open research data? Exploring the relationships between usage trends and metadata quality across scientific disciplines from the Figshare case","volume":"48","author":"Quarati","year":"2022","journal-title":"Journal of Information Science"},{"key":"2024122514555384500_bib40","doi-asserted-by":"publisher","first-page":"588","DOI":"10.12688\/f1000research.11369.2","article-title":"What is open peer review? A systematic review","volume":"6","author":"Ross-Hellauer","year":"2017","journal-title":"F1000Research"},{"key":"2024122514555384500_bib41","doi-asserted-by":"publisher","first-page":"225","DOI":"10.1007\/978-3-030-86668-6_11","article-title":"Detection, analysis, and prediction of research topics with scientific knowledge graphs","volume-title":"Predicting the dynamics of research impact","author":"Salatino","year":"2021"},{"issue":"2","key":"2024122514555384500_bib42","doi-asserted-by":"publisher","first-page":"227","DOI":"10.1177\/0165551519888605","article-title":"A review of author name disambiguation techniques for the PubMed bibliographic database","volume":"47","author":"Sanyal","year":"2021","journal-title":"Journal of Information Science"},{"issue":"9","key":"2024122514555384500_bib43","doi-asserted-by":"publisher","first-page":"e1002663","DOI":"10.1371\/journal.pmed.1002663","article-title":"Science without publication paywalls: cOAlition S for the realisation of full and immediate open access","volume":"15","author":"Schiltz","year":"2018","journal-title":"PLOS Medicine"},{"issue":"1","key":"2024122514555384500_bib44","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1186\/s12874-021-01252-7","article-title":"Facilitating harmonized data quality assessments. A data quality framework for observational health research data collections with software implementations in R","volume":"21","author":"Schmidt","year":"2021","journal-title":"BMC Medical Research Methodology"},{"key":"2024122514555384500_bib45","doi-asserted-by":"publisher","DOI":"10.2777\/445286","volume-title":"Indicator frameworks for fostering open knowledge practices in science and scholarship","author":"Schomberg","year":"2019"},{"issue":"1","key":"2024122514555384500_bib46","doi-asserted-by":"publisher","first-page":"8","DOI":"10.5334\/egems.280","article-title":"Data quality assessment and multi-organizational reporting: Tools to enhance network knowledge","volume":"7","author":"Sengupta","year":"2019","journal-title":"eGEMs"},{"issue":"2","key":"2024122514555384500_bib47","doi-asserted-by":"publisher","first-page":"360","DOI":"10.1073\/pnas.1418218112","article-title":"Measuring the effectiveness of scientific gatekeeping","volume":"112","author":"Siler","year":"2015","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"1","key":"2024122514555384500_bib48","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1002\/asi.23917","article-title":"Theory and practice of data citation","volume":"69","author":"Silvello","year":"2018","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"1","key":"2024122514555384500_bib49","doi-asserted-by":"publisher","first-page":"101109","DOI":"10.1016\/j.joi.2020.101109","article-title":"SciKGraph: A knowledge graph approach to structure a scientific field","volume":"15","author":"Tosi","year":"2021","journal-title":"Journal of Informetrics"},{"key":"2024122514555384500_bib50","doi-asserted-by":"publisher","DOI":"10.31235\/osf.io\/2kxq8","volume-title":"A tale of two \u2018opens\u2019: Intersections between free and open source software and open scholarship","author":"Tennant","year":"2020"},{"key":"2024122514555384500_bib51","doi-asserted-by":"publisher","first-page":"e51987","DOI":"10.3897\/ese.2020.e51987","article-title":"Web of Science and Scopus are not global databases of knowledge","volume":"46","author":"Tennant","year":"2020","journal-title":"European Science Editing"},{"key":"2024122514555384500_bib52","doi-asserted-by":"publisher","first-page":"1059","DOI":"10.1007\/s40747-022-00806-6","article-title":"Scholarly knowledge graphs through structuring scholarly communication: A review","volume":"9","author":"Verma","year":"2023","journal-title":"Complex & Intelligent Systems"},{"key":"2024122514555384500_bib53","doi-asserted-by":"publisher","DOI":"10.12688\/openreseurope.15692.2","article-title":"The ESCAPE open-source software and service repository","author":"Vuillaume","year":"2023","journal-title":"Open Research Europe"},{"issue":"6","key":"2024122514555384500_bib54","doi-asserted-by":"publisher","first-page":"334","DOI":"10.3390\/ijgi11060334","article-title":"Quality assurance for spatial research data","volume":"11","author":"Wagner","year":"2022","journal-title":"ISPRS International Journal of Geo-Information"}],"container-title":["Quantitative Science Studies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/5\/4\/991\/2474893\/qss_a_00322.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/5\/4\/991\/2474893\/qss_a_00322.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,25]],"date-time":"2024-12-25T09:56:06Z","timestamp":1735120566000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/qss\/article\/5\/4\/991\/123928\/Challenges-in-building-scholarly-knowledge-graphs"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":54,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,11,1]]}},"URL":"https:\/\/doi.org\/10.1162\/qss_a_00322","relation":{"has-review":[{"id-type":"doi","id":"10.1162\/QSS_A_00322\/v2\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00322\/v2\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00322\/v1\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00322\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00322\/v1\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00322\/v1\/review2","asserted-by":"object"}]},"ISSN":["2641-3337"],"issn-type":[{"value":"2641-3337","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}