{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T12:31:18Z","timestamp":1772800278062,"version":"3.50.1"},"reference-count":33,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,5,6]],"date-time":"2023-05-06T00:00:00Z","timestamp":1683331200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,5,6]],"date-time":"2023-05-06T00:00:00Z","timestamp":1683331200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000265","name":"Medical Research Council","doi-asserted-by":"publisher","award":["MC_UU_00011\/4"],"award-info":[{"award-number":["MC_UU_00011\/4"]}],"id":[{"id":"10.13039\/501100000265","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Big Data"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Embedding knowledge graphs into low-dimensional spaces is a popular method for applying approaches, such as link prediction or node classification, to these databases. This embedding process is very costly in terms of both computational time and space. Part of the reason for this is the optimisation of hyperparameters, which involves repeatedly sampling, by random, guided, or brute-force selection, from a large hyperparameter space and testing the resulting embeddings for their quality. However, not all hyperparameters in this search space will be equally important. In fact, with prior knowledge of the relative importance of the hyperparameters, some could be eliminated from the search altogether without significantly impacting the overall quality of the outputted embeddings. To this end, we ran a Sobol sensitivity analysis to evaluate the effects of tuning different hyperparameters on the variance of embedding quality. This was achieved by performing thousands of embedding trials, each time measuring the quality of embeddings produced by different hyperparameter configurations. We regressed the embedding quality on those hyperparameter configurations, using this model to generate Sobol sensitivity indices for each of the hyperparameters. By evaluating the correlation between Sobol indices, we find substantial variability in the hyperparameter sensitivities between knowledge graphs with differing dataset characteristics as the probable cause of these inconsistencies. As an additional contribution of this work we identify several relations in the UMLS knowledge graph that may cause data leakage via inverse relations, and derive and present UMLS-43, a leakage-robust variant of that graph.<\/jats:p>","DOI":"10.1186\/s40537-023-00732-5","type":"journal-article","created":{"date-parts":[[2023,5,6]],"date-time":"2023-05-06T04:01:37Z","timestamp":1683345697000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Assessing the effects of hyperparameters on knowledge graph embedding quality"],"prefix":"10.1186","volume":"10","author":[{"given":"Oliver","family":"Lloyd","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yi","family":"Liu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tom","family":"R. Gaunt","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2023,5,6]]},"reference":[{"key":"732_CR1","doi-asserted-by":"crossref","unstructured":"Toutanova K, Chen D. Observed versus latent features for knowledge base and text inference. Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality. 2015","DOI":"10.18653\/v1\/W15-4007"},{"issue":"2","key":"732_CR2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3424672","volume":"15","author":"A Rossi","year":"2021","unstructured":"Rossi A, Barbosa D, Firmani D, Matinata A, Merialdo P. Knowledge graph embedding for link prediction. ACM Trans Knowl Discov Data. 2021;15(2):1\u201349.","journal-title":"ACM Trans Knowl Discov Data"},{"key":"732_CR3","unstructured":"Energy Follow Up Survey: Household Energy Consumption & Affordability. 2021. https:\/\/assets.publishing.service.gov.uk\/government\/uploads\/system\/uploads\/attachment_data\/file\/1018725\/efus-Household-Energy-Consumption-Affordability.pdf. Accessed 8 Aug 2022."},{"key":"732_CR4","unstructured":"DB-Engines Ranking per database model category. Db-engines.com. 2022. https:\/\/db-engines.com\/en\/ranking_categories. Accessed 8 Aug 2022."},{"key":"732_CR5","doi-asserted-by":"crossref","unstructured":"Chen H, Sultan S, Tian Y, Chen M, Skiena S. Fast and accurate network embeddings via very sparse random projection. Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2019.","DOI":"10.1145\/3357384.3357879"},{"key":"732_CR6","doi-asserted-by":"crossref","unstructured":"Zhang Z, Cui P, Li H, Wang X, Zhu W. Billion-scale network embedding with iterative random projection. 2018 IEEE International Conference on Data Mining (ICDM). 2018.","DOI":"10.1109\/ICDM.2018.00094"},{"key":"732_CR7","doi-asserted-by":"crossref","unstructured":"Li Z, Ji J, Fu Z, Ge Y, Xu S, Chen C et al. Efficient Non-Sampling Knowledge Graph Embedding. Proceedings of the Web Conference 2021. 2021.","DOI":"10.1145\/3442381.3449859"},{"key":"732_CR8","doi-asserted-by":"crossref","unstructured":"Peng X, Chen G, Lin C, Stevenson M. Highly efficient knowledge graph embedding learning with orthogonal procrustes analysis. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.","DOI":"10.18653\/v1\/2021.naacl-main.187"},{"key":"732_CR9","doi-asserted-by":"crossref","unstructured":"Broscheit S, Ruffinelli D, Kochsiek A, Betz P, Gemulla R. LibKGE\u2014a knowledge graph embedding library for reproducible research. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2020.","DOI":"10.18653\/v1\/2020.emnlp-demos.22"},{"key":"732_CR10","doi-asserted-by":"crossref","unstructured":"Zheng D, Song X, Ma C, Tan Z, Ye Z, Dong J et al. DGL-KE: Training knowledge graph embeddings at scale. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2020.","DOI":"10.1145\/3397271.3401172"},{"key":"732_CR11","doi-asserted-by":"crossref","unstructured":"Wang H, Wang Y, Lian D, Gao J. A lightweight knowledge graph embedding framework for efficient inference and storage. Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 2021.","DOI":"10.1145\/3459637.3482224"},{"key":"732_CR12","doi-asserted-by":"crossref","unstructured":"Kadlec R, Bajgar O, Kleindienst J. Knowledge Base Completion: Baselines Strike Back. Proceedings of the 2nd Workshop on Representation Learning for NLP. 2017.","DOI":"10.18653\/v1\/W17-2609"},{"key":"732_CR13","unstructured":"Bordes A, Usunier N, Garcia-Dur\u00e1n A, Weston J, Yakhnenko O. Translating embeddings for modeling multi-relational data. Adv Neural Inf Process Syst. 2013; 26."},{"issue":"11","key":"732_CR14","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1145\/219717.219748","volume":"38","author":"G Miller","year":"1995","unstructured":"Miller G. WordNet. Commun ACM. 1995;38(11):39\u201341.","journal-title":"Commun ACM"},{"key":"732_CR15","doi-asserted-by":"crossref","unstructured":"Dettmers T, Minervini P, Stenetorp P, Riedel S. Convolutional 2D knowledge graph embeddings. Proceedings of the AAAI Conference on Artificial Intelligence. 2018;32(1).","DOI":"10.1609\/aaai.v32i1.11573"},{"issue":"90001","key":"732_CR16","doi-asserted-by":"publisher","first-page":"267D","DOI":"10.1093\/nar\/gkh061","volume":"32","author":"O Bodenreider","year":"2004","unstructured":"Bodenreider O. The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Res. 2004;32(90001):267D \u2013 270.","journal-title":"Nucleic Acids Res"},{"key":"732_CR17","doi-asserted-by":"publisher","first-page":"8404","DOI":"10.1109\/ACCESS.2018.2886311","volume":"7","author":"S Sang","year":"2019","unstructured":"Sang S, Yang Z, Liu X, Wang L, Lin H, Wang J, et al. GrEDeL: a knowledge graph embedding based method for drug discovery from biomedical literatures. IEEE Access. 2019;7:8404\u201315.","journal-title":"IEEE Access"},{"key":"732_CR18","unstructured":"Trouillon T, Welbl J, Riedel S, Gaussier U, Bouchard G. Complex embeddings for simple link prediction. 2016. https:\/\/github.com\/ttrouill\/complex"},{"key":"732_CR19","unstructured":"Ruffinelli D, Broscheit S, Gemulla R. YOU can teach an old dog new tricks! ON training knowledge graph embeddings. 2019. https:\/\/github.com\/uma-pi1\/kge"},{"key":"732_CR20","doi-asserted-by":"crossref","unstructured":"Chen S, Liu X, Gao J, Jiao J, Zhang R, Ji Y. Hitter: Hierarchical transformers for knowledge graph embeddings. arXiv preprint arXiv:200812813. 2020;","DOI":"10.18653\/v1\/2021.emnlp-main.812"},{"issue":"4","key":"732_CR21","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1016\/0041-5553(67)90144-9","volume":"7","author":"I Sobol'","year":"1967","unstructured":"Sobol\u2019 I. On the distribution of points in a cube and the approximate evaluation of integrals. USSR Comput Math Math Phys. 1967;7(4):86\u2013112.","journal-title":"USSR Comput Math Math Phys"},{"key":"732_CR22","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa F, Michel V, Grisel O, Blondel M, Prettenhofer P, Weiss R, et al. Scikit-learn: machine learning in python. J Mach Learn Res. 2011;12:2825\u201330.","journal-title":"J Mach Learn Res"},{"issue":"9","key":"732_CR23","doi-asserted-by":"publisher","first-page":"97","DOI":"10.21105\/joss.00097","volume":"2","author":"J Herman","year":"2017","unstructured":"Herman J, Usher W. SALib: an open-source python library for sensitivity analysis. J Open Source Softw. 2017;2(9):97.","journal-title":"J Open Source Softw."},{"issue":"2","key":"732_CR24","doi-asserted-by":"publisher","first-page":"280","DOI":"10.1016\/S0010-4655(02)00280-1","volume":"145","author":"A Saltelli","year":"2002","unstructured":"Saltelli A. Making best use of model evaluations to compute sensitivity indices. Comput Phys Commun. 2002;145(2):280\u201397.","journal-title":"Comput Phys Commun"},{"key":"732_CR25","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1002\/psp4.6","volume":"4","author":"X-Y Zhang","year":"2015","unstructured":"Zhang X-Y, Trame MN, Lesko LJ, Schmidt S. Sobol sensitivity analysis: a tool to guide the development and evaluation of systems pharmacology models. CPT Pharmacometrics Syst Pharmacol. 2015;4:69\u201379.","journal-title":"CPT Pharmacometrics Syst Pharmacol"},{"key":"732_CR26","doi-asserted-by":"publisher","unstructured":"Lloyd O. Zenodo; 2022. https:\/\/doi.org\/10.5281\/zenodo.6627236","DOI":"10.5281\/zenodo.6627236"},{"key":"732_CR27","unstructured":"Papers with code - fb15k-237 benchmark (link prediction). https:\/\/paperswithcode.com\/sota\/link-prediction-on-fb15k-237?metric=MRR"},{"key":"732_CR28","unstructured":"Papers with code - wn18rr benchmark (link prediction). https:\/\/paperswithcode.com\/sota\/link-prediction-on-wn18rr?metric=MRR"},{"key":"732_CR29","unstructured":"Papers with code - umls benchmark (link prediction). https:\/\/paperswithcode.com\/sota\/link-prediction-on-umls?metric=MR"},{"key":"732_CR30","unstructured":"Lacroix T, Usunier N, Obozinski G. Canonical tensor decomposition for knowledge base completion. 35th International Conference on Machine Learning, ICML 2018. 2018;7:4475\u201386. http:\/\/arxiv.org\/abs\/1806.07297"},{"key":"732_CR31","unstructured":"Yu J, Wu L-Y. Understanding the network formation pattern for better link prediction. ArXiv. 2021;abs\/2110.08850."},{"issue":"3","key":"732_CR32","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1145\/3190580.3190586","volume":"51","author":"N Fuhr","year":"2018","unstructured":"Fuhr N. Some common mistakes in IR evaluation, and how they can be avoided. ACM SIGIR Forum. 2018;51(3):32\u201341.","journal-title":"ACM SIGIR Forum"},{"issue":"1","key":"732_CR33","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3451964.3451976","volume":"54","author":"T Sakai","year":"2020","unstructured":"Sakai T. On Fuhr\u2019s guideline for IR evaluation. ACM SIGIR Forum. 2020;54(1):1\u20138.","journal-title":"ACM SIGIR Forum"}],"container-title":["Journal of Big Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s40537-023-00732-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s40537-023-00732-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s40537-023-00732-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,6]],"date-time":"2023-05-06T04:05:43Z","timestamp":1683345943000},"score":1,"resource":{"primary":{"URL":"https:\/\/journalofbigdata.springeropen.com\/articles\/10.1186\/s40537-023-00732-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,6]]},"references-count":33,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["732"],"URL":"https:\/\/doi.org\/10.1186\/s40537-023-00732-5","relation":{},"ISSN":["2196-1115"],"issn-type":[{"value":"2196-1115","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,6]]},"assertion":[{"value":"9 August 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 April 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 May 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not Applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not Applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"59"}}