{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,31]],"date-time":"2026-10-31T19:46:59Z","timestamp":1793476019625,"version":"4.5.0"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,6,17]],"date-time":"2022-06-17T00:00:00Z","timestamp":1655424000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,6,17]],"date-time":"2022-06-17T00:00:00Z","timestamp":1655424000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100010661","name":"EC | Horizon 2020 Framework Programme","doi-asserted-by":"publisher","award":["668353"],"award-info":[{"award-number":["668353"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"publisher"}]},{"name":"netidee grant number 5158"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Data"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Research in artificial intelligence (AI) is addressing a growing number of tasks through a rapidly growing number of models and methodologies. This makes it difficult to keep track of where novel AI methods are successfully \u2013 or still unsuccessfully \u2013 applied, how progress is measured, how different advances might synergize with each other, and how future research should be prioritized. To help address these issues, we created the Intelligence Task Ontology and Knowledge Graph (ITO), a comprehensive, richly structured and manually curated resource on artificial intelligence tasks, benchmark results and performance metrics. The current version of ITO contains 685,560 edges, 1,100 classes representing AI processes and 1,995 properties representing performance metrics. The primary goal of ITO is to enable analyses of the global landscape of AI tasks and capabilities. ITO is based on technologies that allow for easy integration and enrichment with external data, automated inference and continuous, collaborative expert curation of underlying ontological models. We make the ITO dataset and a collection of Jupyter notebooks utilizing ITO openly available.<\/jats:p>","DOI":"10.1038\/s41597-022-01435-x","type":"journal-article","created":{"date-parts":[[2022,6,17]],"date-time":"2022-06-17T10:04:33Z","timestamp":1655460273000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":44,"title":["A curated, ontology-based, large-scale knowledge graph of artificial intelligence tasks and benchmarks"],"prefix":"10.1038","volume":"9","author":[{"given":"Kathrin","family":"Blagec","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adriano","family":"Barbosa-Silva","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Simon","family":"Ott","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4855-2571","authenticated-orcid":false,"given":"Matthias","family":"Samwald","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,6,17]]},"reference":[{"key":"1435_CR1","unstructured":"Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. (2012)."},{"key":"1435_CR2","unstructured":"Vaswani, A. et al. Attention Is All You Need. arXiv (2017)."},{"key":"1435_CR3","unstructured":"Zhang, D. et al. The AI Index 2021 Annual Report. https:\/\/aiindex.stanford.edu\/wp-content\/uploads\/2021\/03\/2021-AI-Index-Report_Master.pdf (2021)."},{"key":"1435_CR4","unstructured":"Hogan, A. et al. Knowledge Graphs. arxiv (2020)."},{"key":"1435_CR5","unstructured":"RDF 1.1 Concepts and Abstract Syntax. https:\/\/www.w3.org\/TR\/rdf11-concepts\/."},{"key":"1435_CR6","unstructured":"OWL 2 Web Ontology Language Primer (Second Edition). https:\/\/www.w3.org\/TR\/owl2-primer\/."},{"key":"1435_CR7","unstructured":"SPARQL 1.1 Overview. https:\/\/www.w3.org\/TR\/sparql11-overview\/."},{"key":"1435_CR8","doi-asserted-by":"crossref","unstructured":"Ruttenberg, A. et al. Advancing translational research with the Semantic Web. BMC Bioinformatics 8 Suppl 3, S2 (2007).","DOI":"10.1186\/1471-2105-8-S3-S2"},{"key":"1435_CR9","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1186\/2041-1480-5-14","volume":"5","author":"M Dumontier","year":"2014","unstructured":"Dumontier, M. et al. The Semanticscience Integrated Ontology (SIO) for biomedical research and knowledge discovery. J. Biomed. Semantics 5, 14 (2014).","journal-title":"J. Biomed. Semantics"},{"key":"1435_CR10","doi-asserted-by":"publisher","unstructured":"Auer, S. et al. Towards a knowledge graph for science. in Proceedings of the 8th International Conference on Web Intelligence, Mining and Semantics - WIMS \u201918 (eds. Akerkar, R. et al.) 1\u20136, https:\/\/doi.org\/10.1145\/3227609.3227689 (ACM Press, 2018).","DOI":"10.1145\/3227609.3227689"},{"key":"1435_CR11","doi-asserted-by":"publisher","first-page":"e2005468","DOI":"10.1371\/journal.pbio.2005468","volume":"16","author":"JPA Ioannidis","year":"2018","unstructured":"Ioannidis, J. P. A. Meta-research: Why research on research matters. PLoS Biol. 16, e2005468 (2018).","journal-title":"PLoS Biol."},{"key":"1435_CR12","unstructured":"Blagec, K., Dorffner, G., Moradi, M. & Samwald, M. A critical analysis of metrics used for measuring progress in artificial intelligence. https:\/\/arxiv.org\/abs\/2008.02577 (2020)."},{"key":"1435_CR13","doi-asserted-by":"crossref","unstructured":"Blagec, K., Kraiger, J., Fr\u00fchwirt, W. & Samwald, M. Benchmark datasets driving artificial intelligence development fail to capture the needs of medical professionals. arXiv (2022).","DOI":"10.1016\/j.jbi.2022.104274"},{"key":"1435_CR14","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.4647824","author":"K Blagec","year":"2021","unstructured":"Blagec, K., Kraiger, J. & Samwald, M. A living catalogue of artificial intelligence datasets and benchmarks for medical decision making. Zenodo https:\/\/doi.org\/10.5281\/zenodo.4647824 (2021)."},{"key":"1435_CR15","doi-asserted-by":"publisher","first-page":"525","DOI":"10.1093\/bioinformatics\/bts718","volume":"29","author":"E Maguire","year":"2013","unstructured":"Maguire, E., Gonz\u00e1lez-Beltr\u00e1n, A., Whetzel, P. L., Sansone, S.-A. & Rocca-Serra, P. OntoMaton: a bioportal powered ontology widget for Google Spreadsheets. Bioinformatics 29, 525\u2013527 (2013).","journal-title":"Bioinformatics"},{"key":"1435_CR16","doi-asserted-by":"publisher","unstructured":"Horridge, M., Gon\u00e7alves, R. S., Nyulas, C. I., Tudorache, T. & Musen, M. A. WebProt\u00e9g\u00e9: A Cloud-Based Ontology Editor. in Companion Proceedings of The 2019 World Wide Web Conference on - WWW \u201919 (eds. Liu, L. & White, R.) 686\u2013689, https:\/\/doi.org\/10.1145\/3308560.3317707 (ACM Press, 2019).","DOI":"10.1145\/3308560.3317707"},{"key":"1435_CR17","doi-asserted-by":"publisher","first-page":"1325","DOI":"10.1093\/bioinformatics\/btt113","volume":"29","author":"J Ison","year":"2013","unstructured":"Ison, J. et al. EDAM: an ontology of bioinformatics operations, types of data and identifiers, topics and formats. Bioinformatics 29, 1325\u20131332 (2013).","journal-title":"Bioinformatics"},{"key":"1435_CR18","doi-asserted-by":"crossref","unstructured":"Tirmizi, S. H. et al. Mapping between the OBO and OWL ontology languages. J. Biomed. Semantics 2 Suppl 1, S3 (2011).","DOI":"10.1186\/2041-1480-2-S1-S3"},{"key":"1435_CR19","unstructured":"Dublin Core Metadata Initiative. Dublin Core Metadata Element Set, Version 1.1. (2012)."},{"key":"1435_CR20","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1300\/J104v43n03_10","volume":"43","author":"M Graves","year":"2007","unstructured":"Graves, M., Constabaris, A. & Brickley, D. FOAF: connecting people on the semantic web. Cataloging & Classification Quarterly 43, 191\u2013202 (2007).","journal-title":"Cataloging & Classification Quarterly"},{"key":"1435_CR21","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.5561989","author":"M Samwald","year":"2021","unstructured":"Samwald, M. & Blagec, K. Intelligence Task Ontology and Knowledge Graph (ITO). Zenodo https:\/\/doi.org\/10.5281\/zenodo.5561989 (2021)."},{"key":"1435_CR22","doi-asserted-by":"publisher","unstructured":"Raad, J. & Cruz, C. A survey on ontology evaluation methods. in Proceedings of the 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management 179\u2013186, https:\/\/doi.org\/10.5220\/0005591001790186 (SCITEPRESS - Science and and Technology Publications, 2015).","DOI":"10.5220\/0005591001790186"},{"key":"1435_CR23","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1145\/2757001.2757003","volume":"1","author":"MA Musen","year":"2015","unstructured":"Musen, M. A., Prot\u00e9g\u00e9 Team. The Prot\u00e9g\u00e9 Project: A Look Back and a Look Forward. AI Matters 1, 4\u201312 (2015).","journal-title":"AI Matters"},{"key":"1435_CR24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10817-013-9296-3","volume":"53","author":"Y Kazakov","year":"2014","unstructured":"Kazakov, Y., Kr\u00f6tzsch, M. & Siman\u010d\u00edk, F. The Incredible ELK. J. Autom. Reasoning 53, 1\u201361 (2014).","journal-title":"J. Autom. Reasoning"},{"key":"1435_CR25","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1016\/j.sbspro.2015.11.313","volume":"212","author":"G Aguado-de-Cea","year":"2015","unstructured":"Aguado-de-Cea, G., Montiel-Ponsoda, E., Poveda-Villal\u00f3n, M. & Giraldo-Pasmin, O. X. Lexicalizing ontologies: the issues behind the labels. Procedia - Social and Behavioral Sciences 212, 151\u2013158 (2015).","journal-title":"Procedia - Social and Behavioral Sciences"},{"key":"1435_CR26","unstructured":"G\u00f3mez-P\u00e9rez, A. Evaluation of Taxonomic Knowledge in Ontologies and Knowledge Bases. in Proceedings of the 12th Banff Knowledge Acquisition for Knowledge-Based Systems Workshop, Banff, Alberta, Canada (1999)."},{"key":"1435_CR27","doi-asserted-by":"publisher","first-page":"7","DOI":"10.4018\/ijswis.2014040102","volume":"10","author":"M Poveda-Villal\u00f3n","year":"2014","unstructured":"Poveda-Villal\u00f3n, M., G\u00f3mez-P\u00e9rez, A. & Su\u00e1rez-Figueroa, M. C. OOPS! (ontology pitfall scanner!). Int. J. Semant. Web Inf. Syst. 10, 7\u201334 (2014).","journal-title":"Int. J. Semant. Web Inf. Syst."},{"key":"1435_CR28","doi-asserted-by":"publisher","unstructured":"Vrande\u010di\u0107, D. Ontology Evaluation. in Handbook on Ontologies (eds. Staab, S. & Studer, R.) 293\u2013313, https:\/\/doi.org\/10.1007\/978-3-540-92673-3_13 (Springer Berlin Heidelberg, 2009).","DOI":"10.1007\/978-3-540-92673-3_13"},{"key":"1435_CR29","doi-asserted-by":"publisher","unstructured":"Lantow, B. OntoMetrics: Putting Metrics into Use for Ontology Evaluation. in Proceedings of the 8th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management 186\u2013191, https:\/\/doi.org\/10.5220\/0006084601860191 (SCITEPRESS - Science and and Technology Publications, 2016).","DOI":"10.5220\/0006084601860191"},{"key":"1435_CR30","doi-asserted-by":"publisher","first-page":"313","DOI":"10.3233\/SW-200422","volume":"12","author":"VA Carriero","year":"2021","unstructured":"Carriero, V. A. et al. Pattern-based design applied to cultural heritage knowledge graphs. SW 12, 313\u2013357 (2021).","journal-title":"SW"},{"key":"1435_CR31","unstructured":"Mart\u00ednez-Plumed, F., Hern\u00e1ndez-Orallo, J. & G\u00f3mez, E. Tracking AI: The Capability Is (Not) Near. in Proceedings of ECAI 2020 2915\u20132916 (IOS Press, 2020)."},{"key":"1435_CR32","doi-asserted-by":"publisher","first-page":"379","DOI":"10.1162\/dint_a_00055","volume":"2","author":"AA Salatino","year":"2020","unstructured":"Salatino, A. A. et al. The Computer Science Ontology: A Comprehensive Automatically-Generated Taxonomy of Research Areas. Data Intelligence 2, 379\u2013416 (2020).","journal-title":"Data Intelligence"},{"key":"1435_CR33","doi-asserted-by":"crossref","unstructured":"Han, K., Yang, P., Mishra, S. & Diesner, J. WikiCSSH: Extracting Computer Science Subject Headings from Wikipedia. in ADBIS, TPDL and EDA 2020 common workshops and doctoral consortium: international workshops: DOING, MADEISD, SKG, BBIGAP, SIMPDA, aiminscience 2020 and doctoral consortium, lyon, france, august 25\u201327, 2020, proceedings (eds. Bellatreche, L. et al.) vol. 1260 207\u2013218 (Springer International Publishing, 2020).","DOI":"10.1007\/978-3-030-55814-7_17"},{"key":"1435_CR34","doi-asserted-by":"publisher","first-page":"17","DOI":"10.3389\/fninf.2011.00017","volume":"5","author":"RA Poldrack","year":"2011","unstructured":"Poldrack, R. A. et al. The cognitive atlas: toward a knowledge foundation for cognitive neuroscience. Front. Neuroinformatics 5, 17 (2011).","journal-title":"Front. Neuroinformatics"},{"key":"1435_CR35","doi-asserted-by":"crossref","unstructured":"Dess\u00ec, D. et al. AI-KG: An Automatically Generated Knowledge Graph of Artificial Intelligence. in The semantic web \u2013 ISWC 2020: 19th international semantic web conference, athens, greece, november 2\u20136, 2020, proceedings, part II (eds. Pan, J. Z. et al.) vol. 12507 127\u2013143 (Springer International Publishing, 2020).","DOI":"10.1007\/978-3-030-62466-8_9"},{"key":"1435_CR36","doi-asserted-by":"publisher","unstructured":"Jaradeh, M. Y. et al. Open research knowledge graph: next generation infrastructure for semantic scholarly knowledge. in Proceedings of the 10th International Conference on Knowledge Capture - K-CAP \u201919 243\u2013246, https:\/\/doi.org\/10.1145\/3360901.3364435 (ACM Press, 2019).","DOI":"10.1145\/3360901.3364435"},{"key":"1435_CR37","doi-asserted-by":"publisher","unstructured":"Kuhn, T. et al. Nanopublications: A Growing Resource of Provenance-Centric Scientific Linked Data. in 2018 IEEE 14th International Conference on e-Science (e-Science) 83\u201392, https:\/\/doi.org\/10.1109\/eScience.2018.00024 (IEEE, 2018).","DOI":"10.1109\/eScience.2018.00024"},{"key":"1435_CR38","doi-asserted-by":"publisher","first-page":"4097","DOI":"10.1093\/bioinformatics\/btaa274","volume":"36","author":"A Breit","year":"2020","unstructured":"Breit, A., Ott, S., Agibetov, A. & Samwald, M. OpenBioLink: a benchmarking framework for large-scale biomedical link prediction. Bioinformatics 36, 4097\u20134098 (2020).","journal-title":"Bioinformatics"},{"key":"1435_CR39","doi-asserted-by":"crossref","unstructured":"Himmelstein, D. S. et al. Systematic integration of biomedical knowledge prioritizes drugs for repurposing. eLife 6 (2017).","DOI":"10.7554\/eLife.26726"},{"key":"1435_CR40","doi-asserted-by":"publisher","DOI":"10.1101\/2020.04.30.071407","author":"TJ Callahan","year":"2020","unstructured":"Callahan, T. J., Tripodi, I. J., Hunter, L. E. & Baumgartner, W. A. A Framework for Automated Construction of Heterogeneous Large-Scale Biomedical Knowledge Graphs. BioRxiv https:\/\/doi.org\/10.1101\/2020.04.30.071407 (2020).","journal-title":"BioRxiv"},{"key":"1435_CR41","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.6566103","author":"M Samwald","year":"2022","unstructured":"Samwald, M. et al. OpenBioLink\/ITO: v1.01. Zenodo https:\/\/doi.org\/10.5281\/zenodo.6566103 (2022)."}],"container-title":["Scientific Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41597-022-01435-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41597-022-01435-x","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41597-022-01435-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,8]],"date-time":"2023-02-08T18:22:26Z","timestamp":1675880546000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41597-022-01435-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,17]]},"references-count":41,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["1435"],"URL":"https:\/\/doi.org\/10.1038\/s41597-022-01435-x","relation":{"references":[{"id-type":"doi","id":"10.5281\/zenodo.4647824","asserted-by":"subject"},{"id-type":"doi","id":"10.5281\/zenodo.5561989","asserted-by":"subject"},{"id-type":"doi","id":"10.5281\/zenodo.6566103","asserted-by":"subject"}]},"ISSN":["2052-4463"],"issn-type":[{"value":"2052-4463","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,6,17]]},"assertion":[{"value":"19 October 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 May 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 June 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"322"}}