{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T16:37:59Z","timestamp":1781109479707,"version":"3.54.1"},"reference-count":50,"publisher":"IGI Global Scientific Publishing","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,1]]},"abstract":"<jats:p>Software engineering employs different benchmarks for a software evaluation. This enables software developers to continuously improve their product. The same needs are intrinsic for software tools in the semantic web field. While there are many different benchmarks already available, there has not been their overview and categorization yet. This work provides such an overview and categorization of benchmarks specifically oriented on benchmarks where an ontology plays an important role. Benchmarks are naturally categorized in line with ontology tool categorization along with an indication which activities those benchmarks are deliberate and which are non-deliberative. Although the article itself can already navigate a reader to an adequate benchmark, we moreover automatically designed a flexible rule-based recommendation tool based on the analysis of existing benchmarks.<\/jats:p>","DOI":"10.4018\/ijswis.2020010103","type":"journal-article","created":{"date-parts":[[2019,12,6]],"date-time":"2019-12-06T10:44:36Z","timestamp":1575629076000},"page":"47-68","source":"Crossref","is-referenced-by-count":15,"title":["A Survey of Ontology Benchmarks for Semantic Web Ontology Tools"],"prefix":"10.4018","volume":"16","author":[{"given":"Ond\u0159ej","family":"Zamazal","sequence":"first","affiliation":[{"name":"University of Economics, Prague, Czech Republic"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"2432","reference":[{"key":"IJSWIS.2020010103-0","unstructured":"Achichi, M., Cheatham, M., Dragisic, Z., Euzenat, J., Faria, D., & Ferrara, A. others (2016). Results of the Ontology Alignment Evaluation Initiative 2016. Proceedings of the 11th ISWC workshop on ontology matching (OM) (pp. 73\u2013129). Academic Press."},{"key":"IJSWIS.2020010103-1","unstructured":"Antunes, G., Bakhshandeh, M., Borbinha, J., Cardoso, J., Dadashnia, S., Di Francescomarino, C., others (2015). The Process Model Matching Contest 2015. GI-Edition\/Proceedings: Lecture notes in informatics, 248, 127\u2013155."},{"key":"IJSWIS.2020010103-2","doi-asserted-by":"crossref","unstructured":"Bail, S., Parsia, B., & Sattler, U. (2010). JustBench: A Framework for OWL Benchmarking. Proceedings of theInternational semantic web conference (pp. 32\u201347). Academic Press.","DOI":"10.1007\/978-3-642-17746-0_3"},{"key":"IJSWIS.2020010103-3","doi-asserted-by":"publisher","DOI":"10.1038\/scientificamerican0501-34"},{"key":"IJSWIS.2020010103-4","unstructured":"Borst, W. N. (19979). Construction of engineering ontologies for knowledge sharing and reuse. 1997."},{"key":"IJSWIS.2020010103-5","doi-asserted-by":"publisher","DOI":"10.4018\/jswis.2009040101"},{"key":"IJSWIS.2020010103-6","first-page":"54","article-title":"Sesame: A Generic Architecture for Storing and Querying RDF and RDF Schema.","author":"J.Broekstra","year":"2002","journal-title":"International semantic web conference"},{"key":"IJSWIS.2020010103-7","doi-asserted-by":"publisher","DOI":"10.1016\/j.websem.2016.06.001"},{"key":"IJSWIS.2020010103-8","unstructured":"Carroll, J. J., & De Roo, J. (2004). OWL Web Ontology Language Test Cases. W3C."},{"key":"IJSWIS.2020010103-9","unstructured":"Cheatham, M., Dragisic, Z., Euzenat, J., Faria, D., Ferrara, A., & Flouris, G. others (2015). Results of the Ontology Alignment Evaluation Initiative 2015. Proceedings of the 10th ISWC workshop on ontology matching (OM) (pp. 60\u2013115). Academic Press."},{"key":"IJSWIS.2020010103-10","doi-asserted-by":"crossref","unstructured":"Cheatham, M., & Hitzler, P. (2014). Conference v2. 0: An uncertain version of the oaei conference benchmark. Proceedings of the International semantic web conference (pp. 33\u201348). Academic Press.","DOI":"10.1007\/978-3-319-11915-1_3"},{"key":"IJSWIS.2020010103-11","unstructured":"Dragisic, Z., Eckert, K., Euzenat, J., Faria, D., Ferrara, A., & Granada, R., others (2014). Results of the ontology alignment evaluation initiative 2014. Proceedings of the 9th international conference on ontology matching (pp. 61\u2013104). Academic Press."},{"key":"IJSWIS.2020010103-12","doi-asserted-by":"publisher","DOI":"10.1186\/s13326-017-0166-5"},{"key":"IJSWIS.2020010103-13","doi-asserted-by":"crossref","unstructured":"Dragisic, Z., Ivanova, V., Lambrix, P., Faria, D., Jim\u00e9nez-Ruiz, E., & Pesquita, C. (2016). User validation in ontology alignment. Proceedings of theInternational semantic web conference (pp. 200\u2013217). Academic Press.","DOI":"10.1007\/978-3-319-46523-4_13"},{"key":"IJSWIS.2020010103-14","unstructured":"Elhaik, Q., Rousset, M.-C., & Ycart, B. (1998). Generating Random Benchmarks for Description Logics. In Description logics. Academic Press."},{"key":"IJSWIS.2020010103-15","doi-asserted-by":"publisher","DOI":"10.1016\/j.websem.2013.05.002"},{"key":"IJSWIS.2020010103-16","unstructured":"Ferrara, A., Lorusso, D., Montanelli, S., & Varese, G. (2008). Towards a benchmark for instance matching. In P. Shvaiko, J. Euzenat, F. Giunchiglia, & H. Stuckenschmidt (Eds.), Ontology matching (OM 2008). CEUR-WS.org."},{"key":"IJSWIS.2020010103-17","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-21064-8_8"},{"key":"IJSWIS.2020010103-18","volume":"Vol. 3","author":"R.Garc\u00eda-Castro","year":"2009","journal-title":"Benchmarking Semantic Web technology"},{"key":"IJSWIS.2020010103-19","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2015.82"},{"key":"IJSWIS.2020010103-20","doi-asserted-by":"publisher","DOI":"10.1016\/j.websem.2011.12.007"},{"key":"IJSWIS.2020010103-21","doi-asserted-by":"crossref","unstructured":"G\u00f6rlitz, O., Thimm, M., & Staab, S. (2012). SPLODGE: Systematic Generation of SPARQL Benchmark Queries for Linked Open Data. Proceedings of the International semantic web conference (pp. 116\u2013132). Academic Press.","DOI":"10.1007\/978-3-642-35176-1_8"},{"issue":"2","key":"IJSWIS.2020010103-22","first-page":"158","article-title":"RDF Test Cases (Tech. Rep.). W3C. Guo, Y., Pan, Z., & Heflin, J. (2005). LUBM: A Benchmark for OWL Knowledge Base Systems.","volume":"3","author":"J.Grant","year":"2004","journal-title":"Journal of Web Semantics"},{"key":"IJSWIS.2020010103-23","doi-asserted-by":"crossref","unstructured":"Haag, F., Lohmann, S., Negru, S., & Ertl, T. (2014a). OntoViBe 2: Advancing the Ontology Visualization Benchmark. Proceedings of the International conference on knowledge engineering and knowledge management (pp. 83\u201398). Academic Press.","DOI":"10.1007\/978-3-319-17966-7_9"},{"key":"IJSWIS.2020010103-24","unstructured":"Haag, F., Lohmann, S., Negru, S., & Ertl, T. (2014b). OntoViBe: An Ontology Visualization Benchmark. In Visual@ ekaw (pp. 14\u201327). Academic Press."},{"key":"IJSWIS.2020010103-25","doi-asserted-by":"publisher","DOI":"10.1186\/s13326-017-0162-9"},{"key":"IJSWIS.2020010103-26","unstructured":"Hawke, S., & Parsia, B. (2009). OWL 2 Web Ontology Language Conformance. W3C."},{"key":"IJSWIS.2020010103-27","unstructured":"Hitzler, P., Kr\u00f6tzsch, M., Parsia, B., Patel-Schneider, P. F., & Rudolph S. (2009). OWL 2 web ontology language primer. W3C recommendation, 27(1), 123."},{"key":"IJSWIS.2020010103-28","unstructured":"Horrocks, I., & Patel-Schneider, P. F. (1998). DL systems comparison. Proc. of the 1998 description logic workshop (DL\u201998) (Vol. 11, pp. 55\u201357). Academic Press."},{"key":"IJSWIS.2020010103-29","first-page":"273","article-title":"Logmap","author":"E.Jim\u00e9nez","year":"2011","journal-title":"Logic-Based and Scalable Ontology Matching."},{"key":"IJSWIS.2020010103-30","first-page":"246","article-title":"Evaluating Mapping Repair Systems with Large Biomedical Ontologies.","volume":"13","author":"E.Jim\u00e9nez-Ruiz","year":"2013","journal-title":"Description Logics"},{"key":"IJSWIS.2020010103-31","doi-asserted-by":"publisher","DOI":"10.1186\/2041-1480-2-S1-S2"},{"key":"IJSWIS.2020010103-32","doi-asserted-by":"crossref","unstructured":"Link, V., Lohmann, S., & Haag, F. (2016). OntoBench: Generating Custom OWL 2 Benchmark Ontologies. Proceedings of the International semantic web conference (pp. 122\u2013130). Academic Press.","DOI":"10.1007\/978-3-319-46547-0_13"},{"key":"IJSWIS.2020010103-33","doi-asserted-by":"publisher","DOI":"10.3233\/SW-150200"},{"key":"IJSWIS.2020010103-34","doi-asserted-by":"crossref","unstructured":"Ma, L., Yang, Y., Qiu, Z., Xie, G., Pan, Y., & Liu, S. (2006). Towards a complete OWL ontology benchmark. Proceedings of theEuropean semantic web conference (pp. 125\u2013139). Academic Press.","DOI":"10.1007\/11762256_12"},{"key":"IJSWIS.2020010103-35","unstructured":"Matentzoglu, N., Tang, D., Parsia, B., & Sattler, U. (2014). The Manchester OWL repository: system description. Proceedings of the 2014 international conference on posters & demonstrations track (pp. 285\u2013288). Academic Press."},{"key":"IJSWIS.2020010103-36","doi-asserted-by":"publisher","DOI":"10.1016\/j.websem.2012.04.001"},{"key":"IJSWIS.2020010103-37","doi-asserted-by":"publisher","DOI":"10.1145\/2757001.2757003"},{"key":"IJSWIS.2020010103-38","unstructured":"Pan, Z. (2005, November). Benchmarking DL Reasoners Using Realistic Ontologies. In OWLED (Vol. 188). Academic Press."},{"key":"IJSWIS.2020010103-39","doi-asserted-by":"crossref","unstructured":"Parsia, B., Matentzoglu, N., Gon\u00e7alves, R. S., Glimm, B., & Steigmiller, A. (2016). The OWL reasoner evaluation (ore) 2015 resources. Proceedings of the International semantic web conference (pp. 159\u2013167). Academic Press.","DOI":"10.1007\/978-3-319-46547-0_17"},{"key":"IJSWIS.2020010103-40","unstructured":"Pesquita, C., Faria, D., Santos, E., & Couto, F. M. (2013). To repair or not to repair: reconciling correctness and coherence in ontology reference alignments. Proceedings of the 8th international conference on ontology matching (pp. 13\u201324). Academic Press."},{"key":"IJSWIS.2020010103-41","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-25073-6_37"},{"issue":"3","key":"IJSWIS.2020010103-42","doi-asserted-by":"crossref","first-page":"775","DOI":"10.1007\/s10115-016-0983-3","article-title":"Minimizing conservativity violations in ontology alignments: Algorithms and evaluation.","volume":"51","author":"A.Solimando","year":"2017","journal-title":"Knowledge and Information Systems"},{"key":"IJSWIS.2020010103-43","author":"S.Staab","year":"2013","journal-title":"Handbook on ontologies"},{"key":"IJSWIS.2020010103-44","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-24794-1_2"},{"key":"IJSWIS.2020010103-45","doi-asserted-by":"publisher","DOI":"10.1007\/11574620_54"},{"key":"IJSWIS.2020010103-46","unstructured":"Weiss, A. R. (2002). Dhrystone benchmark: History, analysis, scores and recommendations."},{"key":"IJSWIS.2020010103-47","doi-asserted-by":"crossref","unstructured":"Yatskevich, M. (2009). A large scale dataset for the evaluation of ontology matching systems. The Knowledge Engineering Review Journal, 24.","DOI":"10.1017\/S026988890900023X"},{"issue":"2","key":"IJSWIS.2020010103-48","first-page":"305","article-title":"Patomat-Versatile Framework for Pattern-Based Ontology Transformation.","volume":"34","author":"O.Zamazal","year":"2015","journal-title":"Computer Information"},{"key":"IJSWIS.2020010103-49","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1016\/j.websem.2017.01.001","article-title":"The ten-year ontofarm and its fertilization within the onto-sphere.","volume":"43","author":"O.Zamazal","year":"2017","journal-title":"Journal of Web Semantics"}],"container-title":["International Journal on Semantic Web and Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.igi-global.com\/viewtitle.aspx?TitleId=244187","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,5,6]],"date-time":"2022-05-06T04:01:03Z","timestamp":1651809663000},"score":1,"resource":{"primary":{"URL":"http:\/\/services.igi-global.com\/resolvedoi\/resolve.aspx?doi=10.4018\/IJSWIS.2020010103"}},"subtitle":[""],"short-title":[],"issued":{"date-parts":[[2020,1]]},"references-count":50,"journal-issue":{"issue":"1"},"URL":"https:\/\/doi.org\/10.4018\/ijswis.2020010103","relation":{},"ISSN":["1552-6283","1552-6291"],"issn-type":[{"value":"1552-6283","type":"print"},{"value":"1552-6291","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1]]}}}