{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,26]],"date-time":"2026-01-26T15:42:58Z","timestamp":1769442178345,"version":"3.49.0"},"reference-count":46,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2018,6,8]],"date-time":"2018-06-08T00:00:00Z","timestamp":1528416000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2018,8,31]]},"abstract":"<jats:p>Ontology alignment is widely used to find the correspondences between different ontologies in diverse fields. After discovering the alignments, several performance scores are available to evaluate them. The scores typically require the identified alignment and a reference containing the underlying actual correspondences of the given ontologies. The current trend in the alignment evaluation is to put forward a new score (e.g., precision, weighted precision, semantic precision, etc.) and to compare various alignments by juxtaposing the obtained scores. However, it is substantially provocative to select one measure among others for comparison. On top of that, claiming if one system has a better performance than one another cannot be substantiated solely by comparing two scalars. In this article, we propose the statistical procedures that enable us to theoretically favor one system over one another. The McNemar\u2019s test is the statistical means by which the comparison of two ontology alignment systems over one matching task is drawn. The test applies to a 2 \u00d7 2 contingency table, which can be constructed in two different ways based on the alignments, each of which has their own merits\/pitfalls. The ways of the contingency table construction and various apposite statistics from the McNemar\u2019s test are elaborated in minute detail. In the case of having more than two alignment systems for comparison, the family wise error rate is expected to happen. Thus, the ways of preventing such an error are also discussed. A directed graph visualizes the outcome of the McNemar\u2019s test in the presence of multiple alignment systems. From this graph, it is readily understood if one system is better than one another or if their differences are imperceptible. The proposed statistical methodologies are applied to the systems participated in the OAEI 2016 anatomy track, and also compares several well-known similarity metrics for the same matching problem.<\/jats:p>","DOI":"10.1145\/3193573","type":"journal-article","created":{"date-parts":[[2018,6,11]],"date-time":"2018-06-11T12:20:54Z","timestamp":1528719654000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Comparison of Ontology Alignment Systems Across Single Matching Task Via the McNemar\u2019s Test"],"prefix":"10.1145","volume":"12","author":[{"given":"Majid","family":"Mohammadi","sequence":"first","affiliation":[{"name":"Delft University of Technology, Zuid-Hollands, The Netherlands"}]},{"given":"Amir Ahooye","family":"Atashin","sequence":"additional","affiliation":[{"name":"Ferdowsi University of Mashhad, Razavi Khorasan, Iran"}]},{"given":"Wout","family":"Hofman","sequence":"additional","affiliation":[{"name":"TNO Research Institute, Zuid-Hollands, The Netherlands"}]},{"given":"Yaohua","family":"Tan","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Zuid-Hollands, The Netherlands"}]}],"member":"320","published-online":{"date-parts":[[2018,6,8]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the 11th ISWC Workshop on Ontology Matching (OM\u201916)","author":"Achichi Manel","year":"2016"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1504\/IJMSO.2016.081585"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-52307-6_8"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-41338-4_19"},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the KDD Workshop on Data Cleaning and Object Consolidation","volume":"3","author":"Cohen William","year":"2003"},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the Twelfth International Workshop on Ontology Matching (OM'17)","author":"da Silva Jomar","year":"2016"},{"key":"e_1_2_1_7_1","volume-title":"Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7(Jan","author":"Dem\u0161ar Janez","year":"2006"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICMWI.2010.5648054"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1961.10482090"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02289261"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-25956-5_6"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.5555\/1625275.1625330"},{"key":"e_1_2_1_13_1","unstructured":"J\u00e9r\u00f4me Euzenat Pavel Shvaiko and others. 2007. Ontology Matching Vol. 18. Springer.   J\u00e9r\u00f4me Euzenat Pavel Shvaiko and others. 2007. Ontology Matching Vol. 18. Springer."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-13-91"},{"key":"e_1_2_1_15_1","volume-title":"Couto","author":"Faria Daniel","year":"2013"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1993.10476358"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/75.4.800"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.2307\/2531823"},{"key":"e_1_2_1_19_1","volume-title":"A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics","author":"Holm Sture","year":"1979"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCC.2012.2209869"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1002\/sim.4780140510"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/2063016.2063035"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/11575832_13"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1961.10482105"},{"key":"e_1_2_1_25_1","first-page":"707","article-title":"Binary codes capable of correcting deletions, insertions, and reversals","volume":"10","author":"Levenshtein Vladimir I.","year":"1966","journal-title":"Soviet Physics Doklady"},{"key":"e_1_2_1_26_1","volume-title":"Shiva: A framework for graph based ontology matching. arxiv:1403.7465.","author":"Mathur Iti","year":"2014"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02295996"},{"key":"e_1_2_1_28_1","volume-title":"LPHOM results for OAEI","author":"Megdiche Imen","year":"2016"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/219717.219748"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2018.2842019"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1137\/0105003"},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the 1st International Workshop on Ontology Matching (OM\u201906)","author":"Nagy Miklos","year":"2006"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/0022-2836(70)90057-4"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-41338-4_25"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1986.10478341"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.ps.46.020195.003021"},{"key":"e_1_2_1_38_1","volume-title":"Handbook of Parametric and Nonparametric Statistical Procedures","author":"Sheskin David J."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/11574620_45"},{"key":"e_1_2_1_40_1","volume-title":"Shuster","author":"Suissa Samy","year":"1991"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the 3rd International Conference on Ontology Matching","volume":"431","author":"Wang Peng","year":"2008"},{"key":"e_1_2_1_42_1","volume-title":"Statistical Research Division","author":"Winkler William E."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1996.8.7.1341"},{"key":"e_1_2_1_44_1","volume-title":"What the no free lunch theorems really mean","author":"Wolpert David H."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/4235.585893"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1177\/0165551514550142"},{"key":"e_1_2_1_47_1","volume-title":"FCA-Map results for OAEI","author":"Zhao Mengyi","year":"2016"}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3193573","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3193573","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T17:49:14Z","timestamp":1750268954000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3193573"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,6,8]]},"references-count":46,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2018,8,31]]}},"alternative-id":["10.1145\/3193573"],"URL":"https:\/\/doi.org\/10.1145\/3193573","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"value":"1556-4681","type":"print"},{"value":"1556-472X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,6,8]]},"assertion":[{"value":"2017-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-06-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}