{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T18:28:44Z","timestamp":1767637724040,"version":"3.48.0"},"reference-count":37,"publisher":"Maximum Academic Press","license":[{"start":{"date-parts":[[2020,7,2]],"date-time":"2020-07-02T00:00:00Z","timestamp":1593648000000},"content-version":"unspecified","delay-in-days":183,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["The Knowledge Engineering Review"],"published-print":{"date-parts":[[2020]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    The quality of a dataset used for evaluating data linking methods, techniques, and tools depends on the availability of a set of mappings, called\n                    <jats:italic>reference alignment<\/jats:italic>\n                    , that is known to be correct. In particular, it is crucial that mappings effectively represent relations between pairs of entities that are indeed similar due to the fact that they denote the same object. Since the reliability of mappings is decisive in order to perform a fair evaluation of automatic linking methods and tools, we call this property of mappings as\n                    <jats:italic>mapping fairness<\/jats:italic>\n                    . In this article, we propose a crowd-based approach, called\n                    <jats:sans-serif>Crowd Quality<\/jats:sans-serif>\n                    (\n                    <jats:sans-serif>CQ<\/jats:sans-serif>\n                    ), for assessing the quality of data linking datasets by measuring the fairness of the mappings in the reference alignment. Moreover, we present a real experiment, where we evaluate two state-of-the-art data linking tools before and after the refinement of the reference alignment based on the\n                    <jats:sans-serif>CQ<\/jats:sans-serif>\n                    approach, in order to present the benefits deriving from the crowd assessment of mapping fairness.\n                  <\/jats:p>","DOI":"10.1017\/s0269888920000363","type":"journal-article","created":{"date-parts":[[2020,7,2]],"date-time":"2020-07-02T04:30:57Z","timestamp":1593664257000},"source":"Crossref","is-referenced-by-count":0,"title":["Crowd-assessing quality in uncertain data linking datasets"],"prefix":"10.48130","volume":"35","author":[{"given":"Daniel","family":"Faria","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4991-4984","authenticated-orcid":false,"given":"Alfio","family":"Ferrara","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9083-4599","authenticated-orcid":false,"given":"Ernesto","family":"Jim\u00e9nez-ruiz","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6594-6644","authenticated-orcid":false,"given":"Stefano","family":"Montanelli","sequence":"additional","affiliation":[]},{"given":"Catia","family":"Pesquita","sequence":"additional","affiliation":[]}],"member":"27968","published-online":{"date-parts":[[2020,7,2]]},"reference":[{"key":"S0269888920000363_ref37","unstructured":"Volz, J. , Bizer, C. , Gaedke, M. & Kobilarov, G. 2009. Silk-a link discovery framework for the web of data. In International Workshop on Linked Data on the Web (LDOW2009), Madrid, Spain. CEUR-WS.org."},{"key":"S0269888920000363_ref34","unstructured":"Saveta, T. , Daskalaki, E. , Flouris, G. , Fundulaki, I. , Herschel, M. & Ngonga Ngomo, A.-C. 2015. Pushing the limits of instance matching systems: a semantics-aware benchmark for linked data. In Proceedings of the 24th International Conference on World Wide Web, ACM, 105\u2013106."},{"key":"S0269888920000363_ref33","unstructured":"Sarasua, C. , Simperl, E. & Noy, N. F. 2012. CrowdMap: crowdsourcing ontology alignment with microtasks. In Proceedings of the 11th International Semantic Web Conference, Boston, MA, USA, 525\u2013541."},{"key":"S0269888920000363_ref32","unstructured":"R\u00f6der, M. , Saveta, T. , Fundulaki, I. & Ngomo, A.-C. N. (2017). Hobbit link discovery benchmarks. 12th International Workshop on Ontology Matching (OM 2017), Vienna, Austria."},{"key":"S0269888920000363_ref31","unstructured":"Paulheim, H. , Hertling, S. & Ritze, D. 2013. Towards evaluating interactive ontology matching tools. In Proceedings of the 10th Extended Semantic Web Conference, Montpellier, France, 31\u201345."},{"key":"S0269888920000363_ref27","unstructured":"Mortensen, J. M. 2013. Crowdsourcing Ontology Verification. In Proceedings of the 12th International Semantic Web Conference, Sydney, Australia, 448\u2013455."},{"key":"S0269888920000363_ref24","unstructured":"Jim\u00e9nez-Ruiz, E. , Grau, B. C. , Horrocks, I. et al. 2012b. Exploiting the UMLS metathesaurus in the ontology alignment evaluation initiative. In 2nd International Workshop on Exploiting Large Knowledge Repositories (E- LKR). CEUR- WS. org."},{"key":"S0269888920000363_ref23","unstructured":"Jim\u00e9nez-Ruiz, E. , Cuenca Grau, B. , Zhou, Y. & Horrocks, I. 2012a. Large-scale interactive ontology matching: algorithms and implementation. In European Conference on Artificial Intelligence (ECAI), 444\u2013449."},{"key":"S0269888920000363_ref10","unstructured":"Cruz, I. F. , Loprete, F. , Palmonari, M. , Stroe, C. & Taheri, A. 2014. Pay-as-you-go multi-user feedback model for ontology matching. In Proceedings of the 19th International Conference on Knowledge Engineering and Knowledge Management, Link\u2019oping, Sweden, 80\u201396."},{"key":"S0269888920000363_ref38","unstructured":"Cuenca Grau, B., Dragisic, Z., Eckert, K., Euzenat, J., Ferrara, A., Granada, R., Ivanova, V., Jim\u00e9nez-Ruiz, E., Kempf, A. O., Lambrix, P., Nikolov, A., Paulheim, H., Ritze, D., Scharffe, F., Shvaiko, P., Trojahn dos Santos, C. & Zamazal, O. 2013. Results of the ontology alignment evaluation initiative 2013. In 8th International Workshop on Ontology Matching (OM 2013), Sydney, Australia, 61\u2013100. CEUR-WS.org"},{"key":"S0269888920000363_ref8","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-25264-3_31"},{"key":"S0269888920000363_ref35","unstructured":"Thaler, S. , Simperl, E. P. B. & Siorpaes, K. 2011. SpotTheLink: a game for ontology alignment. In Proceedings of the 6th Conference on Professional Knowledge Management: From Knowledge to Action, Innsbruck, Austria, 246\u2013253."},{"key":"S0269888920000363_ref25","doi-asserted-by":"publisher","DOI":"10.1017\/S0269888919000080"},{"key":"S0269888920000363_ref6","doi-asserted-by":"publisher","DOI":"10.4135\/9781412985642"},{"key":"S0269888920000363_ref36","unstructured":"Van Dusen, D. A. , Chase, C. & Wise, J. A. 2016. System and method for fuzzy concept mapping, voting ontology crowd sourcing, and technology prediction. US Patent 9461876."},{"key":"S0269888920000363_ref28","unstructured":"Ngomo, A.-C. N. & Auer, S. 2011. Limesa time-efficient approach for large-scale link discovery on the web of data. In 22nd International Joint Conference on Artificial Intelligence, Barcelona, Spain."},{"key":"S0269888920000363_ref7","unstructured":"Castano, S. , Ferrara, A. , Genta, L. & Montanelli, S. 2016. Combining Crowd Consensus and User Trustworthiness for Managing Collective Tasks. Future Generation Computer Systems, 54."},{"key":"S0269888920000363_ref16","unstructured":"Ferrara, A. , Montanelli, S. , Noessner, J. Stuckenschmidt, H. 2011. Benchmarking matching applications on the semantic web. In Extended Semantic Web Conference. Springer, 108\u2013122."},{"key":"S0269888920000363_ref18","unstructured":"Genta, L. , Ferrara, A. & Montanelli, S. 2017. Consensus-based techniques for range-task resolution in crowdsourcing systems. In Proceedings of the 7th EDBT International Workshop on Linked Web Data Management, Venice, Italy."},{"key":"S0269888920000363_ref12","doi-asserted-by":"publisher","DOI":"10.1016\/j.websem.2013.05.002"},{"key":"S0269888920000363_ref1","unstructured":"Achichi, M. , Cheatham, M. , Dragisic, Z. , Euzenat, J. , Faria, D. , Ferrara, A. , Flouris, G. , Fundulaki, I. , Harrow, I. , Ivanova, V. , Jim\u00e9nez-Ruiz, E., Kuss, E., Lambrix, P., Leopold, H., Li, H., Meilicke, C., Montanelli, S., Pesquita, C., Saveta, T., Shvaiko, P., Splendiani, A., Stuckenschmidt, H., Todorov, K., Trojahn dos Santos, C. & Zamazal, O. 2016. Results of the ontology alignment evaluation initiative 2016. In 11th International Workshop on Ontology Matching (OM 2016), Kobe, Japan, 73\u2013129. CEUR-WS.org."},{"key":"S0269888920000363_ref2","unstructured":"Acosta, M. , Zaveri, A. , Simperl, E. , Kontokostas, D. , Auer, S. & Lehmann, J. 2013. Crowdsourcing linked data quality assessment. In Proceedings of the 12th International Semantic Web Conference, Sydney, Australia, 260\u2013276."},{"key":"S0269888920000363_ref30","unstructured":"Noy, N. F. , Mortensen, J. , Musen, M. A. & Alexander, P. R. 2013. Mechanical turk as an ontology engineer?: using microtasks as a component of an ontology-engineering workflow. In Proceedings of the 5th ACM Web Science Conference, Paris, France, 262\u2013271."},{"key":"S0269888920000363_ref9","unstructured":"Cheatham, M. & Hitzler, P. 2014. Conference v2.0: An uncertain version of the OAEI conference benchmark. In Proceedings of the 13th International Semantic Web Conference, Riva del Garda, Italy, 33\u201348."},{"key":"S0269888920000363_ref11","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46523-4_13"},{"key":"S0269888920000363_ref39","unstructured":"Estell\u00e9s-Arolas, E. & Guevara, F. G. L. 2012. Towards an integrated crowdsourcing definition. Journal of Information Science 38(2), 189\u2013200."},{"key":"S0269888920000363_ref3","unstructured":"Algergawy, A. , Cheatham, M., Faria, D., Ferrara, A., Fundulaki, I., Harrow, I., Hertling, S., Jim\u00e9nez-Ruiz, E., Karam, N., Khiat, N., Lambrix, P., Li, H., Montanelli, S., Paulheim, H., Pesquita, C., Saveta, T., Schmidt, D., Shvaiko, P., Splendiani, A., Thi\u00e9blin, E., Trojahn dos Santos, C., Vatascinov\u00e1, J., Zamazal, O. & Zhou, L. 2018. Results of the ontology alignment evaluation initiative 2018. In 13th International Workshop on Ontology Matching (OM 2018), Monterey, CA, USA, 76\u2013116. CEUR-WS.org."},{"key":"S0269888920000363_ref5","unstructured":"Bozzon, A. , Brambilla, M. , Ceri, S. & Mauri, A. 2013. Reactive crowdsourcing. In Proceedings of the 22nd International World Wide Web Conference (WWW 2013), Rio de Janeiro, Brazil, 153\u2013164."},{"key":"S0269888920000363_ref26","doi-asserted-by":"publisher","DOI":"10.1109\/EMR.2010.5559142"},{"key":"S0269888920000363_ref13","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-38721-0"},{"key":"S0269888920000363_ref20","first-page":"1","article-title":"The rise of crowdsourcing","volume":"14","author":"Howe","year":"2006","journal-title":"Wired Magazine"},{"key":"S0269888920000363_ref14","unstructured":"Euzenat, J. & Shvaiko, P. 2007. Ontology Matching, 18. Springer."},{"key":"S0269888920000363_ref17","doi-asserted-by":"publisher","DOI":"10.1038\/075414a0"},{"key":"S0269888920000363_ref21","unstructured":"Jim\u00e9nez-Ruiz, E. & Cuenca Grau, B. 2011. LogMap: logic-based and scalable ontology matching. In Proceedings of the 10th International Semantic Web Conference, Bonn, Germany, 273\u2013288."},{"key":"S0269888920000363_ref29","unstructured":"Noronha, J. , Hysen, E. , Zhang, H. & Gajos, K. Z. 2011. Platemate: crowdsourcing nutritional analysis from food photographs. In Proceeding of the 24th Symposium on User Interface Software and Technology, Santa Barbara, CA, USA, 1\u201312."},{"key":"S0269888920000363_ref15","unstructured":"Faria, D. , Pesquita, C. , Santos, E. , Palmonari, M. , Cruz, I. F. & Couto, F. M. 2013. The AgreementMakerLight ontology matching system. In OTM Conferences - ODBASE, 527\u2013541."},{"key":"S0269888920000363_ref22","doi-asserted-by":"publisher","DOI":"10.1186\/2041-1480-2-S1-S2"}],"container-title":["The Knowledge Engineering Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S0269888920000363","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T14:42:19Z","timestamp":1767624139000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S0269888920000363\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020]]},"references-count":37,"alternative-id":["S0269888920000363"],"URL":"https:\/\/doi.org\/10.1017\/s0269888920000363","relation":{},"ISSN":["0269-8889","1469-8005"],"issn-type":[{"type":"print","value":"0269-8889"},{"type":"electronic","value":"1469-8005"}],"subject":[],"published":{"date-parts":[[2020]]},"article-number":"e33"}}