{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,5]],"date-time":"2026-08-05T21:16:13Z","timestamp":1785964573842,"version":"3.56.0"},"reference-count":27,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2021,2,4]],"date-time":"2021-02-04T00:00:00Z","timestamp":1612396800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100010661","name":"Horizon 2020","doi-asserted-by":"publisher","award":["770158"],"award-info":[{"award-number":["770158"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100013294","name":"Connecting Europe Facility","doi-asserted-by":"publisher","award":["1564820"],"award-info":[{"award-number":["1564820"]}],"id":[{"id":"10.13039\/100013294","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>The lack of granular and rich descriptive metadata highly affects the discoverability and usability of cultural heritage collections aggregated and served through digital platforms, such as Europeana, thus compromising the user experience. In this context, metadata enrichment services through automated analysis and feature extraction along with crowdsourcing annotation services can offer a great opportunity for improving the metadata quality of digital cultural content in a scalable way, while at the same time engaging different user communities and raising awareness about cultural heritage assets. To address this need, we propose the CrowdHeritage open end-to-end enrichment and crowdsourcing ecosystem, which supports an end-to-end workflow for the improvement of cultural heritage metadata by employing crowdsourcing and by combining machine and human intelligence to serve the particular requirements of the cultural heritage domain. The proposed solution repurposes, extends, and combines in an innovative way general-purpose state-of-the-art AI tools, semantic technologies, and aggregation mechanisms with a novel crowdsourcing platform, so as to support seamless enrichment workflows for improving the quality of CH metadata in a scalable, cost-effective, and amusing way.<\/jats:p>","DOI":"10.3390\/info12020064","type":"journal-article","created":{"date-parts":[[2021,2,4]],"date-time":"2021-02-04T12:02:10Z","timestamp":1612440130000},"page":"64","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["CrowdHeritage: Crowdsourcing for Improving the Quality of Cultural Heritage Metadata"],"prefix":"10.3390","volume":"12","author":[{"given":"Eirini","family":"Kaldeli","sequence":"first","affiliation":[{"name":"Artificial Intelligence and Learning Systems Laboratory, School of Electrical and Computer Engineering, National Technical University of Athens, 15780 Athens, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Orfeas","family":"Menis-Mastromichalakis","sequence":"additional","affiliation":[{"name":"Artificial Intelligence and Learning Systems Laboratory, School of Electrical and Computer Engineering, National Technical University of Athens, 15780 Athens, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2722-5421","authenticated-orcid":false,"given":"Spyros","family":"Bekiaris","sequence":"additional","affiliation":[{"name":"Artificial Intelligence and Learning Systems Laboratory, School of Electrical and Computer Engineering, National Technical University of Athens, 15780 Athens, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maria","family":"Ralli","sequence":"additional","affiliation":[{"name":"Artificial Intelligence and Learning Systems Laboratory, School of Electrical and Computer Engineering, National Technical University of Athens, 15780 Athens, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vassilis","family":"Tzouvaras","sequence":"additional","affiliation":[{"name":"Artificial Intelligence and Learning Systems Laboratory, School of Electrical and Computer Engineering, National Technical University of Athens, 15780 Athens, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Giorgos","family":"Stamou","sequence":"additional","affiliation":[{"name":"Artificial Intelligence and Learning Systems Laboratory, School of Electrical and Computer Engineering, National Technical University of Athens, 15780 Athens, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,2,4]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Ganea, O., Ganea, M., Lucchi, A., Eickhoff, C., and Hofmann, T. (2016, January 11\u201315). Probabilistic Bag-Of-Hyperlinks Model for Entity Linking. Proceedings of the 25th International Conference on World Wide Web WWW, Montr\u00e9al, QC, Canada.","DOI":"10.1145\/2872427.2882988"},{"key":"ref_2","unstructured":"Carmel, D., Chang, M., Gabrilovich, E., Hsu, B.P., and Wang, K. (2014, January 11). From TagME to WAT: A new entity annotator. Proceedings of the First ACM International Workshop on Entity Recognition & Disambiguation ERD\u201914, Gold Coast, QLD, Australia."},{"key":"ref_3","unstructured":"Mandalios, A., Tzamaloukas, K., Chortaras, A., and Stamou, G. (2018, January 23). GEEK: Incremental Graph-based Entity Disambiguation. Proceedings of the Linked Data on the Web Co-Located with the 2018 Web Conference-LDOW@WWW, Lyon, France."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Varytimidis, C., Tsatiris, G., Rapantzikos, K., and Kollias, S. (2016, January 6\u20139). A systemic approach to automatic metadata extraction from multimedia content. Proceedings of the 2016 IEEE Symposium Series on Computational Intelligence-SSCI, Athens, Greece.","DOI":"10.1109\/SSCI.2016.7849983"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Amid, E., Mesaros, A., Palom\u00e4ki, K., Laaksonen, J., and Kurimo, M. (2014, January 4\u20139). Unsupervised feature extraction for multimedia event detection and ranking using audio content. Proceedings of the ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing-Proceedings, Florence, Italy.","DOI":"10.1109\/ICASSP.2014.6854743"},{"key":"ref_6","unstructured":"Fillon, T., Simonnot, J., Mifune, M.F., Khoury, S., Pellerin, G., Le Coz, M., Amy de la Bret\u00e8que, E., Doukhan, D., Fourer, D., and Rouas, J.L. (2014, January 12). Telemeta: An open-source web framework for ethnomusicological audio archives management and automatic analysis. Proceedings of the 1st Digital Libraries for Musicology workshop (DLfM 2014), London, UK."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1007\/978-3-319-98192-5_57","article-title":"Exploring Linked Data for the Automatic Enrichment of Historical Archives","volume":"Volume 11155","author":"Munnelly","year":"2018","journal-title":"Lecture Notes in Computer Science, Proceedings of the Semantic Web: ESWC 2018 Satellite Events, Heraklion, Greece, 3\u20137 June 2018"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Dijkshoorn, C., Boer, V., Aroyo, L., and Schreiber, G. (2017). Accurator: Nichesourcing for Cultural Heritage. Hum. Comput., 6.","DOI":"10.15346\/hc.v6i1.91"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ralli, M., Bekiaris, S., Kaldeli, E., Menis-Mastromichalakis, O., Sofou, N., Tzouvaras, V., and Stamou, G. (2020, January 29\u201330). CrowdHeritage: Improving the quality of Cultural Heritage through crowdsourcing methods. Proceedings of the 15th International Workshop on Semantic and Social Media Adaptation and Personalization (SMAP 2020), Zakynthos, Greece.","DOI":"10.1109\/SMAP49528.2020.9248463"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Ioannides, M., Magnenat-Thalmann, N., Fink, E., \u017darni\u0107, R., Yen, A.Y., and Quak, E. (2014). Automatic Enrichments with Controlled Vocabularies in Europeana: Challenges and Consequences. Digital Heritage. Progress in Cultural Heritage: Documentation, Preservation, and Protection, Springer International Publishing.","DOI":"10.1007\/978-3-319-13695-0"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Massi, M., and Marilena Vecco, Y.L. (2020). Crowdsourcing the digital transformation of heritage. Digital Transformation in the Cultural and Creative Industries, Taylor and Francis. Chapter 4.","DOI":"10.4324\/9780429329852"},{"key":"ref_12","first-page":"59","article-title":"A review of online crowdsourcing platforms","volume":"14","author":"Mourelatos","year":"2016","journal-title":"South-East. Eur. J. Econ."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Simpson, R.J., Page, K.R., and Roure, D.D. (2014, January 7\u201311). Zooniverse: Observing the world\u2019s largest citizen science platform. Proceedings of the 23rd International World Wide Web Conference, WWW \u201914, Seoul, Korea.","DOI":"10.1145\/2567948.2579215"},{"key":"ref_14","unstructured":"Ellis, A., Gluckman, D., Cooper, A., and Andrew, G. Your Paintings: A nation\u2019s Oil Paintings Go Online, Tagged by the Public. Museums and the Web Conference, Available online: https:\/\/www.museumsandtheweb.com\/mw2012\/papers\/your_paintings_a_nation_s_oil_paintings_go_onl.html."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1179\/2051819613Z.0000000003","article-title":"Picture This! Community-Led Production of Alternative Views of the Heritage of Gwynedd","volume":"1","author":"Karl","year":"2014","journal-title":"J. Community Archaeol. Herit."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Vincent, M., Flores Guti\u00e9rrez, M., Coughenour, C., Manuel, V., Bendicho, V.M., Remondino, F., and Fritsch, D. (October, January 28). Crowd-sourcing the 3D Digital Reconstructions of Lost Cultural Heritage. Proceedings of the 2015 Digital Heritage, Granada, Spain.","DOI":"10.1109\/DigitalHeritage.2015.7413863"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Dhonju, H., Xiao, W., Mills, J., and Sarhosis, V. (2018). Share Our Cultural Heritage (SOCH): Worldwide 3D Heritage Reconstruction and Visualization via Web and Mobile GIS. ISPRS Int. J. Geo-Inf., 7.","DOI":"10.3390\/ijgi7090360"},{"key":"ref_18","unstructured":"Chun, S., Cherry, R., Hiwiller, D., Trant, J., and Wyman, B. (2006). Steve.museum: An Ongoing Experiment in Social Tagging, Folksonomy, and Museums. Museums and the Web 2006: Proceedings, Archives and Museum Informatics."},{"key":"ref_19","unstructured":"Oomen, J., and Aroyo, L. (July, January 29). Crowdsourcing in the Cultural Heritage Domain: Opportunities and Challenges. Proceedings of the 5th International Conference on Communities and Technologies, Brisbane, Australia."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Gligorov, R., Hildebrand, M., Van Ossenbruggen, J., Aroyo, L., and Schreiber, G. (2013, January 24\u201327). An Evaluation of Labelling-Game Data for Video Retrieval. Proceedings of the 35th European Conference on Advances in Information Retrieval, Moscow, Russia.","DOI":"10.1007\/978-3-642-36973-5_5"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"215","DOI":"10.15514\/ISPRAS-2018-30(2)-11","article-title":"Active learning and crowdsourcing: A survey of annotation optimization methods","volume":"30","author":"Gilyazev","year":"2018","journal-title":"Proc. Inst. Syst. Program. RAS"},{"key":"ref_22","first-page":"486","article-title":"CrowdTruth: Machine-Human Computation Framework for Harnessing Disagreement in Gathering Annotated Data","volume":"Volume 8797","author":"Inel","year":"2014","journal-title":"Lecture Notes in Computer Science, Proceedings of the ISWC 2014-13th International Semantic Web Conference, Riva del Garda, Italy, 19\u201323 October 2014"},{"key":"ref_23","unstructured":"Drosopoulos, N., Tzouvaras, V., Simou, N., Christaki, A., Stabenau, A., Pardalis, K., Xenikoudakis, F., and Kollias, S. (2012). A Metadata Interoperability Platform. Museums and the Web 2012, Springer."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Chortaras, A., Christaki, A., Drosopoulos, N., Kaldeli, E., Ralli, M., Sofou, A., Stabenau, A., Stamou, G., and Tzouvaras, V. (2018, January 23\u201327). WITH: Human-Computer Collaboration for Data Annotation and Enrichment. Proceedings of the Companion of the The Web Conference 2018, Lyon, France.","DOI":"10.1145\/3184558.3191544"},{"key":"ref_25","unstructured":"Giazitzoglou, M., Tzouvaras, V., Chortaras, A., Alfonso, E., and Drosopoulos, N. (2017, January 11\u201314). The WITH platform: Where culture meets creativity. Proceedings of the Posters and Demos Track of the 13th International Conference on Semantic Systems-SEMANTiCS 2017, Amsterdam, The Netherlands."},{"key":"ref_26","unstructured":"Hoffart, J., Yosef, M., Bordino, I., Fuerstenau, H., Pinkal, M., Spaniol, M., Taneva, B., Thater, S., and Weikum, G. (2011, January 27\u201331). Robust Disambiguation of Named Entities in Text. Proceedings of the EMNLP 2011-Conference on Empirical Methods in Natural Language Processing, Edinburgh, Scotland, UK."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Menis-Mastromichalakis, O., Sofou, N., and Stamou, G. (2020, January 19\u201324). Deep Ensemble Art Style Recognition. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.","DOI":"10.1109\/IJCNN48605.2020.9207645"}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/12\/2\/64\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:20:01Z","timestamp":1760160001000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/12\/2\/64"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,4]]},"references-count":27,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2021,2]]}},"alternative-id":["info12020064"],"URL":"https:\/\/doi.org\/10.3390\/info12020064","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,4]]}}}