{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T02:34:24Z","timestamp":1760236464442,"version":"build-2065373602"},"reference-count":49,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2021,12,2]],"date-time":"2021-12-02T00:00:00Z","timestamp":1638403200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Multimedia feature graphs are employed to represent features of images, video, audio, or text. Various techniques exist to extract such features from multimedia objects. In this paper, we describe the extension of such a feature graph to represent the meaning of such multimedia features and introduce a formal context-free PS-grammar (Phrase Structure grammar) to automatically generate human-understandable natural language expressions based on such features. To achieve this, we define a semantic extension to syntactic multimedia feature graphs and introduce a set of production rules for phrases of natural language English expressions. This explainability, which is founded on a semantic model provides the opportunity to represent any multimedia feature in a human-readable and human-understandable form, which largely closes the gap between the technical representation of such features and their semantics. We show how this explainability can be formally defined and demonstrate the corresponding implementation based on our generic multimedia analysis framework. Furthermore, we show how this semantic extension can be employed to increase the effectiveness in precision and recall experiments.<\/jats:p>","DOI":"10.3390\/info12120502","type":"journal-article","created":{"date-parts":[[2021,12,2]],"date-time":"2021-12-02T21:19:08Z","timestamp":1638479948000},"page":"502","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Towards Automated Semantic Explainability of Multimedia Feature Graphs"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2100-7589","authenticated-orcid":false,"given":"Stefan","family":"Wagenpfeil","sequence":"first","affiliation":[{"name":"Faculty of Mathematics and Computer Science, University of Hagen, D-58097 Hagen, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9715-1590","authenticated-orcid":false,"given":"Paul","family":"Mc Kevitt","sequence":"additional","affiliation":[{"name":"Academy for International Science & Research (AISR), Londonderry BT48 7ER, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8293-2802","authenticated-orcid":false,"given":"Matthias","family":"Hemmje","sequence":"additional","affiliation":[{"name":"Faculty of Mathematics and Computer Science, University of Hagen, D-58097 Hagen, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,12,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Kwasnicka, H., and Jain, L.C. (2018). Bridging the Semantic Gap in Image and Video Analysis, Springer.","DOI":"10.1007\/978-3-319-73891-8"},{"key":"ref_2","unstructured":"Clement, J. (2020, August 23). Social Media\u2014Statistics & Facts. Available online: https:\/\/www.statista.com\/topics\/1164\/social-networks\/."},{"key":"ref_3","unstructured":"Sony Electronics Inc (2021, November 22). a7R IV 35 mm Full-Frame Camera with 61.0 MP. Available online: https:\/\/www.sony.com\/electronics\/ interchangeable-lens-cameras\/ilce-7rm4."},{"key":"ref_4","unstructured":"Xiaomi (2021, November 22). Redmi Note 10 Pro\u2014The 108MP Voyager. Available online: https:\/\/www.mi.com\/global\/product\/redmi-note-10-pro\/overview."},{"key":"ref_5","unstructured":"The Washington Post (2021, November 22). Washington Post Archives. Available online: https:\/\/www.washingtonpost.com."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Domingue, J. (2011). Introduction to the Semantic Web Technologies. Handbook of Semantic Web Technologies, Springer.","DOI":"10.1007\/978-3-540-92913-0"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Wagenpfeil, S., Engel, F., Kevitt, P.M., and Hemmje, M. (2021). AI-Based Semantic Multimedia Indexing and Retrieval for Social Media on Smartphones. Information, 12.","DOI":"10.3390\/info12010043"},{"key":"ref_8","unstructured":"Wagenpfeil, S. (2020, August 23). GMAF Prototype. Available online: http:\/\/diss.step2e.de:8080\/GMAFWeb\/."},{"key":"ref_9","unstructured":"Wagenpfeil, S., Engel, F., Kevitt, P.M., and Hemmje, M. (2021, November 28). Github Repository of GMAF and MMFVG. Available online: https:\/\/github.com\/stefanwagenpfeil\/GMAF\/."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Wagenpfeil, S., and Hemmje, M. (2020, January 29\u201330). Towards AI-bases Semantic Multimedia Indexing and Retrieval for Social Media on Smartphones. Proceedings of the 2020 15th International Workshop on Semantic and Social Media Adaptation and Personalization, Zakynthos, Greece.","DOI":"10.1109\/SMAP49528.2020.9248445"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Beyerer, J., Richter, M., and Nagel, M. (2017). Pattern Recognition\u2014Introduction, Walter de Gruyter GmbH & Co KG.","DOI":"10.1515\/9783110537949"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Manotumruksa, J., Macdonald, C., and Ounis, I. (2018, January 8\u201312). A contextual attention recurrent architecture for context-aware venue recommendation. Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, Ann Arbor, MI, USA.","DOI":"10.1145\/3209978.3210042"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Leveling, J. (2013, January 28). Interpretation of coordinations, compound generation, and result fusion for query variants. Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland.","DOI":"10.1145\/2484028.2484115"},{"key":"ref_14","first-page":"35","article-title":"Multimedia Indexing and Retrieval Techniques: A Review","volume":"58","author":"Bhute","year":"2012","journal-title":"Int. J. Comput. Appl."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1126004.1126005","article-title":"Content-based multimedia information retrieval: State of the art and challenges","volume":"2","author":"Lew","year":"2006","journal-title":"ACM Trans. Multimed. Comput. Commun. Appl."},{"key":"ref_16","unstructured":"Hern\u00e1ndez-Gracidas, C., Ju\u00e1rez, A., Sucar, L.E., Montes-y-G\u00f3mez, M., and Villase\u00f1or, L. (2010). Data Fusion and Label Weighting for Image Retrieval Based on Spatio-Conceptual Information. Adapt. Pers. Fusion Heterog. Inf., 76\u201379."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Dufour, R., Esteve, Y., Del\u00e9glise, P., and B\u00e9chet, F. (2009, January 13\u201317). Local and global models for spontaneous speech segment detection and characterization. Proceedings of the Workshop on Automatic Speech Recognition and Understanding, Merano, Italy.","DOI":"10.1109\/ASRU.2009.5372928"},{"key":"ref_18","unstructured":"Subrahmanian, V. (1998). Principles of Multimedia Database Systems, Morgan Kaufmann."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"688","DOI":"10.1109\/76.927421","article-title":"Overview of the MPEG-7 standard","volume":"11","author":"Chang","year":"2001","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_20","unstructured":"FFMpeg.org (2020, August 23). ffmpeg Documentation. Available online: http:\/\/ffmpeg.org."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Mu, X. (2006, January 6\u201311). Content-based video retrieval: Does video\u2019s semantic visual feature matter?. Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, WA, USA.","DOI":"10.1145\/1148170.1148314"},{"key":"ref_22","unstructured":"Wagenpfeil, S., Engel, F., McKevitt, P., and Hemmje, M. (2020, August 26). Graph Codes\u20142D projections of Multimedia Feature Graphs for Fast and Effective Retrieval. Available online: http:\/\/164.177.153.44\/pubs\/icivr2021.pdf."},{"key":"ref_23","unstructured":"yWorks GmbH (2020, August 23). yEd Graph Editor. Available online: https:\/\/www.yworks.com\/products\/yed."},{"key":"ref_24","unstructured":"Wagenpfeil, S., Engel, F., McKevitt, P., and Hemmje, M.L. (2021, July 28). Semantic Query Construction and Result Representation based on Graph Codes. Available online: http:\/\/ceur-ws.org\/Vol-2863\/paper-06."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Kapetanios, E., and Groenewoud, P. (2002). Query Construction through Meaningful Suggestions of Terms. Flexible Query Answering Systems, Springer.","DOI":"10.1007\/3-540-36109-X_18"},{"key":"ref_26","unstructured":"Kapetanios, E., Baer, D., Groenewoud, P., and Mueller, P. (July, January 30). The design and implementation of a meaning driven data query language. Proceedings of the 14th International Conference on Scientific and Statistical Database Management, Aalborg, Denmark."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"347","DOI":"10.14445\/22315381\/IJETT-V41P263","article-title":"Transforming Natural Language Query to SPARQL for Semantic Information Retrieval","volume":"41","author":"Shaik","year":"2016","journal-title":"Int. J. Eng. Trends Technol."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Jung, H., and Kim, W. (2020). Automated conversion from natural language query to SPARQL query. J. Intell. Inf. Syst., 1\u201320.","DOI":"10.1007\/s10844-019-00589-2"},{"key":"ref_29","unstructured":"W3C.org (2020, August 23). W3C Semantic Web Activity. Available online: http:\/\/w3.org\/2001\/sw."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1933","DOI":"10.1109\/TPAMI.2008.125","article-title":"Automatic Semantic Annotation of Real-World Web Images","volume":"30","author":"Wong","year":"2008","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Ni, J., Qian, X., Li, Q., and Xu, X. (2017). Research on Semantic Annotation Based Image Fusion Algorithm. Res. Semant. Annot. Based Image Fusion Algorithm, 945\u2013948.","DOI":"10.1109\/ICCSEC.2017.8446720"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"21662","DOI":"10.1109\/ACCESS.2019.2897849","article-title":"The Use of Ontology in Retrieval: A Study on Textual","volume":"7","author":"Asim","year":"2019","journal-title":"IEEE Access"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Beierle, C., and Kern-Isberner, G. (2019). Methoden Wissensbasierter Systeme\u2013Grundlagen, Springer.","DOI":"10.1007\/978-3-658-27084-1"},{"key":"ref_34","unstructured":"Aho, A. (1999). Compilerbau, Oldenbourg Wissenschaftsverlag."},{"key":"ref_35","unstructured":"Hauser, R. (2010). Principles of Computer Linguistics, Springer Publishing."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Evans, J. (2017). Thinking and Reasoning: A Very Short Introduction, Oxford University Press.","DOI":"10.1093\/actrade\/9780198787259.001.0001"},{"key":"ref_37","unstructured":"W3C (2021, May 28). SKOS Simple Knowledge Organisation System. Available online: https:\/\/www.w3.org\/2004\/02\/skos\/."},{"key":"ref_38","unstructured":"Bochman, A. (2021, October 24). Nonmonotonic Reasoning. Available online: https:\/\/www.sciencedirect.com\/science\/article\/pii\/ S1874585707800124."},{"key":"ref_39","unstructured":"Das, A. (2021, October 28). Knowledge Representation. Available online: https:\/\/www.sciencedirect.com\/science\/article\/pii\/B0122272404001027."},{"key":"ref_40","unstructured":"Reiter, R. (2021, September 20). A Logic for Default Reasoning. Available online: https:\/\/www.sciencedirect.com\/science\/article\/pii\/ 0004370280900144."},{"key":"ref_41","unstructured":"Poole, D. (2021, July 14). A Logical Framework for Default Reasoning. Available online: https:\/\/www.sciencedirect.com\/science\/article\/pii\/ 000437028890077X."},{"key":"ref_42","unstructured":"W3C.org (2020, August 23). SPARQL Query Language for RDF. Available online: https:\/\/www.w3.org\/TR\/sparql11-overview\/."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Wagenpfeil, S., Vu, B., Kevitt, P., and Hemmje, M. (2021). Fast and Effective Retrieval for Large Multimedia Collections. Big Data Cogn. Comput., 5.","DOI":"10.3390\/bdcc5030033"},{"key":"ref_44","unstructured":"LSTM Model (2021, November 28). Long Short-Term Memory. Available online: https:\/\/en.wikipedia.org\/wiki\/Long_short-term_memory."},{"key":"ref_45","unstructured":"Gamma, E., Helm, R., Johnson, R., Vlissides, J., and Patterns, D. (1994). Design Patterns\u2014Elements of Reusable Object Oriented Software, Addison Wesley."},{"key":"ref_46","unstructured":"Wikidata.com (2021, October 24). Wikidata\u2014The Free Knowledgebase. Available online: https:\/\/www.wikidata.org\/wiki\/Wikidata:Main_Page."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1145\/2629489","article-title":"Wikidata: A free collaborative knowledgebase","volume":"57","year":"2014","journal-title":"Commun. ACM"},{"key":"ref_48","unstructured":"(2021, September 14). Text Retrieval Conference, Datasets, Available online: https:\/\/trec.nist.gov\/data.html."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1162\/tacl_a_00166","article-title":"From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions","volume":"2","author":"Young","year":"2014","journal-title":"Trans. Assoc. Comput. Linguist."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/12\/12\/502\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:38:57Z","timestamp":1760168337000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/12\/12\/502"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,2]]},"references-count":49,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["info12120502"],"URL":"https:\/\/doi.org\/10.3390\/info12120502","relation":{},"ISSN":["2078-2489"],"issn-type":[{"type":"electronic","value":"2078-2489"}],"subject":[],"published":{"date-parts":[[2021,12,2]]}}}