{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,11]],"date-time":"2026-08-11T20:02:07Z","timestamp":1786478527095,"version":"3.56.0"},"reference-count":60,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2024,3,13]],"date-time":"2024-03-13T00:00:00Z","timestamp":1710288000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"CHIST-ERA IV","award":["CHISTERA-19-XAI-005"],"award-info":[{"award-number":["CHISTERA-19-XAI-005"]}]},{"name":"FAIR\u2013Future Artificial Intelligence Research","award":["PE00000013"],"award-info":[{"award-number":["PE00000013"]}]},{"DOI":"10.13039\/501100000780","name":"European Commission","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100000780","id-type":"DOI","asserted-by":"crossref"}]},{"name":"ENGINES \u2013 ENGineering INtElligent Systems around intelligent agent technologies","award":["20229ZXBZM"],"award-info":[{"award-number":["20229ZXBZM"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2024,6,30]]},"abstract":"<jats:p>\n            In this article, we focus on the opacity issue of sub-symbolic machine learning predictors by promoting two complementary activities\u2014\n            <jats:italic>symbolic knowledge extraction<\/jats:italic>\n            (SKE) and\n            <jats:italic>symbolic knowledge injection<\/jats:italic>\n            (SKI)\u2014from and into sub-symbolic predictors. We consider as symbolic any language being intelligible and interpretable for both humans and computers. Accordingly, we propose general meta-models for both SKE and SKI, along with two taxonomies for the classification of SKE and SKI methods. By adopting an explainable artificial intelligence (XAI) perspective, we highlight how such methods can be exploited to mitigate the aforementioned opacity issue. Our taxonomies are attained by surveying and classifying existing methods from the literature, following a systematic approach, and by generalising the results of previous surveys targeting specific sub-topics of either SKE or SKI alone. More precisely, we analyse 132 methods for SKE and 117 methods for SKI, and we categorise them according to their purpose, operation, expected input\/output data and predictor types. For each method, we also indicate the presence\/lack of runnable software implementations. Our work may be of interest for data scientists aiming at selecting the most adequate SKE\/SKI method for their needs, and may also work as suggestions for researchers interested in filling the gaps of the current state-of-the-art as well as for developers willing to implement SKE\/SKI-based technologies.\n          <\/jats:p>","DOI":"10.1145\/3645103","type":"journal-article","created":{"date-parts":[[2024,2,8]],"date-time":"2024-02-08T12:08:10Z","timestamp":1707394090000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":30,"title":["Symbolic Knowledge Extraction and Injection with Sub-symbolic Predictors: A Systematic Literature Review"],"prefix":"10.1145","volume":"56","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1841-8996","authenticated-orcid":false,"given":"Giovanni","family":"Ciatto","sequence":"first","affiliation":[{"name":"Dipartimento di Informatica \u2013 Scienza e Ingegneria, Alma Mater Studiorum\u2014Universit\u00e0 di Bologna, Cesena, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0532-6777","authenticated-orcid":false,"given":"Federico","family":"Sabbatini","sequence":"additional","affiliation":[{"name":"Dipartimento di Scienze Pure e Applicate, Universit\u00e0 degli Studi di Urbino Carlo Bo, Urbino, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0531-1978","authenticated-orcid":false,"given":"Andrea","family":"Agiollo","sequence":"additional","affiliation":[{"name":"Dipartimento di Informatica \u2013 Scienza e Ingegneria, Alma Mater Studiorum\u2014Universit\u00e0 di Bologna, Cesena, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9990-420X","authenticated-orcid":false,"given":"Matteo","family":"Magnini","sequence":"additional","affiliation":[{"name":"Dipartimento di Informatica \u2013 Scienza e Ingegneria, Alma Mater Studiorum\u2014Universit\u00e0 di Bologna, Cesena, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6655-3869","authenticated-orcid":false,"given":"Andrea","family":"Omicini","sequence":"additional","affiliation":[{"name":"Dipartimento di Informatica \u2013 Scienza e Ingegneria, Alma Mater Studiorum\u2014Universit\u00e0 di Bologna, Cesena, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,3,13]]},"reference":[{"key":"e_1_3_3_2_2","series-title":"CEUR Workshop Proceedings","first-page":"98","volume-title":"WOA 2021\u201322nd Workshop \u201cFrom Objects to Agents\u201d","volume":"2963","author":"Agiollo Andrea","year":"2021","unstructured":"Andrea Agiollo, Giovanni Ciatto, and Andrea Omicini. 2021. Graph neural networks as the copula mundi between logic and machine learning: A roadmap. In WOA 2021\u201322nd Workshop \u201cFrom Objects to Agents\u201d (Bologna, Italy) (CEUR Workshop Proceedings, Vol. 2963), Roberta Calegari, Giovanni Ciatto, Enrico Denti, Andrea Omicini, and Giovanni Sartor (Eds.). CEUR-WS.org, Bologna, Italy, 98\u2013115. http:\/\/ceur-ws.org\/Vol-2963\/paper18.pdf"},{"key":"e_1_3_3_3_2","series-title":"Proceedings of Machine Learning Research","first-page":"32","volume-title":"Uncertainty in Artificial Intelligence","volume":"180","author":"Agiollo Andrea","year":"2022","unstructured":"Andrea Agiollo and Andrea Omicini. 2022. GNN2GNN: Graph neural networks to generate neural networks. In Uncertainty in Artificial Intelligence(Proceedings of Machine Learning Research, Vol. 180). ML Research Press, 32\u201342. https:\/\/proceedings.mlr.press\/v180\/agiollo22a.htmlProceedings of the 38th Conference on Uncertainty in Artificial Intelligence, UAI 2022, 1-5 August 2022, Eindhoven, The Netherlands."},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0022-0000(05)80071-6"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1016\/0950-7051(96)81920-4"},{"key":"e_1_3_3_6_2","first-page":"43","volume-title":"The Description Logic Handbook: Theory, Implementation, and Applications","author":"Baader Franz","year":"2003","unstructured":"Franz Baader. 2003. Basic description logics. In The Description Logic Handbook: Theory, Implementation, and Applications. Cambridge University Press, USA, 43\u201395."},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2019.12.012"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.3233\/FAIA210348"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1016\/B978-155860932-7\/50101-1"},{"key":"e_1_3_3_10_2","volume-title":"Classification and Regression Trees","author":"Breiman Leo","year":"1984","unstructured":"Leo Breiman, Jerome Friedman, Charles J. Stone, and Richard A. Olshen. 1984. Classification and Regression Trees. CRC Press."},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1177\/2053951715622512"},{"key":"e_1_3_3_12_2","first-page":"528","volume-title":"Encyclopedia of Software Engineering","author":"Caldiera Victor R. Basili, Gianluigi","year":"1994","unstructured":"Victor R. Basili, Gianluigi Caldiera, and H. Dieter Rombach. 1994. The goal question metric approach. In Encyclopedia of Software Engineering, John J. Marciniak (Ed.). John Wiley & Sons, Inc., 528\u2013532."},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.3233\/IA-190036"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-82017-6_20"},{"key":"e_1_3_3_15_2","series-title":"CEUR Workshop Proceedings","first-page":"40","volume-title":"AI&IoT 2019 \u2013 Artificial Intelligence and Internet of Things 2019","volume":"2502","author":"Ciatto Giovanni","year":"2019","unstructured":"Giovanni Ciatto, Roberta Calegari, Andrea Omicini, and Davide Calvaresi. 2019. Towards XMAS: eXplainability through multi-agent systems. In AI&IoT 2019 \u2013 Artificial Intelligence and Internet of Things 2019(CEUR Workshop Proceedings, Vol. 2502), Claudio Savaglio, Giancarlo Fortino, Giovanni Ciatto, and Andrea Omicini (Eds.). CEUR-WS.org, Rende, CS, Italy, 40\u201353. http:\/\/ceur-ws.org\/Vol-2502\/paper3.pdf"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-51924-7_1"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-51924-7_1"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-39252-3"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(00)00077-1"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/2594473.2594475"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10710-017-9314-z"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v38i3.2741"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/C2015-0-00237-4"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/800186.810578"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236009"},{"key":"e_1_3_3_26_2","first-page":"1","article-title":"Rule extraction algorithm for deep neural networks: A review","volume":"1610","author":"Hailesilassie Tameru","year":"2016","unstructured":"Tameru Hailesilassie. 2016. Rule extraction algorithm for deep neural networks: A review. CoRR abs\/1610.05267 (2016), 1\u20136. arXiv:1610.05267http:\/\/arxiv.org\/abs\/1610.05267","journal-title":"CoRR"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.2307\/2268661"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.961358"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.dss.2010.12.003"},{"issue":"3","key":"e_1_3_3_30_2","first-page":"249","article-title":"Supervised machine learning: A review of classification techniques","volume":"31","author":"Kotsiantis Sotiris B.","year":"2007","unstructured":"Sotiris B. Kotsiantis. 2007. Supervised machine learning: A review of classification techniques. Informatica (Slovenia) 31, 3 (2007), 249\u2013268. http:\/\/www.informatica.si\/index.php\/informatica\/article\/view\/148","journal-title":"Informatica (Slovenia)"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2020\/679"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8640.1987.tb00176.x"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236386.3241340"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01246-5_2"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-76274-1"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00146-019-00912-z"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1016\/0022-0000(87)90027-4"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.2307\/2272123"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2018.07.007"},{"key":"e_1_3_3_40_2","volume-title":"Machine Learning (1 ed.)","author":"Mitchell Thomas M.","year":"1997","unstructured":"Thomas M. Mitchell. 1997. Machine Learning (1 ed.). McGraw-Hill, Inc., New York, NY, USA."},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1016\/B978-1-55860-200-7.50040-4"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2896880"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2020.2979670"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.5555\/152181"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-019-0048-x"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1214\/21-SS133"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1038\/323533a0"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098039"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2009.10.018"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-015-9833-0_3"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-76070-9_6"},{"key":"e_1_3_3_52_2","unstructured":"F. Van Veen and S. Leijnen. 2019. The Neural Network Zoo. Retrieved September 17 2021 from https:\/\/www.asimovinstitute.org\/neural-network-zoo"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-57959-7"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2021.3079836"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2017.2754499"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/4235.585893"},{"key":"e_1_3_3_57_2","first-page":"4235","volume-title":"Advances in Neural Information Processing Systems 32 (NeurIPS 2019)","author":"Xie Yaqi","year":"2019","unstructured":"Yaqi Xie, Ziwei Xu, Kuldeep S. Meel, Mohan S. Kankanhalli, and Harold Soh. 2019. Embedding symbolic knowledge into deep networks. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d\u2019Alch\u00e9-Buc, Emily B. Fox, and Roman Garnett (Eds.). Curran Associates, Inc., Vancouver, BC, Canada, 4235\u20134245. https:\/\/proceedings.neurips.cc\/paper\/2019\/hash\/7b66b4fd401a271a1c7224027ce111bc-Abstract.html"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3450068"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-09823-4_6"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2018.2876865"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46307-0_29"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3645103","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3645103","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:03:27Z","timestamp":1750291407000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3645103"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,13]]},"references-count":60,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,6,30]]}},"alternative-id":["10.1145\/3645103"],"URL":"https:\/\/doi.org\/10.1145\/3645103","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,13]]},"assertion":[{"value":"2022-08-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-31","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-03-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}