{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T14:59:26Z","timestamp":1783349966095,"version":"3.54.6"},"reference-count":39,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2024,2,21]],"date-time":"2024-02-21T00:00:00Z","timestamp":1708473600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,2,21]],"date-time":"2024-02-21T00:00:00Z","timestamp":1708473600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"HAT.tec GmbH"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Appl Intell"],"published-print":{"date-parts":[[2024,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Interpretable deep learning models are increasingly important in domains where transparent decision-making is required. In this field, the interaction of the user with the model can contribute to the interpretability of the model. In this research work, we present an innovative approach that combines soft decision trees, neural symbolic learning, and concept learning to create an image classification model that enhances interpretability and user interaction, control, and intervention. The key novelty of our method relies on the fusion of an interpretable architecture with neural symbolic learning, allowing the incorporation of expert knowledge and user interaction. Furthermore, our solution facilitates the inspection of the model through queries in the form of first-order logic predicates. Our main contribution is a human-in-the-loop model as a result of the fusion of neural symbolic learning and an interpretable architecture. We validate the effectiveness of our approach through comprehensive experimental results, demonstrating competitive performance on challenging datasets when compared to state-of-the-art solutions.<\/jats:p>","DOI":"10.1007\/s10489-024-05321-4","type":"journal-article","created":{"date-parts":[[2024,2,21]],"date-time":"2024-02-21T10:02:17Z","timestamp":1708509737000},"page":"3667-3679","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Concept logic trees: enabling user interaction for transparent image classification and human-in-the-loop learning"],"prefix":"10.1007","volume":"54","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2095-8219","authenticated-orcid":false,"given":"David M.","family":"Rodr\u00edguez","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Manuel P.","family":"Cu\u00e9llar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Diego P.","family":"Morales","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,2,21]]},"reference":[{"key":"5321_CR1","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","volume":"58","author":"AB Arrieta","year":"2020","unstructured":"Arrieta AB, D\u00edaz-Rodr\u00edguez N, Del Ser J, Bennetot A, Tabik S, Barbado A, Garc\u00eda S, Gil-L\u00f3pez S, Molina D, Benjamins R et al (2020) Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf Fusion 58:82\u2013115","journal-title":"Inf Fusion"},{"key":"5321_CR2","doi-asserted-by":"publisher","first-page":"103649","DOI":"10.1016\/j.artint.2021.103649","volume":"303","author":"B Samy","year":"2022","unstructured":"Samy B, Garcez ADA, Serafini L, Spranger M (2022) Logic tensor networks. Artif Intell 303:103649","journal-title":"Artif Intell"},{"key":"5321_CR3","doi-asserted-by":"publisher","first-page":"109947","DOI":"10.1016\/j.knosys.2022.109947","volume":"258","author":"A Bennetot","year":"2022","unstructured":"Bennetot A, Franchi G, Del Ser J, Chatila R, D\u00edaz-Rodr\u00edguez N (2022) Greybox xai: a neural-symbolic learning framework to produce interpretable predictions for image classification. Knowl-Based Syst 258:109947","journal-title":"Knowl-Based Syst"},{"key":"5321_CR4","doi-asserted-by":"crossref","unstructured":"Chen L-C, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Computer vision\u2013ECCV 2018: 15th European conference, Munich, Germany, September 8\u201314, 2018, Proceedings, Part VII 15, Springer, pp 833\u2013851","DOI":"10.1007\/978-3-030-01234-2_49"},{"key":"5321_CR5","doi-asserted-by":"crossref","unstructured":"Chen X, Mottaghi R, Liu X, Fidler S, Urtasun R, Yuille A (2014) Detect what you can: detecting and representing objects using holistic models and body parts. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1971\u20131978","DOI":"10.1109\/CVPR.2014.254"},{"key":"5321_CR6","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1016\/j.inffus.2021.09.022","volume":"79","author":"N D\u00edaz-Rodr\u00edguez","year":"2022","unstructured":"D\u00edaz-Rodr\u00edguez N, Lamas A, Sanchez J, Franchi G, Donadello I, Tabik S, Filliat D, Cruz P, Montes R, Herrera F (2022) Explainable neural-symbolic learning (x-nesyl) methodology to fuse deep learning representations with expert knowledge graphs: the monumai cultural heritage use case. Inf Fusion 79:58\u201383","journal-title":"Inf Fusion"},{"key":"5321_CR7","unstructured":"Ding Z (2019) Popular-rl-algorithms. https:\/\/github.com\/quantumiracle\/Popular-RL-Algorithms"},{"key":"5321_CR8","doi-asserted-by":"crossref","unstructured":"Donadello I, Serafini L (2016) Integration of numeric and symbolic information for semantic image interpretation. Intelligenza Artificiale 10(1):33\u201347","DOI":"10.3233\/IA-160093"},{"key":"5321_CR9","doi-asserted-by":"crossref","unstructured":"Donadello I, Serafini L, Garcez ADA (2017) Logic tensor networks for semantic image interpretation. In: Proceedings of the 26th international joint conference on artificial intelligence, pp 1596\u20131602","DOI":"10.24963\/ijcai.2017\/221"},{"key":"5321_CR10","doi-asserted-by":"crossref","unstructured":"Do\u0161ilovi\u0107 FK, Br\u010di\u0107 M, Hlupi\u0107 N (2018) Explainable artificial intelligence: a survey. In: 2018 41st International convention on information and communication technology, electronics and microelectronics (MIPRO), IEEE, pp 0210\u20130215","DOI":"10.23919\/MIPRO.2018.8400040"},{"key":"5321_CR11","unstructured":"Everingham M, Van\u00a0Gool L, Williams CKI, Winn J, Zisserman A (2010) The PASCAL Visual Object Classes Challenge (VOC2010) Results. http:\/\/www.pascal-network.org\/challenges\/VOC\/voc2010\/workshop\/index.html"},{"key":"5321_CR12","unstructured":"Frosst N, Hinton G (2017) Distilling a neural network into a soft decision tree. arXiv:1711.09784"},{"key":"5321_CR13","unstructured":"Ghorbani A, Wexler J, Zou JY, Kim B (2019) Towards automatic concept-based explanations. Adv Neural Inf Process Sys 32"},{"key":"5321_CR14","doi-asserted-by":"crossref","unstructured":"Girshick R (2015) Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 1440\u20131448","DOI":"10.1109\/ICCV.2015.169"},{"key":"5321_CR15","unstructured":"Hazimeh H, Ponomareva N, Mol P, Tan Z, Mazumder R (2020) The tree ensemble layer: differentiability meets conditional computation. In: International conference on machine learning, PMLR, pp 4138\u20134148"},{"key":"5321_CR16","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. IEEE Conference on computer vision and pattern recognition, pp 770\u2013778","DOI":"10.1109\/CVPR.2016.90"},{"key":"5321_CR17","unstructured":"Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. 3rd International conference on learning representations, pp 1\u201315"},{"key":"5321_CR18","unstructured":"Koh PW, Nguyen T, Tang YS, Mussmann S, Pierson E, Kim B, Liang P (2020) Concept bottleneck models. In: International conference on machine learning, PMLR, pp 5338\u20135348"},{"key":"5321_CR19","doi-asserted-by":"crossref","unstructured":"Kontschieder P, Fiterau M, Criminisi A, Bulo SR (2015) Deep neural decision forests. In: Proceedings of the IEEE international conference on computer vision, pp 1467\u20131475","DOI":"10.1109\/ICCV.2015.172"},{"issue":"6","key":"5321_CR20","doi-asserted-by":"publisher","first-page":"574","DOI":"10.1109\/TAI.2021.3111138","volume":"2","author":"A Kumar","year":"2021","unstructured":"Kumar A, Sehgal K, Garg P, Kamakshi V, Krishnan NC (2021) Mace: model agnostic concept extractor for explaining image classification networks. IEEE Trans Artif Intell 2(6):574\u2013583","journal-title":"IEEE Trans Artif Intell"},{"key":"5321_CR21","doi-asserted-by":"publisher","first-page":"266","DOI":"10.1016\/j.neucom.2020.09.041","volume":"420","author":"A Lamas","year":"2021","unstructured":"Lamas A, Tabik S, Cruz P, Montes R, Mart\u00ednez-Sevilla \u00c1, Cruz T, Herrera F (2021) Monumai: dataset, deep learning pipeline and citizen science based app for monumental heritage taxonomy and classification. Neurocomputing 420:266\u2013280","journal-title":"Neurocomputing"},{"key":"5321_CR22","unstructured":"Lamberti F, Morra L, Miro FD (2021) End-to-end training of logic tensor networks for object detection"},{"key":"5321_CR23","doi-asserted-by":"crossref","unstructured":"Lin T-Y, Goyal P, Girshick R, He K, Dollar P (2017) Focal loss for dense object detection. In: Proceedings of the IEEE International conference on computer vision (ICCV), Oct","DOI":"10.1109\/ICCV.2017.324"},{"key":"5321_CR24","unstructured":"Lockhart J, Magazzeni D, Veloso M (2022) Learn to explain yourself, when you can: equipping concept bottleneck models with the ability to abstain on their concept predictions. arXiv:2211.11690"},{"key":"5321_CR25","doi-asserted-by":"crossref","unstructured":"Martone S, Manigrasso F, Lamberti F, Morra L (2022) Prototypical logic tensor networks (proto-ltn) for zero shot learning. In: 2022 26th International conference on pattern recognition (ICPR), IEEE, pp 4427\u20134433","DOI":"10.1109\/ICPR56361.2022.9956239"},{"key":"5321_CR26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.artint.2018.07.007","volume":"267","author":"T Miller","year":"2019","unstructured":"Miller T (2019) Explanation in artificial intelligence: insights from the social sciences. Artif Intell 267:1\u201338","journal-title":"Artif Intell"},{"key":"5321_CR27","unstructured":"Molnar C (2020) Interpretable machine learning. Lulu. com,"},{"key":"5321_CR28","doi-asserted-by":"crossref","unstructured":"Morales\u00a0Rodr\u00edguez D, Pegalajar\u00a0Cuellar M, Morales DP (2023) On the fusion of soft-decision-trees and concept-based models. Available at SSRN 4402768","DOI":"10.2139\/ssrn.4402768"},{"key":"5321_CR29","unstructured":"Mutahar G, Miller T (2022) Concept-based explanations using non-negative concept activation vectors and decision tree for cnn models. arXiv:2211.10807"},{"key":"5321_CR30","doi-asserted-by":"crossref","unstructured":"Posada-Moreno AF, Surya N, Trimpe S (2023) Extracting concepts with local aggregated descriptors. Eclad","DOI":"10.1016\/j.patcog.2023.110146"},{"key":"5321_CR31","doi-asserted-by":"crossref","unstructured":"Ribeiro MT, Singh S, Guestrin C (2016) Why should I trust you? Explaining the predictions of any classifier. ACM SIGKDD international conference on knowledge discovery and data mining, pp 1135\u20131144","DOI":"10.1145\/2939672.2939778"},{"key":"5321_CR32","doi-asserted-by":"crossref","unstructured":"Speith T (2022) A review of taxonomies of explainable artificial intelligence (xai) methods. In: 2022 ACM Conference on fairness, accountability, and transparency, pp 2239\u20132250","DOI":"10.1145\/3531146.3534639"},{"key":"5321_CR33","unstructured":"Touvron H, Cord M, Douze M, Massa F, Sablayrolles A, Jegou H (2021) Training data-efficient image transformers &amp; distillation through attention. In: Meila M, Zhang T (eds) Proceedings of the 38th international conference on machine learning, vol 139 of proceedings of machine learning research, pp 10347\u201310357. PMLR, 18\u201324. https:\/\/proceedings.mlr.press\/v139\/touvron21a.html"},{"key":"5321_CR34","unstructured":"Wagner B, d\u2019Avila Garcez AS (2021) Neural-symbolic integration for fairness in ai. In: CEUR Workshop Proceedings, vol 2846"},{"key":"5321_CR35","unstructured":"Wagner B, Garcez ADA (2022) Neural-Symbolic Integration for Interactive Learning and Conceptual Grounding. http:\/\/arxiv.org\/abs\/2112.11805. arXiv:2112.11805"},{"key":"5321_CR36","unstructured":"Wan A, Dunlap L, Ho D, Yin J, Lee S, Jin H, Petryk S, Bargal SA, Gonzalez JE (2020) Nbdt: neural-backed decision trees"},{"key":"5321_CR37","doi-asserted-by":"crossref","unstructured":"Wang S, Fan Y, Jin S, Takyi-Aninakwa P, Fernandez C (2023) Improved anti-noise adaptive long short-term memory neural network modeling for the robust remaining useful life prediction of lithium-ion batteries. Reliab Eng Syst Saf 230:108920","DOI":"10.1016\/j.ress.2022.108920"},{"key":"5321_CR38","unstructured":"Zarlenga ME, Barbiero P, Shams Z, Kazhdan D, Bhatt U, Jamnik M (2022) On the quality assurance of concept-based representations. https:\/\/openreview.net\/forum?id=Ehhk6jyas6v"},{"key":"5321_CR39","doi-asserted-by":"crossref","unstructured":"Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. IEEE Conference on computer vision and pattern recognition, pp 2921\u20132929","DOI":"10.1109\/CVPR.2016.319"}],"container-title":["Applied Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10489-024-05321-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10489-024-05321-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10489-024-05321-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T13:28:37Z","timestamp":1714397317000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10489-024-05321-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,21]]},"references-count":39,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,3]]}},"alternative-id":["5321"],"URL":"https:\/\/doi.org\/10.1007\/s10489-024-05321-4","relation":{},"ISSN":["0924-669X","1573-7497"],"issn-type":[{"value":"0924-669X","type":"print"},{"value":"1573-7497","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,21]]},"assertion":[{"value":"4 February 2024","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 February 2024","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical and informed consent for data used"}}]}}