{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,8]],"date-time":"2026-01-08T16:38:19Z","timestamp":1767890299550,"version":"3.49.0"},"reference-count":44,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2022,2,14]],"date-time":"2022-02-14T00:00:00Z","timestamp":1644796800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,2,14]],"date-time":"2022-02-14T00:00:00Z","timestamp":1644796800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"lero","award":["13\/RC\/2094"],"award-info":[{"award-number":["13\/RC\/2094"]}]},{"name":"lero","award":["16\/IA\/4605"],"award-info":[{"award-number":["16\/IA\/4605"]}]},{"name":"lero","award":["16\/IA\/4605"],"award-info":[{"award-number":["16\/IA\/4605"]}]},{"DOI":"10.13039\/501100002322","name":"coordena\u00e7\u00e3o de aperfei\u00e7oamento de pessoal de n\u00edvel superior","doi-asserted-by":"publisher","award":["Finance Code 001"],"award-info":[{"award-number":["Finance Code 001"]}],"id":[{"id":"10.13039\/501100002322","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002322","name":"coordena\u00e7\u00e3o de aperfei\u00e7oamento de pessoal de n\u00edvel superior","doi-asserted-by":"publisher","award":["Finance Code 001"],"award-info":[{"award-number":["Finance Code 001"]}],"id":[{"id":"10.13039\/501100002322","id-type":"DOI","asserted-by":"publisher"}]},{"name":"lero","award":["16\/IA\/4605"],"award-info":[{"award-number":["16\/IA\/4605"]}]},{"DOI":"10.13039\/501100001635","name":"University of Limerick","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001635","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["SN COMPUT. SCI."],"published-print":{"date-parts":[[2022,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Fuzzy pattern trees evolved using grammatical evolution, a system we call Fuzzy Grammatical Evolution, are shown to be a robust Explainable Artificial Intelligence technique. Experimental results show Fuzzy Grammatical Evolution achieves competitive results when compared against SVM, Random Forest and Logistic Regression on a set of real world benchmark problems. Fuzzy Grammatical Evolution allows for human input throughout the evolutionary process. Regularization methods and double tournament selection were investigated to determine what method was most successful at finding smaller, more interpretable models. A domain expert was recruited to investigate the interpretability of the models found and to give a confidence score for each model. This expert successfully identified overfit models, validating that Fuzzy Grammatical Evolution can be regarded a powerful Explainable Artificial Intelligence technique.<\/jats:p>","DOI":"10.1007\/s42979-022-01044-w","type":"journal-article","created":{"date-parts":[[2022,2,14]],"date-time":"2022-02-14T16:31:13Z","timestamp":1644856273000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Human in the Loop Fuzzy Pattern Tree Evolution"],"prefix":"10.1007","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6209-4642","authenticated-orcid":false,"given":"Aidan","family":"Murphy","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8166-8499","authenticated-orcid":false,"given":"Gr\u00e1inne","family":"Murphy","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1783-6352","authenticated-orcid":false,"given":"Douglas Mota","family":"Dias","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6580-5668","authenticated-orcid":false,"given":"Jorge","family":"Amaral","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9818-911X","authenticated-orcid":false,"given":"Enrique","family":"Naredo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7002-5815","authenticated-orcid":false,"given":"Conor","family":"Ryan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,2,14]]},"reference":[{"key":"1044_CR1","doi-asserted-by":"publisher","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","volume":"6","author":"A Adadi","year":"2018","unstructured":"Adadi A, Berrada M. Peeking inside the black-box: A survey on explainable artificial intelligence (XAI). IEEE Access. 2018;6:52138\u201360.","journal-title":"IEEE Access"},{"key":"1044_CR2","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","volume":"58","author":"AB Arrieta","year":"2020","unstructured":"Arrieta AB, D\u00edaz-Rodr\u00edguez N, Ser JD, Bennetot A, Tabik S, Barbado A, Garcia S, Gil-Lopez S, Molina D, Benjamins R, Chatila R, Herrera F. Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Inf Fusion. 2020;58:82\u2013115.","journal-title":"Inf Fusion"},{"key":"1044_CR3","doi-asserted-by":"crossref","unstructured":"Azad R.M.A, Ryan C. The best things don\u2019t always come in small packages: Constant creation in grammatical evolution. In: M.\u00a0Nicolau, K.\u00a0Krawiec, M.I. Heywood, M.\u00a0Castelli, P.\u00a0Garc\u00eda-S\u00e1nchez, J.J. Merelo, V.M. Rivas\u00a0Santos, K.\u00a0Sim (eds.) Genetic Programming, pp. 186\u2013197. Springer Berlin Heidelberg, Berlin, Heidelberg 2014.","DOI":"10.1007\/978-3-662-44303-3_16"},{"issue":"8","key":"1044_CR4","doi-asserted-by":"publisher","first-page":"832","DOI":"10.3390\/electronics8080832","volume":"8","author":"DV Carvalho","year":"2019","unstructured":"Carvalho DV, Pereira EM, Cardoso JS. Machine learning interpretability: a survey on methods and metrics. Electronics. 2019;8(8):832.","journal-title":"Electronics"},{"issue":"2","key":"1044_CR5","doi-asserted-by":"publisher","first-page":"182","DOI":"10.1109\/4235.996017","volume":"6","author":"K Deb","year":"2002","unstructured":"Deb K, Pratap A, Agarwal S, Meyarivan T. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans Evol Comput. 2002;6(2):182\u201397.","journal-title":"IEEE Trans Evol Comput"},{"key":"1044_CR6","unstructured":"Doran D, Schulz S, Besold T.R. What does explainable AI really mean? A new conceptualization of perspectives. arXiv preprint arXiv:1710.00794 2017."},{"key":"1044_CR7","unstructured":"Doshi-Velez F, Kim B. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 2017."},{"key":"1044_CR8","volume-title":"Application of an interactive genetic algorithm in the conceptual design of car console","author":"R Dou","year":"2014","unstructured":"Dou R, Zong C, Li M. Application of an interactive genetic algorithm in the conceptual design of car console. Tianjin: Tianjin University; 2014."},{"key":"1044_CR9","doi-asserted-by":"crossref","unstructured":"Fitzgerald J, Ryan C. Exploring boundaries: optimising individual class boundaries for binary classification problem. In: Proceedings of the 14th annual conference on Genetic and evolutionary computation, 2012;pp. 743\u2013750.","DOI":"10.1145\/2330163.2330267"},{"key":"1044_CR10","doi-asserted-by":"crossref","unstructured":"Gilpin L.H, Bau D, Yuan B.Z, Bajwa A, Specter M, Kagal L. Explaining explanations: An overview of interpretability of machine learning. In: 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pp. 80\u201389. IEEE 2018.","DOI":"10.1109\/DSAA.2018.00018"},{"key":"1044_CR11","doi-asserted-by":"crossref","unstructured":"Goodman B, Flaxman S. European union regulations on algorithmic decision-making and a \u201cright to explanation.\u201d AI Magazine. 2017;38(3):50\u20137.","DOI":"10.1609\/aimag.v38i3.2741"},{"issue":"5","key":"1044_CR12","first-page":"93","volume":"51","author":"R Guidotti","year":"2018","unstructured":"Guidotti R, Monreale A, Ruggieri S, Turini F, Giannotti F, Pedreschi D. A survey of methods for explaining black box models. ACM Comput Surv (CSUR). 2018;51(5):93.","journal-title":"ACM Comput Surv (CSUR)"},{"issue":"4","key":"1044_CR13","doi-asserted-by":"publisher","first-page":"958","DOI":"10.1109\/TFUZZ.2008.924348","volume":"16","author":"Z Huang","year":"2008","unstructured":"Huang Z, Gedeon TD, Nikravesh M. Pattern trees induction: A new machine learning method. Trans Fuz Sys. 2008;16(4):958\u201370. https:\/\/doi.org\/10.1109\/TFUZZ.2008.924348.","journal-title":"Trans Fuz Sys"},{"issue":"1","key":"1044_CR14","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1016\/j.dss.2010.12.003","volume":"51","author":"J Huysmans","year":"2011","unstructured":"Huysmans J, Dejaeger K, Mues C, Vanthienen J, Baesens B. An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models. Decis Support Syst. 2011;51(1):141\u201354.","journal-title":"Decis Support Syst"},{"key":"1044_CR15","unstructured":"Ishibuchi H, Tsukamoto N, Nojima Y. Evolutionary many-objective optimization: A short review. In: 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence), pp. 2419\u20132426. IEEE 2008."},{"key":"1044_CR16","volume-title":"Genetic programming: on the programming of computers by means of natural selection","author":"J.R Koza","year":"1992","unstructured":"Koza J.R, Koza J.R. Genetic programming: on the programming of computers by means of natural selection, vol. 1. Cambridge: MIT press; 1992."},{"key":"1044_CR17","unstructured":"Krakovna V, Doshi-Velez F. Increasing the interpretability of recurrent neural networks using hidden markov models. arXiv preprint arXiv:1606.05320 2016."},{"issue":"3","key":"1044_CR18","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1145\/3236386.3241340","volume":"16","author":"ZC Lipton","year":"2018","unstructured":"Lipton ZC. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue. 2018;16(3):31\u201357.","journal-title":"Queue"},{"key":"1044_CR19","doi-asserted-by":"crossref","unstructured":"Luke S, Panait L. Fighting bloat with nonparametric parsimony pressure. In: International Conference on Parallel Problem Solving from Nature, pp. 411\u2013421. Springer 2002.","DOI":"10.1007\/3-540-45712-7_40"},{"key":"1044_CR20","unstructured":"Marcus G. Deep learning: A critical appraisal. arXiv preprint arXiv:1801.00631 2018."},{"key":"1044_CR21","doi-asserted-by":"publisher","unstructured":"Moore A, Murdock V, Cai Y, Jones K. Transparent tree ensembles. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR \u201918, pp. 1241\u20131244. ACM, New York, NY, USA 2018. https:\/\/doi.org\/10.1145\/3209978.3210151.","DOI":"10.1145\/3209978.3210151"},{"key":"1044_CR22","doi-asserted-by":"publisher","unstructured":"Murphy A, Ali M.S, Dias D.M, Amaral J, Naredo E, Ryan C. Grammar-based fuzzy pattern trees for classification problems. In: Proceedings of the 12th International Joint Conference on Computational Intelligence - Volume 1: ECTA,, pp. 71\u201380. INSTICC, SciTePress 2020. https:\/\/doi.org\/10.5220\/0010111900710080","DOI":"10.5220\/0010111900710080"},{"key":"1044_CR23","doi-asserted-by":"publisher","unstructured":"Murphy A, Murphy G, Amaral J.L.M, Dias D.M, Naredo E, Ryan C. Towards incorporating human knowledge in fuzzy pattern tree evolution. In: T.\u00a0Hu, N.\u00a0Louren\u00e7o, E.\u00a0Medvet (eds.) Genetic Programming - 24th European Conference, EuroGP 2021, Held as Part of EvoStar 2021, Virtual Event, April 7-9, 2021, Proceedings, Lecture Notes in Computer Science, vol. 12691, pp. 66\u201381. Springer 2021. https:\/\/doi.org\/10.1007\/978-3-030-72812-0_5.","DOI":"10.1007\/978-3-030-72812-0_5"},{"key":"1044_CR24","doi-asserted-by":"crossref","unstructured":"Murphy A, Ryan C. Improving module identification and use in grammatical evolution. In: Jin Y, editor. 2020 IEEE Congress on Evolutionary Computation, CEC 2020. IEEE Press: IEEE Computational Intelligence Society; 2020.","DOI":"10.1109\/CEC48606.2020.9185571"},{"key":"1044_CR25","first-page":"111","volume":"2","author":"P Nordin","year":"1995","unstructured":"Nordin P, Francone F, Banzhaf W. Explicitly defined introns and destructive crossover in genetic programming. Adv Genet Progr. 1995;2:111\u201334.","journal-title":"Adv Genet Progr"},{"key":"1044_CR26","doi-asserted-by":"crossref","unstructured":"O\u2019Neill M, Ryan C. Grammatical evolution. IEEE Trans. Evolutionary Computation. 2001;5(4):349\u201358.","DOI":"10.1109\/4235.942529"},{"key":"1044_CR27","doi-asserted-by":"publisher","unstructured":"Patten J.V, Ryan C. Attributed grammatical evolution using shared memory spaces and dynamically typed semantic function specification. In: Genetic Programming - 18th European Conference, EuroGP 2015, Copenhagen, Denmark, April 8-10, 2015, Proceedings, 2015; pp. 105\u2013112 . https:\/\/doi.org\/10.1007\/978-3-319-16501-1_9.","DOI":"10.1007\/978-3-319-16501-1_9"},{"key":"1044_CR28","doi-asserted-by":"crossref","unstructured":"Ribeiro M.T, Singh S, Guestrin C. Why should I trust you?: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135\u20131144. ACM 2016.","DOI":"10.1145\/2939672.2939778"},{"key":"1044_CR29","doi-asserted-by":"crossref","unstructured":"Ryan C, Azad R.M.A. Sensible initialisation in grammatical evolution. In: GECCO, pp. 142\u2013145. AAAI 2003.","DOI":"10.1007\/3-540-36599-0_37"},{"key":"1044_CR30","doi-asserted-by":"crossref","unstructured":"Ryan C, Collins J.J, Neill M.O. Grammatical evolution: Evolving programs for an arbitrary language. In: European Conference on Genetic Programming, pp. 83\u201396. Springer 1998.","DOI":"10.1007\/BFb0055930"},{"key":"1044_CR31","doi-asserted-by":"crossref","unstructured":"Ryan C, Collins J.J, O\u2019Neill M. Grammatical evolution: Evolving programs for an arbitrary language. In: W.\u00a0Banzhaf, R.\u00a0Poli, M.\u00a0Schoenauer, T.C. Fogarty (eds.) EuroGP, Lecture Notes in Computer Science, vol. 1391, pp. 83\u201396. Springer 1998.","DOI":"10.1007\/BFb0055930"},{"key":"1044_CR32","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-78717-6","volume-title":"Handbook of grammatical evolution","author":"C Ryan","year":"2018","unstructured":"Ryan C, O\u2019Neill M, Collins J. Handbook of grammatical evolution. Berlin: Springer; 2018."},{"key":"1044_CR33","doi-asserted-by":"crossref","unstructured":"Samek W, Montavon G, Vedaldi A, Hansen L.K, M\u00fcller K.R. Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Springer Nature 2019. Google-Books-ID: j5yuDwAAQBAJ","DOI":"10.1007\/978-3-030-28954-6"},{"issue":"1","key":"1044_CR34","first-page":"52","volume":"22","author":"A.R dos Santos","year":"2015","unstructured":"dos Santos A.R, do\u00a0Amaral J.L.M. Synthesis of fuzzy pattern trees by cartesian genetic programming. Mathw Soft comput. 2015;22(1):52\u20136.","journal-title":"Mathw Soft comput"},{"key":"1044_CR35","doi-asserted-by":"crossref","unstructured":"dos Santos A.R, do\u00a0Amaral J.L.M, Soares C.A.R, de\u00a0Barros A.V. Multi-objective fuzzy pattern trees. In: 2018 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pp. 1\u20136. IEEE 2018.","DOI":"10.1109\/FUZZ-IEEE.2018.8491689"},{"issue":"7837","key":"1044_CR36","doi-asserted-by":"publisher","first-page":"S102","DOI":"10.1038\/d41586-020-03409-8","volume":"588","author":"N Savage","year":"2020","unstructured":"Savage N. The race to the top among the world\u2019s leaders in artificial intelligence. Nature. 2020;588(7837):S102\u20134.","journal-title":"Nature"},{"key":"1044_CR37","unstructured":"Schneider J, Handali J. Personalized explanation in machine learning: A conceptualization. arXiv preprint arXiv:1901.00770 2019"},{"key":"1044_CR38","unstructured":"Senge R. Machine learning methods for fuzzy pattern tree induction. Ph.D. thesis, Philipps-Universit\u00e4t Marburg 2014."},{"issue":"6419","key":"1044_CR39","doi-asserted-by":"publisher","first-page":"1140","DOI":"10.1126\/science.aar6404","volume":"362","author":"D Silver","year":"2018","unstructured":"Silver D, Hubert T, Schrittwieser J, Antonoglou I, Lai M, Guez A, Lanctot M, Sifre L, Kumaran D, Graepel T, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science. 2018;362(6419):1140\u20134.","journal-title":"Science"},{"issue":"9","key":"1044_CR40","doi-asserted-by":"publisher","first-page":"1275","DOI":"10.1109\/5.949485","volume":"89","author":"H Takagi","year":"2001","unstructured":"Takagi H. Interactive evolutionary computation: Fusion of the capabilities of ec optimization and human evaluation. Proc IEEE. 2001;89(9):1275\u201396.","journal-title":"Proc IEEE"},{"key":"1044_CR41","doi-asserted-by":"publisher","first-page":"413","DOI":"10.1142\/S1469026809002679","volume":"8","author":"Y Yi","year":"2009","unstructured":"Yi Y, Fober T, H\u00fcllermeier E. Fuzzy operator trees for modeling rating functions. Int J Comput Intell Appl. 2009;8:413\u201328.","journal-title":"Int J Comput Intell Appl"},{"issue":"3","key":"1044_CR42","doi-asserted-by":"publisher","first-page":"781","DOI":"10.1007\/s11263-020-01405-z","volume":"129","author":"AL Yuille","year":"2021","unstructured":"Yuille AL, Liu C. Deep nets: What have they ever done for vision? Int J Comput Vision. 2021;129(3):781\u2013802.","journal-title":"Int J Comput Vision"},{"issue":"1","key":"1044_CR43","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1162\/evco.1995.3.1.17","volume":"3","author":"BT Zhang","year":"1995","unstructured":"Zhang BT, M\u00fchlenbein H. Balancing accuracy and parsimony in genetic programming. Evol Comput. 1995;3(1):17\u201338.","journal-title":"Evol Comput"},{"key":"1044_CR44","doi-asserted-by":"crossref","unstructured":"Zou J, Schiebinger L. AI can be sexist and racist\u2013it\u2019s time to make it fair 2018.","DOI":"10.1038\/d41586-018-05707-8"}],"container-title":["SN Computer Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42979-022-01044-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s42979-022-01044-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42979-022-01044-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,18]],"date-time":"2024-09-18T14:36:10Z","timestamp":1726670170000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s42979-022-01044-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,14]]},"references-count":44,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,3]]}},"alternative-id":["1044"],"URL":"https:\/\/doi.org\/10.1007\/s42979-022-01044-w","relation":{},"ISSN":["2662-995X","2661-8907"],"issn-type":[{"value":"2662-995X","type":"print"},{"value":"2661-8907","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,14]]},"assertion":[{"value":"17 July 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 January 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 February 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"163"}}