{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T06:13:06Z","timestamp":1774678386305,"version":"3.50.1"},"reference-count":60,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2026,2,14]],"date-time":"2026-02-14T00:00:00Z","timestamp":1771027200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Software"],"abstract":"<jats:p>Machine learning (ML) engineering increasingly incorporates principles from software and requirements engineering to improve development rigor; however, key non-functional requirements (NFRs) such as interpretability and explainability remain difficult to specify and verify using traditional requirements practices. Although prior work defines these qualities conceptually, their lack of measurable criteria prevents systematic verification. This paper presents a novel provenance-driven approach that decomposes ML interpretability and explainability NFRs into verifiable functional requirements (FRs) by leveraging model and data provenance to make model behavior transparent. The approach identifies the specific provenance artifacts required to validate each FR and demonstrates how their verification collectively establishes compliance with interpretability and explainability NFRs. The results show that ML provenance can operationalize otherwise abstract NFRs, transforming interpretability and explainability into quantifiable, testable properties and enabling more rigorous, requirements-based ML engineering.<\/jats:p>","DOI":"10.3390\/software5010009","type":"journal-article","created":{"date-parts":[[2026,2,17]],"date-time":"2026-02-17T09:22:46Z","timestamp":1771320166000},"page":"9","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Verifying Machine Learning Interpretability and Explainability Requirements Through Provenance"],"prefix":"10.3390","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-0555-3640","authenticated-orcid":false,"given":"Lynn","family":"Vonderhaar","sequence":"first","affiliation":[{"name":"Department of Electrical Engineering and Computer Science, Embry-Riddle Aeronautical University, Daytona Beach, FL 32114, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-7969-3277","authenticated-orcid":false,"given":"Juan","family":"Couder","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering and Computer Science, Embry-Riddle Aeronautical University, Daytona Beach, FL 32114, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7801-0124","authenticated-orcid":false,"given":"Tyler Thomas","family":"Procko","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering and Computer Science, Embry-Riddle Aeronautical University, Daytona Beach, FL 32114, USA"}]},{"given":"Eva","family":"Lueddeke","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering and Computer Science, Embry-Riddle Aeronautical University, Daytona Beach, FL 32114, USA"}]},{"given":"Daryela","family":"Cisneros","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering and Computer Science, Embry-Riddle Aeronautical University, Daytona Beach, FL 32114, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2072-5610","authenticated-orcid":false,"given":"Omar","family":"Ochoa","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering and Computer Science, Embry-Riddle Aeronautical University, Daytona Beach, FL 32114, USA"}]}],"member":"1968","published-online":{"date-parts":[[2026,2,14]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Carvalho, D.V., Pereira, E.M., and Cardoso, J.S. (2019). Machine Learning Interpretability: A Survey on Methods and Metrics. Electronics, 8.","DOI":"10.3390\/electronics8080832"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Blanco-Justicia, A., and Domingo-Ferrer, J. (2019, January 26\u201329). Machine Learning Explainability Through Comprehensible Decision Trees. Proceedings of the International Cross-Domain Conference for Machine Learning and Knowledge Extraction (CD-MAKE 2019), Canterbury, UK.","DOI":"10.1007\/978-3-030-29726-8_2"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Habibullah, K.M., Gay, G., and Horkoff, J. (2022, January 19). Non-functional requirements for machine learning: An exploration of system scope and interest. Proceedings of the SE4RAI \u201922: Proceedings of the 1st Workshop on Software Engineering for Responsible AI, Pittsburg, PA, USA.","DOI":"10.1145\/3526073.3527589"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Horkoff, J. (2019, January 23\u201327). Non-Functional Requirements for Machine Learning: Challenges and New Directions. Proceedings of the 2019 IEEE 27th International Requirements Engineering Conference (RE), Jeju, Republic of Korea.","DOI":"10.1109\/RE.2019.00050"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Vonderhaar, L., Elvira, T., and Ochoa, O. (2025, January 1\u20133). Provenance: A Machine Learning Non-Functional Requirement. Proceedings of the IEEE Artificial Intelligence x Software Engineering (AIxSE), Laguna Hills, CA, USA.","DOI":"10.1142\/S1793351X26410060"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1162\/dint_a_00119","article-title":"Provenance Documentation to Enable Explainable and Trustworthy AI: A Literature Review","volume":"5","author":"Kale","year":"2023","journal-title":"Data Intell."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Scherzinger, S., Seifert, C., and Wiese, L. (2019, January 7\u20139). The Best of Both Worlds: Challenges in Linking Provenance and Explainability in Distributed Machine Learning. Proceedings of the IEEE 39th International Conference on Distributed Computing Systems (ICDCS), Dallas, TX, USA.","DOI":"10.1109\/ICDCS.2019.00161"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Nakagawa, T., Narita, K., and Kim, K.-S. (2022, January 12\u201315). How Provenance helps Quality Assurance Activities in AI\/ML Systems. Proceedings of the AIMLSystems \u201922: Proceedings of the Second International Conference on AI-ML Systems, Bangalore, India.","DOI":"10.1145\/3564121.3564801"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1145\/3582302.3582306","article-title":"Management of Machine Learning Lifecycle Artifacts: A Survey","volume":"51","author":"Schlegel","year":"2023","journal-title":"ACM SIGMOD Rec."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1195","DOI":"10.1007\/s10664-012-9199-7","article-title":"Software Bertillonage: Determining the provenance of software development artifacts","volume":"18","author":"Davies","year":"2013","journal-title":"Empir. Softw. Eng."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Davies, J., German, D.M., Godfrey, M.W., and Hindle, A. (2011, January 21\u201322). Software bertillonage: Finding the provenance of an entity. Proceedings of the MSR \u201911: Proceedings of the 8th Working Conference on Mining Software Repositories, Waikiki, Honolulu, HI, USA.","DOI":"10.1145\/1985441.1985468"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Bose, R.P.C., Phokela, K.K., Kaulgud, V., and Podder, S. (2019, January 2\u20135). BLINKER: A Blockchain-Enabled Framework for Software Provenance. Proceedings of the 2019 26th Asia-Pacific Software Engineering Conference (APSEC), Putrajaya, Malaysia.","DOI":"10.1109\/APSEC48747.2019.00010"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1016\/j.scico.2013.11.021","article-title":"Understanding software artifact provenance","volume":"97","author":"Godfrey","year":"2015","journal-title":"Sci. Comput. Program."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Pham, Q., Malik, T., and Foster, I. (2014, January 9\u201313). Auditing and Maintaining Provenance in Software Packages. Proceedings of the Provenance and Annotation of Data and Processes (IPAW), Cologne, Germany.","DOI":"10.1007\/978-3-319-16462-5_8"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Chittimalli, P.K., and Naik, R. (2014, January 3). Variable provenance in software systems. Proceedings of the RSSE 2014: Proceedings of the 4th International Workshop on Recommendation Systems for Software Engineering, Hyderabad, India.","DOI":"10.1145\/2593822.2593826"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Wendel, H., Kunde, M., and Schreiber, A. (2010, January 15\u201316). Provenance of Software Development Processes. Proceedings of the Provenance and Annotation of Data Processes (IPAW), Troy, NY, USA.","DOI":"10.1007\/978-3-642-17819-1_7"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Costa, G.C.B., Dalpra, H.L.O., Teixeira, E.N., Werner, C.M.L., Braga, R.M.M., and Miguel, M.A. (2018, January 28\u201330). Software Processes Analysis with Provenance. Proceedings of the International Conference on Product-Focused Software Process Improvement (PROFES), Wolfsburg, Germany.","DOI":"10.1007\/978-3-030-03673-7_8"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"S\u00e1enz-Ad\u00e1n, C., Moreau, L., P\u00e9rez, B., Miles, S., and Garc\u00eda-Izquierdo, F.J. (2018, January 9\u201310). Automating Provenance Capture in Software Engineering with UML2PROV. Proceedings of the Provenance and Annotation of Data and Processes (IPAW), London, UK.","DOI":"10.1007\/978-3-319-98379-0_5"},{"key":"ref_19","unstructured":"Schreiber, A., von Kurnatowski, L., Meinecke, A., and de Boer, C. (July, January 29). Visualization of Software Development Provenance. Proceedings of the International Conference on Human-Computer Interaction, Washington DC, USA."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Vallayil, M., Nand, P., Yan, W.Q., and Allende-Cid, H. (2023). Explainability of Automated Fact Verification Systems: A Comprehensive Review. Appl. Sci., 13.","DOI":"10.20944\/preprints202310.0366.v1"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Slack, D., Hilgard, S., Jia, E., Singh, S., and Lakkaraju, H. (2020, January 7\u20138). Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods. Proceedings of the AIES \u201920: Proceedings of the AAAI\/ACM Conference on AI, Ethics, and Society,  New York, NY, USA.","DOI":"10.1145\/3375627.3375830"},{"key":"ref_22","unstructured":"Mahmud, S., Saisubramanian, S., and Zilberstein, S. (2024, January 7\u20139). Verification and Validation of AI Systems Using Explanations. Proceedings of the 2024 AAAI Fall Symposia, Arlington, VA, USA."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3236009","article-title":"A Survey of Methods for Explaining Black Box Models","volume":"51","author":"Guidotti","year":"2018","journal-title":"ACM Comput. Surv."},{"key":"ref_24","unstructured":"Doshi-Velez, F., and Kim, B. (2017). Towards A Rigorous Science of Interpretable Machine Learning. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Nuseibeh, B., and Easterbrook, S. (2000, January 4\u201311). Requirements engineering: A roadmap. Proceedings of the Conference on the Future of Software Engineering, Limerick, Ireland.","DOI":"10.1145\/336512.336523"},{"key":"ref_26","unstructured":"Macaulay, L.A. (2012). Requirements Engineering, Springer Science & Business Media."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Villamizar, H., Escovedo, T., and Kalinowski, M. (2021, January 1\u20133). Requirements engineering for machine learning: A systematic mapping study. Proceedings of the 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Palermo, Italy.","DOI":"10.1109\/SEAA53835.2021.00013"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Rahimi, M., Guo, J.L., Kokaly, S., and Chechik, M. (2019, January 23\u201327). Toward requirements specification for machine-learned components. Proceedings of the 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW), Jeju, Republic of Korea.","DOI":"10.1109\/REW.2019.00049"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Pei, Z., Liu, L., Wang, C., and Wang, J. (2022, January 15\u201319). Requirements engineering for machine learning: A review and reflection. Proceedings of the 2022 IEEE 30th International Requirements Engineering Conference Workshops (REW), Melbourne, Australia.","DOI":"10.1109\/REW56159.2022.00039"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1089\/big.2021.0112","article-title":"Structured verification of machine learning models in industrial settings","volume":"11","author":"Kaminwar","year":"2023","journal-title":"Big Data"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1007\/s00766-020-00333-1","article-title":"Explainability as a non-functional requirement: Challenges and recommendations","volume":"25","author":"Chazette","year":"2020","journal-title":"Requir. Eng."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"110866","DOI":"10.1016\/j.knosys.2023.110866","article-title":"An objective metric for explainable AI: How and why to estimate the degree of explainability","volume":"278","author":"Sovrano","year":"2023","journal-title":"Knowl.-Based Syst."},{"key":"ref_33","first-page":"297","article-title":"Interpretability versus Explainability: Classification for Understanding Deep Learning Systems and Models","volume":"29","author":"Sudars","year":"2022","journal-title":"Eng. Optim."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1633","DOI":"10.1111\/coin.12410","article-title":"Interpretability in Healthcare: A Comparative Study of Local Machine Learning Interpretability Techniques","volume":"37","author":"Elshawi","year":"2021","journal-title":"Comput. Intell."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Murdoch, W.J., Singh, C., Kumbier, K., Abbasi-Asl, R., and Yu, B. (2019). Interpretable Machine Learning: Definitions, Methods, and Applications. arXiv.","DOI":"10.1073\/pnas.1900654116"},{"key":"ref_36","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016). Model-Agnostic Interpretability of Machine Learning. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"e1493","DOI":"10.1002\/widm.1493","article-title":"Interpretable and explainable machine learning: A methods-centric overview with concrete examples","volume":"13","author":"Vogt","year":"2023","journal-title":"WIREs Data Min. Knowl. Discov."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., and Kagal, L. (2018, January 1\u20134). Explaining Explanations: An Overview of Interpretability of Machine Learning. Proceedings of the 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), Turin, Italy.","DOI":"10.1109\/DSAA.2018.00018"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"105606","DOI":"10.1016\/j.engappai.2022.105606","article-title":"An analysis of explainability methods for convolutional neural networks","volume":"117","author":"Vonderhaar","year":"2023","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_40","unstructured":"Chang, E. (2025, December 13). MNIST with Keras (Visualization and Saliency Map). Available online: https:\/\/www.kaggle.com\/code\/ernie55ernie\/mnist-with-keras-visualization-and-saliency-map."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Samuel, S., L\u00f6ffler, F., and K\u00f6nig-Ries, B. (2020, January 22). Machine learning pipelines: Provenance, reproducibility and FAIR data principles. Proceedings of the International Provenance and Annotation Workshop, Virtual.","DOI":"10.1007\/978-3-030-80960-7_17"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3503488","article-title":"Establishing data provenance for responsible artificial intelligence systems","volume":"13","author":"Werder","year":"2022","journal-title":"ACM Trans. Manag. Inf. Syst."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Vonderhaar, L., Elvira, T., and Ochoa, O. (2025, January 27\u201328). Generating and Verifying Synthetic Datasets with Requirements Engineering. Proceedings of the 4th International Conference on AI Engineering\u2014Software Engineering for AI (CAIN), Ottawa, ON, Canada.","DOI":"10.1109\/CAIN66642.2025.00032"},{"key":"ref_44","unstructured":"MLflow (2025, December 13). Deliver Production-Ready AI. Available online: https:\/\/mlflow.org\/."},{"key":"ref_45","unstructured":"(2025, December 13). Data Version Control (DVC). Available online: https:\/\/dvc.org\/."},{"key":"ref_46","unstructured":"(2025, December 13). Weights & Biases. Available online: https:\/\/wandb.ai\/site\/."},{"key":"ref_47","unstructured":"Neptune.AI (2025, December 13). Neptune.AI, 2025. Available online: https:\/\/neptune.ai\/."},{"key":"ref_48","unstructured":"(2025, December 13). Where AI Developers Build. Comet, Available online: https:\/\/www.comet.com\/site\/."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"102298","DOI":"10.1016\/j.softx.2025.102298","article-title":"yProv4ML: Effortless provenance tracking for machine learning systems","volume":"31","author":"Padovani","year":"2025","journal-title":"SoftwareX"},{"key":"ref_50","unstructured":"World Wide Web Consortium (W3C) (2025, December 13). PROV-O: The PROV Ontology. Available online: https:\/\/www.w3.org\/TR\/prov-o\/."},{"key":"ref_51","unstructured":"(2025, December 13). PROV-DM: The PROV Data Model. Available online: https:\/\/www.w3.org\/TR\/2013\/REC-prov-dm-20130430\/."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Arp, R., Smith, B., and Spear, A.D. (2015). Building Ontologies with Basic Formal Ontology, MIT Press.","DOI":"10.7551\/mitpress\/9780262527811.001.0001"},{"key":"ref_53","unstructured":"Elmenshawii, F. (2025, December 13). Linear Regression from Scratch. Available online: https:\/\/www.kaggle.com\/code\/fareselmenshawii\/linear-regression-from-scratch\/notebook."},{"key":"ref_54","unstructured":"(2025, December 13). Use the Analysis ToolPak to Perform Complex Data Analysis. Available online: https:\/\/support.microsoft.com\/en-us\/office\/use-the-analysis-toolpak-to-perform-complex-data-analysis-6c67ccf0-f4a9-487c-8dec-bdb5a2cefab6."},{"key":"ref_55","unstructured":"Mothilal, R.K., Sharma, A., and Tan, C. (2025, December 13). Diverse Counterfactual Explanations (DiCE) for ML. Available online: https:\/\/interpret.ml\/DiCE\/."},{"key":"ref_56","unstructured":"Sharma, A. (2025, December 13). Quick Introduction to Generating Counterfactual Explanations Using DiCE. Available online: https:\/\/github.com\/interpretml\/DiCE\/blob\/main\/docs\/source\/notebooks\/DiCE_getting_started.ipynb."},{"key":"ref_57","unstructured":"Molnar, C. (2025). Interpretable Machine Learning: A Guide for Making Black Box Models Explainable, Self-Published."},{"key":"ref_58","unstructured":"Aghili, S.A.S. (2025, December 13). Time Series Decomposition Methods. Available online: https:\/\/www.kaggle.com\/code\/alisadeghiaghili\/time-series-decomposition-methods."},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Alqaraawi, A., Schuessler, M., Wei\u00df, P., Costanza, E., and Berthouze, N. (2020, January 17\u201320). Evaluating saliency map explanations for convolutional neural networks: A user study. Proceedings of the IUI \u201920: Proceedings of the 25th International Conference on Intelligent User Interfaces, Cagliari, Italy.","DOI":"10.1145\/3377325.3377519"},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Bhat, A., Sangbone Assoa, A., and Raychowdhury, A. (2022, January 3\u20135). Gradient Backpropagation based Feature Attribution to Enable Explainable-AI on the Edge. Proceedings of the 2022 IFIP\/IEEE 30th International Conference on Very Large Scale Integration (VLSI-SoC), Patras, Greece.","DOI":"10.1109\/VLSI-SoC54400.2022.9939601"}],"container-title":["Software"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2674-113X\/5\/1\/9\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,17]],"date-time":"2026-02-17T10:01:45Z","timestamp":1771322505000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2674-113X\/5\/1\/9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,14]]},"references-count":60,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,3]]}},"alternative-id":["software5010009"],"URL":"https:\/\/doi.org\/10.3390\/software5010009","relation":{},"ISSN":["2674-113X"],"issn-type":[{"value":"2674-113X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,14]]}}}