{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,13]],"date-time":"2026-10-13T12:31:48Z","timestamp":1791894708802,"version":"4.3.4"},"reference-count":61,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2021,12,30]],"date-time":"2021-12-30T00:00:00Z","timestamp":1640822400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Interpretability is becoming an active research topic as machine learning (ML) models are more widely used to make critical decisions. Tabular data are one of the most commonly used modes of data in diverse applications such as healthcare and finance. Much of the existing interpretability methods used for tabular data only report feature-importance scores\u2014either locally (per example) or globally (per model)\u2014but they do not provide interpretation or visualization of how the features interact. We address this limitation by introducing Feature Vectors, a new global interpretability method designed for tabular datasets. In addition to providing feature-importance, Feature Vectors discovers the inherent semantic relationship among features via an intuitive feature visualization technique. Our systematic experiments demonstrate the empirical utility of this new method by applying it to several real-world datasets. We further provide an easy-to-use Python package for Feature Vectors.<\/jats:p>","DOI":"10.3390\/info13010015","type":"journal-article","created":{"date-parts":[[2021,12,30]],"date-time":"2021-12-30T23:29:07Z","timestamp":1640906947000},"page":"15","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Beyond Importance Scores: Interpreting Tabular ML by Visualizing Feature Semantics"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6465-2321","authenticated-orcid":false,"given":"Amirata","family":"Ghorbani","sequence":"first","affiliation":[{"name":"Department of Electrical Engineering, Stanford University, Stanford, CA 94305, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dina","family":"Berenbaum","sequence":"additional","affiliation":[{"name":"Demystify AI, Tel-Aviv 6701101, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maor","family":"Ivgi","sequence":"additional","affiliation":[{"name":"Demystify AI, Tel-Aviv 6701101, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuval","family":"Dafna","sequence":"additional","affiliation":[{"name":"Demystify AI, Tel-Aviv 6701101, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James Y.","family":"Zou","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Stanford University, Stanford, CA 94305, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,12,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"252","DOI":"10.1038\/s41586-020-2145-8","article-title":"Video-based AI for beat-to-beat assessment of cardiac function","volume":"580","author":"Ouyang","year":"2020","journal-title":"Nature"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-019-0216-8","article-title":"Deep learning interpretation of echocardiograms","volume":"3","author":"Ghorbani","year":"2020","journal-title":"NPJ Digit. Med."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1038\/nature21056","article-title":"Dermatologist-level classification of skin cancer with deep neural networks","volume":"542","author":"Esteva","year":"2017","journal-title":"Nature"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Dixon, M.F., Halperin, I., and Bilokon, P. (2020). Machine Learning in Finance, Springer.","DOI":"10.1007\/978-3-030-41068-1"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1002\/asmb.2209","article-title":"Deep learning for finance: Deep portfolios","volume":"33","author":"Heaton","year":"2017","journal-title":"Appl. Stoch. Model. Bus. Ind."},{"key":"ref_6","first-page":"50","article-title":"European Union regulations on algorithmic decision-making and a \u201cright to explanation\u201d","volume":"38","author":"Goodman","year":"2017","journal-title":"AI Mag."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Jouppi, N.P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., and Borchers, A. (2017, January 24\u201328). In-datacenter performance analysis of a tensor processing unit. Proceedings of the 44th Annual International Symposium on Computer Architecture, New York, NY, USA.","DOI":"10.1145\/3079856.3080246"},{"key":"ref_8","unstructured":"Arik, S.O., and Pfister, T. (2019). Tabnet: Attentive interpretable tabular learning. arXiv."},{"key":"ref_9","unstructured":"Yang, Y., Morillo, I.G., and Hospedales, T.M. (2018). Deep neural decision trees. arXiv."},{"key":"ref_10","unstructured":"Erickson, N., Mueller, J., Shirkov, A., Zhang, H., Larroy, P., Li, M., and Smola, A. (2020). Autogluon-tabular: Robust and accurate automl for structured data. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Carvalho, D.V., Pereira, E.M., and Cardoso, J.S. (2019). Machine learning interpretability: A survey on methods and metrics. Electronics, 8.","DOI":"10.3390\/electronics8080832"},{"key":"ref_12","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016). Model-agnostic interpretability of machine learning. arXiv."},{"key":"ref_13","unstructured":"Sundararajan, M., Taly, A., and Yan, Q. (2017, January 6\u201311). Axiomatic attribution for deep networks. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, Australia."},{"key":"ref_14","unstructured":"Smilkov, D., Thorat, N., Kim, B., Vi\u00e9gas, F., and Wattenberg, M. (2017). Smoothgrad: Removing noise by adding noise. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Poursabzi-Sangdeh, F., Goldstein, D.G., Hofman, J.M., Wortman Vaughan, J.W., and Wallach, H. (2021, January 8\u201313). Manipulating and measuring model interpretability. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan.","DOI":"10.1145\/3411764.3445315"},{"key":"ref_16","unstructured":"Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., and Viegas, F. (2018, January 10\u201315). Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). Proceedings of the International Conference on Machine Learning, PMLR, Stockholm, Sweden."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Zhou, B., Sun, Y., Bau, D., and Torralba, A. (2018, January 8\u201314). Interpretable basis decomposition for visual explanation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01237-3_8"},{"key":"ref_18","unstructured":"Bouchacourt, D., and Denoyer, L. (2019). Educe: Explaining model decisions through unsupervised concepts extraction. arXiv."},{"key":"ref_19","first-page":"9277","article-title":"Towards Automatic Concept-based Explanations","volume":"32","author":"Ghorbani","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Chen, T., and Guestrin, C. (2016, January 13\u201317). Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM Sigkdd International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939785"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/s10994-006-6226-1","article-title":"Extremely randomized trees","volume":"63","author":"Geurts","year":"2006","journal-title":"Mach. Learn."},{"key":"ref_23","first-page":"3146","article-title":"Lightgbm: A highly efficient gradient boosting decision tree","volume":"30","author":"Ke","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_24","unstructured":"Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A.V., and Gulin, A. (2018). CatBoost: Unbiased boosting with categorical features. Adv. Neural Inf. Process. Syst., 31."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1002\/widm.8","article-title":"Classification and regression trees","volume":"1","author":"Loh","year":"2011","journal-title":"Wiley Interdiscip. Rev. Data Min. Knowl. Discov."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Friedman, J., Hastie, T., and Tibshirani, R. (2001). The Elements of Statistical Learning, Springer.","DOI":"10.1007\/978-0-387-21606-5"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"611","DOI":"10.1198\/106186008X344522","article-title":"A bias correction algorithm for the Gini variable importance measure in classification trees","volume":"17","author":"Sandri","year":"2008","journal-title":"J. Comput. Graph. Stat."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1016\/j.chemolab.2010.12.004","article-title":"Empirical comparison of tree ensemble variable importance measures","volume":"105","author":"Auret","year":"2011","journal-title":"Chemom. Intell. Lab. Syst."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2105-9-307","article-title":"Conditional variable importance for random forests","volume":"9","author":"Strobl","year":"2008","journal-title":"BMC Bioinform."},{"key":"ref_30","unstructured":"Lundberg, S.M., and Lee, S.I. (2017, January 4\u20139). A unified approach to interpreting model predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13\u201317). \u201cWhy should i trust you?\u201d Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Fransisco, CA, USA.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2018, January 2\u20137). Anchors: High-precision model-agnostic explanations. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.11491"},{"key":"ref_33","unstructured":"Chen, J., Song, L., Wainwright, M., and Jordan, M. (2018, January 10\u201315). Learning to explain: An information-theoretic perspective on model interpretation. Proceedings of the International Conference on Machine Learning, PMLR, Stockholm, Sweden."},{"key":"ref_34","unstructured":"Ancona, M., Ceolini, E., \u00d6ztireli, C., and Gross, M. (3, January 30). Towards better understanding of gradient-based attribution methods for Deep Neural Networks. Proceedings of the 6th International Conference on Learning Representations, Vancouver, BC, Canada."},{"key":"ref_35","unstructured":"Simonyan, K., Vedaldi, A., and Zisserman, A. (2013). Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_37","unstructured":"Lundberg, S.M., Erion, G.G., and Lee, S.I. (2018). Consistent individualized feature attribution for tree ensembles. arXiv."},{"key":"ref_38","unstructured":"Tsang, M., Rambhatla, S., and Liu, Y. (2020). How does this interaction affect me? Interpretable attribution for feature interactions. arXiv."},{"key":"ref_39","unstructured":"Sundararajan, M., Dhamdhere, K., and Agarwal, A. (2020, January 12\u201318). The Shapley Taylor Interaction Index. Proceedings of the International Conference on Machine Learning, PMLR, Vienna, Austria."},{"key":"ref_40","first-page":"1","article-title":"Explaining explanations: Axiomatic feature interactions for deep networks","volume":"22","author":"Janizek","year":"2021","journal-title":"J. Mach. Learn. Res."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"916","DOI":"10.1214\/07-AOAS148","article-title":"Predictive learning via rule ensembles","volume":"2","author":"Friedman","year":"2008","journal-title":"Ann. Appl. Stat."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Hooker, G. (2004, January 22\u201325). Discovering additive structure in black box functions. Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Seattle, WA, USA.","DOI":"10.1145\/1014052.1014122"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Greenwell, B.M., Boehmke, B.C., and McCarthy, A.J. (2018). A simple and effective model-based variable importance measure. arXiv.","DOI":"10.32614\/CRAN.package.vip"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1943","DOI":"10.1073\/pnas.1711236115","article-title":"Iterative random forests to discover predictive and stable high-order interactions","volume":"115","author":"Basu","year":"2018","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1059","DOI":"10.1111\/rssb.12377","article-title":"Visualizing the effects of predictor variables in black box supervised learning models","volume":"82","author":"Apley","year":"2020","journal-title":"J. R. Stat. Soc. Ser. B Stat. Methodol."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1080\/10618600.2014.907095","article-title":"Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation","volume":"24","author":"Goldstein","year":"2015","journal-title":"J. Comput. Graph. Stat."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"272","DOI":"10.1080\/07350015.2019.1624293","article-title":"Causal interpretations of black-box models","volume":"39","author":"Zhao","year":"2021","journal-title":"J. Bus. Econ. Stat."},{"key":"ref_48","first-page":"1137","article-title":"A neural probabilistic language model","volume":"3","author":"Bengio","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Collobert, R., and Weston, J. (2008, January 5\u20139). A unified architecture for natural language processing: Deep neural networks with multitask learning. Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland.","DOI":"10.1145\/1390156.1390177"},{"key":"ref_50","first-page":"3111","article-title":"Distributed representations of words and phrases and their compositionality","volume":"26","author":"Mikolov","year":"2013","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1162\/tacl_a_00134","article-title":"Improving distributional similarity with lessons learned from word embeddings","volume":"3","author":"Levy","year":"2015","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_52","unstructured":"Wilson, B.J., and Schakel, A.M. (2015). Controlled experiments for word embeddings. arXiv."},{"key":"ref_53","first-page":"2825","article-title":"Scikit-learn: Machine Learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"551","DOI":"10.1111\/rssb.12265","article-title":"Panning for gold:\u2018model-X\u2019knockoffs for high dimensional controlled variable selection","volume":"80","author":"Candes","year":"2018","journal-title":"J. R. Stat. Soc. Ser. B Stat. Methodol."},{"key":"ref_55","unstructured":"10x Genomics (2021, October 18). 3k PBMCs from a Healthy Donor. Available online: https:\/\/www.10xgenomics.com\/resources\/datasets\/3-k-pbm-cs-from-a-healthy-donor-1-standard-1-1-0."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Sudlow, C., Gallacher, J., Allen, N., Beral, V., Burton, P., Danesh, J., Downey, P., Elliott, P., Green, J., and Landray, M. (2015). UK biobank: An open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS Med., 12.","DOI":"10.1371\/journal.pmed.1001779"},{"key":"ref_57","unstructured":"Dua, D., and Graff, C. (2021, October 18). UCI Machine Learning Repository. Available online: https:\/\/archive.ics.uci.edu\/ml\/index.php."},{"key":"ref_58","first-page":"202","article-title":"Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid","volume":"96","author":"Kohavi","year":"1996","journal-title":"Kdd"},{"key":"ref_59","unstructured":"Gimenez, J.R., Ghorbani, A., and Zou, J. (2019;, January 16\u201318). Knockoffs for the mass: New feature importance statistics with false discovery guarantees. Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, PMLR, Naha, Okinawa, Japan."},{"key":"ref_60","unstructured":"Dabkowski, P., and Gal, Y. (2017;, January 4\u20139). Real time image saliency for black box classifiers. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1145\/2641190.2641198","article-title":"OpenML: Networked Science in Machine Learning","volume":"15","author":"Vanschoren","year":"2013","journal-title":"SIGKDD Explor."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/1\/15\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:55:51Z","timestamp":1760169351000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/1\/15"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,30]]},"references-count":61,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,1]]}},"alternative-id":["info13010015"],"URL":"https:\/\/doi.org\/10.3390\/info13010015","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,30]]}}}