{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,19]],"date-time":"2026-06-19T12:08:33Z","timestamp":1781870913828,"version":"3.54.5"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"10","license":[{"start":{"date-parts":[[2021,2,26]],"date-time":"2021-02-26T00:00:00Z","timestamp":1614297600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.springer.com\/tdm"},{"start":{"date-parts":[[2021,2,26]],"date-time":"2021-02-26T00:00:00Z","timestamp":1614297600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.springer.com\/tdm"}],"funder":[{"DOI":"10.13039\/501100012165","name":"Key Technologies Research and Development Program","doi-asserted-by":"crossref","award":["2018YFC0830801"],"award-info":[{"award-number":["2018YFC0830801"]}],"id":[{"id":"10.13039\/501100012165","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Appl Intell"],"published-print":{"date-parts":[[2021,10]]},"DOI":"10.1007\/s10489-021-02255-z","type":"journal-article","created":{"date-parts":[[2021,2,26]],"date-time":"2021-02-26T21:02:40Z","timestamp":1614373360000},"page":"7151-7165","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Interpreting the black box of supervised learning models: Visualizing the impacts of features on prediction"],"prefix":"10.1007","volume":"51","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5315-8712","authenticated-orcid":false,"given":"Xiaohang","family":"Zhang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuan","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhengren","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,2,26]]},"reference":[{"key":"2255_CR1","doi-asserted-by":"crossref","unstructured":"Ancona M, Ceolini E, \u00d6ztireli C, Gross M (2018) Towards better understanding of gradient-based attribution methods for deep neural networks. In: Proceedings of the international conference on learning representation","DOI":"10.1007\/978-3-030-28954-6_9"},{"key":"2255_CR2","unstructured":"Breiman L, Cutler A, Liaw A, Wiener M (2015) Package \u2018randomforest\u2019. https:\/\/cran.r-project.org\/web\/packages\/randomForest\/randomForest.pdf"},{"key":"2255_CR3","volume-title":"Classification and regression trees","author":"L Breiman","year":"1984","unstructured":"Breiman L, Friedman JH, Olshen RA, Stone CJ (1984) Classification and regression trees. Wadsworth, Belmont"},{"issue":"4","key":"2255_CR4","first-page":"1","volume":"20","author":"AM Carrington","year":"2020","unstructured":"Carrington AM, Fieguth PW, et al. (2020) A new concordant partial AUC and partial c statistic for imbalanced data in the evaluation of machine learning algorithms. BMC Med Inform Decis Mak 20 (4):1\u201312","journal-title":"BMC Med Inform Decis Mak"},{"issue":"i06","key":"2255_CR5","first-page":"1","volume":"061","author":"M Charrad","year":"2014","unstructured":"Charrad M, Ghazzali N, Boiteau V, Niknafs A (2014) Nbclust: An R package for determining the relevant number of clusters in a data set. J Stat Softw 061(i06):1\u201336","journal-title":"J Stat Softw"},{"issue":"4","key":"2255_CR6","doi-asserted-by":"publisher","first-page":"547","DOI":"10.1016\/j.dss.2009.05.016","volume":"47","author":"P Cortez","year":"2009","unstructured":"Cortez P, Cerdeira A, Almeida F, Matos T, Reis J (2009) Modeling wine preferences by data mining from physicochemical properties. Decis Support Syst 47(4):547\u2013553","journal-title":"Decis Support Syst"},{"key":"2255_CR7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.ins.2012.10.039","volume":"225","author":"P Cortez","year":"2013","unstructured":"Cortez P, Embrechts MJ (2013) Using sensitivity analysis and visualization techniques to open black box data mining models. Inf Sci 225:1\u201317","journal-title":"Inf Sci"},{"key":"2255_CR8","doi-asserted-by":"crossref","unstructured":"Doron M, Segev I, Shahaf D (2019) Discovering unexpected local nonlinear interactions in scientific black-box models. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery and data mining, pp 425\u201343","DOI":"10.1145\/3292500.3330886"},{"key":"2255_CR9","doi-asserted-by":"publisher","first-page":"95","DOI":"10.1080\/01969727408546059","volume":"4","author":"JC Dunn","year":"1974","unstructured":"Dunn JC (1974) Well-separated clusters and optimal fuzzy partitions. J Cybern 4:95\u2013104","journal-title":"J Cybern"},{"issue":"5","key":"2255_CR10","doi-asserted-by":"publisher","first-page":"1189","DOI":"10.1214\/aos\/1013203451","volume":"29","author":"JH Friedman","year":"2001","unstructured":"Friedman JH (2001) Greedy function approximation: A gradient boosting machine. Ann Stat 29 (5):1189\u20131232","journal-title":"Ann Stat"},{"issue":"1","key":"2255_CR11","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1080\/10618600.2014.907095","volume":"24","author":"A Goldstein","year":"2015","unstructured":"Goldstein A, Kapelner A, Bleich J, Pitkin E (2015) Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation. J Comput Graph Stat 24(1):44\u201365","journal-title":"J Comput Graph Stat"},{"issue":"5","key":"2255_CR12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3236009","volume":"51","author":"R Guidotti","year":"2018","unstructured":"Guidotti R, Monreale A, Ruggieri S, Turini F, Giannotti F, Pedreschi D (2018) A survey of methods for explaining black box models. ACM Comput Surv 51(5):1\u201342","journal-title":"ACM Comput Surv"},{"issue":"6","key":"2255_CR13","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1109\/MIS.2019.2957223","volume":"34","author":"R Guidotti","year":"2019","unstructured":"Guidotti R, Monreale A, Giannotti F, Pedreschi D, Ruggieri S, Turini F (2019) Factual and counterfactual explanations for black box decision making. IEEE Intell Syst 34(6):14\u201323","journal-title":"IEEE Intell Syst"},{"key":"2255_CR14","doi-asserted-by":"publisher","first-page":"1503","DOI":"10.1007\/s10618-014-0368-8","volume":"9","author":"A Henelius","year":"2014","unstructured":"Henelius A, Puolamaki K, Bostrom H, Asker L, Papapetrou P (2014) A peek into the black box: Exploring classifers by randomization. Data Min Knowl Discov 9:1503\u20131529","journal-title":"Data Min Knowl Discov"},{"key":"2255_CR15","doi-asserted-by":"crossref","unstructured":"Henelius A, Puolamaki K, Karlsson I, Zhao J, Asker L, Bostrom H, Papapetrou P (2015) Goldeneye++: A closer look into the black box. In: Proceedings of statistical learning and data sciences: Third international symposium, pp 96\u2013105","DOI":"10.1007\/978-3-319-17091-6_5"},{"key":"2255_CR16","doi-asserted-by":"crossref","unstructured":"Hooker G (2004) Discovering additive structure in black box functions. In: Proceedings of the Tenth ACM SIGKDD international conference on knowledge discovery and data mining , pp 575\u2013580","DOI":"10.1145\/1014052.1014122"},{"issue":"9","key":"2255_CR17","first-page":"1","volume":"1","author":"A Karatzoglou","year":"2004","unstructured":"Karatzoglou A, Smola A, Hornik K, Zeileis A (2004) Kernlab - an S4 package for kernel methods in R. J Stat Softw 1(9):1\u201320","journal-title":"J Stat Softw"},{"key":"2255_CR18","unstructured":"Krause J, Perer A, Bertini E (2016) Using visual analytics to interpret predictive machine learning models. In: ICML workshop on human interpretability in machine learning, pp 1\u20135"},{"key":"2255_CR19","doi-asserted-by":"publisher","unstructured":"Li X-H, Gao CC, Shi Y, Bai W, Gao H, Qiu L, Wang C, Gao Y, Zhang S, Xue X et al (2020) A survey of data-driven and knowledge-aware explainable AI. IEEE Trans Knowl Data Eng https:\/\/doi.org\/10.1109\/TKDE.2020.2983930","DOI":"10.1109\/TKDE.2020.2983930"},{"key":"2255_CR20","unstructured":"Lundberg SM, Lee S-I (2017) A unified approach to interpreting model predictions. In: Advances in neural information processing systems, vol 30, pp 4765\u20134774"},{"key":"2255_CR21","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1016\/j.dss.2014.03.001","volume":"62","author":"S Moro","year":"2014","unstructured":"Moro S, Cortez P, Rita P (2014) A data-driven approach to predict the success of bank telemarketing. Decis Support Syst 62:22\u201331","journal-title":"Decis Support Syst"},{"issue":"23","key":"2255_CR22","doi-asserted-by":"publisher","first-page":"5191","DOI":"10.3390\/app9235191","volume":"9","author":"S Oh","year":"2019","unstructured":"Oh S (2019) Feature interaction in terms of prediction performance. Appl Sci 9(23):5191","journal-title":"Appl Sci"},{"key":"2255_CR23","first-page":"1833","volume":"11","author":"M Ojala","year":"2010","unstructured":"Ojala M, Garriga GC (2010) Permutation tests for studying classifier performance. J Mach Learn Res 11:1833\u20131633","journal-title":"J Mach Learn Res"},{"key":"2255_CR24","doi-asserted-by":"crossref","unstructured":"Ribeiro MT, Singh S, Guestrin C (2016) Why should I trust you?: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp 1135\u20131144","DOI":"10.1145\/2939672.2939778"},{"key":"2255_CR25","unstructured":"Ripley B, Venables W (2016) Package \u2018nnet\u2019. https:\/\/cran.r-project.org\/web\/packages\/nnet\/nnet.pdf"},{"key":"2255_CR26","unstructured":"Sundararajan M, Taly A, Yan Q (2017) Axiomatic attribution for deep networks. In: Proceedings of the 34th international conference on machine learning, vol 70, pp 3319\u20133328"},{"issue":"2","key":"2255_CR27","doi-asserted-by":"publisher","first-page":"411","DOI":"10.1111\/1467-9868.00293","volume":"63","author":"R Tibshirani","year":"2001","unstructured":"Tibshirani R, Walther G, Hastie T (2001) Estimating the number of clusters in a data set via the gap statistic. J R Stat Soc Ser B-Stat Methodol 63(2):411\u2013423","journal-title":"J R Stat Soc Ser B-Stat Methodol"}],"container-title":["Applied Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10489-021-02255-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10489-021-02255-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10489-021-02255-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,4]],"date-time":"2021-09-04T04:33:11Z","timestamp":1630729991000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10489-021-02255-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,26]]},"references-count":27,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2021,10]]}},"alternative-id":["2255"],"URL":"https:\/\/doi.org\/10.1007\/s10489-021-02255-z","relation":{},"ISSN":["0924-669X","1573-7497"],"issn-type":[{"value":"0924-669X","type":"print"},{"value":"1573-7497","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,26]]},"assertion":[{"value":"1 February 2021","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 February 2021","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"no conflicts of interest and no computing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"<!--Emphasis Type='Bold' removed-->Conflict of Interests"}}]}}