{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T16:57:57Z","timestamp":1774630677848,"version":"3.50.1"},"reference-count":48,"publisher":"Emerald","issue":"7","license":[{"start":{"date-parts":[[2017,8,14]],"date-time":"2017-08-14T00:00:00Z","timestamp":1502668800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IMDS"],"published-print":{"date-parts":[[2017,8,14]]},"abstract":"<jats:sec>\n<jats:title content-type=\"abstract-subheading\">Purpose<\/jats:title>\n<jats:p>The purpose of this paper is to address the problem of weak acceptance of machine learning (ML) models in business. The proposed framework of top-performing ML models coupled with general explanation methods provides additional information to the decision-making process. This builds a foundation for sustainable organizational learning.<\/jats:p>\n<\/jats:sec>\n<jats:sec>\n<jats:title content-type=\"abstract-subheading\">Design\/methodology\/approach<\/jats:title>\n<jats:p>To address user acceptance, participatory approach of action design research (ADR) was chosen. The proposed framework is demonstrated on a B2B sales forecasting process in an organizational setting, following cross-industry standard process for data mining (CRISP-DM) methodology.<\/jats:p>\n<\/jats:sec>\n<jats:sec>\n<jats:title content-type=\"abstract-subheading\">Findings<\/jats:title>\n<jats:p>The provided ML model explanations efficiently support business decision makers, reduce forecasting error for new sales opportunities, and facilitate discussion about the context of opportunities in the sales team.<\/jats:p>\n<\/jats:sec>\n<jats:sec>\n<jats:title content-type=\"abstract-subheading\">Research limitations\/implications<\/jats:title>\n<jats:p>The quality and quantity of available data affect the performance of models and explanations.<\/jats:p>\n<\/jats:sec>\n<jats:sec>\n<jats:title content-type=\"abstract-subheading\">Practical implications<\/jats:title>\n<jats:p>The application in the real-world company demonstrates the utility of the approach and provides evidence that transparent explanations of ML models contribute to individual and organizational learning.<\/jats:p>\n<\/jats:sec>\n<jats:sec>\n<jats:title content-type=\"abstract-subheading\">Social implications<\/jats:title>\n<jats:p>All used methods are available as an open-source software and can improve the acceptance of ML in data-driven decision making.<\/jats:p>\n<\/jats:sec>\n<jats:sec>\n<jats:title content-type=\"abstract-subheading\">Originality\/value<\/jats:title>\n<jats:p>The proposed framework incorporates existing ML models and general explanation methodology into a decision-making process. To the authors\u2019 knowledge, this is the first attempt to support organizational learning with a framework combining ML explanations, ADR, and data mining methodology based on the CRISP-DM industry standard.<\/jats:p>\n<\/jats:sec>","DOI":"10.1108\/imds-09-2016-0409","type":"journal-article","created":{"date-parts":[[2017,7,3]],"date-time":"2017-07-03T07:51:12Z","timestamp":1499068272000},"page":"1389-1406","source":"Crossref","is-referenced-by-count":42,"title":["Decision-making framework with double-loop learning through interpretable black-box machine learning models"],"prefix":"10.1108","volume":"117","author":[{"given":"Marko","family":"Bohanec","sequence":"first","affiliation":[]},{"given":"Marko","family":"Robnik-\u0160ikonja","sequence":"additional","affiliation":[]},{"given":"Mirjana","family":"Kljaji\u0107 Bor\u0161tnar","sequence":"additional","affiliation":[]}],"member":"140","reference":[{"key":"key2020120613313549100_ref001","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1016\/j.chb.2014.03.047","article-title":"Reliance, trust and heuristics in judgmental forecasting","volume":"36","year":"2014","journal-title":"Computers in Human Behavior"},{"key":"key2020120613313549100_ref002","volume-title":"On Organizational Learning","year":"1992"},{"issue":"8","key":"key2020120613313549100_ref003","doi-asserted-by":"crossref","first-page":"1717","DOI":"10.1016\/j.jbusres.2015.03.031","article-title":"Golden rule of forecasting: be conservative","volume":"68","year":"2015","journal-title":"Journal of Business Research"},{"issue":"1","key":"key2020120613313549100_ref004","doi-asserted-by":"crossref","first-page":"79","DOI":"10.2307\/25148718","article-title":"The differential use and effect of knowledge-based system explanations in novice and expert judgment decisions","volume":"30","year":"2006","journal-title":"MIS Quarterly"},{"key":"key2020120613313549100_ref005","unstructured":"Bohanec, M. (2016), \u201cB2B data set in CSV format available for download\u201d, available at: www.salvirt.com\/research\/B2Bdataset (accessed September 1, 2016)."},{"key":"key2020120613313549100_ref006","article-title":"Modeling attributes for forecasting B2B opportunities acquisition","year":"2015"},{"key":"key2020120613313549100_ref007","doi-asserted-by":"crossref","unstructured":"Bohanec, M., Bor\u0161tnar, M.K. and Robnik-\u0160ikonja, M. (2016), \u201cIntegration of machine learning insights into organizational learning: a case of B2B sales forecasting\u201d, in D\u2019Ascenzo, F., Magni, M., Lazazzara, A. and Za, S. (Eds), Blurring the Boundaries Through Digital Innovation, Vol. 19, Lecture Notes in Information Systems and Organisation, Springer, Cham, pp. 71-85.","DOI":"10.1007\/978-3-319-38974-5_7"},{"key":"key2020120613313549100_ref008","doi-asserted-by":"crossref","first-page":"416","DOI":"10.1016\/j.eswa.2016.11.010","article-title":"Explaining machine learning models in sales predictions","volume":"71","year":"2017","journal-title":"Expert Systems with Applications"},{"issue":"3","key":"key2020120613313549100_ref009","doi-asserted-by":"crossref","first-page":"270","DOI":"10.1002\/sdr.460","article-title":"The relevance of facilitation in group decision making supported by a simulation model","volume":"27","year":"2011","journal-title":"System Dynamics Review"},{"key":"key2020120613313549100_ref010","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","year":"2001","journal-title":"Machine Learning Journal"},{"key":"key2020120613313549100_ref011","article-title":"Strength in numbers: how does data-driven decision-making affect firm performance?","year":"2011"},{"key":"key2020120613313549100_ref012","first-page":"161","article-title":"An empirical comparison of supervised learning algorithms","year":"2006"},{"issue":"4","key":"key2020120613313549100_ref013","doi-asserted-by":"crossref","first-page":"544","DOI":"10.1016\/j.indmarman.2013.03.006","article-title":"Model-supported business-to-business prospect prediction based on an iterative customer acquisition framework","volume":"42","year":"2013","journal-title":"Industrial Marketing Management"},{"key":"key2020120613313549100_ref014","volume-title":"Out of the Crisis","year":"1986"},{"key":"key2020120613313549100_ref015","volume-title":"How Organizations Learn: An Integrated Strategy for Building Learning Capability","year":"1998"},{"issue":"3","key":"key2020120613313549100_ref016","doi-asserted-by":"crossref","first-page":"1481","DOI":"10.1016\/j.dss.2005.12.003","article-title":"The effect of structural characteristics of explanations on use of a DSS","volume":"42","year":"2006","journal-title":"Decision Support Systems"},{"key":"key2020120613313549100_ref017","volume-title":"Probability and the Weight of Evidence","year":"1950"},{"issue":"2","key":"key2020120613313549100_ref018","first-page":"1","article-title":"Interactive machine learning for health informatics: when do we need the human-in-the-loop?","volume":"3","year":"2016","journal-title":"Brain Informatics"},{"issue":"6","key":"key2020120613313549100_ref019","doi-asserted-by":"crossref","first-page":"1223","DOI":"10.1162\/0899766053630350","article-title":"Rule extraction from recurrent neural networks: a taxonomy and review","volume":"17","year":"2005","journal-title":"Neural Computation"},{"key":"key2020120613313549100_ref020","first-page":"108","article-title":"Nomograms for visualizing support vector machines","year":"2005"},{"issue":"3","key":"key2020120613313549100_ref048","doi-asserted-by":"crossref","first-page":"270","DOI":"10.1002\/sdr.460","article-title":"The relevance of facilitation in group decision making supported by a simulation model","volume":"27","year":"2011","journal-title":"System Dynamics Review"},{"key":"key2020120613313549100_ref021","volume-title":"Machine Learning and Data Mining: Introduction to Principles and Algorithms","year":"2007"},{"issue":"5","key":"key2020120613313549100_ref022","first-page":"1","article-title":"Caret package","volume":"28","year":"2008","journal-title":"Journal of Statistical Software"},{"issue":"3","key":"key2020120613313549100_ref023","doi-asserted-by":"crossref","first-page":"543","DOI":"10.1016\/j.ijresmar.2016.01.003","article-title":"The B2B knowledge gap","volume":"33","year":"2016","journal-title":"International Journal of Research in Marketing"},{"issue":"10","key":"key2020120613313549100_ref024","first-page":"61","article-title":"Big data. The management revolution","volume":"90","year":"2012","journal-title":"Harvard Business Review"},{"key":"key2020120613313549100_ref025","article-title":"A survey of the application of machine learning in decision support systems","year":"2015"},{"key":"key2020120613313549100_ref026","article-title":"e1071: misc functions of the Department of Statistics (e1071)","year":"2016"},{"issue":"2","key":"key2020120613313549100_ref027","doi-asserted-by":"crossref","first-page":"178","DOI":"10.1108\/02634501111117610","article-title":"Industrial sales lead conversion modeling","volume":"29","year":"2011","journal-title":"Marketing Intelligence and Planning"},{"key":"key2020120613313549100_ref028","doi-asserted-by":"crossref","unstructured":"Mo\u017eina, M., Dem\u0161ar, J., Kattan, M. and Zupan, B. (2004), \u201cNomograms for visualization of naive Bayesian classifier\u201d, in Boulicaut, J.F., Esposito, F., Giannotti, F. and Pedreschi, D. (Eds), European Conference on Principles of Data Mining and Knowledge Discovery, Springer, Berlin and Heidelberg, pp. 337-348.","DOI":"10.1007\/978-3-540-30116-5_32"},{"issue":"2","key":"key2020120613313549100_ref029","doi-asserted-by":"crossref","first-page":"2592","DOI":"10.1016\/j.eswa.2008.02.021","article-title":"Application of data mining techniques in customer relationship management: a literature review and classification","volume":"36","year":"2009","journal-title":"Expert Systems with Applications"},{"key":"key2020120613313549100_ref030","volume-title":"The Knowledge-Creating Company: How Japanese Companies Create the Dynamics of Innovation","year":"1995"},{"key":"key2020120613313549100_ref031","doi-asserted-by":"crossref","first-page":"222","DOI":"10.1016\/j.procir.2016.05.024","article-title":"Applying action design research (ADR) to develop concept generation and selection methods","volume":"50","year":"2016","journal-title":"Procedia CIRP"},{"issue":"1","key":"key2020120613313549100_ref032","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1089\/big.2013.1508","article-title":"Data science and its relationship to big data and data-driven decision making","volume":"1","year":"2013","journal-title":"Big Data"},{"key":"key2020120613313549100_ref033","first-page":"445","article-title":"The case against accuracy estimation for comparing induction algorithms","year":"1998"},{"key":"key2020120613313549100_ref034","article-title":"Package nnet","year":"2016"},{"key":"key2020120613313549100_ref035","article-title":"CORElean: classification, regression and feature evaluation","year":"2016"},{"key":"key2020120613313549100_ref036","article-title":"Explain prediction: explanation of predictions for classification and regression","year":"2016"},{"issue":"5","key":"key2020120613313549100_ref037","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1109\/TKDE.2007.190734","article-title":"Explaining classifications for individual instances","volume":"20","year":"2008","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"key2020120613313549100_ref038","volume-title":"Sensitivity Analysis","year":"2000"},{"issue":"1","key":"key2020120613313549100_ref039","doi-asserted-by":"crossref","first-page":"37","DOI":"10.2307\/23043488","article-title":"Action design research","volume":"35","year":"2011","journal-title":"MIS Quarterly"},{"key":"key2020120613313549100_ref040","first-page":"1","article-title":"An efficient explanation of individual classifications using game theory","volume":"11","year":"2010","journal-title":"Journal of Machine Learning Research"},{"issue":"10","key":"key2020120613313549100_ref041","doi-asserted-by":"crossref","first-page":"886","DOI":"10.1016\/j.datak.2009.01.004","article-title":"Explaining instance classifications with interactions of subsets of feature values","volume":"68","year":"2009","journal-title":"Data & Knowledge Engineering"},{"key":"key2020120613313549100_ref042","volume-title":"The Nature of Statistical Learning Theory","year":"1995"},{"issue":"2","key":"key2020120613313549100_ref043","doi-asserted-by":"crossref","first-page":"330","DOI":"10.1016\/j.patcog.2010.08.011","article-title":"Mining data with random forests: a survey and results of new tests","volume":"44","year":"2011","journal-title":"Pattern Recognition"},{"issue":"6","key":"key2020120613313549100_ref044","doi-asserted-by":"crossref","first-page":"80","DOI":"10.2307\/3001968","article-title":"Individual comparisons by ranking methods","volume":"1","year":"1945","journal-title":"Biometrics Bulletin"},{"key":"key2020120613313549100_ref045","first-page":"29","article-title":"CRISP-DM: towards a standard process model for data mining","year":"2000"},{"key":"key2020120613313549100_ref046","volume-title":"Data Mining \u2013 Practical Machine Learning Tools and Techniques","year":"2011","edition":"3rd ed."},{"key":"key2020120613313549100_ref047","first-page":"1945","article-title":"On machine learning towards predictive sales pipeline analytics","year":"2015"}],"container-title":["Industrial Management &amp; Data Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/IMDS-09-2016-0409\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/IMDS-09-2016-0409\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T21:53:25Z","timestamp":1753394005000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/imds\/article\/117\/7\/1389-1406\/187134"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,8,14]]},"references-count":48,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2017,8,14]]}},"alternative-id":["10.1108\/IMDS-09-2016-0409"],"URL":"https:\/\/doi.org\/10.1108\/imds-09-2016-0409","relation":{},"ISSN":["0263-5577"],"issn-type":[{"value":"0263-5577","type":"print"}],"subject":[],"published":{"date-parts":[[2017,8,14]]}}}