{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T14:49:50Z","timestamp":1784213390825,"version":"3.55.0"},"reference-count":26,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2024,5,18]],"date-time":"2024-05-18T00:00:00Z","timestamp":1715990400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,5,18]],"date-time":"2024-05-18T00:00:00Z","timestamp":1715990400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Johannes Kepler University Linz"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["SN COMPUT. SCI."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Organizations employ data mining to discover patterns in historic data in order to learn predictive models. Depending on the predictive model the predictions may be more or less accurate, raising the question about the reliability of individual predictions. This paper proposes a reference process aligned with the CRISP-DM to enable the assessment of reliability of individual predictions obtained from a predictive model. The reference process describes activities along the different stages of the development process required to establish a reliability assessment approach for a predictive model. The paper then presents in more detail two specific approaches for reliability assessment: perturbation of input cases and local quality measures. Furthermore, this paper describes elements of a knowledge graph to capture important metadata about the development process and training data. The knowledge graph serves to properly configure and employ the reliability assessment approaches.<\/jats:p>","DOI":"10.1007\/s42979-024-02892-4","type":"journal-article","created":{"date-parts":[[2024,5,18]],"date-time":"2024-05-18T06:01:46Z","timestamp":1716012106000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["A Reference Process for Assessing the Reliability of Predictive Analytics Results"],"prefix":"10.1007","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8045-2239","authenticated-orcid":false,"given":"Simon","family":"Staudinger","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0955-8647","authenticated-orcid":false,"given":"Christoph G.","family":"Schuetz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1741-0252","authenticated-orcid":false,"given":"Michael","family":"Schrefl","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,5,18]]},"reference":[{"key":"2892_CR1","unstructured":"Wirth R, Hipp J. CRISP-DM: towards a Standard process model for data mining. 2000; 11"},{"key":"2892_CR2","doi-asserted-by":"publisher","unstructured":"Caetano N, Cortez P, Laureano RMS. Using data mining for prediction of hospital length of stay: an application of the CRISP-DM methodology. In: Cordeiro J, Hammoudi S, Maciaszek LA, Camp O, Filipe J editors. Enterprise information systems\u201416th international conference, ICEIS 2014, Lisbon, Portugal, April 27\u201330, 2014, Revised Selected Papers. Lecture Notes in Business Information Processing, vol. 227. Heidelberg: Springer; 2014. p. 149\u2013166. https:\/\/doi.org\/10.1007\/978-3-319-22348-3_9","DOI":"10.1007\/978-3-319-22348-3_9"},{"key":"2892_CR3","unstructured":"Moro S, Laureano R, Cortez P. Using data mining for bank direct marketing: an application of the CRISP-DM methodology 2011. EUROSIS-ETI"},{"issue":"5","key":"2892_CR4","first-page":"162","volume":"2","author":"BC da Rocha","year":"2010","unstructured":"da Rocha BC, de Sousa Junior RT. Identifying bank frauds using CRISP-DM and decision trees. Int J Comput Sci Inf Technol. 2010;2(5):162\u20139.","journal-title":"Int J Comput Sci Inf Technol"},{"key":"2892_CR5","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1016\/j.dss.2014.03.001","volume":"62","author":"S Moro","year":"2014","unstructured":"Moro S, Cortez P, Rita P. A data-driven approach to predict the success of bank telemarketing. Decis Support Syst. 2014;62:22\u201331. https:\/\/doi.org\/10.1016\/j.dss.2014.03.001.","journal-title":"Decis Support Syst"},{"key":"2892_CR6","volume-title":"Numerical mathematics and computing","author":"EW Cheney","year":"2012","unstructured":"Cheney EW, Kincaid DR. Numerical mathematics and computing. Boston: Cengage Learning; 2012."},{"key":"2892_CR7","doi-asserted-by":"crossref","unstructured":"Moroney N. Local color correction using non-linear masking. In: The Eighth color imaging conference: color science and engineering systems, technologies, applications, CIC 2000, Scottsdale, Arizona, USA, November 7\u201310. 2000. p. 108\u2013111. IS &T-The Society for Imaging Science and Technology, Springfield. 2000. http:\/\/www.ingentaconnect.com\/content\/ist\/cic\/2000\/00002000\/00000001\/art00021","DOI":"10.2352\/CIC.2000.8.1.art00021"},{"key":"2892_CR8","doi-asserted-by":"publisher","unstructured":"Capra A, Castorina A, Corchs S, Gasparini F, Schettini R. Dynamic range optimization by local contrast correction and histogram image analysis. In: 2006 digest of technical papers international conference on consumer electronics. Las Vegas: IEEE; 2006. p. 309\u2013310. https:\/\/doi.org\/10.1109\/ICCE.2006.1598434","DOI":"10.1109\/ICCE.2006.1598434"},{"key":"2892_CR9","doi-asserted-by":"publisher","unstructured":"Reimer U, T\u00f6dtli B, Maier E. How to induce trust in medical AI systems. In: Grossmann G, Ram S editors. Advances in conceptual modeling\u2013ER 2020 Workshops CMAI, CMLS, CMOMM4FAIR, CoMoNoS, EmpER, Vienna, Austria, November 3\u20136, 2020, Proceedings. Lecture notes in computer science, vol. 12584. Heidelberg: Springer; 2020. p. 5\u201314. https:\/\/doi.org\/10.1007\/978-3-030-65847-2_1","DOI":"10.1007\/978-3-030-65847-2_1"},{"key":"2892_CR10","doi-asserted-by":"publisher","unstructured":"Vriesmann Jr LM, Oliveira LS, Koerich AL, Sabourin R. Combining overall and local class accuracies in an oracle-based method for dynamic ensemble selection. In: 2015 International joint conference on neural networks, IJCNN 2015, Killarney, Ireland, July 12\u201317, 2015. New York: IEEE; 2015. p. 1\u20137. https:\/\/doi.org\/10.1109\/IJCNN.2015.7280340","DOI":"10.1109\/IJCNN.2015.7280340"},{"key":"2892_CR11","unstructured":"Sahoo S, Lebo T, McGuinness D. PROV-o: The PROV ontology. W3C recommendation, W3C. 2013. https:\/\/www.w3.org\/TR\/2013\/REC-prov-o-20130430\/. Accessed Oct 2021."},{"issue":"2","key":"2892_CR12","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1145\/3436897","volume":"2","author":"TD Huynh","year":"2021","unstructured":"Huynh TD, Tsakalakis N, Helal A, Stalla-Bourdillon S, Moreau L. Addressing regulatory requirements on explanations for automated decisions with provenance\u2014a case study. Digit Gov Res Pract. 2021;2(2):16\u201311614. https:\/\/doi.org\/10.1145\/3436897.","journal-title":"Digit Gov Res Pract"},{"key":"2892_CR13","doi-asserted-by":"publisher","unstructured":"Staudinger S, Schuetz C, Schrefl M. A reference process for judging reliability of classification results in predictive analytics. In: Proceedings of the 10th international conference on data science, technology and applications\u2014DATA. Set\u00fabal: SciTePress; 2021. p. 124\u2013134, https:\/\/doi.org\/10.5220\/0010620501240134","DOI":"10.5220\/0010620501240134"},{"key":"2892_CR14","unstructured":"Winter PM, Eder S, Weissenb\u00f6ck J, Schwald C, Doms T, Vogt T, Hochreiter S, Nessler B. Trusted artificial intelligence: towards certification of machine learning applications. CoRR. 2021. arXiv:abs\/2103.16910."},{"issue":"10","key":"2892_CR15","doi-asserted-by":"publisher","first-page":"64","DOI":"10.1145\/3448248","volume":"64","author":"JM Wing","year":"2021","unstructured":"Wing JM. Trustworthy AI. Commun ACM. 2021;64(10):64\u201371. https:\/\/doi.org\/10.1145\/3448248.","journal-title":"Commun. ACM"},{"issue":"5","key":"2892_CR16","doi-asserted-by":"publisher","first-page":"507","DOI":"10.1109\/TSE.2014.2372785","volume":"41","author":"ET Barr","year":"2015","unstructured":"Barr ET, Harman M, McMinn P, Shahbaz M, Yoo S. The oracle problem in software testing: a survey. IEEE Trans Softw Eng. 2015;41(5):507\u201325. https:\/\/doi.org\/10.1109\/TSE.2014.2372785.","journal-title":"IEEE Trans. Softw. Eng."},{"issue":"1","key":"2892_CR17","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1145\/3143561","volume":"51","author":"TY Chen","year":"2018","unstructured":"Chen TY, Kuo F-C, Liu H, Poon P-L, Towey D, Tse TH, Zhou ZQ. Metamorphic testing: a review of challenges and opportunities. ACM Comput Surv. 2018;51(1):4\u20131427. https:\/\/doi.org\/10.1145\/3143561.","journal-title":"ACM Comput. Surv."},{"issue":"9","key":"2892_CR18","doi-asserted-by":"publisher","first-page":"805","DOI":"10.1109\/TSE.2016.2532875","volume":"42","author":"S Segura","year":"2016","unstructured":"Segura S, Fraser G, S\u00e1nchez AB, Cort\u00e9s AR. A survey on metamorphic testing. IEEE Trans Softw Eng. 2016;42(9):805\u201324. https:\/\/doi.org\/10.1109\/TSE.2016.2532875.","journal-title":"IEEE Trans. Softw. Eng."},{"issue":"4","key":"2892_CR19","doi-asserted-by":"publisher","first-page":"544","DOI":"10.1016\/j.jss.2010.11.920","volume":"84","author":"X Xie","year":"2011","unstructured":"Xie X, Ho JWK, Murphy C, Kaiser GE, Xu B, Chen TY. Testing and validating machine learning classifiers by metamorphic testing. J Syst Softw. 2011;84(4):544\u201358. https:\/\/doi.org\/10.1016\/j.jss.2010.11.920.","journal-title":"J. Syst. Softw."},{"key":"2892_CR20","doi-asserted-by":"publisher","unstructured":"Saha P, Kanewala U. Fault detection effectiveness of metamorphic relations developed for testing supervised classifiers. In: IEEE international conference on artificial intelligence testing, AITest 2019, Newark, CA, USA, April 4\u20139, 2019. New York: IEEE; 2019. p. 157\u2013164. https:\/\/doi.org\/10.1109\/AITest.2019.00019.","DOI":"10.1109\/AITest.2019.00019"},{"key":"2892_CR21","doi-asserted-by":"publisher","unstructured":"Moreira D, Furtado AP, Nogueira SC. Testing acoustic scene classifiers using metamorphic relations. In: IEEE international conference on artificial intelligence testing, AITest 2020, Oxford, August 3-6, 2020. New York: IEEE; 2020. p. 47\u201354. https:\/\/doi.org\/10.1109\/AITEST49225.2020.00014.","DOI":"10.1109\/AITEST49225.2020.00014"},{"key":"2892_CR22","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1016\/j.inffus.2017.09.010","volume":"41","author":"RMO Cruz","year":"2018","unstructured":"Cruz RMO, Sabourin R, Cavalcanti GDC. Dynamic classifier selection: recent advances and perspectives. Inf Fusion. 2018;41:195\u2013216. https:\/\/doi.org\/10.1016\/j.inffus.2017.09.010.","journal-title":"Inf. Fusion"},{"issue":"11","key":"2892_CR23","doi-asserted-by":"publisher","first-page":"2188","DOI":"10.1016\/j.patcog.2005.02.010","volume":"38","author":"L Didaci","year":"2005","unstructured":"Didaci L, Giacinto G, Roli F, Marcialis GL. A study on the performances of dynamic classifier selection based on local accuracy estimation. Pattern Recognit. 2005;38(11):2188\u201391. https:\/\/doi.org\/10.1016\/j.patcog.2005.02.010.","journal-title":"Pattern Recognit."},{"issue":"5","key":"2892_CR24","doi-asserted-by":"publisher","first-page":"970","DOI":"10.1109\/TSMCB.2006.872269","volume":"36","author":"H Guo","year":"2006","unstructured":"Guo H, Shi W, Deng Y. Evaluating sensor reliability in classification problems based on evidence theory. IEEE Trans Syst Man Cybern Part B. 2006;36(5):970\u201381. https:\/\/doi.org\/10.1109\/TSMCB.2006.872269.","journal-title":"IEEE Trans. Syst. Man Cybern. Part B"},{"key":"2892_CR25","doi-asserted-by":"publisher","unstructured":"Bardach M, Gringinger E, Schrefl M, Schuetz CG. Predicting flight delay risk using a random forest classifier based on air traffic scenarios and environmental conditions. In: 2020 AIAA\/IEEE 39th digital avionics systems conference (DASC). 2020. p. 1\u20138. https:\/\/doi.org\/10.1109\/DASC50938.2020.9256474.","DOI":"10.1109\/DASC50938.2020.9256474"},{"issue":"4","key":"2892_CR26","doi-asserted-by":"publisher","first-page":"405","DOI":"10.1109\/34.588027","volume":"19","author":"KS Woods","year":"1997","unstructured":"Woods KS, Kegelmeyer WP, Bowyer KW. Combination of multiple classifiers using local accuracy estimates. IEEE Trans Pattern Anal Mach Intell. 1997;19(4):405\u201310. https:\/\/doi.org\/10.1109\/34.588027.","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."}],"container-title":["SN Computer Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42979-024-02892-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s42979-024-02892-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42979-024-02892-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,18]],"date-time":"2024-05-18T06:15:01Z","timestamp":1716012901000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s42979-024-02892-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,18]]},"references-count":26,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2024,6]]}},"alternative-id":["2892"],"URL":"https:\/\/doi.org\/10.1007\/s42979-024-02892-4","relation":{},"ISSN":["2661-8907"],"issn-type":[{"value":"2661-8907","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,18]]},"assertion":[{"value":"14 February 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 April 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 May 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"On behalf of all authors, the corresponding author states that there is no Conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"563"}}