{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,3,25]],"date-time":"2025-03-25T14:38:14Z","timestamp":1742913494381,"version":"3.40.3"},"publisher-location":"Cham","reference-count":18,"publisher":"Springer International Publishing","isbn-type":[{"type":"print","value":"9783030985806"},{"type":"electronic","value":"9783030985813"}],"license":[{"start":{"date-parts":[[2022,1,1]],"date-time":"2022-01-01T00:00:00Z","timestamp":1640995200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,3,24]],"date-time":"2022-03-24T00:00:00Z","timestamp":1648080000000},"content-version":"vor","delay-in-days":82,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>While a few approaches to online predictive monitoring have focused on concept drift model adaptation, none have considered in depth the issue of performance evaluation for online process outcome prediction. Without such a continuous evaluation, users may be unaware of the performance of predictive models, resulting in inaccurate and misleading predictions. This paper fills this gap by proposing a framework for evaluating online process outcome predictions, comprising two different evaluation methods. These methods are partly inspired by the literature on streaming classification with delayed labels and complement each other to provide a comprehensive evaluation of process monitoring techniques: one focuses on real-time performance evaluation, i.e., evaluating the performance of the most recent predictions, whereas the other focuses on progress-based evaluation, i.e., evaluating the ability of a model to output correct predictions at different prefix lengths. We present an evaluation involving three publicly available event logs, including a log characterised by concept drift.<\/jats:p>","DOI":"10.1007\/978-3-030-98581-3_18","type":"book-chapter","created":{"date-parts":[[2022,3,23]],"date-time":"2022-03-23T18:03:23Z","timestamp":1648058603000},"page":"237-249","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Continuous Performance Evaluation for Business Process Outcome Monitoring"],"prefix":"10.1007","author":[{"given":"Suhwan","family":"Lee","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marco","family":"Comuzzi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xixi","family":"Lu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,3,24]]},"reference":[{"key":"18_CR1","doi-asserted-by":"crossref","unstructured":"Baier, L., Reimold, J., K\u00fchl, N.: Handling concept drift for predictions in business process mining. In: 2020 IEEE 22nd Conference on Business Informatics (CBI), vol. 1, pp. 76\u201383. IEEE (2020)","DOI":"10.1109\/CBI49978.2020.00016"},{"key":"18_CR2","doi-asserted-by":"crossref","unstructured":"Batyuk, A., Voityshyn, V.: Streaming process discovery for lambda architecture-based process monitoring platform. In: 2018 IEEE 13th International Scientific and Technical Conference on Computer Sciences and Information Technologies (CSIT), vol. 1, pp. 298\u2013301. IEEE (2018)","DOI":"10.1109\/STC-CSIT.2018.8526592"},{"key":"18_CR3","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1007\/978-3-642-03915-7_22","volume-title":"Advances in Intelligent Data Analysis VIII","author":"A Bifet","year":"2009","unstructured":"Bifet, A., Gavald\u00e0, R.: Adaptive learning from evolving data streams. In: Adams, N.M., Robardet, C., Siebes, A., Boulicaut, J.-F. (eds.) IDA 2009. LNCS, vol. 5772, pp. 249\u2013260. Springer, Heidelberg (2009). https:\/\/doi.org\/10.1007\/978-3-642-03915-7_22"},{"key":"18_CR4","series-title":"Lecture Notes in Business Information Processing","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1007\/978-3-319-74030-0_12","volume-title":"Business Process Management Workshops","author":"A Burattin","year":"2018","unstructured":"Burattin, A., Carmona, J.: A framework for online conformance checking. In: Teniente, E., Weidlich, M. (eds.) BPM 2017. LNBIP, vol. 308, pp. 165\u2013177. Springer, Cham (2018). https:\/\/doi.org\/10.1007\/978-3-319-74030-0_12"},{"key":"18_CR5","unstructured":"Burattin, A., Sperduti, A., van der Aalst, W.M.: Heuristics miners for streaming event data. arXiv preprint: arXiv:1212.6383 (2012)"},{"key":"18_CR6","doi-asserted-by":"crossref","unstructured":"Burattin, A., Sperduti, A., van der Aalst, W.M.: Control-flow discovery from event streams. In: 2014 IEEE Congress on Evolutionary Computation (CEC), pp. 2420\u20132427. IEEE (2014)","DOI":"10.1109\/CEC.2014.6900341"},{"key":"18_CR7","doi-asserted-by":"crossref","unstructured":"Domingos, P., Hulten, G.: Mining high-speed data streams. In: Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 71\u201380 (2000)","DOI":"10.1145\/347090.347107"},{"issue":"5","key":"18_CR8","doi-asserted-by":"publisher","first-page":"1237","DOI":"10.1007\/s10618-019-00654-y","volume":"34","author":"M Grzenda","year":"2019","unstructured":"Grzenda, M., Gomes, H.M., Bifet, A.: Delayed labelling evaluation for data streams. Data Mining Knowl. Disc. 34(5), 1237\u20131266 (2019). https:\/\/doi.org\/10.1007\/s10618-019-00654-y","journal-title":"Data Mining Knowl. Disc."},{"key":"18_CR9","doi-asserted-by":"crossref","unstructured":"Hulten, G., Spencer, L., Domingos, P.: Mining time-changing data streams. In: Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 97\u2013106 (2001)","DOI":"10.1145\/502512.502529"},{"issue":"1","key":"18_CR10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2674026.2674028","volume":"16","author":"G Krempl","year":"2014","unstructured":"Krempl, G., et al.: Open challenges for data stream mining research. ACM SIGKDD Explor. Newsl. 16(1), 1\u201310 (2014)","journal-title":"ACM SIGKDD Explor. Newsl."},{"key":"18_CR11","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"297","DOI":"10.1007\/978-3-319-23063-4_21","volume-title":"Business Process Management","author":"A Leontjeva","year":"2015","unstructured":"Leontjeva, A., Conforti, R., Di Francescomarino, C., Dumas, M., Maggi, F.M.: Complex symbolic sequence encodings for predictive monitoring of business processes. In: Motahari-Nezhad, H.R., Recker, J., Weidlich, M. (eds.) BPM 2015. LNCS, vol. 9253, pp. 297\u2013313. Springer, Cham (2015). https:\/\/doi.org\/10.1007\/978-3-319-23063-4_21"},{"key":"18_CR12","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"406","DOI":"10.1007\/978-3-319-23063-4_27","volume-title":"Business Process Management","author":"A Maaradji","year":"2015","unstructured":"Maaradji, A., Dumas, M., La Rosa, M., Ostovar, A.: Fast and accurate business process drift detection. In: Motahari-Nezhad, H.R., Recker, J., Weidlich, M. (eds.) BPM 2015. LNCS, vol. 9253, pp. 406\u2013422. Springer, Cham (2015). https:\/\/doi.org\/10.1007\/978-3-319-23063-4_27"},{"key":"18_CR13","first-page":"1","volume":"17","author":"M Maisenbacher","year":"2017","unstructured":"Maisenbacher, M., Weidlich, M.: Handling concept drift in predictive process monitoring. SCC 17, 1\u20138 (2017)","journal-title":"SCC"},{"key":"18_CR14","doi-asserted-by":"crossref","unstructured":"Manapragada, C., Webb, G.I., Salehi, M.: Extremely fast decision tree. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1953\u20131962 (2018)","DOI":"10.1145\/3219819.3220005"},{"key":"18_CR15","unstructured":"Montiel, J., et al.: River: machine learning for streaming data in python (2020)"},{"issue":"2","key":"18_CR16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3301300","volume":"13","author":"I Teinemaa","year":"2019","unstructured":"Teinemaa, I., Dumas, M., Rosa, M.L., Maggi, F.M.: Outcome-oriented predictive process monitoring: review and benchmark. ACM Trans. Knowl. Disc. Data (TKDD) 13(2), 1\u201357 (2019)","journal-title":"ACM Trans. Knowl. Disc. Data (TKDD)"},{"issue":"3","key":"18_CR17","doi-asserted-by":"publisher","first-page":"269","DOI":"10.1007\/s41060-017-0078-6","volume":"8","author":"SJ van Zelst","year":"2017","unstructured":"van Zelst, S.J., Bolt, A., Hassani, M., van Dongen, B.F., van der Aalst, W.M.P.: Online conformance checking: relating event streams to process models using prefix-alignments. Int. J. Data Sci. Anal. 8(3), 269\u2013284 (2017). https:\/\/doi.org\/10.1007\/s41060-017-0078-6","journal-title":"Int. J. Data Sci. Anal."},{"key":"18_CR18","doi-asserted-by":"crossref","unstructured":"\u017dliobaite, I.: Change with delayed labeling: when is it detectable? In: 2010 IEEE International Conference on Data Mining Workshops, pp. 843\u2013850. IEEE (2010)","DOI":"10.1109\/ICDMW.2010.49"}],"container-title":["Lecture Notes in Business Information Processing","Process Mining Workshops"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-030-98581-3_18","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,3,23]],"date-time":"2022-03-23T18:06:30Z","timestamp":1648058790000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-030-98581-3_18"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"ISBN":["9783030985806","9783030985813"],"references-count":18,"URL":"https:\/\/doi.org\/10.1007\/978-3-030-98581-3_18","relation":{},"ISSN":["1865-1348","1865-1356"],"issn-type":[{"type":"print","value":"1865-1348"},{"type":"electronic","value":"1865-1356"}],"subject":[],"published":{"date-parts":[[2022]]},"assertion":[{"value":"24 March 2022","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"ICPM","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Process Mining","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Eindhoven","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"The Netherlands","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2021","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"31 October 2021","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"4 November 2021","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"3","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"icpm2021a","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/icpmconference.org\/2021\/category\/calls\/workshops\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Single-blind","order":1,"name":"type","label":"Type","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"EasyChair","order":2,"name":"conference_management_system","label":"Conference Management System","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"65","order":3,"name":"number_of_submissions_sent_for_review","label":"Number of Submissions Sent for Review","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"28","order":4,"name":"number_of_full_papers_accepted","label":"Number of Full Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"0","order":5,"name":"number_of_short_papers_accepted","label":"Number of Short Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"43% - The value is computed by the equation \"Number of Full Papers Accepted \/ Number of Submissions Sent for Review * 100\" and then rounded to a whole number.","order":6,"name":"acceptance_rate_of_full_papers","label":"Acceptance Rate of Full Papers","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"3","order":7,"name":"average_number_of_reviews_per_paper","label":"Average Number of Reviews per Paper","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"2","order":8,"name":"average_number_of_papers_per_reviewer","label":"Average Number of Papers per Reviewer","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"Yes","order":9,"name":"external_reviewers_involved","label":"External Reviewers Involved","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"The proceedings also include one survey paper on the results of the XES 2.9 Workshop.","order":10,"name":"additional_info_on_review_process","label":"Additional Info on Review Process","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}}]}}