{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,12]],"date-time":"2026-06-12T07:34:33Z","timestamp":1781249673146,"version":"3.54.1"},"reference-count":31,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,8,14]],"date-time":"2024-08-14T00:00:00Z","timestamp":1723593600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Digit. Health"],"abstract":"<jats:sec><jats:title>Background<\/jats:title><jats:p>The use of smartphone apps in cancer patients undergoing systemic treatment can promote the early detection of symptoms and therapy side effects and may be supported by machine learning (ML) for timely adaptation of therapies and reduction of adverse events and unplanned admissions.<\/jats:p><\/jats:sec><jats:sec><jats:title>Objective<\/jats:title><jats:p>We aimed to create an Early Warning System (EWS) to predict situations where supportive interventions become necessary to prevent unplanned visits. For this, dynamically collected standardized electronic patient reported outcome (ePRO) data were analyzed in context with the patient's individual journey. Information on well-being, vital parameters, medication, and free text were also considered for establishing a hybrid ML model. The goal was to integrate both the strengths of ML in sifting through large amounts of data and the long-standing experience of human experts. Given the limitations of highly imbalanced datasets (where only very few adverse events are present) and the limitations of humans in overseeing all possible cause of such events, we hypothesize that it should be possible to combine both in order to partially overcome these limitations.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>The prediction of unplanned visits was achieved by employing a white-box ML algorithm (i.e., rule learner), which learned rules from patient data (i.e., ePROs, vital parameters, free text) that were captured via a medical device smartphone app. Those rules indicated situations where patients experienced unplanned visits and, hence, were captured as alert triggers in the EWS. Each rule was evaluated based on a cost matrix, where false negatives (FNs) have higher costs than false positives (FPs, i.e., false alarms). Rules were then ranked according to the costs and priority was given to the least expensive ones. Finally, the rules with higher priority were reviewed by two oncological experts for plausibility check and for extending them with additional conditions. This hybrid approach comprised the application of a sensitive ML algorithm producing several potentially unreliable, but fully human-interpretable and -modifiable rules, which could then be adjusted by human experts.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>From a cohort of 214 patients and more than 16\u2019000 available data entries, the machine-learned rule set achieved a recall of 19% on the entire dataset and a precision of 5%. We compared this performance to a set of conditions that a human expert had defined to predict adverse events. This \u201chuman baseline\u201d did not discover any of the adverse events recorded in our dataset, i.e., it came with a recall and precision of 0%. Despite more plentiful results were expected by our machine learning approach, the involved medical experts a) had understood and were able to make sense of the rules and b) felt capable to suggest modification to the rules, some of which could potentially increase their precision. Suggested modifications of rules included e.g., adding or tightening certain conditions to make them less sensitive or changing the rule consequences: sometimes further monitoring the situation, applying certain test (such as a CRP test) or applying some simple pain-relieving measures was deemed sufficient, making a costly consultation with the physician unnecessary. We can thus conclude that it is possible to apply machine learning as an inspirational tool that can help human experts to formulate rules for an EWS. While humans seem to lack the ability to define such rules without such support, they are capable of modifying the rules to increase their precision and generalizability.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>Learning rules from dynamic ePRO datasets may be used to assist human experts in establishing an early warning system for cancer patients in outpatient settings.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fdgth.2024.1443987","type":"journal-article","created":{"date-parts":[[2024,8,14]],"date-time":"2024-08-14T05:10:25Z","timestamp":1723612225000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Towards an early warning system for monitoring of cancer patients using hybrid interactive machine learning"],"prefix":"10.3389","volume":"6","author":[{"given":"Andreas","family":"Trojan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emanuele","family":"Laurenzi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephan","family":"J\u00fcngling","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sven","family":"Roth","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Kiessling","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ziad","family":"Atassi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yannick","family":"Kadvany","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Meinrad","family":"Mannhart","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christian","family":"Jackisch","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gerd","family":"Kullak-Ublick","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hans Friedrich","family":"Witschel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2024,8,14]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"6593","DOI":"10.3390\/s20226593","article-title":"Vital signs prediction and early warning score calculation based on continuous monitoring of hospitalised patients using wearable technology","volume":"20","author":"Youssef Ali Amer","year":"2020","journal-title":"Sensors"},{"key":"B2","doi-asserted-by":"publisher","first-page":"105458","DOI":"10.1016\/j.compbiomed.2022.105458","article-title":"Machine learning in medical applications: a review of state-of-the-art methods","volume":"145","author":"Shehab","year":"2022","journal-title":"Comput Biol Med"},{"key":"B3","doi-asserted-by":"publisher","first-page":"100370","DOI":"10.1016\/j.cosrev.2021.100370","article-title":"Role of machine learning in medical research: a survey","volume":"40","author":"Garg","year":"2021","journal-title":"Comput Sci Rev"},{"key":"B4","doi-asserted-by":"publisher","first-page":"106","DOI":"10.1016\/j.radonc.2019.01.003","article-title":"Predicting radiation pneumonitis in locally advanced stage II\u2013III non-small cell lung cancer using machine learning","volume":"133","author":"Luna","year":"2019","journal-title":"Radiother Oncol"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.3389\/fimmu.2023.1138489","article-title":"Identification and prediction of immune checkpoint inhibitors-related pneumonitis by machine learning","volume":"14","author":"Gong","year":"2023","journal-title":"Front Immunol"},{"key":"B6","doi-asserted-by":"publisher","first-page":"1477","DOI":"10.1097\/CCM.0000000000003803","article-title":"Clinician perception of a machine learning-based early warning system designed to predict severe sepsis and septic shock","volume":"47","author":"Ginestra","year":"2019","journal-title":"Crit Care Med"},{"key":"B7","doi-asserted-by":"publisher","first-page":"2049","DOI":"10.3390\/math10122049","article-title":"Machine learning-based cardiac arrest prediction for early warning system","volume":"10","author":"Chae","year":"2022","journal-title":"Mathematics"},{"key":"B8","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1007\/978-3-319-62701-4_14","article-title":"Predicting hospital re-admissions from nursing care data of hospitalized patients","author":"Lodhi","year":"2017","journal-title":"Adv Data Mining"},{"key":"B9","doi-asserted-by":"publisher","first-page":"405","DOI":"10.1038\/s41390-022-02380-6","article-title":"The use of machine learning and artificial intelligence within pediatric critical care","volume":"93","author":"Shah","year":"2023","journal-title":"Pediatr Res"},{"key":"B10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12911-021-01564-0","article-title":"Electronic patient-reported outcomes and machine learning in predicting immune-related adverse events of immune checkpoint inhibitor therapies","volume":"21","author":"Iivanainen","year":"2021","journal-title":"BMC Med Inform Decis Mak"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.2196\/26950","article-title":"Effect of collaborative review of electronic patient-reported outcomes for shared reporting in breast cancer patients: descriptive comparative study","volume":"7","author":"Trojan","year":"2021","journal-title":"JMIR Cancer"},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.2196\/29271","article-title":"The effect of collaborative reviews of electronic patient-reported outcomes on the congruence of patient-and clinician-reported toxicity in cancer patients receiving systemic therapy: prospective, multicenter, observational clinical trial","volume":"23","author":"Trojan","year":"2021","journal-title":"J Med Internet Res"},{"key":"B13","doi-asserted-by":"publisher","first-page":"763","DOI":"10.1038\/nrclinonc.2017.153","article-title":"Patient-reported outcomes in cancer care\u2014hearing the patient voice at greater volume","volume":"14","author":"LeBlanc","year":"2017","journal-title":"Nat Rev Clin Oncol"},{"key":"B14","doi-asserted-by":"publisher","first-page":"e18655","DOI":"10.2196\/18655","article-title":"Digital monitoring and management of patients with advanced or metastatic non-small cell lung cancer treated with cancer immunotherapy and its impact on quality of clinical care: interview and survey study among health care professionals and patients","volume":"22","author":"Schmalz","year":"2020","journal-title":"J Med Internet Res"},{"key":"B15","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1001\/jama.2017.7156","article-title":"Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment","volume":"318","author":"Basch","year":"2017","journal-title":"JAMA"},{"key":"B16","doi-asserted-by":"publisher","first-page":"3084","DOI":"10.1002\/cncr.32898","article-title":"Congruence of patient-and clinician-reported toxicity in women receiving chemotherapy for early breast cancer","volume":"126","author":"Nyrop","year":"2020","journal-title":"Cancer"},{"key":"B17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s41512-020-0070-9","article-title":"Harnessing repeated measurements of predictor variables for clinical risk prediction: a review of existing methods","volume":"4","author":"Bull","year":"2020","journal-title":"Diagn Progn Res"},{"key":"B18","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1038\/s42256-019-0048-x","article-title":"Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead","volume":"1","author":"Rudin","year":"2019","journal-title":"Nat Mach Intell"},{"key":"B19","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1631\/jzus.C1200205","article-title":"Credit scoring by feature-weighted support vector machines","volume":"14","author":"Shi","year":"2013","journal-title":"J Zhejiang Univ Sci C"},{"key":"B20","doi-asserted-by":"publisher","first-page":"4571","DOI":"10.1073\/pnas.1906831117","article-title":"Expert-augmented machine learning","volume":"117","author":"Gennatas","year":"2020","journal-title":"Proc Natl Acad Sci USA"},{"key":"B21","first-page":"173","article-title":"Patient-reported outcomes and mobile applications. A review of their impact on patients\u2019 health outcomes","volume":"46","author":"Collado-Borrell","year":"2022","journal-title":"Farm Hosp"},{"key":"B22","doi-asserted-by":"publisher","first-page":"e48763","DOI":"10.2196\/48763","article-title":"Consolidated reporting guidelines for prognostic and diagnostic machine learning modeling studies: development and validation","volume":"25","author":"Klement","year":"2023","journal-title":"J Med Internet Res"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1301.6770","article-title":"An alternative text representation to tf-idf and bag-of-words","author":"Chen","year":"2013","journal-title":"arXiv Preprint ArXiv:13016770"},{"key":"B24","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1007\/s40708-016-0042-6","article-title":"Interactive machine learning for health informatics: when do we need the human-in-the-loop?","volume":"3","author":"Holzinger","year":"2016","journal-title":"Brain Inform"},{"key":"B25","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1016\/B978-1-55860-377-6.50023-2","article-title":"Fast effective rule induction","author":"Cohen","year":"1995","journal-title":"Mach Learn"},{"key":"B26","first-page":"155","article-title":"Metacost: a general method for making classifiers cost-sensitive","author":"Domingos","year":"1999"},{"key":"B27","first-page":"1269","article-title":"Weka-a machine learning workbench for data mining","volume-title":"Data Mining and Knowledge Discovery Handbook","author":"Frank","year":"2010"},{"key":"B28","doi-asserted-by":"publisher","first-page":"38","DOI":"10.2214\/AJR.18.20224","article-title":"Peering into the black box of artificial intelligence: evaluation metrics of machine learning methods","volume":"212","author":"Handelman","year":"2019","journal-title":"AJR Am Roentgen Ray Soc"},{"key":"B29","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1472-6947-12-1","article-title":"Mining geriatric assessment data for in-patient fall prediction models and high-risk subgroups","volume":"12","author":"Marschollek","year":"2012","journal-title":"BMC Med Inform Decis Mak"},{"key":"B30","doi-asserted-by":"publisher","first-page":"2401","DOI":"10.1007\/s10489-018-1361-5","article-title":"Interactive machine learning: experimental evidence for the human in the algorithmic loop","volume":"49","author":"Holzinger","year":"2019","journal-title":"Appl Intell"},{"key":"B31","first-page":"1","article-title":"Machine learning based early detection system of cardiac arrest","author":"Liu","year":"2019"}],"container-title":["Frontiers in Digital Health"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2024.1443987\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,14]],"date-time":"2024-08-14T05:10:29Z","timestamp":1723612229000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2024.1443987\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,14]]},"references-count":31,"alternative-id":["10.3389\/fdgth.2024.1443987"],"URL":"https:\/\/doi.org\/10.3389\/fdgth.2024.1443987","relation":{},"ISSN":["2673-253X"],"issn-type":[{"value":"2673-253X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,14]]},"article-number":"1443987"}}