{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T15:43:58Z","timestamp":1779896638737,"version":"3.53.1"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"12","license":[{"start":{"date-parts":[[2021,10,18]],"date-time":"2021-10-18T00:00:00Z","timestamp":1634515200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,10,18]],"date-time":"2021-10-18T00:00:00Z","timestamp":1634515200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100003958","name":"Stichting voor de Technische Wetenschappen","doi-asserted-by":"publisher","award":["629.004.013"],"award-info":[{"award-number":["629.004.013"]}],"id":[{"id":"10.13039\/501100003958","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100010409","name":"Nationaal Regieorgaan Praktijkgericht Onderzoek SIA","doi-asserted-by":"publisher","award":["629.004.013"],"award-info":[{"award-number":["629.004.013"]}],"id":[{"id":"10.13039\/501100010409","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Med Syst"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Mobile health (mHealth) intervention systems can employ adaptive strategies to interact with users. Instead of designing such complex strategies manually, reinforcement learning (RL) can be used to adaptively optimize intervention strategies concerning the user\u2019s context. In this paper, we focus on the issue of overwhelming interactions when learning a good adaptive strategy for the user in RL-based mHealth intervention agents. We present a data-driven approach integrating psychological insights and knowledge of historical data. It allows RL agents to optimize the strategy of delivering context-aware notifications from empirical data when counterfactual information (user responses when receiving notifications) is missing. Our approach also considers a constraint on the frequency of notifications, which reduces the interaction burden for users. We evaluated our approach in several simulation scenarios using real large-scale running data. The results indicate that our RL agent can deliver notifications in a manner that realizes a higher behavioral impact than context-blind strategies.<\/jats:p>","DOI":"10.1007\/s10916-021-01773-0","type":"journal-article","created":{"date-parts":[[2021,10,25]],"date-time":"2021-10-25T14:27:15Z","timestamp":1635172035000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":35,"title":["Optimizing Adaptive Notifications in Mobile Health Interventions Systems: Reinforcement Learning from a Data-driven Behavioral Simulator"],"prefix":"10.1007","volume":"45","author":[{"given":"Shihan","family":"Wang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chao","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ben","family":"Kr\u00f6se","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Herke","family":"van Hoof","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,10,18]]},"reference":[{"issue":"3","key":"1773_CR1","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1023\/B:PREV.0000037641.26017.00","volume":"5","author":"LM Collins","year":"2004","unstructured":"Collins, L.M., Murphy, S.A., Bierman, K.L.: A conceptual framework for adaptive preventive interventions. Prevention science 5(3), 185\u2013196 (2004)","journal-title":"Prevention science"},{"key":"1773_CR2","doi-asserted-by":"crossref","unstructured":"Hardeman, W., Houghton, J., Lane, K., Jones, A., Naughton, F.: A systematic review of just-in-time adaptive interventions (Jitais) to promote physical activity. International Journal of Behavioral Nutrition and Physical Activity 16(1), 31 (2019)","DOI":"10.1186\/s12966-019-0792-7"},{"issue":"6","key":"1773_CR3","doi-asserted-by":"publisher","first-page":"446","DOI":"10.1007\/s12160-016-9830-8","volume":"52","author":"I Nahum-Shani","year":"2017","unstructured":"Nahum-Shani, I., Smith, S.N., Spring, B.J., Collins, L.M., Witkiewitz, K., Tewari, A.,\u00a0Murphy, S.A.: Just-in-time adaptive interventions (Jitais) in mobile health: key components\u00a0and design principles for ongoing health behavior support. Annals of Behavioral\u00a0Medicine 52(6), 446\u2013462 (2017)","journal-title":"Annals of Behavioral Medicine"},{"issue":"5","key":"1773_CR4","doi-asserted-by":"publisher","first-page":"673","DOI":"10.1007\/s12529-016-9627-y","volume":"24","author":"SP Goldstein","year":"2017","unstructured":"Goldstein, S.P., Evans, B.C., Flack, D., Juarascio, A., Manasse, S., Zhang, F., Forman, E.M.: Return of the Jitai: applying a just-in-time adaptive intervention framework to the development of m-health solutions for addictive behaviors. International journal of behavioral medicine 24(5), 673\u2013682 (2017)","journal-title":"International journal of behavioral medicine"},{"key":"1773_CR5","doi-asserted-by":"crossref","unstructured":"Sarker, H., Sharmin, M., Ali, A.A., Rahman, M.M., Bari, R., Hossain, S.M., Kumar, S.: Assessing the availability of users to engage in just-in-time intervention in the natural environment. In: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 909\u2013920 (2014)","DOI":"10.1145\/2632048.2636082"},{"key":"1773_CR6","doi-asserted-by":"crossref","unstructured":"Thomas, J.G., Bond, D.S.: Behavioral response to a just-in-time adaptive intervention (Jitai) to reduce sedentary behavior in obese adults: Implications for Jitai optimization. Health Psychology 34(S), 1261 (2015)","DOI":"10.1037\/hea0000304"},{"issue":"10","key":"1773_CR7","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.7994","volume":"19","author":"E Yom-Tov","year":"2017","unstructured":"Yom-Tov, E., Feraru, G., Kozdoba, M., Mannor, S., Tennenholtz, M., Hochberg, I.: Encouraging\u00a0physical activity in patients with diabetes: intervention using a reinforcement\u00a0learning system. Journal of medical Internet research 19(10), e338 (2017)","journal-title":"Journal of medical Internet research"},{"key":"1773_CR8","doi-asserted-by":"crossref","unstructured":"G\u00f6n\u00fcl, S., Naml\u0131, T., Co\u015far, A., and Toroslu, \u0130.H.: A reinforcement learning based algorithm for personalization of digital, just-in-time, adaptive interventions. Artificial Intelligence in Medicine 115, 102062 (2021)","DOI":"10.1016\/j.artmed.2021.102062"},{"key":"1773_CR9","doi-asserted-by":"crossref","unstructured":"Aguilera, A., Figueroa, C.A., Hernandez-Ramos, R., Sarkar, U., Cemballi, A., Gomez-Pathak, L., Miramontes, J., Yom-Tov, E., Chakraborty, B., Yan, X., et al.: mhealth app using machine learning to increase physical activity in diabetes and depression: clinical trial protocol for the diamante study. BMJ open 10(8), e034723 (2020)","DOI":"10.1136\/bmjopen-2019-034723"},{"issue":"2","key":"1773_CR10","doi-asserted-by":"publisher","first-page":"276","DOI":"10.1007\/s10865-018-9964-1","volume":"42","author":"EM Forman","year":"2019","unstructured":"Forman, E.M., Kerrigan, S.G., Butryn, M.L., Juarascio, A.S., Manasse, S.M., Onta\u00f1\u00f3n, S., Dallal, D.H., Crochiere, R.J., Moskow, D.: Can the artificial intelligence technique of reinforcement learning use continuously-monitored digital data to optimize treatment for weight loss? Journal of behavioral medicine 42(2), 276\u2013290 (2019)","journal-title":"Journal of behavioral medicine"},{"key":"1773_CR11","doi-asserted-by":"crossref","unstructured":"Rabbi, M., Aung, M.H., Zhang, M., Choudhury, T.: My behavior: automatic personalized health feedback from user behaviors and preferences using smartphones. In: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 707\u2013718. ACM (2015)","DOI":"10.1145\/2750858.2805840"},{"key":"1773_CR12","doi-asserted-by":"publisher","first-page":"13","DOI":"10.2147\/AHCT.S57844","volume":"1","author":"CA Pellegrini","year":"2015","unstructured":"Pellegrini, C.A., Pfammatter, A.F., Conroy, D.E., Spring, B.: Smartphone applications\u00a0to support weight loss: current perspectives. Advanced health care technologies 1, 13\u00a0(2015)","journal-title":"Advanced health care technologies"},{"key":"1773_CR13","unstructured":"Saunders, W., Sastry, G., Stuhlm\u00fcller, A., Evans, O.: Trial without error: Towards safe reinforcement learning via human intervention. In: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2067\u20132069 (2018)"},{"key":"1773_CR14","unstructured":"Zhou, M., Mintz, Y., Fukuoka, Y., Goldberg, K., Flowers, E., Kaminsky, P., Castillejo, A., Aswani, A.: Personalizing mobile fitness apps using reinforcement learning. In: CEUR workshop proceedings, vol. 2068. NIH Public Access (2018)"},{"key":"1773_CR15","doi-asserted-by":"crossref","unstructured":"Tabatabaei, S.A., Hoogendoorn, M., van Halteren, A.: Narrowing reinforcement learning: Overcoming the cold start problem for personalized health interventions. In: International Conference on Principles and Practice of Multi-Agent Systems, pp. 312\u2013327. Springer (2018)","DOI":"10.1007\/978-3-030-03098-8_19"},{"key":"1773_CR16","unstructured":"Tomkins, S., Liao, P., Yeung, S., Klasnja, P., Murphy, S.: Intelligent pooling in thompson sampling for rapid personalization in mobile health (2019)"},{"key":"1773_CR17","doi-asserted-by":"crossref","unstructured":"Gonul, S., Namli, T., Baskaya, M., Sinaci, A.A., Cosar, A., Toroslu, I.H.: Optimization of just-in-time adaptive interventions using reinforcement learning. In: International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, pp. 334\u2013341. Springer (2018)","DOI":"10.1007\/978-3-319-92058-0_32"},{"issue":"1","key":"1773_CR18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3381007","volume":"4","author":"P Liao","year":"2020","unstructured":"Liao, P., Greenewald, K., Klasnja, P., Murphy, S.: Personalized heartsteps: A reinforcement\u00a0learning algorithm for optimizing physical activity. Proceedings of the ACM on\u00a0Interactive, Mobile, Wearable and Ubiquitous Technologies 4(1), 1\u201322 (2020)","journal-title":"Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"},{"key":"1773_CR19","doi-asserted-by":"crossref","unstructured":"Ameko, M.K., Beltzer, M.L., Cai, L., Boukhechba, M., Teachman, B.A., Barnes, L.E.: Online contextual multi-armed bandits for mobile health interventions: A case study on emotion regulation. In: Fourteenth ACM Conference on Recommender Systems, pp. 249\u2013258 (2020)","DOI":"10.1145\/3383313.3412244"},{"key":"1773_CR20","unstructured":"Sutton, R.S., Barto, A.G.: Reinforcement learning: An introduction. MIT press (2018)"},{"issue":"1","key":"1773_CR21","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1007\/s00146-013-0532-5","volume":"30","author":"B Kamphorst","year":"2015","unstructured":"Kamphorst, B., Kalis, A.: Why option generation matters for the design of autonomous\u00a0e-coaching systems. AI & SOCIETY 30(1), 77\u201388 (2015)","journal-title":"AI & SOCIETY"},{"issue":"2","key":"1773_CR22","doi-asserted-by":"publisher","first-page":"408","DOI":"10.1037\/a0015512","volume":"116","author":"R Tobias","year":"2009","unstructured":"Tobias, R.: Changing behavior by memory aids: A social psychological model of prospective\u00a0memory and habit development tested with dynamic field data. Psychological\u00a0review 116(2), 408\u2013438 (2009)","journal-title":"Psychological review"},{"key":"1773_CR23","doi-asserted-by":"crossref","unstructured":"Zhang, C., Lakens, D., IJsselsteijn, W.A.: Theory integration for lifestyle behavior change in the digital age: An adaptive decision-making framework. Journal of Medical Internet Research 23(4), e17127 (2021)","DOI":"10.2196\/17127"},{"issue":"5","key":"1773_CR24","first-page":"1161","volume":"25","author":"DC Rubin","year":"1999","unstructured":"Rubin, D.C., Hinton, S., Wenzel, A.: The precise time course of retention. Journal of\u00a0Experimental Psychology: Learning, Memory, and Cognition 25(5), 1161\u20131176 (1999)","journal-title":"Journal of Experimental Psychology: Learning, Memory, and Cognition"},{"key":"1773_CR25","unstructured":"Savage, L.J.: The foundations of statistics. Courier Corporation (1972)"},{"issue":"7","key":"1773_CR26","doi-asserted-by":"publisher","first-page":"498","DOI":"10.1090\/S0002-9904-1945-08391-8","volume":"51","author":"J Von Neumann","year":"1945","unstructured":"Von Neumann, J., Morgenstern, O.: Theory of games and economic behavior. Bull.\u00a0Amer. Math. Soc 51(7), 498{504 (1945)","journal-title":"Bull. Amer. Math. Soc"},{"key":"1773_CR27","doi-asserted-by":"crossref","unstructured":"Dunton, G.F., Liao, Y., Intille, S., Huh, J., Leventhal, A.: Momentary assessment of contextual influences on affective response during physical activity. Health Psychology 34(12), 1145 (2015)","DOI":"10.1037\/hea0000223"},{"issue":"2","key":"1773_CR28","doi-asserted-by":"publisher","first-page":"174","DOI":"10.1037\/0022-3514.93.2.174","volume":"93","author":"MJ Louro","year":"2007","unstructured":"Louro, M.J., Pieters, R., Zeelenberg, M.: Dynamics of multiple-goal pursuit. Journal of\u00a0personality and social psychology 93(2), 174 (2007)","journal-title":"Journal of personality and social psychology"},{"key":"1773_CR29","volume-title":"and Petkovic, M","author":"V Mihajlovic","year":"2001","unstructured":"Mihajlovic, V., Petkovic, M.: Dynamic bayesian networks: A state of the art. University\u00a0of Twente Document Repository (2001)"},{"key":"1773_CR30","unstructured":"Kaplan, W.: Advanced calculus. Pearson Education India (1952)"},{"issue":"3\u20134","key":"1773_CR31","first-page":"229","volume":"8","author":"RJ Williams","year":"1992","unstructured":"Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement\u00a0learning. Machine learning 8(3\u20134), 229\u2013256 (1992)","journal-title":"Machine learning"},{"key":"1773_CR32","unstructured":"Fujita, Y., Maeda, S.i.: Clipped action policy gradient. In: Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 80, pp. 1597\u20131606. PMLR, Stockholmsm\u00e4ssan, Stockholm Sweden (2018)"},{"key":"1773_CR33","first-page":"985","volume":"8","author":"S Wang","year":"2021","unstructured":"Wang, S., Scheider, S., Sporrel, K., Deutekom, M., Timmer, J., Kr\u00f6se, B.: What are\u00a0good situations for running? a machine learning study using mobile and geographical\u00a0data. Frontiers in Public Health 8, 985 (2021)","journal-title":"Frontiers in Public Health"},{"key":"1773_CR34","doi-asserted-by":"crossref","unstructured":"Wang, S., Sporrel, K., van Hoof, H., Simons, M., de Boer, R.D., Ettema, D., Nibbeling, N., Deutekom, M. and Kr\u00f6se, B.: Reinforcement learning to send reminders at right moments in smartphone exercise application: A feasibility study. International Journal of Environmental Research and Public Health, 18(11), 6059 (2021)","DOI":"10.3390\/ijerph18116059"}],"container-title":["Journal of Medical Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10916-021-01773-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10916-021-01773-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10916-021-01773-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,11,19]],"date-time":"2021-11-19T01:03:12Z","timestamp":1637283792000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10916-021-01773-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,18]]},"references-count":34,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["1773"],"URL":"https:\/\/doi.org\/10.1007\/s10916-021-01773-0","relation":{},"ISSN":["0148-5598","1573-689X"],"issn-type":[{"value":"0148-5598","type":"print"},{"value":"1573-689X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,18]]},"assertion":[{"value":"4 June 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 September 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 October 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This article does not contain any studies with human participants or animals performed by any of the authors.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Research involving human and animal participants"}},{"value":"The authors declare that they have no conflict of interest.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of interest"}}],"article-number":"102"}}