{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,2]],"date-time":"2026-06-02T14:32:51Z","timestamp":1780410771106,"version":"3.54.1"},"reference-count":46,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2022,7,22]],"date-time":"2022-07-22T00:00:00Z","timestamp":1658448000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"NIH","award":["IUG3DE028723"],"award-info":[{"award-number":["IUG3DE028723"]}]},{"name":"NIH","award":["P50DA054039"],"award-info":[{"award-number":["P50DA054039"]}]},{"name":"NIH","award":["P41EB028242"],"award-info":[{"award-number":["P41EB028242"]}]},{"name":"NIH","award":["U01CA229437"],"award-info":[{"award-number":["U01CA229437"]}]},{"name":"NIH","award":["UH3DE028723"],"award-info":[{"award-number":["UH3DE028723"]}]},{"name":"NIH","award":["R01MH123804"],"award-info":[{"award-number":["R01MH123804"]}]},{"name":"NIH","award":["NSF CBET\u20132112085"],"award-info":[{"award-number":["NSF CBET\u20132112085"]}]},{"name":"NIH","award":["DGE1745303"],"award-info":[{"award-number":["DGE1745303"]}]},{"name":"National Science Foundation","award":["IUG3DE028723"],"award-info":[{"award-number":["IUG3DE028723"]}]},{"name":"National Science Foundation","award":["P50DA054039"],"award-info":[{"award-number":["P50DA054039"]}]},{"name":"National Science Foundation","award":["P41EB028242"],"award-info":[{"award-number":["P41EB028242"]}]},{"name":"National Science Foundation","award":["U01CA229437"],"award-info":[{"award-number":["U01CA229437"]}]},{"name":"National Science Foundation","award":["UH3DE028723"],"award-info":[{"award-number":["UH3DE028723"]}]},{"name":"National Science Foundation","award":["R01MH123804"],"award-info":[{"award-number":["R01MH123804"]}]},{"name":"National Science Foundation","award":["NSF CBET\u20132112085"],"award-info":[{"award-number":["NSF CBET\u20132112085"]}]},{"name":"National Science Foundation","award":["DGE1745303"],"award-info":[{"award-number":["DGE1745303"]}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation Graduate Research Fellowship","doi-asserted-by":"publisher","award":["IUG3DE028723"],"award-info":[{"award-number":["IUG3DE028723"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation Graduate Research Fellowship","doi-asserted-by":"publisher","award":["P50DA054039"],"award-info":[{"award-number":["P50DA054039"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation Graduate Research Fellowship","doi-asserted-by":"publisher","award":["P41EB028242"],"award-info":[{"award-number":["P41EB028242"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation Graduate Research Fellowship","doi-asserted-by":"publisher","award":["U01CA229437"],"award-info":[{"award-number":["U01CA229437"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation Graduate Research Fellowship","doi-asserted-by":"publisher","award":["UH3DE028723"],"award-info":[{"award-number":["UH3DE028723"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation Graduate Research Fellowship","doi-asserted-by":"publisher","award":["R01MH123804"],"award-info":[{"award-number":["R01MH123804"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation Graduate Research Fellowship","doi-asserted-by":"publisher","award":["NSF CBET\u20132112085"],"award-info":[{"award-number":["NSF CBET\u20132112085"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation Graduate Research Fellowship","doi-asserted-by":"publisher","award":["DGE1745303"],"award-info":[{"award-number":["DGE1745303"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Online reinforcement learning (RL) algorithms are increasingly used to personalize digital interventions in the fields of mobile health and online education. Common challenges in designing and testing an RL algorithm in these settings include ensuring the RL algorithm can learn and run stably under real-time constraints, and accounting for the complexity of the environment, e.g., a lack of accurate mechanistic models for the user dynamics. To guide how one can tackle these challenges, we extend the PCS (predictability, computability, stability) framework, a data science framework that incorporates best practices from machine learning and statistics in supervised learning to the design of RL algorithms for the digital interventions setting. Furthermore, we provide guidelines on how to design simulation environments, a crucial tool for evaluating RL candidate algorithms using the PCS framework. We show how we used the PCS framework to design an RL algorithm for Oralytics, a mobile health study aiming to improve users\u2019 tooth-brushing behaviors through the personalized delivery of intervention messages. Oralytics will go into the field in late 2022.<\/jats:p>","DOI":"10.3390\/a15080255","type":"journal-article","created":{"date-parts":[[2022,7,22]],"date-time":"2022-07-22T12:53:45Z","timestamp":1658494425000},"page":"255","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":30,"title":["Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-Implementation Guidelines"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4779-9115","authenticated-orcid":false,"given":"Anna L.","family":"Trella","sequence":"first","affiliation":[{"name":"School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02420, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0850-4978","authenticated-orcid":false,"given":"Kelly W.","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02420, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6138-9089","authenticated-orcid":false,"given":"Inbal","family":"Nahum-Shani","sequence":"additional","affiliation":[{"name":"Institute for Social Research, University of Michigan, Ann Arbor, MI 48109, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3167-3318","authenticated-orcid":false,"given":"Vivek","family":"Shetty","sequence":"additional","affiliation":[{"name":"Schools of Dentistry & Engineering, University of California, Los Angeles, CA 90095, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2886-3898","authenticated-orcid":false,"given":"Finale","family":"Doshi-Velez","sequence":"additional","affiliation":[{"name":"School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02420, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2032-4286","authenticated-orcid":false,"given":"Susan A.","family":"Murphy","sequence":"additional","affiliation":[{"name":"School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02420, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,7,22]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3381007","article-title":"Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activity","volume":"4","author":"Liao","year":"2020","journal-title":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"e338","DOI":"10.2196\/jmir.7994","article-title":"Encouraging physical activity in patients with diabetes: Intervention using a reinforcement learning system","volume":"19","author":"Feraru","year":"2017","journal-title":"J. Med. Internet Res."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"276","DOI":"10.1007\/s10865-018-9964-1","article-title":"Can the artificial intelligence technique of reinforcement learning use continuously-monitored digital data to optimize treatment for weight loss?","volume":"42","author":"Forman","year":"2019","journal-title":"J. Behav. Med."},{"key":"ref_4","unstructured":"Allen, S. (2022, June 01). Stanford Computational Policy Lab Pretrial Nudges. Available online: https:\/\/policylab.stanford.edu\/projects\/nudge.html."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2389","DOI":"10.1007\/s10994-021-05983-y","article-title":"Bandit algorithms to personalize educational chatbots","volume":"110","author":"Cai","year":"2021","journal-title":"Mach. Learn."},{"key":"ref_6","unstructured":"Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (2018). Bandit Learning with Implicit Feedback. Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_7","unstructured":"Bezos, J.P. (2022, June 01). 1997 Letter to Amazon Shareholders, Available online: https:\/\/www.sec.gov\/Archives\/edgar\/data\/1018724\/000119312516530910\/d168744dex991.htm."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"3920","DOI":"10.1073\/pnas.1901326117","article-title":"Veridical data science","volume":"117","author":"Yu","year":"2020","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_9","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"107","DOI":"10.3233\/DS-200028","article-title":"Reinforcement learning for personalization: A systematic literature review","volume":"3","author":"Grua","year":"2020","journal-title":"Data Sci."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"338","DOI":"10.1109\/TAC.2005.844079","article-title":"Bandit problems with side observations","volume":"50","author":"Wang","year":"2005","journal-title":"IEEE Trans. Autom. Control"},{"key":"ref_12","first-page":"96","article-title":"The epoch-greedy algorithm for contextual multi-armed bandits","volume":"20","author":"Langford","year":"2007","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Tewari, A., and Murphy, S.A. (2017). From ads to interventions: Contextual bandits in mobile health. Mobile Health, Springer.","DOI":"10.1007\/978-3-319-51394-2_25"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1207\/s15327744joce1603&4_2","article-title":"What is personalization? Perspectives on the design and implementation of personalization in information systems","volume":"16","author":"Fan","year":"2006","journal-title":"J. Organ. Comput. Electron. Commer."},{"key":"ref_15","unstructured":"Thomas, P., and Brunskill, E. (2016, January 20\u201322). Data-efficient off-policy policy evaluation for reinforcement learning. Proceedings of the International Conference on Machine Learning. PMLR, New York, NY, USA."},{"key":"ref_16","unstructured":"Levine, S., Kumar, A., Tucker, G., and Fu, J. (2020). Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1112","DOI":"10.1080\/01621459.2017.1305274","article-title":"Assessing time-varying causal effect moderation in mobile health","volume":"113","author":"Boruvka","year":"2018","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"e2014602118","DOI":"10.1073\/pnas.2014602118","article-title":"Confidence intervals for policy evaluation in adaptive experiments","volume":"118","author":"Hadad","year":"2021","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_19","first-page":"209","article-title":"Power Constrained Bandits","volume":"Volume 149","author":"Jung","year":"2021","journal-title":"Proceedings of the 6th Machine Learning for Healthcare Conference, PMLR"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Murnane, E.L., Huffaker, D., and Kossinets, G. (2015, January 7\u201311). Mobile Health Apps: Adoption, Adherence, and Abandonment. Proceedings of the Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers, Osaka, Japan. UbiComp\/ISWC\u201915 Adjunct.","DOI":"10.1145\/2800835.2800943"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"e86","DOI":"10.2196\/jmir.2583","article-title":"Opportunities and Challenges for Smartphone Applications in Supporting Health Behavior Change: Qualitative Study","volume":"15","author":"Dennison","year":"2013","journal-title":"J. Med. Internet Res."},{"key":"ref_22","unstructured":"Agarwal, A., Alomar, A., Alumootil, V., Shah, D., Shen, D., Xu, Z., and Yang, C. (2021). PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1225","DOI":"10.1093\/jamia\/ocab001","article-title":"Adaptive learning algorithms to optimize mobile applications for behavioral health: Guidelines for design decisions","volume":"28","author":"Figueroa","year":"2021","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Wei, H., Chen, C., Liu, C., Zheng, G., and Li, Z. (2020). Learning to simulate on sparse trajectory data. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer.","DOI":"10.1007\/978-3-030-67667-4_32"},{"key":"ref_25","unstructured":"Ie, E., Hsu, C.W., Mladenov, M., Jain, V., Narvekar, S., Wang, J., Wu, R., and Boutilier, C. (2019). RecSim: A Configurable Simulation Platform for Recommender Systems. arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Santana, M.R.O., Melo, L.C., Camargo, F.H.F., Brand\u00e3o, B., Soares, A., Oliveira, R.M., and Caetano, S. (2020). MARS-Gym: A Gym framework to model, train, and evaluate Recommender Systems for Marketplaces. arXiv.","DOI":"10.1109\/ICDMW51313.2020.00035"},{"key":"ref_27","unstructured":"Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). OpenAI Gym. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10916-021-01773-0","article-title":"Optimizing Adaptive Notifications in Mobile Health Interventions Systems: Reinforcement Learning from a Data-driven Behavioral Simulator","volume":"45","author":"Wang","year":"2021","journal-title":"J. Med. Syst."},{"key":"ref_29","unstructured":"Singh, A., Halpern, Y., Thain, N., Christakopoulou, K., Chi, E., Chen, J., and Beutel, A. (2020, January 26\u201327). Building healthy recommendation sequences for everyone: A safe reinforcement learning approach. Proceedings of the FAccTRec Workshop, Online."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Korzepa, M., Petersen, M.K., Larsen, J.E., and M\u00f8rup, M. (2020, January 14\u201317). Simulation Environment for Guiding the Design of Contextual Personalization Systems in the Context of Hearing Aids. Proceedings of the Adjunct Publication of the 28th ACM Conference on User Modeling, Adaptation and Personalization, Genoa, Italy. UMAP \u201920 Adjunct.","DOI":"10.1145\/3386392.3399291"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Miller, T., Oren, N., Sakurai, Y., Noda, I., Savarimuthu, B.T.R., and Cao Son, T. (2018). Personalization of Health Interventions Using Cluster-Based Reinforcement Learning. PRIMA 2018: Principles and Practice of Multi-Agent Systems, Springer International Publishing.","DOI":"10.1007\/978-3-030-03098-8"},{"key":"ref_32","unstructured":"Hassouni, A.E., Hoogendoorn, M., van Otterlo, M., Eiben, A.E., Muhonen, V., and Barbaro, E. (2018). A clustering-based reinforcement learning approach for tailored personalization of e-Health interventions. arXiv."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"S135","DOI":"10.1111\/insr.12427","article-title":"Stable Discovery of Interpretable Subgroups via Calibration in Causal Studies","volume":"88","author":"Dwivedi","year":"2020","journal-title":"Int. Stat. Rev."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1002\/sam.11486","article-title":"Next waves in veridical network embedding","volume":"14","author":"Ward","year":"2021","journal-title":"Stat. Anal. Data Min. ASA Data Sci. J."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"621","DOI":"10.3390\/ai2040037","article-title":"A new method to compare the interpretability of rule-based algorithms","volume":"2","author":"Margot","year":"2021","journal-title":"AI"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"e17347","DOI":"10.2196\/17347","article-title":"A Scalable System for Passively Monitoring Oral Health Behaviors Using Electronic Toothbrushes in the Home Setting: Development and Feasibility Study","volume":"8","author":"Shetty","year":"2020","journal-title":"JMIR Mhealth Uhealth"},{"key":"ref_37","unstructured":"Jiang, N., Kulesza, A., Singh, S., and Lewis, R. (2015, January 4\u20138). The dependence of effective planning horizon on model accuracy. Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, Istanbul, Turkey."},{"key":"ref_38","unstructured":"Russo, D., Roy, B.V., Kazerouni, A., and Osband, I. (2022, June 01). A Tutorial on Thompson Sampling, Available online: http:\/\/xxx.lanl.gov\/abs\/1707.02038."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Zhu, F., Guo, J., Xu, Z., Liao, P., Yang, L., and Huang, J. (2018). Group-driven reinforcement learning for personalized mhealth intervention. International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.","DOI":"10.1007\/978-3-030-00928-1_67"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"2685","DOI":"10.1007\/s10994-021-05995-8","article-title":"IntelligentPooling: Practical Thompson sampling for mHealth","volume":"110","author":"Tomkins","year":"2021","journal-title":"Mach. Learn."},{"key":"ref_41","first-page":"4848","article-title":"Multi-task learning for contextual bandits","volume":"30","author":"Deshmukh","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_42","unstructured":"Vaswani, S., Schmidt, M., and Lakshmanan, L. (2017, January 20\u201322). Horde of bandits using gaussian markov random fields. Proceedings of the Artificial Intelligence and Statistics, PMLR, Fort Lauderdale, FL, USA."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s40488-021-00121-4","article-title":"A comparison of zero-inflated and hurdle models for modeling zero-inflated count data","volume":"8","author":"Feng","year":"2021","journal-title":"J. Stat. Distrib. Appl."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1093\/ije\/dyp334","article-title":"Illustrating bias due to conditioning on a collider","volume":"39","author":"Cole","year":"2010","journal-title":"Int. J. Epidemiol."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"640","DOI":"10.1093\/ije\/dyy275","article-title":"Educational Note: Paradoxical collider effect in the analysis of non-communicable disease epidemiological data: A reproducible illustration and web application","volume":"48","author":"Schomaker","year":"2019","journal-title":"Int. J. Epidemiol."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1146\/annurev-soc-071913-043455","article-title":"Endogenous selection bias: The problem of conditioning on a collider variable","volume":"40","author":"Elwert","year":"2014","journal-title":"Annu. Rev. Sociol."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/15\/8\/255\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:54:51Z","timestamp":1760140491000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/15\/8\/255"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,22]]},"references-count":46,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2022,8]]}},"alternative-id":["a15080255"],"URL":"https:\/\/doi.org\/10.3390\/a15080255","relation":{"has-preprint":[{"id-type":"doi","id":"10.20944\/preprints202206.0028.v1","asserted-by":"object"}]},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,22]]}}}