{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,5]],"date-time":"2026-08-05T08:23:29Z","timestamp":1785918209773,"version":"3.56.0"},"reference-count":55,"publisher":"MDPI AG","issue":"18","license":[{"start":{"date-parts":[[2020,9,6]],"date-time":"2020-09-06T00:00:00Z","timestamp":1599350400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/P00993X\/1"],"award-info":[{"award-number":["EP\/P00993X\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>(1) Background: People living with type 1 diabetes (T1D) require self-management to maintain blood glucose (BG) levels in a therapeutic range through the delivery of exogenous insulin. However, due to the various variability, uncertainty and complex glucose dynamics, optimizing the doses of insulin delivery to minimize the risk of hyperglycemia and hypoglycemia is still an open problem. (2) Methods: In this work, we propose a novel insulin bolus advisor which uses deep reinforcement learning (DRL) and continuous glucose monitoring to optimize insulin dosing at mealtime. In particular, an actor-critic model based on deep deterministic policy gradient is designed to compute mealtime insulin doses. The proposed system architecture uses a two-step learning framework, in which a population model is first obtained and then personalized by subject-specific data. Prioritized memory replay is adopted to accelerate the training process in clinical practice. To validate the algorithm, we employ a customized version of the FDA-accepted UVA\/Padova T1D simulator to perform in silico trials on 10 adult subjects and 10 adolescent subjects. (3) Results: Compared to a standard bolus calculator as the baseline, the DRL insulin bolus advisor significantly improved the average percentage time in target range (70\u2013180 mg\/dL) from 74.1%\u00b18.4% to 80.9%\u00b16.9% (p&lt;0.01) and 54.9%\u00b112.4% to 61.6%\u00b114.1% (p&lt;0.01) in the the adult and adolescent cohorts, respectively, while reducing hypoglycemia. (4) Conclusions: The proposed algorithm has the potential to improve mealtime bolus insulin delivery in people with T1D and is a feasible candidate for future clinical validation.<\/jats:p>","DOI":"10.3390\/s20185058","type":"journal-article","created":{"date-parts":[[2020,9,6]],"date-time":"2020-09-06T23:12:49Z","timestamp":1599433969000},"page":"5058","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":74,"title":["An Insulin Bolus Advisor for Type 1 Diabetes Using Deep Reinforcement Learning"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9782-3470","authenticated-orcid":false,"given":"Taiyu","family":"Zhu","sequence":"first","affiliation":[{"name":"Centre for Bio-Inspired Technology, Department of Electrical and Electronic Engineering, Imperial College London, London SW7 2AZ, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3073-3128","authenticated-orcid":false,"given":"Kezhi","family":"Li","sequence":"additional","affiliation":[{"name":"Centre for Bio-Inspired Technology, Department of Electrical and Electronic Engineering, Imperial College London, London SW7 2AZ, UK"},{"name":"Institute of Health Informatics, University College London, London NW1 2DA, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3415-3748","authenticated-orcid":false,"given":"Lei","family":"Kuang","sequence":"additional","affiliation":[{"name":"Centre for Bio-Inspired Technology, Department of Electrical and Electronic Engineering, Imperial College London, London SW7 2AZ, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7088-5807","authenticated-orcid":false,"given":"Pau","family":"Herrero","sequence":"additional","affiliation":[{"name":"Centre for Bio-Inspired Technology, Department of Electrical and Electronic Engineering, Imperial College London, London SW7 2AZ, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2476-3857","authenticated-orcid":false,"given":"Pantelis","family":"Georgiou","sequence":"additional","affiliation":[{"name":"Centre for Bio-Inspired Technology, Department of Electrical and Electronic Engineering, Imperial College London, London SW7 2AZ, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,9,6]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"American Diabetes Association (2017). 2. Classification and diagnosis of diabetes. Diabetes Care, 40, S11\u2013S24.","DOI":"10.2337\/dc17-S005"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"425","DOI":"10.1038\/nrendo.2012.28","article-title":"Management of diabetes mellitus: Is the pump mightier than the pen?","volume":"8","author":"Pickup","year":"2012","journal-title":"Nat. Rev. Endocrinol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1052","DOI":"10.1001\/jama.2015.9536","article-title":"Diabetes: Advances in diagnosis and treatment","volume":"314","author":"Nathan","year":"2015","journal-title":"JAMA"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"190","DOI":"10.1177\/1932296817718213","article-title":"Bolus advisors: Sources of error, targets for improvement","volume":"12","author":"Walsh","year":"2018","journal-title":"J. Diabetes Sci. Technol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"eaaw4949","DOI":"10.1126\/scitranslmed.aaw4949","article-title":"Advances in artificial pancreas systems","volume":"11","author":"Boughton","year":"2019","journal-title":"Sci. Transl. Med."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"996","DOI":"10.1109\/TBME.2006.872818","article-title":"Run-to-run control of blood glucose concentrations for people with type 1 diabetes mellitus","volume":"53","author":"Owens","year":"2006","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1131","DOI":"10.2337\/dc06-2115","article-title":"Prandial insulin dosing using run-to-run control: Application of clinical data and medical expertise to define a suitable performance metric","volume":"30","author":"Palerm","year":"2007","journal-title":"Diabetes Care"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1109\/JBHI.2015.2464088","article-title":"An advanced bolus calculator for type 1 diabetes: System architecture and usability results","volume":"20","author":"Pesl","year":"2015","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"487","DOI":"10.1089\/dia.2015.0413","article-title":"Clinical safety and feasibility of the advanced bolus calculator for type 1 diabetes based on case-based reasoning: A 6-week nonrandomized single-arm pilot study","volume":"18","author":"Reddy","year":"2016","journal-title":"Diabetes Technol. Ther."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"e10775","DOI":"10.2196\/10775","article-title":"Artificial intelligence for diabetes management and decision support: Literature review","volume":"20","author":"Contreras","year":"2018","journal-title":"J. Med. Internet Res."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"612","DOI":"10.1038\/s42255-020-0212-y","article-title":"An artificial intelligence decision support system for the management of type 1 diabetes","volume":"2","author":"Tyler","year":"2020","journal-title":"Nat. Metab."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"3934","DOI":"10.3390\/s100403934","article-title":"A monitoring and advisory system for diabetes patient management using a rule-based method and KNN","volume":"10","author":"Lee","year":"2010","journal-title":"Sensors"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1109\/LCSYS.2018.2844179","article-title":"Postprandial glucose regulation via KNN meal classification in type 1 diabetes","volume":"3","author":"Aiello","year":"2018","journal-title":"IEEE Control Syst. Lett."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1177\/1932296818759558","article-title":"A neural-network-based approach to personalize insulin bolus calculation using continuous glucose monitoring","volume":"12","author":"Cappon","year":"2018","journal-title":"J. Diabetes Sci. Technol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"2633","DOI":"10.1109\/JBHI.2018.2887067","article-title":"A dual mode adaptive basal-bolus advisor based on reinforcement learning","volume":"23","author":"Sun","year":"2018","journal-title":"IEEE J. Biomed. Health Inf."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"549","DOI":"10.1177\/1932296815583333","article-title":"GoCARB in the context of an artificial pancreas","volume":"9","author":"Agianniotis","year":"2015","journal-title":"J. Diabetes Sci. Technol."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"661","DOI":"10.1586\/17434440.2013.827515","article-title":"The use of reinforcement learning algorithms to meet the challenges of an artificial pancreas","volume":"10","author":"Bothe","year":"2013","journal-title":"Expert Rev. Med. Devices"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"101836","DOI":"10.1016\/j.artmed.2020.101836","article-title":"Reinforcement learning application in diabetes blood glucose control: A systematic review","volume":"104","author":"Tejedor","year":"2020","journal-title":"Artif. Intell. Med."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Lee, S., Kim, J., Park, S.W., Jin, S.M., and Park, S.M. (2020). Toward a fully automated artificial pancreas system Using a bioinspired reinforcement learning design: In silico validation. IEEE J. Biomed. Health Inform.","DOI":"10.1109\/JBHI.2020.3002022"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1038\/nature24270","article-title":"Mastering the game of go without human knowledge","volume":"550","author":"Silver","year":"2017","journal-title":"Nature"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"70","DOI":"10.2352\/ISSN.2470-1173.2017.19.AVM-023","article-title":"Deep reinforcement learning framework for autonomous driving","volume":"2017","author":"Sallab","year":"2017","journal-title":"Electron. Imaging"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Nemati, S., Ghassemi, M.M., and Clifford, G.D. (2016, January 17). Optimal medication dosing from suboptimal clinical examples: A deep reinforcement learning approach. Proceedings of the 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA.","DOI":"10.1109\/EMBC.2016.7591355"},{"key":"ref_23","unstructured":"Zhu, T., Li, K., Herrero, P., Chen, J., and Georgiou, P. (2018, January 13). A deep learning algorithm for personalized blood glucose prediction. Proceedings of the 3rd International Workshop on Knowledge Discovery in Healthcare Data, IJCAI-ECAI, Stockholm, Schweden."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1109\/JBHI.2019.2931842","article-title":"GluNet: A deep learning framework for accurate glucose forecasting","volume":"24","author":"Li","year":"2019","journal-title":"IEEE J. Biomed. Health Inf."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Mirshekarian, S., Shen, H., Bunescu, R., and Marling, C. (2019, January 23\u201327). LSTMs and neural Attention models for blood glucose prediction: Comparative experiments on real and synthetic Data. Proceedings of the 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany.","DOI":"10.1109\/EMBC.2019.8856940"},{"key":"ref_26","first-page":"1","article-title":"Dilated recurrent neural networks for glucose forecasting in type 1 diabetes","volume":"2020","author":"Zhu","year":"2020","journal-title":"J. Healthc. Inform. Res."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1177\/1932296813514502","article-title":"The UVA\/PADOVA type 1 diabetes simulator: New features","volume":"8","author":"Micheletto","year":"2014","journal-title":"J. Diabetes Sci. Technol."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"428","DOI":"10.1089\/dia.2013.0377","article-title":"The university of Virginia\/Padova type 1 diabetes simulator matches the glucose traces of a clinical trial","volume":"16","author":"Visentin","year":"2014","journal-title":"Diabetes Technol. Ther."},{"key":"ref_29","first-page":"1087","article-title":"Advanced insulin bolus advisor based on run-to-run control and case-based reasoning","volume":"19","author":"Herrero","year":"2014","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Sun, Q., Jankovic, M.V., Bally, L., and Mougiakakou, S.G. (2018, January 20\u201321). Predicting blood glucose with an LSTM and Bi-LSTM based deep neural network. Proceedings of the IEEE 2018 14th Symposium on Neural Networks and Applications (NEUREL), Belgrade, Serbia.","DOI":"10.1109\/NEUREL.2018.8586990"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_32","unstructured":"Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014, January 21\u201326). Deterministic policy gradient algorithms. Proceedings of the 31st ICML, ICML\u201914, Beijing, China."},{"key":"ref_33","unstructured":"Bengio, Y., and LeCun, Y. (2016, January 2\u20134). Continuous control with deep reinforcement learning. Proceedings of the 4th International Conference on Learning Representations, ICLR, San Juan, Puerto Rico."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1089\/dia.2007.0284","article-title":"Bolus calculator: A review of four \u201csmart\u201d insulin pumps","volume":"10","author":"Zisser","year":"2008","journal-title":"Diabetes Technol. Ther."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1035","DOI":"10.1177\/1932296814532906","article-title":"Bolus calculators","volume":"8","author":"Schmidt","year":"2014","journal-title":"J. Diabetes Sci. Technol."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Bertachi, A., Biagi, L., Beneyto, A., and Veh\u00ed, J. (2020). Dynamic rule-based algorithm to tune insulin-on-board constraints for a hybrid artificial pancreas system. J. Healthc. Eng., 2020.","DOI":"10.1155\/2020\/1414597"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Al-Taee, A.M., Al-Taee, M.A., Al-Nuaimy, W., Muhsin, Z.J., and AlZu\u2019bi, H. (2015, January 26\u201328). Smart bolus estimation taking into account the amount of insulin on board. Proceedings of the IEEE International Conference on Computer and Information Technology, Liverpool, UK.","DOI":"10.1109\/CIT\/IUCC\/DASC\/PICOM.2015.358"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"928","DOI":"10.1177\/193229681300700415","article-title":"Dynamic insulin on board: Incorporation of circadian insulin sensitivity variation","volume":"7","author":"Toffanin","year":"2013","journal-title":"J. Diabetes Sci. Technol."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1175","DOI":"10.2337\/dc15-2716","article-title":"Outcome measures for artificial pancreas clinical trials: A consensus report","volume":"39","author":"Maahs","year":"2016","journal-title":"Diabetes Care"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"S104","DOI":"10.1016\/j.jcjd.2017.10.010","article-title":"Hypoglycemia","volume":"42","author":"Yale","year":"2018","journal-title":"Can. J. Diabetes"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1177\/1932296819851135","article-title":"A modular safety system for an insulin dose recommender: A feasibility study","volume":"14","author":"Liu","year":"2020","journal-title":"J. Diabetes Sci. Technol."},{"key":"ref_43","unstructured":"Bengio, Y., and LeCun, Y. (2016, January 2\u20134). Prioritized experience replay. Proceedings of the 4th International Conference on Learning Representations, ICLR, San Juan, Puerto Rico."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"e11933","DOI":"10.2196\/11933","article-title":"Impact of use frequency of a mobile diabetes management app on blood glucose control: Evaluation study","volume":"7","author":"Vehi","year":"2019","journal-title":"JMIR mHealth uHealth"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Keith-Hynes, P., Guerlain, S., Mize, B., Hughes-Karvetski, C., Khan, M., McElwee-Malloy, M., and Kovatchev, B.P. (2013). DiAs user interface: A patient-centric interface for mobile artificial pancreas systems. J. Diabetes Sci. Technol.","DOI":"10.1177\/193229681300700602"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1089\/dia.2018.0278","article-title":"Design and clinical evaluation of the interoperable artificial pancreas system (iAPS) smartphone app: Interoperable components with modular design for progressive artificial pancreas research and development","volume":"21","author":"Deshpande","year":"2019","journal-title":"Diabetes Technol. Ther."},{"key":"ref_47","unstructured":"Spence, R., Li, K., Uduku, C., Zhu, T., Redmond, L., Herrero, P., Oliver, N., and Georgiou, P. (2020, January 19\u201322). A novel hand-held interface supporting the self-management of type 1 diabetes. Proceedings of the 13th International Conference on Advanced Technologies & Treatments for Diabetes (ATTD 2020), Madrid, Spain."},{"key":"ref_48","unstructured":"Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., and Isard, M. (2016, January 2\u20134). Tensorflow: A system for large-scale machine learning. Proceedings of the 12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16), Savannah, GA, USA."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1109\/JBHI.2019.2908488","article-title":"Convolutional recurrent neural networks for glucose prediction","volume":"24","author":"Li","year":"2019","journal-title":"IEEE J. Biomed. Health Inf."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.cmpb.2015.02.003","article-title":"Method for automatic adjustment of an insulin bolus calculator: In silico robustness evaluation under intra-day variability","volume":"119","author":"Herrero","year":"2015","journal-title":"Comput. Methods Progr. Biomed."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"630","DOI":"10.1177\/193229680800200414","article-title":"Evaluating the efficacy of closed-loop glucose regulation via control-variability grid analysis","volume":"2","author":"Magni","year":"2008","journal-title":"J. Diabetes Sci. Technol."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Zhu, T., Li, K., Herrero, P., and Georgiou, P. (2020). Basal Glucose Control in Type 1 Diabetes using Deep Reinforcement Learning: An In Silico Validation. IEEE J. Biomed. Health Inform.","DOI":"10.1109\/JBHI.2020.3014556"},{"key":"ref_53","unstructured":"Gu, S., Lillicrap, T., Sutskever, I., and Levine, S. (2016, January 20\u201322). Continuous deep q-learning with model-based acceleration. Proceedings of the International Conference on Machine Learning, New York, NY, USA."},{"key":"ref_54","unstructured":"Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. (2016). Safe and efficient off-policy reinforcement learning. Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_55","unstructured":"Dulac-Arnold, G., Mankowitz, D., and Hester, T. (2019, January 14). Challenges of real-world reinforcement learning. Proceedings of the Reinforcement Learning for Real Life (RL4RealLife) Workshop in the 36th International Conference on Machine Learning (ICML), Long Beach, CA, USA."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/18\/5058\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:07:23Z","timestamp":1760177243000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/18\/5058"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,6]]},"references-count":55,"journal-issue":{"issue":"18","published-online":{"date-parts":[[2020,9]]}},"alternative-id":["s20185058"],"URL":"https:\/\/doi.org\/10.3390\/s20185058","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,9,6]]}}}