{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T16:25:59Z","timestamp":1790353559869,"version":"4.1.0"},"reference-count":42,"publisher":"Cambridge University Press (CUP)","issue":"4","license":[{"start":{"date-parts":[[2022,11,17]],"date-time":"2022-11-17T00:00:00Z","timestamp":1668643200000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotica"],"published-print":{"date-parts":[[2023,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Instrument delivery is critical part in vascular intervention surgery. Due to the soft-body structure of instruments, the relationship between manipulation commands and instrument motion is non-linear, making instrument delivery challenging and time-consuming. Reinforcement learning has the potential to learn manipulation skills and automate instrument delivery with enhanced success rates and reduced workload of physicians. However, due to the sample inefficiency when using high-dimensional images, existing reinforcement learning algorithms are limited on realistic vascular robotic systems. To alleviate this problem, this paper proposes discrete soft actor-critic with auto-encoder (DSAC-AE) that augments SAC-discrete with an auxiliary reconstruction task. The algorithm is applied with distributed sample collection and parameter update in a robot-assisted preclinical environment. Experimental results indicate that guidewire delivery can be automatically implemented after 50k sampling steps in less than 15 h, demonstrating the proposed algorithm has the great potential to learn manipulation skill for vascular robotic systems.<\/jats:p>","DOI":"10.1017\/s0263574722001527","type":"journal-article","created":{"date-parts":[[2022,11,17]],"date-time":"2022-11-17T08:05:57Z","timestamp":1668672357000},"page":"1115-1126","source":"Crossref","is-referenced-by-count":13,"title":["Discrete soft actor-critic with auto-encoder on vascular robotic system"],"prefix":"10.1017","volume":"41","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1948-8283","authenticated-orcid":false,"given":"Hao","family":"Li","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7602-4848","authenticated-orcid":false,"given":"Xiao-Hu","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiao-Liang","family":"Xie","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shi-Qi","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mei-Jiang","family":"Gui","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tian-Yu","family":"Xiang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5387-7648","authenticated-orcid":false,"given":"Jin-Li","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zeng-Guang","family":"Hou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"56","published-online":{"date-parts":[[2022,11,17]]},"reference":[{"key":"S0263574722001527_ref5","doi-asserted-by":"crossref","first-page":"1081","DOI":"10.4244\/EIJV7I9A172","article-title":"Brain tumours among interventional cardiologists: A cause for alarm? Report of four new cases from two cities and a review of the literature","volume":"7","author":"Roguin","year":"2012","journal-title":"EuroIntervention"},{"key":"S0263574722001527_ref14","doi-asserted-by":"crossref","first-page":"eaam8638","DOI":"10.1126\/scirobotics.aam8638","article-title":"Medical robotics-regulatory, ethical, and legal considerations for increasing levels of autonomy","volume":"2","author":"Yang","year":"2017","journal-title":"Sci. Robot."},{"key":"S0263574722001527_ref27","doi-asserted-by":"crossref","unstructured":"[27] Chi, W. , Liu, J. , Abdelaziz, M. E. M. K. , Dagnino, G. , Riga, C. V. , Bicknell, C. D. and Yang, G.-Z. , \u201cTrajectory Optimization of Robot-Assisted Endovascular Catheterization with Reinforcement Learning,\u201d In: Proceedings of 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (Institute\u00a0of\u00a0Electrical and Electronics Engineers Inc., Piscataway, 2018) pp. 3875\u20133881.","DOI":"10.1109\/IROS.2018.8593421"},{"key":"S0263574722001527_ref20","doi-asserted-by":"crossref","unstructured":"[20] Guo, J. , Feng, S. and Guo, S. , \u201cStudy on the Automatic Surgical Method of the Vascular Interventional Surgical Robot Based on Deep Learning,\u201d In: Proceedings of 2021 IEEE International Conference on Mechatronics and Automation (Institute of Electrical and Electronics Engineers Inc., Piscataway, 2021) pp. 1076\u20131081.","DOI":"10.1109\/ICMA52036.2021.9512672"},{"key":"S0263574722001527_ref25","doi-asserted-by":"crossref","first-page":"20200007","DOI":"10.1515\/cdbme-2020-0007","article-title":"Autonomous guidewire navigation in a two dimensional vascular phantom","volume":"6","author":"Karstensen","year":"2020","journal-title":"Curr. Dir. Biomed. Eng."},{"key":"S0263574722001527_ref4","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1109\/TMRB.2022.3141313","article-title":"Surgical skill assessment based on dynamic warping manipulations","volume":"4","author":"Zhou","year":"2022","journal-title":"IEEE Trans. Med. Robot. Bionics."},{"key":"S0263574722001527_ref21","doi-asserted-by":"crossref","unstructured":"[21] Chi, W. , Dagnino, G. , Kwok, T. M. Y. , Nguyen, A. , Kundrat, D. , Abdelaziz, M. E. M. K. , Riga, C. , Bicknell, C. and Yang, G.-Z. , \u201cCollaborative Robot-Assisted Endovascular Catheterization with Generative Adversarial Imitation Learning,\u201d In: Proceedings of 2020 IEEE International Conference on Robotics and Automation (Institute of Electrical and Electronics Engineers Inc., Piscataway, 2020) pp. 2414\u20132420.","DOI":"10.1109\/ICRA40945.2020.9196912"},{"key":"S0263574722001527_ref11","doi-asserted-by":"crossref","first-page":"685","DOI":"10.1109\/TRO.2019.2896763","article-title":"A novel robot-assisted endovascular catheterization system with haptic force feedback","volume":"35","author":"Guo","year":"2019","journal-title":"IEEE Trans. Robot."},{"key":"S0263574722001527_ref40","article-title":"Ray: A distributed framework for emerging ai applications","author":"Moritz","year":"2017","journal-title":"CoRR"},{"key":"S0263574722001527_ref13","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1016\/j.eclinm.2019.07.017","article-title":"Long distance tele-robotic-assisted percutaneous coronary intervention: A report of first-in-human experience","volume":"14","author":"Patel","year":"2019","journal-title":"EClinicalMedicine"},{"key":"S0263574722001527_ref19","doi-asserted-by":"crossref","first-page":"1875","DOI":"10.1007\/s11517-019-02002-0","article-title":"A CNN-based prototype method of unstructured surgical state perception and navigation for an endovascular surgery robot","volume":"57","author":"Zhao","year":"2019","journal-title":"Med. Biol. Eng. Comput."},{"key":"S0263574722001527_ref26","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1515\/cdbme-2019-0002","article-title":"Deep reinforcement learning for the navigation of neurovascular catheters","volume":"5","author":"Behr","year":"2019","journal-title":"Curr. Dir. Biomed. Eng."},{"key":"S0263574722001527_ref22","doi-asserted-by":"crossref","first-page":"604","DOI":"10.1038\/s41586-020-03051-4","article-title":"Mastering Atari, Go, chess and shogi by planning with a learned model","volume":"588","author":"Schrittwieser","year":"2020","journal-title":"Nature"},{"key":"S0263574722001527_ref30","unstructured":"[30] Mnih, V. , Badia, A. P. , Mirza, M. , Graves, A. , Lillicrap, T. P. , Harley, T. , Silver, D. and Kavukcuoglu, K. , \u201cAsynchronous Methods for Deep Reinforcement Learning,\u201d In: Proceedings of the 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research, New York, 2016) pp. 1928\u20131937."},{"key":"S0263574722001527_ref2","doi-asserted-by":"crossref","unstructured":"[2] Zhou, X.-H. , Xie, X.-L. , Liu, S.-Q. , Ni, Z.-L. , Zhou, Y.-J. , Li, R.-Q. , Gui, M.-J. , Fan, C.-C. , Feng, Z.-Q. , Bian, G.-B. and Hou, Z.-G. , \u201cLearning skill characteristics from manipulations,\u201d IEEE Trans. Neural Netw. Learn. Syst., 1\u201315 (2022). doi: 10.1109\/TNNLS.2022.3160159.","DOI":"10.1109\/TNNLS.2022.3160159"},{"key":"S0263574722001527_ref38","article-title":"Soft actor-critic for discrete action settings","author":"Christodoulou","year":"2019","journal-title":"CoRR"},{"key":"S0263574722001527_ref12","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TIM.2022.3147904","article-title":"Design and experiments of a novel Halbach-cylinder-based magnetic skin: A preliminary study","volume":"71","author":"Gui","year":"2022","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"S0263574722001527_ref42","doi-asserted-by":"crossref","first-page":"946","DOI":"10.1093\/europace\/eut409","article-title":"Practical ways to reduce radiation dose for patients and staff during device implantations and electrophysiological procedures","volume":"16","author":"Heidbuchel","year":"2014","journal-title":"Europace"},{"key":"S0263574722001527_ref17","doi-asserted-by":"crossref","unstructured":"[17] Rafii-Tari, H. , Liu, J. , Payne, C. J. , Bicknell, C. D. and Yang, G.-Z. , \u201cHierarchical HMM Based Learning of Navigation Primitives for Cooperative Robotic Endovascular Catheterization,\u201d In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (Springer, Berlin, 2014) pp. 496\u2013503.","DOI":"10.1007\/978-3-319-10404-1_62"},{"key":"S0263574722001527_ref8","doi-asserted-by":"crossref","first-page":"913","DOI":"10.1002\/ccd.25927","article-title":"Occupational health hazards of interventional cardiologists in the current decade: Results of the 2014 SCAI membership survey","volume":"86","author":"Klein","year":"2015","journal-title":"Catheter. Cardiovasc. Interv."},{"key":"S0263574722001527_ref6","doi-asserted-by":"crossref","first-page":"647","DOI":"10.1002\/ccd.27173","article-title":"Radiation-associated lens changes in the cardiac catheterization laboratory: Results from the IC-CATARACT(CATaracts Attributed to RAdiation in the CaTh lab) study","volume":"91","author":"Karatasakis","year":"2018","journal-title":"Catheter. Cardiovasc. Interv."},{"key":"S0263574722001527_ref28","doi-asserted-by":"crossref","first-page":"806","DOI":"10.1109\/JBHI.2017.2703903","article-title":"Three-dimensional intravascular reconstruction techniques based on intravascular ultrasound: A technical review","volume":"22","author":"Shi","year":"2018","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"S0263574722001527_ref31","doi-asserted-by":"crossref","first-page":"5415","DOI":"10.1007\/s12206-019-1036-0","article-title":"Automatic control of cardiac ablation catheter with deep reinforcement learning method","volume":"33","author":"You","year":"2019","journal-title":"J. Mech. Sci. Technol."},{"key":"S0263574722001527_ref41","doi-asserted-by":"crossref","first-page":"13","DOI":"10.3389\/fneng.2012.00013","article-title":"Detection of self-paced reaching movement intention from EEG signals","volume":"5","author":"Lew","year":"2012","journal-title":"Front. Neuroeng."},{"key":"S0263574722001527_ref7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1002\/ccd.27114","article-title":"Risk of cataract among interventional cardiologists and catheterization lab staff: A systematic review and meta-analysis","volume":"90","author":"Elmaraezy","year":"2017","journal-title":"Catheter. Cardiovasc. Interv."},{"key":"S0263574722001527_ref37","article-title":"Soft actor-critic algorithms and applications","author":"Haarnoja","year":"2018","journal-title":"CoRR"},{"key":"S0263574722001527_ref15","first-page":"1","article-title":"Rotate-on-retract procedural automation for robotic-assisted percutaneous coronary intervention: First clinical experience","volume":"2018","author":"Nooryani","year":"2018","journal-title":"Case Rep. Cardiol."},{"key":"S0263574722001527_ref16","doi-asserted-by":"crossref","unstructured":"[16] Rafii-Tari, H. , Liu, J. , Lee, S.-L. , Bicknell, C. D. , and Yang, G.-Z. , \u201cLearning-Based Modeling of Endovascular Navigation for Collaborative Robotic Catheterization,\u201d In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (Springer, Berlin, 2013) pp. 369\u2013377.","DOI":"10.1007\/978-3-642-40763-5_46"},{"key":"S0263574722001527_ref3","doi-asserted-by":"crossref","first-page":"697","DOI":"10.1007\/s10439-013-0946-8","article-title":"Current and emerging robot-assisted endovascular catheterization technologies: A review","volume":"42","author":"Rafii-Tari","year":"2014","journal-title":"Ann. Biomed. Eng."},{"key":"S0263574722001527_ref23","doi-asserted-by":"crossref","first-page":"1558","DOI":"10.1017\/S0263574719001632","article-title":"Experimental study of reinforcement learning in mobile robots through spiking architecture of Thalamo-Cortico-Thalamic circuitry of mammalian brain","volume":"38","author":"Azimirad","year":"2020","journal-title":"Robotica"},{"key":"S0263574722001527_ref34","unstructured":"[34] Sr, A. inivas, , Laskin, M. and Abbeel, P. , \u201cCURL: Contrastive Unsupervised Representations for Reinforcement Learning,\u201d In: Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, NewYork, 2020) pp. 5639\u20135650."},{"key":"S0263574722001527_ref10","doi-asserted-by":"crossref","first-page":"4305","DOI":"10.3390\/app9204305","article-title":"Advantage of steerable catheter and haptic feedback for a 5-DOF vascular intervention robot system","volume":"9","author":"Woo","year":"2019","journal-title":"Appl. Sci."},{"key":"S0263574722001527_ref32","doi-asserted-by":"crossref","unstructured":"[32] M, F. eng, , Guo, S. , Zhou, W. and Chen, Z. , \u201cEvaluation of a Reinforcement Learning Algorithm for Vascular Intervention Surgery,\u201d In: Proceedings of 2021 IEEE International Conference on Mechatronics and Automation (Institute\u00a0of\u00a0Electrical and Electronics Engineers Inc., Piscataway, 2021) pp. 1033\u20131037.","DOI":"10.1109\/ICMA52036.2021.9512675"},{"key":"S0263574722001527_ref33","doi-asserted-by":"crossref","unstructured":"[33] Yarats, D. , Zhang, A. , Kostrikov, I. , Amos, B. , Pineau, J. and Fergus, R. , \u201cImproving Sample Efficiency in Model-Free Reinforcement Learning from Images,\u201d In: Proceedings of the 35th AAAI Conference on Artificial Intelligence (Association for the Advancement of Artificial Intelligence, Menlo Park, 2021) pp. 10674\u201310681.","DOI":"10.1609\/aaai.v35i12.17276"},{"key":"S0263574722001527_ref29","article-title":"Dueling network architectures for deep reinforcement learning","author":"Wang","year":"2015","journal-title":"CoRR"},{"key":"S0263574722001527_ref36","unstructured":"[36] Zie, B. D. bart, , Maas, A. L. , Bagnell, J. A. and Dey, A. K. , \u201cMaximum Entropy Inverse Reinforcement Learning,\u201d In: Proceedings of the 23rd AAAI Conference on Artificial Intelligence (Association for the Advancement of Artificial Intelligence, Menlo Park, 2008) pp. 1433\u20131438."},{"key":"S0263574722001527_ref35","doi-asserted-by":"crossref","unstructured":"[35] , H.- Zhao, L. , Liu, S.-Q. , Zhou, X.-H. , Xie, X.-L. , Hou, Z.-G. , Zhou, Y.-J. , Zhang, L.-S. , Gui, M.-J. and Wang, J.-L. , \u201cDesign and Performance Evaluation of a Novel Vascular Robotic System for Complex Percutaneous Coronary Interventions,\u201d In: Proceedings of 2021 43rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (Institute\u00a0of\u00a0Electrical and Electronics Engineers Inc., Piscataway, 2021) pp. 4679\u20134682.","DOI":"10.1109\/EMBC46164.2021.9629943"},{"key":"S0263574722001527_ref24","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1017\/S0263574711000452","article-title":"Optimal motion planning by reinforcement learning in autonomous mobile vehicles","volume":"30","author":"G\u00f3mez","year":"2012","journal-title":"Robotica"},{"key":"S0263574722001527_ref1","doi-asserted-by":"crossref","first-page":"2529","DOI":"10.1016\/j.jacc.2019.10.009","article-title":"The global burden of cardiovascular diseases and risk factors: 2020 and beyond","volume":"74","author":"Mensah","year":"2019","journal-title":"J. Am. Coll. Cardiol."},{"key":"S0263574722001527_ref18","doi-asserted-by":"crossref","first-page":"855","DOI":"10.1007\/s11548-018-1743-5","article-title":"Learning-based endovascular navigation through the use of non-rigid registration for collaborative robotic catheterization","volume":"13","author":"Chi","year":"2018","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"S0263574722001527_ref9","doi-asserted-by":"crossref","first-page":"460","DOI":"10.1016\/j.jcin.2010.12.007","article-title":"First-in-human evaluation of a novel robotic-assisted coronary angioplasty system","volume":"4","author":"Granada","year":"2011","journal-title":"J. Am. Coll. Cardiol. Cardiovas. Interv."},{"key":"S0263574722001527_ref39","doi-asserted-by":"crossref","first-page":"318","DOI":"10.1109\/TPAMI.2018.2858826","article-title":"Focal loss for dense object detection","volume":"42","author":"Lin","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."}],"container-title":["Robotica"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S0263574722001527","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,4]],"date-time":"2023-12-04T10:13:58Z","timestamp":1701684838000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S0263574722001527\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,17]]},"references-count":42,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,4]]}},"alternative-id":["S0263574722001527"],"URL":"https:\/\/doi.org\/10.1017\/s0263574722001527","relation":{},"ISSN":["0263-5747","1469-8668"],"issn-type":[{"value":"0263-5747","type":"print"},{"value":"1469-8668","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,17]]}}}