{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,18]],"date-time":"2025-12-18T14:18:33Z","timestamp":1766067513118,"version":"3.37.3"},"reference-count":25,"publisher":"Springer Science and Business Media LLC","issue":"8","license":[{"start":{"date-parts":[[2022,5,3]],"date-time":"2022-05-03T00:00:00Z","timestamp":1651536000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,5,3]],"date-time":"2022-05-03T00:00:00Z","timestamp":1651536000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100004440","name":"Wellcome Trust","doi-asserted-by":"publisher","award":["203145Z\/16\/Z"],"award-info":[{"award-number":["203145Z\/16\/Z"]}],"id":[{"id":"10.13039\/100004440","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/P027938\/1","EP\/R004080\/1"],"award-info":[{"award-number":["EP\/P027938\/1","EP\/R004080\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/P012841\/1"],"award-info":[{"award-number":["EP\/P012841\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J CARS"],"published-print":{"date-parts":[[2022,8]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Purpose<\/jats:title>\n                <jats:p>Automation of sub-tasks during robotic surgery is challenging due to the high variability of the surgical scenes intra- and inter-patients. For example, the pick and place task can be executed different times during the same operation and for distinct purposes. Hence, designing automation solutions that can generalise a skill over different contexts becomes hard. All the experiments are conducted using the Pneumatic Attachable Flexible (PAF) rail, a novel surgical tool designed for robotic-assisted intraoperative organ manipulation.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>We build upon previous open-source surgical Reinforcement Learning (RL) training environment to develop a new RL framework for manipulation skills, <jats:italic>rlman<\/jats:italic>. In <jats:italic>rlman<\/jats:italic>, contextual RL agents are trained to solve different aspects of the pick and place task using the PAF rail system. <jats:italic>rlman<\/jats:italic> is implemented to support both low- and high-dimensional state information to solve surgical sub-tasks in a simulation environment.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>We use <jats:italic>rlman<\/jats:italic> to train state of the art RL agents to solve four different surgical sub-tasks involving manipulation skills using the PAF rail. We compare the results with state-of-the-art benchmarks found in the literature. We evaluate the ability of the agent to be able to generalise over different aspects of the targeted surgical environment.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusion<\/jats:title>\n                <jats:p>We have shown that the <jats:italic>rlman<\/jats:italic> framework can support the training of different RL algorithms for solving surgical sub-task, analysing the importance of context information for generalisation capabilities. We are aiming to deploy the trained policy on the real da Vinci using the dVRK and show that the generalisation of the trained policy can be transferred to the real world.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1007\/s11548-022-02630-2","type":"journal-article","created":{"date-parts":[[2022,5,3]],"date-time":"2022-05-03T05:02:40Z","timestamp":1651554160000},"page":"1419-1427","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Learning intraoperative organ manipulation with context-based reinforcement learning"],"prefix":"10.1007","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2874-1583","authenticated-orcid":false,"given":"Claudia","family":"D\u2019Ettorre","sequence":"first","affiliation":[]},{"given":"Silvia","family":"Zirino","sequence":"additional","affiliation":[]},{"given":"Neri Niccol\u00f2","family":"Dei","sequence":"additional","affiliation":[]},{"given":"Agostino","family":"Stilli","sequence":"additional","affiliation":[]},{"given":"Elena","family":"De Momi","sequence":"additional","affiliation":[]},{"given":"Danail","family":"Stoyanov","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,5,3]]},"reference":[{"key":"2630_CR1","doi-asserted-by":"crossref","unstructured":"Kazanzides P, Chen Z, Deguet A, Fischer GS, Taylor RH, DiMaio SP (2014) An open-source research kit for the da Vinci\u00ae Surgical System. In: 2014 IEEE Int Conf Robot Autom","DOI":"10.1109\/ICRA.2014.6907809"},{"key":"2630_CR2","doi-asserted-by":"crossref","unstructured":"D\u2019Ettorre C, Mariani A, Stilli A, Rodriguez y Baena F, Valdastri P, Deguet A, Kazanzides P, Taylor RH, Fischer GS, DiMaio SP, Menciassi A, Stoyanov D (2021) Accelerating surgical robotics research: a review of 10 years with the da vinci research kit. IEEE Robot Autom Mag","DOI":"10.1109\/MRA.2021.3101646"},{"key":"2630_CR3","unstructured":"Richter F, Orosco RK, Yip MC (2019) Open-sourced reinforcement learning environments for surgical robotics. In: Proceedings IEEE\/RSJ international conference on intelligent robots and systems (IROS). Macau, China"},{"key":"2630_CR4","doi-asserted-by":"crossref","unstructured":"Stilli A, Dimitrakakis E, D\u2019Ettorre C, Tran M, Stoyanov D (2019) Pneumatically attachable flexible rails for track-guided ultrasound scanning in robotic-assisted partial nephrectomy\u2013a preliminary design study. IEEE Robot Autom Lett 4(2)","DOI":"10.1109\/LRA.2019.2894499"},{"key":"2630_CR5","doi-asserted-by":"crossref","unstructured":"Xu J, Li B, Lu B, Liu Y-H, Dou Q, Heng P-A (2021) SurRoL: an open-source reinforcement learning centered and dVRK compatible platform for surgical robot learning. In: 2021 IEEE\/RSJ Int Conf Intell Robot Syst","DOI":"10.1109\/IROS51168.2021.9635867"},{"key":"2630_CR6","doi-asserted-by":"crossref","unstructured":"D\u2019Ettorre C, Stilli A, Dwyer G, Neves JB, Tran M, Stoyanov, D (2019) Semi-autonomous interventional manipulation using pneumatically attachable flexible rails*. In: 2019 IEEE\/RSJ Int Conf Intell Robot Syst","DOI":"10.1109\/IROS40897.2019.8967789"},{"key":"2630_CR7","doi-asserted-by":"crossref","unstructured":"D\u2019Ettorre C, Stilli A, Dwyer G, Tran M, Stoyanov D (2021) Autonomous pick-and-place using the dVRK. Int J Comput Assist Radiol Surg 16(7):1141\u20131149","DOI":"10.1007\/s11548-021-02397-y"},{"key":"2630_CR8","unstructured":"Sutton RS, Barto AG (2018) Reinforcement learning: an introduction. MIT Press"},{"key":"2630_CR9","doi-asserted-by":"crossref","unstructured":"Tobin J, Fong R, Ray A, Schneider J, Zaremba W, Abbeel P (2017) Domain randomization for transferring deep neural networks from simulation to the real world. In: 2017 IEEE\/RSJ Int Conf Intell Robot Syst","DOI":"10.1109\/IROS.2017.8202133"},{"key":"2630_CR10","doi-asserted-by":"crossref","unstructured":"Fontanelli GA, Selvaggio M, Ferro M, Ficuciello F, Vendittelli M, Siciliano B (2018) A V-REP Simulator for the da vinci research kit robotic platform. In: 2018 7th IEEE Int Conf Biomed Robot Biomechatronics","DOI":"10.1109\/BIOROB.2018.8487187"},{"key":"2630_CR11","doi-asserted-by":"crossref","unstructured":"Munawar A, Wang Y, Gondokaryono R, Fischer, GS (2019) A real-time dynamic simulator and an associated front-end representation format for simulating complex robots and environments. In: 2019 IEEE\/RSJ Int Conf Intell Robot Syst","DOI":"10.1109\/IROS40897.2019.8968568"},{"key":"2630_CR12","unstructured":"Tagliabue E, Pore A, Dall\u2019Alba D, Piccinelli M, Fiorini P (2020) UnityFlexML: training reinforcement learning agents in a simulated surgical environment*. ArXiv"},{"key":"2630_CR13","doi-asserted-by":"crossref","unstructured":"Tagliabue E, Pore A, Dall\u2019Alba D, Magnabosco E, Piccinelli M, Fiorini, P (2020) Soft tissue simulation environment to learn manipulation tasks in autonomous robotic surgery *. In: 2020 IEEE\/RSJ Int Conf Intell Robot Syst","DOI":"10.1109\/IROS45743.2020.9341710"},{"key":"2630_CR14","doi-asserted-by":"crossref","unstructured":"Thach B, Cho BY, Kuntz A, Hermans T (2021) learning visual shape control of novel 3D deformable objects from partial-view point clouds. ArXiv","DOI":"10.1109\/ICRA46639.2022.9812215"},{"key":"2630_CR15","unstructured":"Sodhani S, Zhang A, Pineau J (2021) Multi-task reinforcement learning with context-based representations. In: 38th Int Conf Mach Learn (ICML 2021)"},{"key":"2630_CR16","unstructured":"Hallak A, Di Castro D, Mannor S (2015) Contextual markov decision processes. arXiv preprint arXiv:1502.02259"},{"key":"2630_CR17","doi-asserted-by":"crossref","unstructured":"Hallak A, Tamar A, Munos R, Mannor S (2015) Generalized emphatic temporal difference learning: bias-variance analysis. ArXiv","DOI":"10.1609\/aaai.v30i1.10227"},{"key":"2630_CR18","unstructured":"Dhariwal P, Hesse C, Klimov O, Nichol A, Matthias P, Radford A, Schulman J, Sidor S, Wu Y, Zhokhov P (2017) OpenAI Baselines, Github repository. https:\/\/github.com\/openai\/baselines"},{"key":"2630_CR19","doi-asserted-by":"crossref","unstructured":"Ibarz J, Tan J, Finn C, Kalakrishnan M, Pastor P, Levine S (2021) How to train your robot with deep reinforcement learning: lessons we have learned. Int J Rob Res 14(4\u20135):698\u2013721","DOI":"10.1177\/0278364920987859"},{"key":"2630_CR20","unstructured":"Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal Policy Optimization Algorithms. ArXiv"},{"key":"2630_CR21","unstructured":"Lillicrap TP, Hunt JJ, Pritzel A, Heess N, Erez T, Tassa Y, Silver D, Wierstra D (2016) Continuous control with deep reinforcement learning. In: Int Conf Learn Represent (ICLR)"},{"key":"2630_CR22","unstructured":"Andrychowicz M, Wolski F, Ray A, Schneider J, Fong R, Welinder P, McGrew B, Tobin J, Abbeel P, Zaremba W (2018) Hindsight Experience Replay. ArXiv"},{"key":"2630_CR23","unstructured":"Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wiestra D, Riedmiller M (2013) Playing atari with deep reinforcement learning. ArXiv"},{"key":"2630_CR24","doi-asserted-by":"crossref","unstructured":"Mishra RK (2013) Textbook of practical laparoscopic surgery. JP Medical Ltd","DOI":"10.5005\/jp\/books\/11753"},{"key":"2630_CR25","doi-asserted-by":"crossref","unstructured":"Nair A, McGrew B, Andrychowicz M, Zaremba W, Abbeel, P (2018) Overcoming exploration in reinforcement learning with demonstrations. In: 2018 IEEE Int Conf Robot Autom","DOI":"10.1109\/ICRA.2018.8463162"}],"container-title":["International Journal of Computer Assisted Radiology and Surgery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11548-022-02630-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11548-022-02630-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11548-022-02630-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,7,23]],"date-time":"2022-07-23T03:40:55Z","timestamp":1658547655000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11548-022-02630-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,3]]},"references-count":25,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2022,8]]}},"alternative-id":["2630"],"URL":"https:\/\/doi.org\/10.1007\/s11548-022-02630-2","relation":{},"ISSN":["1861-6429"],"issn-type":[{"type":"electronic","value":"1861-6429"}],"subject":[],"published":{"date-parts":[[2022,5,3]]},"assertion":[{"value":"1 March 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 March 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 May 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"This article does not contain any studies with human participants or animals performed by any of the authors.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"This articles does not contain patient data","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}}]}}