{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T06:55:31Z","timestamp":1775199331570,"version":"3.50.1"},"reference-count":43,"publisher":"MDPI AG","issue":"24","license":[{"start":{"date-parts":[[2023,12,16]],"date-time":"2023-12-16T00:00:00Z","timestamp":1702684800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002241","name":"Japan Science and Technology Agency (JST) CREST","doi-asserted-by":"publisher","award":["JPMJCR20D5"],"award-info":[{"award-number":["JPMJCR20D5"]}],"id":[{"id":"10.13039\/501100002241","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In the context of Minimally Invasive Surgery, surgeons mainly rely on visual feedback during medical operations. In common procedures such as tissue resection, the automation of endoscopic control is crucial yet challenging, particularly due to the interactive dynamics of multi-agent operations and the necessity for real-time adaptation. This paper introduces a novel framework that unites a Hierarchical Quadratic Programming controller with an advanced interactive perception module. This integration addresses the need for adaptive visual field control and robust tool tracking in the operating scene, ensuring that surgeons and assistants have optimal viewpoint throughout the surgical task. The proposed framework handles multiple objectives within predefined thresholds, ensuring efficient tracking even amidst changes in operating backgrounds, varying lighting conditions, and partial occlusions. Empirical validations in scenarios involving single, double, and quadruple tool tracking during tissue resection tasks have underscored the system\u2019s robustness and adaptability. The positive feedback from user studies, coupled with the low cognitive and physical strain reported by surgeons and assistants, highlight the system\u2019s potential for real-world application.<\/jats:p>","DOI":"10.3390\/s23249865","type":"journal-article","created":{"date-parts":[[2023,12,18]],"date-time":"2023-12-18T11:28:07Z","timestamp":1702898887000},"page":"9865","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Endoscope Automation Framework with Hierarchical Control and Interactive Perception for Multi-Tool Tracking in Minimally Invasive Surgery"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7475-787X","authenticated-orcid":false,"given":"Khusniddin","family":"Fozilov","sequence":"first","affiliation":[{"name":"Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Furo-cho, Chikusa-ku, Nagoya 464-8603, Aichi, Japan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8833-2215","authenticated-orcid":false,"given":"Jacinto","family":"Colan","sequence":"additional","affiliation":[{"name":"Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Furo-cho, Chikusa-ku, Nagoya 464-8603, Aichi, Japan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2076-6842","authenticated-orcid":false,"given":"Ana","family":"Davila","sequence":"additional","affiliation":[{"name":"Institutes of Innovation for Future Society, Nagoya University, Furo-cho, Chikusa-ku, Nagoya 464-8603, Aichi, Japan"}]},{"given":"Kazunari","family":"Misawa","sequence":"additional","affiliation":[{"name":"Aichi Cancer Center Hospital, Chikusa Ward, Nagoya 464-8681, Aichi, Japan"}]},{"given":"Jie","family":"Qiu","sequence":"additional","affiliation":[{"name":"Graduate School of Informatics, Nagoya University, Chikusa Ward, Nagoya 464-8601, Aichi, Japan"}]},{"given":"Yuichiro","family":"Hayashi","sequence":"additional","affiliation":[{"name":"Graduate School of Informatics, Nagoya University, Chikusa Ward, Nagoya 464-8601, Aichi, Japan"}]},{"given":"Kensaku","family":"Mori","sequence":"additional","affiliation":[{"name":"Graduate School of Informatics, Nagoya University, Chikusa Ward, Nagoya 464-8601, Aichi, Japan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9917-098X","authenticated-orcid":false,"given":"Yasuhisa","family":"Hasegawa","sequence":"additional","affiliation":[{"name":"Institutes of Innovation for Future Society, Nagoya University, Furo-cho, Chikusa-ku, Nagoya 464-8603, Aichi, Japan"}]}],"member":"1968","published-online":{"date-parts":[[2023,12,16]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1007\/s00464-017-5796-2","article-title":"Robotic versus laparoscopic versus open colorectal surgery: Towards defining criteria to the right choice","volume":"32","author":"Zelhart","year":"2018","journal-title":"Surg. Endosc."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"e2405","DOI":"10.1002\/rcs.2405","article-title":"Preoperative positioning planning for a robot-assisted minimally invasive surgical system based on accuracy and safety","volume":"18","author":"Su","year":"2022","journal-title":"Int. J. Med. Robot. Comput. Assist. Surg."},{"key":"ref_3","first-page":"5022019","article-title":"Robotic-Assisted Laparoscopic Adjustment: A Meta-Analysis and Review","volume":"72","author":"Wang","year":"2023","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_4","first-page":"1729881420912871","article-title":"Corrigendum to Requirements elicitation for robotic and computer-assisted minimally invasive surgery","volume":"17","author":"Nakawala","year":"2020","journal-title":"Int. J. Adv. Robot. Syst."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1267","DOI":"10.1007\/s00464-001-9174-7","article-title":"Controlled trial of the introduction of a robotic camera assistant (Endo Assist) for laparoscopic cholecystectomy","volume":"16","author":"Aiono","year":"2002","journal-title":"Surg. Endosc. Other Interv. Tech."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"707704","DOI":"10.3389\/frobt.2021.707704","article-title":"Automating Endoscope Motion in Robotic Surgery: A Usability Study on da Vinci-Assisted Ex Vivo Neobladder Reconstruction","volume":"8","author":"Col","year":"2021","journal-title":"Front. Robot. AI"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"196","DOI":"10.1016\/j.media.2017.11.011","article-title":"Gaze gesture based human robot interaction for laparoscopic surgery","volume":"44","author":"Fujii","year":"2018","journal-title":"Med. Image Anal."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"72614","DOI":"10.1109\/ACCESS.2021.3079427","article-title":"Autonomous endoscope robot positioning using instrument segmentation with virtual reality visualization","volume":"9","author":"Zinchenko","year":"2021","journal-title":"IEEE Access"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Qian, L., Deguet, A., Wang, Z., Liu, Y.H., and Kazanzides, P. (2019, January 20\u201324). Augmented reality assisted instrument insertion and tool manipulation for the first assistant in robotic surgery. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.","DOI":"10.1109\/ICRA.2019.8794263"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"651","DOI":"10.1146\/annurev-control-062420-090543","article-title":"Autonomy in Surgical Robotics","volume":"4","author":"Attanasio","year":"2021","journal-title":"Annu. Rev. Control Robot. Auton. Syst."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"91206","DOI":"10.1109\/ACCESS.2023.3308619","article-title":"Toward Autonomous Robotic Minimally Invasive Surgery: A Hybrid Framework Combining Task-Motion Planning and Dynamic Behavior Trees","volume":"11","author":"Fozilov","year":"2023","journal-title":"IEEE Access"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1007\/s004649900920","article-title":"Perceptual-motor coordination in an endoscopic surgery simulation","volume":"13","author":"Holden","year":"1999","journal-title":"Surg. Endosc."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1080\/136457001753192277","article-title":"Eye-hand coordination in laparoscopy-an overview of experiments and supporting aids","volume":"10","author":"Wentink","year":"2001","journal-title":"Minim. Invasive Ther. Allied Technol."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1109\/TMRB.2019.2949881","article-title":"Adaptive fov control of laparoscopes with programmable composed constraints","volume":"1","author":"Yang","year":"2019","journal-title":"IEEE Trans. Med. Robot. Bionics"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"7934","DOI":"10.1109\/LRA.2022.3186507","article-title":"Autonomous Laparoscope Control for Minimally Invasive Surgery with Intuition and RCM Constraints","volume":"7","author":"Yang","year":"2022","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Qureshi, R., Ragab, M.G., Abdulkader, S.J., Muneer, A., Alqushaib, A., Sumiea, E.H., and Alhussian, H. (2023). A Comprehensive Systematic Review of YOLO for Medical Object Detection (2018 to 2023). TechRxiv.","DOI":"10.36227\/techrxiv.23681679"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"5007913","DOI":"10.1109\/TIM.2022.3162596","article-title":"Autonomous Recognition of Multiple Surgical Instruments Tips Based on Arrow OBB-YOLO Network","volume":"71","author":"Peng","year":"2022","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"832208","DOI":"10.3389\/frobt.2022.832208","article-title":"Robotic endoscope control via autonomous instrument tracking","volume":"9","author":"Gruijthuijsen","year":"2022","journal-title":"Front. Robot. AI"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"733","DOI":"10.1109\/TMECH.2021.3070553","article-title":"Autonomous multiple instruments tracking for robot-assisted laparoscopic surgery with visual tracking space vector method","volume":"27","author":"Li","year":"2021","journal-title":"IEEE\/ASME Trans. Mechatron."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1002\/rcs.453","article-title":"Kinematic design considerations for minimally invasive surgical robots: An overview","volume":"8","author":"Kuo","year":"2012","journal-title":"Int. J. Med. Robot. Comput. Assist. Surg."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"901","DOI":"10.1007\/s10846-018-0927-0","article-title":"Constrained kinematic control in minimally invasive robotic surgery subject to remote center of motion constraint","volume":"95","author":"Sadeghian","year":"2019","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Colan, J., Nakanishi, J., Aoyama, T., and Hasegawa, Y. (2021). Optimization-based constrained trajectory generation for robot-assisted stitching in endonasal surgery. Robotics, 10.","DOI":"10.3390\/robotics10010027"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Colan, J., Davila, A., Fozilov, K., and Hasegawa, Y. (2023). A Concurrent Framework for Constrained Inverse Kinematics of Minimally Invasive Surgical Robots. Sensors, 23.","DOI":"10.3390\/s23063328"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Li, B., Lu, B., Lu, Y., Dou, Q., and Liu, Y.H. (June, January 30). Data-driven holistic framework for automated laparoscope optimal view control with learning-based depth perception. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi\u2019an, China.","DOI":"10.1109\/ICRA48506.2021.9562083"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"3522916","DOI":"10.1109\/TIM.2022.3204086","article-title":"Endoscope FOV Autonomous Tracking Method for Robot-Assisted Surgery Considering Pose Control, Hand\u2013Eye Coordination, and Image Definition","volume":"71","author":"Peng","year":"2022","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"112658","DOI":"10.1016\/j.measurement.2023.112658","article-title":"Visual servo control of endoscope-holding robot based on multi-objective optimization: System modeling and instrument tracking","volume":"211","author":"Zhang","year":"2023","journal-title":"Measurement"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"9246","DOI":"10.1109\/TIE.2021.3114674","article-title":"Accelerated Dual Neural Network Controller for Visual Servoing of Flexible Endoscopic Robot with Tracking Error, Joint Motion, and RCM Constraints","volume":"69","author":"Cui","year":"2022","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"2250","DOI":"10.1109\/LRA.2022.3143305","article-title":"A Surgeon Preference-Guided Autonomous Instrument Tracking Method with a Robotic Flexible Endoscope Based on dVRK Platform","volume":"7","author":"Huang","year":"2022","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"356","DOI":"10.1109\/TMRB.2022.3155254","article-title":"Augmented Reality-Assisted Autonomous View Adjustment of a 6-DOF Robotic Stereo Flexible Endoscope","volume":"4","author":"Ma","year":"2022","journal-title":"IEEE Trans. Med. Robot. Bionics"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Huang, Y., Li, W., Zhang, X., Li, J., Li, Y., Sun, Y., Chiu, P.W.Y., and Li, Z. (2023). 4-DOF Visual Servoing of a Robotic Flexible Endoscope with a Predefined-Time Convergent and Noise-Immune Adaptive Neural Network. IEEE\/ASME Trans. Mechatron., 1\u201312.","DOI":"10.1109\/TMECH.2023.3286850"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., and Lo, W.Y. (2023). Segment anything. arXiv.","DOI":"10.1109\/ICCV51070.2023.00371"},{"key":"ref_32","unstructured":"Cheng, H.K., and Schwing, A.G. (2022). European Conference on Computer Vision, Springer."},{"key":"ref_33","unstructured":"Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.Y., and Berger, E. (2009, January 12). ROS: An open-source Robot Operating System. Proceedings of the ICRA Workshop on Open Source Software, Kobe, Japan."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"102918","DOI":"10.1016\/j.media.2023.102918","article-title":"Segment anything model for medical image analysis: An experimental study","volume":"89","author":"Mazurowski","year":"2023","journal-title":"Med. Image Anal."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Colan, J., Davila, A., and Hasegawa, Y. (2023, January 21\u201323). Constrained Motion Planning for a Robotic Endoscope Holder Based on Hierarchical Quadratic Programming. Proceedings of the 2023 8th International Conference on Control and Robotics Engineering (ICCRE), Niigata, Japan.","DOI":"10.1109\/ICCRE57112.2023.10155579"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"398","DOI":"10.1109\/70.585902","article-title":"Singularity-robust task-priority redundancy resolution for real-time kinematic control of robot manipulators","volume":"13","author":"Chiaverini","year":"1997","journal-title":"IEEE Trans. Robot. Autom."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"785","DOI":"10.1109\/TRO.2011.2142450","article-title":"Kinematic Control of Redundant Manipulators: Generalizing the Task-Priority Framework to Inequality Task","volume":"27","author":"Kanoun","year":"2011","journal-title":"IEEE Trans. Robot."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"651","DOI":"10.1109\/70.538972","article-title":"A tutorial on visual servo control","volume":"12","author":"Hutchinson","year":"1996","journal-title":"IEEE Trans. Robot. Autom."},{"key":"ref_39","unstructured":"Sol\u00e0, J., Deray, J., and Atchuthan, D. (2018). A micro Lie theory for state estimation in robotics. arXiv."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Carpentier, J., Saurel, G., Buondonno, G., Mirabel, J., Lamiraux, F., Stasse, O., and Mansard, N. (2019, January 14\u201316). The Pinocchio C++ library: A fast and flexible implementation of rigid body dynamics algorithms and their analytical derivatives. Proceedings of the 2019 IEEE\/SICE International Symposium on System Integration (SII), Paris, France.","DOI":"10.1109\/SII.2019.8700380"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s12532-018-0139-4","article-title":"CasADi: A software framework for nonlinear optimization and optimal control","volume":"11","author":"Andersson","year":"2019","journal-title":"Math. Program. Comput."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"637","DOI":"10.1007\/s12532-020-00179-2","article-title":"OSQP: An operator splitting solver for quadratic programs","volume":"12","author":"Stellato","year":"2020","journal-title":"Math. Program. Comput."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"904","DOI":"10.1177\/154193120605000909","article-title":"NASA-task load index (NASA-TLX); 20 years later","volume":"Volume 50","author":"Hart","year":"2006","journal-title":"Proceedings of the Human Factors and Ergonomics Society Annual Meeting"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/24\/9865\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:39:57Z","timestamp":1760132397000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/24\/9865"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,16]]},"references-count":43,"journal-issue":{"issue":"24","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["s23249865"],"URL":"https:\/\/doi.org\/10.3390\/s23249865","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,16]]}}}