{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,5]],"date-time":"2026-02-05T11:23:42Z","timestamp":1770290622603,"version":"3.49.0"},"reference-count":34,"publisher":"MDPI AG","issue":"13","license":[{"start":{"date-parts":[[2024,6,27]],"date-time":"2024-06-27T00:00:00Z","timestamp":1719446400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Spanish Ministry of Science, Innovation, and Universities","award":["PID2022-138206OB-C33"],"award-info":[{"award-number":["PID2022-138206OB-C33"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Partially automated robotic systems, such as camera holders, represent a pivotal step towards enhancing efficiency and precision in surgical procedures. Therefore, this paper introduces an approach for real-time tool localization in laparoscopy surgery using convolutional neural networks. The proposed model, based on two Hourglass modules in series, can localize up to two surgical tools simultaneously. This study utilized three datasets: the ITAP dataset, alongside two publicly available datasets, namely Atlas Dione and EndoVis Challenge. Three variations of the Hourglass-based models were proposed, with the best model achieving high accuracy (92.86%) and frame rates (27.64 FPS), suitable for integration into robotic systems. An evaluation on an independent test set yielded slightly lower accuracy, indicating limited generalizability. The model was further analyzed using the Grad-CAM technique to gain insights into its functionality. Overall, this work presents a promising solution for automating aspects of laparoscopic surgery, potentially enhancing surgical efficiency by reducing the need for manual endoscope manipulation.<\/jats:p>","DOI":"10.3390\/s24134191","type":"journal-article","created":{"date-parts":[[2024,6,27]],"date-time":"2024-06-27T13:26:43Z","timestamp":1719494803000},"page":"4191","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Real-Time Tool Localization for Laparoscopic Surgery Using Convolutional Neural Network"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1468-7363","authenticated-orcid":false,"given":"Diego","family":"Benavides","sequence":"first","affiliation":[{"name":"Instituto de las Tecnolog\u00edas Avanzadas de la Producci\u00f3n (ITAP), Escuela de Ingenier\u00edas Industriales, Universidad de Valladolid, Paseo Prado de la Magdalena 3-5, 47011 Valladolid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1556-7179","authenticated-orcid":false,"given":"Ana","family":"Cisnal","sequence":"additional","affiliation":[{"name":"Instituto de las Tecnolog\u00edas Avanzadas de la Producci\u00f3n (ITAP), Escuela de Ingenier\u00edas Industriales, Universidad de Valladolid, Paseo Prado de la Magdalena 3-5, 47011 Valladolid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2131-7483","authenticated-orcid":false,"given":"Carlos","family":"Font\u00farbel","sequence":"additional","affiliation":[{"name":"Instituto de las Tecnolog\u00edas Avanzadas de la Producci\u00f3n (ITAP), Escuela de Ingenier\u00edas Industriales, Universidad de Valladolid, Paseo Prado de la Magdalena 3-5, 47011 Valladolid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5837-3510","authenticated-orcid":false,"given":"Eusebio","family":"de la Fuente","sequence":"additional","affiliation":[{"name":"Instituto de las Tecnolog\u00edas Avanzadas de la Producci\u00f3n (ITAP), Escuela de Ingenier\u00edas Industriales, Universidad de Valladolid, Paseo Prado de la Magdalena 3-5, 47011 Valladolid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Juan Carlos","family":"Fraile","sequence":"additional","affiliation":[{"name":"Instituto de las Tecnolog\u00edas Avanzadas de la Producci\u00f3n (ITAP), Escuela de Ingenier\u00edas Industriales, Universidad de Valladolid, Paseo Prado de la Magdalena 3-5, 47011 Valladolid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,6,27]]},"reference":[{"key":"ref_1","first-page":"2258","article-title":"A Computerized Analysis of Robotic versus Laparoscopic Task Performance","volume":"21","author":"Narula","year":"2007","journal-title":"Surg. Endosc. Other Interv. Tech."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"S5","DOI":"10.1038\/d41586-020-01037-w","article-title":"Worth the Cost? A Closer Look at the Da Vinci Robot\u2019s Impact on Prostate Cancer Surgery","volume":"580","author":"Crew","year":"2020","journal-title":"Nature"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1007\/978-3-642-19457-3_16","article-title":"Towards the Robotic Co-Worker","volume":"Volume 90","author":"Haddadin","year":"2011","journal-title":"Towards Safe Robots. Springer Tracts in Advanced Robotics"},{"key":"ref_4","first-page":"16","article-title":"Perioperative Management of Unicompartmental Knee Arthroplasty Using the MAKO Robotic Arm System (MAKOplasty)","volume":"38","author":"Pearle","year":"2009","journal-title":"Am. J. Orthop."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"529","DOI":"10.11622\/smedj.2014136","article-title":"Early Experiences with Robot-Assisted Total Knee Arthroplasty Using the DigiMatchTM ROBODOC\u00ae Surgical System","volume":"55","author":"Liow","year":"2014","journal-title":"Singapore Med. J."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"899","DOI":"10.1080\/17434440.2016.1236680","article-title":"Evaluation of the ROSATM Spine Robot for Minimally Invasive Surgical Procedures","volume":"13","author":"Lefranc","year":"2016","journal-title":"Expert Rev. Med. Devices"},{"key":"ref_7","unstructured":"Hurteau, R., DeSantis, S., Begin, E., and Gagner, M. (1994, January 8\u201313). Laparoscopic Surgery Assisted by a Robotic Cameraman: Concept and Experimental Results. Proceedings of the 1994 IEEE International Conference on Robotics and Automation, San Diego, CA, USA."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1109\/51.391776","article-title":"A Telerobotic Assistant for Laparoscopic Surgery","volume":"14","author":"Taylor","year":"1995","journal-title":"IEEE Eng. Med. Biol. Mag."},{"key":"ref_9","first-page":"879","article-title":"ViKY Robotic Scope Holder: Initial Clinical Experience and Preliminary Results Using Instrument Tracking","volume":"15","author":"Voros","year":"2010","journal-title":"IEEE\/ASME Trans. Mechatron."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Fujii, K., Salerno, A., Sriskandarajah, K., Kwok, K.W., Shetty, K., and Yang, G.Z. (2013, January 3\u20137). Gaze Contingent Cartesian Control of a Robotic Arm for Laparoscopic Surgery. Proceedings of the 2013 IEEE\/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan.","DOI":"10.1109\/IROS.2013.6696867"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23\u201328). Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.81"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","article-title":"Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks","volume":"39","author":"Ren","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1542","DOI":"10.1109\/TMI.2017.2665671","article-title":"Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection","volume":"36","author":"Sarikaya","year":"2017","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Jin, A., Yeung, S., Jopling, J., Krause, J., Azagury, D., Milstein, A., and Fei-Fei, L. (2018, January 12\u201315). Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks. Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018, Lake Tahoe, NV, USA.","DOI":"10.1109\/WACV.2018.00081"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016). You Only Look Once: Unified, Real-Time Object Detection. arXiv, 779\u2013788.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Choi, B., Jo, K., Choi, S., and Choi, J. (2017, January 11\u201315). Surgical-Tools Detection Based on Convolutional Neural Network in Laparoscopic Robot-Assisted Surgery. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, Jeju, Republic of Korea.","DOI":"10.1109\/EMBC.2017.8037183"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"78193","DOI":"10.1109\/ACCESS.2020.2989807","article-title":"An Anchor-Free Convolutional Neural Network for Real-Time Surgical Tool Detection in Robot-Assisted Surgery","volume":"8","author":"Liu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Newell, A., Yang, K., and Deng, J. (2016). Stacked Hourglass Networks for Human Pose Estimation. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Springer. 9912 LNCS.","DOI":"10.1007\/978-3-319-46484-8_29"},{"key":"ref_19","first-page":"234","article-title":"U-Net: Convolutional Networks for Biomedical Image Segmentation","volume":"Volume 9351","author":"Ronneberger","year":"2015","journal-title":"Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref_20","unstructured":"Gao, C., Unberath, M., Taylor, R., and Armand, M. (2019). Localizing Dexterous Surgical Tools in X-Ray for Image-Based Navigation. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Ni, Z.L., Bian, G.B., Xie, X.L., Hou, Z.G., Zhou, X.H., and Zhou, Y.J. (2019, January 23\u201327). RASNet: Segmentation for Tracking Surgical Instruments in Surgical Videos Using Refined Attention Segmentation Network. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, Berlin, Germany.","DOI":"10.1109\/EMBC.2019.8856495"},{"key":"ref_22","first-page":"505","article-title":"Simultaneous Recognition and Pose Estimation of Instruments in Minimally Invasive Surgery","volume":"Volume 10434","author":"Kurmann","year":"2017","journal-title":"Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref_23","first-page":"664","article-title":"Concurrent Segmentation and Localization for Tracking of Surgical Instruments","volume":"10434","author":"Laina","year":"2017","journal-title":"Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1145265","DOI":"10.3389\/frobt.2023.1145265","article-title":"Force-Based Control Strategy for a Collaborative Robotic Camera Holder in Laparoscopic Surgery Using Pivoting Motion","volume":"10","author":"Cisnal","year":"2023","journal-title":"Front. Robot. AI"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"S\u00e1nchez-Brizuela, G., Santos-Criado, F.J., Sanz-Gobernado, D., de la Fuente-L\u00f3pez, E., Fraile, J.C., P\u00e9rez-Turiel, J., and Cisnal, A. (2022). Gauze Detection and Segmentation in Minimally Invasive Surgery Video Using Convolutional Neural Networks. Sensors, 22.","DOI":"10.3390\/s22145180"},{"key":"ref_26","unstructured":"(2023, July 10). Data-Grand Challenge. Available online: https:\/\/endovissub-instrument.grand-challenge.org\/Data\/."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Chollet, F. (2017, January 21). Xception: Deep Learning with Depthwise Separable Convolutions. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.195"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"642","DOI":"10.1007\/s11263-019-01204-1","article-title":"CornerNet: Detecting Objects as Paired Keypoints","volume":"128","author":"Law","year":"2018","journal-title":"Int. J. Comput. Vis."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"318","DOI":"10.1109\/TPAMI.2018.2858826","article-title":"Focal Loss for Dense Object Detection","volume":"42","author":"Lin","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1335","DOI":"10.1007\/s11548-020-02214-y","article-title":"Object Extraction via Deep Learning-Based Marker-Free Tracking Framework of Surgical Instruments for Laparoscope-Holder Robots","volume":"15","author":"Zhang","year":"2020","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Padilla, R., Netto, S.L., and da Silva, E.A.B. (2020, January 1\u20133). A Survey on Performance Metrics for Object-Detection Algorithms. Proceedings of the 2020 International Conference on Systems, Signals and Image Processing (IWSSIP), Niteroi, Brazil.","DOI":"10.1109\/IWSSIP48289.2020.9145130"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1095","DOI":"10.1109\/LRA.2020.2967715","article-title":"Automatic Design of Compliant Surgical Forceps with Adaptive Grasping Functions","volume":"5","author":"Sun","year":"2020","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"014502","DOI":"10.1115\/1.4034879","article-title":"Design of a Two Degree-of-Freedom Compliant Tool Tip for a Handheld Powered Surgical Tool","volume":"11","author":"Chandrasekaran","year":"2017","journal-title":"J. Med. Devices Trans. ASME"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/13\/4191\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:06:39Z","timestamp":1760108799000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/13\/4191"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,27]]},"references-count":34,"journal-issue":{"issue":"13","published-online":{"date-parts":[[2024,7]]}},"alternative-id":["s24134191"],"URL":"https:\/\/doi.org\/10.3390\/s24134191","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,27]]}}}