{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T20:56:13Z","timestamp":1789419373877,"version":"build-2803163510"},"reference-count":41,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2025,1,15]],"date-time":"2025-01-15T00:00:00Z","timestamp":1736899200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Bioengineering"],"abstract":"<jats:p>Accurate identification of surgical instruments is crucial for efficient workflows and patient safety within the operating room, particularly in preventing complications such as retained surgical instruments. Artificial Intelligence (AI) models have shown the potential to automate this process. This study evaluates the accuracy of publicly available Large Language Models (LLMs)\u2014ChatGPT-4, ChatGPT-4o, and Gemini\u2014and a specialized commercial mobile application, Surgical-Instrument Directory (SID 2.0), in identifying surgical instruments from images. The study utilized a dataset of 92 high-resolution images of 25 surgical instruments (retractors, forceps, scissors, and trocars) photographed from multiple angles. Model performance was evaluated using accuracy, weighted precision, recall, and F1 score. ChatGPT-4o exhibited the highest accuracy (89.1%) in categorizing instruments (e.g., scissors, forceps). SID 2.0 (77.2%) and ChatGPT-4 (76.1%) achieved comparable accuracy, while Gemini (44.6%) demonstrated lower accuracy in this task. For precise subtype identification of instrument names (like \u201cMayo scissors\u201d or \u201cKelly forceps\u201d), all models had low accuracy, with SID 2.0 having an accuracy of 39.1%, followed by ChatGPT-4o (33.69%). Subgroup analysis revealed ChatGPT-4 and 4o recognized trocars in all instances. Similarly, Gemini identified surgical scissors in all instances. In conclusion, publicly available LLMs can reliably identify surgical instruments at the category level, with ChatGPT-4o demonstrating an overall edge. However, precise subtype identification remains a challenge for all models. These findings highlight the potential of AI-driven solutions to enhance surgical-instrument management and underscore the need for further refinements to improve accuracy and support patient safety.<\/jats:p>","DOI":"10.3390\/bioengineering12010072","type":"journal-article","created":{"date-parts":[[2025,1,15]],"date-time":"2025-01-15T05:59:42Z","timestamp":1736920782000},"page":"72","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Use of Multimodal Artificial Intelligence in Surgical Instrument Recognition"],"prefix":"10.3390","volume":"12","author":[{"given":"Syed Ali","family":"Haider","sequence":"first","affiliation":[{"name":"Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Olivia A.","family":"Ho","sequence":"additional","affiliation":[{"name":"Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7845-7356","authenticated-orcid":false,"given":"Sahar","family":"Borna","sequence":"additional","affiliation":[{"name":"Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cesar A.","family":"Gomez-Cabello","sequence":"additional","affiliation":[{"name":"Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sophia M.","family":"Pressman","sequence":"additional","affiliation":[{"name":"Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-2065-5089","authenticated-orcid":false,"given":"Dave","family":"Cole","sequence":"additional","affiliation":[{"name":"Center for Digital Health, Mayo Clinic, Rochester, MN 55905, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-7562-8758","authenticated-orcid":false,"given":"Ajai","family":"Sehgal","sequence":"additional","affiliation":[{"name":"Center for Digital Health, Mayo Clinic, Rochester, MN 55905, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0043-6659","authenticated-orcid":false,"given":"Bradley C.","family":"Leibovich","sequence":"additional","affiliation":[{"name":"Center for Digital Health, Mayo Clinic, Rochester, MN 55905, USA"},{"name":"Department of Urology, Mayo Clinic, Rochester, MN 55905, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2004-7538","authenticated-orcid":false,"given":"Antonio Jorge","family":"Forte","sequence":"additional","affiliation":[{"name":"Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA"},{"name":"Center for Digital Health, Mayo Clinic, Rochester, MN 55905, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,1,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1177\/1553350607301746","article-title":"Improving Operating Room and Perioperative Safety: Background and Specific Recommendations","volume":"14","author":"Schimpff","year":"2007","journal-title":"Surg. Innov."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1080\/13645700009169645","article-title":"Systems technology in the operating theatre: A prerequisite for the use of advanced devices in surgery","volume":"9","author":"Schurr","year":"2000","journal-title":"Minim. Invasive Ther. Allied Technol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"93","DOI":"10.3121\/cmr.2014.1250.b1-1","article-title":"B1-1: Predictive Modeling to Identify Patients at Risk for Index Hospitalization","volume":"12","author":"Palen","year":"2014","journal-title":"Clin. Med. Res."},{"key":"ref_4","unstructured":"ShahabiKargar, Z., Khanna, S., Good, N., Sattar, A., Lind, J., and O\u2019Dwyer, J. (2014). Predicting Procedure Duration to Improve Scheduling of Elective Surgery. PRICAI 2014: Trends in Artificial Intelligence: 13th Pacific Rim International Conference on Artificial Intelligence, Gold Coast, QLD, Australia, 1\u20135 December 2014, Springer International Publishing. Proceedings 13."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1885","DOI":"10.1093\/jamia\/ocaa140","article-title":"Probabilistic forecasting of surgical case duration using machine learning: Model development and validation","volume":"27","author":"Jiao","year":"2020","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_6","first-page":"471","article-title":"Artificial Intelligence in Endoscopy","volume":"37","author":"Messmann","year":"2021","journal-title":"Bildverarb. Med."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"42010","DOI":"10.1109\/ACCESS.2020.2977072","article-title":"Automatic Markerless Registration and Tracking of the Bone for Computer-Assisted Orthopaedic Surgery","volume":"8","author":"Liu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1016\/j.isurg.2021.03.001","article-title":"Intraoperative video analysis and machine learning models will change the future of surgical training","volume":"1","author":"Kawka","year":"2021","journal-title":"Intell. Surg."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1080\/13645706.2019.1584116","article-title":"Machine and deep learning for workflow recognition during surgery","volume":"28","author":"Padoy","year":"2019","journal-title":"Minim. Invasive Ther. Allied Technol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1253","DOI":"10.1016\/j.surg.2020.10.039","article-title":"Computer vision in surgery","volume":"169","author":"Ward","year":"2020","journal-title":"Surgery"},{"key":"ref_11","first-page":"442","article-title":"Manual Care & Handling of Basic Surgical Instruments-A Prospective","volume":"15","author":"Johnson","year":"2021","journal-title":"Int. J. Clin. Ski."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1016\/j.aorn.2007.07.023","article-title":"Counting difficulties: Retained instruments, sponges, and needles","volume":"87","author":"Jackson","year":"2008","journal-title":"AORN J."},{"key":"ref_13","first-page":"14","article-title":"Missed Instrument and Surgical Sponge (Gauze and Pack)","volume":"20","author":"Nasir","year":"2008","journal-title":"Internet J. Surg."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1515\/cdbme-2021-1001","article-title":"Design and Evaluation of a Novel Instrument Gripper for Handling of Surgical Instruments","volume":"7","author":"Heibeyn","year":"2021","journal-title":"Curr. Dir. Biomed. Eng."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"E275","DOI":"10.1503\/cjs.022720","article-title":"Optimizing the surgical instrument tray to immediately increase efficiency and lower costs in the operating room","volume":"65","author":"Toor","year":"2022","journal-title":"Can. J. Surg."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Ran, B., Huang, B., Liang, S., and Hou, Y. (2023). Surgical Instrument Detection Algorithm Based on Improved YOLOv7x. Sensors, 23.","DOI":"10.3390\/s23115037"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1080\/20476965.2018.1496875","article-title":"Inventory management of surgical supplies and sterile instruments in hospitals: A literature review","volume":"8","author":"Ahmadi","year":"2019","journal-title":"Health Syst."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"153.e151","DOI":"10.1016\/j.jpurol.2019.01.013","article-title":"Decreasing operating room costs via reduction of surgical instruments","volume":"15","author":"Nast","year":"2019","journal-title":"J. Pediatr. Urol."},{"key":"ref_19","unstructured":"(2024, July 07). Tracking Surgical Instruments with AI: A New Approach to Patient Safety. Available online: https:\/\/www.healthcareitnews.com\/news\/tracking-surgical-instruments-ai-new-approach-patient-safety."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"725","DOI":"10.1016\/j.jamcollsurg.2020.01.037","article-title":"Automated Surgical Instrument Detection from Laparoscopic Gastrectomy Video Images Using an Open Source Convolutional Neural Network Platform","volume":"230","author":"Yamazaki","year":"2020","journal-title":"J. Am. Coll. Surg."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"923","DOI":"10.1007\/s10278-022-00616-9","article-title":"Lightweight Deep Neural Network for Articulated Joint Detection of Surgical Instrument in Minimally Invasive Surgical Robot","volume":"35","author":"Sun","year":"2022","journal-title":"J. Digit. Imaging"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Nema, S., and Vachhani, L. (2022). Surgical instrument detection and tracking technologies: Automating dataset labeling for surgical skill assessment. Front. Robot. AI, 9.","DOI":"10.3389\/frobt.2022.1030846"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"103945","DOI":"10.1016\/j.robot.2021.103945","article-title":"Visual detection and tracking algorithms for minimally invasive surgical instruments: A comprehensive review of the state-of-the-art","volume":"149","author":"Wang","year":"2021","journal-title":"Robot. Auton. Syst."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"23748","DOI":"10.1109\/ACCESS.2020.2969885","article-title":"Surgical Tools Detection Based on Modulated Anchoring Network in Laparoscopic Videos","volume":"8","author":"Zhang","year":"2020","journal-title":"IEEE Access"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1186\/s13037-024-00406-y","article-title":"Artificial intelligence model for automated surgical instrument detection and counting: An experimental proof-of-concept study","volume":"18","author":"Deol","year":"2024","journal-title":"Patient Saf. Surg."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1929","DOI":"10.1007\/s11548-024-03226-8","article-title":"Towards multimodal graph neural networks for surgical instrument anticipation","volume":"19","author":"Wagner","year":"2024","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1217","DOI":"10.1007\/s11548-019-01995-1","article-title":"Video-based surgical skill assessment using 3D convolutional neural networks","volume":"14","author":"Funke","year":"2019","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Lavanchy, J.L., Zindel, J., Kirtac, K., Twick, I., Hosgor, E., Candinas, D., and Beldi, G. (2021). Automation of surgical skill assessment using a three-stage machine learning algorithm. Sci. Rep., 11.","DOI":"10.1038\/s41598-021-84295-6"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"679","DOI":"10.1007\/s00464-021-08336-x","article-title":"A contextual detector of surgical tools in laparoscopic videos using deep learning","volume":"36","author":"Namazi","year":"2022","journal-title":"Surg. Endosc."},{"key":"ref_30","unstructured":"(2024, July 07). SAVI transforms surgical instrument tracking with Google Cloud | Google Cloud Blog. Available online: https:\/\/cloud.google.com\/blog\/products\/ai-machine-learning\/savi-transforms-surgical-instrument-tracking-with-google-cloud."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1173","DOI":"10.1177\/0278364907083395","article-title":"Automatic Detection of Instruments in Laparoscopic Images: A First Step Towards High-level Command of Robotic Endoscopic Holders","volume":"26","author":"Voros","year":"2007","journal-title":"Int. J. Robot. Res."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1007\/s11548-011-0662-5","article-title":"Online recognition of surgical instruments by information fusion","volume":"7","author":"Neumuth","year":"2012","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_33","unstructured":"(2024, January 13). ChatGPT. Available online: https:\/\/openai.com\/chatgpt\/overview\/."},{"key":"ref_34","unstructured":"OpenAI (2025, January 09). Hello GPT-4o. Available online: https:\/\/openai.com\/index\/hello-gpt-4o."},{"key":"ref_35","unstructured":"Google (2024, July 07). Gemini. Available online: https:\/\/gemini.google.com\/app\/1018718432ef9dbf."},{"key":"ref_36","unstructured":"(2024, July 07). SID by LayerJot. Available online: https:\/\/www.layerjot.com\/sid."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"e462","DOI":"10.1097\/PTS.0000000000000365","article-title":"Evaluating the Impact of Radio Frequency Identification Retained Surgical Instruments Tracking on Patient Safety: Literature Review","volume":"17","author":"Schnock","year":"2017","journal-title":"J. Patient Saf."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"686","DOI":"10.1016\/j.jamcollsurg.2021.08.689","article-title":"Novel Computer-Aided Diagnosis Software for the Prevention of Retained Surgical Items","volume":"233","author":"Yamaguchi","year":"2021","journal-title":"J. Am. Coll. Surg."},{"key":"ref_39","unstructured":"Nadeau, K. (2025, January 13). The Sterile Processing Department Digital Transformation. Available online: https:\/\/www.hpnonline.com\/sterile-processing\/article\/53083618\/the-sterile-processing-department-digital-transformation."},{"key":"ref_40","unstructured":"(2025, January 13). Revolutionizing Sterile Processing Management with AI | CensisAI2. Available online: https:\/\/censis.com\/solutions\/ai2\/."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1038\/s41746-022-00712-8","article-title":"Multimodal machine learning in precision health: A scoping review","volume":"5","author":"Kline","year":"2022","journal-title":"NPJ Digit. Med."}],"container-title":["Bioengineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2306-5354\/12\/1\/72\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,8]],"date-time":"2025-10-08T10:28:56Z","timestamp":1759919336000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2306-5354\/12\/1\/72"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,15]]},"references-count":41,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,1]]}},"alternative-id":["bioengineering12010072"],"URL":"https:\/\/doi.org\/10.3390\/bioengineering12010072","relation":{},"ISSN":["2306-5354"],"issn-type":[{"value":"2306-5354","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,15]]}}}