{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,14]],"date-time":"2026-07-14T14:54:13Z","timestamp":1784040853673,"version":"3.55.0"},"reference-count":22,"publisher":"Springer Science and Business Media LLC","issue":"7","license":[{"start":{"date-parts":[[2024,5,23]],"date-time":"2024-05-23T00:00:00Z","timestamp":1716422400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,5,23]],"date-time":"2024-05-23T00:00:00Z","timestamp":1716422400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001665","name":"Agence Nationale de la Recherche","doi-asserted-by":"publisher","award":["ANR-20-CHIA-0029-01"],"award-info":[{"award-number":["ANR-20-CHIA-0029-01"]}],"id":[{"id":"10.13039\/501100001665","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J CARS"],"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Purpose<\/jats:title>\n                <jats:p>The modern operating room is becoming increasingly complex, requiring innovative intra-operative support systems. While the focus of surgical data science has largely been on video analysis, integrating surgical computer vision with natural language capabilities is emerging as a necessity. Our work aims to advance visual question answering (VQA) in the surgical context with scene graph knowledge, addressing two main challenges in the current surgical VQA systems: removing question\u2013condition bias in the surgical VQA dataset and incorporating scene-aware reasoning in the surgical VQA model design.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>First, we propose a surgical scene graph-based dataset, SSG-VQA, generated by employing segmentation and detection models on publicly available datasets. We build surgical scene graphs using spatial and action information of instruments and anatomies. These graphs are fed into a question engine, generating diverse QA pairs. We then propose SSG-VQA-Net, a novel surgical VQA model incorporating a lightweight Scene-embedded Interaction Module, which integrates geometric scene knowledge in the VQA model design by employing cross-attention between the textual and the scene features.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>Our comprehensive analysis shows that our SSG-VQA dataset provides a more complex, diverse, geometrically grounded, unbiased and surgical action-oriented dataset compared to existing surgical VQA datasets and SSG-VQA-Net outperforms existing methods across different question types and complexities. We highlight that the primary limitation in the current surgical VQA systems is the lack of scene knowledge to answer complex queries.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusion<\/jats:title>\n                <jats:p>We present a novel surgical VQA dataset and model and show that results can be significantly improved by incorporating geometric scene features in the VQA model design. We point out that the bottleneck of the current surgical visual question\u2013answer model lies in learning the encoded representation rather than decoding the sequence. Our SSG-VQA dataset provides a diagnostic benchmark to test the scene understanding and reasoning capabilities of the model. The source code and the dataset will be made publicly available at: <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/CAMMA-public\/SSG-VQA\">https:\/\/github.com\/CAMMA-public\/SSG-VQA<\/jats:ext-link>.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1007\/s11548-024-03141-y","type":"journal-article","created":{"date-parts":[[2024,5,23]],"date-time":"2024-05-23T12:01:41Z","timestamp":1716465701000},"page":"1409-1417","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["Advancing surgical VQA with scene graph knowledge"],"prefix":"10.1007","volume":"19","author":[{"given":"Kun","family":"Yuan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Manasi","family":"Kattel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jo\u00ebl L.","family":"Lavanchy","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nassir","family":"Navab","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vinkle","family":"Srivastav","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicolas","family":"Padoy","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,5,23]]},"reference":[{"key":"3141_CR1","volume":"76","author":"L Maier-Hein","year":"2022","unstructured":"Maier-Hein L, Eisenmann M, Sarikaya D, M\u00e4rz K, Collins T, Malpani A, Fallert J, Feussner H, Giannarou S, Mascagni P (2022) Surgical data science-from concepts toward clinical translation. Media 76:102306","journal-title":"Media"},{"issue":"2","key":"3141_CR2","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1080\/13645706.2019.1584116","volume":"28","author":"N Padoy","year":"2019","unstructured":"Padoy N (2019) Machine and deep learning for workflow recognition during surgery. Minim Invasive Ther Allied Technol 28(2):82\u201390","journal-title":"Minim Invasive Ther Allied Technol"},{"key":"3141_CR3","volume":"78","author":"CI Nwoye","year":"2022","unstructured":"Nwoye CI, Yu T, Gonzalez C, Seeliger B, Mascagni P, Mutter D, Marescaux J, Padoy N (2022) Rendezvous: attention mechanisms for the recognition of surgical action triplets in endoscopic videos. Media 78:102433","journal-title":"Media"},{"issue":"1","key":"3141_CR4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41597-022-01719-2","volume":"10","author":"M Carstens","year":"2023","unstructured":"Carstens M, Rinner FM, Bodenstedt S, Jenke AC, Weitz J, Distler M, Speidel S, Kolbinger FR (2023) The Dresden surgical anatomy dataset for abdominal organ segmentation in surgical data science. Sci Data 10(1):1\u20138","journal-title":"Sci Data"},{"issue":"1","key":"3141_CR5","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1109\/TMI.2016.2593957","volume":"36","author":"AP Twinanda","year":"2016","unstructured":"Twinanda AP, Shehata S, Mutter D, Marescaux J, De Mathelin M, Padoy N (2016) EndoNet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging 36(1):86\u201397","journal-title":"IEEE Trans Med Imaging"},{"key":"3141_CR6","doi-asserted-by":"crossref","unstructured":"Antol S, Agrawal A, Lu J, Mitchell M, Batra D, Zitnick CL, Parikh D (2015) VQA: visual question answering. In: ICCV, pp 2425\u20132433","DOI":"10.1109\/ICCV.2015.279"},{"key":"3141_CR7","doi-asserted-by":"crossref","unstructured":"Hudson DA, Manning CD (2019) GQA: a new dataset for real-world visual reasoning and compositional question answering. In: CVPR, pp 6700\u20136709","DOI":"10.1109\/CVPR.2019.00686"},{"key":"3141_CR8","doi-asserted-by":"crossref","unstructured":"Seenivasan L, Islam M, Krishna AK, Ren H (2022) Surgical-VQA: visual question answering in surgical scenes using transformer. In: MICCAI, pp 33\u201343","DOI":"10.1007\/978-3-031-16449-1_4"},{"key":"3141_CR9","doi-asserted-by":"crossref","unstructured":"Seenivasan L, Islam M, Kannan G, Ren H (2023) SurgicalGPT: end-to-end language-vision GPT for visual question answering in surgery. In: International conference on medical image computing and computer-assisted intervention, pp 281\u2013290","DOI":"10.1007\/978-3-031-43996-4_27"},{"key":"3141_CR10","doi-asserted-by":"crossref","unstructured":"Krishna R, Zhu Y, Groth O, Johnson J, Hata K, Kravitz J, Chen S, Kalantidis Y, Li L-J, Shamma DA (2017) Visual genome: connecting language and vision using crowdsourced dense image annotations. Int J Comput Vis 123:32\u201373","DOI":"10.1007\/s11263-016-0981-7"},{"key":"3141_CR11","doi-asserted-by":"crossref","unstructured":"Jin A, Yeung S, Jopling J, Krause J, Azagury D, Milstein A, Fei-Fei L (2018) Tool detection and operative skill assessment in surgical videos using region-based convolutional neural networks. In: IEEE WACV","DOI":"10.1109\/WACV.2018.00081"},{"key":"3141_CR12","unstructured":"Hong, W.-Y, Kao, C-L, Kuo Y-H, Wang J-R, Chang W-L, Shih C-S (2020) CholecSeg8k: a semantic segmentation dataset for laparoscopic cholecystectomy based on Cholec80. arXiv:2012.12453"},{"key":"3141_CR13","doi-asserted-by":"crossref","unstructured":"Wang C-Y, Bochkovskiy A, Liao H-YM (2023) Yolov7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In: CVPR, pp 7464\u20137475","DOI":"10.1109\/CVPR52729.2023.00721"},{"key":"3141_CR14","doi-asserted-by":"crossref","unstructured":"He K, Gkioxari G, Doll\u00e1r P, Girshick R (2017) Mask R-CNN. In: ICCV, pp 2961\u20132969","DOI":"10.1109\/ICCV.2017.322"},{"key":"3141_CR15","doi-asserted-by":"publisher","unstructured":"Jocher G YOLOv5 by ultralytics. https:\/\/doi.org\/10.5281\/zenodo.3908559. https:\/\/github.com\/ultralytics\/yolov5","DOI":"10.5281\/zenodo.3908559"},{"key":"3141_CR16","doi-asserted-by":"crossref","unstructured":"Chen L-C, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: ECCV","DOI":"10.1007\/978-3-030-01234-2_49"},{"key":"3141_CR17","doi-asserted-by":"crossref","unstructured":"Johnson J, Hariharan B, Maaten L, Fei-Fei L, Zitnick CL, Girshick R (2017) CLEVR: a diagnostic dataset for compositional language and elementary visual reasoning. In: CVPR","DOI":"10.1109\/CVPR.2017.215"},{"key":"3141_CR18","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: CVPR, pp 770\u2013778","DOI":"10.1109\/CVPR.2016.90"},{"key":"3141_CR19","unstructured":"Kenton JDM-WC, Toutanova LK (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT, pp 4171\u20134186"},{"key":"3141_CR20","doi-asserted-by":"crossref","unstructured":"Beltagy I, Lo K, Cohan A (2019) SciBERT: a pretrained language model for scientific text. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 3615\u20133620","DOI":"10.18653\/v1\/D19-1371"},{"key":"3141_CR21","unstructured":"Huang K, Altosaar J, Ranganath R (2019) ClinicalBERT: modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342"},{"key":"3141_CR22","unstructured":"Li LH, Yatskar M, Yin D, Hsieh C-J, Chang K-W (2019) VisualBERT: a simple and performant baseline for vision and language. arXiv:1908.03557"}],"container-title":["International Journal of Computer Assisted Radiology and Surgery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11548-024-03141-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11548-024-03141-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11548-024-03141-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,8]],"date-time":"2024-07-08T17:16:09Z","timestamp":1720458969000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11548-024-03141-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,23]]},"references-count":22,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2024,7]]}},"alternative-id":["3141"],"URL":"https:\/\/doi.org\/10.1007\/s11548-024-03141-y","relation":{},"ISSN":["1861-6429"],"issn-type":[{"value":"1861-6429","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,23]]},"assertion":[{"value":"19 January 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 April 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 May 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}