{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T19:20:43Z","timestamp":1792783243494,"version":"4.5.0"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2023,10,12]],"date-time":"2023-10-12T00:00:00Z","timestamp":1697068800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,10,12]],"date-time":"2023-10-12T00:00:00Z","timestamp":1697068800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"publisher","award":["16SV8088"],"award-info":[{"award-number":["16SV8088"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006785","name":"Google","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006785","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100008894","name":"Stryker","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100008894","id-type":"DOI","asserted-by":"publisher"}]},{"name":"J &J Robotics & Digital Solutions"},{"DOI":"10.13039\/501100005713","name":"Technische Universit\u00e4t M\u00fcnchen","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005713","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J CARS"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Purpose<\/jats:title>\n                    <jats:p>Surgical procedures take place in highly complex operating rooms\u00a0(OR), involving medical staff, patients, devices and their interactions. Until now, only medical professionals are capable of comprehending these intricate links and interactions. This work advances the field toward automated, comprehensive and semantic understanding and modeling of the OR domain by introducing semantic scene graphs (SSG) as a novel approach to describing and summarizing surgical environments in a structured and semantically rich manner.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We create the first open-source 4D SSG dataset. 4D-OR includes simulated total knee replacement surgeries captured by RGB-D sensors in a realistic OR simulation center. It includes annotations for SSGs, human and object pose, clinical roles and surgical phase labels. We introduce a neural network-based SSG generation pipeline for semantic reasoning in the OR and apply our approach to two downstream tasks: clinical role prediction and surgical phase recognition.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We show that our pipeline can successfully reason within the OR domain. The capabilities of our scene graphs are further highlighted by their successful application to clinical role prediction and surgical phase recognition tasks.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>This work paves the way for multimodal holistic operating room modeling, with the potential to significantly enhance the state of the art in surgical data analysis, such as enabling more efficient and precise decision-making during surgical procedures, and ultimately improving patient safety and surgical outcomes. We release our code and dataset at github.com\/egeozsoy\/4D-OR.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1007\/s11548-023-03022-w","type":"journal-article","created":{"date-parts":[[2023,10,12]],"date-time":"2023-10-12T11:01:56Z","timestamp":1697108516000},"page":"791-799","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Holistic OR domain modeling: a semantic scene graph approach"],"prefix":"10.1007","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2841-3838","authenticated-orcid":false,"given":"Ege","family":"\u00d6zsoy","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tobias","family":"Czempiel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Evin P\u0131nar","family":"\u00d6rnek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ulrich","family":"Eck","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Federico","family":"Tombari","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nassir","family":"Navab","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,10,12]]},"reference":[{"issue":"9","key":"3022_CR1","doi-asserted-by":"publisher","first-page":"691","DOI":"10.1038\/s41551-017-0132-7","volume":"1","author":"L Maier-Hein","year":"2017","unstructured":"Maier-Hein L, Vedula SS, Speidel S, Navab N, Kikinis R, Park A, Eisenmann M, Feussner H, Forestier G, Giannarou S, Hashizume M, Katic D, Kenngott H, Kranzfelder M, Malpani A, M\u00e4rz K, Neumuth T, Padoy N, Pugh C, Schoch N, Stoyanov D, Taylor R, Wagner M, Hager GD, Jannin P (2017) Surgical data science for next-generation interventions. Nat Biomed Eng 1(9):691\u2013696","journal-title":"Nat Biomed Eng"},{"key":"3022_CR2","doi-asserted-by":"crossref","unstructured":"Kennedy-Metz LR, Mascagni P, Torralba A, Dias RD, Perona P, Shah JA, Padoy N, Zenati MA (2020) Computer vision in the operating room: opportunities and caveats. IEEE transactions on medical robotics and bionics 3(1):2\u201310","DOI":"10.1109\/TMRB.2020.3040002"},{"key":"3022_CR3","doi-asserted-by":"publisher","unstructured":"Li Z, Shaban A, Simard J, Rabindran D, DiMaio SP, Mohareri O (2020) A robotic 3d perception system for operating room environment awareness. https:\/\/doi.org\/10.48550\/arXiv.2003.09487","DOI":"10.48550\/arXiv.2003.09487"},{"key":"3022_CR4","doi-asserted-by":"crossref","unstructured":"Lalys F, Jannin P (2014) Surgical process modelling: a review. Surgical process modelling: a review. Int J Comput Assist Radiol Surg 9:495\u2013511","DOI":"10.1007\/s11548-013-0940-5"},{"issue":"4","key":"3022_CR5","doi-asserted-by":"publisher","first-page":"684","DOI":"10.1097\/SLA.0000000000004425","volume":"273","author":"CR Garrow","year":"2020","unstructured":"Garrow CR, Kowalewski K-F, Li L, Wagner M, Schmidt MW, Engelhardt S, Hashimoto DA, Kenngott HG, Bodenstedt S, Speidel S, M\u00fcller-Stich BP, Nickel F (2020) Machine learning for surgical phase recognition: a systematic review. Ann Surg 273(4):684\u2013693","journal-title":"Ann Surg"},{"key":"3022_CR6","doi-asserted-by":"crossref","unstructured":"Nwoye CI, Mutter D, Marescaux J, Padoy N (2019) Weakly supervised convolutional LSTM approach for tool tracking in laparoscopic videos. International journal of computer assisted radiology and surgery 14:1059\u20131067","DOI":"10.1007\/s11548-019-01958-6"},{"key":"3022_CR7","doi-asserted-by":"publisher","unstructured":"Czempiel T, Paschali M, Keicher M, Simson W, Feussner H, Kim ST, Navab N (2020) TeCNO: surgical phase recognition with multi-stage temporal convolutional networks. In: Martel AL et al. Medical Image Computing and Computer Assisted Intervention - MICCAI 2020. MICCAI 2020. Lecture Notes in Computer Science, vol 12263, pp. 343\u2013352. Springer, Cham. https:\/\/doi.org\/10.1007\/978-3-030-59716-0_33","DOI":"10.1007\/978-3-030-59716-0_33"},{"key":"3022_CR8","doi-asserted-by":"crossref","unstructured":"Bodenstedt S, Rivoir D, Jenke A, Wagner M, Breucha M, M\u00fcller-Stich B, Mees ST, Weitz J, Speidel S (2019) Active learning using deep Bayesian networks for surgical workflow analysis. Int J Comput Assist Radiol Surg 14:1079\u20131087","DOI":"10.1007\/s11548-019-01963-9"},{"key":"3022_CR9","doi-asserted-by":"publisher","unstructured":"Laina I, Rieke N, Rupprecht C, Vizca\u00edno JP, Eslami A, Tombari F, Navab N (2017) Concurrent segmentation and localization for tracking of surgical instruments. In: Descoteaux, M, Maier-Hein L, Franz A, Jannin P, Collins D, Duchesne S (eds) Medical Image Computing and Computer-Assisted Intervention \u2013 MICCAI 2017. MICCAI 2017. Lecture Notes in Computer Science, vol 10434. Springer, Cham. https:\/\/doi.org\/10.1007\/978-3-319-66185-8_75","DOI":"10.1007\/978-3-319-66185-8_75"},{"key":"3022_CR10","unstructured":"Srivastav V, Issenhuth T, Kadkhodamohammadi A, Mathelin M, Gangi A, Padoy N (2018) MVOR: a multi-view RGB-D operating room dataset for 2D and 3D human pose estimation. arXiv preprint arXiv:1808.08180"},{"issue":"1","key":"3022_CR11","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1007\/s11548-022-02761-6","volume":"18","author":"N Louis","year":"2023","unstructured":"Louis N, Zhou L, Yule SJ, Dias RD, Manojlovich M, Pagani FD, Likosky DS, Corso JJ (2023) Temporally guided articulated hand pose tracking in surgical videos. Int J Comput Assist Radiol Surg 18(1):117\u2013125","journal-title":"Int J Comput Assist Radiol Surg"},{"key":"3022_CR12","doi-asserted-by":"crossref","unstructured":"Dias RD, Kennedy-Metz LR, Yule SJ, Gombolay M, Zenati MA (2022) Assessing team situational awareness in the operating room via computer vision. In: 2022 IEEE conference on cognitive and computational aspects of situation management (CogSIMA). IEEE, pp 94\u201396","DOI":"10.1109\/CogSIMA54611.2022.9830664"},{"key":"3022_CR13","doi-asserted-by":"publisher","unstructured":"Johnson J, Krishna R, Stark M, Li L-J, Shamma DA, Bernstein MS, Fei-Fei L (2015) Image retrieval using scene graphs. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Boston, MA, USA, 2015, pp. 3668\u20133678, https:\/\/doi.org\/10.1109\/CVPR.2015.7298990","DOI":"10.1109\/CVPR.2015.7298990"},{"key":"3022_CR14","doi-asserted-by":"crossref","unstructured":"Johnson J, Gupta A, Fei-Fei L (2018) Image generation from scene graphs. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1219\u20131228","DOI":"10.1109\/CVPR.2018.00133"},{"key":"3022_CR15","doi-asserted-by":"crossref","unstructured":"Dhamo H, Farshad A, Laina I, Navab N, Hager GD, Tombari F, Rupprecht C (2020) Semantic image manipulation using scene graphs. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp 5213\u20135222","DOI":"10.1109\/CVPR42600.2020.00526"},{"key":"3022_CR16","doi-asserted-by":"crossref","unstructured":"Ji J, Krishna R, Fei-Fei L, Niebles JC (2020) Action genome: actions as compositions of spatio-temporal scene graphs. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, 10236\u201310247","DOI":"10.1109\/CVPR42600.2020.01025"},{"key":"3022_CR17","doi-asserted-by":"crossref","unstructured":"Wald J, Dhamo H, Navab N, Tombari F (2020) Learning 3d semantic scene graphs from 3d indoor reconstructions. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, 3961\u20133970","DOI":"10.1109\/CVPR42600.2020.00402"},{"key":"3022_CR18","doi-asserted-by":"crossref","unstructured":"Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg A, Fei-Fei L (2015) ImageNet large scale visual recognition challenge. International journal of computer vision 115:211\u2013252","DOI":"10.1007\/s11263-015-0816-y"},{"key":"3022_CR19","doi-asserted-by":"crossref","unstructured":"Dai A, Chang AX, Savva M, Halber M, Funkhouser T, Nie\u00dfner M (2017) ScanNet: richly-annotated 3d reconstructions of indoor scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 5828\u20135839","DOI":"10.1109\/CVPR.2017.261"},{"key":"3022_CR20","doi-asserted-by":"crossref","unstructured":"Caesar H, Bankiti V, Lang AH, Vora S, Liong VE, Xu Q, Krishnan A, Pan Y, Baldan G, Beijbom O (2020) nuScenes: a multimodal dataset for autonomous driving. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, 11621\u201311631","DOI":"10.1109\/CVPR42600.2020.01164"},{"key":"3022_CR21","doi-asserted-by":"crossref","unstructured":"Silberman N, Hoiem D, Kohli P, Fergus R (2012) Indoor segmentation and support inference from RGBD images. In: Computer Vision\u2013ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7\u201313, 2012, Proceedings, Part V 12, Springer, pp. 746\u2013760","DOI":"10.1007\/978-3-642-33715-4_54"},{"key":"3022_CR22","doi-asserted-by":"crossref","unstructured":"Krishna R, Zhu Y, Groth O, Johnson J, Hata K, Kravitz J, Chen S, Kalantidis Y, Li L-J, Shamma DA, Bernstein M, Fei-Fei L (2017) Visual genome: connecting language and vision using crowdsourced dense image annotations. International journal of computer vision 123:32\u201373","DOI":"10.1007\/s11263-016-0981-7"},{"key":"3022_CR23","doi-asserted-by":"crossref","unstructured":"Sharghi A, Haugerud H, Oh D, Mohareri O (2020) Automatic operating room surgical activity recognition for robot-assisted surgery. In: Medical Image Computing and Computer Assisted Intervention\u2013 MICCAI 2020: 23rd International Conference, Lima, Peru, October 4\u20138, 2020, Proceedings, Part III 23, Springer, pp. 385\u2013395","DOI":"10.1007\/978-3-030-59716-0_37"},{"key":"3022_CR24","doi-asserted-by":"crossref","unstructured":"\u00d6zsoy E, \u00d6rnek EP, Eck U, Czempiel T, Tombari F, Navab N (2022) 4D-OR: semantic scene graphs for or domain modeling. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp. 475\u2013485","DOI":"10.1007\/978-3-031-16449-1_45"},{"key":"3022_CR25","doi-asserted-by":"crossref","unstructured":"Tu H, Wang C, Zeng W (2020) VoxelPose: towards multi-camera 3d human pose estimation in wild environment. In: Computer Vision\u2013ECCV 2020: 16th European Conference, Glasgow, UK, August 23\u201328, 2020, Proceedings, Part I 16, Springer, pp. 197\u2013212","DOI":"10.1007\/978-3-030-58452-8_12"},{"key":"3022_CR26","doi-asserted-by":"crossref","unstructured":"Liu Z, Zhang Z, Cao Y, Hu H, Tong X (2021) Group-free 3d object detection via transformers. In: Proceedings of the IEEE\/CVF International Conference on Computer Vision, pp. 2949\u20132958","DOI":"10.1109\/ICCV48922.2021.00294"},{"key":"3022_CR27","unstructured":"Qi CR, Su H, Mo K, Guibas LJ (2017) PointNet: deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652\u2013660"},{"key":"3022_CR28","unstructured":"Ta, M, Le Q (2019) EfficientNet: rethinking model scaling for convolutional neural networks. In: International conference on machine learning. In: International conference on machine learning, PMLR, pp. 6105\u20136114"},{"key":"3022_CR29","unstructured":"Ying C, Cai T, Luo S, Zheng S, Ke G, He D, Shen Y, Liu T-Y (2021) Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems 34:28877\u201328888"},{"key":"3022_CR30","unstructured":"Qi CR, Yi L, Su H, Guibas LJ (2017) PointNet++: deep hierarchical feature learning on point sets in a metric space. Adv Neural Inf Process 30"},{"key":"3022_CR31","doi-asserted-by":"publisher","unstructured":"\u00d6zsoy E, \u00d6rnek EP, Eck U, Tombari F, Navab N (2021) Multimodal semantic scene graphs for holistic modeling of surgical procedures. https:\/\/doi.org\/10.1038\/s42005-022-00818-1","DOI":"10.1038\/s42005-022-00818-1"}],"container-title":["International Journal of Computer Assisted Radiology and Surgery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11548-023-03022-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11548-023-03022-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11548-023-03022-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,16]],"date-time":"2024-05-16T09:14:20Z","timestamp":1715850860000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11548-023-03022-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,12]]},"references-count":31,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2024,5]]}},"alternative-id":["3022"],"URL":"https:\/\/doi.org\/10.1007\/s11548-023-03022-w","relation":{},"ISSN":["1861-6429"],"issn-type":[{"value":"1861-6429","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,12]]},"assertion":[{"value":"31 May 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 September 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 October 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no relevant financial or non-financial interests to disclose.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Consent to participate and publish was obtained.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate and publish"}}]}}