{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T00:45:53Z","timestamp":1774658753244,"version":"3.50.1"},"reference-count":30,"publisher":"MDPI AG","issue":"18","license":[{"start":{"date-parts":[[2021,9,15]],"date-time":"2021-09-15T00:00:00Z","timestamp":1631664000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["1345332282"],"award-info":[{"award-number":["1345332282"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100014188","name":"Ministry of Science and ICT, South Korea","doi-asserted-by":"publisher","award":["2020-0-00161"],"award-info":[{"award-number":["2020-0-00161"]}],"id":[{"id":"10.13039\/501100014188","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Accurate identification of the coronary ostia from 3D coronary computed tomography angiography (CCTA) is a essential prerequisite step for automatically tracking and segmenting three main coronary arteries. In this paper, we propose a novel deep reinforcement learning (DRL) framework to localize the two coronary ostia from 3D CCTA. An optimal action policy is determined using a fully explicit spatial-sequential encoding policy network applying 2.5D Markovian states with three past histories. The proposed network is trained using a dueling DRL framework on the CAT08 dataset. The experiment results show that our method is more efficient and accurate than the other methods. blueFloating-point operations (FLOPs) are calculated to measure computational efficiency. The result shows that there are 2.5M FLOPs on the proposed method, which is about 10 times smaller value than 3D box-based methods. In terms of accuracy, the proposed method shows that 2.22 \u00b1 1.12 mm and 1.94 \u00b1 0.83 errors on the left and right coronary ostia, respectively. The proposed method can be applied to the tasks to identify other target objects by changing the target locations in the ground truth data. Further, the proposed method can be utilized as a pre-processing method for coronary artery tracking methods.<\/jats:p>","DOI":"10.3390\/s21186187","type":"journal-article","created":{"date-parts":[[2021,9,15]],"date-time":"2021-09-15T12:00:44Z","timestamp":1631707244000},"page":"6187","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Deep Reinforcement Learning with Explicit Spatio-Sequential Encoding Network for Coronary Ostia Identification in CT Images"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5805-7494","authenticated-orcid":false,"given":"Yeonggul","family":"Jang","sequence":"first","affiliation":[{"name":"Graduate School of Medical Science, Brain Korea 21 Project, Yonsei University College of Medicine, Seoul 03722, Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0414-1762","authenticated-orcid":false,"given":"Byunghwan","family":"Jeon","sequence":"additional","affiliation":[{"name":"School of Computer Science, Kyungil University, Gyeongsan 38428, Korea"}]}],"member":"1968","published-online":{"date-parts":[[2021,9,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1109\/TPAMI.2017.2782687","article-title":"Multi-scale deep reinforcement learning for real-time 3D-landmark detection in CT scans","volume":"41","author":"Ghesu","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1016\/j.media.2019.02.007","article-title":"Evaluating reinforcement learning agents for anatomical landmark detection","volume":"53","author":"Alansary","year":"2019","journal-title":"Med. Image Anal."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Zhou, S.K., Le, H.N., Luu, K., Nguyen, H.V., and Ayache, N. (2021). Deep reinforcement learning in medical imaging: A literature review. arXiv.","DOI":"10.1016\/j.media.2021.102193"},{"key":"ref_4","unstructured":"Dai, T., Dubois, M., Arulkumaran, K., Campbell, J., Bass, C., Billot, B., Uslu, F., de Paola, V., Clopath, C., and Bharath, A.A. (2019, January 8\u201310). Deep Reinforcement Learning for Subpixel Neural Tracking. Proceedings of the International Conference on Medical Imaging with Deep Learning, London, UK."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Zhang, P., Wang, F., and Zheng, Y. (2018). Deep reinforcement learning for vessel centerline tracing in multi-modality 3d volumes. International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.","DOI":"10.1007\/978-3-030-00937-3_86"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1170","DOI":"10.1109\/TMI.2015.2482920","article-title":"Improving computer-aided detection using convolutional neural networks and random view aggregation","volume":"35","author":"Roth","year":"2015","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Liu, S., Zhang, D., Song, Y., Peng, H., and Cai, W. (2017). Triple-crossing 2.5 d convolutional neural network for detecting neuronal arbours in 3d microscopic images. International Workshop on Machine Learning in Medical Imaging, Springer.","DOI":"10.1007\/978-3-319-67389-9_22"},{"key":"ref_8","unstructured":"Al, W.A., Yun, I.D., and Chun, E.J. (2019). Automatic Left Atrial Appendage Orifice Detection for Preprocedural Planning of Appendage Closure. arXiv."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1016\/j.media.2018.10.005","article-title":"Coronary artery centerline extraction in cardiac CT angiography using a CNN-based orientation classifier","volume":"51","author":"Wolterink","year":"2019","journal-title":"Med. Image Anal."},{"key":"ref_10","unstructured":"Navarro, F., Sekuboyina, A., Waldmannstetter, D., Peeken, J.C., Combs, S.E., and Menze, B.H. (2020, January 6\u20138). Deep reinforcement learning for organ localization in CT. Proceedings of the International Conference on Medical Imaging with Deep Learning, Montreal, QC, Canada."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Leroy, G., Rueckert, D., and Alansary, A. (2020). Communicative reinforcement learning agents for landmark detection in brain images. Machine Learning in Clinical Neuroimaging and Radiogenomics in Neuro-Oncology, Springer.","DOI":"10.1007\/978-3-030-66843-3_18"},{"key":"ref_13","first-page":"1245","article-title":"Partial policy-based reinforcement learning for anatomical landmark localization in 3d medical images","volume":"39","author":"Al","year":"2019","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"222","DOI":"10.1016\/j.patcog.2017.03.005","article-title":"Maximum a posteriori estimation method for aorta localization and coronary seed identification","volume":"68","author":"Jeon","year":"2017","journal-title":"Pattern Recognit."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"106958","DOI":"10.1016\/j.patcog.2019.07.003","article-title":"Identification of coronary arteries in CT images by Bayesian analysis of geometric relations among anatomical landmarks","volume":"96","author":"Jeon","year":"2019","journal-title":"Pattern Recognit."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1217","DOI":"10.1109\/TMI.2016.2538802","article-title":"Marginal space deep learning: Efficient architecture for volumetric image parsing","volume":"35","author":"Ghesu","year":"2016","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1304","DOI":"10.1016\/j.media.2013.02.004","article-title":"Global localization of 3D anatomical structures by pre-filtered Hough Forests and discrete optimization","volume":"17","author":"Donner","year":"2013","journal-title":"Med. Image Anal."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1016\/j.media.2017.09.003","article-title":"Integrating geometric configuration and appearance information into a unified framework for anatomical landmark localization","volume":"43","author":"Urschler","year":"2018","journal-title":"Med. Image Anal."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Kasseroller, K., Thaler, F., Payer, C., and \u0160tern, D. (2021). Collaborative Multi-agent Reinforcement Learning for Landmark Localization Using Continuous Action Space. International Conference on Information Processing in Medical Imaging, Springer.","DOI":"10.1007\/978-3-030-78191-0_59"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1798","DOI":"10.1109\/TPAMI.2013.50","article-title":"Representation learning: A review and new perspectives","volume":"35","author":"Bengio","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1016\/j.media.2018.06.006","article-title":"Automated sub-cortical brain structure segmentation combining spatial and deep convolutional features","volume":"48","author":"Kushibar","year":"2018","journal-title":"Med. Image Anal."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1007\/BF00992698","article-title":"Q-learning","volume":"8","author":"Watkins","year":"1992","journal-title":"Mach. Learn."},{"key":"ref_23","unstructured":"Bellman, R.E., and Dreyfus, S.E. (2015). Applied Dynamic Programming, Princeton University Press."},{"key":"ref_24","unstructured":"Tsitsiklis, J.N., and Van Roy, B. (1996, January 3\u20135). Analysis of temporal-diffference learning with function approximation. Proceedings of the 9th International Conference on Neural Information Processing Systems, Denver, CO, USA."},{"key":"ref_25","unstructured":"Wang, Z., Schaul, T., Hessel, M., Van Hasselt, H., Lanctot, M., and De Freitas, N. (2015). Dueling network architectures for deep reinforcement learning. arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"701","DOI":"10.1016\/j.media.2009.06.003","article-title":"Standardized evaluation methodology and reference database for evaluating coronary artery centerline extraction algorithms","volume":"13","author":"Schaap","year":"2009","journal-title":"Med. Image Anal."},{"key":"ref_27","unstructured":"Menghani, G. (2021). Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Yub Jung, H., Lee, S., Seok Heo, Y., and Dong Yun, I. (2015, January 7\u201312). Random tree walk toward instantaneous 3d human pose estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298861"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Al, W.A., Jung, H.Y., Yun, I.D., Jang, Y., Park, H.B., and Chang, H.J. (2018). Automatic aortic valve landmark localization in coronary CT angiography using colonial walk. PLoS ONE, 13.","DOI":"10.1371\/journal.pone.0200317"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1007\/s10554-015-0793-9","article-title":"Automatic aortic root landmark detection in CTA images for preprocedural planning of transcatheter aortic valve implantation","volume":"32","author":"Elattar","year":"2016","journal-title":"Int. J. Cardiovasc. Imaging"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/18\/6187\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:00:06Z","timestamp":1760166006000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/18\/6187"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,15]]},"references-count":30,"journal-issue":{"issue":"18","published-online":{"date-parts":[[2021,9]]}},"alternative-id":["s21186187"],"URL":"https:\/\/doi.org\/10.3390\/s21186187","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,15]]}}}