{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,9]],"date-time":"2026-04-09T05:09:04Z","timestamp":1775711344848,"version":"3.50.1"},"reference-count":26,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2020,3,4]],"date-time":"2020-03-04T00:00:00Z","timestamp":1583280000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["HE7364\/1-2"],"award-info":[{"award-number":["HE7364\/1-2"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Deformable image registration is still a challenge when the considered images have strong variations in appearance and large initial misalignment. A huge performance gap currently remains for fast-moving regions in videos or strong deformations of natural objects. We present a new semantically guided and two-step deep deformation network that is particularly well suited for the estimation of large deformations. We combine a U-Net architecture that is weakly supervised with segmentation information to extract semantically meaningful features with multiple stages of nonrigid spatial transformer networks parameterized with low-dimensional B-spline deformations. Combining alignment loss and semantic loss functions together with a regularization penalty to obtain smooth and plausible deformations, we achieve superior results in terms of alignment quality compared to previous approaches that have only considered a label-driven alignment loss. Our network model advances the state of the art for inter-subject face part alignment and motion tracking in medical cardiac magnetic resonance imaging (MRI) sequences in comparison to the FlowNet and Label-Reg, two recent deep-learning registration frameworks. The models are compact, very fast in inference, and demonstrate clear potential for a variety of challenging tracking and\/or alignment tasks in computer vision and medical image analysis.<\/jats:p>","DOI":"10.3390\/s20051392","type":"journal-article","created":{"date-parts":[[2020,3,4]],"date-time":"2020-03-04T10:46:08Z","timestamp":1583318768000},"page":"1392","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Semantically Guided Large Deformation Estimation with Deep Networks"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1904-2270","authenticated-orcid":false,"given":"In Young","family":"Ha","sequence":"first","affiliation":[{"name":"Institute of medical informatics, University of Luebeck, 23558 Luebeck, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8845-360X","authenticated-orcid":false,"given":"Matthias","family":"Wilms","sequence":"additional","affiliation":[{"name":"Department of Radiology, University of Calgary, Calgary, AB T2N 4N1, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mattias","family":"Heinrich","sequence":"additional","affiliation":[{"name":"Institute of medical informatics, University of Luebeck, 23558 Luebeck, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,3,4]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., and Brox, T. (2017, January 21\u201326). Flownet 2.0: Evolution of optical flow estimation with deep networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.179"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Qin, C., Bai, W., Schlemper, J., Petersen, S.E., Piechnik, S.K., Neubauer, S., and Rueckert, D. (2018). Joint Learning of Motion Estimation and Segmentation for Cardiac MR Image Sequences. arXiv.","DOI":"10.1007\/978-3-030-00129-2_7"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Hu, Y., Modat, M., Gibson, E., Ghavami, N., Bonmati, E., Moore, C.M., Emberton, M., Noble, J.A., Barratt, D.C., and Vercauteren, T. (2018, January 4\u20137). Label-driven weakly-supervised learning for multimodal deformarle image registration. Proceedings of the 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), Washington, DC, USA.","DOI":"10.1109\/ISBI.2018.8363756"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Sun, D., Yang, X., Liu, M.Y., and Kautz, J. (2018, January 19\u201321). PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00931"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Van Der Smagt, P., Cremers, D., and Brox, T. (2015, January 13\u201316). Flownet: Learning optical flow with convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.316"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Roh\u00e9, M.M., Datar, M., Heimann, T., Sermesant, M., and Pennec, X. (2017, January 11\u201313). SVF-Net: Learning deformable image registration using shape matching. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Quebec City, QC, Canada.","DOI":"10.1007\/978-3-319-66182-7_31"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Krebs, J., Mansi, T., Delingette, H., Zhang, L., Ghesu, F.C., Miao, S., Maier, A.K., Ayache, N., Liao, R., and Kamen, A. (2017, January 11\u201313). Robust non-rigid registration through agent-based action learning. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Quebec City, QC, Canada.","DOI":"10.1007\/978-3-319-66182-7_40"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Sokooti, H., de Vos, B., Berendsen, F., Lelieveldt, B.P., I\u0161gum, I., and Staring, M. (2017, January 11\u201313). Nonrigid image registration using multi-scale 3D convolutional neural networks. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Quebec City, QC, Canada.","DOI":"10.1007\/978-3-319-66182-7_27"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"de Vos, B.D., Berendsen, F.F., Viergever, M.A., Staring, M., and I\u0161gum, I. (2017). End-to-end unsupervised deformable image registration with a convolutional neural network. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer.","DOI":"10.1007\/978-3-319-67558-9_24"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Li, H., and Fan, Y. (2017). Non-rigid image registration using fully convolutional networks with deep self-supervision. arXiv.","DOI":"10.1109\/ISBI.2018.8363757"},{"key":"ref_11","unstructured":"Jaderberg, M., Simonyan, K., Zisserman, A., and Kavukcuoglu, K. (2015). Spatial transformer networks. Advances in Neural Information Processing Systems, Neural Information Processing Systems Foundation, Inc. (NIPS )."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1788","DOI":"10.1109\/TMI.2019.2897538","article-title":"VoxelMorph: A learning framework for deformable medical image registration","volume":"38","author":"Balakrishnan","year":"2019","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_13","unstructured":"Sevilla-Lara, L., Sun, D., Jampani, V., and Black, M.J. (July, January 26). Optical flow with semantic segmentation and localized layers. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Hur, J., and Roth, S. (2016, January 8\u201316). Joint optical flow and temporally consistent semantic segmentation. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46604-0_12"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Cheng, J., Tsai, Y.H., Wang, S., and Yang, M.H. (2017, January 22\u201329). SegFlow: Joint learning for video object segmentation and optical flow. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.81"},{"key":"ref_16","unstructured":"Tsai, Y.H., Yang, M.H., and Black, M.J. (July, January 26). Video segmentation via object flow. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5\u20139). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"712","DOI":"10.1109\/42.796284","article-title":"Nonrigid registration using free-form deformations: Application to breast MR images","volume":"18","author":"Rueckert","year":"1999","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Le, V., Brandt, J., Lin, Z., Bourdev, L., and Huang, T.S. (2012, January 7\u201313). Interactive facial feature localization. Proceedings of the European Conference on Computer Vision, Florence, Italy.","DOI":"10.1007\/978-3-642-33712-3_49"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2514","DOI":"10.1109\/TMI.2018.2837502","article-title":"Deep Learning Techniques for Automatic MRI Cardiac Multi-structures Segmentation and Diagnosis: Is the Problem Solved?","volume":"37","author":"Bernard","year":"2018","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Smith, B.M., Zhang, L., Brandt, J., Lin, Z., and Yang, J. (2013, January 25\u201327). Exemplar-based face parsing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.","DOI":"10.1109\/CVPR.2013.447"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1007\/s11263-013-0667-3","article-title":"Face alignment by explicit shape regression","volume":"107","author":"Cao","year":"2014","journal-title":"Int. J. Comput. Vis."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2262","DOI":"10.1109\/TPAMI.2010.46","article-title":"Point set registration: Coherent point drift","volume":"32","author":"Myronenko","year":"2010","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J., and Dalca, A.V. (2018, January 16\u201320). An Unsupervised Learning Model for Deformable Medical Image Registration. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Granada, Spain.","DOI":"10.1109\/CVPR.2018.00964"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Krebs, J., Mansi, T., Mailh\u00e9, B., Ayache, N., and Delingette, H. (2018). Unsupervised Probabilistic Deformation Modeling for Robust Diffeomorphic Registration. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer.","DOI":"10.1007\/978-3-030-00889-5_12"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"470","DOI":"10.1016\/j.neuroimage.2013.04.114","article-title":"Alzheimer\u2019s Disease Neuroimaging Initiative (ADNI). LCC-Demons: A robust and accurate symmetric diffeomorphic registration algorithm","volume":"81","author":"Lorenzi","year":"2013","journal-title":"NeuroImage"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/5\/1392\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:03:54Z","timestamp":1760173434000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/5\/1392"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,4]]},"references-count":26,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2020,3]]}},"alternative-id":["s20051392"],"URL":"https:\/\/doi.org\/10.3390\/s20051392","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3,4]]}}}