{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,5]],"date-time":"2025-11-05T06:36:42Z","timestamp":1762324602282,"version":"build-2065373602"},"reference-count":49,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2019,11,29]],"date-time":"2019-11-29T00:00:00Z","timestamp":1574985600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>We develop a deep learning-based matching method between an RGB (red, green and blue) and an infrared image that were captured from satellite sensors. The method includes a convolutional neural network (CNN) that compares the RGB and infrared image pair and a template searching strategy that searches the correspondent point within a search window in the target image to a given point in the reference image. A densely-connected CNN is developed to extract common features from different spectral bands. The network consists of a series of densely-connected convolutions to make full use of low-level features and an augmented cross entropy loss to avoid model overfitting. The network takes band-wise concatenated RGB and infrared images as the input and outputs a similarity score of the RGB and infrared image pair. For a given reference point, the similarity scores within the search window are calculated pixel-by-pixel, and the pixel with the highest score becomes the matching candidate. Experiments on a satellite RGB and infrared image dataset demonstrated that our method obtained more than 75% improvement on matching rate (the ratio of the successfully matched points to all the reference points) over conventional methods such as SURF, RIFT, and PSO-SIFT, and more than 10% improvement compared to other most recent CNN-based structures. Our experiments also demonstrated high performance and generalization ability of our method applying to multitemporal remote sensing images and close-range images.<\/jats:p>","DOI":"10.3390\/rs11232836","type":"journal-article","created":{"date-parts":[[2019,11,29]],"date-time":"2019-11-29T10:58:21Z","timestamp":1575025101000},"page":"2836","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":29,"title":["Matching RGB and Infrared Remote Sensing Images with Densely-Connected Convolutional Neural Networks"],"prefix":"10.3390","volume":"11","author":[{"given":"Ruojin","family":"Zhu","sequence":"first","affiliation":[{"name":"School of Remote Sensing and Information Engineering, Wuhan University, 129 Luoyu Road, Wuhan 430079, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3515-1602","authenticated-orcid":false,"given":"Dawen","family":"Yu","sequence":"additional","affiliation":[{"name":"School of Remote Sensing and Information Engineering, Wuhan University, 129 Luoyu Road, Wuhan 430079, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3088-1481","authenticated-orcid":false,"given":"Shunping","family":"Ji","sequence":"additional","affiliation":[{"name":"School of Remote Sensing and Information Engineering, Wuhan University, 129 Luoyu Road, Wuhan 430079, China"}]},{"given":"Meng","family":"Lu","sequence":"additional","affiliation":[{"name":"Department of Physical Geography, Faculty of Geoscience, Utrecht University, Princetonlaan 8, 3584 CB Utrecht, The Netherlands"}]}],"member":"1968","published-online":{"date-parts":[[2019,11,29]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"977","DOI":"10.1016\/S0262-8856(03)00137-9","article-title":"Image registration methods: A survey","volume":"21","author":"Barbara","year":"2003","journal-title":"Image Vis. Comput."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1494","DOI":"10.1109\/TGRS.2007.892599","article-title":"Robust multispectral image registration using mutual-information models","volume":"45","author":"Kern","year":"2007","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_3","first-page":"4379","article-title":"Image registration by automatic subimage selection and maximization of combined mutual information and spatial information","volume":"4","author":"Amankwah","year":"2013","journal-title":"IEEE Geosci. Remote Sens. Sym."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Bleyer, M., Rhemann, C., and Rother, C. (September, January 29). PatchMatch stereo-stereo matching with slanted support windows. Proceedings of the 2011 British Machine Vision Conference (BMVC), Dundee, UK.","DOI":"10.5244\/C.25.14"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1023\/B:VISI.0000029664.99615.94","article-title":"Distinctive image features from scale-invariant keypoints","volume":"60","author":"Lowe","year":"2004","journal-title":"Int. J. Comput. Vis."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"438","DOI":"10.1137\/080732730","article-title":"ASIFT: A new framework for fully affine invariant image comparison","volume":"2","author":"Morel","year":"2009","journal-title":"Siam J. Imaging Sci."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1016\/j.isprsjprs.2017.11.019","article-title":"Uniform competency-based local feature extraction for remote sensing images","volume":"135","author":"Sedaghat","year":"2018","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1109\/LGRS.2016.2600858","article-title":"Remote sensing image registration with modified SIFT and enhanced feature matching","volume":"14","author":"Ma","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2941","DOI":"10.1109\/TGRS.2017.2656380","article-title":"Robust registration of multimodal remote sensing images based on structural similarity","volume":"55","author":"Ye","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_10","unstructured":"Li, J.Y., Hu, Q.W., and Ai, M.Y. (2018). RIFT: Multi-modal image matching based on radiation-invariant feature transform. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Liu, X.Z., Ai, Y.F., Zhang, J.L., and Wang, Z.P. (2018). A novel affine and contrast invariant descriptor for infrared and visible image registration. Remote Sens., 10.","DOI":"10.3390\/rs10040658"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Dong, Y.Y., Jiao, W.L., Long, T.F., He, G.J., and Gong, C.J. (2018). An extension of phase correlation-based image registration to estimate similarity transform using multiple polar fourier transform. Remote Sens., 10.","DOI":"10.3390\/rs10111719"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Yan, L., Wang, Z.Q., Liu, Y., and Ye, Z.Y. (2018). Generic and automatic markov random field-based registration for multimodal remote sensing image using grayscale and gradient information. Remote Sens., 10.","DOI":"10.3390\/rs10081228"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"523","DOI":"10.1007\/s11554-018-0760-5","article-title":"Robust feature matching via Gaussian field criterion for remote sensing image registration","volume":"15","author":"Ma","year":"2018","journal-title":"J. Real Time Image Process."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"3356","DOI":"10.1016\/j.patcog.2008.04.017","article-title":"Multi-sensor image registration based on intensity and edge orientation information","volume":"41","author":"Yong","year":"2008","journal-title":"Pattern Recogn."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"4328","DOI":"10.1109\/TGRS.2013.2281391","article-title":"A novel coarse-to-fine scheme for automatic image registration based on SIFT and mutual information","volume":"52","author":"Gong","year":"2014","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1016\/j.isprsjprs.2016.04.002","article-title":"Registration of multitemporal aerial optical images using line features","volume":"117","author":"Zhao","year":"2016","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"3466","DOI":"10.1016\/j.patcog.2015.04.011","article-title":"Efficient and accurate set-based registration of time-separated aerial images","volume":"48","author":"Pham","year":"2015","journal-title":"Pattern Recogn."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1688","DOI":"10.1109\/JSTARS.2013.2273871","article-title":"Automatic line segment registration using Gaussian mixture model and expectation-maximization algorithm","volume":"7","author":"Long","year":"2014","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Wang, X., and Xu, Q. (2016, January 10\u201315). Multi-sensor optical remote sensing image registration based on Line-Point Invariant. Proceedings of the 2016 Geoscience Remote Sensing Symposium (IGARSS), Beijing, China.","DOI":"10.1109\/IGARSS.2016.7729610"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"6058","DOI":"10.1109\/TGRS.2015.2431498","article-title":"Automatic optical-to-SAR image registration by iterative line extraction and voronoi integrated spectral point matching","volume":"53","author":"Sui","year":"2015","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2881","DOI":"10.1109\/JSTARS.2018.2844295","article-title":"High-resolution remote-sensing image registration based on angle matching of edge point features","volume":"11","author":"Guo","year":"2018","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Zbontar, J., and LeCun, Y. (2015, January 7\u201312). Computing the stereo matching cost with a convolutional neural network. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298767"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Suarez, P.L., Sappa, A.D., and Vintimilla, B.X. (2017, January 24\u201326). Cross-Spectral image patch similarity using convolutional neural network. Proceedings of the 2017 IEEE International Workshop of Electronics, Control, Measurement, Signals and Their Application to Mechatronics (ECMSM), San Sebastian, Spain.","DOI":"10.1109\/ECMSM.2017.7945888"},{"key":"ref_25","unstructured":"Jahrer, M., Grabner, M., and Bischof, H. (2008, January 4\u20136). Learned local descriptors for recognition and matching. Proceedings of the Compute Vision Winter Workshop (CVWW), Moravske Toplice, Slovenija."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"He, H.Q., Chen, M., Chen, T., and Li, D.J. (2018). Matching of remote sensing images with complex background variations via Siamese convolutional neural network. Remote Sens., 10.","DOI":"10.3390\/rs10020355"},{"key":"ref_27","unstructured":"Han, X.F., Leung, T., Jia, Y.Q., Sukthankar, R., and Berg, A.C. (2015, January 7\u201312). MatchNet: Unifying feature and metric learning for patch-based matching. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"516","DOI":"10.1080\/2150704X.2019.1577572","article-title":"Learning to match multitemporal optical satellite images using multi-support-patches Siamese networks","volume":"10","author":"He","year":"2019","journal-title":"Remote Sens. Lett."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Zagoruyko, S., and Komodakis, N. (2015, January 7\u201312). Learning to compare image patches via convolutional neural networks. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7299064"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Liu, W., Xuelun, S., Cheng, W., Zhihong, Z., Chenglu, W., and Jonathan, L. (2018, January 13\u201319). H-Net: Neural network for cross-domain image patch matching. Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI), Stockholm, Sweden.","DOI":"10.24963\/ijcai.2018\/119"},{"key":"ref_31","unstructured":"Aguilera, C.A., Aguilera, F.J., Sappa, A.D., Aguilera, C., and Toledo, R. (July, January 26). Learning cross-spectral similarity measures with deep convolutional neural networks. Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Saxena, S., and Verbeek, J. (2016, January 8\u201316). Heterogeneous face recognition with CNNs. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-49409-8_40"},{"key":"ref_33","first-page":"51","article-title":"Correspondence matching in unorganized 3D point clouds using Convolutional Neural Networks","volume":"83","author":"Alba","year":"2019","journal-title":"Image Vis. Comput."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"e1700578","DOI":"10.1126\/sciadv.1700578","article-title":"Convolutional neural network for earthquake detection and location","volume":"4","author":"Perol","year":"2018","journal-title":"Sci. Adv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"En, S., Lechervy, A., and Jurie, F. (2018, January 7\u201310). TS-NET: Combing modality specific and common features for multimodal patch matching. Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece.","DOI":"10.1109\/ICIP.2018.8451804"},{"key":"ref_36","unstructured":"Baruch, E.B., and Keller, Y. (2018). Multimodal matching using a Hybrid Convolutional Neural Network. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1007\/978-3-319-24261-3_7","article-title":"Deep metric learning using triplet network","volume":"Volume 9370","author":"Feragen","year":"2015","journal-title":"Similarity-Based Pattern Recognition, Simbad 2015"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Aguilera, C.A., Sappa, A.D., Aguilera, C., and Toledo, R. (2017). Cross-spectral local descriptors via quadruplet network. Sensors, 17.","DOI":"10.20944\/preprints201703.0061.v1"},{"key":"ref_39","first-page":"2","article-title":"Stereo matching by training a convolutional neural network to compare image patches","volume":"17","author":"Jure","year":"2015","journal-title":"Comput. Sci."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1016\/j.isprsjprs.2017.12.012","article-title":"A deep learning framework for remote sensing image registration","volume":"145","author":"Wang","year":"2018","journal-title":"ISPRS J. Potogramm."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"He, T., Zhang, Z., and Zhang, H. (2018). Bag of tricks for image classification with convolutional neural networks. arXiv.","DOI":"10.1109\/CVPR.2019.00065"},{"key":"ref_42","first-page":"249","article-title":"Understanding the difficulty of training deep feedforward neural networks","volume":"9","author":"Glorot","year":"2010","journal-title":"J. Mach. Learn Res."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (July, January 26). Rethinking the inception architecture for computer vision. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.308"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"3028","DOI":"10.1109\/JSTARS.2019.2916560","article-title":"Registration of multimodal remote sensing image based on deep fully convolutional neural network","volume":"12","author":"Han","year":"2019","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Bay, H., Tuytelaars, T., and Gool, L.V. (2006, January 7\u201313). SURF: Speeded up robust features. Proceedings of the 9th European Conference on Computer Vision (ECCV), Graz, Austria.","DOI":"10.1007\/11744023_32"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"722","DOI":"10.1109\/TPAMI.2008.300","article-title":"LSD: A fast line segment detector with a false detection control","volume":"32","author":"Gioi","year":"2010","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Brown, M., and Susstrunk, S. (2011, January 20\u201325). Multi-spectral sift for scene category recognition. Proceedings of the 24th Conference on Compute Vision and Pattern Recognition (CVPR), Providence, RI, USA.","DOI":"10.1109\/CVPR.2011.5995637"},{"key":"ref_48","unstructured":"Shao, X. (2013, January 18\u201320). A fast matching algorithm with feature points based on NCC. Proceedings of the 2013 International Academic Workshop on Social Science (IAW-SC), Changsha, China."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image quality assessment: From error visibility to structural similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Process."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/11\/23\/2836\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:38:35Z","timestamp":1760189915000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/11\/23\/2836"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11,29]]},"references-count":49,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2019,12]]}},"alternative-id":["rs11232836"],"URL":"https:\/\/doi.org\/10.3390\/rs11232836","relation":{},"ISSN":["2072-4292"],"issn-type":[{"type":"electronic","value":"2072-4292"}],"subject":[],"published":{"date-parts":[[2019,11,29]]}}}