{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,29]],"date-time":"2026-03-29T16:06:55Z","timestamp":1774800415741,"version":"3.50.1"},"reference-count":46,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2018,5,25]],"date-time":"2018-05-25T00:00:00Z","timestamp":1527206400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>The accurate ground-based cloud classification is a challenging task and still under development. The most current methods are limited to only taking the cloud visual features into consideration, which is not robust to the environmental factors. In this paper, we present the novel joint fusion convolutional neural network (JFCNN) to integrate the multimodal information for ground-based cloud classification. To learn the heterogeneous features (visual features and multimodal features) from the ground-based cloud data, we designed the proposed JFCNN as a two-stream structure which contains the vision subnetwork and multimodal subnetwork. We also proposed a novel layer named joint fusion layer to jointly learn two kinds of cloud features under one framework. After training the proposed JFCNN, we extracted the visual and multimodal features from the two subnetworks and integrated them using a weighted strategy. The proposed JFCNN was validated on the multimodal ground-based cloud (MGC) dataset and achieved remarkable performance, demonstrating its effectiveness for ground-based cloud classification task.<\/jats:p>","DOI":"10.3390\/rs10060822","type":"journal-article","created":{"date-parts":[[2018,5,28]],"date-time":"2018-05-28T03:54:21Z","timestamp":1527479661000},"page":"822","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":46,"title":["Multimodal Ground-Based Cloud Classification Using Joint Fusion Convolutional Neural Network"],"prefix":"10.3390","volume":"10","author":[{"given":"Shuang","family":"Liu","sequence":"first","affiliation":[{"name":"Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin 300387, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mei","family":"Li","sequence":"additional","affiliation":[{"name":"Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin 300387, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2993-8612","authenticated-orcid":false,"given":"Zhong","family":"Zhang","sequence":"additional","affiliation":[{"name":"Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin 300387, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3941-1141","authenticated-orcid":false,"given":"Baihua","family":"Xiao","sequence":"additional","affiliation":[{"name":"The State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaozhong","family":"Cao","sequence":"additional","affiliation":[{"name":"Meteorological Observation Centre, China Meteorological Administration, Beijing 100081, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2018,5,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Tan, K., Zhang, Y., and Tong, X. (2016). Cloud extraction from chinese high resolution satellite imagery by probabilistic latent semantic analysis and object-based machine learning. Remote Sens., 8.","DOI":"10.3390\/rs8110963"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1109\/36.981353","article-title":"Unsupervised segmentation of low clouds from infrared METEOSAT images based on a contextual spatio-temporal labeling approach","volume":"40","author":"Papin","year":"2002","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1356","DOI":"10.1109\/JSTARS.2012.2201449","article-title":"On the use of a cluster ensemble cloud classification technique in satellite precipitation estimation","volume":"5","author":"Mahrooghy","year":"2012","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Holdaway, D., and Yang, Y. (2016). Study of the effect of temporal sampling frequency on DSCOVR observations using the GEOS-5 nature run results (Part II): Cloud Coverage. Remote Sens., 8.","DOI":"10.3390\/rs8050431"},{"key":"ref_5","first-page":"941","article-title":"Cloud mapping from the ground: Use of photogrammetric methods","volume":"68","author":"Seiz","year":"2002","journal-title":"Photogramm. Eng. Remote Sens."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1221","DOI":"10.1175\/JAM2277.1","article-title":"Cloud-base-height estimation from paired ground-based hemispherical observations","volume":"44","author":"Kassianov","year":"2005","journal-title":"J. Appl. Meteorol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1421","DOI":"10.1175\/1520-0450(2003)042<1421:CCBOAI>2.0.CO;2","article-title":"Cloud coverage based on all-sky imaging and its impact on surface solar irradiance","volume":"42","author":"Pfister","year":"2003","journal-title":"J. Appl. Meteorol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1127\/0941-2948\/2008\/0321","article-title":"Estimation of the total cloud cover with high temporal resolution and parametrization of short-term fluctuations of sea surface insolation","volume":"17","author":"Kalisch","year":"2008","journal-title":"Meteorol. Z."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1970","DOI":"10.1016\/j.egypro.2015.03.198","article-title":"Cloud detection methodology based on a sky-imaging system","volume":"69","author":"Chauvin","year":"2015","journal-title":"Energy Procedia"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"392","DOI":"10.1016\/j.solener.2012.11.015","article-title":"Equipment and methodologies for cloud detection and classification: A review","volume":"95","author":"Tapakis","year":"2013","journal-title":"Sol. Energy"},{"key":"ref_11","unstructured":"Shields, J.E., Karr, M.E., Tooman, T.P., Sowle, D.H., and Moore, S.T. (1998, January 23\u201327). The Whole Sky Imager-a Year of Progress. Proceedings of the Atmospheric Radiation Measurement Science Team Meeting, Tucson, AZ, USA."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"633","DOI":"10.1175\/JTECH1875.1","article-title":"Retrieving cloud characteristics from ground-based daytime color all-sky images","volume":"23","author":"Long","year":"2006","journal-title":"J. Atmos. Ocean. Technol."},{"key":"ref_13","unstructured":"Thurairajah, B. (2004). Thermal Infrared Imaging of the Atmosphere: The Infrared Cloud Imager. [Ph.D. Thesis, Montana State University]."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1364\/JOSAA.25.000029","article-title":"Development of a sky imager for cloud cover assessment","volume":"25","author":"Cazorla","year":"2008","journal-title":"J. Opt. Soc. Am. A"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"54","DOI":"10.3724\/SP.J.1010.2009.00054","article-title":"Vicarious calibration on the sensor of whole sky infrared cloud measuring system","volume":"28","author":"Sun","year":"2009","journal-title":"J. Infrared. Millim. Waves"},{"key":"ref_16","unstructured":"Buch, K.A., Sun, C.H., and Thorne, L.R. (1995, January 4\u20137). Cloud Classification Using Whole-Sky Imager Data. Proceedings of the 5th Atmospheric Radiation Measurement Science Team Meeting, San Diego, CA, USA."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"557","DOI":"10.5194\/amt-3-557-2010","article-title":"Automatic cloud classification of whole sky images","volume":"3","author":"Heinle","year":"2010","journal-title":"Atmos. Meas. Technol."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"410","DOI":"10.1175\/2010JTECHA1385.1","article-title":"Cloud classification based on structure features of infrared images","volume":"28","author":"Liu","year":"2011","journal-title":"J. Atmos. Ocean. Technol."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"258","DOI":"10.1007\/s10044-005-0007-5","article-title":"Automated ground-based cloud recognition","volume":"8","author":"Singh","year":"2005","journal-title":"Pattern Anal. Appl."},{"key":"ref_20","unstructured":"Liu, S., Wang, C., Xiao, B., Zhang, Z., and Shao, Y. (2012, January 16\u201318). Ground-Based cloud Classification Using Multiple Random Projections. Proceedings of the International Conference on Computer Vision in Remote Sensing, Xiamen, China."},{"key":"ref_21","unstructured":"Liu, S., Wang, C., Xiao, B., Zhang, Z., and Shao, Y. (2012, January 11\u201315). Soft-Signed Sparse Coding for Ground-Based cloud Classification. Proceedings of the International Conference on Pattern Recognition, Tsukuba, Japan."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s13351-013-0206-8","article-title":"Salient local binary pattern for ground-based cloud classification","volume":"27","author":"Liu","year":"2013","journal-title":"Acta. Meteorol. Sin."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1422","DOI":"10.1587\/transinf.2014EDL8252","article-title":"Learning Discriminative Features for Ground-Based Cloud Classification via Mutual Information Maximization","volume":"98","author":"Liu","year":"2015","journal-title":"IEICE Trans. Inf. Syst."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1175\/JTECH-D-13-00048.1","article-title":"Cloud classification of ground-based images using texture\u2013structure features","volume":"31","author":"Zhuo","year":"2014","journal-title":"J. Atmos. Ocean. Technol."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"789","DOI":"10.1175\/JTECH-D-15-0015.1","article-title":"mCLOUD: A multiview visual feature extraction mechanism for ground-based cloud image categorization","volume":"33","author":"Xiao","year":"2016","journal-title":"J. Atmos. Ocean. Technol."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1023\/B:VISI.0000029664.99615.94","article-title":"Distinctive image features from scale-invariant keypoints","volume":"60","author":"Lowe","year":"2004","journal-title":"Int. J. Comput. Vis."},{"key":"ref_27","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20138). Imagenet Classification with Deep Convolutional Neural Networks. Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Dong, C., Loy, C.C., He, K., and Tang, X. (2014, January 6\u201312). Learning a Deep Convolutional Network for Image Super-Resolution. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10593-2_13"},{"key":"ref_29","unstructured":"Dai, J., Li, Y., He, K., and Sun, J. (2016, January 5\u201310). R-FCN: Object Detection via Region-based Fully Convolutional Networks. Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1038\/nature21056","article-title":"Dermatologist-level classification of skin cancer with deep neural networks","volume":"542","author":"Esteva","year":"2017","journal-title":"Nature"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1109\/MGRS.2016.2540798","article-title":"Deep learning for remote sensing data: A technical tutorial on the state of the art","volume":"4","author":"Zhang","year":"2016","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"7092","DOI":"10.1109\/TGRS.2017.2740362","article-title":"High-resolution aerial image labeling with convolutional neural networks","volume":"55","author":"Maggiori","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"5729","DOI":"10.1109\/TGRS.2017.2712809","article-title":"DeepCloud: Ground-based cloud image categorization using deep convolutional features","volume":"55","author":"Ye","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"816","DOI":"10.1109\/LGRS.2017.2681658","article-title":"Deep Convolutional Activations-Based Features for Ground-Based Cloud Classification","volume":"14","author":"Shi","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Chen, Q., Zhang, G., Yang, X., Li, S., Li, Y., and Wang, H.H. (2017). Single image shadow detection and removal based on feature fusion and multiple dictionary learning. Multimed. Tools Appl., 1\u201324.","DOI":"10.1007\/s11042-017-5299-0"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Tharwat, A., Gaber, T., Awad, Y.M., Dey, N., and Hassanien, A.E. (2016, January 24\u201326). Plants Identification Using Feature Fusion Technique and Bagging Classifier. Proceedings of the International Conference on Advanced Intelligent System and Informatics, Beni Suef, Egypt.","DOI":"10.1007\/978-3-319-26690-9_41"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1016\/j.knosys.2017.03.014","article-title":"Distributed incremental fingerprint identification with reduced database penetration rate using a hierarchical classification based on feature fusion and selection","volume":"126","author":"Peralta","year":"2017","journal-title":"Knowl. Based. Syst."},{"key":"ref_38","unstructured":"Park, T., and Lee, T. (arXiv, 2015). Musical instrument sound classification with deep convolutional neural network using feature fusion approach, arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Yang, X., Yumer, E., Asente, P., Kraley, M., Kifer, D., and Giles, C.L. (arXiv, 2017). Learning to extract semantic structure from documents using multimodal fully convolutional neural networks, arXiv.","DOI":"10.1109\/CVPR.2017.462"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1369","DOI":"10.1016\/S0031-3203(02)00262-5","article-title":"Feature fusion: parallel strategy vs. serial strategy","volume":"36","author":"Yang","year":"2003","journal-title":"Pattern Recogn."},{"key":"ref_41","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_42","first-page":"1929","article-title":"Dropout: A simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1908","DOI":"10.1109\/TIP.2010.2045169","article-title":"Semantics-preserving bag-of-words models and applications","volume":"19","author":"Wu","year":"2010","journal-title":"IEEE Trans. Image Process."},{"key":"ref_44","unstructured":"Lazebnik, S., Schmid, C., and Ponce, J. (2006, January 17\u201322). Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, New York, NY, USA."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"971","DOI":"10.1109\/TPAMI.2002.1017623","article-title":"Multiresolution gray-scale and rotation invariant texture classification with local binary patterns","volume":"24","author":"Ojala","year":"2002","journal-title":"IEEE Trans. Pattern Anal."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1657","DOI":"10.1109\/TIP.2010.2044957","article-title":"A completed modeling of local binary pattern operator for texture classification","volume":"19","author":"Guo","year":"2010","journal-title":"IEEE Trans. Image Process."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/10\/6\/822\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T15:05:52Z","timestamp":1760195152000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/10\/6\/822"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,5,25]]},"references-count":46,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2018,6]]}},"alternative-id":["rs10060822"],"URL":"https:\/\/doi.org\/10.3390\/rs10060822","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,5,25]]}}}