{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:52:52Z","timestamp":1760143972685,"version":"build-2065373602"},"reference-count":36,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2024,3,12]],"date-time":"2024-03-12T00:00:00Z","timestamp":1710201600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Union Horizon 2020 Research and Innovation Programme","award":["825355"],"award-info":[{"award-number":["825355"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>There are several image inverse tasks, such as inpainting or super-resolution, which can be solved using deep internal learning, a paradigm that involves employing deep neural networks to find a solution by learning from the sample itself rather than a dataset. For example, Deep Image Prior is a technique based on fitting a convolutional neural network to output the known parts of the image (such as non-inpainted regions or a low-resolution version of the image). However, this approach is not well adjusted for samples composed of multiple modalities. In some domains, such as satellite image processing, accommodating multi-modal representations could be beneficial or even essential. In this work, Multi-Modal Convolutional Parameterisation Network (MCPN) is proposed, where a convolutional neural network approximates shared information between multiple modes by combining a core shared network with modality-specific head networks. The results demonstrate that these approaches can significantly outperform the single-mode adoption of a convolutional parameterisation network on guided image inverse problems of inpainting and super-resolution.<\/jats:p>","DOI":"10.3390\/jimaging10030069","type":"journal-article","created":{"date-parts":[[2024,3,12]],"date-time":"2024-03-12T03:55:34Z","timestamp":1710215734000},"page":"69","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Multi-Modal Convolutional Parameterisation Network for Guided Image Inverse Problems"],"prefix":"10.3390","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0927-0416","authenticated-orcid":false,"given":"Mikolaj","family":"Czerkawski","sequence":"first","affiliation":[{"name":"Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G1 1XW, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6212-5314","authenticated-orcid":false,"given":"Priti","family":"Upadhyay","sequence":"additional","affiliation":[{"name":"Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G1 1XW, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9450-1791","authenticated-orcid":false,"given":"Christopher","family":"Davison","sequence":"additional","affiliation":[{"name":"Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G1 1XW, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6206-2229","authenticated-orcid":false,"given":"Robert","family":"Atkinson","sequence":"additional","affiliation":[{"name":"Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G1 1XW, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Craig","family":"Michie","sequence":"additional","affiliation":[{"name":"Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G1 1XW, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9093-5245","authenticated-orcid":false,"given":"Ivan","family":"Andonovic","sequence":"additional","affiliation":[{"name":"Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G1 1XW, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4499-4281","authenticated-orcid":false,"given":"Malcolm","family":"Macdonald","sequence":"additional","affiliation":[{"name":"Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G1 1XW, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9284-1899","authenticated-orcid":false,"given":"Javier","family":"Cardona","sequence":"additional","affiliation":[{"name":"Department of Chemical Engineering, University of Strathclyde, Glasgow G1 1XJ, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9150-6805","authenticated-orcid":false,"given":"Christos","family":"Tachtatzis","sequence":"additional","affiliation":[{"name":"Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G1 1XW, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,3,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Gandelsman, Y., Shocher, A., and Irani, M. (2019, January 15\u201320). \u201cDouble-DIP\u201d: Unsupervised Image Decomposition via Coupled Deep-Image-Priors. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.01128"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1867","DOI":"10.1007\/s11263-020-01303-4","article-title":"Deep Image Prior","volume":"128","author":"Ulyanov","year":"2020","journal-title":"Int. J. Comput. Vis."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1109\/MGRS.2016.2637824","article-title":"Hyperspectral and Multispectral Data Fusion: A comparative review of the recent literature","volume":"5","author":"Yokoya","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1656","DOI":"10.1109\/JSTARS.2018.2805923","article-title":"Remote Sensing Image Fusion With Deep Convolutional Neural Network","volume":"11","author":"Shao","year":"2018","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"5","DOI":"10.5194\/isprs-annals-IV-1-5-2018","article-title":"SAR to Optical Image Synthesis for Cloud Removal with Generative Adversarial Networks","volume":"4","author":"Bermudez","year":"2018","journal-title":"ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Singh, P., and Komodakis, N. (2018, January 22\u201327). Cloud-Gan: Cloud Removal for Sentinel-2 Imagery Using a Cyclic Consistent Generative Adversarial Networks. Proceedings of the IGARSS 2018 IEEE International Geoscience and Remote Sensing Symposium, Valencia, Spain.","DOI":"10.1109\/IGARSS.2018.8519033"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1016\/j.isprsjprs.2020.05.013","article-title":"Cloud removal in Sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion","volume":"166","author":"Meraner","year":"2020","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Isola, P., Zhu, J.Y., Zhou, T., and Efros, A.A. (2017, January 21\u201326). Image-to-Image Translation with Conditional Adversarial Networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.632"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1109\/MGRS.2015.2441912","article-title":"Missing Information Reconstruction of Remote Sensing Data: A Technical Review","volume":"3","author":"Shen","year":"2015","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"4274","DOI":"10.1109\/TGRS.2018.2810208","article-title":"Missing data reconstruction in remote sensing image with a unified spatial-temporal-spectral deep convolutional neural network","volume":"56","author":"Zhang","year":"2018","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Kang, S.H., Choi, Y., and Choi, J.Y. (2021). Restoration of missing patterns on satellite infrared sea surface temperature images due to cloud coverage using deep generative inpainting network. J. Mar. Sci. Eng., 9.","DOI":"10.3390\/jmse9030310"},{"key":"ref_12","first-page":"102599","article-title":"Unsupervised missing information reconstruction for single remote sensing image with Deep Code Regression","volume":"105","author":"Gao","year":"2021","journal-title":"Int. J. Appl. Earth Obs. Geoinf."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Ebel, P., Schmitt, M., and Zhu, X.X. (2021, January 11\u201316). Internal Learning for Sequence-to-Sequence Cloud Removal via Synthetic Aperture Radar Prior Information. Proceedings of the 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, Brussels, Belgium.","DOI":"10.1109\/IGARSS47720.2021.9554268"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"5222414","DOI":"10.1109\/TGRS.2022.3146246","article-title":"SEN12MS-CR-TS: A Remote Sensing Data Set for Multi-modal Multi-temporal Cloud Removal","volume":"60","author":"Ebel","year":"2022","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1364\/OPTCON.439671","article-title":"Remote sensing image cloud removal by deep image prior with a multitemporal constraint","volume":"1","author":"Zhang","year":"2022","journal-title":"Opt. Contin."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Czerkawski, M., Upadhyay, P., Davison, C., Werkmeister, A., Cardona, J., Atkinson, R., Michie, C., Andonovic, I., Macdonald, M., and Tachtatzis, C. (2022). Deep Internal Learning for Inpainting of Cloud-Affected Regions in Satellite Imagery. Remote Sens., 14.","DOI":"10.3390\/rs14061342"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"2378","DOI":"10.1109\/TIP.2011.2109730","article-title":"FSIM: A Feature Similarity Index for Image Quality Assessment","volume":"20","author":"Zhang","year":"2011","journal-title":"IEEE Trans. Image Process."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1185","DOI":"10.1109\/TIP.2010.2092435","article-title":"Information Content Weighting for Perceptual Image Quality Assessment","volume":"20","author":"Wang","year":"2011","journal-title":"IEEE Trans. Image Process."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Golestaneh, S.A., Dadsetan, S., and Kitani, K.M. (2022, January 3\u20138). No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistency. Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.","DOI":"10.1109\/WACV51458.2022.00404"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"3474","DOI":"10.1109\/TIP.2021.3061932","article-title":"Uncertainty-aware blind image quality assessment in the laboratory and wild","volume":"30","author":"Zhang","year":"2021","journal-title":"IEEE Trans. Image Process."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7\u201312). Going deeper with convolutions. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Shaham, T.R., Dekel, T., and Michaeli, T. (November, January 27). SinGAN: Learning a Generative Model From a Single Natural Image. Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.","DOI":"10.1109\/ICCV.2019.00467"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Upadhyay, P., Czerkawski, M., Davison, C., Cardona, J., Macdonald, M., Andonovic, I., Michie, C., Atkinson, R., Papadopoulou, N., and Nikas, K. (2022). A Flexible Multi-Temporal and Multi-Modal Framework for Sentinel-1 and Sentinel-2 Analysis Ready Data. Remote Sens., 14.","DOI":"10.3390\/rs14051120"},{"key":"ref_24","first-page":"121","article-title":"Satellite image inpainting with deep generative adversarial neural networks","volume":"10","author":"Zaytar","year":"2021","journal-title":"IAES Int. J. Artif. Intell."},{"key":"ref_25","first-page":"6410","article-title":"Image Inpainting with Bilateral Convolution","volume":"14","author":"Deng","year":"2022","journal-title":"Remote Sens."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Suvorov, R., Logacheva, E., Mashikhin, A., Remizova, A., Ashukha, A., Silvestrov, A., Kong, N., Goka, H., Park, K., and Lempitsky, V. (2022, January 3\u20138). Resolution-robust Large Mask Inpainting with Fourier Convolutions. Proceedings of the 2022 IEEE\/CVF Winter Conference on Applications of Computer Vision, WACV 2022, Waikoloa, HI, USA.","DOI":"10.1109\/WACV51458.2022.00323"},{"key":"ref_27","first-page":"1","article-title":"Palette: Image-to-Image Diffusion Models","volume":"1","author":"Saharia","year":"2022","journal-title":"Proc. ACM SIGGRAPH"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., and Van Gool, L. (2022, January 18\u201324). RePaint: Inpainting using Denoising Diffusion Probabilistic Models. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01117"},{"key":"ref_29","unstructured":"Conover, W.J. (1999). Practical Nonparametric Statistics, Wiley. [3rd ed.]."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"He, K., Sun, J., and Tang, X. (2010, January 5\u201311). Guided Image Filtering. Proceedings of the 11th European Conference on Computer Vision, Crete, Greece.","DOI":"10.1007\/978-3-642-15549-9_1"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Barron, J.T., and Poole, B. (2016, January 11\u201314). The Fast Bilateral Solver. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46487-9_38"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Hui, T.W., Loy, C.C., and Tang, X. (2016, January 11\u201314). Depth Map Super-Resolution by Deep Multi-Scale Guidance. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46487-9_22"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1109\/TPAMI.2017.2669034","article-title":"Robust Guided Image Filtering Using Nonconvex Potentials","volume":"40","author":"Ham","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_34","unstructured":"Lutio, R.D., D\u2019Aronco, S., Wegner, J.D., and Schindler, K. (November, January 27). Guided super-resolution as pixel-to-pixel transformation. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Republic of Korea."},{"key":"ref_35","unstructured":"Lutio, R.D., Becker, A., DAronco, S., Russo, S., Wegner, J.D., and Schindler, K. (2022, January 18\u201324). Learning Graph Regularisation for Guided Super-Resolution. Proceedings of the 2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Lim, B., Son, S., Kim, H., Nah, S., and Lee, K. (2017, January 21\u201326). Enhanced Deep Residual Networks for Single Image Super-Resolution. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Los Alamitos, CA, USA.","DOI":"10.1109\/CVPRW.2017.151"}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/10\/3\/69\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:12:14Z","timestamp":1760105534000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/10\/3\/69"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,12]]},"references-count":36,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,3]]}},"alternative-id":["jimaging10030069"],"URL":"https:\/\/doi.org\/10.3390\/jimaging10030069","relation":{},"ISSN":["2313-433X"],"issn-type":[{"type":"electronic","value":"2313-433X"}],"subject":[],"published":{"date-parts":[[2024,3,12]]}}}