{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,18]],"date-time":"2026-02-18T04:32:18Z","timestamp":1771389138482,"version":"3.50.1"},"reference-count":60,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2020,11,29]],"date-time":"2020-11-29T00:00:00Z","timestamp":1606608000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>In this paper we introduce a novel machine learning-based fully automatic approach for the semantic analysis and documentation of masonry wall images, performing in parallel automatic detection and virtual completion of occluded or damaged wall regions, and brick segmentation leading to an accurate model of the wall structure. For this purpose, we propose a four-stage algorithm which comprises three interacting deep neural networks and a watershed transform-based brick outline extraction step. At the beginning, a U-Net-based sub-network performs initial wall segmentation into brick, mortar and occluded regions, which is followed by a two-stage adversarial inpainting model. The first adversarial network predicts the schematic mortar-brick pattern of the occluded areas based on the observed wall structure, providing in itself valuable structural information for archeological and architectural applications. The second adversarial network predicts the pixels\u2019 color values yielding a realistic visual experience for the observer. Finally, using the neural network outputs as markers in a watershed-based segmentation process, we generate the accurate contours of the individual bricks, both in the originally visible and in the artificially inpainted wall regions. Note that while the first three stages implement a sequential pipeline, they interact through dependencies of their loss functions admitting the consideration of hidden feature dependencies between the different network components. For training and testing the network a new dataset has been created, and an extensive qualitative and quantitative evaluation versus the state-of-the-art is given. The experiments confirmed that the proposed method outperforms the reference techniques both in terms of wall structure estimation and regarding the visual quality of the inpainting step, moreover it can be robustly used for various different masonry wall types.<\/jats:p>","DOI":"10.3390\/rs12233918","type":"journal-article","created":{"date-parts":[[2020,11,29]],"date-time":"2020-11-29T21:00:57Z","timestamp":1606683657000},"page":"3918","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":25,"title":["Deep Learning-Based Masonry Wall Image Analysis"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0131-985X","authenticated-orcid":false,"given":"Yahya","family":"Ibrahim","sequence":"first","affiliation":[{"name":"3in-PPCU Research Group, P\u00e9ter P\u00e1zm\u00e1ny Catholic University, H-2500 Esztergom, Hungary"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0338-863X","authenticated-orcid":false,"given":"Bal\u00e1zs","family":"Nagy","sequence":"additional","affiliation":[{"name":"3in-PPCU Research Group, P\u00e9ter P\u00e1zm\u00e1ny Catholic University, H-2500 Esztergom, Hungary"},{"name":"Institute for Computer Science and Control (SZTAKI), H-1111 Budapest, Hungary"},{"name":"Faculty of Informatics, University of Debrecen, 4028 Debrecen, Hungary"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3203-0741","authenticated-orcid":false,"given":"Csaba","family":"Benedek","sequence":"additional","affiliation":[{"name":"3in-PPCU Research Group, P\u00e9ter P\u00e1zm\u00e1ny Catholic University, H-2500 Esztergom, Hungary"},{"name":"Institute for Computer Science and Control (SZTAKI), H-1111 Budapest, Hungary"},{"name":"Faculty of Informatics, University of Debrecen, 4028 Debrecen, Hungary"}]}],"member":"1968","published-online":{"date-parts":[[2020,11,29]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Pierdicca, R., Paolanti, M., Matrone, F., Martini, M., Morbidoni, C., Malinverni, E.S., Frontoni, E., and Lingua, A.M. (2020). Point Cloud Semantic Segmentation Using a Deep Learning Framework for Cultural Heritage. Remote Sens., 12.","DOI":"10.3390\/rs12061005"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Gallwey, J., Eyre, M., Tonkins, M., and Coggan, J. (2019). Bringing Lunar LiDAR Back Down to Earth: Mapping Our Industrial Heritage through Deep Transfer Learning. Remote Sens., 11.","DOI":"10.3390\/rs11171994"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"de Lima-Hernandez, R., and Vergauwen, M. (2020). A Hybrid Approach to Reassemble Ancient Decorated Block Fragments through a 3D Puzzling Engine. Remote Sens., 12.","DOI":"10.3390\/rs12162526"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Teruggi, S., Grilli, E., Russo, M., Fassi, F., and Remondino, F. (2020). A Hierarchical Machine Learning Approach for Multi-Level and Multi-Resolution 3D Point Cloud Classification. Remote Sens., 12.","DOI":"10.3390\/rs12162598"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1007\/s00138-016-0819-x","article-title":"Image-based search and retrieval for biface artefacts using features capturing archaeologically significant characteristics","volume":"28","author":"Eramian","year":"2016","journal-title":"Mach. Vis. Appl."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"642","DOI":"10.1007\/s11036-016-0805-6","article-title":"Mobile Application for Archaeological Site Image Content Retrieval and Automated Generating Image Descriptions with Neural Network","volume":"22","author":"Prasomphan","year":"2017","journal-title":"Mob. Netw. Appl."},{"key":"ref_7","unstructured":"van der Maaten, L., Boon, P., Lange, G., Paijmans, J., and Postma, E. (2020, November 27). Computer Vision and Machine Learning for Archaeology. Available online: https:\/\/publikationen.uni-tuebingen.de\/xmlui\/bitstream\/handle\/10900\/61550\/CD49_Maaten_et_al_CAA2006.pdf."},{"key":"ref_8","first-page":"883","article-title":"Classification and reconstruction algorithms for the archaeological fragments","volume":"32","author":"Rasheed","year":"2018","journal-title":"J. King Saud Univ.-Comput. Inf. Sci."},{"key":"ref_9","first-page":"331","article-title":"4D reconstruction of the past: The image retrieval and 3D model construction pipeline","volume":"Volume 9229","author":"Hadjimitsis","year":"2014","journal-title":"Proceedings of the Second International Conference on Remote Sensing and Geoinformation of the Environment (RSCy2014)"},{"key":"ref_10","first-page":"78","article-title":"Stability Analysis of Masonry Structure in Angkor Ruin Considering the Construction Quality of the Foundation","volume":"4","author":"Hashimoto","year":"2014","journal-title":"J. Civ. Eng. Res."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"F\u0131rat, S., Kinuthia, J., and Abu-Tair, A. (2017, January 15\u201317). Structural Stability Analysis of Large-Scale Masonry Historic City Walls. Proceedings of the 3rd International Sustainable Buildings Symposium (ISBS 2017), Dubai, UAE.","DOI":"10.1007\/978-3-319-63709-9"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Shen, Y., Lindenbergh, R., Wang, J., and Ferreira, V.G. (2018). Extracting Individual Bricks from a Laser Scan Point Cloud of an Unorganized Pile of Bricks. Remote Sens., 10.","DOI":"10.3390\/rs10111709"},{"key":"ref_13","first-page":"227","article-title":"Damage detection of concrete masonry structures by enhancing deformation measurement using DIC","volume":"Volume 9804","author":"Yu","year":"2016","journal-title":"Nondestructive Characterization and Monitoring of Advanced Materials, Aerospace, and Civil Infrastructure 2016"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"98","DOI":"10.21660\/2019.59.8272","article-title":"Damage Detection and Localization in Masonry Structure Using Faster Region Convolutional Networks","volume":"17","author":"Ali","year":"2019","journal-title":"Int. J. GEOMATE"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Aguilar, R., Torrealva, D., Moreira, S., Pando, M.A., and Ramos, L.F. (2019). Historic Digital Survey: Reality Capture and Automatic Data Processing for the Interpretation and Analysis of Historic Architectural Rubble Masonry. Structural Analysis of Historical Constructions, Springer.","DOI":"10.1007\/978-3-319-99441-3"},{"key":"ref_16","first-page":"234","article-title":"U-Net: Convolutional Networks for Biomedical Image Segmentation","volume":"Volume 9351","author":"Ronneberger","year":"2015","journal-title":"Medical Image Computing and Computer-Assisted Intervention"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Suchocki, C., Dami\u0119cka-Suchocka, M., Katzer, J., Janicka, J., Rapi\u0144ski, J., and Sta\u0142owska, P. (2020). Remote Detection of Moisture and Bio-Deterioration of Building Walls by Time-Of-Flight and Phase-Shift Terrestrial Laser Scanners. Remote Sens., 12.","DOI":"10.3390\/rs12111708"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Duan, C., Pan, J., and Li, R. (2020). Thick Cloud Removal of Remote Sensing Images Using Temporal Smoothness and Sparsity Regularized Tensor Optimization. Remote Sens., 12.","DOI":"10.3390\/rs12203446"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Dai, P., Ji, S., and Zhang, Y. (2020). Gated Convolutional Networks for Cloud Removal From Bi-Temporal Remote Sensing Images. Remote Sens., 12.","DOI":"10.3390\/rs12203427"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Ibrahim, Y., Nagy, B., and Benedek, C. (2019, January 27\u201329). CNN-Based watershed Marker Extraction for Brick Segmentation in Masonry Walls. Proceedings of the International Conference Image Analysis and Recognition, Waterloo, ON, Canada.","DOI":"10.1007\/978-3-030-27202-9_30"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Ibrahim, Y., Nagy, B., and Benedek, C. (2021, January 10\u201315). A GAN-based Blind Inpainting Method for Masonry Wall Images. Proceedings of the International Conference on Pattern Recognition, Milan, Italy.","DOI":"10.1109\/ICPR48806.2021.9413009"},{"key":"ref_22","first-page":"567","article-title":"Detection of bricks in a masonry wall","volume":"XXXVII","author":"Sithole","year":"2008","journal-title":"Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci."},{"key":"ref_23","unstructured":"Hemmleb, M., Weritz A, F., Schiemenz B, A., Grote C, A., and Maierhofer, C. (2006, January 25\u201327). Multi-spectral data acquisition and processing techniques for damage detection on building surfaces. Proceedings of the ISPRS Commission V Symposium, Dresden, Germany."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"3","DOI":"10.5194\/isprsannals-II-3-W5-3-2015","article-title":"Automatic creation of structural models from point cloud data: The case of masonry structures","volume":"II-3\/W5","author":"Riveiro","year":"2015","journal-title":"ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1863","DOI":"10.3390\/rs6031863","article-title":"Image-Based Delineation and Classification of Built Heritage Masonry","volume":"6","author":"Oses","year":"2014","journal-title":"Remote Sens."},{"key":"ref_26","first-page":"592","article-title":"Evaluation of historic masonry substrates: Towards greater objectivity and efficiency","volume":"102","author":"Valero","year":"2016","journal-title":"Constr. Build. Mater."},{"key":"ref_27","unstructured":"Xie, J., Xu, L., and Chen, E. (2012). Image Denoising and Inpainting with Deep Neural Networks. Advances in Neural Information Processing Systems 25 (NIPS), MIT Press."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Liu, Y., Pan, J., and Su, Z. (2019, January 17\u201320). Deep Blind Image Inpainting. Proceedings of the Intelligence Science and Big Data Engineering, Visual Data Engineering, Nanjing, China.","DOI":"10.1007\/978-3-030-36189-1_11"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"K\u00f6hler, R., Schuler, C., Sch\u00f6lkopf, B., and Harmeling, S. (2014, January 2\u20135). Mask-Specific Inpainting with Deep Neural Networks. Proceedings of the German Conference on Pattern Recognition, LNCS, Munster, Germany.","DOI":"10.1007\/978-3-319-11752-2_43"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1017\/S0956792502004904","article-title":"Digital inpainting based on the Mumford-Shah-Euler image model","volume":"13","author":"Esedoglu","year":"2002","journal-title":"Eur. J. Appl. Math."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1273","DOI":"10.1109\/TCSVT.2007.903663","article-title":"Image Compression with Edge-Based Inpainting","volume":"17","author":"Liu","year":"2007","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1145\/1531326.1531330","article-title":"PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing","volume":"28","author":"Barnes","year":"2009","journal-title":"ACM Trans. Graph."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"82:1","DOI":"10.1145\/2185520.2185578","article-title":"Image Melding: Combining Inconsistent Images using Patch-based Synthesis","volume":"31","author":"Darabi","year":"2012","journal-title":"ACM Trans. Graph."},{"key":"ref_34","unstructured":"Martin, D., Fowlkes, C., Tal, D., and Malik, J. (2001, January 7\u201314). A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics. Proceedings of the 8th International Conference Computer Vision, Vancouver, BC, Canada."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., and Catanzaro, B. (2018, January 8\u201314). Image Inpainting for Irregular Holes Using Partial Convolutions. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01252-6_6"},{"key":"ref_36","unstructured":"Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T. (November, January 27). Free-Form Image Inpainting With Gated Convolution. Proceedings of the International Conference on Computer Vision (ICCV), Seoul, Korea."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Pathak, D., Kr\u00e4henb\u00fchl, P., Donahue, J., Darrell, T., and Efros, A.A. (2016, January 27\u201330). Context Encoders: Feature Learning by Inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.278"},{"key":"ref_38","unstructured":"Wang, Y., Tao, X., Qi, X., Shen, X., and Jia, J. (2018). Image Inpainting via Generative Multi-column Convolutional Neural Networks. arXiv."},{"key":"ref_39","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Zheng, C., Cham, T., and Cai, J. (2019, January 16\u201320). Pluralistic Image Completion. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00153"},{"key":"ref_41","first-page":"2672","article-title":"Generative Adversarial Nets","volume":"Volume 27","author":"Goodfellow","year":"2014","journal-title":"Advances in Neural Information Processing Systems"},{"key":"ref_42","unstructured":"Nazeri, K., Ng, E., Joseph, T., Qureshi, F., and Ebrahimi, M. (2019, January 27\u201328). EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning. Proceedings of the International Conference on Computer Vision Workshop (ICCVW), Seoul, Korea."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"679","DOI":"10.1109\/TPAMI.1986.4767851","article-title":"A computational approach to edge-detection","volume":"8","author":"Canny","year":"1986","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Xie, S., and Tu, Z. (2015, January 7\u201313). Holistically-Nested Edge Detection. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.164"},{"key":"ref_45","unstructured":"S\u00e1r\u00e1ndi, I., Linder, T., Arras, K.O., and Leibe, B. (2018, January 1\u20135). How Robust is 3D Human Pose Estimation to Occlusion?. Proceedings of the IEEE\/RSJ International Conference Intelligent Robots and Systems Workshop (IROSWS), Madrid, Spain."},{"key":"ref_46","unstructured":"Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., and Zisserman, A. (2020, November 27). The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. Available online: http:\/\/host.robots.ox.ac.uk\/pascal\/VOC\/voc2012\/."},{"key":"ref_47","unstructured":"Jung, A.B., Wada, K., Crall, J., Tanaka, S., Graving, J., Reinders, C., Yadav, S., Banerjee, J., Vecsei, G., and Kraft, A. (2020, November 27). Imgaug. Available online: https:\/\/github.com\/aleju\/imgaug."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_49","unstructured":"Kingma, D.P., and Ba, J. (2014, January 14\u201316). Adam: A Method for Stochastic Optimization. Proceedings of the International Conference on Learning Representations, Banff, AB, Canada."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Johnson, J., Alahi, A., and Fei-Fei, L. (2016, January 8\u201316). Perceptual losses for real-time style transfer and super-resolution. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46475-6_43"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Zhu, J., Park, T., Isola, P., and Efros, A.A. (2017, January 22\u201329). Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.244"},{"key":"ref_52","unstructured":"Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. (2018). Spectral Normalization for Generative Adversarial Networks. arXiv."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Gatys, L.A., Ecker, A.S., and Bethge, M. (2016, January 27\u201330). Image Style Transfer Using Convolutional Neural Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.265"},{"key":"ref_54","first-page":"187","article-title":"The Watershed Transform: Definitions, Algorithms and Parallelization Strategies","volume":"41","author":"Roerdink","year":"2000","journal-title":"Fundam. Inf."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"375","DOI":"10.1016\/S0167-8655(02)00262-3","article-title":"Strategies for image segmentation combining region and boundary information","volume":"24","author":"Freixenet","year":"2003","journal-title":"Pattern Recognit. Lett."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"4522","DOI":"10.1093\/bioinformatics\/btz259","article-title":"Biomedical image augmentation using Augmentor","volume":"35","author":"Bloice","year":"2019","journal-title":"Bioinformatics"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image quality assessment: From error visibility to structural similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_58","unstructured":"Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Klambauer, G., and Hochreiter, S. (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium. arXiv."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1002\/nav.3800020109","article-title":"The Hungarian method for the assignment problem","volume":"2","author":"Kuhn","year":"1955","journal-title":"Nav. Res. Logist. Q."},{"key":"ref_60","unstructured":"Femiani, J., Para, W.R., Mitra, N.J., and Wonka, P. (2018). Facade Segmentation in the Wild. arXiv."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/12\/23\/3918\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:39:14Z","timestamp":1760179154000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/12\/23\/3918"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,29]]},"references-count":60,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2020,12]]}},"alternative-id":["rs12233918"],"URL":"https:\/\/doi.org\/10.3390\/rs12233918","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,11,29]]}}}