{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T02:37:20Z","timestamp":1760236640333,"version":"build-2065373602"},"reference-count":32,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2021,12,9]],"date-time":"2021-12-09T00:00:00Z","timestamp":1639008000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP20K19856"],"award-info":[{"award-number":["JP20K19856"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>This paper presents reliable estimation of deterioration levels via late fusion using multi-view distress images for practical inspection. The proposed method simultaneously solves the following two problems that are necessary to support the practical inspection. Since maintenance of infrastructures requires a high level of safety and reliability, this paper proposes a neural network that can generate an attention map from distress images and text data acquired during the inspection. Thus, deterioration level estimation with high interpretability can be realized. In addition, since multi-view distress images are taken for single distress during the actual inspection, it is necessary to estimate the final result from these images. Therefore, the proposed method integrates estimation results obtained from the multi-view images via the late fusion and can derive an appropriate result considering all the images. To the best of our knowledge, no method has been proposed to solve these problems simultaneously, and this achievement is the biggest contribution of this paper. In this paper, we confirm the effectiveness of the proposed method by conducting experiments using data acquired during the actual inspection.<\/jats:p>","DOI":"10.3390\/jimaging7120273","type":"journal-article","created":{"date-parts":[[2021,12,9]],"date-time":"2021-12-09T21:46:58Z","timestamp":1639086418000},"page":"273","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Reliable Estimation of Deterioration Levels via Late Fusion Using Multi-View Distress Images for Practical Inspection"],"prefix":"10.3390","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8039-3462","authenticated-orcid":false,"given":"Keisuke","family":"Maeda","sequence":"first","affiliation":[{"name":"Office of Institutional Research, Hokkaido University, N-8, W-5, Kita-ku, Sapporo 060-0808, Japan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3884-7325","authenticated-orcid":false,"given":"Naoki","family":"Ogawa","sequence":"additional","affiliation":[{"name":"Graduate School of Information Science and Technology, Hokkaido University, N-14, W-9, Kita-ku, Sapporo 060-0814, Japan"}]},{"given":"Takahiro","family":"Ogawa","sequence":"additional","affiliation":[{"name":"Faculty of Information Science and Technology, Hokkaido University, N-14, W-9, Kita-ku, Sapporo 060-0814, Japan"}]},{"given":"Miki","family":"Haseyama","sequence":"additional","affiliation":[{"name":"Faculty of Information Science and Technology, Hokkaido University, N-14, W-9, Kita-ku, Sapporo 060-0814, Japan"}]}],"member":"1968","published-online":{"date-parts":[[2021,12,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"293","DOI":"10.18770\/KEPCO.2016.02.02.293","article-title":"Development of the Corrosion Deterioration Inspection Tool for Transmission Tower Members","volume":"2","author":"Woo","year":"2016","journal-title":"KEPCO J. Electr. Power Energy"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"527","DOI":"10.1111\/mice.12351","article-title":"Unified vision-based methodology for simultaneous concrete defect detection and geolocalization","volume":"33","author":"Li","year":"2018","journal-title":"Comput.-Aided Civ. Infrastruct. Eng."},{"key":"ref_3","unstructured":"Yasuno, T., Michihiro, N., and Kazuhiro, N. (2020). Per-pixel Classification Rebar Exposures in Bridge Eye-inspection. arXiv."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Maeda, K., Takahashi, S., Ogawa, T., and Haseyama, M. (2017, January 17\u201320). Automatic estimation of deterioration level on transmission towers via deep extreme learning machine based on local receptive field. Proceedings of the IEEE International Conference on Image Processing, Beijing, China.","DOI":"10.1109\/ICIP.2017.8296708"},{"key":"ref_5","unstructured":"(2021, August 29). White Paper on Land, Infrastructure, Transport and Tourism in Japan, 2017; Technical Report; Ministry of Land, Infrastructure Tourism, Transport and Tourism. Available online: http:\/\/www.mlit.go.jp\/common\/001269888.pdf."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"773","DOI":"10.1002\/qre.1634","article-title":"Data analysis for condition-based railway infrastructure maintenance","volume":"31","author":"Bergquist","year":"2015","journal-title":"Qual. Reliab. Eng. Int."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1016\/j.aei.2018.04.014","article-title":"Distress classification of class-imbalanced inspection data via correlation-maximizing weighted extreme learning machine","volume":"37","author":"Maeda","year":"2018","journal-title":"Adv. Eng. Inform."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Kruachottikul, P., Cooharojananone, N., Phanomchoeng, G., Chavarnakul, T., Kovitanggoon, K., Trakulwaranont, D., and Atchariyachanvanich, K. (2019, January 23\u201325). Bridge sub structure defect inspection assistance by using deep learning. Proceedings of the IEEE International Conference on Awareness Science and Technology, Morioka, Japan.","DOI":"10.1109\/ICAwST.2019.8923507"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"991","DOI":"10.1109\/JAS.2020.1003234","article-title":"Concrete defects inspection and 3D mapping using CityFlyer quadrotor robot","volume":"7","author":"Yang","year":"2020","journal-title":"IEEE\/CAA J. Autom. Sin."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1111\/mice.12561","article-title":"Generative adversarial network for road damage detection","volume":"36","author":"Maeda","year":"2021","journal-title":"Comput.-Aided Civ. Infrastruct. Eng."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Arya, D., Maeda, H., Ghosh, S.K., Toshniwal, D., Mraz, A., Kashiyama, T., and Sekimoto, Y. (2020). Transfer learning-based road damage detection for multiple countries. arXiv.","DOI":"10.1016\/j.autcon.2021.103935"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"65234","DOI":"10.1109\/ACCESS.2021.3074019","article-title":"Distress Image Retrieval for Infrastructure Maintenance via Self-Trained Deep Metric Learning Using Experts\u2019 Knowledge","volume":"9","author":"Ogawa","year":"2021","journal-title":"IEEE Access"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"654","DOI":"10.1111\/mice.12451","article-title":"Convolutional sparse coding-based deep random vector functional link network for distress classification of road structures","volume":"34","author":"Maeda","year":"2019","journal-title":"Comput.-Aided Civ. Infrastruct. Eng."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Ogawa, N., Maeda, K., Ogawa, T., and Haseyama, M. (2020, January 10\u201312). Distress Level Classification of Road Infrastructures via CNN Generating Attention Map. Proceedings of the IEEE Global Conference on Life Sciences and Technologies, Kyoto, Japan.","DOI":"10.1109\/LifeTech48969.2020.1570619126"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Ogawa, N., Maeda, K., Ogawa, T., and Haseyama, M. (2021, January 19\u201322). Correlation-Aware Attention Branch Network Using Multi-Modal Data For Deterioration Level Estimation Of Infrastructures. Proceedings of the IEEE International Conference on Image Processing, Anchorage, AK, USA.","DOI":"10.1109\/ICIP42928.2021.9506551"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"6957","DOI":"10.1109\/TIP.2021.3100556","article-title":"Interleaved Deep Artifacts-Aware Attention Mechanism for Concrete Structural Defect Classification","volume":"30","author":"Bhattacharya","year":"2021","journal-title":"IEEE Trans. Image Process."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"3707","DOI":"10.1109\/TCSVT.2020.3028008","article-title":"Multi-deformation aware attention learning for concrete structural defect classification","volume":"31","author":"Bhattacharya","year":"2020","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. (2016, January 27\u201330). Learning deep features for discriminative localization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.319"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_20","unstructured":"Nie, W., Zhang, Y., and Patel, A. (2018, January 10\u201315). A theoretical explanation for perplexing behaviors of backpropagation-based visualizations. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Chattopadhay, A., Sarkar, A., Howlader, P., and Balasubramanian, V.N. (2018, January 12\u201315). Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. Proceedings of the IEEE Winter Conference on Applications of Computer Vision, Lake Tahoe, NV, USA.","DOI":"10.1109\/WACV.2018.00097"},{"key":"ref_22","first-page":"423","article-title":"Multimodal machine learning: A survey and taxonomy","volume":"41","author":"Ahuja","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Maeda, K., Takahashi, S., Ogawa, T., and Haseyama, M. (2017). Distress Classification of Road Structures via Adaptive Bayesian Network Model Selection. J. Comput. Civ. Eng., 31.","DOI":"10.1061\/(ASCE)CP.1943-5487.0000686"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"23091","DOI":"10.1007\/s11042-020-10040-2","article-title":"Deterioration level estimation via neural network maximizing category-based ordinally supervised multi-view canonical correlation","volume":"80","author":"Maeda","year":"2021","journal-title":"Multimed. Tools Appl."},{"key":"ref_25","unstructured":"Guo, M.H., Xu, T.X., Liu, J.J., Liu, Z.N., Jiang, P.T., Mu, T.J., Zhang, S.H., Martin, R.R., Cheng, M.M., and Hu, S.M. (2021). Attention Mechanisms in Computer Vision: A Survey. arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"633","DOI":"10.1109\/JSTSP.2018.2849593","article-title":"Estimation of deterioration levels of transmission towers via deep learning maximizing canonical correlation between heterogeneous features","volume":"12","author":"Maeda","year":"2018","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Fukui, H., Hirakawa, T., Yamashita, T., and Fujiyoshi, H. (2019, January 15\u201320). Attention branch network: Learning of attention mechanism for visual explanation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.01096"},{"key":"ref_28","unstructured":"Mitsuhara, M., Fukui, H., Sakashita, Y., Ogata, T., Hirakawa, T., Yamashita, T., and Fujiyoshi, H. (2019). Embedding Human Knowledge into Deep Neural Network via Attention Map. arXiv."},{"key":"ref_29","unstructured":"Andrew, G., Arora, R., Bilmes, J., and Livescu, K. (2013, January 16\u201321). Deep canonical correlation analysis. Proceedings of the International Conference on Machine Learning, Atlanta, GA, USA."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"17","DOI":"10.3151\/coj.48.5_17","article-title":"Maintenance Management System for Concrete Structures in Expressways\u2014A Case Study of NEXCO East Japan Kanto Branch","volume":"48","author":"Yuji","year":"2010","journal-title":"Concr. J."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_32","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20138). Imagenet classification with deep convolutional neural networks. Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA."}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/7\/12\/273\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:44:31Z","timestamp":1760168671000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/7\/12\/273"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,9]]},"references-count":32,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["jimaging7120273"],"URL":"https:\/\/doi.org\/10.3390\/jimaging7120273","relation":{},"ISSN":["2313-433X"],"issn-type":[{"type":"electronic","value":"2313-433X"}],"subject":[],"published":{"date-parts":[[2021,12,9]]}}}