{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:04:52Z","timestamp":1760144692624,"version":"build-2065373602"},"reference-count":52,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2024,5,10]],"date-time":"2024-05-10T00:00:00Z","timestamp":1715299200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001691","name":"JSPS KAKENHI","doi-asserted-by":"publisher","award":["JP21H03456","JP23K11211","JP23K11141"],"award-info":[{"award-number":["JP21H03456","JP23K11211","JP23K11141"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In our digitally driven society, advances in software and hardware to capture video data allow extensive gathering and analysis of large datasets. This has stimulated interest in extracting information from video data, such as buildings and urban streets, to enhance understanding of the environment. Urban buildings and streets, as essential parts of cities, carry valuable information relevant to daily life. Extracting features from these elements and integrating them with technologies such as VR and AR can contribute to more intelligent and personalized urban public services. Despite its potential benefits, collecting videos of urban environments introduces challenges because of the presence of dynamic objects. The varying shape of the target building in each frame necessitates careful selection to ensure the extraction of quality features. To address this problem, we propose a novel evaluation metric that considers the video-inpainting-restoration quality and the relevance of the target object, considering minimizing areas with cars, maximizing areas with the target building, and minimizing overlapping areas. This metric extends existing video-inpainting-evaluation metrics by considering the relevance of the target object and interconnectivity between objects. We conducted experiment to validate the proposed metrics using real-world datasets from Japanese cities Sapporo and Yokohama. The experiment results demonstrate feasibility of selecting video frames conducive to building feature extraction.<\/jats:p>","DOI":"10.3390\/s24103035","type":"journal-article","created":{"date-parts":[[2024,5,13]],"date-time":"2024-05-13T11:18:17Z","timestamp":1715599097000},"page":"3035","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["A Novel Frame-Selection Metric for Video Inpainting to Enhance Urban Feature Extraction"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-4819-2066","authenticated-orcid":false,"given":"Yuhu","family":"Feng","sequence":"first","affiliation":[{"name":"Graduate School of Information Science and Technology, Hokkaido University, Sapporo 060-0814, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4099-2465","authenticated-orcid":false,"given":"Jiahuan","family":"Zhang","sequence":"additional","affiliation":[{"name":"Graduate School of Information Science and Technology, Hokkaido University, Sapporo 060-0814, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2898-2504","authenticated-orcid":false,"given":"Guang","family":"Li","sequence":"additional","affiliation":[{"name":"Education and Research Center for Mathematical and Data Science, Hokkaido University, Sapporo 060-0812, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4474-3995","authenticated-orcid":false,"given":"Ren","family":"Togo","sequence":"additional","affiliation":[{"name":"Faculty of Information Science and Technology, Hokkaido University, Sapporo 060-0814, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8039-3462","authenticated-orcid":false,"given":"Keisuke","family":"Maeda","sequence":"additional","affiliation":[{"name":"Data-Driven Interdisciplinary Research Emergence Department, Hokkaido University, Sapporo 060-0813, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5332-8112","authenticated-orcid":false,"given":"Takahiro","family":"Ogawa","sequence":"additional","affiliation":[{"name":"Faculty of Information Science and Technology, Hokkaido University, Sapporo 060-0814, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1496-1761","authenticated-orcid":false,"given":"Miki","family":"Haseyama","sequence":"additional","affiliation":[{"name":"Faculty of Information Science and Technology, Hokkaido University, Sapporo 060-0814, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,5,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"652","DOI":"10.1109\/ACCESS.2014.2332453","article-title":"Toward scalable systems for big data analytics: A technology tutorial","volume":"2","author":"Hu","year":"2014","journal-title":"IEEE Access"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"276","DOI":"10.1109\/TBDATA.2016.2586447","article-title":"Visual analytics in urban computing: An overview","volume":"2","author":"Zheng","year":"2016","journal-title":"IEEE Trans. Big Data"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1108\/LHT-12-2017-0274","article-title":"Artificial Intelligence powered Internet of Things and smart public service","volume":"38","author":"Ma","year":"2020","journal-title":"Libr. Hi Tech"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"448","DOI":"10.1093\/comjnl\/bxy082","article-title":"Algorithmic government: Automating public services and supporting civil servants in using data science technologies","volume":"62","author":"Engin","year":"2019","journal-title":"Comput. J."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1016\/j.giq.2016.05.004","article-title":"Universal and contextualized public services: Digital public service innovation framework","volume":"33","author":"Bertot","year":"2016","journal-title":"Gov. Inf. Q."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Nam, T., and Pardo, T.A. (2011, January 26\u201328). Smart city as urban innovation: Focusing on management, policy, and context. Proceedings of the International Conference on Theory and Practice of Electronic Governance, Tallinn, Estonia.","DOI":"10.1145\/2072069.2072100"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Lee, P., Hunter, W.C., and Chung, N. (2020). Smart tourism city: Developments and transformations. Sustainability, 12.","DOI":"10.3390\/su12103958"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Gan, Y., Li, G., Togo, R., Maeda, K., Ogawa, T., and Haseyama, M. (2023). Zero-shot traffic sign recognition based on midlevel feature matching. Sensors, 23.","DOI":"10.3390\/s23239607"},{"key":"ref_9","first-page":"1","article-title":"Urban computing: Concepts, methodologies, and applications","volume":"5","author":"Zheng","year":"2014","journal-title":"ACM Trans. Intell. Syst. Technol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1865","DOI":"10.24294\/jgc.v6i1.1865","article-title":"Digital twins and 3D information modeling in a smart city for traffic controlling: A review","volume":"6","author":"Rezaei","year":"2023","journal-title":"J. Geogr. Cartogr."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Li, X., Lv, Z., Hu, J., Zhang, B., Yin, L., Zhong, C., Wang, W., and Feng, S. (2015, January 4\u20137). Traffic management and forecasting system based on 3d gis. Proceedings of the 2015 15th IEEE\/ACM International Symposium on Cluster, Cloud and Grid Computing, Shenzhen, China.","DOI":"10.1109\/CCGrid.2015.62"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1109\/TITS.2004.837816","article-title":"Spatial-temporal traffic data analysis based on global data management using MAS","volume":"5","author":"Zhang","year":"2004","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Sebe, I.O., Hu, J., You, S., and Neumann, U. (2003, January 7). 3d video surveillance with augmented virtual environments. Proceedings of the First ACM SIGMM international workshop on Video surveillance, Berkeley, CA, USA.","DOI":"10.1145\/982452.982466"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"287","DOI":"10.1111\/cgf.13803","article-title":"A survey on visual traffic simulation: Models, evaluations, and applications in autonomous driving","volume":"Volume 39","author":"Chao","year":"2020","journal-title":"Computer Graphics Forum"},{"key":"ref_15","unstructured":"Gao, G., Gao, J., Liu, Q., Wang, Q., and Wang, Y. (2020). Cnn-based density estimation and crowd counting: A survey. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Kim, D., Woo, S., Lee, J.Y., and Kweon, I.S. (2019, January 15\u201320). Deep video inpainting. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00594"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Zeng, Y., Fu, J., and Chao, H. (2020, January 14\u201319). Learning joint spatial-temporal transformations for video inpainting. Proceedings of the IEEE\/CVF European Conference on Computer Vision, Seattle, WA, USA.","DOI":"10.1007\/978-3-030-58517-4_31"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Li, Z., Lu, C.Z., Qin, J., Guo, C.L., and Cheng, M.M. (2022, January 18\u201324). Towards an end-to-end framework for flow-guided video inpainting. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01704"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"103147","DOI":"10.1016\/j.cviu.2020.103147","article-title":"A comprehensive review of past and present image inpainting methods","volume":"203","author":"Jam","year":"2021","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"102028","DOI":"10.1016\/j.displa.2021.102028","article-title":"Image inpainting based on deep learning: A review","volume":"69","author":"Qin","year":"2021","journal-title":"Displays"},{"key":"ref_21","unstructured":"Zhang, H., Mai, L., Xu, N., Wang, Z., Collomosse, J., and Jin, H. (November, January 27). An internal learning approach to video inpainting. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Republic of Korea."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"e2","DOI":"10.1017\/ATSIP.2012.2","article-title":"An overview on video forensics","volume":"1","author":"Milani","year":"2012","journal-title":"APSIPA Trans. Signal Inf. Process."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Xu, R., Li, X., Zhou, B., and Loy, C.C. (2019, January 15\u201320). Deep flow-guided video inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00384"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1016\/0004-3702(81)90024-2","article-title":"Determining optical flow","volume":"17","author":"Horn","year":"1981","journal-title":"Artif. Intell."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"433","DOI":"10.1145\/212094.212141","article-title":"The computation of optical flow","volume":"27","author":"Beauchemin","year":"1995","journal-title":"ACM Comput. Surv."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Lee, Y.J., Kim, J., and Grauman, K. (2011, January 6\u201313). Key-segments for video object segmentation. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Barcelona, Spain.","DOI":"10.1109\/ICCV.2011.6126471"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1109\/LSP.2012.2227726","article-title":"Making a \u201ccompletely blind\u201d image quality analyzer","volume":"20","author":"Mittal","year":"2012","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"4695","DOI":"10.1109\/TIP.2012.2214050","article-title":"No-reference image quality assessment in the spatial domain","volume":"21","author":"Mittal","year":"2012","journal-title":"IEEE Trans. Image Process."},{"key":"ref_29","unstructured":"Venkatanath, N., Praneeth, D., Bh, M.C., Channappayya, S.S., and Medasani, S.S. (March, January 27). Blind image quality evaluation using perception based features. Proceedings of the National Conference on Communications, Mumbai, India."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"469","DOI":"10.1016\/j.image.2010.05.009","article-title":"No-reference image and video quality estimation: Applications and human-motivated design","volume":"25","author":"Hemami","year":"2010","journal-title":"Signal Process. Image Commun."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1687-5281-2014-40","article-title":"No-reference image and video quality assessment: A classification and review of recent approaches","volume":"2014","author":"Shahid","year":"2014","journal-title":"EURASIP J. Image Video Process."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Zou, X., Yang, L., Liu, D., and Lee, Y.J. (2021, January 20\u201325). Progressive temporal feature alignment network for video inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.01618"},{"key":"ref_33","first-page":"6096","article-title":"Partial convolution for padding, inpainting, and image synthesis","volume":"45","author":"Liu","year":"2022","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_34","unstructured":"Chang, Y.L., Liu, Z.Y., Lee, K.Y., and Hsu, W. (2019). Learnable gated temporal shift module for deep video inpainting. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Hu, Y.T., Wang, H., Ballas, N., Grauman, K., and Schwing, A.G. (2020, January 23\u201328). Proposal-based video completion. Proceedings of the IEEE European Conference on Computer Vision, Glasgow, UK.","DOI":"10.1007\/978-3-030-58583-9_3"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"105778","DOI":"10.1016\/j.knosys.2020.105778","article-title":"Multi-scale generative adversarial inpainting network based on cross-layer attention transfer mechanism","volume":"196","author":"Shao","year":"2020","journal-title":"Knowl. Based Syst."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Yu, B., Li, W., Li, X., Lu, J., and Zhou, J. (2021, January 11\u201317). Frequency-aware spatiotemporal transformers for video inpainting detection. Proceedings of the IEEE International Conference on Computer Vision, Montreal, BC, Canada.","DOI":"10.1109\/ICCV48922.2021.00808"},{"key":"ref_38","unstructured":"Lee, S., Oh, S.W., Won, D., and Kim, S.J. (November, January 27). Copy-and-paste networks for deep video inpainting. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Republic of Korea."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Zhang, K., Fu, J., and Liu, D. (2022, January 23\u201327). Flow-guided transformer for video inpainting. Proceedings of the IEEE European Conference on Computer Vision, Tel Aviv, Israel.","DOI":"10.1007\/978-3-031-19797-0_5"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Wang, X., Chan, K.C., Yu, K., Dong, C., and Change Loy, C. (2019, January 16\u201317). Edvr: Video restoration with enhanced deformable convolutional networks. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA.","DOI":"10.1109\/CVPRW.2019.00247"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Chan, K.C., Wang, X., Yu, K., Dong, C., and Loy, C.C. (2021, January 20\u201325). Basicvsr: The search for essential components in video super-resolution and beyond. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00491"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1007\/BF01420984","article-title":"Performance of optical flow techniques","volume":"12","author":"Barron","year":"1994","journal-title":"Int. J. Comput. Vis."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Van Der Smagt, P., Cremers, D., and Brox, T. (2015, January 7\u201313). Flownet: Learning optical flow with convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.316"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Hui, T.W., Tang, X., and Loy, C.C. (2018, January 18\u201322). Liteflownet: A lightweight convolutional neural network for optical flow estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00936"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Ranjan, A., and Black, M.J. (2017, January 21\u201326). Optical flow estimation using a spatial pyramid network. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.291"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1109\/TIP.2017.2760518","article-title":"Deep neural networks for no-reference and full-reference image quality assessment","volume":"27","author":"Bosse","year":"2017","journal-title":"IEEE Trans. Image Process."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"3129","DOI":"10.1109\/TIP.2012.2190086","article-title":"No-reference image quality assessment using visual codebooks","volume":"21","author":"Ye","year":"2012","journal-title":"IEEE Trans. Image Process."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Fu, Y., and Wang, S. (2016). A no reference image quality assessment metric based on visual perception. Algorithms, 9.","DOI":"10.3390\/a9040087"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1109\/MSP.2011.942471","article-title":"Reduced-and no-reference image quality assessment","volume":"28","author":"Wang","year":"2011","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.cviu.2016.12.009","article-title":"Learning a no-reference quality metric for single-image super-resolution","volume":"158","author":"Ma","year":"2017","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_51","unstructured":"Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., and Zhu, J. (2023). Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., and Lo, W.Y. (2023). Segment anything. arXiv.","DOI":"10.1109\/ICCV51070.2023.00371"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/10\/3035\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:43:45Z","timestamp":1760107425000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/10\/3035"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,10]]},"references-count":52,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2024,5]]}},"alternative-id":["s24103035"],"URL":"https:\/\/doi.org\/10.3390\/s24103035","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2024,5,10]]}}}