{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T10:07:12Z","timestamp":1774519632454,"version":"3.50.1"},"reference-count":62,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2022,1,11]],"date-time":"2022-01-11T00:00:00Z","timestamp":1641859200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Manual visual inspection of civil infrastructure is high-risk, subjective, and time-consuming. The success of deep learning and the proliferation of low-cost consumer robots has spurred rapid growth in research and application of autonomous inspections. The major components of autonomous inspection include data acquisition, data processing, and decision making, which are usually studied independently. However, for robust real-world applicability, these three aspects of the overall process need to be addressed concurrently with end-to-end testing, incorporating scenarios such as variations in structure type, color, damage level, camera distance, view angle, lighting, etc. Developing real-world datasets that span all these scenarios is nearly impossible. In this paper, we propose a framework to create a virtual visual inspection testbed using 3D synthetic environments that can enable end-to-end testing of autonomous inspection strategies. To populate the 3D synthetic environment with virtual damaged buildings, we propose the use of a non-linear finite element model to inform the realistic and automated visual rendering of different damage types, the damage state, and the material textures of what are termed herein physics-based graphics models (PBGMs). To demonstrate the benefits of the autonomous inspection testbed, three experiments are conducted with models of earthquake damaged reinforced concrete buildings. First, we implement the proposed framework to generate a new large-scale annotated benchmark dataset for post-earthquake inspections of buildings termed QuakeCity. Second, we demonstrate the improved performance of deep learning models trained using the QuakeCity dataset for inference on real data. Finally, a comparison of deep learning-based damage state estimation for different data acquisition strategies is carried out. The results demonstrate the use of PBGMs as an effective testbed for the development and validation of strategies for autonomous vision-based inspections of civil infrastructure.<\/jats:p>","DOI":"10.3390\/s22020532","type":"journal-article","created":{"date-parts":[[2022,1,11]],"date-time":"2022-01-11T20:33:04Z","timestamp":1641933184000},"page":"532","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":65,"title":["Physics-Based Graphics Models in 3D Synthetic Environments as Autonomous Vision-Based Inspection Testbeds"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2118-5975","authenticated-orcid":false,"given":"Vedhus","family":"Hoskere","sequence":"first","affiliation":[{"name":"Department of Civil and Environmental Engineering, University of Houston, Houston, TX 77024, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1680-5079","authenticated-orcid":false,"given":"Yasutaka","family":"Narazaki","sequence":"additional","affiliation":[{"name":"Department of Civil and Environmental Engineering, ZJUI-UIUC Institute, Zhejiang University, Hangzhou 310027, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"suffix":"Jr.","given":"Billie F.","family":"Spencer","sequence":"additional","affiliation":[{"name":"Department of Civil and Environmental Engineering, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,1,11]]},"reference":[{"key":"ref_1","unstructured":"Rojahn, C., Bonneville, D.R., Quadri, N.D., Phipps, M.T., Ranous, R.A., Russell, J.E., Staehlin, W.E., and Turner, Z. (2005). ATC-20-1 Field Manual: Postearthquake Safety Evaluation of Buildings, Applied Technology Council."},{"key":"ref_2","unstructured":"(2020, November 16). Mexico City Earthquake: What We Know. Available online: https:\/\/www.usatoday.com\/story\/news\/world\/2017\/09\/20\/what-we-know-mexico-earthquake\/684113001\/."},{"key":"ref_3","unstructured":"Eulich, W. (2018, July 05). Mexico\u2019s Long Wait for Building Inspections. Available online: https:\/\/www.usnews.com\/news\/best-countries\/articles\/2017-09-27\/mexicans-face-a-daunting-wait-for-building-inspections-after-earthquakes."},{"key":"ref_4","unstructured":"(2018, April 01). Colegio de Ingenieros Civiles de M\u00e9xico (CICM) Resumen Preliminar de Danos de los Inmeubles Inspeccionados por las Brigadas del CICM del Sismo del 19\/09\/2017. Available online: https:\/\/www.sismosmexico.org\/informes."},{"key":"ref_5","unstructured":"(2020, November 16). Thousands Homeless after Mexico Earthquake. Available online: https:\/\/www.13abc.com\/content\/news\/Thousands-homeless-after-Mexico-earthquake-447419433.html."},{"key":"ref_6","unstructured":"(2020, November 16). 6 Months after Mexico Quake Some Still Camp Outside Homes\u2013The Denver Post. Available online: https:\/\/www.denverpost.com\/2018\/03\/20\/mexico-earthquake-homeless-camps\/."},{"key":"ref_7","unstructured":"(2020, June 02). 2015 Nepal Earthquake: Facts, FAQs, and How to Help | World Vision. Available online: https:\/\/www.worldvision.org\/disaster-relief-news-stories\/2015-nepal-earthquake-facts."},{"key":"ref_8","unstructured":"(2020, June 02). Italian Earthquake: 40 Dead and 50,000 Homeless-Telegraph. Available online: https:\/\/www.telegraph.co.uk\/news\/worldnews\/europe\/italy\/5113762\/Italian-earthquake-40-dead-and-50000-homeless.html."},{"key":"ref_9","unstructured":"(2020, June 02). New Zealand Earthquake \u201cDamaged 100,000 Homes\u201d-BBC News. Available online: https:\/\/www.bbc.com\/news\/world-asia-pacific-11191105."},{"key":"ref_10","unstructured":"Yeum, C.M. (2016). Computer Vision-Based Structural Assessment Exploiting Large Volumes of Images. [Ph.D. Dissertation, Purdue University]."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Ghosh Mondal, T., Jahanshahi, M.R., Wu, R., and Wu, Z.Y. (2020). Deep learning-based multi-class damage detection for autonomous post-disaster reconnaissance. Struct. Control Heal. Monit., 27.","DOI":"10.1002\/stc.2507"},{"key":"ref_12","first-page":"91","article-title":"Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks","volume":"39","author":"Ren","year":"2015","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"e2313","DOI":"10.1002\/stc.2313","article-title":"Automatic seismic damage identification of reinforced concrete columns from images by a region-based deep convolutional neural network","volume":"26","author":"Xu","year":"2019","journal-title":"Struct. Control Heal. Monit."},{"key":"ref_14","unstructured":"Hoskere, V., Narazaki, Y., Hoang, T.A., and Spencer, B.F. (2017, January 11\u201312). Vision-based Structural Inspection using Multiscale Deep Convolutional Neural Networks. Proceedings of the 3rd Huixian International Forum on Earthquake Engineering for Young Researchers, Urbana, IL, USA."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"757","DOI":"10.1007\/s13349-020-00409-0","article-title":"MaDnet: Multi-task Semantic Segmentation of Multiple types of Structural Materials and Damage in Images of Civil Infrastructure","volume":"10","author":"Hoskere","year":"2020","journal-title":"J. Civ. Struct. Heal. Monit."},{"key":"ref_16","unstructured":"Hoskere, V., Narazaki, Y., Hoang, T.A., and Spencer, B.F. (2018, January 22\u201325). Towards Automated Post-Earthquake Inspections with Deep Learning-based Condition-Aware Models. Proceedings of the 7th World Conference on Structural Control and Monitoring, 7WCSCM, Qingdao, China."},{"key":"ref_17","unstructured":"Narazaki, Y., Hoskere, V., Hoang, T.A., and Spencer, B.F. (2017, January 22\u201323). Vision-based automated bridge component recognition integrated with high-level scene understanding. Proceedings of the 13th International Workshop on Advanced Smart Materials and Smart Structures Technology (ANCRiSST), Tokyo, Japan."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Narazaki, Y., Hoskere, V., Hoang, T.A., Fujino, Y., Sakurai, A., and Spencer, B.F. (2019). Vision-based automated bridge component recognition with high-level scene consistency. Comput. Civ. Infrastruct. Eng., 12505.","DOI":"10.1111\/mice.12505"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Narazaki, Y., Hoskere, V., Hoang, T.A., and Spencer, B.F. (2018, January 22\u201325). Automated Bridge Component Recognition using Video Data. Proceedings of the 7th World Conference on Structural Control and Monitoring, 7WCSCM, Qingdao, China.","DOI":"10.1002\/stc.2207"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Gao, Y., and Mosalam, K.M. (2019). PEER Hub ImageNet (\u03a6-Net): A Large-Scale Multi-Attribute Benchmark Dataset of Structural Images, University of California. PEER Report No. 2019\/07.","DOI":"10.55461\/QFKL9711"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Liang, X. (2018). Image-based post-disaster inspection of reinforced concrete bridge systems using deep learning with Bayesian optimization. Comput. Civ. Infrastruct. Eng.","DOI":"10.1111\/mice.12425"},{"key":"ref_22","first-page":"13","article-title":"3D InspectionNet: A deep 3D convolutional neural networks based approach for 3D defect detection on concrete columns","volume":"Volume 10971","author":"Gyekenyesi","year":"2019","journal-title":"Proceedings of the Nondestructive Characterization and Monitoring of Advanced Materials, Aerospace, Civil Infrastructure, and Transportation XIII"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"495","DOI":"10.1111\/mice.12549","article-title":"Postdisaster image-based damage detection and repair cost estimation of reinforced concrete buildings using dual convolutional neural networks","volume":"35","author":"Pan","year":"2020","journal-title":"Comput. Civ. Infrastruct. Eng."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Spencer Jr., B.F., Hoskere, V., and Narazaki, Y. (2019). Advances in Computer Vision\u2013based Civil Infrastructure Inspection and Monitoring. Engineering.","DOI":"10.1016\/j.eng.2018.11.030"},{"key":"ref_25","unstructured":"(2020, October 26). Gazebo. Available online: http:\/\/gazebosim.org\/."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Shah, S., Dey, D., Lovett, C., and Kapoor, A. (2018). AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles. arXiv.","DOI":"10.1007\/978-3-319-67361-5_40"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Ros, G., Sellart, L., Materzynska, J., Vazquez, D., and Lopez, A.M. (2016, January 27\u201330). The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes. Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.352"},{"key":"ref_28","unstructured":"Dosovitskiy, A., Ros, G., Codevilla, F., L\u00f3pez, A., and Koltun, V. (2017). CARLA: An Open Urban Driving Simulator. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Bewley, A., Rigley, J., Liu, Y., Hawke, J., Shen, R., Lam, V.-D., and Kendall, A. (2018). Learning to Drive from Simulation without Real World Labels. arXiv.","DOI":"10.1109\/ICRA.2019.8793668"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Richter, S.R., Darmstadt, T.U., Anu, Z.H., and Koltun, V. (2017, January 22\u201329). Playing for Benchmarks. Proceedings of the International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.243"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Hoskere, V., Narazaki, Y., Spencer, B.F., and Smith, M.D. (2019, January 10\u201312). Deep learning-based damage detection of miter gates using synthetic imagery from computer graphics. Proceedings of the Structural Health Monitoring, Stanford, CA, USA.","DOI":"10.12783\/shm2019\/32463"},{"key":"ref_32","unstructured":"Hoskere, V., Narazaki, Y., and Spencer, B.F. (2019, January 4\u20137). Learning to Detect Important Visual Changes for Structural Inspections using Physics-based Graphics Models. Proceedings of the Submitted to the 9th International Conference on Structural Health Monitoring of Intelligent Infrastructure (SHMII-9), St Louis, MO, USA."},{"key":"ref_33","unstructured":"Hoskere, V. (2021). Developing autonomy in structural inspections through computer vision and graphics. [Ph.D. Dissertation, University of Illinois at Urbana-Champaign]."},{"key":"ref_34","first-page":"709","article-title":"Vision-based dense displacement and strain estimation of miter gates with the performance evaluation using physics-based graphics models","volume":"24","author":"Narazaki","year":"2019","journal-title":"Smart Struct. Syst."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1841","DOI":"10.1177\/1475921720939522","article-title":"Efficient development of vision-based dense three-dimensional displacement measurement algorithms using physics-based graphics models","volume":"20","author":"Narazaki","year":"2020","journal-title":"J. Struct. Heal. Monit."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Zdziebko, P., and Holak, K. (2021). Synthetic Image Generation Using the Finite Element Method and Blender Graphics Program for Modeling of Vision-Based Measurement Systems. Sensors, 21.","DOI":"10.3390\/s21186046"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"107850","DOI":"10.1016\/j.ymssp.2021.107850","article-title":"Synthetic environments for vision-based structural condition assessment of Japanese high-speed railway viaducts","volume":"160","author":"Narazaki","year":"2021","journal-title":"Mech. Syst. Signal Process."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. (2017). Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. arXiv.","DOI":"10.1109\/IROS.2017.8202133"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Szeliski, R. (2011). Computer Vision, Springer. Texts in Computer Science.","DOI":"10.1007\/978-1-84882-935-0"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1117\/12.959611","article-title":"The Theory and Measurement of Bidirectional Reflectance Distribution Function (Brdf) And Bidirectional Transmittance Distribution Function (BTDF)","volume":"Volume 0257","author":"Hunt","year":"1981","journal-title":"Proceedings of the Radiation Scattering in Optical Systems"},{"key":"ref_41","first-page":"1","article-title":"Physically Based Shading at Disney","volume":"2012","author":"Burley","year":"2012","journal-title":"ACM SIGGRAPH"},{"key":"ref_42","unstructured":"(2020, July 11). CC0 Textures-Public Domain PBR Materials. Available online: https:\/\/cc0textures.com\/."},{"key":"ref_43","unstructured":"(2018, August 01). Blender. Available online: https:\/\/www.blender.org\/."},{"key":"ref_44","unstructured":"(2019, May 16). Adobe Substance by Adobe. Available online: https:\/\/www.substance3d.com\/."},{"key":"ref_45","unstructured":"(2020, March 12). OpenCV. Available online: https:\/\/opencv.org\/."},{"key":"ref_46","unstructured":"Musgrave, F.K., Kolb, C.E., and Mace, R.S. (August, January 31). The synthesis and rendering of eroded fractal terrains. Proceedings of the 16th Annual Conference on Computer Graphics and Interactive Techniques-SIGGRAPH \u201989, New York, NY, USA."},{"key":"ref_47","unstructured":"Reinhard, E., Heidrich, W., Pattanaik, S., Debevec, P., and Ward, G. (2010). High Dynamic Range Imaging: Acquisition, Display, and Image-based Lighting, Elsevier."},{"key":"ref_48","unstructured":"(2017, September 15). \u00a9 Dassault Syst\u00e8mes Abaqus 6.14 Online Documentation. Available online: http:\/\/Abaqus.software.polimi.it\/v6.14\/index.html."},{"key":"ref_49","unstructured":"(2021, September 27). The OpenSeesPy Library\u2014OpenSeesPy 3.3.0.0 Documentation. Available online: https:\/\/openseespydoc.readthedocs.io\/en\/latest\/."},{"key":"ref_50","unstructured":"(2017, August 06). Data-Driven Science. Available online: https:\/\/datacenterhub.org\/."},{"key":"ref_51","unstructured":"(2020, July 12). Discover Street View and Contribute Your Own Imagery to Google Maps. Available online: https:\/\/www.google.com\/streetview\/."},{"key":"ref_52","unstructured":"(2020, July 12). 318-14: Building Code Requirements for Structural Concrete and Commentary. Available online: https:\/\/www.concrete.org\/store\/productdetail.aspx?ItemID=318U14&Language=English."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"892","DOI":"10.1061\/(ASCE)0733-9399(1998)124:8(892)","article-title":"Plastic-Damage Model for Cyclic Loading of Concrete Structures","volume":"124","author":"Lee","year":"1998","journal-title":"J. Eng. Mech."},{"key":"ref_54","unstructured":"Jankowiak, T., and Lodygowski, T. (2005). Identification of parameters of concrete damage plasticity constitutive model. Found. Civ. Environ., 53\u201369."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1016\/j.conbuildmat.2015.03.021","article-title":"Simplified micro modeling of partially grouted masonry assemblages","volume":"83","author":"Bolhassani","year":"2015","journal-title":"Constr. Build. Mater."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"2831","DOI":"10.1002\/eqe.2934","article-title":"Fragility functions for masonry infill walls with in-plane loading","volume":"46","author":"Chiozzi","year":"2017","journal-title":"Earthq. Eng. Struct. Dyn."},{"key":"ref_57","unstructured":"(1998). FEMA 306 Evaluation of Earthquake Damaged Concrete and Masonry Wall Buildings Basic Procedures Manual Applied Technology Council (ATC-43 Project) The Partnership for Response and Recovery, The Partnership for Response and Recovery."},{"key":"ref_58","unstructured":"Tu, Y.H., Ao, L.C., and Jean, W.Y. (2012, January 24\u201328). Study on the Earthquake Damage Evaluation Procedure for RC and Confined Masonry Buildings. Proceedings of the 15th World Conference on Earthquake Engineering, Lisbon, Portugal."},{"key":"ref_59","first-page":"1","article-title":"InstaDam: A software tool for rapid semi-automated pixel-wise annotation of defects in images","volume":"11","author":"Hoskere","year":"2021","journal-title":"Appl. Sci."},{"key":"ref_60","first-page":"1","article-title":"Deep residual learning for image steganalysis","volume":"77","author":"Wu","year":"2017","journal-title":"Multimed. Tools Appl."},{"key":"ref_61","unstructured":"(2021, October 18). PyTorch. Available online: https:\/\/pytorch.org\/."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"640","DOI":"10.1109\/TPAMI.2016.2572683","article-title":"Fully Convolutional Networks for Semantic Segmentation","volume":"39","author":"Shelhamer","year":"2015","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/2\/532\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T14:14:51Z","timestamp":1760364891000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/2\/532"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,11]]},"references-count":62,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2022,1]]}},"alternative-id":["s22020532"],"URL":"https:\/\/doi.org\/10.3390\/s22020532","relation":{"has-preprint":[{"id-type":"doi","id":"10.20944\/preprints202111.0154.v1","asserted-by":"object"}]},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,11]]}}}