{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,24]],"date-time":"2026-03-24T03:35:28Z","timestamp":1774323328590,"version":"3.50.1"},"reference-count":48,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2022,12,30]],"date-time":"2022-12-30T00:00:00Z","timestamp":1672358400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"VIA IMC GmbH"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>This article proposes a novel approach to segment instances of bulk material heaps in aerial data using deep learning-based computer vision and transfer learning to automate material inventory procedures in the construction-, mining-, and material-handling industry. The proposed method uses information about color, texture, and surface topography as input features for a supervised computer vision algorithm. The approach neither relies on hand-crafted assumptions on the general shape of heaps, nor does it solely rely on surface material type recognition. Therefore, the method is able to (1) segment heaps with \u201catypical\u201d shapes, (2) segment heaps that stand on a surface made of the same material as the heap itself, (3) segment individual heaps of the same material type that border each other, and (4) differentiate between artificial heaps and other objects of similar shape like natural hills. To utilize well-established segmentation algorithms for raster-grid-based data structures, this study proposes a pre-processing step to remove all overhanging occlusions from a 3D surface scan and convert it into a 2.5D raster format. Preliminary results demonstrate the general feasibility of the approach. The average F1 score computed on the test set was 0.70 regarding object detection and 0.90 regarding the pixelwise segmentation.<\/jats:p>","DOI":"10.3390\/rs15010211","type":"journal-article","created":{"date-parts":[[2023,1,2]],"date-time":"2023-01-02T02:44:03Z","timestamp":1672627443000},"page":"211","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Automatic Segmentation of Bulk Material Heaps Using Color, Texture, and Topography from Aerial Data and Deep Learning-Based Computer Vision"],"prefix":"10.3390","volume":"15","author":[{"given":"Andreas","family":"Ellinger","sequence":"first","affiliation":[{"name":"Institute of Construction Informatics, TU Dresden, 01062 Dresden, Germany"},{"name":"VIA IMC GmbH, 12489 Berlin, Germany"}]},{"given":"Christian","family":"Woerner","sequence":"additional","affiliation":[{"name":"VIA IMC GmbH, 12489 Berlin, Germany"}]},{"given":"Raimar","family":"Scherer","sequence":"additional","affiliation":[{"name":"Institute of Construction Informatics, TU Dresden, 01062 Dresden, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2022,12,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Tucci, G., Gebbia, A., Conti, A., Fiorini, L., and Lubello, C. (2019). Monitoring and Computation of the Volumes of Stockpiles of Bulk Material by Means of UAV Photogrammetric Surveying. Remote Sens., 11.","DOI":"10.3390\/rs11121471"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"808","DOI":"10.1061\/(ASCE)0733-9364(2005)131:7(808)","article-title":"Fundamental Principles of Site Material Management","volume":"131","author":"Thomas","year":"2005","journal-title":"J. Constr. Eng. Manag."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"100181","DOI":"10.1016\/j.rineng.2020.100181","article-title":"A review on the impact of mining operation: Monitoring, assessment and management","volume":"8","author":"Agboola","year":"2020","journal-title":"Results Eng."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Kovani\u010d, L., Blistan, P., \u0160troner, M., Urban, R., and Blistanova, M. (2021). Suitability of Aerial Photogrammetry for Dump Documentation and Volume Determination in Large Areas. Appl. Sci., 11.","DOI":"10.3390\/app11146564"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Yang, X., Huang, Y., and Zhang, Q. (2020). Automatic Stockpile Extraction and Measurement Using 3D Point Cloud and Multi-Scale Directional Curvature. Remote Sens., 12.","DOI":"10.3390\/rs12060960"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Son, S.W., Kim, D.W., Sung, W.G., and Yu, J.J. (2020). Integrating UAV and TLS Approaches for Environmental Management: A Case Study of a Waste Stockpile Area. Remote Sens., 12.","DOI":"10.3390\/rs12101615"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Liu, J., Liu, X., Lv, X., Wang, B., and Lian, X. (2022). Novel Method for Monitoring Mining Subsidence Featuring Co-Registration of UAV LiDAR Data and Photogrammetry. Appl. Sci., 12.","DOI":"10.3390\/app12189374"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Godone, D., Allasia, P., Borrelli, L., and Gull\u00e0, G. (2020). UAV and Structure from Motion Approach to Monitor the Maierato Landslide Evolution. Remote Sens., 12.","DOI":"10.3390\/rs12061039"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Park, H.C., Rachmawati, T.S.N., and Kim, S. (2022). UAV-Based High-Rise Buildings Earthwork Monitoring\u2014A Case Study. Sustainability, 14.","DOI":"10.3390\/su141610179"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Carabassa, V., Montero, P., Alca\u00f1iz, J.M., and Padr\u00f3, J.-C. (2021). Soil Erosion Monitoring in Quarry Restoration Using Drones. Minerals, 11.","DOI":"10.3390\/min11090949"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"012032","DOI":"10.1088\/1757-899X\/270\/1\/012032","article-title":"Volumetric calculation using low cost unmanned aerial vehicle (UAV) approach","volume":"270","author":"Maulud","year":"2017","journal-title":"IOP Conf. Ser. Mater. Sci. Eng."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Alsayed, A., Yunusa-Kaltungo, A., Quinn, M.K., Arvin, F., and Nabawy, M.R.A. (2021). Drone-Assisted Confined Space Inspection and Stockpile Volume Estimation. Remote Sens., 13.","DOI":"10.3390\/rs13173356"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Jiang, Y., Huang, Y., Liu, J., Li, D., Li, S., Nie, W., and Chung, I.-H. (2022). Automatic Volume Calculation and Mapping of Construction and Demolition Debris Using Drones, Deep Learning, and GIS. Drones, 6.","DOI":"10.3390\/drones6100279"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Sivitskis, A.J., Lehner, J.W., Harrower, M.J., Dumitru, I.A., Paulsen, P.E., Nathan, S., Viete, D.R., Al-Jabri, S., Helwing, B., and Wiig, F. (2019). Detecting and Mapping Slag Heaps at Ancient Copper Production Sites in Oman. Remote Sens., 11.","DOI":"10.3390\/rs11243014"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Finkbeiner, M.S., Uchiyama, N., and Sawodny, O. (2019, January 27\u201328). Shape Recognition of Material Heaps in Outdoor Environments and Optimal Excavation Planning. Proceedings of the 2019 International Electronics Symposium (IES), Surabaya, Indonesia.","DOI":"10.1109\/ELECSYM.2019.8901670"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"679","DOI":"10.1109\/TPAMI.1986.4767851","article-title":"A Computational Approach to Edge Detection","volume":"8","author":"Canny","year":"1986","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Kumar, C., Mathur, Y., and Jannesari, A. (June, January 30). Efficient Volume Estimation for Dynamic Environments using Deep Learning on the Edge. Proceedings of the 2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), Lyon, France.","DOI":"10.1109\/IPDPSW55747.2022.00159"},{"key":"ref_18","first-page":"e01216","article-title":"Rockfill material segmentation and gradation calculation based on deep learning","volume":"17","author":"Fan","year":"2022","journal-title":"Case Stud. Constr. Mater."},{"key":"ref_19","unstructured":"Huang, H., University of Illinois Urbana-Champaign, Tutumluer, E., Luo, J., Ding, K., Qamhia, I., and Hart, J. (2022). 3D Image Analysis Using Deep Learning for Size and Shape Characterization of Stockpile Riprap Aggregates\u2014Phase 2, Illinois Center for Transportation."},{"key":"ref_20","unstructured":"Zalta, E.N. (2018). Sorites Paradox. The Stanford Encyclopedia of Philosophy, Metaphysics Research Lab, Stanford University. Available online: https:\/\/plato.stanford.edu\/archives\/sum2018\/entries\/sorites-paradox\/."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"100011","DOI":"10.1016\/j.ophoto.2021.100011","article-title":"Semantic segmentation of point cloud data using raw laser scanner measurements and deep neural networks","volume":"3","author":"Kaijaluoto","year":"2022","journal-title":"ISPRS Open J. Photogramm. Remote Sens."},{"key":"ref_22","unstructured":"Qi, C.R., Su, H., Mo, K., and Guibas, L.J. (2017, January 21\u201326). PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA. Available online: https:\/\/openaccess.thecvf.com\/content_cvpr_2017\/html\/Qi_PointNet_Deep_Learning_CVPR_2017_paper.html."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Zhao, H., Jiang, L., Fu, C.-W., and Jia, J. (2019, January 15\u201320). PointWeb: Enhancing Local Neighborhood Features for Point Cloud Processing. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00571"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"722","DOI":"10.1109\/TITS.2020.3023541","article-title":"Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review","volume":"23","author":"Cui","year":"2022","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Griffiths, D., and Boehm, J. (2019). A Review on Deep Learning Techniques for 3D Sensed Data Classification. Remote Sens., 11.","DOI":"10.3390\/rs11121499"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Doll\u00e1r, P., and Girshick, R. (2017, January 22\u201329). Mask R-CNN. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1109\/JPROC.2020.3004555","article-title":"A Comprehensive Survey on Transfer Learning","volume":"109","author":"Zhuang","year":"2021","journal-title":"Proc. IEEE"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Kentsch, S., Caceres, M.L.L., Serrano, D., Roure, F., and Diez, Y. (2020). Computer Vision and Deep Learning Techniques for the Analysis of Drone-Acquired Forest Images, a Transfer Learning Study. Remote Sens., 12.","DOI":"10.3390\/rs12081287"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Maqsood, M., Nazir, F., Khan, U., Aadil, F., Jamal, H., Mehmood, I., and Song, O.-Y. (2019). Transfer Learning Assisted Classification and Detection of Alzheimer\u2019s Disease Stages Using 3D MRI Scans. Sensors, 19.","DOI":"10.3390\/s19112645"},{"key":"ref_30","unstructured":"(2022, October 19). COCO\u2014Common Objects in Context. Available online: https:\/\/cocodataset.org\/#explore."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Reina, A.J., Mart\u00ednez, J.L., Mandow, A., Morales, J., Garc\u00eda-Cerezo, A., Mart\u00ednez, J.L., and Morales, J. (2014, January 8\u201311). Collapsible cubes: Removing overhangs from 3D point clouds to build local navigable elevation maps. Proceedings of the 2014 IEEE\/ASME International Conference on Advanced Intelligent Mechatronics, Besan\u00e7on, France.","DOI":"10.1109\/AIM.2014.6878213"},{"key":"ref_32","unstructured":"(2022, October 14). las2dem_README. Available online: http:\/\/www.cs.unc.edu\/~isenburg\/lastools\/download\/las2dem_README.txt."},{"key":"ref_33","unstructured":"(2022, October 14). LAStools. 12 October 2022. Available online: https:\/\/github.com\/LAStools\/LAStools."},{"key":"ref_34","unstructured":"(2022, October 17). OpenCV: Color Conversions. Available online: https:\/\/docs.opencv.org\/3.4\/de\/d25\/imgproc_color_conversions.html."},{"key":"ref_35","unstructured":"G\u00e9ron, A. (2019). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems, O\u2019Reilly Media, Inc.. [2nd ed.]."},{"key":"ref_36","unstructured":"Python, R. (2022, October 17). Image Segmentation Using Color Spaces in OpenCV + Python\u2013Real Python. Available online: https:\/\/realpython.com\/python-opencv-color-spaces\/."},{"key":"ref_37","unstructured":"(2022, October 22). Color Quantization Using K-Means. Scikit-Learn. Available online: https:\/\/scikit-learn\/stable\/auto_examples\/cluster\/plot_color_quantization.html."},{"key":"ref_38","first-page":"3523","article-title":"Image Segmentation Using Deep Learning: A Survey","volume":"44","author":"Minaee","year":"2022","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Girshick, R. (2015, January 7\u201313). Fast R-CNN. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile. Available online: https:\/\/openaccess.thecvf.com\/content_iccv_2015\/html\/Girshick_Fast_R-CNN_ICCV_2015_paper.html.","DOI":"10.1109\/ICCV.2015.169"},{"key":"ref_40","unstructured":"Thoma, M. (2016). A Survey of Semantic Segmentation. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Padilla, R., Netto, S.L., and da Silva, E.A.B. (2020, January 1\u20133). A Survey on Performance Metrics for Object-Detection Algorithms. Proceedings of the 2020 International Conference on Systems, Signals and Image Processing (IWSSIP), Niteroi, Brazil.","DOI":"10.1109\/IWSSIP48289.2020.9145130"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Kirillov, A., He, K., Girshick, R., Rother, C., and Dollar, P. (2019, January 16\u201320). Panoptic Segmentation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA. Available online: https:\/\/openaccess.thecvf.com\/content_CVPR_2019\/html\/Kirillov_Panoptic_Segmentation_CVPR_2019_paper.html.","DOI":"10.1109\/CVPR.2019.00963"},{"key":"ref_43","unstructured":"(2022, October 26). PyTorch. Available online: https:\/\/www.pytorch.org."},{"key":"ref_44","unstructured":"(2022, October 26). OpenCV. Available online: https:\/\/opencv.org\/."},{"key":"ref_45","unstructured":"(2022, October 15). NumPy. Available online: https:\/\/numpy.org\/."},{"key":"ref_46","unstructured":"GDAL-Geospatial Data Abstraction Library (2022, October 26). Open Source Geospatial Foundation, 26 October 2022. Available online: https:\/\/github.com\/OSGeo\/gdal."},{"key":"ref_47","unstructured":"(2022, October 21). Azure VM Comparison. Available online: https:\/\/azureprice.net\/vm\/Standard_NC6."},{"key":"ref_48","unstructured":"(2022, October 15). Open3D: A Modern Library for 3D Data Processing\u2014Open3D 0.15.1 Documentation. Available online: http:\/\/www.open3d.org\/docs\/release\/."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/15\/1\/211\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:48:42Z","timestamp":1760147322000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/15\/1\/211"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,30]]},"references-count":48,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,1]]}},"alternative-id":["rs15010211"],"URL":"https:\/\/doi.org\/10.3390\/rs15010211","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,30]]}}}