{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,22]],"date-time":"2025-12-22T10:55:00Z","timestamp":1766400900706,"version":"3.37.3"},"reference-count":52,"publisher":"Walter de Gruyter GmbH","issue":"3","funder":[{"DOI":"10.13039\/501100002347","name":"Federal Ministry of Education and Research","doi-asserted-by":"crossref","award":["13N14674"],"award-info":[{"award-number":["13N14674"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,3,28]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Robotic systems require holistic capabilities to sense, perceive, and act autonomously within their application environment. A safe and trustworthy autonomous operation is essential, especially in hazardous environments and critical applications like autonomous construction machinery for the decontamination of landfill sites. This article presents an enhanced combination of machine learning (ML) methods with classic artificial intelligence (AI) methods and customized validation methods to ensure highly reliable and accurate sensing and perception of the environment for autonomous construction machinery. The presented methods have been developed, evaluated, and applied within the Competence Center \u00bbRobot Systems for Decontamination in Hazardous Environments\u00ab (ROBDEKON) for investigating and developing robotic systems for autonomous decontamination tasks. The objective of this article is to give a holistic, in-depth overview for the ML-based part of the perception pipeline for an autonomous construction machine working in unstructured environments.<\/jats:p>","DOI":"10.1515\/auto-2022-0054","type":"journal-article","created":{"date-parts":[[2023,3,10]],"date-time":"2023-03-10T00:52:41Z","timestamp":1678409561000},"page":"219-232","source":"Crossref","is-referenced-by-count":6,"title":["Machine learning for the perception of autonomous construction machinery"],"prefix":"10.1515","volume":"71","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0198-8634","authenticated-orcid":false,"given":"Nina Felicitas","family":"Heide","sequence":"first","affiliation":[{"name":"Fraunhofer IOSB, Karlsruhe , Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4715-8908","authenticated-orcid":false,"given":"Janko","family":"Petereit","sequence":"additional","affiliation":[{"name":"Fraunhofer IOSB, Karlsruhe , Germany"}]}],"member":"374","published-online":{"date-parts":[[2023,3,10]]},"reference":[{"key":"2023033111483235852_j_auto-2022-0054_ref_001","unstructured":"P. Woock, N. F. Heide, and D. Kuehn, \u201cRobotersysteme f\u2019\u00fcur die Dekontamination in menschenfeindlichen Umgebungen,\u201d in Proceedings at Leipziger Deponiefachtagung, vol.\u00a02020, 2020."},{"key":"2023033111483235852_j_auto-2022-0054_ref_002","doi-asserted-by":"crossref","unstructured":"J. Petereit, J. Beyerer, T. Asfour, et al.., \u201cROBDEKON: robotic systems for decontamination in hazardous environments,\u201d in IEEE SSRR, 2019.","DOI":"10.1109\/SSRR.2019.8848969"},{"key":"2023033111483235852_j_auto-2022-0054_ref_003","doi-asserted-by":"crossref","unstructured":"S. Khan, H. Rahmani, and S. A. A. Shah, A Guide to Convolutional Neural Networks for Computer Vision (Synthesis Lectures on Computer Vision), Morgan & Claypool Publishers, 2018.","DOI":"10.1007\/978-3-031-01821-3"},{"key":"2023033111483235852_j_auto-2022-0054_ref_004","doi-asserted-by":"crossref","unstructured":"T. Emter, C. Frese, A. Zube, and J. Petereit, \u201cAlgorithm toolbox for autonomous mobile robotic systems,\u201d ATZ Offhighway, vol.\u00a010, pp.\u00a048\u201353, 2017. https:\/\/doi.org\/10.1007\/s41321-017-0037-0.","DOI":"10.1007\/s41321-017-0037-0"},{"key":"2023033111483235852_j_auto-2022-0054_ref_005","doi-asserted-by":"crossref","unstructured":"A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, \u201cVision meets robotics: the KITTI dataset,\u201d Int. J. Robot. Res., vol.\u00a032, no.\u00a011, pp.\u00a01231\u20131237, 2013. https:\/\/doi.org\/10.1177\/0278364913491297.","DOI":"10.1177\/0278364913491297"},{"key":"2023033111483235852_j_auto-2022-0054_ref_006","doi-asserted-by":"crossref","unstructured":"P. Wolf, A. Vierling, J. Husemann, K. Berns, and P. Decker, \u201cExtending skills of autonomous off-road robots on the example of behavior-based edge compaction in a road construction scenario,\u201d in Commercial Vehicle Technology 2020\/2021, Springer Vieweg, 2021, pp. 51\u201362.","DOI":"10.1007\/978-3-658-29717-6_5"},{"key":"2023033111483235852_j_auto-2022-0054_ref_007","doi-asserted-by":"crossref","unstructured":"J. Ma, J. Luo, H. Pu, Y. Peng, S. Xie, and J. Gu, \u201cDesign, simulation and manufacturing of a tracked robot for nuclear accidents,\u201d in IEEE ROBIO, 2014.","DOI":"10.1109\/ROBIO.2014.7090601"},{"key":"2023033111483235852_j_auto-2022-0054_ref_008","unstructured":"H. Osumi, \u201cApplication of robot technologies to the disaster sites,\u201d in Report of JSME Research Committee on the Great East Japan Earthquake Disaster, 2014."},{"key":"2023033111483235852_j_auto-2022-0054_ref_009","unstructured":"F. Mascarich, T. Wilson, T. Dang, S. Khattak, C. Papachristos, and K. Alexis, Towards Robotically Supported Decommissioning of Nuclear Sites, 2017 [Online]. Available at: http:\/\/arxiv.org\/pdf\/1705.06401v1."},{"key":"2023033111483235852_j_auto-2022-0054_ref_010","doi-asserted-by":"crossref","unstructured":"S. Notheis, P. Kern, M. Mende, B. Hein, H. Worn, and S. Gentes, \u201cTowards an autonomous manipulator system for decontamination and release measurement,\u201d in IEEE\/ASME MESA, 2012.","DOI":"10.1109\/MESA.2012.6275572"},{"key":"2023033111483235852_j_auto-2022-0054_ref_011","doi-asserted-by":"crossref","unstructured":"G. Haskins, J. Kruecker, U. Kruger, et al.., \u201cLearning deep similarity metric for 3D MR-TRUS image registration,\u201d Int. J. Comput. Assist. Radiol. Surg., vol.\u00a014, no.\u00a03, pp.\u00a0417\u2013425, 2019. https:\/\/doi.org\/10.1007\/s11548-018-1875-7.","DOI":"10.1007\/s11548-018-1875-7"},{"key":"2023033111483235852_j_auto-2022-0054_ref_012","unstructured":"A. Dhall, K. Chelani, V. Radhakrishnan, and K. M. Krishna, \u201cLiDAR-camera calibration using 3D-3D point correspondences,\u201d arxiv\/1705.09785v1, 2017."},{"key":"2023033111483235852_j_auto-2022-0054_ref_013","doi-asserted-by":"crossref","unstructured":"J. K\u00fcmmerle, T. K\u00fchner, and M. Lauer, \u201cAutomatic calibration of multiple cameras and depth sensors with a spherical target,\u201d in IEEE\/RSJ IROS, 2018.","DOI":"10.1109\/IROS.2018.8593955"},{"key":"2023033111483235852_j_auto-2022-0054_ref_014","doi-asserted-by":"crossref","unstructured":"A. Pujol-Miro, J. Ruiz-Hidalgo, and J. R. Casas, \u201cRegistration of images to unorganized 3D point clouds using contour cues,\u201d in 25th European Signal Processing Conference, 2017.","DOI":"10.23919\/EUSIPCO.2017.8081173"},{"key":"2023033111483235852_j_auto-2022-0054_ref_015","doi-asserted-by":"crossref","unstructured":"N. Schneider, F. Piewak, C. Stiller, and U. Franke, \u201cRegnet: multimodal sensor registration using deep neural networks,\u201d in IEEE IV, 2017.","DOI":"10.1109\/IVS.2017.7995968"},{"key":"2023033111483235852_j_auto-2022-0054_ref_016","doi-asserted-by":"crossref","unstructured":"G. Iyer, R. K. Ram, J. K. Murthy, and K. M. Krishna, \u201cCalibNet: geometrically supervised extrinsic calibration using 3D spatial transformer networks,\u201d in IEEE\/RSJ IROS, 2018.","DOI":"10.1109\/IROS.2018.8593693"},{"key":"2023033111483235852_j_auto-2022-0054_ref_017","doi-asserted-by":"crossref","unstructured":"A. Handa, M. Bloesch, V. P\u0103tr\u0103ucean, S. Stent, J. McCormac, and A. Davison, \u201cGvnn: neural network library for geometric computer vision,\u201d in ECCV, 2016.","DOI":"10.1007\/978-3-319-49409-8_9"},{"key":"2023033111483235852_j_auto-2022-0054_ref_018","doi-asserted-by":"crossref","unstructured":"J. Jiao, R. Wang, W. Wang, S. Dong, Z. Wang, and W. Gao, \u201cLocal stereo matching with improved matching cost and disparity refinement,\u201d IEEE MultiMed., vol.\u00a021, no.\u00a04, pp.\u00a016\u201327, 2014. https:\/\/doi.org\/10.1109\/mmul.2014.51.","DOI":"10.1109\/MMUL.2014.51"},{"key":"2023033111483235852_j_auto-2022-0054_ref_019","unstructured":"H. Hirschm\u00fcller, \u201cAccurate and efficient stereo processing by semi-global matching and mutual information,\u201d in IEEE CVPR, 2005."},{"key":"2023033111483235852_j_auto-2022-0054_ref_020","doi-asserted-by":"crossref","unstructured":"Y. Boykov, O. Veksler, and R. Zabih, \u201cFast approximate energy minimization via graph cuts,\u201d IEEE Trans. Pattern Anal. Mach. Intell., vol.\u00a023, no.\u00a011, pp.\u00a01222\u20131239, 2001. https:\/\/doi.org\/10.1109\/34.969114.","DOI":"10.1109\/34.969114"},{"key":"2023033111483235852_j_auto-2022-0054_ref_021","doi-asserted-by":"crossref","unstructured":"J. \u010cech, J. Sanchez-Riera, and R. Horaud, \u201cScene flow estimation by growing correspondence seeds,\u201d in IEEE CVPR, 2011.","DOI":"10.1109\/CVPR.2011.5995442"},{"key":"2023033111483235852_j_auto-2022-0054_ref_022","unstructured":"J. Zbontar and Y. LeCun, \u201cStereo matching by training a convolutional neural network to compare image patches,\u201d J. Mach. Learn. Res., vol. 17, no. 1, pp. 2287\u20132318, 2016."},{"key":"2023033111483235852_j_auto-2022-0054_ref_023","doi-asserted-by":"crossref","unstructured":"S. Zagoruyko and N. Komodakis, \u201cLearning to compare image patches via convolutional neural networks,\u201d in IEEE CVPR, 2015.","DOI":"10.1109\/CVPR.2015.7299064"},{"key":"2023033111483235852_j_auto-2022-0054_ref_024","doi-asserted-by":"crossref","unstructured":"J. Zbontar and Y. LeCun, \u201cComputing the stereo matching cost with a convolutional neural network,\u201d in IEEE CVPR, 2015.","DOI":"10.1109\/CVPR.2015.7298767"},{"key":"2023033111483235852_j_auto-2022-0054_ref_025","doi-asserted-by":"crossref","unstructured":"W. Luo, A. G. Schwing, and R. Urtasun, \u201cEfficient deep learning for stereo matching,\u201d in IEEE CVPR, 2016.","DOI":"10.1109\/CVPR.2016.614"},{"key":"2023033111483235852_j_auto-2022-0054_ref_026","doi-asserted-by":"crossref","unstructured":"P. Wolf, T. Ropertz, and K. Berns, \u201cBehavior-based obstacle detection in off-road environments considering data quality,\u201d in International Conference on Informatics in Control, Automation and Robotics, 2020, pp.\u00a0786\u2013809.","DOI":"10.1007\/978-3-030-11292-9_39"},{"key":"2023033111483235852_j_auto-2022-0054_ref_027","doi-asserted-by":"crossref","unstructured":"P. Wolf and K. Berns, \u201cData-fusion for robust off-road perception considering data quality of uncertain sensors,\u201d in IEEE\/RSJ IROS, 2021, pp.\u00a06876\u20136883.","DOI":"10.1109\/IROS51168.2021.9636541"},{"key":"2023033111483235852_j_auto-2022-0054_ref_028","doi-asserted-by":"crossref","unstructured":"J. Long, E. Shelhamer, and T. Darrell, \u201cFully convolutional networks for semantic segmentation,\u201d in IEEE CVPR, 2015.","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"2023033111483235852_j_auto-2022-0054_ref_029","doi-asserted-by":"crossref","unstructured":"B. Wu, A. Wan, X. Yue, and K. Keutzer, \u201cSqueezeSeg: convolutional neural nets with recurrent CRF for real-time road-object segmentation from 3D LiDAR point cloud,\u201d in IEEE ICRA, 2018.","DOI":"10.1109\/ICRA.2018.8462926"},{"key":"2023033111483235852_j_auto-2022-0054_ref_030","doi-asserted-by":"crossref","unstructured":"A. Milioto, I. Vizzo, J. Behley, and C. Stachniss, \u201cRangeNet++: fast and accurate LiDAR semantic segmentation,\u201d in IEEE\/RSJ IROS, 2019.","DOI":"10.1109\/IROS40897.2019.8967762"},{"key":"2023033111483235852_j_auto-2022-0054_ref_031","unstructured":"C. R. Qi, H. Su, K. Mo, and L. J. Guibas, \u201cPointNet: deep learning on point sets for 3d classification and segmentation,\u201d in IEEE CVPR, 2017."},{"key":"2023033111483235852_j_auto-2022-0054_ref_032","unstructured":"B. Khaleghi, \u201cThe how of explainable AI: pre-modelling explainability,\u201d 2019 [Online]. Available at: https:\/\/towardsdatascience.com\/the-how-of-explainable-ai-pre-modelling-explainability-699150495fe4."},{"key":"2023033111483235852_j_auto-2022-0054_ref_033","doi-asserted-by":"crossref","unstructured":"J. D. Wang and H. C. Liu, \u201cAn approach to evaluate the fitness of one class structure via dynamic centroids,\u201d Expert Syst. Appl., vol. 38, no. 11, pp. 13764\u201313772, 2011. https:\/\/doi.org\/10.1016\/j.eswa.2011.04.178.","DOI":"10.1016\/j.eswa.2011.04.178"},{"key":"2023033111483235852_j_auto-2022-0054_ref_034","unstructured":"S. Mani, A. Sankaran, S. Tamilselvam, and A. Sethi, \u201cCoverage testing of deep learning models using dataset characterization,\u201d arXiv preprint: 1911.07309, 2019."},{"key":"2023033111483235852_j_auto-2022-0054_ref_035","unstructured":"The Alan Turing Institute, Impact Story: A Right to Explanation, The Alan Turing Institute, 2021 [Online]. Available at: https:\/\/www.turing.ac.uk\/research\/impact-stories\/a-right-to-explanation."},{"key":"2023033111483235852_j_auto-2022-0054_ref_036","unstructured":"European Parliament and the Council of the European Union, \u201cGeneral data protection regulation,\u201d 2016 [Online]. Available at: http:\/\/data.europa.eu\/eli\/reg\/2016\/679\/oj."},{"key":"2023033111483235852_j_auto-2022-0054_ref_037","unstructured":"N. Heide, T. Emter, and J. Petereit, \u201cCalibration of multiple 3D LiDAR sensors to a common vehicle frame,\u201d in ISR 2018; 50th International Symposium on Robotics, 2018."},{"key":"2023033111483235852_j_auto-2022-0054_ref_038","doi-asserted-by":"crossref","unstructured":"N. Heide, C. Frese, T. Emter, and J. Petereit, \u201cReal-time hyperspectral stereo processing for the generation of 3D depth information,\u201d in IEEE ICIP, 2018.","DOI":"10.1109\/ICIP.2018.8451194"},{"key":"2023033111483235852_j_auto-2022-0054_ref_039","doi-asserted-by":"crossref","unstructured":"N. F. Heide, P. Woock, M. Sauer, T. Leitritz, and M. Heizmann, \u201cUCSR: registration and fusion of cross-source 2D and 3D sensor data in unstructured environments,\u201d in Fusion, 2020.","DOI":"10.23919\/FUSION45008.2020.9190307"},{"key":"2023033111483235852_j_auto-2022-0054_ref_040","unstructured":"N. F. Heide, S. Gamer, and M. Heizmann, \u201cUEM-CNN: enhanced stereo matching for unstructured environments with dataset filtering and novel error metrics,\u201d in ISR 2020; 52th International Symposium on Robotics, 2020."},{"key":"2023033111483235852_j_auto-2022-0054_ref_041","doi-asserted-by":"crossref","unstructured":"N. F. Heide, A. Albrecht, and M. Heizmann, \u201cSET: stereo evaluation toolbox for combined performance assessment of camera systems, 3D reconstruction and visual SLAM,\u201d IEEE ICICSP, 2019.","DOI":"10.1109\/ICICSP48821.2019.8958548"},{"key":"2023033111483235852_j_auto-2022-0054_ref_042","doi-asserted-by":"crossref","unstructured":"N. F. Heide, A. Albrecht, and M. Heizmann, \u201cA step towards explainable artificial neural networks in image processing by dataset assessment,\u201d in Forum Bildverarbeitung, Karlsruhe, Germany, KIT Scientific Publishing, 2020.","DOI":"10.58895\/ksp\/1000124383-23"},{"key":"2023033111483235852_j_auto-2022-0054_ref_043","doi-asserted-by":"crossref","unstructured":"N. F. Heide, E. M\u00fcller, J. Petereit, and M. Heizmann, \u201cX3Seg: model-agnostic explanations for the semantic segmentation of 3D point clouds with prototypes and criticism,\u201d in IEEE ICIP, 2021.","DOI":"10.1109\/ICIP42928.2021.9506624"},{"key":"2023033111483235852_j_auto-2022-0054_ref_044","doi-asserted-by":"crossref","unstructured":"S. Kolski, D. Ferguson, M. Bellino, and R. Y. Siegwart, \u201cAutonomous driving in structured and unstructured environments,\u201d in IEEE IV, 2006, pp.\u00a0558\u2013563.","DOI":"10.1109\/IROS.2006.282302"},{"key":"2023033111483235852_j_auto-2022-0054_ref_045","doi-asserted-by":"crossref","unstructured":"N. F. Heide, A. Albrecht, T. Emter, and J. Petereit, \u201cPerformance optimization of autonomous platforms in unstructured outdoor environments using a novel constrained planning approach,\u201d in IEEE IV, 2019.","DOI":"10.1109\/IVS.2019.8813805"},{"key":"2023033111483235852_j_auto-2022-0054_ref_046","doi-asserted-by":"crossref","unstructured":"G. Elbaz, T. Avraham, and A. Fischer, \u201c3D point cloud registration for localization using a deep neural network auto-encoder,\u201d in IEEE CVPR, 2017.","DOI":"10.1109\/CVPR.2017.265"},{"key":"2023033111483235852_j_auto-2022-0054_ref_047","doi-asserted-by":"crossref","unstructured":"X. Huang, J. Zhang, L. Fan, Q. Wu, and C. Yuan, \u201cA systematic approach for cross-source point cloud registration by preserving macro and micro structures,\u201d IEEE Trans. Image Process., vol.\u00a026, no.\u00a07, pp.\u00a03261\u20133276, 2017. https:\/\/doi.org\/10.1109\/tip.2017.2695888.","DOI":"10.1109\/TIP.2017.2695888"},{"key":"2023033111483235852_j_auto-2022-0054_ref_048","unstructured":"P. J. Besl and N. D. McKay, \u201cMethod for registration of 3-d shapes,\u201d in Sensor Fusion IV: Control Paradigms and Data Structures, vol. 1611, SPIE, 1992, pp. 586\u2013606."},{"key":"2023033111483235852_j_auto-2022-0054_ref_049","doi-asserted-by":"crossref","unstructured":"I. Gheta, M. Heizmann, A. Belkin, and J. Beyerer, \u201cWorld modeling for autonomous systems,\u201d in KI 2010: Advances in Artificial Intelligence, vol. 6359, Berlin, Heidelberg, Springer, 2010.","DOI":"10.1007\/978-3-642-16111-7_20"},{"key":"2023033111483235852_j_auto-2022-0054_ref_050","doi-asserted-by":"crossref","unstructured":"M. Heizmann, I. Gheta, F. Puente Le\u00f3n, and J. Beyerer, \u201cInformationsfusion zur Umgebungsexploration,\u201d in Verteilte Messsysteme, Karlsruhe, Germany, KIT Scientific Publishing, 2010, pp. 133\u2013152.","DOI":"10.1524\/teme.2010.0098"},{"key":"2023033111483235852_j_auto-2022-0054_ref_051","doi-asserted-by":"crossref","unstructured":"F. Langer, A. Milioto, A. Haag, J. Behley, and C. Stachniss, \u201cDomain transfer for semantic segmentation of LiDAR data using deep neural networks,\u201d in IEEE\/RSJ IROS, 2020.","DOI":"10.1109\/IROS45743.2020.9341508"},{"key":"2023033111483235852_j_auto-2022-0054_ref_052","doi-asserted-by":"crossref","unstructured":"J. Behley, M. Garbade, A. Milioto, et al.., \u201cSemanticKITTI: a dataset for semantic scene understanding of LiDAR sequences,\u201d in IEEE\/CVF ICCV, 2019.","DOI":"10.1109\/ICCV.2019.00939"}],"container-title":["at - Automatisierungstechnik"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.degruyter.com\/document\/doi\/10.1515\/auto-2022-0054\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.degruyter.com\/document\/doi\/10.1515\/auto-2022-0054\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,16]],"date-time":"2024-10-16T02:55:55Z","timestamp":1729047355000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.degruyter.com\/document\/doi\/10.1515\/auto-2022-0054\/html"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,1]]},"references-count":52,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2023,3,10]]},"published-print":{"date-parts":[[2023,3,28]]}},"alternative-id":["10.1515\/auto-2022-0054"],"URL":"https:\/\/doi.org\/10.1515\/auto-2022-0054","relation":{},"ISSN":["0178-2312","2196-677X"],"issn-type":[{"type":"print","value":"0178-2312"},{"type":"electronic","value":"2196-677X"}],"subject":[],"published":{"date-parts":[[2023,3,1]]}}}