{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,19]],"date-time":"2026-02-19T05:43:20Z","timestamp":1771479800179,"version":"3.50.1"},"reference-count":44,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2021,2,19]],"date-time":"2021-02-19T00:00:00Z","timestamp":1613692800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This paper addresses the validation of a robust vision-based pose estimation technique using a Photonic Mixer Device (PMD) sensor as a single visual sensor in the close-range phase of spacecraft rendezvous. First, it was necessary to integrate the developed hybrid navigation technique for the PMD sensor into the hardware-in-the-loop (HIL) rendezvous system developed by the German Aerospace Center (DLR). Thereafter, HIL tests were conducted using the European Proximity Operation Simulator (EPOS) with sun simulation and in total darkness. For the future missions with an active sensor, e.g., a PMD camera, it could be useful to use only its own illumination during the rendezvous phase in penumbra or umbra, instead of additional flash light. In some tests, the rotational rate of the target object was also tuned. Unlike the rendezvous tests in other works, here we present for the first time closed-loop approaches with only depth and amplitude images of a PMD sensor. For the rendezvous tests in the EPOS laboratory, the Argos3D camera was used at the range of 8 to 5.5 m; the performance showed promising results.<\/jats:p>","DOI":"10.3390\/s21041455","type":"journal-article","created":{"date-parts":[[2021,2,19]],"date-time":"2021-02-19T09:06:10Z","timestamp":1613725570000},"page":"1455","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Hardware-in-the-Loop Simulations with Umbra Conditions for Spacecraft Rendezvous with PMD Visual Sensors"],"prefix":"10.3390","volume":"21","author":[{"given":"Ksenia","family":"Klionovska","sequence":"first","affiliation":[{"name":"German Space Operations Center, German Aerospace Center, M\u00fcnchener Stra\u00dfe 20, 82234 We\u00dfling, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matthias","family":"Burri","sequence":"additional","affiliation":[{"name":"German Space Operations Center, German Aerospace Center, M\u00fcnchener Stra\u00dfe 20, 82234 We\u00dfling, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,2,19]]},"reference":[{"key":"ref_1","unstructured":"(2020). ESA\u2019s Annual Space Environment Report, ESA Space Debris Office."},{"key":"ref_2","unstructured":"Gaias, G., Ardaens, J.S., and Schultz, C. (June, January 29). The AVANTI experiment: Flight results. Proceedings of the 10th International ESA Conference on Guidance, Navigation & Control Systems, Salzburg, Austria."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Sharma, S., and D\u2019Amico, S. (2019, January 13\u201317). Pose estimation for non-cooperative Spacecraft Rendezvous using Neural Networks. Proceedings of the AIAA\/AAS Space Flight Mechanics Meeting, Maui, HI, USA.","DOI":"10.1109\/AERO.2018.8396425"},{"key":"ref_4","unstructured":"Benninghoff, H., Rems, F., Risse, E.A., Irmisch, P., Ernst, I., Brunner, B., Stelzer, M., Lampariello, R., Krenn, R., and Reiner, M. (2018, January 6\u20139). RICADOS-rendezvous, inspection, capturing and detumbling by orbital servicing. Proceedings of the 7th International Conference on Astrodynamics Tools and Techniques, Oberpfaffenhofen, Germany."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Lingenauber, M., Strobl, K.H., Oumer, N.W., and Kriegel, S. (2017, January 4\u201311). Benefits of plenoptic cameras for robot vision during close range on-orbit servicing maneuvers. Proceedings of the 2017 IEEE Aerospace Conference, Big Sky, MT, USA.","DOI":"10.1109\/AERO.2017.7943666"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Shtark, T., and Gurfil, P. (2017). Tracking a non-cooperative target using real-time stereovision-based control: An experimental study. Sensors, 17.","DOI":"10.3390\/s17040735"},{"key":"ref_7","unstructured":"Hanlon, E.A.S., and Piper, G. (2017, January 3\u20136). COTS 3D camera for on-orbit proximity operations demonstration. Proceedings of the 33rd Space Symposium, Technical Track, Colorado Springs, CO, USA."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"418","DOI":"10.1016\/j.actaastro.2018.10.033","article-title":"Proximity operations about and identification of non-cooperative resident space objects using stereo imaging","volume":"155","author":"Davis","year":"2019","journal-title":"Acta Astronaut."},{"key":"ref_9","unstructured":"Yilmaz, O., Aouf, N., Majewski, L., Sanchez-Gestido, M., and Ortega, G. (June, January 29). Using infrared based relative navigation for active debris removal. Proceedings of the 10th International ESA Conference on Guidance, Navigation & Control Systems, Salzburg, Austria."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"100548","DOI":"10.1016\/j.paerosci.2019.05.008","article-title":"Review of the robustness and applicability of monocular pose estimation systems for relative navigation with an uncooperative spacecraft","volume":"110","author":"Cassinis","year":"2019","journal-title":"Prog. Aerosp. Sci."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Cavrois, B., Vergnol, A., Donnard, A., Casiez, P., and Mongrard, O. (2015, January 5\u20139). LIRIS demonstrator on ATV5: A step beyond for european non cooperative navigation system. Proceedings of the AIAA Guidance, Navigation, and Control Conference, Kissimmee, FL, USA.","DOI":"10.2514\/6.2015-0336"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Opromolla, R., Fasano, G., Rufino, G., and Grassi, M. (2017). Hardware in the loop performance assessment of LIDAR-based spacecraft pose determination. Sensors, 17.","DOI":"10.3390\/s17102197"},{"key":"ref_13","unstructured":"Rems, F., Moreno Gonz\u00e1lez, J.A., Boge, T., Tuttas, S., and Stilla, U. (2015, January 11\u201313). Fast initial pose estimation of spacecraft from LIDAR point cloud data. Proceedings of the 13th Symposium on Advanced Space Technologies in Robotics and Automation, Noordwijk, The Netherlands."},{"key":"ref_14","unstructured":"Ventura, J. (2021, February 19). Autonomous proximity operations for noncooperative space target. Dissertation, Technische Universit\u00e4t M\u00fcnchen. Available online: https:\/\/mediatum.ub.tum.de\/doc\/1324836\/1324836.pdf."},{"key":"ref_15","unstructured":"Tzschichholz, T. (2014). Relative pose estimation of known rigid objects using a novel approach to high-level PMD-\/CCD- sensor data fusion with regard to applications in space. [Ph.D. Thesis, Universit\u00e4t W\u00fcrzburg]."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Klionovska, K., Ventura, J., Benninghoff, H., and Huber, F. (2018). Close range tracking of an uncooperative target in a sequence of photonic mixer device (PMD) images. Robotics, 7.","DOI":"10.3390\/robotics7010005"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"214","DOI":"10.1111\/j.1467-8659.2007.01016.x","article-title":"Efficient RANSAC for Point-Cloud Shape Detection","volume":"26","author":"Schnabel","year":"2007","journal-title":"Comput. Graph. Forum"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Jasiobedzki, P., Se, S., Pan, T., Umasuthan, M., and Greenspan, M. (2005, January 20). Autonomous Satellite Rendezvous and Docking using LIDAR and Model Based Vision. Proceedings of the Conference: SPIE Spaceborne Sensors II, Orlando, FL, USA.","DOI":"10.1117\/12.604011"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"298","DOI":"10.1016\/j.actaastro.2016.05.007","article-title":"LIDAR-based Relative Navigation with Respect to Noncooperative Objects","volume":"126","author":"Woods","year":"2016","journal-title":"Acta Astronaut."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Liu, L., Zhao, G., and Bo, Y. (2016). Point Cloud Based Relative Pose Estimation of a Satellite in Close Range. Sensors, 16.","DOI":"10.3390\/s16060824"},{"key":"ref_21","unstructured":"Cropp, A., and Palmer, P. (2002, January 14\u201318). Pose Estimation and Relative Orbit Determination of Nearby Target Microsatellite using Passive Imagery. Proceedings of the 5th Cranfield Conference on Dynamics and Control of Systems and Structures in Space, Cambridge, UK."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1504\/IJSPACESE.2014.060600","article-title":"Pose Estimation of an Uncooperative Spacecraft from Actual Space Imagery","volume":"2","author":"Benn","year":"2014","journal-title":"Int. J. Space Sci. Eng."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Petit, A., March, E., and Kanani, K. (2012, January 7\u201312). Tracking Complex Targets for Space Rendezvous and Debris Removal Applications. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems, IROS\u201912, Vilamoura-Algarve, Portugal.","DOI":"10.1109\/IROS.2012.6386083"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Pressigout, M., Marchand, E., and Memin, E. (2008, January 12\u201315). Hybrid Tracking Approach using Optical Flow and Pose Estimation. Proceedings of the 15th IEEE International Conference on Image Processing, San Diego, CA, USA.","DOI":"10.1109\/ICIP.2008.4712356"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Opromolla, R., Fasano, G., Rufino, G., and Grassi, M. (2015). A Model-Based 3D Template Matching Technique for Pose Acquisition of an Uncooperative Space Object. Sensors, 15.","DOI":"10.3390\/s150306360"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1117\/12.421129","article-title":"Template matching using fast normalized cross correlation","volume":"Volume 4387","author":"Casasent","year":"2001","journal-title":"Optical Pattern Recognition XII"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"4638","DOI":"10.1109\/TAES.2020.2999148","article-title":"Neural Network-Based Pose Estimation for Noncooperative Spacecraft Rendezvous","volume":"56","author":"Sharma","year":"2020","journal-title":"IEEE Trans. Aerosp. Electron. Syst."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Phisannupawong, T., Kamsing, P., Torteeka, P., Channumsin, S., Sawangwit, U., Hematulin, W., Jarawan, T., Somjit, T., Yooyen, S., and Delahaye, D. (2020). Vision-Based Spacecraft Pose Estimation via a Deep Convolutional Neural Network for Noncooperative Docking Operations. Aerospace, 7.","DOI":"10.3390\/aerospace7090126"},{"key":"ref_29","unstructured":"Risse, E.A., Schwenk, K., Bennighoff, H., and Rems, F. (2020, January 1\u20133). Guidance, Navigation and Control for Autonomous Close-Range-Rendezvous. Proceedings of the DLRK Online, Munich, Germany."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"535","DOI":"10.1007\/s12567-017-0192-2","article-title":"End-to-end simulation and verification of GNC and robotic systems considering both space segment and ground segment","volume":"10","author":"Benninghoff","year":"2018","journal-title":"CEAS Space J."},{"key":"ref_31","unstructured":"Rems, F., Risse, E.A., and Benninghoff, H. (June, January 29). Rendezvous GNC-system for autonomous orbital servicing of uncooperative targets. Proceedings of the 10th International ESA Conference on Guidance, Navigation & Control Systems, Salzburg, Austria."},{"key":"ref_32","unstructured":"Vera-Rodriguez, R., Fierrez, J., and Morales, A. (2019). Experimental analysis of measurements fusion for pose estimation using PMD sensor. Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, Springer International Publishing."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"A107","DOI":"10.17815\/jlsrf-3-155","article-title":"European proximity operations simulator 2.0 (EPOS)-a robotic-based rendezvous and docking simulator","volume":"3","author":"Benninghoff","year":"2017","journal-title":"J. Large-Scale Res. Facil."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Rems, F., Bennighoff, H., Mietner, C., Risse, E.A., and Burri, M. (2020, January 1\u20133). 10-Year Anniversary of the European Proximity Operations Simulator 2.0\u2014Looking Back at Test Campaigns, Rendezvous Research and Facility Improvements. Proceedings of the DLRK Online, Munich, Germany.","DOI":"10.3390\/aerospace8090235"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1016\/j.actaastro.2014.05.023","article-title":"Development and hardware-in-the-loop test of a guidance, navigation and control system for on-orbit servicing","volume":"102","author":"Benninghoff","year":"2014","journal-title":"Acta Astronaut."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Langmann, B. (2014). Wide area 2D\/3D Imaging. Development, Analysis and Applications, Springer.","DOI":"10.1007\/978-3-658-06457-0"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"31096","DOI":"10.1364\/OE.25.031096","article-title":"CS-ToF: High-resolution compressive time-of-flight imaging","volume":"25","author":"Li","year":"2017","journal-title":"Opt. Express"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1109\/TCI.2015.2510506","article-title":"A Comparative Error Analysis of Current Time-of-Flight Sensors","volume":"2","author":"Placht","year":"2016","journal-title":"IEEE Trans. Comput. Imaging"},{"key":"ref_39","unstructured":"Strobl, K.H., Sepp, W., Fuchs, S., Paredes, C., Smisek, M., and Arbter, K. (2018). DLR CalDe and DLR CalLab. Institute of Robotics and Mechatronics, German Aerospace Center (DLR)."},{"key":"ref_40","unstructured":"Klionovska, K., Benninghoff, H., and Strobl, K.H. (2017, January 20\u201322). PMD Camera-and HandEye-Calibration for On-Orbit Servicing Test Scenarios On the Ground. Proceedings of the In 14th Symposium on Advanced Space Technologies in Robotis and Automation (ASTRA), Leiden, the Netherlands."},{"key":"ref_41","unstructured":"Grossman, J., Grossman, M., and Katz, R. (1980). The First Systems of Weighted Differential and Integral Calculus, Archimedes Foundation."},{"key":"ref_42","unstructured":"Klionovska, K. (2020). Analysis and Optimization of PMD-Sensor Data for Rendezvous Applications in Space. [Ph.D. Thesis, Universit\u00e4t der Bundeswehr M\u00fcnchen]. Available online: https:\/\/athene-forschung.unibw.de\/85311?show_id=134113."},{"key":"ref_43","unstructured":"Fraunhofer Institute (2021, February 19). Research News. Available online: https:\/\/www.fraunhofer.de\/content\/dam\/zv\/en\/press-media\/2017\/August\/ResearchNews\/rn08_2017_FHR_Defunct%20satellites.pdf."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"3774","DOI":"10.1029\/2019GC008392","article-title":"The Climate Data Toolbox for MATLAB","volume":"20","author":"Greene","year":"2019","journal-title":"Geochem. Geophys. Geosyst."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/4\/1455\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:26:17Z","timestamp":1760160377000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/4\/1455"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,19]]},"references-count":44,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2021,2]]}},"alternative-id":["s21041455"],"URL":"https:\/\/doi.org\/10.3390\/s21041455","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,19]]}}}