{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,5]],"date-time":"2025-11-05T06:25:40Z","timestamp":1762323940075,"version":"build-2065373602"},"reference-count":42,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2016,5,5]],"date-time":"2016-05-05T00:00:00Z","timestamp":1462406400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Sensing techniques are important for solving problems of uncertainty inherent to intelligent grasping tasks. The main goal here is to present a visual sensing system based on range imaging technology for robot manipulation of non-rigid objects. Our proposal provides a suitable visual perception system of complex grasping tasks to support a robot controller when other sensor systems, such as tactile and force, are not able to obtain useful data relevant to the grasping manipulation task. In particular, a new visual approach based on RGBD data was implemented to help a robot controller carry out intelligent manipulation tasks with flexible objects. The proposed method supervises the interaction between the grasped object and the robot hand in order to avoid poor contact between the fingertips and an object when there is neither force nor pressure data. This new approach is also used to measure changes to the shape of an object\u2019s surfaces and so allows us to find deformations caused by inappropriate pressure being applied by the hand\u2019s fingers. Test was carried out for grasping tasks involving several flexible household objects with a multi-fingered robot hand working in real time. Our approach generates pulses from the deformation detection method and sends an event message to the robot controller when surface deformation is detected. In comparison with other methods, the obtained results reveal that our visual pipeline does not use deformations models of objects and materials, as well as the approach works well both planar and 3D household objects in real time. In addition, our method does not depend on the pose of the robot hand because the location of the reference system is computed from a recognition process of a pattern located place at the robot forearm. The presented experiments demonstrate that the proposed method accomplishes a good monitoring of grasping task with several objects and different grasping configurations in indoor environments.<\/jats:p>","DOI":"10.3390\/s16050640","type":"journal-article","created":{"date-parts":[[2016,5,6]],"date-time":"2016-05-06T10:19:23Z","timestamp":1462529963000},"page":"640","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["3D Visual Data-Driven Spatiotemporal Deformations for Non-Rigid Object Grasping Using Robot Hands"],"prefix":"10.3390","volume":"16","author":[{"given":"Carlos","family":"Mateo","sequence":"first","affiliation":[{"name":"Computer Science Research Institute, University of Alicante, San Vicente del Raspeig, Alicante 03690, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9288-0161","authenticated-orcid":false,"given":"Pablo","family":"Gil","sequence":"additional","affiliation":[{"name":"Physics, Systems Engineering and Signal Theory Department, University of Alicante, San Vicente del Raspeig, Alicante 03690, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6261-9939","authenticated-orcid":false,"given":"Fernando","family":"Torres","sequence":"additional","affiliation":[{"name":"Physics, Systems Engineering and Signal Theory Department, University of Alicante, San Vicente del Raspeig, Alicante 03690, Spain"}]}],"member":"1968","published-online":{"date-parts":[[2016,5,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"347","DOI":"10.1016\/j.robot.2011.07.019","article-title":"Efficient models for grasp planning with a multi-fingered hand","volume":"60","author":"Saut","year":"2012","journal-title":"Robot. Auton. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"551","DOI":"10.1016\/j.robot.2010.01.003","article-title":"A strategy for grasping unknown objects based on co-planarity and colour information","volume":"58","author":"Popovic","year":"2010","journal-title":"Robot. Auton. Syst."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"452","DOI":"10.1016\/j.robot.2011.07.013","article-title":"Active learning of visual descriptors for grasping using non-parametric smoothed beta distributions","volume":"60","author":"Montesano","year":"2012","journal-title":"Robot. Auton. Syst."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1109\/TRO.2013.2289018","article-title":"Data-driven grasp synthesis\u2014A survey","volume":"30","author":"Bohg","year":"2014","journal-title":"IEEE Trans. Robot."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"558","DOI":"10.1109\/TRO.2011.2134130","article-title":"Tactile sensing for mobile manipulation","volume":"27","author":"Chitta","year":"2011","journal-title":"IEEE Trans. Robot."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1016\/j.sna.2011.02.038","article-title":"Tactile sensing for dexterous in-hand manipulation in robotics\u2014A review","volume":"167","author":"Yousef","year":"2011","journal-title":"Sens. Actuators A Phys."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1016\/j.robot.2015.07.015","article-title":"Tactile sensing in dexterous robot hands\u2014Review","volume":"74","author":"Kappassov","year":"2015","journal-title":"Robot. Auton. Syst."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1007\/978-1-4471-4664-3_4","article-title":"Sensors and methods for the evaluation of grasping","volume":"10","author":"Morales","year":"2013","journal-title":"Grasping Robot."},{"key":"ref_9","unstructured":"Li, Q., Elbrechter, C., Haschke, R., and Ritter, H. (2013, January 3\u20137). Integrating vision, haptics and proprioception into a feedback controller for in-hand manipulation of unknown objects. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Tokyo, Japan."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1007\/s11370-011-0088-5","article-title":"Towards cognitive grasping: Modelling of unknown objects and its corresponding grasp types","volume":"4","author":"Kim","year":"2011","journal-title":"Intell. Serv. Robot."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Bimbo, J., Seneviratne, L., Althoefer, K., and Liu, H. (2013, January 3\u20137). Combining touch and vision for the estimation of and object's pose during manipulation. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Tokyo, Japan.","DOI":"10.1109\/IROS.2013.6696931"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Mkhirtayan, A., and Burschka, D. (2012, January 28\u201331). Vision based haptic multisensory for manipulation of soft, fragile objects. Proceedings of the IEEE Sensors, Taipei, China.","DOI":"10.1109\/ICSENS.2012.6411035"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Cappelleri, D.J., Piazza, G., and Kumar, V. (2009, January 12\u201317). Two dimensional, vision based N force sensor for microrobotics. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Kobe, Japan.","DOI":"10.1109\/ROBOT.2009.5152647"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1002\/rob.1009","article-title":"Fusing force and vision feedback for manipulating deformable objects","volume":"18","author":"Luo","year":"2001","journal-title":"J. Robot. Syst."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Khalil, F.F., Curtis, P., and Payeur, P. (2010, January 15\u201316). Visual monitoring of Surface deformations on objects manipulated with a robotic hand. Proceedings of the IEEE International Workshop on Robotic and Sensors Environments (ROSE), Phoenix, AZ, USA.","DOI":"10.1109\/ROSE.2010.5675327"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1007\/978-3-642-28572-1_18","article-title":"Using near-field stereo vision for robotic grasping in cluttered environments","volume":"79","author":"Leeper","year":"2014","journal-title":"Exp. Robot."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1446","DOI":"10.1177\/0278364914536939","article-title":"Identification and active exploration of deformable object boundary constraints through robotic manipulation","volume":"33","author":"Boonvisut","year":"2014","journal-title":"Int. J. Robot. Res."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1109\/MRA.2015.2448951","article-title":"Benchmarking in manipulation research","volume":"22","author":"Calli","year":"2015","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"866","DOI":"10.1177\/0278364913512170","article-title":"Grasping deformable planar objects: Squeeze, stick\/slip analysis, and energy-based optimalities","volume":"33","author":"Jia","year":"2014","journal-title":"Int. J. Robot. Res."},{"key":"ref_20","unstructured":"Lin, H., Guo, F., Wang, F., and Jia, Y.-B. (June, January 31). Picking up soft 3D objects with two fingers. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China."},{"key":"ref_21","unstructured":"Shadow Robot, \u201cDexterous Hand\u201d. Available online: http:\/\/www.shadowrobot.com\/products\/dexterous-hand\/."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1016\/j.arcontrol.2010.09.001","article-title":"Multifingered robot hands: Control for grasping and manipulation","volume":"34","author":"Yoshikawa","year":"2010","journal-title":"Ann. Rev. Control"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1457","DOI":"10.1109\/TRO.2013.2275651","article-title":"Model-free Visually servoed deformation control of elastic objects by robot manipulators","volume":"29","author":"Liu","year":"2013","journal-title":"IEEE Trans. Robot."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1462","DOI":"10.1177\/0278364914529355","article-title":"On the visual deformation servoing of compliant objects: Uncalibrated control methods and experiments","volume":"33","author":"Liu","year":"2014","journal-title":"Int. J. Robot. Res."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Berenson, D. (2013, January 3\u20137). Manipulation of deformable objects without modeling and simulating deformation. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan.","DOI":"10.1109\/IROS.2013.6697007"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TIM.2014.2315734","article-title":"Navigation and Manipulation Planning using a Visuo-haptic Sensor on a Mobile Platform","volume":"63","author":"Alt","year":"2014","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Sun, L., Aragon-Camarasa, G., Rogers, S., and Siebert, J.P. (2015, January 26\u201330). Accurate garment surface analysis using and active stereo robot head with application to dual-arm flattening. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Seattle, MA, USA.","DOI":"10.1109\/ICRA.2015.7138998"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1437","DOI":"10.3390\/s120201437","article-title":"Accuracy and Resolution of Depth Data for Indoor Mapping Applications","volume":"12","author":"Khoshelham","year":"2012","journal-title":"Sensors"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1152","DOI":"10.1016\/j.compind.2013.04.005","article-title":"An efficient method for fully automatic 3D digitization of unknown objects","volume":"64","author":"Khalfaoui","year":"2013","journal-title":"Comput. Ind."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Delgado, A., Jara, C.A., Torres, F., and Mateo, C.M. (2015, January 19\u201321). Control of Robot Fingers with Adaptable Tactile Servoing to Manipulate Deformable Objects. Proceedings of the Robot 2015: Second Iberian Robotics Conference, Lisboa, Portugal.","DOI":"10.1007\/978-3-319-27146-0_7"},{"key":"ref_31","unstructured":"Kragic, D., Miller, A.T., and Allen, P.K. (2001, January 21\u201326). Real-time tracking meets online grasp planning. Proceedings of the on IEEE International Conference on Robotics and Automation, Seoul, Korea."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Mateo, C.M., Gil, P., and Torres, F. (2015). Visual perception for the 3D recognition of geometric pieces in robotic manipulation. Int. J. Adv. Manuf. Technol, 1\u201315.","DOI":"10.1007\/s00170-015-7708-8"},{"key":"ref_33","first-page":"1","article-title":"Finger Readjustment Algoritm for Object Manipulation Based on Tactile Information","volume":"10","author":"Corrales","year":"2013","journal-title":"Int. J. Adv. Robot. Syst."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Corrales, J.A., Perderau, V., and Torres, F. (2013, January 6\u201310). Multi-fingered robotic hand planner for object reconfiguration through a rolling contact evolution model. Proccedings of the IEEE International Conference on Robotics and Automation (ICRA), Karlsruhe, Germany.","DOI":"10.1109\/ICRA.2013.6630638"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Mateo, C.M., Gil, P., and Torres, F. (2014, January 2\u20134). A Performance evaluation of surface normal-based descriptors for recognition of objects using CAD-Models. Proccedings of the 11th International Conference on Informatics in Control, Automation and Robotics (ICINCO), Vienna, Austria.","DOI":"10.5220\/0005056504280435"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1109\/MRA.2012.2206675","article-title":"Tutorial: Point cloud library: Three dimensional object recognition and 6DoF pose estimation","volume":"19","author":"Aldoma","year":"2012","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Papon, J., Abramov, A., Schoeler, M., and Woergoetter, F. (2013, January 23\u201328). Voxel Cloud Connectivity Segmentation\u2014Supervoxels from PointClouds. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Porland, OR, USA.","DOI":"10.1109\/CVPR.2013.264"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"741","DOI":"10.1016\/0031-3203(89)90010-1","article-title":"O(log n) Bimodality Analysis","volume":"22","author":"Philps","year":"1989","journal-title":"Pattern Recognit."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"2280","DOI":"10.1016\/j.patcog.2014.01.005","article-title":"Automatic generation and detection of highly reliable fiducial markers under occlusion","volume":"47","year":"2014","journal-title":"Pattern Recognit."},{"key":"ref_40","unstructured":"Fox, D. (2001, January 3\u20138). KLD-Sampling: Adaptive Particle Filters. Proceedings of the Neural Information Processing Systems Conference (NIPS), Vancouver, BC, Canada."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"985","DOI":"10.1177\/0278364903022012001","article-title":"Adapting the sample size in particle filters through KLD-sampling","volume":"22","author":"Fox","year":"2003","journal-title":"Int. J. Robot. Res."},{"key":"ref_42","unstructured":"Mateo, C.M., Gil, P., and Torres, F. (2015, January 21\u201323). Analysis of Shapes to Measure Surface: An approach for detection of deformations. Proceedings of the 12th International Conference on Informatics in Control, Automation and Robotics (ICINCO), Colmar, France."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/16\/5\/640\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T19:23:19Z","timestamp":1760210599000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/16\/5\/640"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,5,5]]},"references-count":42,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2016,5]]}},"alternative-id":["s16050640"],"URL":"https:\/\/doi.org\/10.3390\/s16050640","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2016,5,5]]}}}