{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,11]],"date-time":"2025-11-11T15:53:59Z","timestamp":1762876439606,"version":"build-2065373602"},"reference-count":36,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2023,6,25]],"date-time":"2023-06-25T00:00:00Z","timestamp":1687651200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000780","name":"European Commission","doi-asserted-by":"publisher","award":["101070136"],"award-info":[{"award-number":["101070136"]}],"id":[{"id":"10.13039\/501100000780","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotics"],"abstract":"<jats:p>Robotic manipulation in cluttered environments is one of the challenges roboticists are currently facing. When the objects to handle are delicate fresh fruits, grasping is even more challenging. Detecting and localizing fruits with the accuracy necessary to grasp them is very difficult due to the large variability in the aspect and dimensions of each item. This paper proposes a solution that exploits a state-of-the-art neural network and a novel enhanced 6D pose estimation method that integrates the depth map with the neural network output. Even with an accurate localization, grasping fruits with a suitable force to avoid slippage and damage at the same time is another challenge. This work solves this issue by resorting to a grasp controller based on tactile sensing. Depending on the specific application scenario, grasping a fruit might be impossible without colliding with other objects or other fruits. Therefore, a non-prehensile manipulation action is here proposed to push items hindering the grasp of a detected fruit. The pushing from an initial location to a target one is performed by a model predictive controller taking into account the unavoidable delay in the perception and computing pipeline of the robotic system. Experiments with real fresh fruits demonstrate that the overall proposed approach allows a robot to successfully grasp apples in various situations.<\/jats:p>","DOI":"10.3390\/robotics12040092","type":"journal-article","created":{"date-parts":[[2023,6,26]],"date-time":"2023-06-26T05:39:13Z","timestamp":1687757953000},"page":"92","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Non-Prehensile Manipulation Actions and Visual 6D Pose Estimation for Fruit Grasping Based on Tactile Sensing"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8364-6728","authenticated-orcid":false,"given":"Marco","family":"Costanzo","sequence":"first","affiliation":[{"name":"Dipartimento di Ingegneria, Universit\u00e0 degli Studi della Campania Luigi Vanvitelli, Via Roma 29, 81031 Aversa, Italy"}]},{"given":"Marco","family":"De Simone","sequence":"additional","affiliation":[{"name":"Dipartimento di Ingegneria, Universit\u00e0 degli Studi della Campania Luigi Vanvitelli, Via Roma 29, 81031 Aversa, Italy"}]},{"given":"Sara","family":"Federico","sequence":"additional","affiliation":[{"name":"Dipartimento di Ingegneria, Universit\u00e0 degli Studi della Campania Luigi Vanvitelli, Via Roma 29, 81031 Aversa, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6550-0573","authenticated-orcid":false,"given":"Ciro","family":"Natale","sequence":"additional","affiliation":[{"name":"Dipartimento di Ingegneria, Universit\u00e0 degli Studi della Campania Luigi Vanvitelli, Via Roma 29, 81031 Aversa, Italy"}]}],"member":"1968","published-online":{"date-parts":[[2023,6,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"533","DOI":"10.1177\/027836499601500602","article-title":"Stable Pushing: Mechanics, Controllability, and Planning","volume":"15","author":"Lynch","year":"1999","journal-title":"Int. J. Robot. Res."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Yu, K.T., Bauza, M., Fazeli, N., and Rodriguez, A. (2016, January 9\u201314). More than a million ways to be pushed. A high-fidelity experimental dataset of planar pushing. Proceedings of the 2016 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea.","DOI":"10.1109\/IROS.2016.7758091"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Yu, K.T., Leonard, J., and Rodriguez, A. (October, January 28). Shape and pose recovery from planar pushing. Proceedings of the 2015 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany.","DOI":"10.1109\/IROS.2015.7353523"},{"key":"ref_4","unstructured":"Lynch, K. (1993, January 26\u201330). Estimating the friction parameters of pushed objects. Proceedings of the 1993 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS\u201993), Yokohama, Japan."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1177\/027836498600500303","article-title":"Mechanics and Planning of Manipulator Pushing Operations","volume":"5","author":"Mason","year":"1986","journal-title":"Int. J. Robot. Res."},{"key":"ref_6","unstructured":"Akella, S., and Mason, M. (1992, January 12\u201314). Posing polygonal objects in the plane by pushing. Proceedings of the 1992 IEEE International Conference on Robotics and Automation, Nice, France."},{"key":"ref_7","unstructured":"Lynch, K., Maekawa, H., and Tanie, K. (1992, January 7\u201310). Manipulation And Active Sensing By Pushing Using Tactile Feedback. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems, Raleigh, NC, USA."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"755","DOI":"10.1177\/0278364920913938","article-title":"Reactive planar non-prehensile manipulation with hybrid model predictive control","volume":"39","author":"Hogan","year":"2020","journal-title":"Int. J. Robot. Res."},{"key":"ref_9","unstructured":"Hogan, F.R., and Rodriguez, A. (2020). Algorithmic Foundations of Robotics XII: Proceedings of the Twelfth Workshop on the Algorithmic Foundations of Robotics, Springer International Publishing."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Costanzo, M., Simone, M.D., Federico, S., Natale, C., and Pirozzi, S. (2023, January 3\u20136). Enhanced 6D Pose Estimation for Robotic Fruit Picking. Proceedings of the 9th International Conference on Control, Decision and Information Technologies (CoDIT\u201923), Rome, Italy.","DOI":"10.1109\/CoDIT58514.2023.10284072"},{"key":"ref_11","unstructured":"Mason, M. (1986). Manipulator Grasping and Pushing Operations, Massachusetts Institute of Technology. Technical Report."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1016\/0043-1648(91)90104-3","article-title":"Planar sliding with dry friction Part 1. Limit surface and moment function","volume":"143","author":"Goyal","year":"1991","journal-title":"Wear"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"320","DOI":"10.1115\/1.2899703","article-title":"Fixture planning with friction","volume":"113","author":"Lee","year":"1991","journal-title":"J. Eng. Ind."},{"key":"ref_14","unstructured":"Billard, A., Dragan, A., Peters, J., and Morimoto, J. (2018, January 29\u201331). A Data-Efficient Approach to Precise and Controlled Pushing. Proceedings of the 2nd Conference on Robot Learning, Z\u00fcrich, Switzerland."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Kopicki, M., Zurek, S., Stolkin, R., M\u00f6rwald, T., and Wyatt, J. (2011, January 9\u201313). Learning to predict how rigid objects behave under simple manipulation. Proceedings of the 2011 IEEE International Conference on Robotics and Automation, Shanghai, China.","DOI":"10.1109\/ICRA.2011.5980295"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Byravan, A., and Fox, D. (June, January 29). SE3-nets: Learning rigid body motion using deep neural networks. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.","DOI":"10.1109\/ICRA.2017.7989023"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., and Funkhouser, T. (2018, January 1\u20135). Learning Synergies between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning. Proceedings of the 2018 Proceedings of the IEEE International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.","DOI":"10.1109\/IROS.2018.8593986"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"876","DOI":"10.1109\/TPAMI.2011.206","article-title":"Gradient Response Maps for Real-Time Detection of Textureless Objects","volume":"34","author":"Hinterstoisser","year":"2012","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Li, Y., Wang, G., Ji, X., Xiang, Y., and Fox, D. (2018, January 8\u201314). DeepIM: Deep Iterative Matching for 6D Pose Estimation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01231-1_42"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Rios-Cabrera, R., and Tuytelaars, T. (2013, January 1\u20138). Discriminatively Trained Templates for 3D Object Detection: A Real Time Scalable Approach. Proceedings of the 2013 IEEE International Conference on Computer Vision, Sydney, Australia.","DOI":"10.1109\/ICCV.2013.256"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Xiang, Y., Schmidt, T., Narayanan, V., and Fox, D. (2018, January 26\u201330). PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. Proceedings of the XIV Robotics Science and Systems Conference (RSS), Pittsburgh, PA, USA.","DOI":"10.15607\/RSS.2018.XIV.019"},{"key":"ref_22","unstructured":"Billard, A., Dragan, A., Peters, J., and Morimoto, J. (2018, January 29\u201331). Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects. Proceedings of the 2nd Conference on Robot Learning, Z\u00fcrich, Switzerland."},{"key":"ref_23","unstructured":"Hou, T., Ahmadyan, A., Zhang, L., Wei, J., and Grundmann, M. (2020). MobilePose: Real-Time Pose Estimation for Unseen Objects with Weak Shape Supervision. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Lin, Y., Tremblay, J., Tyree, S., Vela, P.A., and Birchfield, S. (2022, January 23\u201327). Single-Stage Keypoint-based Category-level Object Pose Estimation from an RGB Image. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Philadelphia, PA, USA.","DOI":"10.1109\/ICRA46639.2022.9812299"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Pavlakos, G., Zhou, X., Chan, A., Derpanis, K.G., and Daniilidis, K. (June, January 29). 6-DoF object pose from semantic keypoints. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.","DOI":"10.1109\/ICRA.2017.7989233"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Wang, H., Sridhar, S., Huang, J., Valentin, J., Song, S., and Guibas, L.J. (2019, January 15\u201320). Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00275"},{"key":"ref_27","unstructured":"Manhardt, F., Nickel, M., Meier, S., Minciullo, L., and Navab, N. (2020). CPS: Class-level 6D Pose and Shape Estimation From Monocular Images. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Vedaldi, A., Bischof, H., Brox, T., and Frahm, J.M. (2020). Computer Vision\u2014ECCV 2020, Springer International Publishing.","DOI":"10.1007\/978-3-030-58589-1"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Costanzo, M., De Maria, G., Natale, C., and Pirozzi, S. (2019). Design and Calibration of a Force\/Tactile Sensor for Dexterous Manipulation. Sensors, 19.","DOI":"10.3390\/s19040966"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1109\/TRO.2019.2944130","article-title":"Two-Fingered In-Hand Object Handling Based on Force\/Tactile Feedback","volume":"36","author":"Costanzo","year":"2020","journal-title":"IEEE Trans. Robot."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"102545","DOI":"10.1016\/j.mechatronics.2021.102545","article-title":"Control of robotic object pivoting based on tactile sensing","volume":"76","author":"Costanzo","year":"2021","journal-title":"Mechatronics"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1007\/s11263-008-0152-6","article-title":"EPnP: An accurate O(n) solution to the PnP problem","volume":"81","author":"Lepetit","year":"2009","journal-title":"Int. J. Comput. Vis."},{"key":"ref_33","unstructured":"Morrical, N., Tremblay, J., Lin, Y., Tyree, S., Birchfield, S., Pascucci, V., and Wald, I. (2021). NViSII: A Scriptable Tool for Photorealistic Image Generation. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"723","DOI":"10.1023\/A:1021711402723","article-title":"Application of interior-point methods to model predictive control","volume":"99","author":"Rao","year":"1998","journal-title":"J. Optim. Theory Appl."},{"key":"ref_35","unstructured":"Gurobi Optimization, LLC (2023, June 22). Available online: https:\/\/www.gurobi.com."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1109\/MRA.2005.1577023","article-title":"ViSP for visual servoing: A generic software platform with a wide class of robot control skills","volume":"12","author":"Marchand","year":"2006","journal-title":"IEEE Robot. Autom. Mag."}],"container-title":["Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2218-6581\/12\/4\/92\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:00:30Z","timestamp":1760126430000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2218-6581\/12\/4\/92"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,25]]},"references-count":36,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2023,8]]}},"alternative-id":["robotics12040092"],"URL":"https:\/\/doi.org\/10.3390\/robotics12040092","relation":{},"ISSN":["2218-6581"],"issn-type":[{"type":"electronic","value":"2218-6581"}],"subject":[],"published":{"date-parts":[[2023,6,25]]}}}