{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T03:28:51Z","timestamp":1760239731479,"version":"build-2065373602"},"reference-count":58,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2020,12,26]],"date-time":"2020-12-26T00:00:00Z","timestamp":1608940800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP17K06277"],"award-info":[{"award-number":["JP17K06277"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004423","name":"Waseda University","doi-asserted-by":"publisher","award":["2018A-047, 2020Q-015 and 2020C-780"],"award-info":[{"award-number":["2018A-047, 2020Q-015 and 2020C-780"]}],"id":[{"id":"10.13039\/501100004423","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This paper introduces a system that can estimate the deformation process of a deformed flat object (folded plane) and generate the input data for a robot with human-like dexterous hands and fingers to reproduce the same deformation of another similar object. The system is based on processing RGB data and depth data with three core techniques: a weighted graph clustering method for non-rigid point matching and clustering; a refined region growing method for plane detection on depth data based on an offset error defined by ourselves; and a novel sliding checking model to check the bending line and adjacent relationship between each pair of planes. Through some evaluation experiments, we show the improvement of the core techniques to conventional studies. By applying our approach to different deformed papers, the performance of the entire system is confirmed to have around 1.59 degrees of average angular error, which is similar to the smallest angular discrimination of human eyes. As a result, for the deformation of the flat object caused by folding, if our system can get at least one feature point cluster on each plane, it can get spatial information of each bending line and each plane with acceptable accuracy. The subject of this paper is a folded plane, but we will develop it into a robotic reproduction of general object deformation.<\/jats:p>","DOI":"10.3390\/s21010105","type":"journal-article","created":{"date-parts":[[2020,12,27]],"date-time":"2020-12-27T20:52:21Z","timestamp":1609102341000},"page":"105","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Estimation of Flat Object Deformation Using RGB-D Sensor for Robot Reproduction"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1026-5902","authenticated-orcid":false,"given":"Xin","family":"He","sequence":"first","affiliation":[{"name":"Graduate School of Information, Production and Systems (IPS), Waseda University, Kitakyushu 808-0135, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7776-2984","authenticated-orcid":false,"given":"Takafumi","family":"Matsumaru","sequence":"additional","affiliation":[{"name":"Graduate School of Information, Production and Systems (IPS), Waseda University, Kitakyushu 808-0135, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,12,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1131","DOI":"10.1109\/34.244675","article-title":"Finite-element methods for active contour models and balloons for 2-D and 3-D images","volume":"15","author":"Cohen","year":"1993","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"580","DOI":"10.1109\/34.216727","article-title":"Shape and nonrigid motion estimation through physics-based synthesis","volume":"15","author":"Metaxas","year":"1993","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Urtasun, R., Fleet, D.J., Hertzmann, A., and Fua, P. (2005, January 17\u201321). Priors for people tracking from small training sets. Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV\u201905), Beijing, China.","DOI":"10.1109\/ICCV.2005.193"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Perriollat, M., and Bartoli, A. (2007, January 17\u201322). A Quasi-Minimal Model for Paper-Like Surfaces. Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.","DOI":"10.1109\/CVPR.2007.383356"},{"key":"ref_5","unstructured":"McInerney, T., and Terzopoulos, D. (1993, January 11\u201314). A finite element model for 3D shape reconstruction and nonrigid motion tracking. Proceedings of the 1993 (4th) International Conference on Computer Vision, Berlin, Germany."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Malti, A., Bartoli, A., and Collins, T. (2011, January 6\u201313). A pixel-based approach to template-based monocular 3D reconstruction of deformable surfaces. Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain.","DOI":"10.1109\/ICCVW.2011.6130447"},{"key":"ref_7","unstructured":"Bregler, C., Hertzmann, A., and Biermann, H. (2000, January 15). Recovering non-rigid 3D shape from image streams. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662), Hilton Head Island, SC, USA."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"878","DOI":"10.1109\/TPAMI.2007.70752","article-title":"Nonrigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors","volume":"30","author":"Torresani","year":"2008","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Akhter, I., Sheikh, Y., and Khan, S. (2009, January 20\u201325). In defense of orthonormality constraints for nonrigid structure from motion. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPRW.2009.5206620"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Gotardo, P.F.U., and Martinez, A.M. (2011, January 6\u201313). Kernel non-rigid structure from motion. Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain.","DOI":"10.1109\/ICCV.2011.6126319"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Varol, A., Salzmann, M., Tola, E., and Fua, P. (October, January 29). Template-free monocular reconstruction of deformable surfaces. Proceedings of the 2009 IEEE 12th International Conference on Computer Vision, Kyoto, Japan.","DOI":"10.1109\/ICCV.2009.5459403"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Taylor, J., Jepson, A.D., and Kutulakos, K.N. (2010, January 13\u201318). Non-rigid structure from locally-rigid motion. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA.","DOI":"10.1109\/CVPR.2010.5540002"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Russell, C., Fayad, J., and Agapito, L. (2011). Energy based multiple model fitting for non-rigid structure from motion. CVPR, 3009\u20133016.","DOI":"10.1109\/CVPR.2011.5995383"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Salzmann, M., Hartley, R., and Fua, P. (2007, January 14\u201321). Convex Optimization for Deformable Surface 3-D Tracking. Proceedings of the 2007 IEEE 11th International Conference on Computer Vision, Rio de Janeiro, Brazil.","DOI":"10.1109\/ICCV.2007.4409031"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Wang, C., Li, X., and Liu, Y. (2010, January 23\u201326). Monocular 3D Tracking of Deformable Surfaces Using Linear Programming. Proceedings of the 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey.","DOI":"10.1109\/ICPR.2010.423"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Tsoli, A., and Argyros, A.A. (2019, January 27\u201328). Patch-Based Reconstruction of a Textureless Deformable 3D Surface from a Single RGB Image. Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, Korea.","DOI":"10.1109\/ICCVW.2019.00498"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Bednarik, J., Fua, P., and Salzmann, M. (2018, January 5\u20138). Learning to Reconstruct Texture-Less Deformable Surfaces from a Single View. Proceedings of the 2018 International Conference on 3D Vision (3DV), Verona, Italy.","DOI":"10.1109\/3DV.2018.00075"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Ye, G., Liu, Y., Hasler, N., Ji, X., Dai, Q., and Theobalt, C. (2012, January 7\u201313). Performance capture of interacting characters with handheld kinects. Proceedings of the European Conference on Computer Vision, Florence, Italy.","DOI":"10.1007\/978-3-642-33709-3_59"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Elanattil, S., Moghadam, P., Denman, S., Sridharan, S., and Fookes, C. (2018, January 10\u201313). Skeleton Driven Non-Rigid Motion Tracking and 3D Reconstruction. Proceedings of the 2018 Digital Image Computing: Techniques and Applications (DICTA), Canberra, Australia.","DOI":"10.1109\/DICTA.2018.8615797"},{"key":"ref_20","unstructured":"Lim, J., and Yang, M.-H. (2005, January 20\u201325). A direct method for modeling non-rigid motion with thin plate spline. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\u201905), San Diego, CA, USA."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Kozlov, C., Slavcheva, M., and Ilic, S. (2018, January 5\u20138). Patch-Based Non-rigid 3D Reconstruction from a Single Depth Stream. Proceedings of the 2018 International Conference on 3D Vision (3DV), Verona, Italy.","DOI":"10.1109\/3DV.2018.00016"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Tsoli, A., and Argyros, A.A. (2016, January 25\u201328). Tracking Deformable Surfaces That Undergo Topological Changes Using an RGB-D Camera. Proceedings of the 2016 Fourth International Conference on 3D Vision (3DV), Stanford, CA, USA.","DOI":"10.1109\/3DV.2016.42"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Paulus, C.J., Haouchine, N., Cazier, D., and Cotin, S. (October, January 29). Augmented Reality during Cutting and Tearing of Deformable Objects. Proceedings of the 2015 IEEE International Symposium on Mixed and Augmented Reality, Fukuoka, Japan.","DOI":"10.1109\/ISMAR.2015.19"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Zhang, Q., Fu, B., Ye, M., and Yang, R. (2014, January 24\u201327). Quality Dynamic Human Body Modeling Using a Single Low-cost Depth Camera. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.92"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Dou, M., Taylor, J., Fuchs, H., Fitzgibbon, A., and Izadi, S. (2015, January 7\u201312). 3D scanning deformable objects with a single RGBD sensor. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298647"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"5966","DOI":"10.1109\/TIP.2017.2740624","article-title":"Templateless Non-Rigid Reconstruction and Motion Tracking With a Single RGB-D Camera","volume":"26","author":"Wang","year":"2017","journal-title":"IEEE Trans. Image Process."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Slavcheva, M., Baust, M., Cremers, D., and Ilic, S. (2017, January 21\u201326). KillingFusion: Non-rigid 3D Reconstruction without Correspondences. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.581"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"4746","DOI":"10.1109\/TIP.2019.2909197","article-title":"Global 3D Non-Rigid Registration of Deformable Objects Using a Single RGB-D Camera","volume":"28","author":"Yang","year":"2019","journal-title":"IEEE Trans. Image Process."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"2876","DOI":"10.1109\/TIP.2013.2246521","article-title":"Global Propagation of Affine Invariant Features for Robust Matching","volume":"22","author":"Cui","year":"2013","journal-title":"IEEE Trans. Image Process."},{"key":"ref_30","unstructured":"Chum, O., and Matas, J. (2005, January 20\u201325). Matching with PROSAC - progressive sample consensus. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\u201905), San Diego, CA, USA."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Zhu, J., and Lyu, M.R. (2007, January 17\u201322). Progressive Finite Newton Approach To Real-time Nonrigid Surface Detection. Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.","DOI":"10.1109\/CVPR.2007.383200"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Lowe, D.G. (1999, January 20\u201327). Object recognition from local scale-invariant features. Proceedings of the Seventh IEEE International Conference on Computer Vision, Kerkyra, Greece.","DOI":"10.1109\/ICCV.1999.790410"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1706","DOI":"10.1109\/TIP.2014.2307478","article-title":"Robust Point Matching via Vector Field Consensus","volume":"23","author":"Ma","year":"2014","journal-title":"IEEE Trans. Image Process."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1109\/34.121791","article-title":"A method for registration of 3-D shapes","volume":"14","author":"Besl","year":"1992","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_35","unstructured":"Chui, H., and Rangarajan, A. (2000, January 15). A new algorithm for non-rigid point matching. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662), Hilton Head Island, SC, USA."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Sofka, M., Yang, G., and Stewart, C.V. (2007, January 17\u201322). Simultaneous Covariance Driven Correspondence (CDC) and Transformation Estimation in the Expectation Maximization Framework. Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.","DOI":"10.1109\/CVPR.2007.383166"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"643","DOI":"10.1109\/TPAMI.2006.81","article-title":"Robust point matching for nonrigid shapes by preserving local neighborhood structures","volume":"28","author":"Zheng","year":"2006","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"4369","DOI":"10.1109\/TIP.2012.2208980","article-title":"Robust Weighted Graph Transformation Matching for Rigid and Nonrigid Image Registration","volume":"21","author":"Izadi","year":"2012","journal-title":"IEEE Trans. Image Process."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Leordeanu, M., and Hebert, M. (2005, January 17\u201321). A spectral technique for correspondence problems using pairwise constraints. Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV\u201905), Beijing, China.","DOI":"10.1109\/ICCV.2005.20"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"371","DOI":"10.1109\/TPAMI.2016.2545659","article-title":"Probabilistic Model for Robust Affine and Non-Rigid Point Set Matching","volume":"39","author":"Qu","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Tarashima, S., Shimamura, J., Kinebuchi, T., and Satah, S. (2018, January 7\u201310). Keypoint Matching for Non-Rigid Object via Locally Consistent Visual Pattern Mining. Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece.","DOI":"10.1109\/ICIP.2018.8451016"},{"key":"ref_42","unstructured":"Macropol, K. (2009). Clustering on Graphs: The Markov Cluster Algorithm (MCL), University of Utrecht."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1145\/358669.358692","article-title":"Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography","volume":"24","author":"Fischler","year":"1981","journal-title":"Commun. ACM"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"2771","DOI":"10.1109\/TCYB.2014.2316282","article-title":"NCC-RANSAC: A Fast Plane Extraction Method for 3-D Range Data Segmentation","volume":"44","author":"Qian","year":"2014","journal-title":"IEEE Trans. Cybern."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Yue, W., Lu, J., Zhou, W., and Miao, Y. (2018, January 9\u201311). A new plane segmentation method of point cloud based on mean shift and RANSAC. Proceedings of the 2018 Chinese Control And Decision Conference (CCDC), Shenyang, China.","DOI":"10.1109\/CCDC.2018.8407394"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Xu, D., Li, F., and Wei, H. (2019, January 19\u201321). 3D Point Cloud Plane Segmentation Method Based on RANSAC And Support Vector Machine. Proceedings of the 2019 14th IEEE Conference on Industrial Electronics and Applications (ICIEA), Xi\u2019an, China.","DOI":"10.1109\/ICIEA.2019.8834367"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1016\/j.jvcir.2013.04.001","article-title":"Continuous plane detection in point-cloud data based on 3D Hough Transform","volume":"25","author":"Hulik","year":"2014","journal-title":"J. Vis. Commun. Image Represent."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1016\/0031-3203(91)90073-E","article-title":"A probabilistic Hough transform","volume":"24","author":"Kiryati","year":"1991","journal-title":"Pattern Recognit."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Huang, H., and Brenner, C. (2011, January 11\u201313). Rule-based roof plane detection and segmentation from laser point clouds. Proceedings of the 2011 Joint Urban Remote Sensing Event, Munich, Germany.","DOI":"10.1109\/JURSE.2011.5764777"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1016\/j.patcog.2007.04.003","article-title":"Real-time line detection through an improved Hough transform voting scheme","volume":"41","author":"Fernandes","year":"2008","journal-title":"Pattern Recognit."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1016\/S0921-8890(03)00007-1","article-title":"Learning compact 3D models of indoor and outdoor environments with a mobile robot","volume":"44","author":"Burgard","year":"2003","journal-title":"Robot. Auton. Syst."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Poppinga, J., Vaskevicius, N., Birk, A., and Pathak, K. (2008, January 22\u201326). Fast plane detection and polygonalization in noisy 3D range images. Proceedings of the 2008 IEEE\/RSJ International Conference on Intelligent Robots and Systems, Nice, France.","DOI":"10.1109\/IROS.2008.4650729"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Xiao, J., Adler, B., and Zhang, H. (2012, January 13\u201315). 3D point cloud registration based on planar surfaces. Proceedings of the 2012 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), Hamburg, Germany.","DOI":"10.1109\/MFI.2012.6343035"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Smisek, J., Jancosek, M., and Pajdla, T. (2013). 3D with Kinect. Consumer Depth Cameras for Computer Vision, Springer.","DOI":"10.1007\/978-1-4471-4640-7_1"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1016\/j.cviu.2016.07.005","article-title":"Unsupervised object region proposals for RGB-D indoor scenes","volume":"154","author":"Deng","year":"2017","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"1282","DOI":"10.1016\/j.robot.2014.03.013","article-title":"Approximate triangulation and region growing for efficient segmentation and smoothing of range images","volume":"62","author":"Holz","year":"2014","journal-title":"Robot. Auton. Syst."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1109\/TCSVT.2017.2780181","article-title":"Robust Plane Detection Using Depth Information From a Consumer Depth Camera","volume":"29","author":"Jin","year":"2019","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"1375","DOI":"10.1016\/S0042-6989(03)00119-6","article-title":"Orientation discrimination in foveal and extra-foveal vision: Effects of stimulus bandwidth and contrast","volume":"43","author":"Sally","year":"2003","journal-title":"Vis. Res."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/1\/105\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:46:27Z","timestamp":1760179587000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/1\/105"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12,26]]},"references-count":58,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,1]]}},"alternative-id":["s21010105"],"URL":"https:\/\/doi.org\/10.3390\/s21010105","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2020,12,26]]}}}