{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T16:05:20Z","timestamp":1780502720624,"version":"3.54.1"},"reference-count":45,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2021,3,9]],"date-time":"2021-03-09T00:00:00Z","timestamp":1615248000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"The Ministry of Trade, Industry and Energy grant funded by the Korean government (No.10080686, Development of fundamental technology of tactile sensor using image information)","award":["10080686"],"award-info":[{"award-number":["10080686"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This work describes the development of a vision-based tactile sensor system that utilizes the image-based information of the tactile sensor in conjunction with input loads at various motions to train the neural network for the estimation of tactile contact position, area, and force distribution. The current study also addresses pragmatic aspects, such as choice of the thickness and materials for the tactile fingertips and surface tendency, etc. The overall vision-based tactile sensor equipment interacts with an actuating motion controller, force gauge, and control PC (personal computer) with a LabVIEW software on it. The image acquisition was carried out using a compact stereo camera setup mounted inside the elastic body to observe and measure the amount of deformation by the motion and input load. The vision-based tactile sensor test bench was employed to collect the output contact position, angle, and force distribution caused by various randomly considered input loads for motion in X, Y, Z directions and RxRy rotational motion. The retrieved image information, contact position, area, and force distribution from different input loads with specified 3D position and angle are utilized for deep learning. A convolutional neural network VGG-16 classification modelhas been modified to a regression network model and transfer learning was applied to suit the regression task of estimating contact position and force distribution. Several experiments were carried out using thick and thin sized tactile sensors with various shapes, such as circle, square, hexagon, for better validation of the predicted contact position, contact area, and force distribution.<\/jats:p>","DOI":"10.3390\/s21051920","type":"journal-article","created":{"date-parts":[[2021,3,9]],"date-time":"2021-03-09T21:36:21Z","timestamp":1615325781000},"page":"1920","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":66,"title":["Vision-Based Tactile Sensor Mechanism for the Estimation of Contact Position and Force Distribution Using Deep Learning"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4165-0021","authenticated-orcid":false,"given":"Vijay","family":"Kakani","sequence":"first","affiliation":[{"name":"Information and Communication Engineering, Inha University, 100 Inharo, Nam-gu, Incheon 22212, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xuenan","family":"Cui","sequence":"additional","affiliation":[{"name":"Information and Communication Engineering, Inha University, 100 Inharo, Nam-gu, Incheon 22212, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mingjie","family":"Ma","sequence":"additional","affiliation":[{"name":"VisionIn Inc. Global R&amp;D Center, 704 Ace Gasan Tower, 121 Digital-ro, Geumcheon-gu, Seoul 08505, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4232-3804","authenticated-orcid":false,"given":"Hakil","family":"Kim","sequence":"additional","affiliation":[{"name":"Information and Communication Engineering, Inha University, 100 Inharo, Nam-gu, Incheon 22212, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Umbaugh, S.E. (2010). Digital Image Processing and Analysis: Human and Computer Vision Applications with CVIPtools, CRC Press.","DOI":"10.1201\/9781439802069"},{"key":"ref_2","first-page":"100033","article-title":"A critical review on computer vision and artificial intelligence in food industry","volume":"2","author":"Kakani","year":"2020","journal-title":"J. Agric. Food Res."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Kakani, V., Kim, H., Basivi, P.K., and Pasupuleti, V.R. (2020). Surface Thermo-Dynamic Characterization of Poly (Vinylidene Chloride-Co-Acrylonitrile)(P (VDC-co-AN)) Using Inverse-Gas Chromatography and Investigation of Visual Traits Using Computer Vision Image Processing Algorithms. Polymers, 12.","DOI":"10.3390\/polym12081631"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Shimonomura, K. (2019). Tactile image sensors employing camera: A review. Sensors, 19.","DOI":"10.3390\/s19183933"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Kakani, V., Kim, H., Lee, J., Ryu, C., and Kumbham, M. (2020). Automatic Distortion Rectification of Wide-Angle Images Using Outlier Refinement for Streamlining Vision Tasks. Sensors, 20.","DOI":"10.3390\/s20030894"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Kakani, V., Kim, H., Kumbham, M., Park, D., Jin, C.B., and Nguyen, V.H. (2019). Feasible Self-Calibration of Larger Field-of-View (FOV) Camera Sensors for the Advanced Driver-Assistance System (ADAS). Sensors, 19.","DOI":"10.3390\/s19153369"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1016\/j.mechatronics.2017.11.002","article-title":"Robotic tactile perception of object properties: A review","volume":"48","author":"Luo","year":"2017","journal-title":"Mechatronics"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Li, W., Konstantinova, J., Noh, Y., Alomainy, A., and Althoefer, K. (2018, January 25\u201327). Camera-based force and tactile sensor. Proceedings of the Annual Conference Towards Autonomous Robotic Systems, Bristol, UK.","DOI":"10.1007\/978-3-319-96728-8_37"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Sferrazza, C., and D\u2019Andrea, R. (2019). Design, motivation and evaluation of a full-resolution optical tactile sensor. Sensors, 19.","DOI":"10.3390\/s19040928"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Yuan, W., Mo, Y., Wang, S., and Adelson, E.H. (2018, January 21\u201325). Active clothing material perception using tactile sensing and deep learning. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.","DOI":"10.1109\/ICRA.2018.8461164"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Yuan, W., Li, R., Srinivasan, M.A., and Adelson, E.H. (2015, January 26\u201330). Measurement of shear and slip with a GelSight tactile sensor. Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, USA.","DOI":"10.1109\/ICRA.2015.7139016"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1177\/027836499000900301","article-title":"Tactile sensing mechanisms","volume":"9","author":"Fearing","year":"1990","journal-title":"Int. J. Robot. Res."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"558","DOI":"10.1109\/TRO.2011.2134130","article-title":"Tactile sensing for mobile manipulation","volume":"27","author":"Chitta","year":"2011","journal-title":"IEEE Trans. Robot."},{"key":"ref_14","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"661","DOI":"10.1080\/01691864.2019.1632222","article-title":"Recent progress in tactile sensing and sensors for robotic manipulation: Can we turn tactile sensing into vision?","volume":"33","author":"Yamaguchi","year":"2019","journal-title":"Adv. Robot."},{"key":"ref_16","unstructured":"Hosoda, K., Tada, Y., and Asada, M. (October, January 30). Internal representation of slip for a soft finger with vision and tactile sensors. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems, Lausanne, Switzerland."},{"key":"ref_17","unstructured":"Kolker, A., Jokesch, M., and Thomas, U. (2016, January 21\u201322). An optical tactile sensor for measuring force values and directions for several soft and rigid contacts. Proceedings of the ISR 2016: 47st International Symposium on Robotics, VDE, Munich, Germany."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"3340","DOI":"10.1109\/LRA.2018.2852797","article-title":"Slip detection with a biomimetic tactile sensor","volume":"3","author":"James","year":"2018","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"720","DOI":"10.1016\/j.robot.2007.05.003","article-title":"Neural network models of haptic shape perception","volume":"55","author":"Johnsson","year":"2007","journal-title":"Robot. Auton. Syst."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1881","DOI":"10.1109\/TIM.2019.2919354","article-title":"A novel dynamic-vision-based approach for tactile sensing applications","volume":"69","author":"Naeini","year":"2019","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Ma, D., Donlon, E., Dong, S., and Rodriguez, A. (2019, January 20\u201324). Dense tactile force estimation using GelSlim and inverse FEM. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.","DOI":"10.1109\/ICRA.2019.8794113"},{"key":"ref_22","unstructured":"Wilson, A., Wang, S., Romero, B., and Adelson, E. (2020). Design of a Fully Actuated Robotic Hand With Multiple Gelsight Tactile Sensors. arXiv."},{"key":"ref_23","first-page":"5","article-title":"Event-driven visual-tactile sensing and learning for robots","volume":"4","author":"Taunyazov","year":"2020","journal-title":"Perception"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"473","DOI":"10.1109\/TRO.2011.2125350","article-title":"Tactile-object recognition from appearance information","volume":"27","author":"Pezzementi","year":"2011","journal-title":"IEEE Trans. Robot."},{"key":"ref_25","unstructured":"Zhang, Y., Yuan, W., Kan, Z., and Wang, M.Y. (2020, January 16\u201318). Towards Learning to Detect and Predict Contact Events on Vision-based Tactile Sensors. Proceedings of the Conference on Robot Learning, Boston, MA, USA."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"472","DOI":"10.1109\/56.20431","article-title":"Planar and finger-shaped optical tactile sensors for robotic applications","volume":"4","author":"Begej","year":"1988","journal-title":"IEEE J. Robot. Autom."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Lepora, N.F., and Ward-Cherrier, B. (October, January 28). Superresolution with an optical tactile sensor. Proceedings of the 2015 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany.","DOI":"10.1109\/IROS.2015.7353744"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"2037","DOI":"10.1109\/JSEN.2010.2104316","article-title":"Robust slippage degree estimation based on reference update of vision-based tactile sensor","volume":"11","author":"Ito","year":"2011","journal-title":"IEEE Sens. J."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Yang, X.D., Grossman, T., Wigdor, D., and Fitzmaurice, G. (2012, January 7\u201310). Magic finger: Always-available input through finger instrumentation. Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, Cambridge, MA, USA.","DOI":"10.1145\/2380116.2380137"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s40638-017-0058-2","article-title":"Object recognition combining vision and touch","volume":"4","author":"Corradi","year":"2017","journal-title":"Robot. Biomim."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"993","DOI":"10.1007\/s10514-018-9777-7","article-title":"iCLAP: Shape recognition by combining proprioception and touch sensing","volume":"43","author":"Luo","year":"2019","journal-title":"Auton. Robot."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Piacenza, P., Dang, W., Hannigan, E., Espinal, J., Hussain, I., Kymissis, I., and Ciocarlie, M. (June, January 29). Accurate contact localization and indentation depth prediction with an optics-based tactile sensor. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.","DOI":"10.1109\/ICRA.2017.7989117"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Johnson, M.K., and Adelson, E.H. (2009, January 20\u201325). Retrographic sensing for the measurement of surface texture and shape. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPRW.2009.5206534"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2010324.1964941","article-title":"Microgeometry capture using an elastomeric sensor","volume":"30","author":"Johnson","year":"2011","journal-title":"ACM Trans. Graph. (TOG)"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Yuan, W., Srinivasan, M.A., and Adelson, E.H. (2016, January 9\u201314). Estimating object hardness with a gelsight touch sensor. Proceedings of the 2016 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea.","DOI":"10.1109\/IROS.2016.7759057"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"545","DOI":"10.1109\/TRO.2011.2121130","article-title":"Learning dynamic tactile sensing with robust vision-based training","volume":"27","author":"Kroemer","year":"2011","journal-title":"IEEE Trans. Robot."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Meier, M., Patzelt, F., Haschke, R., and Ritter, H.J. (2016, January 6\u20139). Tactile convolutional networks for online slip and rotation detection. Proceedings of the International Conference on Artificial Neural Networks, Barcelona, Spain.","DOI":"10.1007\/978-3-319-44781-0_2"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Chuah, M.Y., and Kim, S. (2016, January 16\u201321). Improved normal and shear tactile force sensor performance via least squares artificial neural network (lsann). Proceedings of the 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden.","DOI":"10.1109\/ICRA.2016.7487123"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1850001","DOI":"10.1142\/S0219843618500019","article-title":"Active tactile transfer learning for object discrimination in an unstructured environment using multimodal robotic skin","volume":"15","author":"Kaboli","year":"2018","journal-title":"Int. J. Humanoid Robot."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"6872","DOI":"10.1109\/JSEN.2019.2912968","article-title":"CNN-based methods for object recognition with high-resolution tactile sensors","volume":"19","author":"Gandarias","year":"2019","journal-title":"IEEE Sens. J."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Sferrazza, C., and D\u2019Andrea, R. (2018). Transfer learning for vision-based tactile sensing. arXiv.","DOI":"10.1109\/IROS40897.2019.8967571"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1109\/TOH.2009.47","article-title":"Finger-shaped gelforce: Sensor for measuring surface traction fields for robotic hand","volume":"3","author":"Sato","year":"2009","journal-title":"IEEE Trans. Haptics"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"173438","DOI":"10.1109\/ACCESS.2019.2956882","article-title":"Ground truth force distribution for learning-based tactile sensing: A finite element approach","volume":"7","author":"Sferrazza","year":"2019","journal-title":"IEEE Access"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"419","DOI":"10.5254\/1.3547752","article-title":"Durometer hardness and the stress-strain behavior of elastomeric materials","volume":"76","author":"Qi","year":"2003","journal-title":"Rubber Chem. Technol."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Moeslund, T.B. (2012). BLOB analysis. Introduction to Video and Image Processing, Springer.","DOI":"10.1007\/978-1-4471-2503-7"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/5\/1920\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:35:45Z","timestamp":1760160945000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/5\/1920"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,9]]},"references-count":45,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2021,3]]}},"alternative-id":["s21051920"],"URL":"https:\/\/doi.org\/10.3390\/s21051920","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,9]]}}}