{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T18:23:18Z","timestamp":1780338198022,"version":"3.54.1"},"reference-count":38,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2017,10,26]],"date-time":"2017-10-26T00:00:00Z","timestamp":1508976000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In this paper, we present an interaction force estimation method that uses visual information rather than that of a force sensor. Specifically, we propose a novel deep learning-based method utilizing only sequential images for estimating the interaction force against a target object, where the shape of the object is changed by an external force. The force applied to the target can be estimated by means of the visual shape changes. However, the shape differences in the images are not very clear. To address this problem, we formulate a recurrent neural network-based deep model with fully-connected layers, which models complex temporal dynamics from the visual representations. Extensive evaluations show that the proposed learning models successfully estimate the interaction forces using only the corresponding sequential images, in particular in the case of three objects made of different materials, a sponge, a PET bottle, a human arm, and a tube. The forces predicted by the proposed method are very similar to those measured by force sensors.<\/jats:p>","DOI":"10.3390\/s17112455","type":"journal-article","created":{"date-parts":[[2017,10,26]],"date-time":"2017-10-26T06:48:14Z","timestamp":1509000494000},"page":"2455","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":56,"title":["Inferring Interaction Force from Visual Information without Using Physical Force Sensors"],"prefix":"10.3390","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8895-0411","authenticated-orcid":false,"given":"Wonjun","family":"Hwang","sequence":"first","affiliation":[{"name":"Department of Software and Computer Engineering, Ajou University, 206 Worldcup-ro, Yeongtong-gu, Suwon 16499, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Soo-Chul","family":"Lim","sequence":"additional","affiliation":[{"name":"Department of Mechanical, Robotics and Energy Engineering, Dongguk University, 30, Pildong-ro 1gil, Jung-gu, Seoul 04620, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2017,10,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Grosu, V., Grosu, S., Vanderborght, B., Lefeber, D., and Rodriguez-Guerrero, C. (2017). Multi-Axis Force Sensor for Human\u2013Robot Interaction Sensing in a Rehabilitation Robotic Device. Sensors, 17.","DOI":"10.3390\/s17061294"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1109\/LRA.2015.2505061","article-title":"A Conformable Force\/Tactile Skin for Physical Human-Robot Interaction","volume":"1","author":"Cirillo","year":"2016","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_3","unstructured":"Landi, C.T., Ferraguti, F., Sabattini, L., Secchi, C., and Fantuzzi, C. (June, January 29). Admittance Control Parameter Adaptation for Physical Human-Robot Interaction. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Singapore."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"482","DOI":"10.1109\/TCSVT.2014.2359098","article-title":"Fusing Sorted Random Projections for Robust Texture and Material Classification","volume":"25","author":"Liu","year":"2015","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2407","DOI":"10.1109\/TMM.2016.2598140","article-title":"Deep learning for surface material classification using haptic and visual information","volume":"18","author":"Zheng","year":"2016","journal-title":"IEEE Trans. Multimed."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"4899","DOI":"10.3390\/s140304899","article-title":"A Finger-Shaped Tactile Sensor for Fabric Surfaces Evaluation by 2-Dimensional Active Sliding Touch","volume":"14","author":"Hu","year":"2014","journal-title":"Sensors"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"705","DOI":"10.1177\/0278364914549607","article-title":"Deep learning for detecting robotic grasps","volume":"34","author":"Lenz","year":"2015","journal-title":"Int. J. Robot. Res."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Su, Z., Hausman, K., Chebotar, Y., Molchanov, A., Loeb, G.E., Sukhatme, G.S., and Schaal, S. (2015, January 3\u20135). Force estimation and slip detection\/classification for grip control using a biomimetic tactile sensor. Proceedings of the 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), Seoul, Korea.","DOI":"10.1109\/HUMANOIDS.2015.7363558"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Liu, Y., Han, H., Liu, T., Yi, J., Li, Q., and Inoue, Y. (2016). A Novel Tactile Sensor with Electromagnetic Induction and Its Application on Stick-Slip Interaction Detection. Sensors, 16.","DOI":"10.3390\/s16040430"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1007\/s10514-012-9291-2","article-title":"Force feedback exploiting tactile and proximal force\/torque sensing","volume":"33","author":"Fumagalli","year":"2012","journal-title":"Auton. Robots"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Zhang, H., Wu, R., Li, C., Zang, X., Zhang, X., Jin, H., and Zhao, J. (2017). A Force-Sensing System on Legs for Biomimetic Hexapod Robots Interacting with Unstructured Terrain. Sensors, 17.","DOI":"10.3390\/s17071514"},{"key":"ref_12","unstructured":"Dahiya, R.S., Gori, M., Metta, G., and Sandini, G. (2009, January 28). Better manipulation with human inspired tactile sensing. Proceedings of the RSS 2009 workshop on Understanding the Human Hand for Advancing Robotic Manipulation, Seattle, WA, USA."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1214","DOI":"10.1109\/TRO.2015.2473515","article-title":"Force sensor integrated surgical forceps for minimally invasive robotic surgery","volume":"31","author":"Kim","year":"2015","journal-title":"IEEE Trans. Robot."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1080\/13645700310023069","article-title":"Force-sensitive tactile sensor for minimal access surgery","volume":"13","author":"Kattavenos","year":"2004","journal-title":"Minim. Invasive Ther. Allied Technol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"074301","DOI":"10.1063\/1.3606502","article-title":"Development of optical fiber Bragg grating force-reflection sensor system of medical application for safe minimally invasive robotic surgery","volume":"82","author":"Song","year":"2011","journal-title":"Rev. Sci. Instrum."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"He, X., Balicki, M.A., Kang, J.U., Gehlbach, P.L., Handa, J.T., Taylor, R.H., and Iordachita, I.I. (2012, January 21\u201326). Force sensing micro-forceps with integrated fiber bragg grating for vitreoretinal surgery. Proceedings of the SPIE 8218, Optical Fibers and Sensors for Medical Diagnostics and Treatment Applications XII, San Francisco, CA, USA.","DOI":"10.1117\/12.909602"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1235","DOI":"10.1109\/TRO.2007.904891","article-title":"Force feedback benefit depends on experience in multiple degree of freedom robotic surgery task","volume":"23","author":"Wagner","year":"2007","journal-title":"IEEE Trans. Robot."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Geravand, M., Flacco, F., and De Luca, A. (2013, January 6\u201310). Human-robot physical interaction and collaboration using an industrial robot with a closed control architecture. Proceedings of the 2013 IEEE International Conference on Robotics and Automation (ICRA), Karlsruhe, Germany.","DOI":"10.1109\/ICRA.2013.6631141"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"282","DOI":"10.1109\/LRA.2016.2601345","article-title":"Interaction Force Reconstruction for Humanoid Robots","volume":"2","author":"Mattioli","year":"2017","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1312","DOI":"10.1109\/LRA.2017.2666420","article-title":"Gaussian Process Regression for Sensorless Grip Force Estimation of Cable-Driven Elongated Surgical Instruments","volume":"2","author":"Li","year":"2017","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Magrini, E., Flacco, F., and De Luca, A. (2014, January 14\u201318). Estimation of contact forces using a virtual force sensor. Proceedings of the 2014 IEEE\/RSJ International Conference on Intelligent Robots and Systems, Chicago, IL, USA.","DOI":"10.1109\/IROS.2014.6942848"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"431","DOI":"10.1109\/TOH.2016.2640289","article-title":"Towards Retrieving Force Feedback in Robotic-Assisted Surgery: A Supervised Neuro-Recurrent-Vision Approach","volume":"10","author":"Aviles","year":"2016","journal-title":"IEEE Trans. Haptics"},{"key":"ref_23","unstructured":"Aviles, A.I., Alsaleh, S., Sobrevilla, P., and Casals, A. (March, January 27). Exploring the effects of dimensionality reduction in deep networks for force estimation in robotic-assisted surgery. Proceedings of the SPIE Medical Imaging, San Diego, CA, USA."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Zhu, Y., Jiang, C., Zhao, Y., Terzopoulos, D., and Zhu, S.C. (2016, January 27\u201330). Inferring forces and learning human utilities from videos. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.415"},{"key":"ref_25","unstructured":"Pham, T.H., Kheddar, A., Qammaz, A., and Argyros, A.A. (2015, January 7\u201312). Towards force sensing from vision: Observing hand-object interactions to infer manipulation forces. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA."},{"key":"ref_26","unstructured":"Ferm\u00fcller, C., Wang, F., Yang, Y., Zampogiannis, K., Zhang, Y., Barranco, F., and Pfeiffer, M. (2017). Prediction of manipulation actions. Int. J. Comput. Vis., 1\u201317."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T. (2015, January 7\u201312). Long-term recurrent convolutional networks for visual recognition and description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298878"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1016\/j.trc.2015.03.014","article-title":"Long short-term memory neural network for traffic speed prediction using remote microwave sensor data","volume":"54","author":"Ma","year":"2015","journal-title":"Transp. Res. Part C Emerg. Technol."},{"key":"ref_29","unstructured":"Sutskever, I., Martens, J., and Hinton, G.E. (July, January 28). Generating text with recurrent neural networks. Proceedings of the 28th International Conference on Machine Learning (ICML-11), Bellevue, Washington, DC, USA."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Vinyals, O., Ravuri, S.V., and Povey, D. (2012, January 25\u201330). Revisiting recurrent neural networks for robust ASR. Proceedings of the 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Kyoto, Japan.","DOI":"10.1109\/ICASSP.2012.6288816"},{"key":"ref_31","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20136). Imagenet classification with deep convolutional neural networks. Proceedings of the 25th International Conference on Neural Information Processing Systems, Lake Tahoe, Nevada."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1007\/BF00045676","article-title":"Continuous contact force models for impact analysis in multibody systems","volume":"5","author":"Lankarani","year":"1994","journal-title":"Nonlinear Dyn."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Dubitzky, W., Granzow, M., and Berrar, D.P. (2007). Fundamentals of Data Mining in Genomics and Proteomics, Springer Science & Business Media.","DOI":"10.1007\/978-0-387-47509-7"},{"key":"ref_36","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"528","DOI":"10.1161\/01.STR.11.5.528","article-title":"Light touch thresholds in normal persons and cerebral vascular disease patient: Bilateral deficit after unilateral lesion","volume":"11","author":"Essing","year":"1980","journal-title":"Stroke"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"531","DOI":"10.3758\/BF03212187","article-title":"Manual discrimination of force using active finger motion","volume":"49","author":"Pang","year":"1991","journal-title":"Atten. Percept. Psychophys."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/17\/11\/2455\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T18:48:32Z","timestamp":1760208512000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/17\/11\/2455"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,10,26]]},"references-count":38,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2017,11]]}},"alternative-id":["s17112455"],"URL":"https:\/\/doi.org\/10.3390\/s17112455","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,10,26]]}}}