{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,9]],"date-time":"2026-06-09T14:52:37Z","timestamp":1781016757094,"version":"3.54.1"},"reference-count":62,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2023,5,6]],"date-time":"2023-05-06T00:00:00Z","timestamp":1683331200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This article addresses how to tackle one of the most demanding tasks in manufacturing and industrial maintenance sectors: using robots with a novel and robust solution to detect the fastener and its rotation in (un)screwing tasks over parallel surfaces with respect to the tool. To this end, the vision system is based on an industrial camera with a dynamic exposure time, a tunable liquid crystal lens (TLCL), and active near-infrared reflectance (NIR) illumination. Its camera parameters, combined with a fixed value of working distance (WD) and variable or constant field of view (FOV), make it possible to work with a variety of fastener sizes under several lighting conditions. This development also uses a collaborative robot with an embedded force sensor to verify the success of the fastener localization in a real test. Robust algorithms based on segmentation neural networks (SNN) and vision were developed to find the center and rotation of the hexagon fastener in a flawless condition and worn, scratched, and rusty conditions. SNNs were tested using a graphics processing unit (GPU), central processing unit (CPU), and edge devices, such as Jetson Javier Nx (JJNX), Intel Neural Compute Stick 2 (INCS2), and M.2 Accelerator with Dual Edge TPU (DETPU), with optimization parameters, such as the unsigned integer (UINT) and float (FP), to understand their performance. A virtual program logic controller (PLC) was mounted on a personal computer (PC) as the main control to process the images and save the data. Moreover, a mathematical analysis based on the international standard organization (ISO) and patents of the manual socket wrench was performed to determine the maximum error allowed. In addition, the work was substantiated using exhaustive evaluation tests, validating the tolerance errors, robotic forces for successfully completed tasks, and algorithms implemented. As a result of this work, the translation tolerances increase with higher sizes of fasteners from 0.75 for M6 to 2.50 for M24; however, the rotation decreases with the size from 5.5\u00b0 for M6 to 3.5\u00b0 for M24. The proposed methodology is a robust solution to tackle outliers contours and fake vertices produced by distorted masks present in non-constant illumination; it can reach an average accuracy to detect the vertices of 99.86% and the center of 100%, also, the time consumed by the SNN and the proposed algorithms is 73.91 ms on an Intel Core I9 CPU. This work is an interesting contribution to industrial robotics and improves current applications.<\/jats:p>","DOI":"10.3390\/s23094527","type":"journal-article","created":{"date-parts":[[2023,5,8]],"date-time":"2023-05-08T02:29:22Z","timestamp":1683512962000},"page":"4527","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Robust Fastener Detection Based on Force and Vision Algorithms in Robotic (Un)Screwing Applications"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9290-1081","authenticated-orcid":false,"given":"Paul","family":"Espinosa Peralta","sequence":"first","affiliation":[{"name":"Centre for Automation and Robotics (CAR) UPM-CSIC, Universidad Polit\u00e9cnica de Madrid, 28006 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0030-1551","authenticated-orcid":false,"given":"Manuel","family":"Ferre","sequence":"additional","affiliation":[{"name":"Centre for Automation and Robotics (CAR) UPM-CSIC, Universidad Polit\u00e9cnica de Madrid, 28006 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6652-0090","authenticated-orcid":false,"given":"Miguel \u00c1ngel","family":"S\u00e1nchez-Ur\u00e1n","sequence":"additional","affiliation":[{"name":"Centre for Automation and Robotics (CAR) UPM-CSIC, Universidad Polit\u00e9cnica de Madrid, 28006 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,5,6]]},"reference":[{"key":"ref_1","unstructured":"Coloma Chac\u00f3n, S. (2020). Methods, Strategies and Application Cases for Robotic Telemanipulation in Hazardous Environments. [Ph.D. Thesis, Universidad Polit\u00e9cnica de Madrid]."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Tardioli, D., Matell\u00e1n, V., Heredia, G., Silva, M.F., and Marques, L. (2022, January 23\u201325). ROSE: Robot for Automatic Spacer Installation in Overhead Power Lines. Proceedings of the ROBOT2022: Fifth Iberian Robotics Conference, Zaragoza, Spain.","DOI":"10.1007\/978-3-031-21062-4"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Zhong, Y., Fu, Z., Su, M., Guan, Y., Zhu, H., and Zhong, L. (2019, January 6\u20138). Development of A Robot System Performing Maintenance Tasks on High-Voltage Power Transmission Lines. Proceedings of the 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO), Dali, China.","DOI":"10.1109\/ROBIO49542.2019.8961863"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1016\/j.engappai.2019.01.008","article-title":"Railway track fastener defect detection based on image processing and deep learning techniques: A comparative study","volume":"80","author":"Wei","year":"2019","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_5","unstructured":"De Ruvo, G., De Ruvo, P., Marino, F., Mastronardi, G., Mazzeo, P., and Stella, E. (2005, January 4\u20136). A FPGA-based architecture for automatic hexagonal bolts detection in railway maintenance. Proceedings of the Seventh International Workshop on Computer Architecture for Machine Perception (CAMP\u201905), Palermo, Italy."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"De Ruvo, P., Distante, A., Stella, E., and Marino, F. (2009, January 7\u201310). A GPU-based vision system for real time detection of fastening elements in railway inspection. Proceedings of the 2009 16th IEEE International Conference on Image Processing (ICIP), Cairo, Egypt.","DOI":"10.1109\/ICIP.2009.5414438"},{"key":"ref_7","first-page":"1455","article-title":"Unfastening of Hexagonal Headed Screws by a Collaborative Robot","volume":"17","author":"Li","year":"2020","journal-title":"IEEE Trans. Autom. Sci. Eng."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"287","DOI":"10.1016\/j.procir.2016.02.135","article-title":"Simplifying robot tools by taking advantage of sensor integration in human collaboration robots","volume":"44","author":"Gerbers","year":"2016","journal-title":"Procedia CIRP"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Rastegarpanah, A., Ner, R., Stolkin, R., and Marturi, N. (2021). Nut Unfastening by Robotic Surface Exploration. Robotics, 10.","DOI":"10.3390\/robotics10030107"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1729881420923997","DOI":"10.1177\/1729881420923997","article-title":"Case study: Performance analysis and development of robotized screwing application with integrated vision sensing system for automotive industry","volume":"17","author":"Bulej","year":"2020","journal-title":"Int. J. Adv. Robot. Syst."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"107","DOI":"10.2478\/msr-2022-0014","article-title":"Real-Time Instance Segmentation of Metal Screw Defects Based on Deep Learning Approach","volume":"22","author":"Chen","year":"2022","journal-title":"Meas. Sci. Rev."},{"key":"ref_12","unstructured":"Liu, X.J., Nie, Z., Yu, J., Xie, F., and Song, R. (2021, January 22\u201325). Semi-autonomous Robotic Manipulation by Tele-Operation with Master-Slave Robots and Autonomy Based on Vision and Force Sensing. Proceedings of the Intelligent Robotics and Applications, Yantai, China."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Chen, W.H., Wegener, K., and Dietrich, F. (2014, January 5\u201310). A robot assistant for unscrewing in hybrid human-robot disassembly. Proceedings of the 2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014), Bali, Indonesia.","DOI":"10.1109\/ROBIO.2014.7090386"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"887","DOI":"10.1109\/TASE.2017.2679720","article-title":"A System Combining Force and Vision Sensing for Automated Screw Removal on Laptops","volume":"15","author":"DiFilippo","year":"2018","journal-title":"IEEE Trans. Autom. Sci. Eng."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"031008","DOI":"10.1115\/1.4056074","article-title":"Automatic Screw Detection and Tool Recommendation System for Robotic Disassembly","volume":"145","author":"Zhang","year":"2022","journal-title":"J. Manuf. Sci. Eng."},{"key":"ref_16","unstructured":"Ratchev, S. (2020, January 14\u201315). Strategies for Dealing with Problems in Robotised Unscrewing Operations. Proceedings of the Smart Technologies for Precision Assembly, Virtual Event."},{"key":"ref_17","unstructured":"(2005). Assembly Tools for Screws and Nuts\u2014Wrench and Socket Openings\u2014Tolerances for General Use (Standard No. ISO 691:2005(E))."},{"key":"ref_18","unstructured":"(1996). Assembly Tools for Screws and Nuts\u2014Driving Squares (Standard No. IS0 1174-l:1996(E))."},{"key":"ref_19","unstructured":"(1996). Assembly Tools for Screws and Nuts\u2014Square Drive sockets (Standard No. IS0 2725-1:1996(E))."},{"key":"ref_20","unstructured":"(1979). Hexagon Head Bolts\u2014Product Grades A and B (Standard No. IS0 4014-1979(E))."},{"key":"ref_21","unstructured":"Knudsen, R.G., Price, J.W., and Olson, G.E. (1970). Wrench Sockets, Socket Drives and Similar Couplers. (3,495,485), U.S. Patent."},{"key":"ref_22","unstructured":"Wright, R.B., and Vozenilek, T.M. (1994). Socket Wrench Opening. (5,284,073 994), U.S. Patent."},{"key":"ref_23","unstructured":"Dossier, M. (1986). Tightening Tool for Nuts or Bolts. (4,581,957), U.S. Patent."},{"key":"ref_24","unstructured":"Mader, W.G., and Peppel, P. (1992). Wrench Openings. (5,092,203), U.S. Patent."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1109\/34.121783","article-title":"Single lens stereo with a plenoptic camera","volume":"14","author":"Adelson","year":"1992","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Khoo, I.C. (2022). Liquid Crystals, John Wiley & Sons.","DOI":"10.1002\/9781119705819"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1128","DOI":"10.1063\/1.1779954","article-title":"Variable-focus liquid lens for miniature cameras","volume":"85","author":"Kuiper","year":"2004","journal-title":"Appl. Phys. Lett."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1773","DOI":"10.1163\/156856111X600514","article-title":"Liquid lenses and driving mechanisms: A review","volume":"26","author":"Chiu","year":"2012","journal-title":"J. Adhes. Sci. Technol."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Shih, L. (2007, January 29\u201331). Autofocus survey: A comparison of algorithms. Proceedings of the Electronic Imaging 2007, San Jose, CA, USA.","DOI":"10.1117\/12.705386"},{"key":"ref_30","unstructured":"Glavin, L., MacKinnon, J., and Varghese, S. (February, January 28). Autofocus changes the paradigm for camera technology. Proceedings of the Physics and Simulation of Optoelectronic Devices XXV, San Francisco, CA, USA."},{"key":"ref_31","first-page":"1165","article-title":"Autofocus system and evaluation methodologies: A literature review","volume":"30","author":"Zhang","year":"2018","journal-title":"Sens. Mater"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1330","DOI":"10.1109\/34.888718","article-title":"A flexible new technique for camera calibration","volume":"22","author":"Zhang","year":"2000","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"3586","DOI":"10.1109\/TIP.2015.2448356","article-title":"Enhancing Color Images of Extremely Low Light Scenes Based on RGB\/NIR Images Acquisition With Different Exposure Times","volume":"24","author":"Sugimura","year":"2015","journal-title":"IEEE Trans. Image Process."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Echtler, F., Sielhorst, T., Huber, M., and Klinker, G. (2009, January 16\u201318). A short guide to modulated light. Proceedings of the 3rd International Conference on Tangible and Embedded Interaction, Cambridge, UK.","DOI":"10.1145\/1517664.1517742"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"693","DOI":"10.1109\/TIM.2019.2905309","article-title":"A multifeature hierarchical locating algorithm for hexagon nut of railway fasteners","volume":"69","author":"Peng","year":"2019","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Shim, I., Lee, J.Y., and Kweon, I.S. (2014, January 14\u201318). Auto-adjusting camera exposure for outdoor robotics using gradient information. Proceedings of the 2014 IEEE\/RSJ International Conference on Intelligent Robots and Systems, Chicago, IL, USA.","DOI":"10.1109\/IROS.2014.6942682"},{"key":"ref_37","unstructured":"Nourani-Vatani, N., and Roberts, J. (2007, January 10\u201312). Automatic camera exposure control. Proceedings of the Australasian Conference on Robotics and Automation 2007, Brisbane, Australia."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Yildiz, E., and W\u00f6rg\u00f6tter, F. (2020, January 4\u20136). DCNN-based Screw Classification in Automated Disassembly Processes. Proceedings of the ROBOVIS, Budapest, Hungary.","DOI":"10.5220\/0009979900610068"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1177","DOI":"10.1080\/0951192X.2021.1963476","article-title":"Accurate screw detection method based on faster R-CNN and rotation edge similarity for automatic screw disassembly","volume":"34","author":"Li","year":"2021","journal-title":"Int. J. Comput. Integr. Manuf."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Feng, Y., Yang, B., Li, X., Fu, C.W., Cao, R., Chen, K., Dou, Q., Wei, M., Liu, Y.H., and Heng, P.A. (2022). Towards Robust Part-aware Instance Segmentation for Industrial Bin Picking. arXiv.","DOI":"10.1109\/ICRA46639.2022.9811728"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5\u20139). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_42","unstructured":"Abdulla, W. (2023, May 05). Mask R-CNN for Object Detection and Instance Segmentation on Keras and TensorFlow. Available online: https:\/\/github.com\/matterport\/Mask_RCNN."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18\u201322). Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00474"},{"key":"ref_44","unstructured":"Bolya, D., Zhou, C., Xiao, F., and Lee, Y.J. (November, January 27). Yolact: Real-time instance segmentation. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Republic of Korea."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1016\/0734-189X(85)90016-7","article-title":"Topological structural analysis of digitized binary images by border following","volume":"30","author":"Suzuki","year":"1985","journal-title":"Comput. Vision Graph. Image Process."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1016\/0167-8655(82)90016-2","article-title":"Finding the convex hull of a simple polygon","volume":"1","author":"Sklansky","year":"1982","journal-title":"Pattern Recognit. Lett."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1006\/cviu.1999.0831","article-title":"Robust detection of lines using the progressive probabilistic hough transform","volume":"78","author":"Matas","year":"2000","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_48","unstructured":"(1998). Manipulating Industrial Robots\u2014Performance Criteria and Related Test Methods (Standard No. ISO 9283)."},{"key":"ref_49","unstructured":"(2023, May 05). Omron Industrial Camera. Available online: https:\/\/industrial.omron.es\/es\/products\/F430-F000L12M-SWV."},{"key":"ref_50","unstructured":"(2023, May 05). Intel CPU. Available online: https:\/\/ark.intel.com\/content\/www\/fr\/fr\/ark\/products\/199328\/intel-core-i910900-processor-20m-cache-up-to-5-20-ghz.html."},{"key":"ref_51","unstructured":"(2023, May 05). Jetson Xavier NX Developer Kit. Available online: https:\/\/developer.nvidia.com\/embedded\/learn\/get-started-jetson-xavier-nx-devkit."},{"key":"ref_52","unstructured":"(2023, May 05). Intel Neural Compute Stick 2. Available online: https:\/\/www.intel.com\/content\/www\/us\/en\/developer\/articles\/tool\/neural-compute-stick.html."},{"key":"ref_53","unstructured":"(2023, May 05). M.2 Accelerator with Dual Edge TPU. Available online: https:\/\/coral.ai\/products\/m2-accelerator-dual-edgetpu\/."},{"key":"ref_54","unstructured":"(2023, May 05). Servomotor Qrob70I. Available online: https:\/\/en.zeroerr.cn\/rotary_actuators\/erob70i."},{"key":"ref_55","unstructured":"Wu, S.T., and Marquez, M.R.G. (2003, January 12\u201315). A non-self-intersection Douglas-Peucker algorithm. Proceedings of the 16th Brazilian Symposium on Computer Graphics and Image Processing (SIBGRAPI 2003), Sao Carlos, Brazil."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s40537-016-0043-6","article-title":"A survey of transfer learning","volume":"3","author":"Weiss","year":"2016","journal-title":"J. Big Data"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"1538","DOI":"10.1214\/009053605000000255","article-title":"Boosting with early stopping: Convergence and consistency","volume":"33","author":"Zhang","year":"2005","journal-title":"Ann. Stat."},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., and Savarese, S. (2019, January 15\u201320). Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00075"},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s40537-019-0197-0","article-title":"A survey on image data augmentation for deep learning","volume":"6","author":"Shorten","year":"2019","journal-title":"J. Big Data"},{"key":"ref_60","unstructured":"Harris, C., and Stephens, M. (September, January 31). A combined corner and edge detector. Proceedings of the Alvey Vision Conference, Manchester, UK."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"105205","DOI":"10.1088\/1361-6501\/ac750e","article-title":"Automatic optimal camera exposure time control for digital image correlation","volume":"33","author":"Pan","year":"2022","journal-title":"Meas. Sci. Technol."},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Buslaev, A., Iglovikov, V.I., Khvedchenya, E., Parinov, A., Druzhinin, M., and Kalinin, A.A. (2020). Albumentations: Fast and flexible image augmentations. Information, 11.","DOI":"10.3390\/info11020125"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/9\/4527\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:30:35Z","timestamp":1760124635000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/9\/4527"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,6]]},"references-count":62,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2023,5]]}},"alternative-id":["s23094527"],"URL":"https:\/\/doi.org\/10.3390\/s23094527","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,6]]}}}