{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,28]],"date-time":"2026-02-28T18:27:25Z","timestamp":1772303245826,"version":"3.50.1"},"reference-count":27,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2024,1,4]],"date-time":"2024-01-04T00:00:00Z","timestamp":1704326400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Science Foundation of China","doi-asserted-by":"publisher","award":["52075260"],"award-info":[{"award-number":["52075260"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Science Foundation of China","doi-asserted-by":"publisher","award":["BE2023086"],"award-info":[{"award-number":["BE2023086"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Key Research and Development Program of Jiangsu Province, China","award":["52075260"],"award-info":[{"award-number":["52075260"]}]},{"name":"Key Research and Development Program of Jiangsu Province, China","award":["BE2023086"],"award-info":[{"award-number":["BE2023086"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Robot arm monitoring is often required in intelligent industrial scenarios. A two-stage method for robot arm attitude estimation based on multi-view images is proposed. In the first stage, a super-resolution keypoint detection network (SRKDNet) is proposed. The SRKDNet incorporates a subpixel convolution module in the backbone neural network, which can output high-resolution heatmaps for keypoint detection without significantly increasing the computational resource consumption. Efficient virtual and real sampling and SRKDNet training methods are put forward. The SRKDNet is trained with generated virtual data and fine-tuned with real sample data. This method decreases the time and manpower consumed in collecting data in real scenarios and achieves a better generalization effect on real data. A coarse-to-fine dual-SRKDNet detection mechanism is proposed and verified. Full-view and close-up dual SRKDNets are executed to first detect the keypoints and then refine the results. The keypoint detection accuracy, PCK@0.15, for the real robot arm reaches up to 96.07%. In the second stage, an equation system, involving the camera imaging model, the robot arm kinematic model and keypoints with different confidence values, is established to solve the unknown rotation angles of the joints. The proposed confidence-based keypoint screening scheme makes full use of the information redundancy of multi-view images to ensure attitude estimation accuracy. Experiments on a real UR10 robot arm under three views demonstrate that the average estimation error of the joint angles is 0.53 degrees, which is superior to that achieved with the comparison methods.<\/jats:p>","DOI":"10.3390\/s24010305","type":"journal-article","created":{"date-parts":[[2024,1,4]],"date-time":"2024-01-04T09:47:32Z","timestamp":1704361652000},"page":"305","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Accurate Robot Arm Attitude Estimation Based on Multi-View Images and Super-Resolution Keypoint Detection Networks"],"prefix":"10.3390","volume":"24","author":[{"given":"Ling","family":"Zhou","sequence":"first","affiliation":[{"name":"College of Mechanical & Electrical Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China"}]},{"given":"Ruilin","family":"Wang","sequence":"additional","affiliation":[{"name":"College of Mechanical & Electrical Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8846-1666","authenticated-orcid":false,"given":"Liyan","family":"Zhang","sequence":"additional","affiliation":[{"name":"College of Mechanical & Electrical Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China"}]}],"member":"1968","published-online":{"date-parts":[[2024,1,4]]},"reference":[{"key":"ref_1","unstructured":"Lin, L., Yang, Y., Song, Y., Nemec, B., Ude, A., Rytz, J.A., Buch, A.G., Kr\u00fcger, N., and Savarimuthu, T.R. (July, January 29). Peg-in-Hole assembly under uncertain pose estimation. Proceedings of the 11th World Congress on Intelligent Control and Automation, Shenyang, China."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"180","DOI":"10.36548\/jismac.2019.3.005","article-title":"Robot assisted sensing, control and manufacture in automobile industry","volume":"1","author":"Smys","year":"2019","journal-title":"J. ISMAC"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"107092","DOI":"10.1016\/j.compag.2022.107092","article-title":"Design and evaluation of a robotic apple harvester using optimized picking patterns","volume":"198","author":"Bu","year":"2022","journal-title":"Comput. Electron. Agric."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Lu, G., Li, Y., Jin, S., Zheng, Y., Chen, W., and Zheng, X. (2011, January 19\u201320). A realtime motion capture framework for synchronized neural decoding. Proceedings of the 2011 IEEE International Symposium on VR Innovation, Singapore.","DOI":"10.1109\/ISVRI.2011.5759656"},{"key":"ref_5","unstructured":"Verma, A., Kofman, J., and Wu, X. (2004, January 17\u201319). Application of Markerless Image-Based Arm Tracking to Robot-Manipulator Teleoperation. Proceedings of the 2004 First Canadian Conference on Computer and Robot Vision, 2004, Proceedings, London, ON, Canada."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1016\/j.autcon.2019.04.004","article-title":"A vision-based marker-less pose estimation system for articulated construction robots","volume":"104","author":"Liang","year":"2019","journal-title":"Autom. Constr."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Toshev, A., and Szegedy, C. (2014, January 23\u201328). DeepPose: Human pose estimation via deep neural networks. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.214"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Newell, A., Yang, K., and Deng, J. (2016, January 11\u201314). Stacked hourglass networks for human pose estimation. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46484-8_29"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Chen, Y., Wang, Z., Peng, Y., Zhang, Z., Yu, G., and Sun, J. (2018, January 18\u201323). Cascaded pyramid network for multi-person pose estimation. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00742"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Peng, S., Liu, Y., Huang, Q., Zhou, X., and Bao, H. (2019, January 15\u201320). PVNet: Pixel-wise voting network for 6DoF pose estimation. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00469"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Sun, K., Xiao, B., Liu, D., and Wang, J. (2019, January 15\u201320). Deep high-resolution representation learning for human pose estimation. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00584"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1109\/TPAMI.2015.2439281","article-title":"Image super-resolution using deep convolutional networks","volume":"38","author":"Dong","year":"2016","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Kim, J., Lee, J., and Lee, K. (2016, January 27\u201330). Accurate image super-resolution using very deep convolutional networks. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.182"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Lim, B., Son, S., Kim, H., Nah, S., and Mu Lee, K. (2017, January 21\u201326). Enhanced deep residual networks for single image super-resolution. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.","DOI":"10.1109\/CVPRW.2017.151"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Shi, W., Caballero, J., Husz\u00e1r, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., and Wang, Z. (2016, January 27\u201330). Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.207"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Widmaier, F., Kappler, D., Schaal, S., and Bohg, J. (2016, January 16\u201321). Robot arm pose estimation by pixel-wise regression of joint angles. Proceedings of the 2016 IEEE International Conference on Robotics and Automation, Stockholm, Sweden.","DOI":"10.1109\/ICRA.2016.7487185"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Labb\u00e9, Y., Carpentier, J., Aubry, M., and Sivic, J. (2021, January 20\u201325). Single-view robot pose and joint angle estimation via render & compare. Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00170"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Zuo, Y., Qiu, W., Xie, L., Zhong, F., Wang, Y., and Yuille, A.L. (2019, January 15\u201320). CRAVES: Controlling robotic arm with a vision-based economic system. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00434"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Liu, Q., Yang, D., Hao, W., and Wei, Y. (2018, January 14\u201316). Research on Kinematic Modeling and Analysis Methods of UR Robot. Proceedings of the 2018 IEEE 4th Information Technology and Mechatronics Engineering Conference (ITOEC), Chongqing, China.","DOI":"10.1109\/ITOEC.2018.8740681"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Sanders, A. (2017). An Introduction to Unreal Engine 4, Taylor & Francis Group.","DOI":"10.1201\/9781315382555"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll\u00e1r, P., and Zitnick, C.L. (2014, January 6\u201312). Microsoft COCO: Common objects in context. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Qiu, W., Zhong, F., Zhang, Y., Qiao, S., Xiao, Z., Kim, T.S., and Wang, Y. (2017, January 18\u201320). UnrealCV: Virtual worlds for computer vision. Proceedings of the 2017 ACM, Tacoma, WA, USA.","DOI":"10.1145\/3123266.3129396"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1330","DOI":"10.1109\/34.888718","article-title":"A flexible new technique for camera calibration","volume":"22","author":"Zhang","year":"2000","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Strobl, K., and Hirzinger, G. (2006, January 9\u201313). Optimal hand-eye calibration. Proceedings of the 2006 IEEE\/RSJ International Conference on Intelligent Robots and Systems, Beijing, China.","DOI":"10.1109\/IROS.2006.282250"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"717","DOI":"10.1109\/70.326576","article-title":"Robot sensor calibration: Solving AX=XB on the euclidean group","volume":"10","author":"Park","year":"1994","journal-title":"IEEE Trans. Robot. Autom."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1090\/qam\/10666","article-title":"A method for the solution of certain problems in least squares","volume":"2","author":"Levenberg","year":"1944","journal-title":"Quart. Appl. Mach."},{"key":"ref_27","first-page":"1097","article-title":"ImageNet classification with deep convolutional neural networks","volume":"25","author":"Krizhevsky","year":"2012","journal-title":"Adv. Neural Inf. Process. Syst."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/1\/305\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T13:40:01Z","timestamp":1760103601000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/1\/305"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,4]]},"references-count":27,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,1]]}},"alternative-id":["s24010305"],"URL":"https:\/\/doi.org\/10.3390\/s24010305","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,4]]}}}