{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T17:04:41Z","timestamp":1774631081771,"version":"3.50.1"},"reference-count":33,"publisher":"MDPI AG","issue":"18","license":[{"start":{"date-parts":[[2021,9,17]],"date-time":"2021-09-17T00:00:00Z","timestamp":1631836800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>As a complex task, robot sorting has become a research hotspot. In order to enable robots to perform simple, efficient, stable and accurate sorting operations for stacked multi-objects in unstructured scenes, a robot multi-object sorting system is built in this paper. Firstly, the training model of rotating target detection is constructed, and the placement state of five common objects in unstructured scenes is collected as the training set for training. The trained model is used to obtain the position, rotation angle and category of the target object. Then, the instance segmentation model is constructed, and the same data set is made, and the instance segmentation network model is trained. Then, the optimized Mask R-CNN instance segmentation network is used to segment the object surface pixels, and the upper surface point cloud is extracted to calculate the normal vector. Then, the angle obtained by the normal vector of the upper surface and the rotation target detection network is fused with the normal vector to obtain the attitude of the object. At the same time, the grasping order is calculated according to the average depth of the surface. Finally, after the obtained object posture, category and grasping sequence are fused, the performance of the rotating target detection network, the instance segmentation network and the robot sorting system are tested on the established experimental platform. Based on this system, this paper carried out an experiment on the success rate of object capture in a single network and an integrated network. The experimental results show that the multi-object sorting system based on deep learning proposed in this paper can sort stacked objects efficiently, accurately and stably in unstructured scenes.<\/jats:p>","DOI":"10.3390\/s21186238","type":"journal-article","created":{"date-parts":[[2021,9,22]],"date-time":"2021-09-22T03:47:35Z","timestamp":1632282455000},"page":"6238","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["Research on Multi-Object Sorting System Based on Deep Learning"],"prefix":"10.3390","volume":"21","author":[{"given":"Hongyan","family":"Zhang","sequence":"first","affiliation":[{"name":"School of Mechanical and Aerospace Engineering, Jilin University, Changchun 130022, China"}]},{"given":"Huawei","family":"Liang","sequence":"additional","affiliation":[{"name":"School of Mechanical and Aerospace Engineering, Jilin University, Changchun 130022, China"}]},{"given":"Tao","family":"Ni","sequence":"additional","affiliation":[{"name":"School of Vehicle and Energy, Yanshan University, Qinhuangdao 066004, China"}]},{"given":"Lingtao","family":"Huang","sequence":"additional","affiliation":[{"name":"School of Mechanical and Aerospace Engineering, Jilin University, Changchun 130022, China"}]},{"given":"Jinsong","family":"Yang","sequence":"additional","affiliation":[{"name":"School of Mechanical and Aerospace Engineering, Jilin University, Changchun 130022, China"}]}],"member":"1968","published-online":{"date-parts":[[2021,9,17]]},"reference":[{"key":"ref_1","first-page":"2817","article-title":"Recent researches on robot autonomous grasp technology","volume":"35","author":"Liu","year":"2020","journal-title":"Control. Decis."},{"key":"ref_2","unstructured":"Rahardja, K., and Kosaka, A. (1996, January 4\u20138). Recognition and localization of multiple complex objects using simple visual cues. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems, Las Vegas, NV, USA."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1007\/s10846-005-3895-0","article-title":"Sensory-based colour sorting automated robotic cell","volume":"43","author":"Ata","year":"2005","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"923","DOI":"10.1177\/027836490302210009","article-title":"Robust visual servoing","volume":"22","author":"Kragic","year":"2003","journal-title":"Int. J. Robot. Res."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Saxena, A., Driemeyer, J., Kearns, J., Osondu, C., and Ng, A.Y. (2008). Learning to grasp novel objects using vision. Experimental Robotics, Springer.","DOI":"10.1177\/0278364907087172"},{"key":"ref_6","unstructured":"Wang, J.S. (2016). Research on Object Detection and Sort Algorithm on Vision System of Mechanical Arm. [Master\u2019s Thesis, Zhejiang University]."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"421","DOI":"10.1177\/0278364917710318","article-title":"Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection","volume":"37","author":"Levine","year":"2016","journal-title":"Int. J. Robot. Res."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Hossain, D., Capi, G., and Jindai, M. (2016, January 17\u201320). Evolution of deep belief neural network parameters for robot object recognition and grasping. Proceedings of the Procedia Computer Science, Tokyo, Japan.","DOI":"10.1016\/j.procs.2017.01.195"},{"key":"ref_9","unstructured":"Viereck, U., Pas, A., Saenko, K., and Platt, R. (2017, January 13\u201315). Learning a visuomotor controller for real world robotic grasping using simulated depth images. Proceedings of the Conference on Robot Learning, Mountain View, CA, USA."},{"key":"ref_10","unstructured":"Zhang, X.Y. (2020). Unknown Stack Object Grasping Method Based on 3D Construction and Reasoning. [Master\u2019s Thesis, Harbin institute of Technology]."},{"key":"ref_11","unstructured":"Li, Y.N. (2018). Research on Robust Grasping System Based on SSD Object Detection Technology. [Master\u2019s Thesis, Jilin University]."},{"key":"ref_12","unstructured":"Wang, B. (2020). Research on Robot Intelligent Grasping and Flexible Placement Technology Based on Vision and Force Sense. [Master\u2019s Thesis, Jilin University]."},{"key":"ref_13","unstructured":"Yang, X., Liu, Q., Yan, J., Li, A., Zhang, Z., and Yu, G. (2019). R3det: Refined single-stage detector with feature refinement for rotating object. arXiv."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Jiang, Y., Zhu, X., Wang, X., Yang, S., Li, W., Wang, H., Fu, P., and Luo, Z. (2017). R2cnn: Rotational region cnn for orientation robust scene text detection. arXiv.","DOI":"10.1109\/ICPR.2018.8545598"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Ding, J., Xue, N., Long, Y., Xia, G.S., and Lu, Q. (2018). Learning roi transformer for detecting oriented objects in aerial images. arXiv.","DOI":"10.1109\/CVPR.2019.00296"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1452","DOI":"10.1109\/TPAMI.2020.2974745","article-title":"Gliding vertex on the horizontal bounding box for multi-oriented object detection","volume":"43","author":"Xu","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_17","unstructured":"Qian, W., Yang, X., Peng, S., Guo, Y., and Yan, J. (2019). Learning modulated loss for rotated object detection. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Li, Y., Qi, H., Dai, J., Ji, X., and Wei, Y. (2017, January 21\u201326). Fully convolutional instance-aware semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New York, NY, USA.","DOI":"10.1109\/CVPR.2017.472"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Long, J., Shelhamer, E., and Darrell, T. (2015, January 7\u201312). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Berkeley, CA, USA.","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Shen, X., and Stamos, I. (2020). 3D Object Detection and Instance Segmentation from 3D Range and 2D Color Images. Sensors, 21.","DOI":"10.3390\/s21041213"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Doll\u00e1r, P., and Girshick, R. (2017, January 22\u201329). Mask r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_22","unstructured":"Bolya, D., Zhou, C., Xiao, F., and Lee, Y.J. (November, January 27). Yolact: Real-time instance segmentation. Proceedings of the IEEE\/CVF International Conference on Computer Vision. University of California, Davis, CA, USA."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Peng, S., Jiang, W., Pi, H., Li, X., Bao, H., and Zhou, X. (2020, January 13\u201319). Deep snake for real-time instance segmentation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Hangzhou, China.","DOI":"10.1109\/CVPR42600.2020.00856"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Wang, X., Kong, T., Shen, C., Jiang, Y., and Li, L. (2020, January 23\u201328). Solo: Segmenting objects by locations. Proceedings of the European Conference on Computer Vision, Adelaide, Australia.","DOI":"10.1007\/978-3-030-58523-5_38"},{"key":"ref_25","unstructured":"Xia, J. (2019). Deep-Learing Based Robot Grasp Detection Using RGB-D Sensor. [Master\u2019s Thesis, Southeast University]."},{"key":"ref_26","unstructured":"Zheng, S.S. (2020). Research on Grasp Detection of Industrial Robot Based on Deep Learning. [Master\u2019s Thesis, Harbin Institute of technology]."},{"key":"ref_27","unstructured":"Xiao, C.J. (2020). Research on Mechanical Arm Grasping Detection Using Deep Learning. [Master\u2019s Thesis, Harbin institute of technology]."},{"key":"ref_28","unstructured":"Jia, M.F. (2018). Research and Application of Robot Automatic Spraying System Based on Machine Vision. [Master\u2019s Thesis, South China University of Technology]."},{"key":"ref_29","first-page":"139","article-title":"The Research on Orientation and Scraping of Workpiece with Asymmetry Structure","volume":"30","author":"Wang","year":"2017","journal-title":"Electronic Sci. Technol."},{"key":"ref_30","first-page":"108","article-title":"Research on Straightness Detection Method of Cartesian Robot Based on Machine Vision","volume":"46","author":"Cao","year":"2018","journal-title":"Mach. Tool Hydraul."},{"key":"ref_31","unstructured":"Li, H. (2016). Design and Research on Vision Sorting and Tracking System of Delta Parallel Robot. [Master\u2019s Thesis, South China University of Technology]."},{"key":"ref_32","first-page":"127","article-title":"Design of Visual Servo Sorting Robot Based on SVM Classification","volume":"48","author":"Zheng","year":"2019","journal-title":"ME Eng. Technol."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Liu, Z.C., Lin, G.S., Yang, S., Feng, J.S., and Lin, W.S. (2018, January 18\u201323). Learning markov clustering networks for scene text detection. Proceedings of the 31st Meeting of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00725"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/18\/6238\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:01:15Z","timestamp":1760166075000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/18\/6238"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,17]]},"references-count":33,"journal-issue":{"issue":"18","published-online":{"date-parts":[[2021,9]]}},"alternative-id":["s21186238"],"URL":"https:\/\/doi.org\/10.3390\/s21186238","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,17]]}}}