{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T13:05:26Z","timestamp":1781615126734,"version":"3.54.5"},"reference-count":40,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2020,3,12]],"date-time":"2020-03-12T00:00:00Z","timestamp":1583971200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Image based human behavior and activity understanding has been a hot topic in the field of computer vision and multimedia. As an important part, skeleton estimation, which is also called pose estimation, has attracted lots of interests. For pose estimation, most of the deep learning approaches mainly focus on the joint feature. However, the joint feature is not sufficient, especially when the image includes multi-person and the pose is occluded or not fully visible. This paper proposes a novel multi-task framework for the multi-person pose estimation. The proposed framework is developed based on Mask Region-based Convolutional Neural Networks (R-CNN) and extended to integrate the joint feature, body boundary, body orientation and occlusion condition together. In order to further improve the performance of the multi-person pose estimation, this paper proposes to organize the different information in serial multi-task models instead of the widely used parallel multi-task network. The proposed models are trained on the public dataset Common Objects in Context (COCO), which is further augmented by ground truths of body orientation and mutual-occlusion mask. Experiments demonstrate the performance of the proposed method for multi-person pose estimation and body orientation estimation. The proposed method can detect 84.6% of the Percentage of Correct Keypoints (PCK) and has an 83.7% Correct Detection Rate (CDR). Comparisons further illustrate the proposed model can reduce the over-detection compared with other methods.<\/jats:p>","DOI":"10.3390\/s20061593","type":"journal-article","created":{"date-parts":[[2020,3,13]],"date-time":"2020-03-13T08:58:59Z","timestamp":1584089939000},"page":"1593","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Multi-Person Pose Estimation using an Orientation and Occlusion Aware Deep Learning Network"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9708-7429","authenticated-orcid":false,"given":"Yanlei","family":"Gu","sequence":"first","affiliation":[{"name":"College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Shiga 525-8577, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huiyang","family":"Zhang","sequence":"additional","affiliation":[{"name":"Institute of Industrial Science, The University of Tokyo, Tokyo 153-8505, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shunsuke","family":"Kamijo","sequence":"additional","affiliation":[{"name":"Institute of Industrial Science, The University of Tokyo, Tokyo 153-8505, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,3,12]]},"reference":[{"key":"ref_1","unstructured":"Sminchisescu, C., and Telea, A. (2002, January 4\u20138). Human Pose Estimation from Silhouettes. A Consistent Approach Using Distance Level Sets. Proceedings of the 10th international conference in central Europe on computer graphics, visualization and computer vision, Bory, Czech Republic."},{"key":"ref_2","unstructured":"Mittal, A., Zhao, L., and Davis, L.S. (2003, January 22\u201322). Human body pose estimation using silhouette shape analysis. Proceedings of the IEEE Conference on Advanced Video and Signal Based Surveillance, Miami, FL, USA."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1109\/TPAMI.2007.250600","article-title":"Tracking people by learning their appearance","volume":"29","author":"Ramanan","year":"2006","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_4","unstructured":"Weiss, D., Sapp, B., and Taskar, B. (2010, January 6\u201311). Sidestepping intractable inference with structured ensemble cascades. Proceedings of the 24th International Conference on Neural Information Processing Systems, Vancouver, BC, Canada."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"443","DOI":"10.1109\/ACCESS.2016.2643439","article-title":"Human Pose Estimation by Exploiting Spatial and Temporal Constraints in Body-Part Configurations","volume":"5","author":"Li","year":"2016","journal-title":"IEEE Access"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1023\/B:VISI.0000042934.15159.49","article-title":"Pictorial structures for object recognition","volume":"61","author":"Felzenszwalb","year":"2005","journal-title":"Int. J. Comput. Vision"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"2878","DOI":"10.1109\/TPAMI.2012.261","article-title":"Articulated human detection with flexible mixtures of parts","volume":"35","author":"Yang","year":"2012","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Pishchulin, L., Andriluka, M., Gehler, P., and Schiele, B. (2013, January 23\u201328). Poselet Conditioned Pictorial Structures. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.","DOI":"10.1109\/CVPR.2013.82"},{"key":"ref_9","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20138). Imagenet classification with deep convolutional neural networks. Proceedings of the 26th International Conference on Neural Information Processing Systems 2012, Lake Tahoe, CA, USA."},{"key":"ref_10","unstructured":"Simonyan, K., and Zisserman, A. (2019, December 01). Very deep convolutional networks for large-scale image recognition. Available online: arXivpreprintarXiv:1409.1556."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7\u201312). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Girshick, R. (2015, January 7\u201313). Fast R-CNN. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.169"},{"key":"ref_14","unstructured":"Redmon, J., and Farhadi, A. (2019, December 01). Yolov3: An incremental improvement. Available online: arXivpreprintarXiv:1804.02767."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Doll\u00e1r, P., and Girshick, R. (2017, January 22\u201329). Mask R-CNN. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Toshev, A., and Szegedy, C. (2014, January 23\u201328). DeepPose: Human pose estimation via deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.214"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Haque, A., Peng, B., Luo, Z., Alahi, A., Yeung, S., and Fei-Fei, L. (2016, January 8\u201316). Towards viewpoint invariant 3D human pose estimation. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46448-0_10"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Carreira, J., Agrawal, P., Fragkiadaki, K., and Malik, J. (2016, January 27\u201330). Human pose estimation with iterative error feedback. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.512"},{"key":"ref_19","unstructured":"Tompson, J.J., Jain, A., LeCun, Y., and Bregler, C. (2014, January 8\u201313). Joint training of a convolutional network and a graphical model for human pose estimation. Proceedings of the Neural Information Processing Systems 2014, Montr\u00e9al, QC, Canada."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Wei, S.E., Ramakrishna, V., Kanade, T., and Sheikh, Y. (2016, January 27\u201330). Convolutional pose machines. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.511"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Newell, A., Yang, K., and Deng, J. (2016, January 8\u201316). Stacked hourglass networks for human pose estimation. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46484-8_29"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"58056","DOI":"10.1109\/ACCESS.2018.2874307","article-title":"A parameter efficient human pose estimation method based on densely connected convolutional module","volume":"6","author":"Wang","year":"2018","journal-title":"IEEE Access"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"71158","DOI":"10.1109\/ACCESS.2019.2919154","article-title":"Human pose estimation with deeply learned multi-scale compositional models","volume":"7","author":"Wang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Chen, Y., Shen, C., Wei, X.S., Liu, L., and Yang, J. (2017, January 22\u201329). Adversarial posenet: A structure-aware convolutional network for human pose estimation. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.137"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s11042-019-7433-7","article-title":"Deep neural networks for human pose estimation from a very low resolution depth image","volume":"78","author":"Szczuko","year":"2019","journal-title":"Multimed. Tools Appl."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Pishchulin, L., Insafutdinov, E., Tang, S., Andres, B., Andriluka, M., Gehler, P.V., and Schiele, B. (2016, January 27\u201330). DeepCut: Joint subset partition and labeling for multi person pose estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.533"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M., and Schiele, B. (2016, January 8\u201316). DeeperCut: A deeper, stronger, and faster multi-person pose estimation model. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46466-4_3"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Kocabas, M., Karagoz, S., and Akbas, E. (2018, January 8\u201314). MultiPoseNet: Fast multi-person pose estimation using pose residual network. Proceedings of the European Conference on Computer Vision, Munich, Germany.","DOI":"10.1007\/978-3-030-01252-6_26"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Cao, Z., Simon, T., Wei, S.E., and Sheikh, Y. (2017, January 21\u201326). Realtime multi-person 2D pose estimation using part affinity fields. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.143"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Papandreou, G., Zhu, T., Kanazawa, N., Toshev, A., Tompson, J., Bregler, C., and Murphy, K. (2017, January 21\u201326). Towards accurate multi-person pose estimation in the wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.395"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., and Guadarrama, S. (2017, January 21\u201326). Speed\/Accuracy trade-offs for modern convolutional object detectors. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.351"},{"key":"ref_32","unstructured":"Ren, S., He, K., Girshick, R., and Sun, J. (2014, January 8\u201313). Faster R-CNN: Towards real-time object detection with region proposal networks. Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Azizpour, H., and Laptev, I. (2012, January 7\u201313). Object detection using strongly-supervised deformable part models. Proceedings of the European Conference on Computer Vision, Florence, Italy.","DOI":"10.1007\/978-3-642-33718-5_60"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Ghiasi, G., Yang, Y., Ramanan, D., and Fowlkes, C.C. (2014, January 23\u201328). Parsing occluded people. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.308"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Rafi, U., Gall, J., and Leibe, B. (2015, January 7\u201312). A semantic occlusion model for human pose estimation from a single depth image. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Boston, MA, USA.","DOI":"10.1109\/CVPRW.2015.7301338"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"26107","DOI":"10.1007\/s11042-018-5839-2","article-title":"Integral customer pose estimation using body orientation and visibility mask","volume":"77","author":"Liu","year":"2018","journal-title":"Multimed. Tools Appl."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Zhang, H., Gu, Y., and Kamijo, S. (2019, January 11\u201313). Orientation and occlusion aware multi-person pose estimation using multi-task deep learning network. Proceedings of the IEEE International Conference on Consumer Electronics, Las Vegas, NV, USA.","DOI":"10.1109\/ICCE.2019.8662041"},{"key":"ref_38","first-page":"640","article-title":"Fully convolutional networks for semantic segmentation","volume":"39","author":"Long","year":"2015","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll\u00e1r, P., and Zitnick, C.L. (2014, January 6\u201312). Microsoft COCO: Common objects in context. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"6595","DOI":"10.1007\/s11042-016-3342-1","article-title":"Customer behavior classification using surveillance camera for marketing","volume":"76","author":"Liu","year":"2017","journal-title":"Multimed. Tools Appl."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/6\/1593\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:06:29Z","timestamp":1760173589000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/6\/1593"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,12]]},"references-count":40,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2020,3]]}},"alternative-id":["s20061593"],"URL":"https:\/\/doi.org\/10.3390\/s20061593","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3,12]]}}}