{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T23:20:37Z","timestamp":1789514437999,"version":"build-2803163510"},"reference-count":86,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2020,5,22]],"date-time":"2020-05-22T00:00:00Z","timestamp":1590105600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Peking University Tencent Rhino Bird Innovation Fund"},{"name":"Beijing Natural Science Foundation","award":["4192025"],"award-info":[{"award-number":["4192025"]}]},{"DOI":"10.13039\/100016909","name":"Microsoft Research Asia","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100016909","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61772043"],"award-info":[{"award-number":["61772043"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2020,5,31]]},"abstract":"<jats:p>Large-scale benchmarks provide a solid foundation for the development of action analytics. Most of the previous activity benchmarks focus on analyzing actions in RGB videos. There is a lack of large-scale and high-quality benchmarks for multi-modal action analytics. In this article, we introduce PKU Multi-Modal Dataset (PKU-MMD), a new large-scale benchmark for multi-modal human action analytics. It consists of about 28,000 action instances and 6.2 million frames in total and provides high-quality multi-modal data sources, including RGB, depth, infrared radiation (IR), and skeletons. To make PKU-MMD more practical, our dataset comprises two subsets under different settings for action understanding, namely Part I and Part II. Part I contains 1,076 untrimmed video sequences with 51 action classes performed by 66 subjects, while Part II contains 1,009 untrimmed video sequences with 41 action classes performed by 13 subjects. Compared to Part I, Part II is more challenging due to short action intervals, concurrent actions and heavy occlusion. PKU-MMD can be leveraged in two scenarios: action recognition with trimmed video clips and action detection with untrimmed video sequences. For each scenario, we provide benchmark performance on both subsets by conducting different methods with different modalities under two evaluation protocols, respectively. Experimental results show that PKU-MMD is a significant challenge to many state-of-the-art methods. We further illustrate that the features learned on PKU-MMD can be well transferred to other datasets. We believe this large-scale dataset will boost the research in the field of action analytics for the community.<\/jats:p>","DOI":"10.1145\/3365212","type":"journal-article","created":{"date-parts":[[2020,5,25]],"date-time":"2020-05-25T18:07:21Z","timestamp":1590430041000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":89,"title":["A Benchmark Dataset and Comparison Study for Multi-modal Human Action Analytics"],"prefix":"10.1145","volume":"16","author":[{"given":"Jiaying","family":"Liu","sequence":"first","affiliation":[{"name":"Institute of Computer Science and Technology, Peking University, Haidian, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sijie","family":"Song","sequence":"additional","affiliation":[{"name":"Institute of Computer Science and Technology, Peking University, Haidian, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chunhui","family":"Liu","sequence":"additional","affiliation":[{"name":"Institute of Computer Science and Technology, Peking University, Haidian, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yanghao","family":"Li","sequence":"additional","affiliation":[{"name":"Institute of Computer Science and Technology, Peking University, Haidian, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yueyu","family":"Hu","sequence":"additional","affiliation":[{"name":"Institute of Computer Science and Technology, Peking University, Haidian, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,5,22]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the IEEE International Conference on E-health Networking, Application 8 Services (Healthcom\u201913)","author":"Amiri S. Mohsen"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2012.6239175"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298698"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11042-016-3374-6"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.502"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICIP.2015.7350781"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33868-7_6"},{"key":"e_1_2_1_8_1","unstructured":"CMU. 2003. CMU Graphics Lab Motion Capture Database. Retrieved from http:\/\/mocap.cs.cmu.edu\/.  CMU. 2003. CMU Graphics Lab Motion Capture Database. Retrieved from http:\/\/mocap.cs.cmu.edu\/."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46454-1_17"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298878"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2016.2552404"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46487-9_47"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-009-0275-4"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.213"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208303"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00502"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-013-0683-3"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2016.2640292"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.5244\/C.31.72"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3177757"},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2571--2578","author":"Jain Arpit"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 740--747","author":"Jain Manan"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.5555\/2789272.2789280"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2015.2456412"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2017.44"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.223"},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the IEEE International Conference on Computer Vision. 2556--2563","author":"Kuehne H."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2008.4587756"},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the IEEE International Conference on Multimedia Expo Workshops. 613--616","author":"Li Bo","year":"2017"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the IEEE International Conference on Multimedia Expo Workshops. 597--600","author":"Li Chao","year":"2017"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2018\/109"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2010.5543273"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46478-7_13"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.109"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.sigpro.2014.08.038"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132734.3132739"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01264-9_11"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.333"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2015.2490551"},{"key":"e_1_2_1_40_1","volume-title":"Technical Report CG-2007-2.","author":"M\u00fcller M.","year":"2007"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/WACV.2013.6474999"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2013.98"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-15561-1_11"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2016.2533389"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10605-2_48"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2012.6247807"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0816-y"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2011.6126349"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.115"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/WACV.2015.138"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.498"},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the Advances in Neural Information Processing Systems. 568--576","author":"Simonyan Karen","year":"2014"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v31i1.11212"},{"key":"e_1_2_1_54_1","volume-title":"Amir Roshan Zamir, and Mubarak Shah","author":"Soomro Khurram","year":"2012"},{"key":"e_1_2_1_55_1","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence. 47--55","author":"Sung Jaeyong","year":"2011"},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the IEEE International Conference on Robotics and Automation. 842--849","author":"Sung Jaeyong","year":"2012"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2013.341"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.484"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.441"},{"key":"e_1_2_1_60_1","volume-title":"Proceedings of the IEEE International Conference on Multimedia Expo Workshops. 591--596","author":"Wang Hongsong","year":"2017"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2012.6247813"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.339"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/2647868.2654912"},{"key":"e_1_2_1_64_1","volume-title":"Action recognition and detection by combining motion and appearance features. THUMOS","author":"Wang Limin","year":"2014"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2013.345"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10602-1_37"},{"key":"e_1_2_1_67_1","volume-title":"CUHK8SIAT submission for THUMOS15 action recognition challenge. THUMOS","author":"Wang Limin","year":"2015"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.678"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46484-8_2"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.406"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.389"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299065"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/2733373.2806222"},{"key":"e_1_2_1_74_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 20--27","author":"Xia Lu"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.12328"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.108"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038917"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2012.6239234"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.342"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2016.05.019"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2017.2721108"},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.233"},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.317"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.442"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v30i1.10451"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.316"}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3365212","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3365212","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:01:30Z","timestamp":1750183290000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3365212"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5,22]]},"references-count":86,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,5,31]]}},"alternative-id":["10.1145\/3365212"],"URL":"https:\/\/doi.org\/10.1145\/3365212","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"value":"1551-6857","type":"print"},{"value":"1551-6865","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,5,22]]},"assertion":[{"value":"2019-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-05-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}