{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,3]],"date-time":"2026-03-03T06:28:32Z","timestamp":1772519312938,"version":"3.50.1"},"reference-count":65,"publisher":"Association for Computing Machinery (ACM)","issue":"9","license":[{"start":{"date-parts":[[2024,8,16]],"date-time":"2024-08-16T00:00:00Z","timestamp":1723766400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000181","name":"Air Force Office of Scientific Research","doi-asserted-by":"crossref","award":["FA9550-22-1-0475"],"award-info":[{"award-number":["FA9550-22-1-0475"]}],"id":[{"id":"10.13039\/100000181","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2024,9,30]]},"abstract":"<jats:p>\n            Action segmentation consists of temporally segmenting a video and labeling each segmented interval with a specific action label. In this work, we propose a novel action segmentation method that requires no initial video analysis and no annotated data. Our proposal involves extracting features from videos using several pre-trained deep-learning models, including spatiotemporal and self-supervised methods. Data is then transformed using a positional encoder, and finally, a clustering algorithm is applied, where each produced cluster presumably corresponds to a different single and distinguishable action. For self-supervised features, we explored DINO, and for spatiotemporal features, we investigated I3D and SlowFast methods. Moreover, two different clustering algorithms (FINCH and KMeans) were investigated, and we also explored how varying the length of video snippets that generate the feature vectors affected the quality of the segmentation. Experiments show that our method produces competitive results on the\n            <jats:italic>Breakfast<\/jats:italic>\n            and\n            <jats:italic>INRIA Instructional Videos<\/jats:italic>\n            dataset benchmarks. Our best result was produced using a composition of self-supervised features generated by DINO, FINCH clustering, and positional encoding.\n          <\/jats:p>","DOI":"10.1145\/3649465","type":"journal-article","created":{"date-parts":[[2024,2,24]],"date-time":"2024-02-24T09:17:08Z","timestamp":1708766228000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Action Segmentation through Self-Supervised Video Features and Positional-Encoded Embeddings"],"prefix":"10.1145","volume":"20","author":[{"given":"Guilherme de A. P.","family":"Marques","sequence":"first","affiliation":[{"name":"Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jos\u00e9 Matheus C.","family":"Boaro","sequence":"additional","affiliation":[{"name":"Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Antonio Jos\u00e9 G.","family":"Busson","sequence":"additional","affiliation":[{"name":"BTG Pactual, S\u00e3o Paulo, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alan L. V.","family":"Guedes","sequence":"additional","affiliation":[{"name":"Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Julio Cesar","family":"Duarte","sequence":"additional","affiliation":[{"name":"Military Institute of Engineering (IME), Rio de Janeiro, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"S\u00e9rgio","family":"Colcher","sequence":"additional","affiliation":[{"name":"Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,8,16]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00129"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.495"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","unstructured":"Federico Becattini Tiberio Uricchio Lorenzo Seidenari Lamberto Ballan and Alberto Del Bimbo. 2020. Am I done? Predicting action progress in videos. ACM Trans. Multimedia Comput. Commun. Appl. 16 4 Article 119 (Dec.2020) 24 pages. DOI:10.1145\/3402447","DOI":"10.1145\/3402447"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10602-1_41"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00951"},{"key":"e_1_3_2_7_2","unstructured":"Jo\u00e3o Carreira Eric Noland Andras Banki-Horvath Chloe Hillier and Andrew Zisserman. 2018. A short note about Kinetics-600. CoRR abs\/1808.01340 (2018) 6 pages. arXiv:1808.01340http:\/\/arxiv.org\/abs\/1808.01340"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.502"},{"key":"e_1_3_2_9_2","doi-asserted-by":"crossref","unstructured":"Chien-Yi Chang De-An Huang Yanan Sui Li Fei-Fei and Juan Carlos Niebles. 2019. D3TW: Discriminative differentiable dynamic time warping for weakly supervised action alignment and segmentation. arXiv:1901.02598 [cs] abs\/1901.02598 (April2019) 10 pages. http:\/\/arxiv.org\/abs\/1901.02598arXiv: 1901.02598.","DOI":"10.1109\/CVPR.2019.00366"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","unstructured":"Weidong Chen Guorong Li Xinfeng Zhang Shuhui Wang Liang Li and Qingming Huang. 2023. Weakly supervised text-based actor-action video segmentation by clip-level multi-instance learning. ACM Trans. Multimedia Comput. Commun. Appl. 19 1 Article 12 (Jan.2023) 22 pages. DOI:10.1145\/3514250","DOI":"10.1145\/3514250"},{"key":"e_1_3_2_11_2","unstructured":"Junyoung Chung Caglar Gulcehre KyungHyun Cho and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. ArXiv abs\/1412.3555 (2014) 9 pages. https:\/\/api.semanticscholar.org\/CorpusID:5201925"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524273.3528187"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","unstructured":"Jeff Donahue Lisa Anne Hendricks Marcus Rohrbach Subhashini Venugopalan Sergio Guadarrama Kate Saenko and Trevor Darrell. 2017. Long-term recurrent convolutional networks for visual recognition and description. IEEE Transactions on Pattern Analysis and Machine Intelligence 39 4 (2017) 677\u2013691. DOI:10.1109\/TPAMI.2016.2599174","DOI":"10.1109\/TPAMI.2016.2599174"},{"key":"e_1_3_2_14_2","unstructured":"Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai Thomas Unterthiner Mostafa Dehghani Matthias Minderer Georg Heigold Sylvain Gelly Jakob Uszkoreit and Neil Houlsby. 2021. An Image is Worth \\(16\\times 16\\) Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations. OpenReview.net Virtual Event Austria 22 pages."},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00058"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00028"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00630"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.5555\/3305381.3305510"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.622"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICME51207.2021.9428317"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","unstructured":"Samitha Herath Mehrtash Harandi and Fatih Porikli. 2017. Going deeper into action recognition. Image Vision Comput. 60 C (Apr.2017) 4\u201321. DOI:10.1016\/j.imavis.2017.01.010","DOI":"10.1016\/j.imavis.2017.01.010"},{"key":"e_1_3_2_22_2","doi-asserted-by":"crossref","unstructured":"Sepp Hochreiter and J\u00fcrgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9 8 (1997) 32 pages.","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","unstructured":"Berthold K. P. Horn and Brian G. Schunck. 1981. Determining optical flow. Artificial Intelligence 17 1 (1981) 185\u2013203. DOI:10.1016\/0004-3702(81)90024-2","DOI":"10.1016\/0004-3702(81)90024-2"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","unstructured":"Fairouz Hussein and Massimo Piccardi. 2017. V-JAUNE: A framework for joint action recognition and video summarization. ACM Trans. Multimedia Comput. Commun. Appl. 13 2 Article 20 (Apr.2017) 19 pages. DOI:10.1145\/3063532","DOI":"10.1145\/3063532"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.223"},{"key":"e_1_3_2_26_2","unstructured":"Will Kay Joao Carreira Karen Simonyan Brian Zhang Chloe Hillier Sudheendra Vijayanarasimhan Fabio Viola Tim Green Trevor Back Paul Natsev Mustafa Suleyman and Andrew Zisserman. 2017. The kinetics human action video dataset. arXiv:1705.06950 [cs] abs\/1705.06950 (May2017) 22 pages. http:\/\/arxiv.org\/abs\/1705.06950arXiv: 1705.06950."},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.5244\/C.22.99"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.364"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01576"},{"key":"e_1_3_2_30_2","volume-title":"Time Warps, String Edits, and Macromolecules: The Theory and Practice of Sequence Comparison","author":"Kruskal J. B.","year":"1983","unstructured":"J. B. Kruskal and Mark Liberman. 1983. The symmetric time-warping problem: From continuous to discrete. In Time Warps, String Edits, and Macromolecules: The Theory and Practice of Sequence Comparison. Addison-Wesley, Massachussetts."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.105"},{"key":"e_1_3_2_32_2","doi-asserted-by":"crossref","unstructured":"Hilde Kuehne Alexander Richard and Juergen Gall. 2017. Weakly supervised learning of actions from transcripts. Computer Vision and Image Understanding 163 (2017) 78\u201389.","DOI":"10.1016\/j.cviu.2017.06.004"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","unstructured":"Hilde Kuehne Alexander Richard and Juergen Gall. 2020. A hybrid RNN-HMM approach for weakly supervised temporal action segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 42 4 (April2020) 765\u2013779. DOI:10.1109\/TPAMI.2018.2884469arXiv: 1906.01028.","DOI":"10.1109\/TPAMI.2018.2884469"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","unstructured":"H. W. Kuhn. 1955. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly 2 1-2 (1955) 83\u201397. DOI:10.1002\/nav.3800020109","DOI":"10.1002\/nav.3800020109"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01234"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58517-4_21"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00634"},{"key":"e_1_3_2_38_2","doi-asserted-by":"crossref","unstructured":"Jun Li and Sinisa Todorovic. 2021. Action shuffle alternating learning for unsupervised action segmentation. arXiv:2104.02116 [cs] abs\/2104.02116 (April2021) 9 pages. http:\/\/arxiv.org\/abs\/2104.02116arXiv: 2104.02116.","DOI":"10.1109\/CVPR46437.2021.01244"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00718"},{"key":"e_1_3_2_40_2","first-page":"281","volume-title":"In 5th Berkeley Symposium on Mathematical Statistics and Probability","author":"MacQueen J.","year":"1967","unstructured":"J. MacQueen. 1967. Some methods for classification and analysis of multivariate observations. In In 5th Berkeley Symposium on Mathematical Statistics and Probability. University of California Press, Berkeley, CA, 281\u2013297."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3428658.3430967"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.5753\/webmedia_estendido.2020.13082"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","unstructured":"Francisco Mic\u00f3-Engu\u00eddanos Wilmer Moina-Rivera Juan Guti\u00e9rrez-Aguado and Miguel Garcia-Pineda. 2023. Per-title and per-segment CRF estimation using DNNs for quality-based video coding. Expert Systems with Applications 227 (2023) 120289. DOI:10.1016\/j.eswa.2023.120289","DOI":"10.1016\/j.eswa.2023.120289"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","unstructured":"Ronald Poppe. 2010. A survey on vision-based human action recognition. Image and Vision Computing 28 6 (2010) 976\u2013990. DOI:10.1016\/j.imavis.2009.11.014","DOI":"10.1016\/j.imavis.2009.11.014"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.140"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00771"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3323503.3345029"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00914"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01107"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2008.4587730"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00873"},{"key":"e_1_3_2_52_2","doi-asserted-by":"crossref","unstructured":"Peter Shaw Jakob Uszkoreit and Ashish Vaswani. 2018. Self-attention with relative position representations. CoRR abs\/1803.02155 (2018) 5 pages. arXiv:1803.02155http:\/\/arxiv.org\/abs\/1803.02155","DOI":"10.18653\/v1\/N18-2074"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","unstructured":"Qinghongya Shi Hong-Bo Zhang Zhe Li Ji-Xiang Du Qing Lei and Jing-Hua Liu. 2022. Shuffle-invariant network for action recognition in videos. ACM Trans. Multimedia Comput. Commun. Appl. 18 3 Article 69 (Mar.2022) 18 pages. DOI:10.1145\/3485665","DOI":"10.1145\/3485665"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46448-0_31"},{"key":"e_1_3_2_55_2","first-page":"568","volume-title":"Advances in Neural Information Processing Systems 27","author":"Simonyan Karen","year":"2014","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (Eds.). Curran Associates, Inc., Cambridge, MA, USA, 568\u2013576. http:\/\/papers.nips.cc\/paper\/5353-two-stream-convolutional-networks-for-action-recognition-in-videos.pdf"},{"key":"e_1_3_2_56_2","doi-asserted-by":"crossref","unstructured":"Yaser Souri Mohsen Fayyaz Luca Minciullo Gianpiero Francesca and Juergen Gall. 2021. Fast weakly supervised action segmentation using mutual consistency. IEEE Transactions on Pattern Analysis and Machine Intelligence 44 10 (2021) 6196\u20136208.","DOI":"10.1109\/TPAMI.2021.3089127"},{"key":"e_1_3_2_57_2","unstructured":"Deqing Sun Xiaodong Yang Ming-Yu Liu and Jan Kautz. 2017. PWC-Net: CNNs for optical flow using pyramid warping and cost volume. CoRR abs\/1709.02371 (2017) 18 pages. arXiv:1709.02371http:\/\/arxiv.org\/abs\/1709.02371"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","unstructured":"Javier S\u00e1nchez P\u00e9rez Enric Meinhardt-Llopis and Gabriele Facciolo. 2013. TV-L1 optical flow estimation. Image Processing On Line 3 (2013) 137\u2013150. 10.5201\/ipol.2013.26","DOI":"10.5201\/ipol.2013.26"},{"key":"e_1_3_2_59_2","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez \u0141ukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017) 11 pages."},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/WACV48630.2021.00128"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.441"},{"key":"e_1_3_2_62_2","unstructured":"Xiaolong Wang Ross B. Girshick Abhinav Gupta and Kaiming He. 2017. Non-local neural networks. CoRR abs\/1711.07971 (2017) 10 pages. arXiv:1711.07971http:\/\/arxiv.org\/abs\/1711.07971"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00631"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","unstructured":"Huifen Xia and Yongzhao Zhan. 2020. A survey on temporal action localization. IEEE Access 8 (2020) 70477\u201370487. DOI:10.1109\/ACCESS.2020.2986861","DOI":"10.1109\/ACCESS.2020.2986861"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00681"},{"key":"e_1_3_2_66_2","first-page":"9","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR \u201915)","author":"Ng Joe Yue-Hei","year":"2015","unstructured":"Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. 2015. Beyond short snippets: Deep networks for video classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR \u201915). IEEE, Boston, MA, USA, 9 pages."}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3649465","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3649465","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:03:17Z","timestamp":1750291397000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3649465"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,16]]},"references-count":65,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2024,9,30]]}},"alternative-id":["10.1145\/3649465"],"URL":"https:\/\/doi.org\/10.1145\/3649465","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"value":"1551-6857","type":"print"},{"value":"1551-6865","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,16]]},"assertion":[{"value":"2023-04-22","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-09","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-08-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}