{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,3]],"date-time":"2026-03-03T13:33:27Z","timestamp":1772544807895,"version":"3.50.1"},"reference-count":52,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2025,1,10]],"date-time":"2025-01-10T00:00:00Z","timestamp":1736467200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada (NSERC)","doi-asserted-by":"publisher","award":["ALLRP 577388-22"],"award-info":[{"award-number":["ALLRP 577388-22"]}],"id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada (NSERC)","doi-asserted-by":"publisher","award":["ALLRP 570580-2021"],"award-info":[{"award-number":["ALLRP 570580-2021"]}],"id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>The safety and efficiency of assembly lines are critical to manufacturing, but human supervisors cannot oversee all activities simultaneously. This study addresses this challenge by performing a comparative study to construct an initial real-time, semi-supervised temporal action recognition setup for monitoring worker actions on assembly lines. Various feature extractors and localization models were benchmarked using a new assembly dataset, with the I3D model achieving an average mAP@IoU=0.1:0.7 of 85% without optical flow or fine-tuning. The comparative study was extended to self-supervised learning via a modified SPOT model, which achieved a mAP@IoU=0.1:0.7 of 65% with just 10% of the data labeled using extractor architectures from the fully-supervised portion. Milestones include high scores for both fully and semi-supervised learning on this dataset and improved SPOT performance on ANet1.3. This study identified the particularities of the problem, which were leveraged and referenced to explain the results observed in semi-supervised scenarios. The findings highlight the potential for developing a scalable solution in the future, providing labour efficiency and safety compliance for manufacturers.<\/jats:p>","DOI":"10.3390\/jimaging11010017","type":"journal-article","created":{"date-parts":[[2025,1,10]],"date-time":"2025-01-10T09:24:42Z","timestamp":1736501082000},"page":"17","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Supervised and Self-Supervised Learning for Assembly Line Action Recognition"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-3327-0218","authenticated-orcid":false,"given":"Christopher","family":"Indris","sequence":"first","affiliation":[{"name":"Department of Computer Science, Toronto Metropolitan University, Toronto, ON M5B 2K3, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-6534-598X","authenticated-orcid":false,"given":"Fady","family":"Ibrahim","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Toronto Metropolitan University, Toronto, ON M5B 2K3, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8722-3300","authenticated-orcid":false,"given":"Hatem","family":"Ibrahem","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Toronto Metropolitan University, Toronto, ON M5B 2K3, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7060-8124","authenticated-orcid":false,"given":"G\u00f6tz","family":"Bramesfeld","sequence":"additional","affiliation":[{"name":"Department of Aerospace Engineering, Toronto Metropolitan University, Toronto, ON M5B 2K3, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jie","family":"Huo","sequence":"additional","affiliation":[{"name":"IFIVEO Canada Inc., Windsor, ON N8W 0A6, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1648-7071","authenticated-orcid":false,"given":"Hafiz Mughees","family":"Ahmad","sequence":"additional","affiliation":[{"name":"IFIVEO Canada Inc., Windsor, ON N8W 0A6, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Syed Khizer","family":"Hayat","sequence":"additional","affiliation":[{"name":"IFIVEO Canada Inc., Windsor, ON N8W 0A6, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3182-104X","authenticated-orcid":false,"given":"Guanghui","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Toronto Metropolitan University, Toronto, ON M5B 2K3, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2025,1,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Rahman, R., Indris, C., Bramesfeld, G., Zhang, T., Li, K., Chen, X., Grijalva, I., McCornack, B., Flippo, D., and Sharda, A. (2024). A New Dataset and Comparative Study for Aphid Cluster Detection and Segmentation in Sorghum Fields. J. Imaging, 10.","DOI":"10.3390\/jimaging10050114"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1564","DOI":"10.1002\/ohn.411","article-title":"Interpretable computer vision to detect and classify structural laryngeal lesions in digital flexible laryngoscopic images","volume":"169","author":"Bur","year":"2023","journal-title":"Otolaryngol.-Head Neck Surg."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Xiao, X., Hu, Q.V., and Wang, G. (2023). Edge-aware multi-task network for integrating quantification segmentation and uncertainty prediction of liver tumor on multi-modality non-contrast MRI. Medical Image Computing and Computer-Assisted Intervention, Springer.","DOI":"10.1007\/978-3-031-43901-8_62"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Patel, K., Li, F., and Wang, G. (2024, January 14\u201317). Multi-Layer Dense Attention Decoder for Polyp Segmentation. Proceedings of the 2024 14th International Conference on Biomedical Engineering and Technology, Seoul, Republic of Korea.","DOI":"10.1145\/3678935.3678955"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1680","DOI":"10.1109\/TCI.2024.3488563","article-title":"Occcasnet: Occlusion-aware cascade cost volume for light field depth estimation","volume":"10","author":"Chao","year":"2024","journal-title":"IEEE Trans. Comput. Imaging"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Cheng, S., Jin, Y., Harrison, S.P., Quilodr\u00e1n-Casas, C., Prentice, I.C., Guo, Y.K., and Arcucci, R. (2022). Parameter flexible wildfire prediction using machine learning techniques: Forward and inverse modelling. Remote Sens., 14.","DOI":"10.3390\/rs14133228"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"117339","DOI":"10.1016\/j.cma.2024.117339","article-title":"Dynamical system prediction from sparse observations using deep neural networks with Voronoi tessellation and physics constraint","volume":"432","author":"Wang","year":"2024","journal-title":"Comput. Methods Appl. Mech. Eng."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"711","DOI":"10.1016\/j.procir.2019.01.106","article-title":"Vision sensor based action recognition for improving efficiency and quality under the environment of industry 4.0","volume":"80","author":"Wang","year":"2019","journal-title":"Procedia CIRP"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Reining, C., Niemann, F., Moya Rueda, F., Fink, G.A., and Ten Hompel, M. (2019). Human Activity Recognition for Production and Logistics\u2014A Systematic Literature Review. Information, 10.","DOI":"10.3390\/info10080245"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1192","DOI":"10.1109\/SURV.2012.110112.00192","article-title":"A Survey on Human Activity Recognition using Wearable Sensors","volume":"15","author":"Lara","year":"2013","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1177","DOI":"10.1016\/j.procir.2019.03.288","article-title":"Activity recognition in manual manufacturing: Detecting screwing processes from sensor data","volume":"81","author":"Bauernhansl","year":"2019","journal-title":"Procedia CIRP"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1016\/j.promfg.2020.01.288","article-title":"Action recognition in manufacturing assembly using multimodal sensor fusion","volume":"39","author":"Tao","year":"2019","journal-title":"Procedia Manuf."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2014191","DOI":"10.1080\/08839514.2021.2014191","article-title":"Hand gesture recognition of methods-time measurement-1 motions in manual assembly tasks using graph convolutional networks","volume":"36","author":"Riedel","year":"2022","journal-title":"Appl. Artif. Intell."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"633","DOI":"10.1007\/s10845-021-01815-x","article-title":"An individualized system of skeletal data-based CNN classifiers for action recognition in manufacturing assembly","volume":"34","author":"Qin","year":"2023","journal-title":"J. Intell. Manuf."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1145\/3065386","article-title":"ImageNet classification with deep convolutional neural networks","volume":"60","author":"Krizhevsky","year":"2017","journal-title":"Commun. ACM"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"4302","DOI":"10.1109\/TPAMI.2022.3193611","article-title":"Deep Learning-Based Action Detection in Untrimmed Videos: A Survey","volume":"45","author":"Vahdani","year":"2023","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_17","unstructured":"Indris, C. (2024, December 20). ActionFormer. Available online: https:\/\/github.com\/chrisindris\/actionformer_release."},{"key":"ref_18","unstructured":"Tang, T., Kim, K., and Sohn, K. (2023). TemporalMaxer: Maximize Temporal Context with only Max Pooling for Temporal Action Localization. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Wang, L., Huang, B., Zhao, Z., Tong, Z., He, Y., Wang, Y., Wang, Y., and Qiao, Y. (2023, January 17\u201324). VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking. Proceedings of the 2023 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.01398"},{"key":"ref_20","unstructured":"Wang, Y., Li, K., Li, Y., He, Y., Huang, B., Zhao, Z., Zhang, H., Xu, J., Liu, Y., and Wang, Z. (2022). InternVideo: General Video Foundation Models via Generative and Discriminative Learning. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Carreira, J., and Zisserman, A. (2017, January 21\u201326). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.502"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., and Paluri, M. (2018, January 18\u201323). A Closer Look at Spatiotemporal Convolutions for Action Recognition. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00675"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Ji, J., Cao, K., and Niebles, J.C. (November, January 27). Learning Temporal Action Proposals With Fewer Labels. Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.","DOI":"10.1109\/ICCV.2019.00717"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Wang, X., Zhang, S., Qing, Z., Shao, Y., Gao, C., and Sang, N. (2021, January 20\u201325). Self-Supervised Learning for Semi-Supervised Temporal Action Proposal. Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00194"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Avidan, S., Brostow, G., Ciss\u00e9, M., Farinella, G.M., and Hassner, T. (2021, January 20\u201325). Semi-Supervised Temporal Action Detection with Proposal-Free Masking. Proceedings of the Computer Vision\u2014ECCV 2022, Nashville, TN, USA.","DOI":"10.1007\/978-3-031-19800-7"},{"key":"ref_26","unstructured":"Indris, C., and Other contributors (2024, December 20). SPOT. Available online: https:\/\/github.com\/chrisindris\/SPOT."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Gudlin, M., Hegedi\u0107, M., Golec, M., and Kolar, D. (2024). Improving Time Study Methods Using Deep Learning-Based Action Segmentation Models. Appl. Sci., 14.","DOI":"10.3390\/app14031185"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Jo, H., Gwon, H., Jo, S., and Jung, C. (Preprint, 2024). An Empirical Study on Factors of Influence for Single-Frame Supervised Temporal Action Detection, Preprint.","DOI":"10.21203\/rs.3.rs-4620944\/v1"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Li, N., Zhang, D., Li, Y., and Zhang, Q. (2024). Repetitive assembly basic action detection and standard work measurement based on deep learning. Int. J. Comput. Integr. Manuf., 1\u201319.","DOI":"10.1080\/0951192X.2024.2372281"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"201","DOI":"10.57062\/ijpem-st.2023.0073","article-title":"AI-assisted Monitoring of Human-centered Assembly: A Comprehensive Review","volume":"1","author":"Selvaraj","year":"2023","journal-title":"Int. J. Precis. Eng.-Manuf.-Smart Technol."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Heilbron, F.C., Escorcia, V., Ghanem, B., and Niebles, J.C. (2015, January 7\u201312). ActivityNet: A large-scale video benchmark for human activity understanding. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298698"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.cviu.2016.10.018","article-title":"The THUMOS challenge on action recognition for videos \u201cin the wild\u201d","volume":"155","author":"Idrees","year":"2017","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Alwassel, H., Giancola, S., and Ghanem, B. (2021, January 11\u201317). TSP: Temporally-Sensitive Pretraining of Video Encoders for Localization Tasks. Proceedings of the 2021 IEEE\/CVF International Conference on Computer Vision Workshops (ICCVW), Montreal, BC, Canada.","DOI":"10.1109\/ICCVW54120.2021.00356"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Vedaldi, A., Bischof, H., Brox, T., and Frahm, J.M. (2020, January 23\u201328). RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. Proceedings of the Computer Vision\u2014ECCV 2020, Glasgow, UK.","DOI":"10.1007\/978-3-030-58548-8"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Avidan, S., Brostow, G., Ciss\u00e9, M., Farinella, G.M., and Hassner, T. (2022, January 23\u201327). ActionFormer: Localizing Moments of Actions with Transformers. Proceedings of the Computer Vision\u2014ECCV 2022, Tel Aviv, Israel.","DOI":"10.1007\/978-3-031-19800-7"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7\u201312). Going deeper with convolutions. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"ImageNet Large Scale Visual Recognition Challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Wang, L., Xiong, Y., Lin, D., and Gool, L.V. (2017, January 21\u201326). UntrimmedNets for Weakly Supervised Action Recognition and Detection. Proceedings of the 2017 IEEE conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.678"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Zhai, X., Oliver, A., Kolesnikov, A., and Beyer, L. (2019). S4L: Self-Supervised Semi-Supervised Learning. arXiv.","DOI":"10.1109\/ICCV.2019.00156"},{"key":"ref_41","unstructured":"Dubois, Y., Hashimoto, T., Ermon, S., and Liang, P. (2022). Improving Self-Supervised Learning by Characterizing Idealized Representations. arXiv."},{"key":"ref_42","unstructured":"Zaiem, S., Parcollet, T., and Essid, S. (2024). Less Forgetting for Better Generalization: Exploring Continual-learning Fine-tuning Methods for Speech Self-supervised Representations. arXiv."},{"key":"ref_43","unstructured":"Zakarias, G.W., Hansen, L.K., and Tan, Z.H. (2024). BiSSL: Bilevel Optimization for Self-Supervised Pre-Training and Fine-Tuning. arXiv."},{"key":"ref_44","unstructured":"Li, X., Xiong, H., An, H., Xu, C., and Dou, D. (2020). RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr. arXiv."},{"key":"ref_45","unstructured":"Hamilton, M., Shelhamer, E., and Freeman, W.T. (2020). It Is Likely That Your Loss Should be a Likelihood. arXiv."},{"key":"ref_46","unstructured":"Li, J., Zhou, P., Xiong, C., and Hoi, S.C.H. (2021). Prototypical Contrastive Learning of Unsupervised Representations. arXiv."},{"key":"ref_47","unstructured":"Iashin, V., and Other contributors (2024, November 19). Video Features. Available online: https:\/\/github.com\/v-iashin\/video_features."},{"key":"ref_48","unstructured":"Indris, C. (2024, November 19). VideoMAEv2. Available online: https:\/\/github.com\/chrisindris\/VideoMAEv2."},{"key":"ref_49","unstructured":"Indris, C. (2024, November 19). Video Features. Available online: https:\/\/github.com\/chrisindris\/video_features."},{"key":"ref_50","unstructured":"Indris, C. (2024, November 19). TSP. Available online: https:\/\/github.com\/chrisindris\/TSP."},{"key":"ref_51","unstructured":"Ibrahim, F. (2024, November 19). TemporalMaxer. Available online: https:\/\/github.com\/f1ibrahim-tmu\/TemporalMaxer."},{"key":"ref_52","unstructured":"Loshchilov, I., and Hutter, F. (2019). Decoupled Weight Decay Regularization. arXiv."}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/11\/1\/17\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,8]],"date-time":"2025-10-08T10:26:36Z","timestamp":1759919196000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/11\/1\/17"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,10]]},"references-count":52,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,1]]}},"alternative-id":["jimaging11010017"],"URL":"https:\/\/doi.org\/10.3390\/jimaging11010017","relation":{},"ISSN":["2313-433X"],"issn-type":[{"value":"2313-433X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,10]]}}}