{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T10:16:36Z","timestamp":1785406596863,"version":"3.56.0"},"reference-count":61,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T00:00:00Z","timestamp":1784505600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004032","name":"Jilin University","doi-asserted-by":"publisher","award":["2025SZQHYB03"],"award-info":[{"award-number":["2025SZQHYB03"]}],"id":[{"id":"10.13039\/501100004032","id-type":"DOI","asserted-by":"publisher"}]},{"award":["2025SZQHYB03"],"award-info":[{"award-number":["2025SZQHYB03"]}],"id":[{"id":"https:\/\/ror.org\/00js3aw79","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Data"],"abstract":"<jats:p>We integrated communication theory with advanced computer vision techniques to propose a novel approach for fine-grained content extraction from short videos. Unlike methods focused on summarization or subtitle generation for longer videos, our approach emphasizes extracting detailed content and understanding the intricate narrative structure of short videos. By employing scene segmentation, similarity-based filtering algorithms, and support vector machines, the method identifies keyframes that capture precise visual details. Further, it generates semantically accurate textual descriptions using the mPLUG model, enabling an in-depth understanding of video content. Using a dataset of short videos from the cultural and tourism domain, we validated the proposed method. Experimental results demonstrate that our approach achieves high precision in identifying and understanding detailed visual elements, effectively bridging the gap between visual representation and semantic meaning. Additionally, the study explores the influence of different video content types, interference factors, and image description models on fine-grained content extraction, highlighting its potential for improving intelligent analysis of short-video data.<\/jats:p>","DOI":"10.3390\/data11070179","type":"journal-article","created":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T06:09:09Z","timestamp":1784527749000},"page":"179","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["An Algorithm for Fine-Grained Content Extraction and Understanding in Short Videos"],"prefix":"10.3390","volume":"11","author":[{"given":"Yanqi","family":"Wan","sequence":"first","affiliation":[{"name":"School of Journalism and Communication, Jilin University, Changchun 130012, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-5272-6203","authenticated-orcid":false,"given":"Shuya","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Journalism and Information Communication, Huazhong University of Science and Technology, Wuhan 430074, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yi","family":"Xu","sequence":"additional","affiliation":[{"name":"School of Public Administration, Jilin University, Changchun 130012, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kunfang","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Economics, Jilin University, Changchun 130012, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Heyi","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Resources and Environment, Jilin Agricultural University, Changchun 130118, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4878-2323","authenticated-orcid":false,"given":"Mingzheng","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Journalism and Communication, Jilin University, Changchun 130012, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,7,20]]},"reference":[{"key":"ref_1","unstructured":"Shutsko, A. (2020). User-generated short video content in social media. A case study of TikTok. Proceedings of the Social Computing and Social Media. Participation, User Experience, Consumer Experience, and Applications of Social Computing: 12th International Conference, SCSM 2020, Held as Part of the 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, 19\u201324 July 2020, Springer. Proceedings, Part II 22."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1177\/2050157920952120","article-title":"The co-evolution of two Chinese mobile short video apps: Parallel platformization of Douyin and TikTok","volume":"9","author":"Kaye","year":"2021","journal-title":"Mob. Media Commun."},{"key":"ref_3","unstructured":"Yang, S., Zhao, Y., and Ma, Y. (2019, January 12\u201314). Analysis of the reasons and development of short video application-Taking Tik Tok as an example. Proceedings of the 2019 9th International Conference on Information and Social Science (ICISS 2019), Manila, Philippines."},{"key":"ref_4","unstructured":"(2024, July 16). Social Media-Statistics & Facts. Available online: https:\/\/www.statista.com\/topics\/1164\/social-networks\/#topicOverview."},{"key":"ref_5","first-page":"1","article-title":"Social media use in 2021","volume":"1","author":"Auxier","year":"2021","journal-title":"Pew Res. Cent."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Chen, T.S., Siarohin, A., Menapace, W., Deyneka, E., Chao, H.w., Jeon, B.E., Fang, Y., Lee, H.Y., Ren, J., and Yang, M.H. (2024, January 24\u201327). Panda-70m: Captioning 70m videos with multiple cross-modality teachers. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR52733.2024.01265"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1080\/10584609.2020.1723752","article-title":"In validations we trust? The impact of imperfect human annotations as a gold standard on the quality of validation of automated content analysis","volume":"37","author":"Song","year":"2020","journal-title":"Political Commun."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1080\/19312458.2023.2277956","article-title":"Automated visual analysis for the study of social media effects: Opportunities, approaches, and challenges","volume":"18","author":"Peng","year":"2024","journal-title":"Commun. Methods Meas."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Nyhuis, D., Ringwald, T., Rittmann, O., Gschwend, T., and Stiefelhagen, R. (2021). Automated video analysis for social science research 1. Handbook of Computational Social Science, Volume 2, Routledge.","DOI":"10.4324\/9781003025245-26"},{"key":"ref_10","unstructured":"Nassauer, A., and Legewie, N.M. (2022). Video Data Analysis: How to Use 21st Century Video in the Social Sciences, Sage."},{"key":"ref_11","unstructured":"Ye, K., Buettner, K., and Kovashka, A. (2018). Story understanding in video advertisements. arXiv."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2183","DOI":"10.1080\/13683500.2018.1501009","article-title":"A discussion of video as a data collection tool","volume":"22","author":"Griffin","year":"2019","journal-title":"Curr. Issues Tour."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"101179","DOI":"10.1016\/j.cogsys.2023.101179","article-title":"WordIllusion: An adversarial text generation algorithm based on human cognitive system","volume":"83","author":"Fu","year":"2024","journal-title":"Cogn. Syst. Res."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"188415","DOI":"10.1109\/ACCESS.2020.3028392","article-title":"Research on the motives affecting the behavior of short video\u2019s creators","volume":"8","author":"Bi","year":"2020","journal-title":"IEEE Access"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1007\/s41109-019-0226-0","article-title":"Movienet: A movie multilayer network model using visual and textual semantic cues","volume":"4","author":"Mourchid","year":"2019","journal-title":"Appl. Netw. Sci."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Bu, Y., Sheng, Q., Cao, J., Qi, P., Wang, D., and Li, J. (2024). FakingRecipe: Detecting Fake News on Short Video Platforms from the Perspective of Creative Process. arXiv.","DOI":"10.1145\/3664647.3680663"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1838","DOI":"10.1109\/JPROC.2021.3117472","article-title":"Video summarization using deep neural networks: A survey","volume":"109","author":"Apostolidis","year":"2021","journal-title":"Proc. IEEE"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"7021","DOI":"10.1007\/s11042-023-15859-z","article-title":"A keyframe selection for summarization of informative activities using clustering in surveillance videos","volume":"83","author":"Pandian","year":"2024","journal-title":"Multimed. Tools Appl."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"4881","DOI":"10.1007\/s00521-021-06132-1","article-title":"Key moment extraction for designing an agglomerative clustering algorithm-based video summarization framework","volume":"35","author":"Yasmin","year":"2023","journal-title":"Neural Comput. Appl."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2761","DOI":"10.2174\/2666255813999200710131444","article-title":"Comparative analysis of keyframe extraction techniques for video summarization","volume":"14","author":"Parikh","year":"2021","journal-title":"Recent Adv. Comput. Sci. Commun. (Former. Recent Patents Comput. Sci.)"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"3013","DOI":"10.1109\/TIP.2023.3275069","article-title":"Video summarization with spatiotemporal vision transformer","volume":"32","author":"Hsu","year":"2023","journal-title":"IEEE Trans. Image Process."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1711","DOI":"10.1007\/s13042-023-01992-6","article-title":"Fine-tuning pretrained transformer encoders for sequence-to-sequence learning","volume":"15","author":"Bao","year":"2024","journal-title":"Int. J. Mach. Learn. Cybern."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Feng, S., Xie, Y., Wei, Y., Yan, J., and Wang, Q. (2022). Transformer-based video summarization with spatial-temporal representation. Proceedings of the 2022 8th International Conference on Big Data and Information Analytics (BigDIA), IEEE.","DOI":"10.1109\/BigDIA56350.2022.9874248"},{"key":"ref_24","unstructured":"Liu, Y.T., Li, Y.J., and Wang, Y.C.F. (December, January 30). Transforming multi-concept attention into video summarization. Proceedings of the Asian Conference on Computer Vision, Kyoto, Japan."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"107677","DOI":"10.1016\/j.patcog.2020.107677","article-title":"Exploring global diverse attention via pairwise temporal relation for video summarization","volume":"111","author":"Li","year":"2021","journal-title":"Pattern Recognit."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"445","DOI":"10.1109\/TCSVT.2022.3197819","article-title":"Unsupervised video summarization via deep reinforcement learning with shot-level semantics","volume":"33","author":"Yuan","year":"2022","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"3278","DOI":"10.1109\/TCSVT.2020.3037883","article-title":"AC-SUM-GAN: Connecting actor-critic and generative adversarial networks for unsupervised video summarization","volume":"31","author":"Apostolidis","year":"2020","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Li, H., Zhu, J., Ma, C., Zhang, J., and Zong, C. (2017, January 7\u201311). Multi-modal summarization for asynchronous collection of text, image, audio and video. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.","DOI":"10.18653\/v1\/D17-1114"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Huang, J.H., Murn, L., Mrak, M., and Worring, M. (2021, January 16\u201319). Gpt2mvs: Generative pre-trained transformer-2 for multi-modal video summarization. Proceedings of the 2021 International Conference on Multimedia Retrieval, Taipei, Taiwan.","DOI":"10.1145\/3460426.3463662"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"3017","DOI":"10.1109\/TIP.2022.3163855","article-title":"Relational reasoning over spatial-temporal graphs for video summarization","volume":"31","author":"Zhu","year":"2022","journal-title":"IEEE Trans. Image Process."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"483","DOI":"10.1016\/j.ins.2021.07.012","article-title":"Graph-based structural difference analysis for video summarization","volume":"577","author":"Chai","year":"2021","journal-title":"Inf. Sci."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"32635","DOI":"10.1007\/s11042-023-14925-w","article-title":"Data-driven enabled approaches for criteria-based video summarization: A comprehensive survey, taxonomy, and future directions","volume":"82","author":"Sabha","year":"2023","journal-title":"Multimed. Tools Appl."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1016\/j.visinf.2019.12.002","article-title":"Examining the use of narrative constructs in data videos","volume":"4","author":"Cao","year":"2020","journal-title":"Vis. Inform."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"425","DOI":"10.1109\/OJSP.2021.3090333","article-title":"RAPIQUE: Rapid and accurate video quality prediction of user generated content","volume":"2","author":"Tu","year":"2021","journal-title":"IEEE Open J. Signal Process."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"505","DOI":"10.1002\/hipo.23483","article-title":"Significance of visual scene-based learning in the hippocampal systems across mammalian species","volume":"33","author":"Lee","year":"2023","journal-title":"Hippocampus"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3167\/proj.2012.060102","article-title":"The attentional theory of cinematic continuity","volume":"6","author":"Smith","year":"2012","journal-title":"Projections"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1111\/tops.12455","article-title":"The scene perception & event comprehension theory (SPECT) applied to visual narratives","volume":"12","author":"Loschky","year":"2020","journal-title":"Top. Cogn. Sci."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Xu, X., Yang, L., Yip, D., Fan, M., Wei, Z., and Qu, H. (May, January 30). From \u2018wow\u2019to \u2018why\u2019: Guidelines for creating the opening of a data video with cinematic styles. Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA.","DOI":"10.1145\/3491102.3501896"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Kress, G., and Van Leeuwen, T. (2020). Reading Images: The Grammar of Visual Design, Routledge.","DOI":"10.4324\/9781003099857"},{"key":"ref_40","unstructured":"Kress, G. (2009). Multimodality: A Social Semiotic Approach to Contemporary Communication, Routledge."},{"key":"ref_41","unstructured":"Barthes, R. (1968). Elements of Semiology, Jonathan Cape."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"100502","DOI":"10.1016\/j.dcm.2021.100502","article-title":"What are digital media?","volume":"41","author":"Bateman","year":"2021","journal-title":"Discourse Context Media"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"73","DOI":"10.16875\/stem.2021.22.2.73","article-title":"Critical literacy and semiotic juxtaposition: The possibilities of multimodal pedagogy in korea","volume":"22","author":"Pederson","year":"2021","journal-title":"J. Engl. Teach. Through Movies Media"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1016\/j.neucom.2020.10.042","article-title":"New ideas and trends in deep multimodal content understanding: A review","volume":"426","author":"Chen","year":"2021","journal-title":"Neurocomputing"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"871","DOI":"10.1007\/s11165-021-10038-2","article-title":"Meaning making with multiple representations: A case study of a preservice teacher creating a digital explanation","volume":"52","author":"Nielsen","year":"2022","journal-title":"Res. Sci. Educ."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Gibson, J.J. (2014). The Ecological Approach to Visual Perception: Classic Edition, Psychology Press.","DOI":"10.4324\/9781315740218"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"233","DOI":"10.4135\/9781473984066.n14","article-title":"The affordances of social media platforms","volume":"1","author":"Bucher","year":"2018","journal-title":"SAGE Handb. Soc. Media"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18146\/tmg.815","article-title":"What to do with 2.000.000 historical press photos? The challenges and opportunities of applying a scene detection algorithm to a digitised press photo collection","volume":"25","author":"Wevers","year":"2022","journal-title":"TMG J. Media Hist."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"363","DOI":"10.2307\/2529786","article-title":"An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"409","DOI":"10.1016\/j.neucom.2020.04.035","article-title":"Survey on visual rhythms: A spatio-temporal representation for video sequences","volume":"402","author":"Vieira","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_51","unstructured":"(2024, June 10). PySceneDetect. Available online: https:\/\/github.com\/Breakthrough\/PySceneDetect\/."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Hussain, Z., Zhang, M., Zhang, X., Ye, K., Thomas, C., Agha, Z., Ong, N., and Kovashka, A. (2017, January 21\u201326). Automatic understanding of image and video advertisements. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.123"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Li, C., Xu, H., Tian, J., Wang, W., Yan, M., Bi, B., Ye, J., Chen, H., Xu, G., and Cao, Z. (2022). mplug: Effective and efficient vision-language learning by cross-modal skip-connections. arXiv.","DOI":"10.18653\/v1\/2022.emnlp-main.488"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7\u201312). Bleu: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.","DOI":"10.3115\/1073083.1073135"},{"key":"ref_55","unstructured":"Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., and Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv."},{"key":"ref_56","unstructured":"Anderson, P., Fernando, B., Johnson, M., and Gould, S. (2016). Spice: Semantic propositional image caption evaluation. Proceedings of the Computer Vision\u2013ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, 11\u201314 October 2016, Springer. Proceedings, Part V 14."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Vedantam, R., Lawrence Zitnick, C., and Parikh, D. (2015, January 7\u201312). Cider: Consensus-based image description evaluation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7299087"},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Popovi\u0107, M. (2015, January 17\u201318). chrF: Character n-gram F-score for automatic MT evaluation. Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisbon, Portugal.","DOI":"10.18653\/v1\/W15-3049"},{"key":"ref_59","unstructured":"Li, J., Li, D., Xiong, C., and Hoi, S. (2022). Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. Proceedings of the International Conference on Machine Learning, PMLR."},{"key":"ref_60","unstructured":"Kumar, A. (2026, June 21). The Illustrated Image Captioning Using Transformers. Ankur | NLP Enthusiast. Available online: https:\/\/ankur3107.github.io\/blogs\/the-illustrated-image-captioning-using-transformers\/."},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., and Saenko, K. (2015, January 7\u201313). Sequence to sequence-video to text. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.515"}],"container-title":["Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2306-5729\/11\/7\/179\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T06:36:55Z","timestamp":1784529415000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2306-5729\/11\/7\/179"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,20]]},"references-count":61,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2026,7]]}},"alternative-id":["data11070179"],"URL":"https:\/\/doi.org\/10.3390\/data11070179","relation":{},"ISSN":["2306-5729"],"issn-type":[{"value":"2306-5729","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,20]]}}}