{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,22]],"date-time":"2026-07-22T15:03:04Z","timestamp":1784732584476,"version":"3.55.0"},"reference-count":53,"publisher":"Association for Computing Machinery (ACM)","issue":"8","license":[{"start":{"date-parts":[[2026,7,22]],"date-time":"2026-07-22T00:00:00Z","timestamp":1784678400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"name":"National Key Research and Development Program of China","award":["2025ZD1008102"],"award-info":[{"award-number":["2025ZD1008102"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2026,8,31]]},"abstract":"<jats:p>\n                    This article targets to enhance the diffusion-based text-to-video generation by improving the two input prompts, including the noise and the text. Accommodated with this goal, we propose POS, a\n                    <jats:italic toggle=\"yes\">P<\/jats:italic>\n                    rompt\n                    <jats:italic toggle=\"yes\">O<\/jats:italic>\n                    ptimization\n                    <jats:italic toggle=\"yes\">S<\/jats:italic>\n                    uite to boost text-to-video models. POS is motivated by two observations: (1)\n                    <jats:italic toggle=\"yes\">Video generation shows instability in terms of noise<\/jats:italic>\n                    . Given the same text, different noises lead to videos that differ significantly in terms of both frame quality and temporal consistency. This observation implies that there exists an optimal noise matched to each textual input; To capture the potential noise, we propose an optimal noise approximator to approach the potential optimal noise. Particularly, the optimal noise approximator initially searches a video that closely relates to the text prompt and then inverts it into the noise space to serve as an improved noise prompt for the textual input. (2)\n                    <jats:italic toggle=\"yes\">Improving the text prompt via LLMs often causes semantic deviation<\/jats:italic>\n                    . Many existing text-to-vision works have utilized LLMs to improve the text prompts for generation enhancement. However, existing methods often neglect the semantic alignment between the original text and the rewritten one. In response to this issue, we design a semantic-preserving rewriter to impose constraints in both rewriting and denoising phrases to preserve semantic consistency. Extensive experiments on popular benchmarks show that our POS can improve the text-to-video models with a clear margin.\n                  <\/jats:p>","DOI":"10.1145\/3820056","type":"journal-article","created":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T13:45:57Z","timestamp":1781099157000},"page":"1-17","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["POS: A Prompts Optimization Suite for Augmenting Text-to-Video Generation"],"prefix":"10.1145","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-1400-6005","authenticated-orcid":false,"given":"Shijie","family":"Ma","sequence":"first","affiliation":[{"name":"Zhejiang Lab, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1428-4417","authenticated-orcid":false,"given":"Huayi","family":"Xu","sequence":"additional","affiliation":[{"name":"Zhejiang Lab, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-9698-8508","authenticated-orcid":false,"given":"Mengjian","family":"Li","sequence":"additional","affiliation":[{"name":"Zhejiang Lab, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-0618-7859","authenticated-orcid":false,"given":"Haiyang","family":"Li","sequence":"additional","affiliation":[{"name":"Hefei University of Technology, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4036-1391","authenticated-orcid":false,"given":"Xuemei","family":"Yang","sequence":"additional","affiliation":[{"name":"Hefei University of Technology, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-8960-8279","authenticated-orcid":false,"given":"Yongrui","family":"Li","sequence":"additional","affiliation":[{"name":"Hefei University of Technology, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6366-9834","authenticated-orcid":false,"given":"Yujiao","family":"Wu","sequence":"additional","affiliation":[{"name":"CSIRO, Hobart, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6596-8117","authenticated-orcid":false,"given":"Yaxiong","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Computer Science, Hefei University of Technology, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,7,22]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00175"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3689641"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.02161"},{"key":"e_1_3_2_5_2","first-page":"4171","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171\u20134186."},{"key":"e_1_3_2_6_2","unstructured":"Weixi Feng Jiachen Li Michael Saxon Tsu Jui Fu Wenhu Chen and William Yang Wang. 2024. TC-Bench: Benchmarking temporal compositionality in text-to-video and image-to-video generation. arXiv:2406.08656. Retrieved from https:\/\/arxiv.org\/abs\/2406.08656"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.02096"},{"key":"e_1_3_2_8_2","first-page":"66923","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","volume":"36","author":"Hao Yaru","year":"2023","unstructured":"Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. 2023. Optimizing prompts for text-to-image generation. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 36, 66923\u201366939."},{"key":"e_1_3_2_9_2","unstructured":"Yingqing He Tianyu Yang Yong Zhang Ying Shan and Qifeng Chen. 2022. Latent video diffusion models for high-fidelity long video generation. arXiv:2211.13221. Retrieved from https:\/\/arxiv.org\/abs\/2211.13221"},{"key":"e_1_3_2_10_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","volume":"30","author":"Heusel Martin","year":"2017","unstructured":"Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 30."},{"key":"e_1_3_2_11_2","unstructured":"Jonathan Ho William Chan Chitwan Saharia Jay Whang Ruiqi Gao Alexey Gritsenko Diederik P. Kingma Ben Poole Mohammad Norouzi David J. Fleet et al. 2022. Imagen video: High definition video generation with diffusion models. arXiv:2210.02303. Retrieved from https:\/\/arxiv.org\/abs\/2210.02303"},{"key":"e_1_3_2_12_2","first-page":"6840","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","volume":"33","author":"Ho Jonathan","year":"2020","unstructured":"Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 33, 6840\u20136851."},{"key":"e_1_3_2_13_2","first-page":"8633","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","volume":"35","author":"Ho Jonathan","year":"2022","unstructured":"Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. 2022. Video diffusion models. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 8633\u20138646."},{"key":"e_1_3_2_14_2","volume-title":"Proceedings of the 1st Workshop on Controllable Video Generation@ ICML24","author":"Hong Susung","year":"2024","unstructured":"Susung Hong, Junyoung Seo, Heeseong Shin, Sunghwan Hong, and Seungryong Kim. 2024. Large language models are frame-level directors for zero-shot text-to-video generation. In Proceedings of the 1st Workshop on Controllable Video Generation@ ICML24."},{"key":"e_1_3_2_15_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations","author":"Hong Wenyi","year":"2023","unstructured":"Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. 2023. CogVideo: Large-scale pretraining for text-to-video generation via transformers. In Proceedings of the 11th International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=rB6TpjAuSRy"},{"key":"e_1_3_2_16_2","unstructured":"Will Kay Joao Carreira Karen Simonyan Brian Zhang Chloe Hillier Sudheendra Vijayanarasimhan Fabio Viola Tim Green Trevor Back Paul Natsev et al. 2017. The kinetics human action video dataset. arXiv:1705.06950. Retrieved from https:\/\/arxiv.org\/abs\/1705.06950"},{"key":"e_1_3_2_17_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations","author":"Kynk\u00e4\u00e4nniemi Tuomas","year":"2023","unstructured":"Tuomas Kynk\u00e4\u00e4nniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. 2023. The role of ImageNet classes in Fr\u00e9chet inception distance. In Proceedings of the 11th International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=4oXTQ6m_ws8"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3638556"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52733.2024.00821"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.00218"},{"key":"e_1_3_2_22_2","first-page":"10209","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Luo Zhengxiong","year":"2023","unstructured":"Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. 2023. VideoFusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, 10209\u201310218."},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.00585"},{"key":"e_1_3_2_24_2","unstructured":"OpenAI. 2023. DALL-E3. Retrieved from https:\/\/cdn.openai.com\/papers\/DALL_E_3_System_Card.pdf"},{"key":"e_1_3_2_25_2","unstructured":"OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774. Retrieved from https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_2_26_2","volume-title":"Proceedings of the 12th International Conference on Learning Representations","author":"Podell Dustin","year":"2024","unstructured":"Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M\u00fcller, Joe Penna, and Robin Rombach. 2024. SDXL: Improving latent diffusion models for high-resolution image synthesis. In Proceedings of the 12th International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=di52zR8xgf"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.01460"},{"key":"e_1_3_2_28_2","first-page":"8748","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning. PMLR, 8748\u20138763."},{"key":"e_1_3_2_29_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","volume":"32","author":"Razavi Ali","year":"2019","unstructured":"Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. 2019. Generating diverse high-fidelity images with VQ-VAE-2. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 32."},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1410"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01042"},{"key":"e_1_3_2_32_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","volume":"29","author":"Salimans Tim","year":"2016","unstructured":"Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training GANs. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 29."},{"key":"e_1_3_2_33_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations","author":"Singer Uriel","year":"2023","unstructured":"Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. 2023. Make-A-Video: Text-to-video generation without text-video data. In Proceedings of the 11th International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=nJfylDvgzlq"},{"key":"e_1_3_2_34_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Song Jiaming","year":"2021","unstructured":"Jiaming Song, Chenlin Meng, and Stefano Ermon. 2021. Denoising diffusion implicit models. In Proceedings of the International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=St1giarCHLP"},{"key":"e_1_3_2_35_2","unstructured":"Khurram Soomro Amir Roshan Zamir and Mubarak Shah. 2012. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv:1705.06950. Retrieved from https:\/\/arxiv.org\/abs\/1705.06950"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3648368"},{"key":"e_1_3_2_37_2","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288. Retrieved from https:\/\/arxiv.org\/abs\/2307.09288"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.510"},{"key":"e_1_3_2_39_2","unstructured":"Thomas Unterthiner Sjoerd Van Steenkiste Karol Kurach Raphael Marinier Marcin Michalski and Sylvain Gelly. 2018. Towards accurate generative models of video: A new metric & challenges. arXiv:1812.01717. Retrieved from https:\/\/arxiv.org\/abs\/1812.01717"},{"key":"e_1_3_2_40_2","doi-asserted-by":"crossref","unstructured":"Quo Vadis Joao Carreira and Andrew Zisserman. 2017. Action recognition? A new model and the kinetics dataset. In CVPR 4724\u20134733.","DOI":"10.1109\/CVPR.2017.502"},{"key":"e_1_3_2_41_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 30."},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.02158"},{"key":"e_1_3_2_43_2","unstructured":"Ang Wang Baole Ai Bin Wen Chaojie Mao Chen-Wei Xie Di Chen Feiwu Yu Haiming Zhao Jianxiao Yang Jianyuan Zeng et al. 2025. Wan: Open and advanced large-scale video generative models. arXiv:2503.20314. Retrieved from https:\/\/arxiv.org\/abs\/2503.20314"},{"key":"e_1_3_2_44_2","unstructured":"Jiuniu Wang Hangjie Yuan Dayou Chen Yingya Zhang Xiang Wang and Shiwei Zhang. 2023. ModelScope text-to-video technical report. arXiv:2308.06571. Retrieved from https:\/\/arxiv.org\/abs\/2308.06571"},{"key":"e_1_3_2_45_2","unstructured":"Wenjing Wang Huan Yang Zixi Tuo Huiguo He Junchen Zhu Jianlong Fu and Jiaying Liu. 2023. VideoFactory: Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation. arXiv: 2305.10874. Retrieved from https:\/\/arxiv.org\/abs\/2305.10874"},{"key":"e_1_3_2_46_2","volume-title":"Proceedings of the 12th International Conference on Learning Representations","author":"Wang Yi","year":"2024","unstructured":"Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Yu Qiao. 2024. InternVid: A large-scale video-text dataset for multimodal understanding and generation. In Proceedings of the 12th International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=MLBdiWu4Fw"},{"key":"e_1_3_2_47_2","unstructured":"Chenfei Wu Lun Huang Qianxi Zhang Binyang Li Lei Ji Fan Yang Guillermo Sapiro and Nan Duan. 2021. GODIVA: Generating open-domain videos from natural descriptions. arXiv:2104.14806. Retrieved from https:\/\/arxiv.org\/abs\/2104.14806"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.00701"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.571"},{"key":"e_1_3_2_50_2","unstructured":"Wilson Yan Yunzhi Zhang Pieter Abbeel and Aravind Srinivas. 2021. VideoGPT: Video generation using VQ-VAE and transformers. arXiv:2104.10157. Retrieved from https:\/\/arxiv.org\/abs\/2104.10157"},{"key":"e_1_3_2_51_2","first-page":"67","volume-title":"Transactions of the Association for Computational Linguistics","volume":"2","author":"Young Peter","year":"2014","unstructured":"Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2 (2014), 67\u201378."},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3749844"},{"key":"e_1_3_2_53_2","unstructured":"Daquan Zhou Weimin Wang Hanshu Yan Weiwei Lv Yizhe Zhu and Jiashi Feng. 2022. MagicVideo: Efficient video generation with latent diffusion models. arXiv:2211.11018. Retrieved from https:\/\/arxiv.org\/abs\/2211.11018"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3581783.3612707"}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3820056","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,22]],"date-time":"2026-07-22T14:27:32Z","timestamp":1784730452000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3820056"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,22]]},"references-count":53,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2026,8,31]]}},"alternative-id":["10.1145\/3820056"],"URL":"https:\/\/doi.org\/10.1145\/3820056","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"value":"1551-6857","type":"print"},{"value":"1551-6865","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,22]]},"assertion":[{"value":"2025-12-07","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-04-29","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-07-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}