{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T12:28:58Z","timestamp":1780316938801,"version":"3.54.1"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,8,26]],"date-time":"2024-08-26T00:00:00Z","timestamp":1724630400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,8,26]],"date-time":"2024-08-26T00:00:00Z","timestamp":1724630400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62202064"],"award-info":[{"award-number":["62202064"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Cybersecurity"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Text-to-image (TTI) models provide huge innovation ability for many industries, while the content security triggered by them has also attracted wide attention. Considerable research has focused on content security threats of large language models (LLMs), yet comprehensive studies on the content security of TTI models are notably scarce. This paper introduces a systematic tool, named EvilPromptFuzzer, designed to fuzz evil prompts in TTI models. For 15 kinds of fine-grained risks, EvilPromptFuzzer employs the strong knowledge-mining ability of LLMs to construct seed banks, in which the seeds cover various types of characters, interrelations, actions, objects, expressions, body parts, locations, surroundings, etc. Subsequently, these seeds are fed into the LLMs to build scene-diverse prompts, which can weaken the semantic sensitivity related to the fine-grained risks. Hence, the prompts can bypass the content audit mechanism of the TTI model, and ultimately help to generate images with inappropriate content. For the risks of violence, horrible, disgusting, animal cruelty, religious bias, political symbol, and extremism, the efficiency of EvilPromptFuzzer for generating inappropriate images based on DALL.E 3 are greater than 30%, namely, more than 30 generated images are malicious among 100 prompts. Specifically, the efficiency of horrible, disgusting, political symbols, and extremism up to 58%, 64%, 71%, and 50%, respectively. Additionally, we analyzed the vulnerability of existing popular content audit platforms, including Amazon, Google, Azure, and Baidu. Even the most effective Google SafeSearch cloud platform identifies only 33.85% of malicious images across three distinct categories.<\/jats:p>","DOI":"10.1186\/s42400-024-00279-9","type":"journal-article","created":{"date-parts":[[2024,8,26]],"date-time":"2024-08-26T04:02:31Z","timestamp":1724644951000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["EvilPromptFuzzer: generating inappropriate content based on text-to-image models"],"prefix":"10.1186","volume":"7","author":[{"given":"Juntao","family":"He","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Haoran","family":"Dai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Runqi","family":"Sui","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-1866-5828","authenticated-orcid":false,"given":"Xuejing","family":"Yuan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dun","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hao","family":"Feng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xinyue","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenchuan","family":"Yang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Baojiang","family":"Cui","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kedan","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,8,26]]},"reference":[{"key":"279_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1561\/3300000041","volume":"6","author":"C Barrett","year":"2023","unstructured":"Barrett C, Boyd B, Bursztein E, Carlini N, Chen B, Choi J, Chowdhury AR, Christodorescu M, Datta A, Feizi S et al (2023) Identifying and mitigating the security risks of generative AI. Found Trends Privacy Security 6:1\u201352","journal-title":"Found Trends Privacy Security"},{"key":"279_CR2","unstructured":"Bellagente M, Brack M, Teufel H, Friedrich F, Deiseroth B, Eichenberg C, Dai AM, Baldock R, Nanda S, Oostermeijer K et al (2023) Multifusion: fusing pre-trained models for multi-lingual, multi-modal image generation. Adv Neural Inf Process Syst 36"},{"key":"279_CR3","doi-asserted-by":"crossref","unstructured":"Bird C, Ungless E, Kasirzadeh A (2023) Typology of risks of generative text-to-image models. In: Proceedings of the 2023 AAAI\/ACM conference on AI, ethics, and society, pp 396\u2013410","DOI":"10.1145\/3600211.3604722"},{"key":"279_CR4","doi-asserted-by":"crossref","unstructured":"Cho J, Zala A, Bansal M (2023) Dall-eval: probing the reasoning skills and social biases of text-to-image generation models. In: Proceedings of the IEEE\/CVF international conference on computer vision, pp 3043\u20133054","DOI":"10.1109\/ICCV51070.2023.00283"},{"key":"279_CR5","unstructured":"Friedrich F, H\u00e4mmerl K, Schramowski P, Libovicky J, Kersting K, Fraser A (2024) Multilingual text-to-image generation magnifies gender stereotypes and prompt engineering may not help you. arXiv preprint arXiv:2401.16092"},{"key":"279_CR6","unstructured":"Gal R, Alaluf Y, Atzmon Y, Patashnik O, Bermano AH, Chechik G, Cohen-or D (2022) An image is worth one word: personalizing text-to-image generation using textual inversion. In: The eleventh international conference on learning representationse"},{"issue":"1","key":"279_CR7","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1109\/TPAMI.2022.3152247","volume":"45","author":"K Han","year":"2022","unstructured":"Han K, Wang Y, Chen H, Chen X, Guo J, Liu Z, Tang Y, Xiao A, Xu C, Xu Y et al (2022) A survey on vision transformer. IEEE Trans Pattern Anal Mach Intell 45(1):87\u2013110","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"279_CR8","unstructured":"Hinz M (2023) Risks the metaverse poses for children and adolescents: an exploratory content analysis. B.S. thesis, University of Twente"},{"key":"279_CR9","doi-asserted-by":"crossref","unstructured":"Hussain A, Alipour MA (2021) DIAR: removing uninteresting bytes from seeds in software fuzzing. arxiv: 2112.13297","DOI":"10.1109\/ICSTW55395.2022.00058"},{"key":"279_CR10","unstructured":"Hutchinson B, Baldridge J, Prabhakaran V (2022) Underspecification in scene description-to-depiction tasks. In: Proceedings of the 2nd conference of the asia-pacific chapter of the association for computational linguistics and the 12th international joint conference on natural language processing, pp 1172\u20131184"},{"key":"279_CR11","doi-asserted-by":"crossref","unstructured":"Jawahar G, Sagot B, Seddah D (2019) What does bert learn about the structure of language? In: ACL 2019-57th annual meeting of the association for computational linguistics","DOI":"10.18653\/v1\/P19-1356"},{"key":"279_CR12","doi-asserted-by":"crossref","unstructured":"Kieslich K, Diakopoulos N, Helberger N (2023) Anticipating impacts: Using large-scale scenario writing to explore diverse implications of generative AI in the news environment. arXiv preprint arXiv:2310.06361","DOI":"10.1007\/s43681-024-00497-4"},{"key":"279_CR13","unstructured":"Korbak T, Shi K, Chen A, Bhalerao RV, Buckley C, Phang J, Bowman SR, Perez E (2023) Pretraining language models with human preferences. In: International conference on machine learning, pp 17506\u201317533. PMLR"},{"key":"279_CR14","doi-asserted-by":"crossref","unstructured":"Liu Y, Deng G, Xu Z, Li Y, Zheng Y, Zhang Y, Zhao L, Zhang T, Liu Y (2023) Jailbreaking chatgpt via prompt engineering: an empirical study. arXiv preprint arXiv:2305.13860","DOI":"10.1145\/3663530.3665021"},{"key":"279_CR15","doi-asserted-by":"crossref","unstructured":"Ma Y, Xu G, Sun X, Yan M, Zhang J, Ji R (2022) X-clip: End-to-end multi-grained contrastive learning for video-text retrieval. In: Proceedings of the 30th ACM international conference on multimedia, pp 638\u2013647","DOI":"10.1145\/3503161.3547910"},{"key":"279_CR16","unstructured":"Meng C, He Y, Song Y, Song J, Wu J, Zhu J-Y, Ermon S (2021) Sdedit: guided image synthesis and editing with stochastic differential equations. In: International conference on learning representations"},{"key":"279_CR17","doi-asserted-by":"crossref","unstructured":"Qi X, Huang K, Panda A, Henderson P, Wang M, Mittal P (2024) Visual adversarial examples jailbreak aligned large language models. In: Proceedings of the AAAI conference on artificial intelligence, vol 38, pp 21527\u201321536","DOI":"10.1609\/aaai.v38i19.30150"},{"key":"279_CR18","doi-asserted-by":"crossref","unstructured":"Qu Y, Shen X, He X, Backes M, Zannettou S, Zhang Y (2023) Unsafe diffusion: On the generation of unsafe images and hateful memes from text-to-image models. In: Proceedings of the 2023 ACM SIGSAC conference on computer and communications security, pp 3403\u20133417","DOI":"10.1145\/3576915.3616679"},{"key":"279_CR19","unstructured":"Rando J, Paleka D, Lindner D, Heim L, Tramer F (2022) Red-teaming the stable diffusion safety filter. In: NeurIPS ML safety workshop"},{"key":"279_CR20","doi-asserted-by":"crossref","unstructured":"Rassin R, Ravfogel S, Goldberg Y (2022) Dalle-2 is seeing double: flaws in word-to-concept mapping in text2image models. In: Proceedings of the Fifth BlackboxNLP workshop on analyzing and interpreting neural networks for NLP, pp 335\u2013345","DOI":"10.18653\/v1\/2022.blackboxnlp-1.28"},{"key":"279_CR21","doi-asserted-by":"crossref","unstructured":"Ruiz N, Li Y, Jampani V, Pritch Y, Rubinstein M, Aberman K (2023) Dreambooth: fine tuning text-to-image diffusion models for subject-driven generation. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp 22500\u201322510","DOI":"10.1109\/CVPR52729.2023.02155"},{"key":"279_CR22","doi-asserted-by":"crossref","unstructured":"Saha BK (2024) Generative artificial intelligence for industry: opportunities, challenges, and impact. In: 2024 international conference on artificial intelligence in information and communication (ICAIIC), pp 081\u2013086. IEEE","DOI":"10.1109\/ICAIIC60209.2024.10463245"},{"key":"279_CR23","doi-asserted-by":"crossref","unstructured":"Schramowski P, Tauchmann C, Kersting K (2022) Can machines help us answering question 16 in datasheets, and in turn reflecting on inappropriate content? In: Proceedings of the 2022 ACM conference on fairness, accountability, and transparency, pp 1350\u20131361","DOI":"10.1145\/3531146.3533192"},{"key":"279_CR24","doi-asserted-by":"crossref","unstructured":"Sha Z, Li Z, Yu N, Zhang Y (2023) De-fake: detection and attribution of fake images generated by text-to-image generation models. In: Proceedings of the 2023 ACM SIGSAC conference on computer and communications security, pp 3418\u20133432","DOI":"10.1145\/3576915.3616588"},{"key":"279_CR25","doi-asserted-by":"crossref","unstructured":"Struppek L, Hintersdorf D, Kersting K (2023) Rickrolling the artist: injecting backdoors into text encoders for text-to-image synthesis. In: Proceedings of the IEEE\/CVF international conference on computer vision, pp 4584\u20134596","DOI":"10.1109\/ICCV51070.2023.00423"},{"key":"279_CR26","unstructured":"Touvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, Bashlykov N, Batra S, Bhargava P, Bhosale S et al (2023) Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288"},{"key":"279_CR27","doi-asserted-by":"crossref","unstructured":"Yang Y, Hui B, Yuan H, Gong N, Cao Y (2024) Sneakyprompt: jailbreaking text-to-image generative models. In: 2024 IEEE symposium on security and privacy (SP), pp 1\u201316","DOI":"10.1109\/SP54263.2024.00123"},{"key":"279_CR28","doi-asserted-by":"crossref","unstructured":"Yao D, Zhang J, Harris IG, Carlsson M (2024) Fuzzllm: A novel and universal fuzzing framework for proactively discovering jailbreak vulnerabilities in large language models. In: ICASSP 2024-2024 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 4485\u20134489. IEEE","DOI":"10.1109\/ICASSP48485.2024.10448041"},{"key":"279_CR29","doi-asserted-by":"crossref","unstructured":"Ye F, Liu G, Wu X, Wu L (2024) Altdiffusion: a multilingual text-to-image diffusion model. In: Proceedings of the AAAI conference on artificial intelligence, vol 38, pp 6648\u20136656","DOI":"10.1609\/aaai.v38i7.28487"},{"key":"279_CR30","unstructured":"Yu J, Lin X, Xing X (2023) Gptfuzzer: red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253"},{"key":"279_CR31","doi-asserted-by":"crossref","unstructured":"Zhang L, Rao A, Agrawala M (2023) Adding conditional control to text-to-image diffusion models. In: Proceedings of the IEEE\/CVF international conference on computer vision, pp. 3836\u20133847","DOI":"10.1109\/ICCV51070.2023.00355"},{"issue":"2","key":"279_CR32","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3639372","volume":"15","author":"H Zhao","year":"2024","unstructured":"Zhao H, Chen H, Yang F, Liu N, Deng H, Cai H, Wang S, Yin D, Du M (2024) Explainability for large language models: a survey. ACM Trans Intell Syst Technol 15(2):1\u201338","journal-title":"ACM Trans Intell Syst Technol"}],"container-title":["Cybersecurity"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s42400-024-00279-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s42400-024-00279-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s42400-024-00279-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,27]],"date-time":"2024-11-27T06:18:19Z","timestamp":1732688299000},"score":1,"resource":{"primary":{"URL":"https:\/\/cybersecurity.springeropen.com\/articles\/10.1186\/s42400-024-00279-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,26]]},"references-count":32,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["279"],"URL":"https:\/\/doi.org\/10.1186\/s42400-024-00279-9","relation":{},"ISSN":["2523-3246"],"issn-type":[{"value":"2523-3246","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,26]]},"assertion":[{"value":"10 April 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 June 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 August 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"We obtained the IRB Exempt certificates from our institute.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"The authors declare that we have no conflicts of interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interest"}}],"article-number":"70"}}