{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,13]],"date-time":"2026-10-13T17:54:04Z","timestamp":1791914044830,"version":"4.3.4"},"reference-count":25,"publisher":"PeerJ","license":[{"start":{"date-parts":[[2024,10,22]],"date-time":"2024-10-22T00:00:00Z","timestamp":1729555200000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Biomedical Sensors & Systems Lab"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"abstract":"<jats:p>\n                    Large language models (LLMs) have become transformative tools in areas like text generation, natural language processing, and conversational AI. However, their widespread use introduces security risks, such as jailbreak attacks, which exploit LLM\u2019s vulnerabilities to manipulate outputs or extract sensitive information. Malicious actors can use LLMs to spread misinformation, manipulate public opinion, and promote harmful ideologies, raising ethical concerns. Balancing safety and accuracy require carefully weighing potential risks against benefits. Prompt Guarding (Prompt-G) addresses these challenges by using vector databases and embedding techniques to assess the credibility of generated text, enabling real-time detection and filtering of malicious content. We collected and analyzed a dataset of Self Reminder attacks to identify and mitigate jailbreak attacks, ensuring that the LLM generates safe and accurate responses. In various attack scenarios, Prompt-G significantly reduced jailbreak success rates and effectively identified prompts that caused confusion or distraction in the LLM. Integrating our model with Llama 2 13B chat reduced the attack success rate (ASR) to 2.08%. The source code is available at:\n                    <jats:uri xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/doi.org\/10.5281\/zenodo.13501821\">https:\/\/doi.org\/10.5281\/zenodo.13501821<\/jats:uri>\n                    .\n                  <\/jats:p>","DOI":"10.7717\/peerj-cs.2374","type":"journal-article","created":{"date-parts":[[2024,10,22]],"date-time":"2024-10-22T05:10:58Z","timestamp":1729573858000},"page":"e2374","source":"Crossref","is-referenced-by-count":19,"title":["Mitigating adversarial manipulation in LLMs: a prompt-based approach to counter Jailbreak attacks (Prompt-G)"],"prefix":"10.7717","volume":"10","author":[{"given":"Bhagyajit","family":"Pingua","sequence":"first","affiliation":[{"name":"School of Computer Sciences, Odisha University of Technology and Research, Bhubaneswar, Odisha, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Deepak","family":"Murmu","sequence":"additional","affiliation":[{"name":"School of Computer Sciences, Odisha University of Technology and Research, Bhubaneswar, Odisha, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Meenakshi","family":"Kandpal","sequence":"additional","affiliation":[{"name":"School of Computer Sciences, Odisha University of Technology and Research, Bhubaneswar, Odisha, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2747-3919","authenticated-orcid":true,"given":"Jyotirmayee","family":"Rautaray","sequence":"additional","affiliation":[{"name":"School of Computer Sciences, Odisha University of Technology and Research, Bhubaneswar, Odisha, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pranati","family":"Mishra","sequence":"additional","affiliation":[{"name":"School of Computer Sciences, Odisha University of Technology and Research, Bhubaneswar, Odisha, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rabindra Kumar","family":"Barik","sequence":"additional","affiliation":[{"name":"School of Computer Applications, KIIT Deemed to be University, Bhubaneswar, Odisha, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6656-4333","authenticated-orcid":true,"given":"Manob Jyoti","family":"Saikia","sequence":"additional","affiliation":[{"name":"Electrical and Computer Engineering, The University of Memphis, Memphis, TN, United States"},{"name":"Biomedical Sensors & Systems Lab, The University of Memphis, Memphis, FL, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"4443","published-online":{"date-parts":[[2024,10,22]]},"reference":[{"key":"10.7717\/peerj-cs.2374\/ref-1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1904.03323","article-title":"Publicly available clinical bert embeddings","author":"Alsentzer","year":"2019"},{"key":"10.7717\/peerj-cs.2374\/ref-2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2312.01648","article-title":"Characterizing large language model geometry solves toxicity detection and generation","author":"Balestriero","year":"2023"},{"key":"10.7717\/peerj-cs.2374\/ref-3","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2402.05668","article-title":"Comprehensive assessment of jailbreak attacks against LLMs","author":"Chu","year":"2024"},{"key":"10.7717\/peerj-cs.2374\/ref-4","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2310.12773","article-title":"Safe rlhf: safe reinforcement learning from human feedback","author":"Dai","year":"2023"},{"key":"10.7717\/peerj-cs.2374\/ref-5","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2402.00888","article-title":"Security and privacy challenges of large language models: a survey","author":"Das","year":"2024"},{"key":"10.7717\/peerj-cs.2374\/ref-6","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2401.10862","article-title":"Pruning for protection: increasing jailbreak resistance in aligned LLMs without fine-tuning","author":"Hasan","year":"2024"},{"key":"10.7717\/peerj-cs.2374\/ref-7","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2311.05232","article-title":"A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions","author":"Huang","year":"2023"},{"key":"10.7717\/peerj-cs.2374\/ref-8","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2309.00614","article-title":"Baseline defenses for adversarial attacks against aligned language models","author":"Jain","year":"2023"},{"key":"10.7717\/peerj-cs.2374\/ref-9","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2402.03299","article-title":"Guard: role-playing to generate natural-language jailbreakings to test guideline adherence of large language models","author":"Jin","year":"2024"},{"key":"10.7717\/peerj-cs.2374\/ref-10","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2402.01763","article-title":"When large language models meet vector databases: a survey","author":"Jing","year":"2024"},{"key":"10.7717\/peerj-cs.2374\/ref-11","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2402.15180","article-title":"Break the breakout: reinventing lm defense against jailbreak attacks with self-refinement","author":"Kim","year":"2024"},{"key":"10.7717\/peerj-cs.2374\/ref-12","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2401.02038","article-title":"Understanding llms: a comprehensive overview from training to inference","author":"Liu","year":"2024"},{"key":"10.7717\/peerj-cs.2374\/ref-13","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2310.03684","article-title":"Smoothllm: defending large language models against jailbreaking attacks","author":"Robey","year":"2023"},{"key":"10.7717\/peerj-cs.2374\/ref-14","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2308.03825","article-title":"\u201cdo anything now\u201d: characterizing and evaluating in-the-wild jailbreak prompts on large language models","author":"Shen","year":"2023"},{"key":"10.7717\/peerj-cs.2374\/ref-15","doi-asserted-by":"publisher","first-page":"80716","DOI":"10.1109\/ACCESS.2020.2988796","article-title":"Unsupervised k-means clustering algorithm","volume":"8","author":"Sinaga","year":"2020","journal-title":"IEEE Access"},{"key":"10.7717\/peerj-cs.2374\/ref-16","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2311.11855","article-title":"Evil geniuses: delving into the safety of llm-based agents","author":"Tian","year":"2023"},{"key":"10.7717\/peerj-cs.2374\/ref-17","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2307.09288","article-title":"Llama 2: open foundation and fine-tuned chat models","author":"Touvron","year":"2023"},{"key":"10.7717\/peerj-cs.2374\/ref-18","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2307.02483","article-title":"Jailbroken: how does LLM safety training fail?","volume":"36","author":"Wei","year":"2024","journal-title":"Advances in Neural Information Processing Systems"},{"key":"10.7717\/peerj-cs.2374\/ref-19","doi-asserted-by":"publisher","first-page":"1486","DOI":"10.1038\/s42256-023-00765-8","article-title":"Defending chatgpt against jailbreak attack via self-reminder","volume":"5","author":"Wu","year":"2023","journal-title":"Nature Machine Intelligence"},{"key":"10.7717\/peerj-cs.2374\/ref-20","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2309.13256","article-title":"Defending pre-trained language models as few-shot learners against backdoor attacks","volume":"36","author":"Xi","year":"2024","journal-title":"Advances in Neural Information Processing Systems"},{"key":"10.7717\/peerj-cs.2374\/ref-21","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2404.08517","article-title":"Online safety analysis for llms: a benchmark, an assessment, and a path forward","author":"Xie","year":"2024"},{"key":"10.7717\/peerj-cs.2374\/ref-22","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2402.13457","article-title":"LLM jailbreak attack versus defense techniques\u2013a comprehensive study","author":"Xu","year":"2024"},{"key":"10.7717\/peerj-cs.2374\/ref-23","doi-asserted-by":"publisher","first-page":"19377","DOI":"10.1609\/aaai.v38i17.29908","article-title":"Exploring post-training quantization in llms from comprehensive study to low rank compensation","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","volume":"38","author":"Yao","year":"2024"},{"key":"10.7717\/peerj-cs.2374\/ref-24","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2311.09096","article-title":"Defending large language models against jailbreaking attacks through goal prioritization","author":"Zhang","year":"2023"},{"key":"10.7717\/peerj-cs.2374\/ref-25","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2402.13148","article-title":"Defending jailbreak prompts via in-context adversarial game","author":"Zhou","year":"2024"}],"container-title":["PeerJ Computer Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/peerj.com\/articles\/cs-2374.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-2374.xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-2374.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-2374.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,22]],"date-time":"2024-10-22T05:11:09Z","timestamp":1729573869000},"score":1,"resource":{"primary":{"URL":"https:\/\/peerj.com\/articles\/cs-2374"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,22]]},"references-count":25,"alternative-id":["10.7717\/peerj-cs.2374"],"URL":"https:\/\/doi.org\/10.7717\/peerj-cs.2374","archive":["CLOCKSS","LOCKSS","Portico"],"relation":{},"ISSN":["2376-5992"],"issn-type":[{"value":"2376-5992","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,22]]},"article-number":"e2374"}}