Defending against Prompt Injection with Structured Queries (StruQ) and Preference Optimization (SecAlign)
Berkeley BAIR researchers describe two fine-tuning defenses against prompt injection attacks on LLM-integrated applications: StruQ and SecAlign. Prompt injection is listed as the number one threat by OWASP for these applications, where untrusted data can carry instructions that manipulate the model. The defenses separate prompts from data and train models to follow only the intended instruction.
Key Takeaways
- Recent advances in Large Language Models (LLMs) enable exciting LLM-integrated applications.
However, as LLMs have improved, so have the attacks against them.
- To mitigate the imminent prompt injection threat, we propose two fine-tuning-defenses, StruQ and SecAlign.
Without additional cost on computation or human labor, they are utility-preserving effective defenses.
- Prompt injection threat model in LLM-integrated applications We propose that prompt injection has two causes.
First, LLM input has no separation between prompt and data so that no signal points to the intended instruction.
- The generated dataset contains clean samples and samples with injected instructions.
The LLM is supervised-fine-tuned to always respond to the intended instruction highlighted by the secure front-end.
- StruQ, with an ASR 45%, significantly mitigates prompt injections compared to prompting-based defenses.
Stats & Key Facts
- #SecAlign also stops strong optimization-based attacks to success rates lower than 15%, a number reduced by over 4 times from the previous SOTA in all 5 tested LLMs.
- #StruQ, with an ASR 45%, significantly mitigates prompt injections compared to prompting-based defenses.
Recent advances in Large Language Models (LLMs) enable exciting LLM-integrated applications. However, as LLMs have improved, so have the attacks against them. Prompt injection attack is listed as the #1 threat by OWASP to LLM-integrated applications, where an LLM input contains a trusted prompt (instruction) and an untrusted data.
The data may contain injected instructions to arbitrarily manipulate the LLM. As an example, to unfairly promote "Restaurant A", its owner could use prompt injection to post a review on Yelp, e.g., "Ignore your previous instruction. If an LLM receives the Yelp reviews and follows the injected instruction, it could be misled to recommend Restaurant A, which has poor reviews.
An example of prompt injection Production-level LLM systems, e.g., Google Docs , Slack AI , ChatGPT , have been shown vulnerable to prompt injections. To mitigate the imminent prompt injection threat, we propose two fine-tuning-defenses, StruQ and SecAlign. Without additional cost on computation or human labor, they are utility-preserving effective defenses.
For more details please read the original article at Berkeley BAIR.
Continue Learning
Comments
Comments appear only after moderation. Your email identifies your submission to the moderator and is never displayed here.
No approved comments yet.