LLM Security
StaleDescription
Novel approaches to breaking app-integrated LLMs — a collection of research and tools covering prompt injection, indirect prompt injection, and other LLM attack vectors.
Key Features
- Demonstrates indirect prompt injection attacks on application-integrated LLMs like Bing Chat
- Proof-of-concept for remote control of compromised LLMs via command-and-control servers
- Persistent cross-session compromise through memory poisoning of agent key-value stores
- Spread injection attacks via email processing pipelines to compromise other LLMs
- Code completion engine attacks through context window manipulation in IDEs
- Research paper (ArXiv 2302.12173) with reproducible demos using GPT-3, GPT-4, and LangChain
Use Cases
💡 Security research and red-teaming to evaluate LLM application robustness against injection attacks
💡 Understanding attack vectors before deploying LLMs in production environments
💡 Training AI safety teams on prompt injection defense strategies and threat modeling
💡 Auditing code completion tools and AI assistants for vulnerability to context manipulation
💡 Building defensive frameworks informed by real-world attack demonstrations
Strengths & Limitations
✅ Strengths
- • High community interest (2.1k stars)
- • Permissive open-source license (MIT)
- • Established track record (3 years in production)
- • Responsive to issues, low backlog
⚠️ Limitations
- • No updates in over 12 months
Categories
Quick Start
Clone the repository and set your OpenAI API key. Run demos with `python scenarios/main.py` for GPT-4 scenarios, or try the GPT-3 + LangChain scenarios in `scenarios/gpt3langchain`. Code completion attacks require an IDE with LLM autocompletion support.