PromptTools
StaleDescription
PromptTools provides open-source tools for prompt testing and experimentation, supporting multiple LLMs (OpenAI, LLaMA) and vector databases (Chroma, Weaviate, LanceDB) to help developers systematically evaluate and optimize RAG systems.
Key Features
- Open-source prompt testing and experimentation across multiple LLM providers
- Supports OpenAI, Anthropic, LLaMA, Mistral, Gemini, HuggingFace, and more
- Vector database evaluation: Chroma, Weaviate, Qdrant, LanceDB, Pinecone
- Code-first experiments with Jupyter notebook and Streamlit playground interfaces
- Export results to CSV, JSON, or MongoDB for persistence and analysis
- Self-hostable with no data forwarded to third-party servers
Use Cases
π‘ Compare prompt performance across different LLM models side by side
π‘ Evaluate retrieval accuracy of vector database configurations
π‘ A/B test prompt templates with temperature and parameter sweeps
π‘ Benchmark RAG pipeline components for optimal system design
π‘ Prototype and validate LLM applications before production deployment
Categories
Quick Start
```bash
pip install prompttools
```
```python
from prompttools.experiment import OpenAIChatExperiment
messages = [[{"role": "user", "content": "Tell me a joke."}]]
models = ["gpt-3.5-turbo", "gpt-4"]
experiment = OpenAIChatExperiment(models, messages, temperature=[0.0])
experiment.run()
experiment.visualize()
```
Playground: `streamlit run prompttools/playground/playground.py`