Text Embeddings Inference
ActiveDescription
A blazing fast inference solution for text embeddings models built in Rust, serving as core infrastructure for building RAG systems and vector retrieval pipelines with high throughput and low latency.
Key Features
- Blazing fast Rust-based inference with no model graph compilation step
- Supports 15+ model families including Qwen3, GTE, BERT, JinaBERT, ModernBERT
- Token-based dynamic batching with Flash Attention and cuBLASLt optimization
- Production-ready with distributed tracing (OpenTelemetry) and Prometheus metrics
- Docker images with fast boot times — true serverless deployment ready
- Safetensors and ONNX weight loading with Metal support on Mac
Use Cases
💡 Building high-throughput embedding services for RAG pipelines
💡 Serving vector retrieval systems at scale with low latency
💡 Deploying reranking and sequence classification models in production
💡 Running embedding inference on both GPU and Apple Silicon locally
💡 Integrating embedding generation into real-time search applications
Tags
Categories
Quick Start
```bash
docker run --gpus all -p 8080:80 \
-v $PWD/data:/data \
ghcr.io/huggingface/text-embeddings-inference:latest \
--model-id Qwen/Qwen3-Embedding-0.6B
curl http://localhost:8080/embed \
-X POST \
-H 'Content-Type: application/json' \
-d '{"inputs": "What is deep learning?"}'
```