mistral.rs

Active
GitHub Rust MIT

Description

Fast, flexible LLM inference engine built in Rust — supports multiple model architectures and quantization schemes for high-performance local LLM deployment.

Key Features

  • Zero-Config Model Loading — auto-detect architecture, quantization format, and chat template from any Hugging Face model
  • True Multimodality — text, vision, video, audio input, speech generation, image generation, and embeddings in one engine
  • Smart Quantization — UQFF prebuilt quantizations and ISQ with automatic best-format selection at each level
  • OpenAI + Anthropic Compatible Serving — single process exposes both /v1 and /v1/messages endpoints
  • Built-in Web UI — served at /ui with reasoning, code execution, plots, and inline file display; edit messages to branch conversations
  • Agentic Runtime — web search, local Python code execution with model feedback, session management, and custom tool hooks

Use Cases

💡 Run local LLM inference with high performance on CUDA, Metal, and CPU
💡 Deploy OpenAI/Anthropic-compatible API servers for existing applications
💡 Build agentic workflows with web search and code execution capabilities
💡 Quantize and optimize models for specific hardware with mistralrs tune
💡 Generate images, speech, and embeddings through a unified inference engine

Tags

rust llm tools framework

Categories

⚡ Agent Tools

Quick Start

Install via curl: curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.sh | sh. Then run: mistralrs run -m Qwen/Qwen3-4B for interactive chat, or mistralrs serve -m google/gemma-4-E4B-it to start an API server with the built-in web UI at localhost:1234/ui.
Visit GitHub

Related Projects