Chonkie
ActiveDescription
The lightweight ingestion library for fast, efficient and robust RAG pipelines. Supports multiple chunking strategies and embedding models to significantly improve retrieval-augmented generation results.
Key Features
- 11 chunking strategies: Token, Fast (SIMD 100+ GB/s), Semantic, Recursive, Code, Neural, Slumber, and more
- End-to-end pipeline API for fetching, chunking, refining, embedding, and shipping to vector DBs
- 45+ integrations: tokenizers, embedding providers, LLMs, refineries, porters, and vector databases
- Multilingual support for 56 languages out of the box
- Self-hosted REST API server with pipeline management stored in SQLite
- Lightweight at 505KB with async support for high-throughput applications
Use Cases
π‘ Preparing document chunks for RAG pipelines with semantic or recursive splitting
π‘ Building production document ingestion services with the REST API and pipeline management
π‘ Code-aware chunking for developer documentation and repository analysis
π‘ High-throughput text processing with async chunking and embedding pipelines
Categories
Quick Start
1. Install: pip install chonkie. 2. Use: from chonkie import RecursiveChunker; chunker = RecursiveChunker(); chunks = chunker('Your text here'). 3. For pipelines: from chonkie import Pipeline; pipe = Pipeline().chunk_with('semantic', chunk_size=512); doc = pipe.run(texts='...').