LanceDB

Active
GitHub Rust Apache-2.0

Description

An open-source embedded retrieval library for multimodal AI with zero server configuration, using the Lance columnar format for efficient vector search and filtering, ideal for agent memory and RAG applications.

Key Features

  • Millisecond vector search with state-of-the-art indexing for billions of vectors
  • Comprehensive search: vector similarity, full-text search, and SQL queries in one platform
  • Multimodal support — store and query text, images, videos, point clouds, and more
  • Zero-copy, automatic versioning — manage data versions without extra infrastructure
  • GPU-accelerated vector index building for dramatically faster large-scale data processing
  • Rich ecosystem: LangChain, LlamaIndex, Apache Arrow, Pandas, DuckDB integrations

Use Cases

💡 Build long-term memory stores for AI agents with efficient semantic retrieval and context recall
💡 Create RAG (Retrieval-Augmented Generation) applications for multimodal knowledge-base Q&A
💡 Build recommendation systems using vector similarity for personalized content matching
💡 Process large-scale multimodal datasets with cross-modal search and analysis capabilities
💡 Embed as a database in Python/TypeScript/Rust apps with zero server deployment

Strengths & Limitations

Strengths

  • Actively maintained, recent updates
  • High community interest (11.2k stars)
  • Permissive open-source license (Apache-2.0)
  • Established track record (3 years in production)

⚠️ Limitations

  • High issue backlog (618 open issues)

Quick Start

Install with pip install lancedb. Connect: import lancedb; db = lancedb.connect('~/.lancedb'). Create table: table = db.create_table('my_table', data). Search: results = table.search(query).limit(10).to_pandas(). Quickstart at docs.lancedb.com/quickstart.

Related Projects

pgvector

22.6k · C
Active A+

Open-source vector similarity search extension for PostgreSQL, enabling native vector storage and ANN retrieval in relational databases, a foundational component for building agent memory and RAG systems.

vector-databasepostgresqlsimilarity-search +2
  • · PostgreSQL native extension — Store and query vectors directly in relational database, no extra infrastructure
  • · Multiple distance metrics — L2, cosine distance, inner product, L1, Hamming, and Jaccard distance
  • · Exact and approximate search — HNSW and IVFFlat indexes for ANN approximate nearest neighbor search

HelixDB

5.7k · Rust
Active A+

An open-source graph-vector database built from scratch in Rust, combining graph database and vector retrieval capabilities to provide AI agents with unified storage for both knowledge graphs and semantic search.

graph-databasevector-databaserag +2
  • · Graph-vector database combining graph traversal and vector similarity search in one system
  • · Rust-based engine for high performance with low resource consumption
  • · TypeScript and Rust SDKs with declarative query DSL for building queries

Embedchain

63.4k · Python
Active A+

Embedchain is a universal memory layer for AI agents, enabling quick integration of diverse data sources into LLMs for context-aware AI applications.

memoryragembeddings +2
  • · Universal memory layer — quickly integrates diverse data sources (web, PDFs, YouTube, Notion) into LLMs for context
  • · Vectorized storage — auto-chunks, embeds, and stores data in vector databases with semantic retrieval support
  • · Multi-LLM backend support — compatible with OpenAI, Cohere, Ollama and other LLM and embedding models

HyperDB

1.4k · Python
Stale C

A hyper-fast local vector database for use with LLM Agents, providing lightweight vector storage and similarity search capabilities for embedding as instant memory and knowledge retrieval components in agent applications.

vector-databasepythonagent +2
  • · Hyper-fast local vector database designed specifically for LLM Agents
  • · C++ high-performance backend with hardware-accelerated operations via MKL BLAS
  • · Simple unified interface compatible with all major LLM agent frameworks