HomeProjectsArticlesPK BoardAbout
Explore Projects

Pathway

Active
GitHub Python NOASSERTION

Description

Pathway is a Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG applications.

Key Features

  • Rust-powered engine — Scalable Rust engine based on Differential Dataflow with multithreading, multiprocessing, and distributed computation
  • Unified batch & streaming — Same code works for local dev, batch jobs, stream replays, and live data processing
  • Rich connectors — Built-in connectors for Kafka, GDrive, PostgreSQL, SharePoint; Airbyte connector covers 300+ data sources
  • LLM & RAG toolchain — LLM wrappers, parsers, embedders, splitters, and in-memory real-time Vector Index with LlamaIndex/LangChain integration
  • Stateful transformations — Joins, windowing, sorting with Rust-accelerated implementations for high performance
  • Persistence & consistency — Computation state can be persisted for crash recovery; at-least-once (community) and exactly-once (enterprise) consistency

Use Cases

💡 Build real-time ETL pipelines to continuously process and transform data streams from message queues like Kafka
💡 Set up private RAG applications with Ollama and Mistral AI for document-based real-time Q&A systems
💡 Convert unstructured data to structured SQL queries on-the-fly for dynamic data analysis
💡 Build adaptive RAG systems that automatically select retrieval strategies based on query complexity
💡 Implement multimodal RAG combining text and image processing with GPT-4o for mixed-document analysis

Strengths & Limitations

✅ Strengths

  • • Actively maintained, recent updates
  • • High community interest (62.5k stars)
  • • Established track record (3 years in production)
  • • Responsive to issues, low backlog

⚠️ Limitations

  • • No clear open-source license

Tags

etl streaming rag real-time python

Categories

📚 RAG Tools

Quick Start

pip install -U pathway

import pathway as pw

# Read data from a CSV file
table = pw.io.csv.read("data/input.csv")

# Transform the data
result = table.select(col1=pw.this.col1, col2=pw.this.col2 * 2)

# Write results to output
pw.io.jsonlines.write(result, "output.jsonl")

# Start the real-time processing pipeline
pw.run()
Visit GitHub Visit Website

Related Projects

RAGatouille

4.0k · Python
Stale C

Easily use and train state of the art late-interaction retrieval methods (ColBERT) in any RAG pipeline. Designed for modularity and ease-of-use, backed by research.

ragpythonembedding +1
  • · ColBERT late-interaction retrieval: wraps advanced ColBERT models into simple APIs, enabling late-interaction retrieval methods in RAG pipelines without deep IR research knowledge
  • · End-to-end training and fine-tuning: built-in RAGTrainer and TrainingDataProcessor with automatic deduplication, positive/negative pairing and hard negative mining, supporting training from scratch or fine-tuning pretrained models
  • · Modular composable architecture: DataProcessor, NegativeMiner and other components can be used independently, with support for custom NegativeMiner integration into training pipelines
View on GitHub

MemAgent

1.1k · Python
Stale B

A MemAgent framework that can extrapolate to 3.5M context tokens, along with a training framework for RL training of any agent workflow.

memoryagentrag +2
  • · Ultra-Long Context Processing: Extrapolate from 8K training context to 3.5M tokens with performance loss under 5%
  • · Reinforcement Learning Driven: Trained with RLVR (Reinforcement Learning from Verifiable Rewards), extends DAPO algorithm for end-to-end multi-turn conversation optimization
  • · Linear Time Complexity: Breaks through computational bottlenecks in long-text processing with linear resource scaling
View on GitHub

LightRAG

38.9k · Python
Active A+

LightRAG is a simple and fast Retrieval-Augmented Generation framework using graph-enhanced retrieval, published at EMNLP 2025.

raggraphretrieval +2
  • · Graph-enhanced retrieval — Dual-level retrieval (local/global) using knowledge graph entities and relationships, more precise than vector search
  • · Four text chunking strategies — Fixed, Recursive, Vector, and Paragraph chunking adapted to different document types
  • · Multi-backend storage — Neo4j, PostgreSQL, MongoDB, OpenSearch, JSON KV Store and other storage backends supported
View full details → View on GitHub