Guidance

Normal
GitHub Jupyter Notebook MIT

Description

Guidance is a programming framework for controlling LLM output, supporting structured generation, constrained decoding, and templated prompts to ensure outputs conform to predefined formats.

Key Features

  • Constrained decoding — Control LLM output format via regex and context-free grammars
  • Pythonic interface — Write LLM control logic with Python context managers and decorators
  • Selection function — select() constrains model output to predefined options
  • Offline grammar debugging — Validate constraint grammars locally with Mock model, no API calls
  • Custom Guidance functions — Create reusable LLM control functions with @guidance decorator
  • Multi-backend support — Supports Transformers, llama.cpp, OpenAI and other inference backends

Use Cases

💡 Ensure LLM output conforms to JSON or specific data structure formats
💡 Build structured data extraction pipelines requiring precise format control
💡 Guarantee correct function parameter formats in agent tool calls
💡 Reduce LLM output latency and cost through constrained decoding

Tags

structured-output constrained-decoding templating python llm

Categories

⚡ Agent Tools

Quick Start

pip install guidance

from guidance import system, user, assistant, gen
from guidance.models import Transformers

lm = Transformers("microsoft/Phi-4-mini-instruct")

with system():
    lm += "You are a helpful assistant"
with user():
    lm += "Hello. What is your name?"
with assistant():
    lm += gen(max_tokens=20)

print(lm)
Visit GitHub

Related Projects