W&B Weave is an open-source observability, evaluation, and monitoring toolkit by Weights & Biases for developing reliable LLM and generative AI applications. It provides full tracing of LLM calls, systematic evaluation frameworks, production monitoring with guardrails, and deep integration with the broader W&B ecosystem for end-to-end AI development.

Weave addresses the fundamental challenges of LLM application development: non-determinism, subjective output quality, and sensitivity to prompt changes. It provides visibility into every LLM call, versions all artifacts (prompts, datasets, models, configs), and enables systematic experimentation and comparison.

Core capabilities:

• Tracing – Full visibility into every LLM call, input, output, latency, and token cost
• Evaluation – Systematic benchmarking with custom scorers, datasets, and aggregated metrics
• Monitoring – Production guardrails for toxicity, hallucination, and quality issues on live traffic
• Versioning – Automatic versioning of prompts, datasets, models, and configurations
• Feedback – Human annotation collection via UI or API
• Leaderboards – Color-coded comparison matrices for ranking models and prompts

pip install weave

import weave
 
# Initialize a Weave project
weave.init("my-llm-project")

The @weave.op() decorator is the foundation of Weave. It turns any function into a traceable, versioned operation that logs inputs, outputs, and metadata automatically.

import weave
import openai
 
weave.init("my-app")
client = openai.OpenAI()
 
@weave.op()
def generate_answer(question: str) -> str:
    "Generate an answer using an LLM. Automatically traced by Weave."
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": question}
        ]
    )
    return response.choices[0].message.content
 
@weave.op()
def check_relevance(question: str, answer: str) -> dict:
    "Score whether the answer is relevant to the question."
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "user",
             "content": f"Is this answer relevant? "
                        f"Question: {question} Answer: {answer} "
                        f"Reply yes or no."}
        ]
    )
    is_relevant = "yes" in response.choices[0].message.content.lower()
    return {"relevant": is_relevant}
 
# Every call is traced with full I/O logging
answer = generate_answer("What is retrieval-augmented generation?")
relevance = check_relevance("What is RAG?", answer)

Weaves evaluation system benchmarks LLM applications against datasets using custom scorers:

import weave
from weave import Evaluation, Model
import asyncio
 
weave.init("eval-project")
 
 
# Define a Model with a predict method
class QAModel(Model):
    model_name: str
    temperature: float = 0.0
 
    @weave.op()
    def predict(self, question: str) -> dict:
        response = client.chat.completions.create(
            model=self.model_name,
            temperature=self.temperature,
            messages=[{"role": "user", "content": question}]
        )
        return {"answer": response.choices[0].message.content}
 
 
# Define scorers
@weave.op()
def exact_match(expected: str, output: dict) -> dict:
    return {"match": expected.lower() == output["answer"].lower()}
 
@weave.op()
def length_check(output: dict) -> dict:
    return {"reasonable_length": 10 < len(output["answer"]) < 2000}
 
 
# Create evaluation dataset
dataset = [
    {"question": "What is the capital of France?", "expected": "Paris"},
    {"question": "What is 2 + 2?", "expected": "4"},
    {"question": "Who wrote Hamlet?", "expected": "Shakespeare"},
]
 
# Run evaluation
model = QAModel(model_name="gpt-4o-mini")
evaluation = Evaluation(dataset=dataset, scorers=[exact_match, length_check])
asyncio.run(evaluation.evaluate(model))
# Results appear in Weave UI with aggregated metrics

Weave Leaderboards provide color-coded comparison matrices for ranking models and prompts:

1. Run evaluations across multiple model variants
2. Select evaluation runs in the UI and click Compare
3. View performance matrices with baseline highlighting
4. Identify regressions and improvements at a glance

This is particularly valuable for systematic prompt optimization and model selection.

Weave includes production-ready scorers for common quality checks:

• Hallucination detection – Checks outputs against source context
• Toxicity/moderation – Flags harmful or inappropriate content
• Context precision – Measures retrieval relevance in RAG pipelines
• Token cost tracking – Monitors spending across calls
• Latency measurement – Tracks response times

Weave applies evaluation scorers to live production traffic:

• Set guardrails that alert on quality degradation
• Monitor token costs and latency in real-time
• Collect human feedback through the UI or API
• Compare production behavior against evaluation baselines

Weave connects with the broader Weights & Biases platform:

• W&B Models – Track model training and fine-tuning
• W&B Registry – Version datasets, models, prompts, and code
• Experiments – Traditional ML experiment tracking
• Serverless fine-tuning – RL-based model improvement

This enables end-to-end workflows from model development through production monitoring.

• Weave on GitHub
• Weave Documentation
• Weave Product Page
• Weights & Biases

• LangSmith – LangChains observability platform
• Langfuse – Open-source LLM observability
• MLflow – Open-source ML lifecycle management
• Weights & Biases – ML experiment tracking platform