Evaluation and Observability for LLM Products¶

You cannot ship an LLM feature on vibes. Because the model's output is probabilistic (limitation #2 from AI limitations), "I tried it a few times and it looked good" is not evidence — the next call can format differently, hallucinate, or regress after a one-line prompt edit. Two disciplines turn a demo into a product you can defend:

For non-engineers: you can't tell whether an AI feature works by trying it once — the same input can give a different answer next time. Evaluation is grading the model against a set of known-good examples, so you can prove a change actually helped and didn't quietly break something else. Observability is recording every real call in production so you can see what it did, what it cost, and why. This page is about building both.

Evaluation (evals) — measuring quality against a known-good benchmark, so you can prove a change is an improvement and not a regression.
Observability — recording every model call in production so you can see what happened, what it cost, and why it behaved the way it did.

This page covers both, with the durable patterns first and AuditAgent as a worked example. AuditAgent is just an illustration; the practices apply to any LLM-backed product. This page is the companion to Patterns for building real products on LLM APIs.

Part 1 — Evaluation¶

Why evals, not vibes¶

A unit test asserts an exact value. An LLM eval can't, because there's no single correct string — so evals measure quality statistically across many cases. Anthropic's guidance in Define success criteria and build evaluations boils down to a few principles worth internalizing:

Be empirical. Score on a measurable scale (correct/incorrect, 1–5, a match rate) rather than a subjective gut read.
Be task-specific. Your eval set should mirror your real input distribution, edge cases included.
Automate and prioritize volume. Many cases with automated grading beat a handful of hand-graded ones — noise averages out across volume.
Be multidimensional. Most features need several criteria (e.g. accuracy and format and safety), not one number.

The building blocks¶

Block	What it is	Why it matters
Golden dataset	A curated set of inputs with known-good expected outputs ("ground truth").	The fixed yardstick every change is measured against.
Scorer	A function that compares model output to ground truth and emits a number.	Turns fuzzy quality into a trackable metric.
Regression check	Re-running the whole eval suite on every prompt/model change and gating on a threshold.	Stops a "small improvement" from silently breaking ten other cases.
Held-out / anti-overfit guard	Rules and data that keep you from tuning to the benchmark instead of the task.	A prompt that memorizes your test set helps no real user.

Grading can be code-based (exact/structural match), model-based (an LLM-as-judge scores against a rubric — scales to thousands of cases), or human (the gold standard for the hard cases). Most mature setups mix all three. Anthropic's cookbook has runnable recipes for each grader style.

Scoring is rarely binary¶

Real outputs aren't simply right or wrong. A robust scorer usually buckets results — and weights them by how much you trust the source.

AuditAgent in practice. Its validation harness scores every produced finding into three buckets against a golden dataset: true positive (matches a known vulnerability), acceptable (a real but not-in-ground-truth issue), and false positive (noise). It tracks not just precision but recall — "distinct ground-truth issues covered" — because a security tool that misses real bugs is worse than one that's merely noisy. Separately, its deterministic audit-score algorithm weights findings by severity and by detector confidence: LLM-based context scans are multiplied by 0.70 while deterministic static analysis gets 0.95, encoding directly into the math that probabilistic output is lower-trust evidence.

The transferable idea: don't collapse quality into a single pass/fail. Separate recall from precision, separate "wrong" from "extra," and weight signals by their reliability.

The tuning loop (and the overfitting trap)¶

The workflow for improving an LLM feature is a tight loop: change one thing → run the eval suite → compare metrics to the previous best → keep or revert. Gate it on explicit thresholds so "better" is objective, and snapshot the prompt before each edit so you can roll back.

The danger is overfitting to your own benchmark: tweaking a prompt until it nails the exact test repos but generalizes worse. The guard is a discipline rule — changes must address categories of failure, never name specific test cases.

AuditAgent in practice. Its prompt-tuning loop runs exactly this cycle against a multi-repo gold set, gating on concrete thresholds (e.g. true-positives kept > 80% AND surviving false-positives < 30% in aggregate). Each iteration snapshots the prompt file before editing, logs what was tried, and follows a hard rule: prompt changes must be general principles, never references to specific repos, contracts, or findings — explicitly to avoid overfitting the benchmark.

Anthropic's Evaluation Tool supports this loop in the Console (test cases, side-by-side prompt comparison, re-run on change), and their statistical approach to model evals is the reference for treating eval scores as measurements with noise rather than exact truths.

Don't forget end-to-end evals¶

Component evals (does this prompt return good findings?) don't catch breakage in the plumbing around the model — auth, the UI flow, the request pipeline. A thin layer of end-to-end tests on the critical user path catches those.

AuditAgent in practice. A separate qa-agent drives real browser missions (via Playwright) through the live product — log in, request a scan, fill every step of the scan stepper — verifying the whole flow a user actually touches, not just the model's output.

Part 2 — Observability¶

You can't fix what you can't see¶

In production, the model is a black box running inside an even bigger black box (retries, tool calls, multi-step pipelines). Without instrumentation, a slow or expensive or wrong response is impossible to diagnose after the fact. Tracing records the full lifecycle of each request — every model call, tool execution, and retrieval step — with timing, inputs, outputs, token counts, and cost.

The core data model (as standardized by tools like Langfuse) is worth knowing:

Trace — one end-to-end request (e.g. one scan, one chat turn).
Observation / span — an individual step within a trace; these nest.
Generation — a special span for a single LLM call, carrying model name, prompt, completion, token usage, and cost.
Score — a quality measurement attached to a trace, which is what links observability back to evals.

What to instrument¶

Every model call, tagged with which step it belongs to — so you're not staring at a wall of anonymous "completion" entries.
Token usage and cost, per call and per trace, so you can find the expensive steps.
Latency, including time-to-first-token for streaming features.
Tool calls and retries, so a runaway loop is visible.
Production scores — sample real traffic and run evals on it, not just your offline set.

AuditAgent in practice. Every model call is wrapped in a named Langfuse span tagged <strategy>:<stage> (e.g. deep_triage:verify, taint_path:analyze). The team learned this the hard way: before the tags, every call showed up as an anonymous complete, making it impossible to attribute cost or diagnose which stage was misbehaving. Production runs against Langfuse Cloud; local development points at a self-hosted Langfuse stack — the same instrumentation either way.

Tracing closes the loop with evals¶

The two disciplines aren't separate. Observability gives you a stream of real production traces; you sample those traces, score them (with code, model-as-judge, or human review), and feed the hard cases back into your golden dataset. Langfuse is built around exactly this loop — tracing, scores, datasets, and experiments in one place — and it batches trace ingestion in the background so instrumentation doesn't slow the request path.

The mental model. Evals tell you whether a change is good before you ship. Observability tells you whether it's still good after real users hit it — and supplies the next round of eval cases. A product that does both can improve its LLM features deliberately instead of guessing.