Patterns for Building Real Products on LLM APIs¶

This page is about the API path: using a model API (Claude, GPT, Gemini, …) as a component inside a product you ship to users — not using AI as a coding assistant at your desk. If you want the distinction spelled out, see Two ways to use AI. Here the model is a runtime dependency, like a database or a payment provider, and your code has to call it reliably, cheaply, and safely thousands of times a day.

For non-engineers: this page is about using an AI model as a part inside a product you ship — the way you'd use a database or a payment provider — not as a coding assistant. A raw model is just "text in, text out": no memory, no ability to act. Everything useful comes from the code built around it. The rest of this section is a tour of those durable building blocks.

This page is the map of the section. It introduces the mental model and summarizes each durable pattern in a paragraph, then links to a dedicated deep-dive page. Read it top to bottom for the lay of the land, then follow the links that matter to what you're building.

Prefer to watch first? You Can Learn AI Agent Harness & Loop Engineering In 19 Min (Sean's AI Stories, YouTube) walks almost exactly the territory of this section — the agent loop and its harness, memory, retrieval/RAG, tool calling and its guardrails, then tracing and evals — in plain language in 19 minutes. A good primer before the deep-dive pages, whether you're technical or not.

The one mental model to start from¶

A raw LLM call is a pure function: text in, text out, no memory, no side effects, no ability to act. Everything interesting about an LLM product comes from the scaffolding you build around that function. The model is the engine; the product is the car.

Three hard limitations shape almost every pattern below. They are covered in depth in AI limitations, and this page uses the same numbering:

Limited memory — the model can only "see" a fixed window of tokens at once, and each call is stateless, so you can neither feed it everything nor rely on it to remember across runs.
No real thinking (no compounding learning) — output is probabilistic; the model can be wrong, drift in format, or be manipulated, it can't reliably check its own work, and it doesn't learn from one run to the next.
Outdated knowledge — training has a cutoff, so the model doesn't know your private data or anything recent unless you supply it.

Most patterns in this section exist to work around one of those three limits. Each one is illustrated with how AuditAgent — an in-house AI smart-contract security auditor — handles it. AuditAgent is just a worked example; the pattern is the point, and every one reappears in chatbots, RAG systems, coding agents, and data-extraction pipelines.

The patterns at a glance¶

Pattern	What it solves	Deep dive
Tool-use loop / agents	A bare model can only emit text; a loop lets it act — look things up, run analyzers, decide its next step.	Tool loops & agents →
Chunking & retrieval	You can't fit a whole codebase/corpus in the window, so you index it and retrieve only the relevant slice (limitation #1).	Chunking & retrieval →
Persistent memory	Each call is stateless, so durable knowledge across runs has to be stored and re-injected by you (limitations #1 & #2).	Persistent memory →
Structured output & verification	Make the model fill a typed schema so code can act on it, and check its work with a second model (limitation #2).	Structured output, verification & cost →
Decomposition into a pipeline	One giant prompt is hard to control; split the job into specialized, individually-testable steps.	Below
Model selection in product code	At scale, model choice is an economic decision: cost, latency, and capability trade off.	Below + Choosing models →
Reliability engineering	Non-determinism, prompt injection, runaway cost, retries — concerns a shipped product can't skip.	Below
Evaluation & observability	You can't verify probabilistic behavior by clicking once; you need evals and tracing.	Evaluation & observability →

The four patterns linked to dedicated pages each get the full treatment there. The remaining three — decomposition, model selection, and reliability — are cross-cutting, so they live here.

Decomposition into a pipeline¶

One giant prompt that tries to do everything at once is hard to control, hard to debug, and degrades as context fills up. The durable move is to break the job into specialized steps, each with its own focused prompt, model, and output schema, and chain them together. Anthropic calls this prompt chaining in Building Effective Agents: each step processes the output of the previous one, and you can insert programmatic checks (gates) between steps.

Benefits: each step gets the model's full attention on a narrow task, you can use a cheaper model for easy steps, you can test and trace each step independently, and you can retry a failed step without re-running the whole job.

A closely related principle: do as much work as possible without the LLM. Parsing, scoring, graph walks, deduplication, and filtering are cheaper, faster, and reproducible in plain code. Reserve model calls for the genuinely judgment-based steps, and pre-process everything around them deterministically.

AuditAgent in practice. A scan is a multi-phase pipeline — context generation, then hypothesis generation, then findings + validation — rather than one prompt. Within the analysis stage, the code agent runs several specialized strategies (a broad triage pass, a cross-boundary data-flow analysis, a memory-safety detector, and an open-ended agentic loop), each decomposed into its own stages such as triage → analyze → verify → generalize. Raw findings then flow through a verifier, a variant search, and a dedup pass before scoring. No single prompt is asked to "audit the whole repo," and the deterministic parts (parsing, scoring, dedup) run in ordinary code.

Model selection in product code¶

At your desk, you reach for the most capable model and don't think about cost. In a product that makes millions of calls, model choice is an engineering and economic decision: per-token cost at scale, latency budgets, and capability all trade off against each other. The pattern is to match the model to the step, often using a cheap/fast model for high-volume narrow work and a frontier model only where reasoning quality pays for itself.

Cost at scale — a 10× cheaper model on a step you call a million times a day dominates your bill.
Latency — interactive features have a budget; smaller models and streaming help.
Capability — reserve the expensive model for steps that genuinely need it.
Abstraction — route through a thin internal layer so swapping a model (or provider) is a config change, not a refactor, and so callers can't pick an unsafe/expensive option directly.

For the capability/price/latency landscape and how to choose, see Choosing models. The cost levers specific to a high-volume product — prompt caching, model tiering — are covered in Structured output, verification & cost.

AuditAgent in practice. Calls go through an internal llm_router that abstracts multiple providers (Claude and GPT model families, among others) behind one interface. Scan tiers (QUICK / MEDIUM / FULL) map to fixed strategy + model + budget combinations; clients pick a tier and never see the underlying model knobs, so they can't disable verification or force a cheaper model. A cheap model handles high-volume per-file annotation while a stronger model does the deep analysis — the textbook "right model for the step" split.

Reliability engineering¶

When the model is a production dependency, a category of problems appears that you never face using AI as a dev tool. None of these are optional for a shipped product.

Concern	Why it bites on the API path	Mitigation pattern
Non-determinism	Same input can yield different output; "it worked once" proves nothing.	Schema-validate every output; build evals; treat low-confidence model output as a lower-trust signal.
Prompt injection & safety	Untrusted content in your context (user input, scraped docs, a repo under analysis) can carry instructions that hijack the model.	Never blindly execute model-requested actions; sandbox tool execution; separate trusted instructions from untrusted data; least-privilege tools.
Cost control	A runaway loop or an oversized context can be expensive fast.	Turn/tool-call budgets; prompt caching for stable prefixes; the Batch API (~50% off) for non-latency-sensitive work; token accounting; per-tier limits.
Retries & timeouts	Networks fail, providers rate-limit, slow calls stall the request.	Bounded retries with backoff that honors the provider's `Retry-After`; connect vs request timeouts; per-tool timeouts.
Versioning prompts	A "small" prompt edit silently changes behavior for everyone.	Version prompts; snapshot before changes; gate changes behind regression evals.

AuditAgent in practice. The llm_router ships a with_retry layer that honors a provider Retry-After value before falling back to exponential backoff, separate connect_timeout (5s) and request_timeout (300s), and a per-tool-call timeout. Prompt caching is its single largest cost lever — caching the large static system prompt took a representative scan from ~$13 to a ~$4–6 envelope. And it treats model output as lower-trust evidence: in its scoring algorithm, findings from LLM-based scans carry a lower confidence multiplier than findings from deterministic static analysis — a concrete way of encoding "the model can be wrong" into the product.

Prompt caching caveat. Caching only helps if the cached prefix is byte-stable. A timestamp in the system prompt, reordered tool definitions, or an edited parameter all bust the cache. Keep volatile data out of the static prefix and pass it in later messages instead. Anthropic's Lessons from building Claude Code is a good field guide.

You can't ship LLM features on vibes¶

Every pattern above produces probabilistic behavior, which means you cannot verify it by clicking around once and declaring victory. Shipping requires evaluations (golden datasets, scoring, regression checks) and observability (tracing every call so you can diagnose cost and behavior). That is its own discipline — see Evaluation and observability.