Skip to content

Chunking & Retrieval: Feeding a Model More Than Fits

This is the canonical illustration of the design principle from the introduction: pick your approach based on the limitation it works around. The limitation here is #1 — the context window. You cannot paste an entire codebase, document corpus, or knowledge base into a single model call. So you split it, index it, and retrieve only the pieces relevant to the question at hand.

For non-engineers: imagine asking a consultant to review a 10,000-page legal archive, but they can only hold 30 pages in their hands at once. You don't hand them the archive — you build an index, find the relevant pages, and give them just those. That's retrieval. The whole craft is picking the right 30 pages.

The problem, concretely

A model has a fixed context window (tens to hundreds of thousands of tokens depending on the model — see choosing models). A real software project is far larger. Even when a codebase would technically fit a very large window, stuffing everything in is a bad idea:

  • It's expensive (you pay per token, every call).
  • It's slow.
  • Quality drops — models attend less reliably to information buried in a huge context (the "lost in the middle" effect). More context is not more understanding.

So the goal is never "give the model everything." It's "give the model exactly what it needs, and nothing else."

Two families of retrieval

There are two broad ways to find the relevant pieces:

Semantic (vector) retrieval Structural retrieval
How Embed chunks as vectors; find chunks "similar in meaning" to the query Parse the code into symbols and a graph; answer exact structural queries ("what calls this function?", "what does this contract inherit?")
Good for Fuzzy natural-language questions over prose/docs Code, where relationships are exact and you want precision
Cost Embedding model + vector DB A parser + an in-memory index

A lot of teams reach for a vector database reflexively. It's worth asking whether you actually need fuzzy similarity, or whether your data has exact structure you can exploit instead.

Where embeddings come from. Semantic retrieval needs a model to turn text into vectors, and Anthropic has no first-party embeddings endpoint — you pair Claude with a dedicated embedding provider (Voyage AI has historically been the one Anthropic recommends) or an open-source model. Check Anthropic's embeddings guide before committing, since the recommendation can change.

Case study: AuditAgent's structural code index

When AuditAgent audits a smart-contract repo, it needs to give the model the right slice of code for each potential finding. It deliberately does not use a vector DB. Instead it builds a structural index:

  1. Scan & parse every file with tree-sitter (a fast, language-aware parser supporting Solidity, Rust, TypeScript, Python, Go, and more).
  2. Extract symbols (functions, contracts, structs), imports, inheritance, calls, and modifiers.
  3. Store them in an in-memory SQLite database with a generic edges table (source → destination with an edge type like calls, inherits, imports). New relationship types are a one-row insert, not a schema migration.
  4. Resolve cross-file references in a second pass — a single pass can't, because a function may call something defined in a file not yet parsed.
  5. Serve structural queries: "find this symbol," "who calls it," "what's its inheritance chain," "which tests exercise it."

The design rationale is instructive — each choice is a limitation-driven trade-off:

  • SQLite, not a vector DB, because the questions are exact ("what calls transfer?"), not fuzzy. The standard library handles it; no extra infrastructure.
  • Source code is not duplicated into the index. Only one- or two-line previews plus line ranges are stored; full function bodies are read from disk on demand and cached. This keeps memory bounded — a direct response to "we can't hold it all at once."
  • No LLM in the retrieval path. An earlier version used an LLM to filter/crop retrieved context; it was removed after testing showed it added latency and cost without measurably improving relevance. (A great example of measuring instead of assuming.)
  • Ephemeral. The index is rebuilt fresh per scan and lives only in RAM — no cache-invalidation bugs, no cross-job pollution.

The result is that for any given finding, the system assembles a tight "context bundle" — the function itself, its callers, the modifiers guarding it, relevant tests — and a bounded graph "expander" walks a few hops out to include closely-related code, ranked by relevance, capped by a budget. The model gets a focused, high-signal slice instead of a haystack.

Chunking: how to split

Whatever the retrieval method, you have to decide how to cut the source into pieces. Bad chunking quietly destroys quality.

  • Chunk on natural boundaries, not arbitrary character counts. For code, that's functions/classes/contracts. For prose, that's headings/sections. A chunk that splits a function in half is worse than useless.
  • Keep chunks self-contained. A chunk should carry enough context to be understood alone — e.g. include the function signature and a line or two of its docblock, not just the body.
  • Watch the edge cases. AuditAgent's markdown chunker, for instance, had to correctly handle fenced code blocks (```) so it wouldn't terminate a chunk mid-block. These details matter more than they look.
  • Mirror your source structure when you store chunks, so a retrieved chunk can be traced back to its origin (file, line range).

Context budgeting

Once you can retrieve, you still have to fit the result into a budget. Patterns:

  • Cap how much you include. AuditAgent ships a file's full source only when it's under a token threshold (e.g. ~6,000 tokens); above that it falls back to a compact "header + relevant snippets" representation.
  • Rank, then truncate. Score candidate chunks by relevance and keep the top N. Don't include something just because it matched.
  • Leave room for the answer. The output and any reasoning also consume the window. Budget input so the model has room to respond.
  • Use breadcrumbs for breadth. When you can't afford full bodies for everything in scope, include one-line "here's what exists and where" pointers, and let a tool loop fetch full bodies on demand. This trades depth for completeness at low cost.

Retrieval-Augmented Generation (RAG), in one line

"RAG" is just this pattern with a name: Retrieve relevant context, Augment the prompt with it, then Generate. Whether you retrieve via vectors or structure, the shape is the same — and it's the standard answer to "the model doesn't know my data" (limitation #3).

Takeaways

  • The context window is a hard limit; design around it from the start.
  • More context ≠ better output. Aim for high signal, not high volume.
  • Don't default to a vector DB — ask whether your data has exact structure you can query instead.
  • Chunk on natural boundaries; keep chunks self-contained.
  • Budget the window: rank, cap, and leave room for the answer.

Sources