Layered RAG: why vector search is the wrong tool for half your questions

The first version of my document RAG system worked beautifully in the demo and fell over on the second real question.

The demo question was "what are the termination conditions in the consulting agreement?" — a meaning question. Embed it, find the chunks that talk about termination, hand them to the model. It works because the question and the answer are semantically close.

The second question was "how many contracts did we sign in Q3?"

Vector search does something for that question. It retrieves chunks that look like contracts and mention dates. The model then counts them and answers with total confidence. The number is wrong, because the retrieved set was never the complete set — it was the top k most similar chunks, and similarity has nothing to do with completeness.

That is the whole problem in one example. Vector search returns things that are similar. A counting question needs things that are complete. No amount of better embeddings fixes that, because it is not a ranking failure.

Two kinds of questions

Once I sorted real user questions into buckets, the split was close to even.

Metadata questions. How many, since when, which ones, by whom, what is the total. These are answerable exactly from structured fields — dates, parties, types, amounts — if those fields exist. They have a correct answer, and being approximately right is the same as being wrong.

Semantic questions. What does this clause mean, does this contradict the other agreement, what were the terms about liability. These need the text, and similarity is genuinely the right retrieval signal.

The mistake was building one pipeline and pointing every question at it. The fix was not a better retriever. It was admitting there are two problems.

The layers

question
   ↓
classify  ──→  metadata?  ──→  SQL over extracted fields
   │
   └────────→  semantic?  ──→  hybrid search (BM25 + vectors)
   │
   └────────→  both?     ──→  SQL to filter, then search inside the result

The third branch is the one that made the system actually useful. "What did we agree about liability in the contracts we signed with vendors last year?" is both: a filter (type = vendor, year = 2025) and a meaning question inside that filtered set. Answering it with pure vector search means hoping the top k happens to land inside the right subset. Answering it with SQL first means the search space is already correct before similarity is consulted.

Extraction is the actual work

The SQL layer only exists if the fields exist, which means extraction at ingest time. For each document: type, parties, dates, amounts, references to other documents. This is unglamorous and it is where most of the effort went.

Two things that mattered more than I expected:

Extract into a schema and validate it. A model asked for a date returns 2025-03-14, March 14, 2025, 14/3/25, and occasionally a sentence explaining that the date is unclear. Validate at the boundary and reject rather than store:

const extractedDocument = z.object({
    documentType: z.enum(['contract', 'invoice', 'report', 'correspondence', 'other']),
    parties: z.array(z.string().min(1)).max(20),
    signedAt: z.iso.date().nullable(),
    totalAmount: z.number().nonnegative().nullable(),
    currency: z.string().length(3).nullable(),
});
 
const extract = async (text: string): Promise<IExtracted | null> => {
    const parsed = extractedDocument.safeParse(await model.json(EXTRACTION_PROMPT, text));
 
    if (!parsed.success) {
        await quarantine.record(text, parsed.error.issues);
        return null;
    }
 
    return parsed.data;
};

A document that fails extraction goes to a quarantine queue, not into the index with null fields. A null field is indistinguishable from "this document genuinely has no amount", and once that ambiguity is in the database every count built on it is unreliable.

Store what you could not extract. The quarantine queue turned out to be the best signal I had about where the pipeline was weak. Documents cluster in there, and the clusters are real categories nobody told me about.

Hebrew broke my assumptions

The corpus is mixed Hebrew and English, and that changed the semantic layer.

Hebrew is morphologically rich — prefixes attach directly to words, so the same root appears in many surface forms. Pure BM25 underperforms because the token that matches the query is a different string from the token in the document. Multilingual embeddings handle that better, but they blur precise identifiers: a contract number or a party name is a string that has to match exactly, and embeddings are specifically good at not caring about exact strings.

Hybrid retrieval, weighted by which language the query is in, beat either alone:

const search = async (query: string, filter: IFilter): Promise<IChunk[]> => {
    const hebrew = HEBREW_PATTERN.test(query);
    const weights = hebrew ? { lexical: 0.3, semantic: 0.7 } : { lexical: 0.5, semantic: 0.5 };
 
    const [lexical, semantic] = await Promise.all([bm25.search(query, filter), vectors.search(await embed(query), filter)]);
 
    return fuseReciprocalRank([lexical, semantic], weights).slice(0, TOP_K);
};

Neither number is principled. They came from a labelled set of about 200 real questions and a sweep, and they are re-checked when the corpus changes.

The router is a classifier, not a prompt

The obvious first version asks the model to pick a route in the same call that answers the question. Do not do this. It couples two failures — the model can route correctly and answer badly, or route badly and answer correctly, and you cannot tell which happened from the output.

Routing is a separate call with a constrained output, which makes it independently measurable:

const route = async (question: string): Promise<Route> => {
    const decision = await model.json(ROUTER_PROMPT, question);
    const parsed = routeSchema.safeParse(decision);
 
    if (!parsed.success) {
        return 'hybrid';
    }
 
    return parsed.data.route;
};

The fallback is hybrid, deliberately. When the router is unsure, the more expensive path that consults both layers is the safe failure — the wrong answer is much more expensive than the extra query.

Once routing was separate I could measure it, and routing accuracy was the single biggest lever on end-to-end quality. It was also the cheapest thing to improve: a handful of labelled examples in the prompt moved it more than any change to the embedding model.

What I would tell myself at the start

Sort your questions before you build a retriever. A hundred real questions, sorted by hand into metadata and semantic, tells you what to build. I built the semantic half first because it is the interesting half, and then discovered that half my traffic was counting questions I was answering wrong.

"Retrieval augmented" is not the same as "vector database". The retrieval that answers a counting question is a SELECT. That is still RAG. Nothing about the pattern requires embeddings.

A confident wrong number is the worst failure mode. A system that says "I cannot answer that exactly" is more useful than one that returns a number that is close. Users check a hedge. They do not check a number.