When a retrieval-augmented system gives a bad answer, the instinct is almost always to blame the model — try the newest release, add a bigger context window, tune the prompt harder. In our experience, the actual fault sits upstream of the model far more often than it sits in the model itself: the retrieval step handed the model the wrong context, and no amount of prompting fixes a model reasoning correctly over the wrong document. Get retrieval right and a modest model performs well. Get it wrong and the best model available will confidently answer from the wrong source.
01Chunking strategy is the highest-leverage decision, and it's usually made carelessly
A common default is fixed-size chunks — 512 tokens, 50-token overlap — applied uniformly regardless of document structure. This works reasonably for prose. It works badly for structured documents: a numbered clause in a policy document, a table with a header row, a legal contract where clause 4.2 references "the terms defined in section 2" — split those across a chunk boundary and the retrieved chunk loses the antecedent it depends on. The model then either hallucinates a plausible-sounding completion or answers from a chunk that's technically relevant but missing the qualifying detail that changes the answer. Structure-aware chunking — splitting on actual document boundaries (sections, clauses, table rows) rather than a fixed token count — costs more engineering time upfront and consistently outperforms fixed-size chunking on structured content, in our experience by a wide enough margin that it's worth the extra build time on any corpus that isn't pure prose.
02Recall, not similarity score, is what to measure
A common failure is trusting cosine similarity scores as a proxy for retrieval quality without ever measuring whether the right document actually appears in the top-k results. The two aren't the same thing — a system can return high-similarity-score chunks that are topically related but don't contain the specific fact the query needs, every time, and the similarity scores will look fine throughout. The metric that matters is recall@k against a held-out set of real queries with known correct source documents: does the right document actually appear in the top 5 (or whatever k you retrieve)? If recall@5 sits below roughly 85% on a representative query set, no amount of prompt engineering downstream will reliably fix the answer quality, because the model is being asked to reason over context that doesn't contain the answer around 15% of the time or worse.
03Hybrid search — dense plus sparse — outperforms pure vector search on most real corpora
Dense embeddings are excellent at semantic similarity and weak at exact-match precision: a query for a specific product SKU, error code, or defined term often retrieves semantically related-but-wrong results from a pure vector search, because embedding models compress exact tokens into the same fuzzy semantic space as everything else. Pairing dense retrieval with a sparse method (BM25 or similar keyword-based scoring) and combining the two result sets closes this gap — the sparse side catches exact-match cases the dense side misses, and the dense side still catches paraphrased queries the sparse side would miss. This is a well-established pattern in information retrieval generally, not a novel technique, and it's still frequently skipped in RAG builds that reach for vector search alone because it's the default in most getting-started tutorials.
04Reranking is a cheap fix for a common problem: the right document retrieved, ranked too low
Initial retrieval (dense, sparse, or hybrid) is optimized for speed across a large corpus, which trades off some precision. A second-stage reranker — a smaller, more expensive-per-query model that re-scores the top 20–50 initial candidates — recovers a meaningful amount of that precision for a small latency cost, since it only runs on a short list rather than the whole corpus. If your top-k results often contain the right document but not in the first two or three positions, a reranking stage is usually a better investment than a bigger initial retrieval window, since the model tends to weight earlier context in the prompt more heavily regardless of what's technically included further down.
05Metadata filtering deserves to happen before semantic search, not after
A query with a clear scoping signal — a date range, a product category, a document type — should filter the candidate set before the similarity search runs, not retrieve broadly and hope the semantic ranking sorts it out. Filtering after retrieval means the top-k slots are being contested by documents that should have been excluded from consideration entirely, which quietly lowers recall for the documents that actually matter.
None of this requires a frontier model. A well-architected retrieval pipeline with structure-aware chunking, hybrid search, a reranking stage, and a real recall metric will outperform a poorly-architected one paired with the newest, most expensive model available — because the model can only work with what it's given, and no model reasons its way to a fact that was never in its context window. Model choice is a real decision and it matters. It's just rarely the decision that's actually broken when a RAG system gives a wrong answer.