The baseline that beats most stacks
Most production RAG systems in 2026 still underperform a shockingly simple baseline: BM25 (lexical) + a modern reranker + top-5 chunks stuffed into a good model. No vector DB required.
Before adding embeddings, benchmark this. Half the RAG teams we talk to are running a $300/month Pinecone bill and getting worse recall than they'd get from Elasticsearch and Cohere Rerank.
Hybrid retrieval
Once you exceed a few million chunks or need semantic recall on paraphrases, hybrid (BM25 + dense embeddings) becomes the right default. Reciprocal Rank Fusion combines the two rankings without hyperparameter tuning.
Use a strong embedding model — Voyage-3, OpenAI text-embedding-3-large, or Cohere embed v4. Cheap embeddings save $10/month and cost you 15 points of recall.
Store metadata alongside vectors: document ID, section, timestamp, author. Filter aggressively before ranking — a 100K-chunk index filtered to the right document is faster and more accurate than semantic search over everything.
Reranking is not optional
Retrieve 50 candidates, rerank to top 5. Cohere Rerank 3, Voyage Rerank, or a Jina reranker all move quality dramatically — often more than any embedding upgrade.
Reranker latency is ~100-300ms for 50 candidates. Budget for it. The alternative is stuffing 20 mediocre chunks into your prompt, paying for the tokens, and getting a worse answer.
When long-context wins
For single-document Q&A where the whole doc fits in context (say < 200K tokens on Claude, < 1M on Gemini 2.5), skip RAG entirely. Stuff the doc, ask the question, cache the prompt.
This is faster to build, cheaper per query than a well-run RAG stack once you factor in caching, and eliminates the whole class of 'we retrieved the wrong chunk' bugs.
Break glass: use RAG when your corpus is large, changes frequently, or spans documents. Long-context wins when the corpus is small, stable, and per-session.
Evaluate retrieval, not just answers
Most RAG eval suites grade the final answer. That confounds retrieval quality with generation quality. Grade them separately.
For retrieval: build a golden set of {query → relevant chunk IDs}. Measure recall@5, recall@10, and MRR. When retrieval regresses, you catch it before it poisons the answer.
For generation: give the model the correct chunks and grade the answer. If answers are bad with perfect retrieval, the prompt or model is the problem, not the pipeline.