rag-architectureRAG system architecture and design decisions. Covers naive vs advanced vs agentic RAG, decision trees for RAG vs fine-tuning vs long context, production topology, latency budgets, and component sequencing. USE WHEN: user mentions "RAG architecture", "RAG design", "naive RAG", "advanced RAG", "agentic RAG", "RAG vs fine-tuning", "RAG vs long context", "prod…
Independent GitHub snapshot · Automated classification · Original documentation governs
name: rag-architecture description: | RAG system architecture and design decisions. Covers naive vs advanced vs agentic RAG, decision trees for RAG vs fine-tuning vs long context, production topology, latency budgets, and component sequencing.
USE WHEN: user mentions "RAG architecture", "RAG design", "naive RAG", "advanced RAG", "agentic RAG", "RAG vs fine-tuning", "RAG vs long context", "production RAG"
chunking-strategies;
query rewriting - use query-transformations; retrieval algorithms - use advanced-retrieval;
evaluation - use rag-evaluation; agent loops - use agentic-rag
allowed-tools: Read, Grep, Glob, Write, Edit| Tier | Components | Best For | Complexity |
|---|---|---|---|
| Naive RAG | Chunk + embed + top-K + stuff | Prototype, < 10k docs, homogeneous content | Low |
| Advanced RAG | + query rewriting, hybrid search, reranking, metadata filters | Production, heterogeneous content, > 10k docs | Medium |
| Agentic RAG | + self-reflection, retrieval-as-tool, multi-hop, corrective fallback | Complex research, multi-source synthesis, high-stakes answers | High |
[Docs] -> [Splitter] -> [Embedder] -> [Vector DB]
|
[Query] -> [Embedder] -> [Top-K Search] --+-> [Prompt Stuffer] -> [LLM] -> [Answer]
Single failure mode: bad retrieval = bad answer. No recovery path. Works for well-scoped FAQ bots on small corpora.
pre-retrieval retrieval post-retrieval
[Query] -> [Router] -> [Rewrite/HyDE/Multi-Q] -> [Hybrid Search] -> [Rerank] -> [Compress] -> [LLM]
|
[BM25] + [Dense Vector] + [Metadata Filter]
Each stage is independently replaceable and measurable. See query-transformations, hybrid-search, reranking.
[Query] -> [Planner Agent]
|
v
+-----+-----+-------------------+
| | |
[Retrieve] [Web Search] [Code Tool]
| | |
+-----+-----+-------------------+
|
[Critic / Self-reflection]
|
+-----+-----+
| |
[Answer] [Replan / More retrieval]
Dynamic step count, dynamic tool selection, self-correction. See agentic-rag.
Is the knowledge dynamic (changes > monthly)?
Yes -> RAG
No -> Continue
|
Is it style/format/persona (not facts)?
Yes -> Fine-tuning (SFT or DPO)
No -> Continue
|
Does total corpus fit in 200k-1M tokens?
Yes -> Long context with prompt caching (cheaper than RAG at small scale)
No -> RAG
|
Need factual grounding with citations?
Yes -> RAG (mandatory for auditability)
No -> Hybrid: long context for recent + RAG for archive
Rules of thumb:
| Stage | Budget | Optimization |
|---|---|---|
| Query embedding | 50-150ms | Batch + local model for simple queries |
| Query rewriting (optional) | 300-800ms | Skip for short factual queries |
| Vector search (top-50) | 20-100ms | HNSW with ef_search tuned |
| BM25 search | 10-50ms | Parallel with vector search |
| Fusion (RRF) | < 5ms | In-memory |
| Reranking (top-50 -> top-5) | 100-400ms | Cohere/Voyage API or local BGE |
| LLM generation | 1000-2000ms | Streaming, prompt caching |
| Total | 1500-3500ms |
Parallelize embedding + BM25. Skip rewriting for short queries. Cache query embeddings for hot terms.
+------------------+
| Ingestion API |
+--------+---------+
|
+-----------v------------+
| Chunker + Embedder |
| (batch worker, queue) |
+-----------+------------+
|
+-------------------+---------------------+
| |
+------v------+ +-------v-------+
| Vector DB | | Document Store|
| (HNSW) | | (Postgres/S3) |
+------+------+ +-------+-------+
| |
| +------------------+ |
+----> Retrieval Service <----------------+
| (hybrid + rerank)|
+--------+---------+
|
+--------v---------+
| LLM Gateway |
| (Claude/GPT) |
+--------+---------+
|
+--------v---------+
| API / UI |
+------------------+
Separate the ingestion path from the query path. Never block queries on indexing.
from dataclasses import dataclass
from langchain_anthropic import ChatAnthropic
from langchain_openai import OpenAIEmbeddings
from langchain_qdrant import QdrantVectorStore
from langchain_community.retrievers import BM25Retriever
from langchain.retrievers import EnsembleRetriever, ContextualCompressionRetriever
from langchain_cohere import CohereRerank
@dataclass
class RAGConfig:
top_k_retrieve: int = 50
top_n_rerank: int = 5
alpha: float = 0.5 # dense weight in hybrid
rerank_model: str = "rerank-english-v3.0"
llm_model: str = "claude-sonnet-4-5-20250929"
def build_pipeline(docs, cfg: RAGConfig):
emb = OpenAIEmbeddings(model="text-embedding-3-small")
vstore = QdrantVectorStore.from_documents(docs, emb, collection_name="kb")
dense = vstore.as_retriever(search_kwargs={"k": cfg.top_k_retrieve})
sparse = BM25Retriever.from_documents(docs); sparse.k = cfg.top_k_retrieve
hybrid = EnsembleRetriever(retrievers=[sparse, dense], weights=[1 - cfg.alpha, cfg.alpha])
reranker = CohereRerank(model=cfg.rerank_model, top_n=cfg.top_n_rerank)
retriever = ContextualCompressionRetriever(base_compressor=reranker, base_retriever=hybrid)
llm = ChatAnthropic(model=cfg.llm_model, max_tokens=1024)
return retriever, llm
| Regime | Index | Strategy |
|---|---|---|
| < 100k chunks | HNSW in-memory (FAISS, Chroma) | Single node |
| 100k-10M chunks | Managed HNSW (Qdrant, Pinecone, Weaviate) | Replicate reads |
| 10M-1B chunks | Sharded HNSW + IVF coarse filter | Partition by tenant/namespace |
| > 1B chunks | Disk-ANN / SPTAG / hierarchical | Custom; hire infra |
Namespace per tenant avoids noisy-neighbor retrieval in multi-tenant SaaS.
| Anti-Pattern | Fix |
|---|---|
| Starting with agentic RAG | Start naive, measure, add complexity only when recall < 70% |
| Treating RAG as a solved box | Every stage needs its own eval; see rag-evaluation |
| Single index for heterogeneous content | Separate indexes per content type with a router |
| Synchronous indexing in query path | Queue ingestion; queries never block on embedding |
| No reranking above 10k docs | Reranking recovers 10-30% recall@5 vs raw vector search |
Hardcoded top_k everywhere | Parametrize; tune with eval set |
| Storing chunks only in vector DB | Keep source of truth in document DB; vector DB holds refs |