---

name: retrieval-rag-and-data-pipelines description: "Document preprocessing, LocalDB persistence, retriever indexing, and RAG context assembly." disable-model-invocation: true metadata: disco-role: operating license: MIT

Retrieval, RAG, and Data Pipelines

Use this sub-skill when the task is to prepare documents, split text, persist local data, build retriever indexes, convert retrieval output into context, or assemble a retrieval-first RAG flow.

Route here for

• Document modeling for text corpora and chunked passages.
• TextSplitter chunking rules, separator choices, overlap settings, and invalid chunk-shape troubleshooting.
• ToEmbeddings and RetrieverOutputToContextStr as retrieval-side data transforms.
• LocalDB load/extend/transform/save/load flows for documents and chunked corpora.
• Retriever base contracts and concrete retrieval backends: BM25Retriever, FAISSRetriever, LanceDBRetriever, QdrantRetriever, and PostgresRetriever.
• Retrieval-oriented RAG assembly patterns: split, embed, persist, retrieve, deduplicate, and pass context downstream.

Do not handle here

• Provider/client selection, embedding model setup, or generator configuration: route to the model-client-and-generator-workflows sub-skill.
• Metrics, evaluator loops, prompt optimization, or retriever scoring analysis: route to the evaluation-and-optimization sub-skill.
• Agentic RAG, tool use, runner orchestration, or streaming event handling: route to the agents-tools-and-streaming sub-skill.
• Model training or optimization of retriever/generator parameters: route to the evaluation-and-optimization sub-skill.

Operating workflow

1. Normalize inputs into Document objects and keep raw metadata with the corpus.
2. Choose the lightest valid preprocessing path:
   • TextSplitter for chunking plain text.
   • ToEmbeddings when chunks need vectors.
   • LocalDB when you need persistence, transform reuse, or staged filtering.
3. Pick the retrieval backend:
   • BM25Retriever for lexical or keyword-first retrieval.
   • FAISSRetriever for local vector search.
   • LanceDBRetriever, QdrantRetriever, or PostgresRetriever when the index lives in an optional store.
4. Use RetrieverOutputToContextStr to assemble retrieved chunks into a downstream context string.
5. Build the RAG boundary with a retrieval step, a context builder, and a downstream answer component from another sub-skill.
6. When retrieval fails, consult troubleshooting before changing the corpus or the retrieval contract.

Bundled scripts

• scripts/text_splitter_smoke.py: deterministic TextSplitter smoke check with a tiny Document.
• scripts/localdb_smoke.py: LocalDB load/transform/save/load smoke using a safe bundled transform.

Start with data pipelines for document and transform contracts, retrievers for backend selection, and RAG recipes for assembly patterns.