---

name: llamaindex description: >- LlamaIndex is an open-source Python framework for connecting LLMs to your own data: it loads and chunks documents, indexes them in a vector store, and answers questions over them with retrieval (RAG) and tool-calling agents. Use when ingesting documents, configuring retrieval strategies, building query engines, creating multi-step agents, or evaluating answer quality. Trigger words: llamaindex, llama-index, rag, retrieval augmented generation, vector index, query engine, document loader, knowledge base. license: Apache-2.0 compatibility: "Python 3.10+; an LLM and embedding provider (OpenAI by default, or local models through Ollama)" metadata: author: terminal-skills version: "1.1.0" category: data-ai tags: ["llamaindex", "rag", "llm", "retrieval", "knowledge-base"] repository: https://github.com/run-llama/llama_index

LlamaIndex

Overview

LlamaIndex is a data framework for building RAG pipelines, knowledge assistants, and data-augmented LLM applications. It provides document loaders, chunking strategies, several index types, hybrid retrieval with reranking, tool-calling agents, and evaluation tools for question-answering systems. The Python library is llama-index-core plus one pip package per integration (300+). This skill follows the 0.14 API: agents are workflow-based (FunctionAgent), and the agent classes from before 0.13 no longer exist.

Instructions

Install and configure models

pip install llama-index                 # llama-index-core + the OpenAI LLM and embedding integrations
pip install llama-index-readers-file    # PDF, DOCX, PPTX, CSV parsing for SimpleDirectoryReader
pip install llama-index-vector-stores-chroma llama-index-retrievers-bm25 llama-index-postprocessor-cohere-rerank

The package name spells the import: llama-index-llms-ollama is from llama_index.llms.ollama import Ollama. Set models once on Settings; every index, query engine, and evaluator picks them up.

from llama_index.core import Settings
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI

# Both read OPENAI_API_KEY from the environment. Left unset, the defaults are the
# older gpt-3.5-turbo and text-embedding-ada-002.
Settings.llm = OpenAI(model="gpt-4o-mini")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

# Local models instead: pip install llama-index-llms-ollama llama-index-embeddings-ollama
# from llama_index.llms.ollama import Ollama
# from llama_index.embeddings.ollama import OllamaEmbedding
# Settings.llm = Ollama(model="llama3.1", request_timeout=360.0, context_window=8000)
# Settings.embed_model = OllamaEmbedding(model_name="nomic-embed-text")

Load, index, query

from llama_index.core import SimpleDirectoryReader, StorageContext, VectorStoreIndex, load_index_from_storage

documents = SimpleDirectoryReader(
    "docs/handbook", recursive=True, required_exts=[".md", ".pdf"], filename_as_id=True
).load_data()
index = VectorStoreIndex.from_documents(documents)   # chunks, embeds, stores in memory
index.storage_context.persist(persist_dir="storage")

# Later runs: reload instead of re-embedding
index = load_index_from_storage(StorageContext.from_defaults(persist_dir="storage"))
response = index.as_query_engine(similarity_top_k=4).query("How many vacation days do new hires get?")
print(response)
for hit in response.source_nodes:
    print(hit.score, hit.metadata["file_name"])

• When chunking, start with SentenceSplitter at 1024 tokens with 200 token overlap (its defaults), use MarkdownNodeParser for structured documents, CodeSplitter for code (needs tree_sitter_language_pack), and adjust based on evaluation results.
• When indexing, use VectorStoreIndex as the default for most RAG, PropertyGraphIndex for entity relationships (KnowledgeGraphIndex is deprecated), and DocumentSummaryIndex for per-document summaries.
• When building query engines, use RetrieverQueryEngine for standard RAG, CitationQueryEngine for responses with source attribution, and SubQuestionQueryEngine for complex multi-part queries.
• Set similarity_top_k based on context window: 3-5 chunks for large models, 2-3 for smaller models.

Incremental ingestion into a vector store

import os
import chromadb
from llama_index.core.extractors import TitleExtractor
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.vector_stores.chroma import ChromaVectorStore

collection = chromadb.PersistentClient(path="chroma_db").get_or_create_collection("handbook")
vector_store = ChromaVectorStore(chroma_collection=collection)

pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(chunk_size=1024, chunk_overlap=200),
        TitleExtractor(),            # per document: one LLM call per chunk (first 5) plus one to merge; adds document_title
        Settings.embed_model,        # must be a stage when a vector store is attached
    ],
    docstore=SimpleDocumentStore(),  # remembers a hash per document id
    vector_store=vector_store,
)
if os.path.exists("pipeline_storage"):
    pipeline.load("pipeline_storage")
nodes = pipeline.run(documents=documents)   # unchanged files are skipped, edited ones are replaced
pipeline.persist("pipeline_storage")
index = VectorStoreIndex.from_vector_store(vector_store)

Deduplication keys on the document id, so load with filename_as_id=True; otherwise every run generates new ids and re-embeds everything.

Hybrid retrieval and reranking

from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.postprocessor.cohere_rerank import CohereRerank
from llama_index.retrievers.bm25 import BM25Retriever

chunks = vector_store.get_nodes(node_ids=None)   # every chunk in the Chroma collection
hybrid = QueryFusionRetriever(
    [index.as_retriever(similarity_top_k=10), BM25Retriever.from_defaults(nodes=chunks, similarity_top_k=10)],
    similarity_top_k=10,
    num_queries=1,               # 1 = no LLM-generated query variations
    mode="reciprocal_rerank",
)
engine = RetrieverQueryEngine.from_args(
    hybrid,
    node_postprocessors=[CohereRerank(top_n=4)],   # reads COHERE_API_KEY
)

Agents

import asyncio
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.core.tools import QueryEngineTool
from llama_index.core.workflow import Context

handbook_tool = QueryEngineTool.from_defaults(
    query_engine=engine,
    name="handbook_search",
    description="Answers questions about the employee handbook: time off, expenses, hardware.",
)
agent = FunctionAgent(
    tools=[handbook_tool],
    llm=Settings.llm,
    system_prompt="Answer HR questions. Search the handbook before answering.",
)

async def main():
    ctx = Context(agent)   # carries chat history between runs
    print(await agent.run("How many vacation days do new hires get?", ctx=ctx))
    print(await agent.run("And how many of them carry over?", ctx=ctx))

asyncio.run(main())

FunctionAgent relies on the provider's native tool calling. For models without it, use ReActAgent from the same module; AgentWorkflow(agents=[...]) lets several agents hand off to each other. Plain Python functions with type hints and a docstring are valid tools.

Evaluation

from llama_index.core.evaluation import BatchEvalRunner, FaithfulnessEvaluator, RelevancyEvaluator

async def evaluate(engine, questions):
    judge = OpenAI(model="gpt-4o")
    runner = BatchEvalRunner(
        {"faithfulness": FaithfulnessEvaluator(llm=judge), "relevancy": RelevancyEvaluator(llm=judge)},
        workers=4,
    )
    results = await runner.aevaluate_queries(engine, queries=questions)
    for name, evals in results.items():
        print(name, sum(e.passing for e in evals) / len(evals))

Faithfulness checks the answer against the retrieved context and relevancy against the question; neither needs labels. CorrectnessEvaluator scores 1-5 against a reference answer you supply.

Examples

Example 1: Build a RAG pipeline over company documentation

User request: "Create a question-answering system over our internal docs"

With Settings, documents, and the ingestion pipeline above in ask_handbook.py, add a citation engine:

import sys
from llama_index.core.query_engine import CitationQueryEngine

engine = CitationQueryEngine.from_args(index, similarity_top_k=4, citation_chunk_size=512)
response = engine.query(sys.argv[1])
print(response)
for i, hit in enumerate(response.source_nodes, start=1):
    print(f"[{i}] {hit.metadata['file_name']}")

python ask_handbook.py "How many vacation days do new hires get, and do unused days carry over?"

Output (wording varies by model):

New hires receive 25 vacation days per calendar year, prorated from the start date [1].
Up to 5 unused days carry over and expire on March 31 [1].
[1] time-off.md
[2] expenses.md

The second run of the script embeds no documents (only the question), because the pipeline's document store finds every file unchanged.

Example 2: Create a multi-source research agent

User request: "Build an agent that can search our handbook and our runbooks, and also do date math"

from datetime import date
from llama_index.core.agent.workflow import FunctionAgent, ToolCallResult
from llama_index.core.tools import QueryEngineTool

def days_between(start: str, end: str) -> int:
    """Number of days between two ISO dates (YYYY-MM-DD)."""
    return (date.fromisoformat(end) - date.fromisoformat(start)).days

tools = [
    QueryEngineTool.from_defaults(handbook_index.as_query_engine(), name="handbook_search",
        description="HR policies: time off, expenses, hardware purchases."),
    QueryEngineTool.from_defaults(runbook_index.as_query_engine(), name="runbook_search",
        description="Engineering runbooks: deploys, incident response, on-call."),
    days_between,
]
agent = FunctionAgent(tools=tools, llm=Settings.llm, system_prompt="Use the tools; cite which one you used.")

async def ask(question: str):
    handler = agent.run(question)
    async for event in handler.stream_events():
        if isinstance(event, ToolCallResult):
            print("tool:", event.tool_name, event.tool_kwargs)
    print(await handler)

Output for asyncio.run(ask("Vacation carry-over expires March 31. How many days is that from 2026-10-01?")) (which tools are called, and with what arguments, varies by model): one tool: line per call — for example handbook_search, then days_between {'start': '2026-10-01', 'end': '2027-03-31'} — followed by an answer that uses the returned 181 days.

Guidelines

• Use SentenceSplitter with 1024 token chunks and 200 token overlap as the starting point.
• Install llama-index-readers-file before loading PDFs or Office files: since 0.14.18 pip install llama-index no longer pulls it in, and without it SimpleDirectoryReader only logs a warning and reads those files as raw text.
• Code written for 0.12 or earlier breaks: ServiceContext raises (use Settings), and ReActAgent.from_tools, AgentRunner, FunctionCallingAgent, OpenAIAgent, and QueryPipeline were removed in 0.13. Agents are async; call them with await agent.run(...).
• Use hybrid retrieval (vector + keyword) for production; pure vector search misses exact term matches. QueryFusionRetriever runs its retrievers asynchronously, so a vector store created with only a sync client (Qdrant, for one) fails there — pass its async client too or set use_async=False.
• Add a reranker (CohereRerank) after retrieval to improve result relevance for small cost: retrieve 10, keep 4.
• Metadata extractors (TitleExtractor, SummaryExtractor) improve retrieval but cost LLM calls on every new or changed document; add them once the plain pipeline is measured.
• Evaluate with BatchEvalRunner on a fixed question set before deploying and after every change to chunking or retrieval; subjective quality assessment does not scale.
• Use IngestionPipeline with a docstore for incremental updates; do not re-embed unchanged documents. Persist both the pipeline state and the vector store.
• Documents and questions are sent to the configured LLM and embedding provider. For data that must stay on the machine, set local models on Settings before building any index, and re-embed after changing the embedding model — vectors from different models are not comparable.
• The maintainers now focus on the hosted LlamaParse platform and keep the framework as an open toolkit. For hard PDFs (scans, tables) the built-in readers are basic; for a single prompt over a small file, call the model API directly instead of building an index.