---

name: data-ingestion description: "Load local files, directories, and websites into DeepSearcher collections while validating inputs, loader choices, chunking, and vector database implications." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Data Ingestion

Use this sub-skill when the user needs to load documents or web pages into a DeepSearcher collection, choose an ingestion loader or crawler, tune chunking and batching, prevent collection overwrite mistakes, or validate ingestion inputs before indexing.

Prefer DeepSearcher's Python APIs for ingestion workflows:

• Local files and directories: deepsearcher.offline_loading.load_from_local_files(...)
• Websites and web-readable documents: deepsearcher.offline_loading.load_from_website(...)
• Configuration: deepsearcher.configuration.Configuration and deepsearcher.configuration.init_config(...)

For command-line deepsearcher load ... syntax, route to cli-and-service. For provider credentials and model/provider selection, route to provider-configuration. For querying or retrieval after ingestion, route to rag-query. For evaluation dataset loading, route to evaluation.

Fast Routing

User intent	Read
Load PDFs, Markdown, text, JSON/JSONL, or a directory	references/local-loading.md
Crawl or scrape websites into a collection	references/web-loading.md
Pick PDF/Text/JSON/Unstructured/Docling loaders or tune chunks	references/data-formats-and-chunking.md
Diagnose missing files, empty chunks, FireCrawl key errors, Milvus Lite locks, or version incompatibilities	references/troubleshooting.md
Validate paths, URLs, chunk parameters, and collection-name normalization without indexing	scripts/validate_ingestion_inputs.py

Operating Checklist

1. Confirm the ingestion source type: local path(s), directory, URL(s), or a mix that should be split into separate local and web ingestion calls.
2. Confirm the intended collection name and overwrite policy. For local loading, names are normalized by replacing spaces and hyphens with underscores; set force_new_collection=True only when the existing collection should be dropped.
3. Choose the configured file loader or web crawler before calling init_config(config). Do not initialize configuration just to validate user input.
4. Validate local paths, URL shape, chunk parameters, and optional JSON text key with the bundled helper when practical. The helper performs no indexing, no network calls, and no provider initialization.
5. Initialize DeepSearcher only after credentials, optional dependencies, and vector DB readiness are accounted for.
6. Load data, then route post-load search, retrieval, answer generation, and token accounting to rag-query.

Minimal API Pattern

from deepsearcher.configuration import Configuration, init_config
from deepsearcher.offline_loading import load_from_local_files

config = Configuration()
config.set_provider_config("file_loader", "PDFLoader", {})
config.set_provider_config("vector_db", "Milvus", {"uri": "./milvus.db", "token": ""})
init_config(config)

load_from_local_files(
    paths_or_directory=["docs/guide.pdf", "notes/summary.md", "notes/plain.txt"],
    collection_name="project_knowledge",
    collection_description="Project PDFs, Markdown, and text notes",
    force_new_collection=False,
    chunk_size=1500,
    chunk_overlap=100,
    batch_size=256,
)

Treat credentialed, networked, remote-service, and expensive ingestion as optional unless the user explicitly asks to run it and the required provider configuration is ready.