---

name: evaluation description: "Plan, validate, and safely run DeepSearcher retrieval evaluations, especially 2WikiMultiHopQA recall workflows, with explicit cost, resume, and corpus-loading controls." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Evaluation

Use this sub-skill for retrieval-quality evaluation with DeepSearcher. The supported benchmark workflow is 2WikiMultiHopQA, comparing DeepSearcher retrieval with the naive RAG baseline using Recall@2 and Recall@5.

Route here

• Validate a 2Wiki-style corpus/questions pair before spending embedding or LLM tokens.
• Plan a benchmark run with dataset, config_yaml, pre_num, max_iter, output_dir, skip_load, and flag.
• Interpret details.csv, statistics.json, retrieved-title metadata, recall@K, token usage, and resumable partial output.
• Resume an interrupted run without reloading a compatible, already-populated vector collection.

Route elsewhere

• Provider names, credentials, YAML provider dictionaries, optional SDKs, or version selection: use ../provider-configuration/SKILL.md.
• General local/website indexing, collection naming, chunking, or loader behavior: use ../data-ingestion/SKILL.md.
• query, retrieve, naive_retrieve, RAG agent selection, or iteration behavior outside the benchmark: use ../rag-query/SKILL.md.
• The deepsearcher console command or HTTP service: use ../cli-and-service/SKILL.md.

Start safely

1. Run the no-network validator before initialization or indexing. It checks the dataset name, both JSON shapes, required YAML feature sections, and the output path without importing DeepSearcher or invoking providers:

   python scripts/check_evaluation_inputs.py \
     --dataset 2wikimultihopqa \
     --corpus path/to/2wikimultihopqa_corpus.json \
     --questions path/to/2wikimultihopqa.json \
     --config-yaml path/to/eval_config.yaml \
     --output-dir path/to/eval-output
   

   Add --create-output-dir only when it is safe to create the planned output directories. See data formats for custom-fixture requirements.

2. Confirm that config_yaml contains all five provide_settings feature sections and that the file loader can read the corpus (JsonFileLoader with text_key: text for the standard corpus). Check provider credentials and vector DB readiness separately; the validator intentionally does not test them.

3. Budget the run before launching it. A full run initializes an LLM, embedding model, loader/crawler providers, and vector DB, embeds the corpus on the first load, and invokes the retrieval pipeline for each evaluated question. More samples improve stability; higher max_iter generally increases token use and elapsed time. Use a small positive pre_num first, record the provider/model and configuration, and expand only after the smoke run succeeds.

4. Invoke the available evaluation runner using its public argument contract. The source-independent command shape is:

   python path/to/evaluation_runner.py \
     --dataset 2wikimultihopqa \
     --config_yaml path/to/eval_config.yaml \
     --pre_num 5 \
     --max_iter 3 \
     --output_dir path/to/eval-output \
     --flag result
   

   The full workflow is optional and credentialed/expensive. Do not treat a validator pass as proof that providers, embeddings, Milvus, or the benchmark runner are operational.

5. On the first run, allow corpus loading into a fresh or intentionally replaced collection. The benchmark loader uses the pre-chunked corpus as one large chunk per record (chunk_size=999999, chunk_overlap=0) and replaces the target collection in its normal first-run path. Treat that replacement as destructive; isolate the evaluation collection or vector DB from useful data.

6. After a successful load, resume or run another sample range with --skip_load only when the same compatible collection is still available: same vector DB endpoint/URI, collection, embedding dimension/model, and corpus. skip_load skips indexing; it does not verify that the collection exists or matches the files. Do not use it after changing embeddings, vector DB, collection, corpus, or an isolated working directory.

7. Use a distinct flag for logically separate reports. Results are written below output_dir/<flag>/; a flag separates report files, not vector collections. Avoid concurrent runs against the same local Milvus Lite database because the cwd-relative database can lock.

Outputs and handoff

A completed or partial run writes:

• details.csv: one row per processed question, including the index, question, DeepSearcher and naive-RAG recall dictionaries, gold titles, and retrieved titles.
• statistics.json: aggregate Recall@2/Recall@5 for both systems, DeepSearcher token usage, error count, sample count, and token usage per sample.

The runner resumes from the number of existing CSV rows and combines saved statistics with newly processed rows. Preserve details.csv and statistics.json as a pair; back them up before manual edits. Inspect both files and record the dataset/config/model, pre_num, max_iter, flag, whether loading was skipped, and any pipeline errors when reporting results.

References and helper

• Evaluation workflow — parameter meanings, safe phases, resume procedure, and output lifecycle.
• Data formats — accepted JSON/YAML contracts and Recall@K calculation.
• Troubleshooting — validation, compatibility, cost, vector DB, and resume failures.
• scripts/check_evaluation_inputs.py — deterministic, no-network input checker with --help and optional output-directory creation.