---

name: evaluation-reproduction description: "Use PixelRAG evaluation scripts and paper reproduction harnesses safely, including reader/search serve preflights and graders." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

PixelRAG Evaluation and Reproduction

Use this sub-skill when the task asks to reproduce PixelRAG paper results, run eval/ benchmark cells, debug reproduce.sh, configure reader/search serves, compare retrieval modes, or understand grader/output metadata.

Start Here

1. Decide whether the user wants a quick pipeline smoke or a paper-matching run.
   • Quick smoke can use a public search API or a tiny self-hosted index.
   • Paper Table 1 reproduction requires specific search indexes, reader model, large tile/index data, and grader credentials.
2. Validate services before running a benchmark. Use pixelrag_eval_preflight.py or reproduce the shell preflight manually.
3. Confirm credentials:
   • OpenAI key/base URL for LLM judge on NQ/NQ-Tables and some VQA grading paths.
   • Optional MiniMax key for MiniMax reader experiments.
4. Record the run metadata stamped by the harness: dataset/split/n, reader, retrieval URL/status, top-k, instruction, and grader.

Read or Run

• Read paper-reproduction.md for reproduce.sh, ports, roles, expected resources, and public-API caveats.
• Read eval-api-reference.md for run_bench.py, retrieval flags, model config, output files, and graders.
• Read troubleshooting.md for empty retrieval, wrong index, judge failures, query-image path issues, and slow on-demand rendering.
• Run pixelrag_eval_preflight.py before an expensive eval.

Common Routes

Request	Action
"Reproduce Table 1 NQ base"	Check reader Qwen/Qwen3.5-4B, base pixel serve vector count, OpenAI judge, then run the nq/base cell.
"Try a public API smoke"	Use direct run_bench.py --local-api-url mode; warn that public base endpoint may not match normed paper index.
"Why is score near zero?"	Check preflight, retrieval timeout, grader key/base URL, reader model, and whether closed-book fallback occurred.
"Run MiniMax reader"	Use model config rules and set MiniMax endpoint/key/context length explicitly.
"Need full training checkpoint eval"	Route to ../training-and-data/SKILL.md for training outputs, then return here for eval harness.

Boundaries

• Do not download hundreds of GB of indexes or tiles unless the user explicitly approves.
• Do not launch vLLM, search serves, or long benchmark jobs silently.
• Do not treat strict exact-match NQ/NQ-Tables scores as paper numbers; the paper used an LLM judge for those cells.
• Keep benchmark outputs and reports outside the runtime skill tree.