---

name: evaluation description: "Guide VLA-Adapter LIBERO and CALVIN evaluation, command generation, outputs, prerequisites, metrics, and benchmark cautions." disable-model-invocation: true metadata: disco-role: operating license: MIT

Evaluation (external-checkout adapter)

This sub-skill renders plans for native LIBERO/CALVIN evaluators in a separate checkout. It contains only documentation and a command renderer: it does not contain experiments/robot/ or vla-scripts/, does not run an evaluation, and is not a self-contained benchmark runtime.

Set the absolute source root and enter it before any native workflow:

export VLA_ADAPTER_REPO_ROOT=/abs/path/to/VLA-Adapter
cd "$VLA_ADAPTER_REPO_ROOT"
python -m pip install -e "$VLA_ADAPTER_REPO_ROOT"

Native entrypoints are experiments/robot/libero/run_libero_eval.py and vla-scripts/evaluate_calvin.py. The bundled builder only prints commands beginning with cd <absolute-repo-root> &&; it never imports the model or launches a rollout. External prerequisites are CUDA-capable PyTorch, the base VLA-Adapter package, a complete compatible checkpoint, the selected LIBERO or CALVIN checkout and assets/data/config layout, and a working EGL/OpenGL/MuJoCo or pybullet renderer. Video runs may additionally require imageio/ffmpeg.

Read first

• Read references/evaluation-workflows.md for checkpoint-to-command workflows, expected outputs, and benchmark-specific runtime behavior.
• Read references/benchmarks.md for suite names, checkpoint names, default evaluation volumes, and published LIBERO/CALVIN metrics.
• Read references/troubleshooting.md for dependency, renderer, checkpoint, unnormalization-key, log, and video failures.
• Run python "$VLA_ADAPTER_SKILL_ROOT/sub-skills/evaluation/scripts/build_eval_command.py" --repo-root "$VLA_ADAPTER_REPO_ROOT" --help to inspect the non-executing command renderer.

Operating route

1. Identify the benchmark: use LIBERO for libero_spatial, libero_object, libero_goal, or libero_10; use CALVIN for calvin_abc.
2. Confirm the checkpoint family and suite match. Prefer the Pro checkpoints for published reproduction unless the user explicitly asks for the original release or a custom checkpoint.
3. Confirm downstream prerequisites before launching: CUDA-capable PyTorch, the base VLA-Adapter package dependencies, the external benchmark stack, benchmark assets, and the checkpoint files.
4. Generate the command with the bundled builder, then review the resulting CUDA_VISIBLE_DEVICES=... python ... invocation before execution.
5. Capture stdout/stderr into an eval_logs/ file for long runs. LIBERO also writes internal text logs and rollout videos; CALVIN writes evaluation result files and per-subtask videos.
6. Interpret only full-volume runs against published metrics: LIBERO defaults to 500 episodes per suite, while CALVIN reports average successful sequence length across 1,000 five-instruction sequences.

Critical cautions

• LIBERO evaluation depends on the repository's image preprocessing: both third-person and wrist camera images are rotated 180 degrees before policy input.
• LIBERO/OpenVLA action execution normalizes the gripper action to [-1, +1], binarizes it, then inverts the OpenVLA gripper sign for environment execution.
• CALVIN reports average sequence length, not a single binary episode success rate; also inspect chain success rates for 1 through 5 consecutive instructions.
• External LIBERO and CALVIN stacks were not part of the base package import surface. Treat them as downstream prerequisites, not guaranteed installed dependencies.