---

name: evaluation-tools description: "Use when evaluating RobustVideoMatting alpha/foreground predictions, preparing LR or HR metric directories, generating evaluation composites, or interpreting speed benchmarks." disable-model-invocation: true metadata: disco-role: operating license: GPL 3.0

RobustVideoMatting Evaluation Tools

Use this sub-skill when the task is about measuring RVM predictions, preparing prediction/ground-truth trees, understanding LR versus HR metrics, or explaining published speed numbers.

Read this when

• The user asks to compute pha_mad, pha_mse, pha_grad, pha_conn, pha_dtssd, fgr_mad, or fgr_mse.
• The task mentions evaluate_lr.py, evaluate_hr.py, videomatte_512x288, videomatte_1920x1080, or Excel metric output.
• The user needs to generate synthetic evaluation composites from matte and background datasets.
• The user asks why converter FPS differs from the README speed table.

Route other tasks elsewhere:

• Producing predictions from a video or frame directory: inference-workflows.
• Dataset acquisition/training data layouts: training-data.
• Model forward internals: model-api.

Evaluation workflow

1. Ensure predictions and ground truth use the exact same dataset/clip/frame tree. The evaluator expects alpha under pha/ and foreground under fgr/ when foreground metrics are selected.

2. Choose LR or HR flow:

   • LR evaluation is CPU/NumPy/OpenCV oriented and includes pha_conn.
   • HR evaluation hardcodes CUDA tensors and Kornia for high-resolution gradient metrics.

3. For a safe tiny assertion check, use the bundled JSON evaluator:

   python scripts/rvm_evaluate_lr_tiny.py \
     --pred-dir pred/videomatte_512x288 \
     --true-dir true/videomatte_512x288 \
     --metrics pha_mad pha_mse pha_dtssd fgr_mad fgr_mse
   

4. For full repo-style evaluation, mirror the metric names and directory constraints in references/evaluation-reference.md.

5. If the user asks for speed, separate tensor throughput from media IO. The README speed table is based on a CUDA tensor loop, not the Python converter's full decode/encode pipeline.

Bundled references and script

• Read references/evaluation-reference.md for LR/HR metrics, directory schemas, compositing scripts, and speed-test caveats.
• Read references/troubleshooting.md for empty result directories, frame mismatches, missing dependencies, HR CUDA-only behavior, deprecated NumPy aliases, and foreground-mask issues.
• Run scripts/rvm_evaluate_lr_tiny.py for a self-contained CPU JSON summary on small prediction/ground-truth folders. It intentionally does not depend on the original checkout.

Key decisions

• Use LR/tiny evaluation for quick validation of directory matching and metric plumbing.
• Use HR evaluation only in a CUDA environment with Kornia and matching high-resolution data; it is not a CPU fallback.
• Do not run synthetic composite generation as a default check. Those scripts require large external matte/background datasets and can create many files.
• If foreground metrics are requested, require matching fgr/ frames and a non-empty alpha mask.

Acceptance check for evaluation answers

A good answer states the exact directory shape, names the metrics and runtime requirements, checks frame-name equality, chooses LR/HR appropriately, and explains whether any skipped compositing or CUDA benchmark is outside the safe verification scope.