---

name: evaluation description: "Select, prepare, and interpret ECCV2022-RIFE benchmark and evaluation workflows." disable-model-invocation: true metadata: disco-role: operating license: MIT

ECCV2022-RIFE evaluation sub-skill

Use this sub-skill when the task is about RIFE benchmarks, reported evaluation metrics, benchmark dataset layout, throughput smoke checks, or deciding which evaluation case is safe to run.

Natural triggers include: "run RIFE benchmarks", "UCF101 evaluation", "Vimeo90K evaluation", "MiddleBury IE", "ATD12K", "HD benchmark", "HD 4X", "PSNR/SSIM/IE", "testtime smoke", "which benchmark is safe", and "dataset layout".

Route away

• Image-pair, video, PNG-sequence, --ratio, --exp, --fps, --scale, --fp16, --montage, or output generation questions belong to the interpolation sub-skill.
• Vimeo triplet training, distributed launch, TensorBoard logs, checkpoint production, or reproduction training belongs to the training sub-skill.
• Do not download benchmark datasets or checkpoints automatically. Official metrics require user-provided external assets.

Core workflow

1. Classify the request. Decide whether the user needs benchmark selection, layout validation, command construction, metric interpretation, or verification skip classification.

2. Gate assets before commands. Full UCF101, Vimeo90K, MiddleBury, ATD12K, HD, and HD 4X evaluations require external datasets plus checkpoints. The benchmark scripts expect checkpoint directories containing flownet.pkl.

3. Gate backend criticality.

   • benchmark/testtime.py is the only no-dataset/no-checkpoint source benchmark candidate. It times random 480x640 tensor inference and is not an official quality metric.
   • UCF101, Vimeo90K, MiddleBury, and ATD12K use torch.device("cuda" if available else "cpu"); CPU is functionally possible but may be slow.
   • benchmark/HD.py and benchmark/HD_multi_4X.py contain explicit .cuda() calls and have no CPU substitute. Treat them as required-CUDA cases.

4. Validate layout safely. Use the bundled validator before running any source benchmark:

   python scripts/check_benchmark_layout.py --repo-root <checkout> --benchmark <name>
   

   The validator checks expected source scripts, dataset files, checkpoint files, and HD YUV sizes. It performs no downloads and does not execute benchmarks.

5. Use benchmark commands only after gates pass. The command, metric, dataset, checkpoint, dependency, backend, and skip rationale matrix is in references/benchmarks.md.

6. Interpret signals carefully. PSNR and SSIM are higher-is-better image fidelity metrics; MiddleBury IE is lower-is-better interpolation error; testtime.py prints seconds per inference and says nothing about checkpoint quality.

7. Troubleshoot by symptom. Use references/troubleshooting.md for missing files, checkpoint load errors, CUDA-only HD failures, scikit-image/YUV issues, slow CPU runs, shape/padding failures, and metric interpretation traps.

Verification skip decisions

Use these labels consistently when planning final verification:

• safe smoke candidate: benchmark/testtime.py when repo dependencies are installed and the runtime budget can tolerate 200 random 480x640 inferences. Prefer CUDA for speed; on CPU, mark SKIP_EXPENSIVE if the budget is tight.
• SKIP_DATA: any full benchmark whose dataset directory or checkpoint file is absent, or whose setup would require network downloads.
• SKIP_EXPENSIVE: full dataset benchmarks when assets exist but running all samples is outside the approved time/GPU budget; also use for CPU-only throughput timing if too slow.
• required-CUDA: HD and HD 4X benchmarks if CUDA is unavailable, even if CPU imports work.
• not verified: never claim README-reported official metrics were reproduced unless the matching dataset, checkpoint, backend, and command were actually run later.