name: benchmark-design
description: Use when designing an evaluation plan for a CS, ML, or agent project and you need to choose datasets, baselines, metrics, ablations, compute budgets, and reporting rules.
Purpose
Design a benchmark that is fair, reproducible, and aligned with the paper or project claim.
When To Use
- defining a new evaluation protocol for a paper or prototype
- choosing baselines, metrics, and data splits
- planning ablations and error analysis before running experiments
- checking whether reported claims are actually testable
Inputs
- task definition and target claims
- candidate datasets, models, or systems
- compute budget, runtime limits, and hardware constraints
- prior papers or existing benchmark conventions if available
Outputs
- benchmark matrix of tasks, datasets, metrics, and baselines
- evaluation protocol and run rules
- ablation plan and error-analysis checklist
- reporting template for tables and plots
- risk notes about confounders or unfair comparisons
Workflow
- Translate the project claim into measurable questions.
- Select datasets and splits that match the intended claim.
- Define baseline tiers: trivial, standard, and strong reference systems.
- Choose primary and secondary metrics, then add ablations and sanity checks.
- Record compute, prompt, seed, and implementation fairness rules.
Constraints
- do not compare systems under hidden budget mismatches
- avoid metric choices that only flatter the proposed method
- separate headline metrics from diagnostic metrics
- make ablations answer specific causal questions, not random variations