---

name: task-authoring description: "Create, validate, and debug lm-evaluation-harness YAML tasks, task groups, filters, metrics, Jinja prompts, includes, and external task directories." disable-model-invocation: true metadata: disco-role: operating license: MIT

Task Authoring

Use this sub-skill when an agent needs to add or repair lm-evaluation-harness task definitions. It covers YAML TaskConfig and group config authoring, !function helpers, few-shot settings, local/custom datasets, filters, metrics, and static validation. Route full evaluation execution to ../evaluation-runs/, model/backend setup to ../model-backends/, and result interpretation to ../result-logging/.

Start Here

1. Identify whether the user is creating a leaf task, a Python-backed task, a group, or a tag-based collection.
2. Draft the YAML from references/task-schema.md and prefer a small local fixture while iterating.
3. Use scripts/create_task_skeleton.py for a self-contained external task directory when starting from scratch.
4. Run scripts/lint_task_yaml.py before any command that downloads datasets or imports arbitrary !function code.
5. Hand off runtime validation with lm-eval validate --tasks TASK --include_path DIR; run only after confirming dataset access and any unsafe/custom code implications.

Safe Workflow

• Static checks first: parse YAML, resolve include structure, inspect keys, metrics, filters, template strings, !function references, and common YAML quoting mistakes.
• Runtime validation second: lm-eval validate confirms task discovery for the installed package; the command advertises broader validation concerns, but version 0.4.13.dev0 primarily checks that requested tasks/groups/tags resolve through TaskManager.
• Full evaluation last: only use evaluation commands after routing to the evaluation-runs sub-skill and confirming model/backend cost, dataset download, and cache behavior.

References

• references/task-schema.md — YAML fields, groups/tags, includes, !function, local datasets, few-shot config.
• references/task-workflows.md — creation, external task package, validation, debugging, and handoff workflows.
• references/filters-and-metrics.md — registered filters, metric/output-type compatibility, and pipeline shape rules.
• references/troubleshooting.md — common failures for YAML quoting, includes, splits, filters, metrics, and unsafe code.

Bundled Scripts

• scripts/create_task_skeleton.py creates a minimal external task directory with YAML, utils.py, and optional tiny JSON fixture.
• scripts/lint_task_yaml.py performs safe static checks without importing task functions or downloading datasets.

Boundaries

• Do not copy or summarize the full upstream task catalog; use only task names/examples needed for the user's task.
• Do not run full evaluations from this sub-skill; produce the task and validation handoff.
• Do not implement model backends or analyze benchmark scores here.
• Treat Python !function, Python task classes, local loading scripts, and unsafe_code: true as code execution surfaces that require explicit confirmation before runtime execution.