---

name: evaluation description: "Draft sam eval suites, test cases, scoring configs, and offline preflight checks." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Evaluation

Use this sub-skill to configure and review sam eval runs, score test suites, interpret result trees, and validate inputs before live evaluation.

Covers

• Local and remote evaluation mode selection
• Test suite and test case input shaping
• Scoring and result interpretation
• Offline validation of file references and environment-variable references

Start here

• Evaluation workflows
• Data formats
• Troubleshooting
• Offline validator

Excludes

• Runtime task submission or agent execution, which belongs in runtime-operations
• Workflow YAML authoring, which belongs in workflow-authoring
• Project scaffolding or bootstrap flows, which belongs in project-bootstrap

Notes

• The current live loader expects JSON syntax for sam eval; the bundled validator accepts JSON or YAML for preflight only.
• Keep results_dir_name explicit so result trees are predictable and easy to compare.
• Use the bundled validator first when a suite looks wrong, missing, or mixed between local and remote modes.