executing-plans
executing-plans
Use when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available
スキルを探す
生産性・共同作業 · SKILL.md
flan_heldout_instruction_evaluator
flan_heldout_instruction_evaluatorEvaluate held-out direct and CoT instruction examples, compute accuracy delta, and emit FLAN recovery mechanism checks.
Independent GitHub snapshot · Automated classification · Original documentation governs
スキルの説明
GitHubname: flan_heldout_instruction_evaluator description: Evaluate held-out direct and CoT instruction examples, compute accuracy delta, and emit FLAN recovery mechanism checks.
FLAN Held-Out Instruction Evaluator
Use this skill after an instruction-finetuning run to measure held-out generalization.
Inputs
- Held-out examples and predictions before/after finetuning.
- Mixture audit, training trace, and target metadata.
Outputs
- Accuracy before/after/delta and mechanism checks.
Workflow
- Check held-out ids are absent from training ids.
- Compute exact-match accuracy before and after.
- Emit mechanism checks for CoT, optimizer, and target matching.
Validation
Run python tests/test_evaluator.py.
Limitations
Supports deterministic proxy metrics, not full benchmark evaluation.