evolve-skill-from-tracesEvolve SKILL.md files from agent execution traces using a three-stage pipeline: trajectory collection from observed runs, parallel multi-agent patch proposal for error and success analysis, and conflict-free consolidation of overlapping edits via prevalence-weighting. Based on the Trace2Skill methodology.
Independent GitHub snapshot · Automated classification · Original documentation governs
Raw traces → validated SKILL.md via 3-stage pipeline: trajectory collect, parallel multi-agent patch proposal, conflict-free consolidation. Bridges observed behavior + documented procedures. Successful runs → reproducible skills.
traces — agent logs/transcripts (min 10 successful recommended)target_skill — path to existing SKILL.md or "new" for from-scratchanalyst_count — parallel analysts (default: 4)held_out_ratio — fraction reserved for validation (default: 0.2)Gather logs/tool-call sequences/transcripts. Filter successful. Normalize → (state, action, outcome) triples w/ timestamps.
trace_entry:
state: <context before the action>
action: <tool call, command, or decision made>
outcome: <result, output, or state change>
timestamp: <ISO 8601>
held_out_ratio (default 20%) for Step 7 validation, rest for Steps 2-6# Example: count available traces and compute partition
total_traces=$(ls traces/*.json | wc -l)
held_out=$(echo "$total_traces * 0.2 / 1" | bc)
drafting=$((total_traces - held_out))
echo "Drafting: $drafting traces, Held-out: $held_out traces"
→ Normalized traces partitioned drafting (80%) + held-out (20%). Each entry has state, action, outcome, timestamp.
If err: <10 successful → collect more. Small sets → overfit → fail on novel. No timestamps → ordinal sequence.
Group by outcome pattern. Identify invariant core (all successful) vs variant branches (diff across runs). Invariant core = skeleton.
invariant_core:
- action: "read_input_file"
frequency: 100%
- action: "validate_schema"
frequency: 100%
- action: "transform_data"
frequency: 100%
variant_branches:
- action: "retry_on_timeout"
frequency: 35%
condition: "network latency > 2s"
- action: "fallback_to_cache"
frequency: 15%
condition: "API returns 503"
→ Clear separation invariant (all) vs variant (subset). Each variant has frequency + condition.
If err: no invariant emerges (heterogeneous) → target may be multiple distinct skills. Split by outcome type, process each.
Invariant core → initial SKILL.md w/ frontmatter, When to Use (entry conditions), Inputs (varied params), Procedure (1 step per invariant action).
# Scaffold the skeleton if creating a new skill
mkdir -p skills/<skill-name>/
# Skeleton structure
## When to Use
- <derived from common entry conditions>
## Inputs
- **Required**: <parameters present in all traces>
- **Optional**: <parameters present in some traces>
## Procedure
### Step N: <invariant action label>
<most common implementation from traces>
**Expected:** <most common success outcome>
**On failure:** <placeholder -- refined in Steps 4-6>
→ Valid SKILL.md skeleton w/ frontmatter, When to Use, Inputs, Procedure (1 step per invariant). Expected = observed, On failure = placeholder.
If err: skeleton >500 lines pre-variants → invariant too granular. Merge adjacent actions always together. Target 5-10 steps.
Spawn N analysts (4-6), each reviews traces vs skeleton from diff lens. Each → structured patch: section, old, new, rationale.
Lens per analyst:
| Analyst | Lens | Focus |
|---|---|---|
| 1 | Correctness | All success paths captured? Any invariant missing? |
| 2 | Efficiency | Redundant steps? Merge/parallelize? |
| 3 | Robustness | Failure modes unhandled? On failure content? |
| 4 | Edge Cases | Variants become conditional steps or pitfalls? |
| 5 (optional) | Clarity | Each step unambiguous? Mechanically followable? |
| 6 (optional) | Generalizability | Trace-specific artifacts should abstract? |
Each analyst gets:
Each → structured patches:
patch:
analyst: "robustness"
section: "Procedure > Step 3"
old_text: "**On failure:** <placeholder>"
new_text: "**On failure:** If the API returns 503, wait 5 seconds and retry up to 3 times. If retries are exhausted, fall back to the cached response from the previous successful run."
rationale: "Traces #4, #7, #12 show 503 errors resolved by retry. Trace #15 shows cache fallback when retries fail."
supporting_traces: [4, 7, 12, 15]
→ Each analyst returns 3-10 patches w/ section, old/new, rationale, trace IDs. All collected into single set.
If err: analyst returns no patches → lens may not apply. OK — not every lens surfaces issues. Vague patches no trace refs → reject + re-prompt w/ supporting_traces req.
Compare all patches for overlapping edits. Classify each pair.
| Conflict | Def | Resolution |
|---|---|---|
| Compatible | Diff sections, no overlap | Merge directly |
| Complementary | Same section, additive (both add, no contradiction) | Combine text |
| Contradictory | Same section, mutually exclusive (add X, remove X or add Y instead) | Step 6 resolution |
conflict_report:
total_patches: 24
compatible: 18
complementary: 4
contradictory: 2
contradictions:
- section: "Procedure > Step 5"
patch_a: {analyst: "efficiency", action: "remove step"}
patch_b: {analyst: "robustness", action: "add retry logic"}
supporting_traces_a: [2, 8, 11]
supporting_traces_b: [4, 7, 12, 15]
→ Conflict report: all pairs classified, contradictions have supporting trace counts.
If err: ambiguous (patch adds + modifies same section) → split into 2 (additive + modifying). Re-classify.
Merge all → single SKILL.md via 3-tier resolution.
argumentation skill to eval which better serves purposeconsolidation_log:
applied_directly: 18
combined: 4
resolved_by_prevalence: 1
resolved_by_argumentation: 1
rejected_alternatives_documented: 2
Post-consolidation verify:
→ Single consolidated SKILL.md incorporating all analysts. Contradictions resolved w/ rationale. Rejected alts → pitfall/note.
If err: internally inconsistent (Step 3 assumes file exists but Step 2 removed by efficiency) → revert conflicting edit, keep original skeleton for section. Flag for manual review.
Run consolidated against held-out (20% Step 1). Verify Expected/On failure match observed in unseen traces.
validation_results:
held_out_traces: 5
full_match: 4
partial_match: 1
no_match: 0
mismatches:
- trace_id: 23
step: 4
expected: "API returns 200"
actual: "API returns 429 (rate limited)"
action: "Add rate-limit handling to On failure block"
create-skill for dir, registry, symlinksevolve-skill for version + translation sync# Final validation: line count
lines=$(wc -l < skills/<skill-name>/SKILL.md)
[ "$lines" -le 500 ] && echo "OK ($lines lines)" || echo "FAIL: $lines lines > 500"
→ ≥80% held-out match end-to-end. Registered in skills/_registry.yml w/ correct metadata.
If err: >20% mismatch → overfit to drafting. Add mismatched to drafting, re-run Step 2. Still fail after 2 iterations → too variable for single skill → split by outcome type.
evolve-skill — simpler human-directed (complementary: no traces)create-skill — newly extracted from scratch; Step 7 registrationreview-skill-format — validation post-consolidation → agentskills.ioargumentation — Step 6 resolve contradictions when prevalence tiedverify-agent-output — evidence trails for patches; validates analyst outputs Step 4