---

name: llm-distillation description: > Distill knowledge from large teacher LLMs into small student models via sequence-level (response-based) knowledge distillation. Covers teacher selection and licensing checks, generating distillation data with Bedrock serverless teachers (DeepSeek-R1, Claude via converse), reasoning distillation (keeping /chain-of-thought for math/code/puzzle domains, stripping it for simple QA), sampling and temperature choices, batch generation with retry/backoff, cost estimation, data curation (dedup, decontamination, verifiable-reward filtering, LLM-judge filtering), producing TRL-ready JSONL for SFT/QLoRA, and evaluating distillation quality with student-vs-teacher relative gates. Use when creating a small specialized model from a large model's outputs, building R1-style distilled reasoners, or preparing teacher-generated training data. license: Apache-2.0 metadata: author: llmops-skills version: "1.0" category: llmops

LLM Distillation

Overview

Sequence-level knowledge distillation (KD) trains a small student model on the outputs of a large teacher model: generate responses with the teacher, curate them, then fine-tune the student on the curated pairs with plain SFT. No logits, no special loss — the entire "distillation" lives in the data. This is how the official DeepSeek-R1-Distill models were made, and it is the cheapest path to a small model that inherits a large model's behavior on a narrow domain.

This skill owns the data side: teacher generation, curation, and quality gates. The training job itself (QLoRA/SFT on SageMaker or local GPU) belongs to llm-fine-tuning — this skill's deliverable is a curated JSONL that skill can consume directly.

When to Use This Skill

• Creating a small, cheap, self-hostable model that mimics a large teacher on a specific domain
• Building an R1-style distilled reasoner for math, code, or structured-puzzle tasks
• Generating teacher-labeled training data when human labels are unavailable
• Cutting inference cost by replacing serverless frontier-model calls with a distilled student
• Deciding whether to keep or strip chain-of-thought in distillation targets

Distillation Pipeline

Select Teacher → Check License → Generate → Curate → Format → [llm-fine-tuning] → Evaluate
      │              │              │          │        │            │               │
  Reasoning vs    MIT/Apache OK  Bedrock    Dedup    TRL        QLoRA SFT      Student vs
  non-reasoning   ToS may forbid converse   Decontam messages   (out of        teacher gates
  teacher         distillation   + retry    Verify   JSONL      scope here)    (RELATIVE)
                                 + backoff  Judge

Step-by-Step Instructions

1. Teacher Selection and License Check

Pick the teacher by domain, then verify you are allowed to distill from it before generating a single token:

Teacher (Bedrock serverless)	Model / profile ID	Distillation allowed?
DeepSeek-R1 (reasoning)	us.deepseek.r1-v1:0	Yes — MIT license explicitly permits distillation
Claude (general)	global.anthropic.claude-sonnet-5	Check current Anthropic ToS — historically restricts training competing models
Llama 3.x/4 (general)	us.meta.llama3-3-70b-instruct-v1:0	Yes with attribution — community license requires "Built with Llama" naming for derivatives

Rule of thumb: open-weight models with MIT/Apache-2.0 (DeepSeek, Qwen) are safe teachers; proprietary API models often forbid using outputs to train models — read the provider ToS, not just the marketing page. Record the teacher ID and license decision in the dataset card.

# Confirm the inference profile is available in your region (region-agnostic: uses AWS_REGION)
aws bedrock list-inference-profiles \
  --query "inferenceProfileSummaries[?contains(inferenceProfileId, 'deepseek')].inferenceProfileId"

2. Teacher Generation via Bedrock Converse

import time
import boto3
from botocore.config import Config

# Region comes from AWS_REGION/profile - never hardcode
bedrock = boto3.client("bedrock-runtime",
                       config=Config(retries={"max_attempts": 0}))  # we own retry policy

TEACHER_ID = "us.deepseek.r1-v1:0"  # cross-region inference profile

def generate_teacher_response(prompt, max_retries=6):
    """One teacher call with exponential backoff on throttling."""
    for attempt in range(max_retries):
        try:
            resp = bedrock.converse(
                modelId=TEACHER_ID,
                messages=[{"role": "user", "content": [{"text": prompt}]}],
                inferenceConfig={
                    "maxTokens": 8192,      # reasoning traces are LONG; don't truncate mid-think
                    "temperature": 0.6,     # R1 recommended range 0.5-0.7; 0 causes repetition loops
                    "topP": 0.95,
                },
            )
            blocks = resp["output"]["message"]["content"]
            # R1 via Converse returns reasoning in a reasoningContent block + answer in text
            reasoning = next((b["reasoningContent"]["reasoningText"]["text"]
                              for b in blocks if "reasoningContent" in b), "")
            answer = next((b["text"] for b in blocks if "text" in b), "")
            return {"reasoning": reasoning, "answer": answer,
                    "usage": resp["usage"], "stop": resp["stopReason"]}
        except bedrock.exceptions.ThrottlingException:
            time.sleep(min(2 ** attempt, 60))   # 1,2,4,...60s
    raise RuntimeError(f"Throttled {max_retries} times - lower concurrency")

Generation guidance:

• Prompt format: give the teacher the exact task format the student will see at inference (same system prompt, same answer-format instruction, e.g. "put the final answer in \boxed{}"). Distillation copies the whole behavior, format included.
• Temperature: 0.6-0.7 for reasoning teachers (R1 loops at temperature 0); 0.2-0.3 for deterministic QA/summarization teachers. Generate 2-4 samples per prompt when you have ground truth to filter against (rejection sampling) — keep only correct ones.
• stopReason check: drop any sample with stopReason == "max_tokens" — a truncated reasoning chain teaches the student to trail off.
• Checkpoint every N samples to JSONL so a throttling storm or spot interruption never loses completed generations (see scripts/generate_teacher_data.py).

Cost estimation per 1k samples (do this before launching a 100k-sample run):

cost_per_1k = 1000 * (avg_input_tokens * price_in + avg_output_tokens * price_out) / 1e6
# Example: DeepSeek-R1 on Bedrock ~$1.35/M in, ~$5.40/M out (verify current pricing).
# Reasoning outputs average 2k-6k tokens: 1000 * (300*1.35 + 4000*5.40)/1e6 ≈ $22 per 1k samples.
# 3 samples/prompt for rejection sampling triples that. Budget first.

3. Reasoning Mode Decision: Keep or Strip the Chain-of-Thought

The single most important design choice in sequence-level KD:

What is the student's task domain?
├─ Multi-step reasoning (math, code gen, ARC-style grid puzzles, planning)
│  └─ KEEP the teacher's <think>/reasoning in the training target
│     (this is exactly how R1-Distill-Qwen models were trained; the student
│      learns to reason, not just to answer)
│     Target = "<think>\n{reasoning}\n</think>\n\n{answer}"
├─ Simple QA, summarization, classification, extraction
│  └─ STRIP reasoning, train on final answers only
│     (CoT adds latency and tokens at inference with no accuracy gain on
│      tasks the student can do in one step)
└─ Mixed / unsure
   └─ Run a 200-sample pilot both ways, compare student accuracy AND
      inference token cost; reasoning targets typically cost 5-20x more
      output tokens at inference

Two hard constraints when keeping reasoning:

1. Length vs student context: if the teacher's reasoning + answer exceeds the student's training max_length, the sample is truncated and the student learns unterminated rambling. Filter out samples longer than ~90% of the student's context window (in student-tokenizer tokens, not teacher tokens).
2. Consistent format: every training target must use the same reasoning delimiters (<think>...</think>); mixed formats destroy format-validity at inference.

4. Data Curation

Apply in this order (each stage feeds the next — see scripts/curate_distillation_data.py):

1. Format validation: reasoning block present and closed (if reasoning mode), answer non-empty, stopReason was end_turn. Reject malformed samples outright.
2. Deduplication: exact dedup on prompt, then MinHash near-dup at threshold 0.8 on prompt+answer (reuse the machinery from llm-data-preparation).
3. Decontamination vs eval set: run n-gram/MinHash overlap between training prompts and every eval set you will report on. A distilled student that memorized its eval set is the most common fake success in KD.
4. Correctness filtering — prefer verifiable reward:
   • Ground truth exists (math answers, unit tests, puzzle solutions): exact-match / execution-based validation. Deterministic, free, and un-gameable — always beats an LLM judge when available. Keep only samples where the teacher's final answer verifies.
   • No ground truth (open QA, summarization): LLM-judge scoring (1-5 scale, keep >= 4). Use a different model as judge than the teacher — a teacher judging its own outputs inflates scores. Spot-check 50 judge decisions by hand before trusting it at scale.
5. Train/val split: 95/5 split by prompt (never let the same prompt land in both), stratified by task subtype if the domain is heterogeneous.

5. Student Training Handoff (Boundary with llm-fine-tuning)

This skill produces data; llm-fine-tuning owns the training job. The contract is a JSONL in TRL messages format that SFTTrainer consumes without modification:

{"messages": [
  {"role": "user", "content": "Solve: If 3x + 7 = 22, what is x? Put the final answer in \\boxed{}."},
  {"role": "assistant", "content": "<think>\n3x + 7 = 22, so 3x = 15, x = 5.\n</think>\n\n\\boxed{5}"}
]}

Handoff notes for the fine-tuning side:

• KD data is plain SFT data — QLoRA works unchanged (llm-fine-tuning sections 1-3). No KL loss, no teacher logits, no special trainer.
• Set the training max_length to cover the 95th-percentile sample length measured in the student's tokenizer; you already filtered outliers in step 3.
• Reasoning distillation benefits from higher LoRA rank (r=64) than plain style transfer (r=16) — reasoning is a capability, not a formatting tweak.
• 1 epoch is usually right for >= 50k samples; 2-3 epochs for 5-20k. Overfit shows up as the student parroting teacher phrasing on val prompts.

6. Evaluating Distillation Quality

Gates are relative to the teacher, never absolute — a 1.7B student will not match a 671B teacher's absolute scores, and pretending otherwise makes every distillation "fail":

Gate	Threshold	How
Relative solve rate	student >= 0.80 x teacher solve rate on held-out set	Same prompts, same verifier, both models
Format validity	>= 0.98 of student outputs parse (closed </think>, extractable answer)	Regex/parser over student generations
Decontamination proof	0 eval prompts with >= 13-gram overlap vs training set	Re-run decontamination against final training JSONL
No judge-only wins	Verifiable metrics agree with judge direction	If judge says "improved" but exact-match dropped, trust exact-match

Report both numbers ("teacher 84%, student 71%, ratio 0.85 — pass"), and evaluate at the same sampling settings you will deploy the student with. Baseline the un-tuned student too: if base student = 45% and distilled = 71%, distillation added 26 points; without the baseline you cannot attribute the gain.

7. Driving Student Eval as an Autonomous Agent (live-verified 2026-07, us-east-1)

Lessons from an agent-orchestrated QLoRA distillation run (Qwen3-1.7B student, SageMaker ml.g5.2xlarge) where the eval is a verifiable-reward scorer, not an LLM judge:

• Never trust training loss — decide "more epochs?" only from a held-out eval. The run finished 1 epoch at train loss 0.21 / eval loss 0.50; neither number predicts the solve rate. Eval the held-out tasks BEFORE spending on epoch 2.
• Split generation (GPU) from scoring (CPU) into separate processes. The GPU job loads the merged model and writes generations.jsonl incrementally (an interrupted run still yields a scorable partial file); the scorer runs anywhere without torch. A scoring bug then never costs GPU time to re-fix, and the scorer stays unit-testable.
• Greedy decoding for the gate. The eval gate must be reproducible; sampling makes the solve rate a distribution rather than a number. (This intentionally departs from the temperature-0.5–0.7 guidance above, which is for teacher generation, not student gating.)
• The base-model baseline needs its thinking mode suppressed. An un-tuned reasoning student (Qwen3) opens a <think> chain and can spend the entire token budget inside it, emitting no answer — scored naively that reads as "the base model can't do the task" when the real cause is decoding budget, which inflates the apparent distillation lift. Pass enable_thinking=False for the baseline, and verify the chat template actually honors the flag before spending GPU (templates accept unknown Jinja vars in silence; on the real Qwen3 template only False changes the rendering — True and absent are identical).
• Render eval prompts through the SAME chat template used in training. A mismatch silently tanks the solve rate while the model is fine.
• Launch-and-release, never wait in-session. A multi-hour training/generation job must not be awaited inside an agent turn: record the job name to the manifest, exit the turn, and let an event (EventBridge job-state rule → Step Functions task token) re-invoke a fresh session on completion. Budget-bound the job itself (max_train_seconds) so a hang converts to a checkpointed stop, not an open-ended bill.
• Validate the scorer with negative controls before trusting any lift number — see llm-evaluation §6 (Negative Controls): an oracle self-test cannot catch a scorer that accepts too much, and format-validity is the tiebreak metric precisely when both solve rates are 0.000 — the expected case for a small student on a hard benchmark.

Anti-Patterns

1. Distilling from a forbidden teacher — check the license/ToS first. DeepSeek-R1 (MIT) explicitly allows it; several proprietary providers' terms prohibit training on outputs. This is a legal defect in the model, not a data bug you can filter out later.
2. Training on contaminated data — skipping decontamination against eval sets produces scores that evaporate in production. Decontaminate before reporting anything.
3. Reasoning chains longer than the student's context — truncated targets teach the student to produce unterminated reasoning. Filter by student-tokenizer length.
4. Judge-gaming — filtering with the teacher as its own judge, or optimizing until the judge score is high while exact-match falls. Verifiable reward > judge, always.
5. Absolute-performance gates — demanding the student match the teacher's absolute score guarantees failure; gate on the relative ratio and honest baselines.
6. Temperature 0 with reasoning teachers — R1-class models degenerate into repetition loops at temperature 0; stay in the 0.5-0.7 band the model card recommends.

Best Practices

1. License check is step zero — record teacher ID + license decision in the dataset card
2. Budget before generating — estimate cost per 1k samples; reasoning outputs are 2k-6k tokens
3. Checkpoint generation — resume-safe JSONL appends, never lose paid teacher tokens
4. Rejection-sample when ground truth exists — 2-4 generations per prompt, keep verified ones
5. Drop truncated generations — stopReason == "max_tokens" samples are poison
6. Verifiable reward over LLM judge whenever an exact answer or test suite exists
7. Different model for judging than the teacher being distilled
8. Gate relative, report absolute — ratio >= 0.80 with both numbers shown
9. Baseline the un-tuned student to attribute the distillation gain

Scripts

• scripts/generate_teacher_data.py - Batch teacher generation via Bedrock converse with retry/backoff, checkpointing, and cost estimation
• scripts/curate_distillation_data.py - Curation pipeline: format validation, dedup, decontamination, answer verification, judge filtering, TRL-format export

References

See references/REFERENCE.md for teacher/license comparisons, reasoning-mode guidance, cost math, and quality-gate details.

Related skills

Upstream: llm-data-preparation (seed prompts, dedup machinery) · Downstream: llm-fine-tuning (trains the student on this skill's JSONL), then llm-evaluation (student-vs-teacher gates) See also: llm-cost-optimization — distillation is itself a cost lever (replace frontier calls with the student) · llm-deployment for serving the distilled student