---

name: data-preparation description: "Prepare and validate flat pretraining, packed SFT, preference, and RL prompt datasets for train-llm-from-scratch." disable-model-invocation: true metadata: disco-role: operating license: MIT

Data Preparation Router

Use this sub-skill when a task is about building, inspecting, validating, or triaging datasets for this repository's LLM pipeline. Keep training, checkpoint, evaluation, chat, and UI work routed to the sibling skills listed below.

Decide The Dataset Shape First

1. Base pretraining text -> flat HDF5 tokens array. Use for next-token base-model pretraining.
2. SFT/instruction data -> packed HDF5 with aligned tokens and loss_mask. Use for assistant-only supervised fine-tuning.
3. Preference data -> JSONL rows with prompt, chosen, and rejected. Use for reward-model training and DPO/ORPO/KTO.
4. RL prompts -> JSONL rows with prompt and numeric-or-null gold; the arithmetic curriculum is the same schema. Use for PPO and GRPO/RLVR.

Read references/data-formats.md for exact schema, mask, tokenizer, and data-loader contracts before accepting a user-provided file. Read references/workflows.md for distilled command recipes and safe small-data checks. Read references/troubleshooting.md when validation fails or data generation is blocked by dependencies, cache, network, or context limits.

Safe Validators Bundled Here

These scripts do not download data, train models, modify input files, or require a repository checkout. Run them before launching any stage that consumes the data.

• scripts/inspect_h5_tokens.py — inspect a flat pretraining HDF5 tokens dataset: shape, dtype, min/max, EOT count, and optional head decode.
• scripts/validate_sft_h5.py — validate packed SFT HDF5 tokens and loss_mask schema, binary mask, token range, and trained-token fraction.
• scripts/validate_preference_jsonl.py — validate preference JSONL rows and reject missing/empty fields or degenerate chosen == rejected pairs.
• scripts/validate_rl_prompts_jsonl.py — validate prompt/gold JSONL rows, numeric/null gold policy, and optional arithmetic-curriculum consistency.

Routing Boundaries

• Base model training, DDP/bf16 pretraining, memory flags, checkpointing, and resume behavior -> ../model-pretraining/SKILL.md.
• SFT, reward model, DPO/ORPO/KTO, PPO, GRPO, rollout, KL, and optimizer logic -> ../post-training-rlhf/SKILL.md.
• GSM8K scoring, answer parsing during evaluation, stage tables, generation, and chat -> ../evaluation-chat/SKILL.md.
• JSON config editing, CLI override precedence, smoke configs, and Streamlit UI -> ../configuration-ui/SKILL.md.

Operating Checklist

1. Ask which downstream stage the data will feed and record the intended context length, data directory, cache policy, and whether network access is available.
2. Pick the matching dataset shape from this router; do not mix HDF5 pretraining tokens, packed SFT rows, preference pairs, and RL prompts.
3. If generating public data, use the distilled command recipe in references/workflows.md, starting with a tiny limit when practical.
4. Validate the resulting file with the bundled script for that shape.
5. If validation fails, stop before training and use references/troubleshooting.md to identify whether the fix is regeneration, path creation, cache/network repair, context length change, or schema cleanup.