---

name: data-preparation description: "Prepare and validate OpenRLHF datasets for SFT, reward modeling, DPO, PPO/RL prompts, chat-template use, dataset mixing, packing-sensitive length checks, and VLM image placeholders before expensive training." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

OpenRLHF Data Preparation

Use this sub-skill when the task is about dataset schemas, key mapping, chat-template choices, prompt/reward/preference records, tiny dry-run validation, or VLM image placeholder alignment for OpenRLHF.

Route by Training Mode

• SFT: prepare prompt/response or multiturn chat records; see references/data-formats.md#sft-records and validate with scripts/validate_openrlhf_dataset.py --mode sft.
• Reward model: prepare chosen/rejected pairs, optional prompt prefixes, and optional margin values; see references/data-formats.md#reward-model-records and validate with --mode rm.
• DPO: prepare chosen/rejected preferences with optional shared prompt; see references/data-formats.md#dpo-records and validate with --mode dpo.
• PPO/RL prompts: prepare prompt-only records with optional labels and datasource fields; see references/data-formats.md#ppo-and-rl-prompt-records and validate with --mode ppo.
• VLM prompts: verify <image> placeholders line up with the configured image key before Ray/vLLM training; see references/data-formats.md#vlm-image-prompts and references/troubleshooting.md#multimodal-image-mismatches.

Core Decisions

• Match CLI key flags (--data.input_key, --data.output_key, --data.prompt_key, --data.chosen_key, --data.rejected_key, --data.label_key, --data.image_key) to the actual record fields before launching training.
• Choose exactly one prompt formatting path: use --data.apply_chat_template for chat-message lists, or --data.input_template for plain string prompts.
• For SFT multiturn data, enable --data.multiturn only with --data.apply_chat_template, and put the full trajectory under --data.input_key rather than splitting assistant turns into --data.output_key.
• For mixed datasets, keep comma-separated --data.dataset and --data.dataset_probs counts aligned; probabilities are parsed as floats and must match the dataset count.
• Treat --data.max_len, prompt length, packing, ring attention, and VLM image counts as preflight risks; see references/validation.md before running expensive GPU jobs.

Safe Local Validator

Run the bundled validator on tiny local JSON or JSONL samples without importing OpenRLHF, Ray, torch, datasets, or transformers:

python skills/openrlhf/sub-skills/data-preparation/scripts/validate_openrlhf_dataset.py \
  --mode sft \
  --input sample.jsonl \
  --input-key question \
  --output-key response \
  --max-samples 20

The validator checks required keys, null/empty values, chat-message shape, chosen/rejected confusion, dataset probability counts, rough length limits, and VLM placeholder/image alignment. It is a schema preflight, not a tokenizer-accurate training simulation.

Boundaries

• For full SFT/RM/DPO training command construction, hand off to supervised-preference-training after data validates.
• For PPO/Ray/vLLM rollout setup, agent functions, or reward services, hand off to rl-agent-training after prompt data validates.
• For installation, CUDA, flash-attn, Ray, vLLM, DeepSpeed, checkpoint, and environment issues, hand off to operations-and-utilities.