---

name: post-training-alignment description: "Helps with LMFlow reward modeling, DPO, DPOv2, iterative DPO, RAFT, and LoRA merge workflows." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Post-Training Alignment

Use this sub-skill when the task is about reward models, preference optimization, RAFT, iterative DPO, or merging LoRA adapters after a training run.

Typical Triggers

• reward model, rm_tuner, rm_inferencer
• DPO, DPOv2, iterative DPO
• RAFT, alignment, preference optimization
• merge LoRA, lora_model_path, reward_model_name_or_path

What This Sub-Skill Owns

• Reward-model training and inference guidance.
• Preference-data layouts used by DPO-style workflows.
• RAFT command patterns and data cleanup notes.
• LoRA merge command construction.

Read These First

• references/workflows.md for the workflow overview.
• references/data-and-config.md for preference dataset and config shapes.
• references/api-reference.md for the alignment dataclass families.
• references/troubleshooting.md for trl, Ray, reward-data, and merge issues.
• scripts/build_alignment_command.py to render a copyable command.

Cross-Links

• Base fine-tuning lives in ../training-and-optimization/SKILL.md.
• Dataset schema details live in ../data-and-templates/SKILL.md.
• Engine-specific inference details that iterative DPO may reuse live in ../inference-and-evaluation/SKILL.md.

Workflow

1. Identify whether the user wants reward modeling, pairwise preference optimization, RAFT, or a merge step.
2. Confirm the dataset type and whether a separate reward/reference model is needed.
3. Check the selected extras (trl, ray, optional engine extras) before promising runtime success.
4. Render the command and call out any blocking hardware or download requirements.

Common Decisions

• Use reward-model training when the task is to score responses.
• Use DPO/DPOv2 when the task is to optimize pairwise preferences.
• Use iterative DPO when the task repeatedly generates, scores, and aligns.
• Use RAFT when the user explicitly wants reward-ranked fine-tuning.
• Use merge-lora when the task is to combine a base model and a LoRA adapter.

What Not To Do

• Do not mix this route with ordinary supervised fine-tuning.
• Do not hide the difference between training a reward model and using it for inference.
• Do not treat missing trl, Ray, or engine extras as optional if the user requested the corresponding path.
• Do not promise a GPU runtime for RAFT or iterative DPO without the relevant backend evidence.