---

name: trlx description: "Guides trlX LLM post-training, RLHF, Accelerate, configuration, sweeps, and optional NeMo/Megatron workflows." disable-model-invocation: true metadata: disco-role: operating license: MIT

trlX repo skill

Use this skill when a task involves trlX (trlx), the CarperAI Transformer Reinforcement Learning X package for fine-tuning language models with RLHF-style workflows.

Read this skill for

• trlx.train usage, signatures, argument modes, callbacks, and trainer return objects.
• PPO, ILQL, SFT, and RFT post-training recipes for causal or seq2seq language models.
• Reward-function training, reward-labeled sample training, prompt/completion SFT, PEFT, checkpoints, logging, and generation evaluation.
• TRLConfig/YAML/default config editing, trainer/method/pipeline matching, optimizer/scheduler names, and data pipeline contracts.
• Hugging Face Accelerate and DeepSpeed launch planning for trlX training scripts.
• Ray Tune sweeps with python -m trlx.sweep and trlX dot-path config updates.
• Optional NVIDIA NeMo/Megatron workflows, .nemo or rank-sharded checkpoints, and NeMo config wiring.

Do not use this skill for

• General RL libraries such as Gymnasium, Stable-Baselines3, CleanRL, or Tianshou unless the task explicitly uses trlX.
• Generic NeMo workflows that do not touch trlX trainer/model wrappers.
• Long maintainer benchmark/report workflows that clone branches, run full GPU examples, and publish W&B reports; those were excluded from runtime guidance.
• Training execution itself when dependencies, datasets, credentials, or hardware are unavailable; first use the checks and troubleshooting references.

Fast route map

If the task mentions...	Go to
trlx.train, reward_fn, prompts, samples, rewards, PPO, ILQL, SFT, RFT, PEFT, checkpoints, Accelerate, DeepSpeed, or sweeps	training sub-skill
NeMo, Megatron, Apex, .nemo, NeMoPPOTrainer, NeMoILQLTrainer, NeMoSFTTrainer, PPOGPT, ILQLGPT, SFTGPT, mp_rank_XX, or NeMo YAMLs	nemo sub-skill
Package purpose, install shape, supported algorithms/backends, root route selection, or verified inspection facts	package overview
Import failures, dependency conflicts, CUDA/DeepSpeed setup, missing NeMo, W&B/Ray import behavior, or launcher environment mismatch	root troubleshooting
Staleness, source commit, package version, evidence paths, or whether to refresh this skill	repo provenance

Minimal install and import check

For full training stacks, use a Python 3.9-3.11 environment. Install a CUDA-capable PyTorch matching the host hardware when GPU training is needed, then install trlX from a public source or a current checkout. A typical source-checkout path is:

python -m pip install -r requirements.txt
python -m pip install -e .

When installing directly from the public repository, install the appropriate PyTorch wheel first because trlX's package metadata does not fully replace the repository's pinned runtime requirements.

After installation, run the bundled safe helper from this skill tree:

python scripts/check_trlx_install.py
python scripts/check_trlx_install.py --cuda
python scripts/check_trlx_install.py --json

The helper imports trlX, prints the trlx.train signature, registries, default config summaries, and optionally performs a tiny CUDA allocation. It does not download models or start training.

Backend notes

• Accelerate-backed trlX APIs and CUDA-capable PyTorch import were verified for this skill's source snapshot.
• NeMo/Apex/Megatron was intentionally not installed in the minimum inspection environment. The NeMo sub-skill is source-backed and must not be treated as verified backend coverage until a NeMo/Apex environment is prepared.
• Many public examples require Hugging Face model/dataset downloads, W&B, Triton, Ray, large GPU memory, or multi-node launchers. The skill distills their patterns instead of making future agents run original source examples.