---

name: torchrl description: "Use TorchRL for TensorDict-first reinforcement-learning environments, collectors, replay buffers, modules, objectives, LLM/RLHF/VLA workflows, services, rendering, and maintainer-safe repository changes." disable-model-invocation: true metadata: disco-role: operating license: MIT

TorchRL

Use this repo skill when the task involves TorchRL (torchrl), the PyTorch reinforcement-learning library built around TensorDict data, composable environments, collectors, replay buffers, modules, losses, trainers, LLM/RLHF/VLA extensions, services, rendering, or contributing to the pytorch/rl repository.

First checks

1. Confirm the installed package and backend scope before making claims:

   python - <<'PY'
   import torch, tensordict, torchrl
   print('torch', torch.__version__, 'cuda', torch.version.cuda, torch.cuda.is_available())
   print('tensordict', tensordict.__version__)
   print('torchrl', torchrl.__version__)
   PY
   

2. For a reusable base smoke, run scripts/check_torchrl_env.py. It imports the major TorchRL surfaces, runs a native PendulumEnv rollout, samples a small replay buffer, inspects rlrender help, and reports optional backend availability without downloading models or starting services.

3. If the task depends on Gym, MuJoCo, DM Control, IsaacLab, VMAS, Ray, vLLM, SGLang, LeRobot/OpenX, video codecs, or CUDA kernels, read backend compatibility and the owning sub-skill's troubleshooting file before deciding whether a CPU result is enough.

4. If you are working in a source checkout, compare it with repository provenance. Refresh this skill if commit, package version, public entry points, or dirty source state differ materially.

Route by task

Task signal	Read
EnvBase, PendulumEnv, GymEnv, specs, TransformedEnv, Compose, transforms, check_env_specs, step_mdp, SerialEnv, ParallelEnv, simulator wrappers	envs-and-transforms
Collector, rollout loops, evaluator, frames_per_batch, sync, backend selection, replay buffers, storages, samplers, prioritized replay, HER, memmap, Ray replay	collectors-and-replay
Actor, ProbabilisticActor, ValueOperator, QValueActor, TensorDictModule keys, specs, distributions, recurrent GRU/LSTM modules, multi-agent models, model-based wrappers	modules-and-policies
PPO/SAC/DQN/DDPG/TD3/IQL/CQL/MAPPO losses, value estimators, set_keys, target updaters, trainers, Hydra configs, SOTA algorithm recipes	objectives-and-training
LLM post-training, RLHF/GRPO/SFT, ChatEnv, LLMCollector, vLLM/SGLang wrappers, VLA schemas/actions, service registry, render CLI, video/checkpoint surfaces	llm-vla-and-services
Editing TorchRL source, adding public APIs, tests, docs, benchmarks, deprecations, optional-dep CI labels, GPU markers, Hydra config parity	development-and-testing

Install and dependency stance

• General users: pip install torchrl with a PyTorch build appropriate for the task. Match PyTorch and TensorDict versions; TorchRL releases are synchronized with the PyTorch ecosystem.
• Source contributors: use an editable install only in a checkout, after installing the intended PyTorch build. When using uv with a preselected PyTorch/nightly build, use --no-deps for editable installs to avoid unintended framework downgrades.
• Install optional extras narrowly. Examples: torchrl[dm_control], torchrl[gym_continuous], torchrl[marl], torchrl[offline-data], torchrl[llm], torchrl[llm-vllm], torchrl[llm-sglang], torchrl[grpo], torchrl[vla], torchrl[rendering], torchrl[video].
• Do not install broad dev/test/LLM/simulator extras just to answer a CPU-verifiable API question. Document unverified optional backend limits instead.

Read install and extras for the package metadata, console entry points, and safe install/probe commands.

Core mental model

TorchRL components pass structured TensorDict objects through the whole loop:

TensorDict -> policy/module writes action/log_prob -> environment writes next/reward/done
           -> collector batches trajectories -> replay buffer stores/samples
           -> loss reads named keys -> optimizer updates ordinary PyTorch parameters

Keep keys explicit, prefer NestedKey tuples for nested data, validate specs early, and route optional backend failures to the narrow owner rather than rewriting the full pipeline.

Tiny CPU integration smoke

For a no-download, no-simulator sanity check across the main RL path, run these bundled helpers from their local skill directories after installing TorchRL:

python scripts/check_torchrl_env.py --steps 3 --check-cli
python sub-skills/envs-and-transforms/scripts/smoke_env_rollout.py --steps 3 --check-specs
python sub-skills/modules-and-policies/scripts/smoke_actor.py
python sub-skills/collectors-and-replay/scripts/smoke_collector.py
python sub-skills/objectives-and-training/scripts/inspect_loss_keys.py --loss ClipPPOLoss

When wiring PPO, remember that ClipPPOLoss defaults sample_log_prob to action_log_prob; make the actor write that key or remap the loss with set_keys(sample_log_prob=...).

Cross-cutting troubleshooting

Read troubleshooting for install/import failures, version mismatches, optional dependency errors, CLI misuse, backend claims, and when to stop instead of silently falling back. Workflow-specific failure matrices live in each sub-skill's references/troubleshooting.md.