---

name: rl-training description: "Routes Hydra and RSL-RL policy training, evaluation, checkpoint loading, task configuration, observation and action contracts, WeightPolicy bridging, and TensorBoard workflows for this quadruped MPC project." disable-model-invocation: true metadata: disco-role: operating license: MIT

RL training

Use this sub-skill when the task involves training or evaluating the learned MPC-weight policy, selecting A1/Aliengo/Go1, loading or resuming a checkpoint, checking observations/actions/rewards, or viewing RSL-RL TensorBoard logs.

Read first

1. Read workflows for the public package/repository installation, Hydra overrides, and the train/test/resume/checkpoint decision.

2. Read configuration before changing task, device, environment count, or learning settings.

3. Read API and bridge contracts before wiring a policy to an MPC runner or diagnosing tensor shape errors.

4. From this installed sub-skill directory, run the bundled safe command/config checker when planning a run:

   python scripts/validate_rl_config.py --task Aliengo --num-envs 1
   python scripts/run_training.py --help
   

   The installed-package launcher is a dry run unless --run is supplied.

5. Read troubleshooting when an import, Hydra resolution, checkpoint, device, or logging step fails.

Hard backend gate

RL environment construction, training, evaluation, and WeightPolicy import require NVIDIA Isaac Gym Preview 4 in addition to the documented Python, PyTorch/CUDA, and RSL-RL stack. The current inspection handoff proved PyTorch 1.10/CUDA 11.3 on an A100, package imports, and mpc_osqp, but did not prove Isaac Gym; isaacgym is unavailable. Treat all RL/simulation execution as blocked until the official SDK is installed and imported. Do not substitute a CPU import or a CUDA tensor smoke test for this gate. Use isaac-gym-simulation for SDK/API and hardware diagnosis, and mpc-control for the controller side of the bridge.

Scope and routing

• Training/evaluation: Hydra overrides select task, headless, num_envs, checkpoint, test, devices, seed, and max_iterations. Use the bundled launcher only after the backend gate passes; public commands and fallback behavior are in workflows.
• Task/config contract: valid task names, defaults, observation/reward layout, and PPO settings are in configuration.
• Policy API and bridge: the 48-value observation and 12-value action contracts, policy architecture, checkpoint keys, and MPC weight mapping are in api-reference.
• Low-level controller math/FSM: do not duplicate it here; route to mpc-control.
• Isaac Gym lifecycle, assets, viewer, PhysX, and GPU pipeline: route to isaac-gym-simulation.

Safe operating rules

• Install the current project package through its public interface, then use the bundled scripts/run_training.py launcher. It resolves the installed training entry point and does not require opening a checkout-local script. Hydra uses key=value overrides, not an argparse option catalog.
• Choose a user-owned working directory and explicit run root for every experiment. Keep checkpoints under user-supplied paths; do not rely on a default run directory from a source checkout.
• Prefer an explicit absolute checkpoint file. A missing explicit file causes the source loader to try its configured latest-run fallback, which can select an unintended run or fail if no model exists. The bundled checker validates paths without importing or deserializing a checkpoint.
• Keep num_obs=48 and num_actions=12. A checkpoint with incompatible actor/critic dimensions is not made safe by changing num_envs.
• Training uses the MPC bridge by default in the public training entry point: policy actions are transformed into 12 MPC parameters and the controller produces torques. This is not the same as direct joint-position control.
• Do not change observation ordering, action scaling, robot task, or policy hidden dimensions for an existing checkpoint without treating it as a new model and validating it from scratch.
• Training is long-running and evaluation loops for ten episode horizons; do not use either as a smoke test. Use the bundled checker and the setup/backend route first.

Completion checklist

Before handing off a requested run, confirm:

• the task is exactly A1, Aliengo, or Go1;
• Isaac Gym is importable on the requested GPU, or the request is explicitly reported as blocked;
• num_envs is positive and fits GPU memory;
• an explicit user-supplied checkpoint exists and is structurally compatible when loading;
• test=True is used only for evaluation and test=False for training/resume;
• the user-owned run directory and TensorBoard log directory are unambiguous; and
• any MPC deployment request is cross-linked to the controller bridge route.