---

name: td3-training description: "Guide TD3 model construction, bounded training adaptations, replay-buffer use, checkpoints, and TensorBoard outputs for the DRL robot-navigation implementation without requiring ROS or Gazebo for model checks." disable-model-invocation: true metadata: disco-role: operating license: MIT

TD3 training

Use this skill when a researcher needs to understand or adapt the repository's TD3 training contract. It is an operating guide, not a simulator launcher and not an evaluation-loop guide. Keep ROS/Gazebo startup and policy evaluation in the repository's corresponding sibling skills; use the cross-links supplied by the top-level router rather than duplicating them here.

Operating contract

• The intended observation has 20 Velodyne bins followed by four robot values: distance to goal, relative heading, linear action, and angular action. The model therefore uses state_dim=24 and action_dim=2.
• Actor outputs are normalized actions in [-1, 1]^2. Before the environment step, convert them to [linear=(a0+1)/2, angular=a1]; do not store this converted pair as the replay action.
• Training is off-policy TD3 over transitions (state, normalized_action, reward, done_for_bootstrap, next_state).
• The implementation assumes a process working directory containing pytorch_models/, results/, and runs/. Treat those as configurable output locations in any safe adaptation, rather than relying on a checkout path.

Read the focused references before changing behavior:

• api-reference.md — architecture and method contracts, including source quirks.
• workflows.md — bounded model smoke, training adaptation, artifacts, and TensorBoard workflow.
• hyperparameters.md — exact defaults and the distinction between method defaults and training-call values.
• troubleshooting.md — predictable failures and diagnosis without launching the simulator.

Safe use sequence

1. Establish the required state/action dimensions and selected device. Run the bundled scripts/td3_model_smoke.py first; it reimplements only the Actor and Critic shape contract and never imports ROS, Gazebo, or repository modules.
2. Validate replay semantics with scripts/replay_buffer_smoke.py. In particular, verify eviction, the underfilled-batch behavior, done flags, and the state/action array shapes.
3. For real training, provide a ROS/Gazebo-capable environment through the separate simulator workflow. This skill does not claim that simulator execution is available on a CPU-only inspection host.
4. Preserve normalized actions in the buffer, use a separate environment-action conversion, and preserve the time-limit bootstrap convention described in the references.
5. Keep training bounded during development: use a small step/episode budget, a prefilled synthetic or mock transition source, an explicit log directory, and a temporary checkpoint/result directory. Do not use the repository's multi-million-step defaults for a smoke test.
6. Check the resulting actor/critic state-dict shapes and output files before attempting any evaluation. Evaluation cadence and ROS launch belong to the sibling evaluation/simulator skills.

Algorithm decisions to preserve

The actor is 24 -> 800 -> 600 -> 2, with ReLU on the two hidden layers and tanh on the output. The critic has two independent Q branches. Each branch projects state through 24 -> 800, combines a 600-wide state projection with a 600-wide action projection, and emits one scalar. Actor and critic targets are initialized from their online networks. Adam is used with PyTorch's implicit constructor defaults.

Each critic update forms a target action from the actor target, adds Gaussian policy noise clipped to [-noise_clip, noise_clip], clamps the action to [-max_action, max_action], and uses the smaller target Q. The critic minimizes the sum of two MSE losses. On every policy_freq-th local iteration, the actor is updated through the first critic Q and both target networks receive a soft update. The source resets this local iteration at each call to TD3.train, so record that quirk when reproducing cadence.

Exploration adds Gaussian noise to the normalized actor action and clips it. When enabled, the near-obstacle branch can hold one random angular action for 8–14 steps while forcing normalized linear action to -1. See the exact slice and probability conditions in the hyperparameter reference; do not silently replace this with environment-space noise.

Outputs and compatibility

Training creates actor and critic state-dict files under pytorch_models, a NumPy evaluation history under results, and scalar events under runs. Names, relative-path assumptions, missing optimizer state, and load-device compatibility are intentional review points. A portable adaptation should use map_location when loading, validate state-dict keys/shapes, and pass an explicit TensorBoard log_dir.

The checked host lacks ROS Noetic, Gazebo, roscore, roslaunch, catkin_make, and ROS message modules. Therefore only CPU model/replay checks (and optional CUDA tensor checks when independently available) are in scope here. Never report a successful simulator training run based on these bundled smokes.

Evidence anchors

The operating contract is distilled from the repository README.md, TD3/train_velodyne_td3.py, TD3/replay_buffer.py, and the output-directory notes in TD3/runs/description and TD3/pytorch_models/description. TD3/test_velodyne_td3.py was used only to confirm the actor checkpoint-loading shape contract. These paths are provenance, not runtime dependencies.

Handoff checklist

Before handing a TD3 adaptation to another skill, report:

• state/action dimensions and device;
• whether transitions were normalized and whether the buffer was sufficiently populated;
• exact bounded iteration/episode budget and random seeds;
• actor/critic output and checkpoint shape checks;
• TensorBoard and result paths, with no accidental checkout-path dependency;
• any unresolved ROS/Gazebo or PyTorch serialization limitation.