---

name: training description: "Plan, validate, launch, monitor, and safely resume NanoTrack training with explicit data, configuration, CUDA, and runtime gates." disable-model-invocation: true metadata: disco-role: operating license: NO_LICENSE

NanoTrack Training

Scope

Use this operating skill for NanoTrack training requests in the SiamTrackers collection: preparing cropped training data, editing a copied YAML config, validating model/data compatibility, selecting a safe launch, interpreting loss and checkpoint behavior, or planning recovery from a failed run.

NanoTrack is the maintained end-to-end workflow represented here. Treat other tracker directories as historical snapshots unless another skill says otherwise.

Route these requests instead:

• Tracking initialization, template/track, and frame-by-frame use: inference.
• Benchmark result production or metric interpretation: evaluation.
• ONNX, mobile conversion, FLOPs, latency, or deployment: export.
• Differences in V1/V2/V3 code selection or training another snapshot: variant-catalog.

Hard Limits

• Training is CUDA-only in the stock control flow. It constructs the model with .cuda() and moves all four training tensors with .cuda(); CUDA: false does not create a CPU training path.
• The stock entry point initializes rank=0, world_size=1 through a stub. Distributed classes are present, but stock multi-process training is not enabled. Do not put the unmodified entry point behind torchrun.
• A real run needs cropped images, an annotation JSON, compatible weights when configured, and substantial time. None is bundled with this skill.
• No full data, training, checkpoint-resume, metric, or export run was verified during skill construction. Never promise an accuracy, duration, or output.
• Historical dependency pins describe an old environment. Do not reproduce those versions blindly on modern Python/CUDA.

Operating Sequence

1. Establish a NanoTrack-compatible project root, copied config path, requested variant, data ownership, GPU allocation, run directory, and completion criterion. Keep every relative config path anchored to the launch root.

2. If the request changes V1/V2/V3 architecture or head selection, consult variant-catalog first. A YAML filename alone does not select every variant-specific implementation detail.

3. Read configuration.md. Confirm BAN.BAN, backbone/adjust/head channels, training geometry, enabled dataset records, augmentation rates, target counts, scheduler, and output paths.

4. Run the bundled read-only checker before importing the training package:

   SKILL_DIR=/path/to/this/training
   python "$SKILL_DIR/scripts/check_training_config.py" \
     --config "$CFG" \
     --project-root "$NANOTRACK_ROOT" \
     --world-size 1 \
     --require-cuda
   

5. Treat every checker error as a launch blocker. Review warnings manually; geometry warnings can be intentional for a matched variant, but unchecked files and non-divisible epoch sizing need explicit acceptance.

6. Perform the data and CUDA gates in workflows.md. In particular, probe one annotation record and its exact crop file, then load one BANDataset sample before a long run. Do not use a benchmark dataset layout as a training crop layout.

7. Start with single-process execution and a unique copied config. Set TRAIN.LOG_DIR and TRAIN.SNAPSHOT_DIR to run-specific locations. Use an explicitly selected free GPU; an available GPU is not necessarily idle.

8. Watch startup through config merge, model creation, data indexing, first forward/backward, first scalar write, and first periodic log. Stop on silent invalid-loss skipping, repeated worker failures, or unexpected output-path reuse.

9. Resume only after matching the config, variant, optimizer parameter groups, and checkpoint epoch. Read the transition-epoch warning in workflows.md; scheduler state is reconstructed, not restored.

10. Record the exact config, command, environment probe, data identity, seed, GPU assignment, and accepted warnings with the run outputs.

Input Contract

The dataset returns a mapping containing:

Key	Meaning
template	Float32 CHW crop, normally 3 x TRAIN.EXEMPLAR_SIZE x TRAIN.EXEMPLAR_SIZE
search	Float32 CHW crop, normally 3 x TRAIN.SEARCH_SIZE x TRAIN.SEARCH_SIZE
label_cls	Int64 point labels: -1 ignored, 0 negative, 1 positive
label_loc	Float32 4 x OUTPUT_SIZE x OUTPUT_SIZE left/top/right/bottom distances
bbox	Augmented search-box corners; returned for inspection, not consumed by the model loss

The training forward consumes the first four keys, moves them to CUDA, and returns total_loss, cls_loss, and loc_loss. The weighted total is:

total_loss = TRAIN.CLS_WEIGHT * cls_loss
           + TRAIN.LOC_WEIGHT * loc_loss

The loader uses TRAIN.BATCH_SIZE, TRAIN.NUM_WORKERS, pinned memory, and an internally shuffled index list. It does not pass shuffle=True to the DataLoader.

Dataset Gate

For each name in DATASET.NAMES, require a non-empty ROOT, ANNO, FRAME_RANGE, and NUM_USE. Relative paths resolve from the NanoTrack launch root. The maintained GOT record defaults to:

ROOT: data/GOT-10k/crop511
ANNO: data/GOT-10k/train.json

A crop is addressed as:

ROOT/<video>/<six-digit-frame>.<track>.x.jpg

The annotation must map video to track to six-digit frame keys and valid boxes. Read the exact schema and sampling behavior in configuration.md.

Safe Single-Process Launch

The historical training launcher is a long-running, CUDA/data-bound source entry point. This skill intentionally does not bundle or execute it. After the static and real-batch gates pass, an operator with the selected source implementation may invoke its documented launcher from that checkout using an explicit config, a selected physical GPU exposed as logical device 0, and run-specific log/snapshot paths. The source launcher is an external execution dependency, not a runtime file in this skill.

This command starts a long-running job and must be approved after the gates pass. Use the bundled checker for dry-run/config validation; the historical launcher has no safe dry-run or max-step option.

Output Contract

Rank 0 creates TRAIN.LOG_DIR, writes logs.txt, and opens a TensorBoard writer there. It creates TRAIN.SNAPSHOT_DIR and saves checkpoint_e<N>.pth at epoch transitions with:

• epoch
• state_dict
• optimizer

The scheduler is absent from the checkpoint. TRAIN.RESUME restores model, optimizer, and epoch; TRAIN.PRETRAINED loads model weights only and is ignored when resume is set. BACKBONE.PRETRAINED, when non-empty, is loaded earlier.

Failure Routing

Use troubleshooting.md for missing crops, annotation schema failures, worker crashes, CUDA/OOM errors, geometry or channel mismatches, invalid losses, scheduler/resume issues, and unsafe distributed launches. Escalate architecture-selection uncertainty to variant-catalog; do not repair it by guessing channels or output sizes.