---

name: evaluation description: "Test NanoTrack-compatible trackers on named datasets, validate result layouts, dispatch benchmark protocols, interpret tracking metrics, and plan bounded hyperparameter searches." disable-model-invocation: true metadata: disco-role: operating license: NO_LICENSE

NanoTrack Evaluation

Use this sub-skill to plan or diagnose NanoTrack benchmark testing, validate already-produced result files, choose the matching evaluator, interpret metrics, or bound a hyperparameter search. NanoTrack is the collection's maintained workflow; treat other tracker snapshots as catalogue entries unless explicitly routed elsewhere.

Route First

• Model/config/checkpoint loading, device placement, tracker initialization, or frame-by-frame prediction: route to inference.
• Dataset cropping, losses, optimization, checkpoint creation, or resume: route to training.
• ONNX/NCNN conversion, FLOPs, latency, or deployment speed claims: route to export.
• Any non-NanoTrack snapshot or comparison across snapshots: route to variant-catalog.

Do not convert an evaluation request into training, and do not treat benchmark speed output as a controlled export/deployment benchmark.

Read By Need

• Dataset names, factory behavior, on-disk data assumptions, CLI flag contracts, and exact result paths: datasets and results
• Evaluator dispatch, metric meaning, GOT-10k wrappers, and bounded parameter search: benchmarks
• Extension ABI, imports, data/path mismatches, workers, GUI, CUDA, and known dispatch defects: troubleshooting
• Offline structural preflight: check_result_layout.py

Hard Gates

A full run is not self-contained in this skill. Establish all of these before claiming test or evaluation success:

1. The named dataset and its annotations/images are present in the layout expected by the selected wrapper.
2. A matching NanoTrack config and checkpoint exist; evaluation-only work may instead start from complete result files.
3. The model runtime can import its dependencies and, for stock testing, use an explicitly selected CUDA device. A merely visible GPU is not proof it is free or compatible.
4. The Cython extension exposing toolkit.utils.region was built for the active Python ABI. Never accept a copied prebuilt region.so as proof.
5. The result root, dataset name, and tracker name resolve to the same directory contract for result production and result consumption.
6. Multiprocessing and any plotting/visualization requirements are acceptable for the host. Use a headless path unless visualization was requested.

No dataset or checkpoint is bundled. Full tracking, training, benchmark, parameter-search, plotting, and export execution were not verified during skill construction. Never invent scores or imply that a layout-only pass validates tracking quality.

Operating Sequence

1. Normalize the request

Record:

• test-and-evaluate, evaluate-existing-results, layout-only, or hyperparameter search;
• exact dataset spelling and protocol;
• dataset root, result root, tracker name/pattern, and expected sequences;
• config/checkpoint pair and GPU allocation when predictions must be produced;
• required metrics, worker limit, visualization policy, and completion proof.

If the user asks for a single video, remember that result production may filter to it while the evaluator still expects a complete dataset unless separately constrained. Prefer layout validation over launching a full evaluator on partial results.

2. Select dataset and protocol

Use DatasetFactory.create_dataset(name, dataset_root, load_img=False) for the maintained test-side contract. load_img=False avoids caching every image, but wrappers still open frames while iterating and some inspect the first image at construction. It is not a metadata-only mode.

Choose one protocol:

• OPE: OTB, DTB70, UAVDT, VisDrone, LaSOT, UAV123/UAV20L, or NFS.
• GOT-10k validation/reporting: GOT-10k.
• VOT restart: VOT2016, VOT2017, VOT2018, or VOT2019.
• VOT long-term: VOT2018-LT.

Read the dispatch asymmetries before launch: VOT2017 is evaluator-only in this snapshot; TrackingNet is factory-only; UAVDT and VisDrone need an evaluator import correction in an active runtime.

3. Separate production from evaluation

Result production consumes images, config, checkpoint, tracker code, OpenCV, PyTorch, and CUDA. Evaluation consumes images/annotations plus result files and may start process pools. Keep their evidence distinct:

• a result file exists: production wrote something;
• bundled validator passes: names, row types, and selected counts are coherent;
• evaluator completes: protocol code consumed data and results;
• metric is credible: protocol, dataset split, result completeness, and runtime were documented.

4. Validate result files offline

Run the bundled standard-library checker before importing the evaluation stack:

SKILL_DIR=/path/to/evaluation
python "$SKILL_DIR/scripts/check_result_layout.py" --list-datasets
python "$SKILL_DIR/scripts/check_result_layout.py" \
  --dataset OTB100 --results-root ./results --tracker-name nanotrack \
  --sequence FleetFace:707

For VOT restart, the default check enforces the stock loss/restart marker schedule. Use --relax-vot-restart only when deliberately validating a non-stock but protocol-compatible producer. Add --json for machine-readable output. The checker performs no downloads, imports, multiprocessing, metrics, or GUI work.

5. Evaluate with the matching benchmark

• OPE: success curve/AUC and 20-pixel precision; LaSOT additionally reports normalized precision at threshold 0.20.
• VOT restart: accuracy, robustness/lost count, and EAO.
• VOT2018-LT: confidence-threshold precision, recall, and maximum F1.
• GOT-10k validation: AO, success rate at IoU greater than 0.5, and speed from positive timing rows.

Interpret only metrics actually emitted. Do not compare numbers across different protocols as if they had the same denominator or restart behavior. Read benchmarks before explaining metric trade-offs.

6. Search hyperparameters safely

Search only after one fixed configuration produces valid results and metrics. Freeze dataset split, checkpoint, variant, device policy, and evaluator. Compute the Cartesian trial count before launch, use a small coarse grid, allocate a unique result directory per trial, and never treat an Occ placeholder as a completed result. Select on a validation set; reserve test/hidden-test results for final reporting. Record the full tuple (penalty_k, lr, window_influence, instance_size) with every score.

Completion Evidence

Return a concise record containing:

• exact dataset/protocol and whether results were produced or only inspected;
• data, checkpoint, CUDA, extension, worker, and GUI gate status;
• result-root/dataset/tracker path contract;
• validator command and exit status;
• evaluator class and emitted metric names, with scores only when truly run;
• missing sequences/files, warnings, patches, and unresolved limitations.