---

name: gpu-and-resource-operations description: "Detect and select local NVIDIA GPUs, report their resource status, and explain the explicitly opt-in GPU keep-alive without hiding hardware or CUDA limitations." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

GPU and resource operations

Use this skill for local, read-only GPU inspection and selection before an experiment. It covers NVIDIA visibility, memory-based availability, the reserve_last policy, and the optional keep-alive's safety contract. It does not launch training, reserve a device through a scheduler, or start a keeper implicitly.

Safety contract

• Begin with the bundled scripts/gpu_status.py; it only invokes nvidia-smi queries and never allocates CUDA memory.
• Treat an absent, timed-out, or failed nvidia-smi query as no usable GPU. Never infer free devices from configuration, PyTorch importability, or a missing status result.
• “Free” means reported memory_used_mb < 1000 by default, not that a device is exclusively reserved. Confirm ownership before launching work.
• reserve_last=true is a selection policy for a keep-alive slot, not a lock. With one detected GPU it remains usable; with two or more, the highest listed index is excluded from experiment candidates.
• Do not run the keep-alive during inspection. It allocates a CUDA tensor and performs repeated operations, so it is an explicit side effect requiring user approval.

Operating flow

1. Run python scripts/gpu_status.py --help if the interface is unfamiliar; help must not probe hardware.
2. Run python scripts/gpu_status.py (or add --json for machine-readable output). Record the status reason when no GPU is reported.
3. Select only from usable_gpus and then filter to free_gpus. Keep the reserved device out of training when the default policy excludes it.
4. If the user explicitly chooses a device, report its gpu_id, name, memory, utilization, and temperature before passing its ID to the experiment launcher. Local launchers express this as CUDA_VISIBLE_DEVICES.
5. Before CUDA work, verify both a working NVIDIA driver (nvidia-smi) and a PyTorch build for which torch.cuda.is_available() is true. A Python package alone is not proof of CUDA support.
6. For SSH/Slurm status, scheduler allocation, or transport details, route to execution-and-monitoring. For loop launch and experiment lifecycle, route to autonomous-experiments. For installing external skills, route to skills-and-installation.

Bundled references

• GPU status and selection — exact detector signatures, output shapes, query fields, thresholds, and edge cases.
• Keep-alive contract — explicit side effects, PyTorch/CUDA prerequisites, interval, signals, cleanup, and invocation rules.
• Troubleshooting — predictable driver, CUDA, parsing, selection, and keeper failures.

Output contract

A normal report should include:

• detected: whether a complete local NVIDIA status was obtained;
• one row per GPU with ID, name, used/total memory in MB, utilization percent, and temperature in °C;
• usable_gpus after reserve_last policy;
• free_gpus after the strict 1000-MB default threshold (or the stated override);
• an explicit reason when detection/status is unavailable; and
• whether a last device was excluded, without claiming a reservation lock.

When detected is false, both usable and free lists must be empty. Do not silently substitute Slurm queue occupancy or a remote host's status for local NVIDIA measurements.

If a report is used to choose a launch mask, preserve the host GPU IDs and the selection policy in the handoff. Re-run the read-only probe immediately before a long launch because memory and utilization are live observations, not a reservation. Never claim that a keeper is active unless its explicit process and stop/cleanup outcome have been checked.