---

name: off-policy-value description: "Guides minimalRL DQN, ACER, and V-trace replay, target-network, and off-policy correction workflows for discrete control." disable-model-invocation: true metadata: disco-role: operating license: MIT

Off-Policy Value Workflows

Use this sub-skill when a task asks about minimalRL's discrete value-learning, replay-buffer, target-network, ACER, or V-trace correction logic.

Route by task

• DQN / replay buffer / target network: read references/api-reference.md for ReplayBuffer, Qnet, and train contracts, then references/workflows.md for the safe update flow.
• ACER: read references/workflows.md for sequence replay and truncated importance sampling, then references/troubleshooting.md for probability-ratio failures.
• V-trace off-policy correction: use the V-trace section in references/api-reference.md and references/workflows.md; route single-process policy/value architecture questions to ../on-policy-discrete/SKILL.md.
• Shape-only or pre-training checks: run scripts/smoke_off_policy_value.py.

Do not use this sub-skill for

• REINFORCE, actor-critic, PPO, or recurrent PPO route selection; use ../on-policy-discrete/SKILL.md.
• DDPG, continuous PPO, or SAC; use ../continuous-control/SKILL.md.
• A2C/A3C multiprocessing and worker lifecycle; use ../parallel-actor-critic/SKILL.md.

Quick operating workflow

1. Identify whether the user is debugging value targets, replay sampling, behavior-policy probabilities, or target-network synchronization.

2. Check tuple contracts in references/api-reference.md before changing replay buffer contents.

3. Use references/workflows.md to preserve the update order: sample replay, compute detached targets, optimize current network, then update target/behavior state as appropriate.

4. Run the bundled smoke helper before long training:

   python sub-skills/off-policy-value/scripts/smoke_off_policy_value.py --algorithm all
   

5. For ValueError: Sample larger than population, NaN ratios, gather shape failures, or stale target behavior, read references/troubleshooting.md.

Bundled runtime files

• references/api-reference.md records replay tuple contracts, classes, functions, hyperparameters, and tensor shapes.
• references/workflows.md explains DQN, ACER, and V-trace update sequences and adaptation rules.
• references/troubleshooting.md gives symptom-to-fix guidance for replay and correction failures.
• scripts/smoke_off_policy_value.py validates replay sampling and off-policy tensor contracts without importing the original repository or running full training.