---

name: positive_advantage_sil_loss description: Compute Self-Imitation Learning policy and value losses using the paper's positive-advantage gate.

Positive-Advantage SIL Loss

Use this skill when a recovery or implementation needs the exact SIL objective from Equations 1-3 of "Self-Imitation Learning". It is suitable for checking full actor-critic implementations or reduced scalar proxies.

Do not use this skill for ordinary actor-critic advantages that allow negative policy pressure; SIL clamps negative advantages to zero.

Inputs

• returns: Monte Carlo returns from replay records.
• values: current value estimates for the same states.
• action_probabilities or selected-action probabilities from the policy.
• beta: value-loss coefficient.

Outputs

• Positive advantages.
• Valid-sample mask/count.
• Policy loss, value loss, total loss.

Workflow

1. Compute advantage = return - value.
2. Clamp to positive_advantage = max(advantage, 0).
3. Compute policy_loss = mean(-log(prob_action) * positive_advantage).
4. Compute value_loss = mean(0.5 * positive_advantage^2).
5. Return policy_loss + beta * value_loss and diagnostics.

Validation

Run:

python tests/test_sil_loss.py

The test verifies exact loss values and zero contribution for non-positive advantages.

Limitations

• This script computes scalar diagnostics, not automatic differentiation.
• Callers are responsible for mapping full policy distributions to selected-action probabilities.

Zero-Pressure Ablation

If every replay return is less than or equal to the current value estimate, SIL must report zero valid samples and apply no imitation pressure. Treat this as expected behavior, not a metric failure.