---

name: ppo_clipped_surrogate_objective description: Compute and validate PPO clipped probability-ratio surrogate losses and diagnostics for policy update recovery or implementation.

PPO Clipped Surrogate Objective

When To Use

Use this skill when implementing, auditing, or recovering the core objective from Proximal Policy Optimization. It is appropriate when you have action log probabilities under an old behavior policy, action log probabilities under a candidate new policy, and aligned advantage estimates.

Do not use it to collect trajectories or update model parameters; those are separate responsibilities.

Inputs

• old_log_probs: aligned numeric list of sampled-action log probabilities under the old policy.
• new_log_probs: aligned numeric list under the candidate policy.
• advantages: aligned numeric list of advantages.
• clip_epsilon: positive clipping width, typically 0.2.

Outputs

The script returns JSON with ratios, unclipped, clipped, objective_terms, mean_objective, loss, clip_fraction, and approx_kl.

Workflow

1. Freeze old-policy log probabilities before computing ratios.
2. Compute ratio = exp(new_log_prob - old_log_prob).
3. Compute unclipped = ratio * advantage.
4. Compute clipped = clip(ratio, 1-epsilon, 1+epsilon) * advantage.
5. Use minimum(unclipped, clipped) per sample and average it.
6. Minimize loss = -mean_objective if using a minimization optimizer.
7. Inspect clip_fraction and approx_kl to ensure the update is auditable.

Validation

Run:

python tests/test_clipped_surrogate.py
python scripts/compute_clipped_surrogate.py --old-log-probs '[0,0]' --new-log-probs '[0.3,-0.3]' --advantages '[1,-1]' --clip-epsilon 0.2

Limitations

This skill does not estimate advantages, sample environments, or train parameters. It focuses only on the PPO objective and diagnostics.