---

name: jailbreak_proxy_evaluator description: Compute held-out benign-versus-adversarial obedience deltas and mechanism checks for safe jailbreak proxies.

Jailbreak Proxy Evaluator

Use this skill to evaluate whether an optimized visual prompt changes held-out behavior in a safe proxy experiment. It mirrors the paper's comparison between benign image and adversarial image conditions without generating or scoring harmful text.

Inputs

• target: metadata from module_plan.json.fast_recovery_target.
• protocol: output from the safe corpus protocol skill.
• benign_scores: prompt IDs mapped to numeric baseline scores.
• adversarial_scores: prompt IDs mapped to numeric optimized-prompt scores.
• Optional optimizer diagnostics such as loss decrease and prompt change.

Outputs

• metrics.obedience_delta: mean adversarial score minus mean benign score.
• category_metrics: optional per-category means and deltas.
• mechanism_checks: booleans for held-out split validity, prompt change, loss decrease, target match, and threshold success.

Workflow

1. Confirm the target metric is obedience_delta for the proxy recovery.
2. Align score dictionaries by held-out prompt ID.
3. Compute mean benign score, mean adversarial score, and their delta.
4. Aggregate category deltas using protocol categories.
5. Combine score evidence with optimizer diagnostics to decide whether the proxy threshold is met.

Validation

Run:

python scripts/proxy_evaluator.py --self-test

The tests cover metric calculation, category aggregation, and missing-score rejection.

Limitations

This evaluator cannot establish real-world jailbreak harm. It validates a mechanism-faithful, harmless proxy under soft-mode constraints.