name: qa-resilience
description: "Designs and tests distributed-system resilience. Use when adding retries, deadlines, hedging, circuit breakers, overload protection, chaos experiments, or SLO reliability gates."
compatibility: Portable core. Works on Claude Code and Codex.
version: "1.2"
last_validated: 2026-07-11
QA Resilience
Quick Reference
| Symptom | Start With |
|---|
| slow or hanging dependency | deadline and timeout budget |
| transient dependency failure | bounded retry with jitter and retry budget |
| sustained dependency failure | circuit breaker and fallback |
| rate-limited dependency | honor Retry-After, expose degraded behavior, and test quota paths intentionally |
| one bad host in a healthy pool | outlier detection or endpoint ejection |
| queue or pool saturation | bulkheads, concurrency limits, load shedding |
| static concurrency limit keeps going stale under changing capacity | adaptive concurrency limit with a static backstop; verify the goodput plateau — load-shedding-backpressure.md |
| non-critical feature outage | graceful degradation or feature flag fallback |
| resilience validation | deterministic fault injection before chaos |
When to Use This Skill
- retries, deadlines, hedging, breakers, bulkheads, and overload protection
- degraded-mode UX or API behavior
- service-mesh or gateway resilience policy
- chaos engineering, game days, DR drills, and fault injection
- release gates based on failure behavior, not only happy-path load tests
Route Elsewhere
Workflow
- Identify the critical user journeys and the dependencies that can break them.
- Define the contract per dependency:
- timeout or deadline budget
- retry ownership
- breaker or outlier policy
- concurrency and queue limits
- degraded behavior if the dependency is unavailable
- Decide where the policy lives:
- app code
- client library
- mesh or gateway
- Test in stages:
- design review or checker output establishes policy coherence only
- deterministic fault injection proves the response to a named controlled fault
- staged chaos in non-production proves bounded behavior and recovery in that environment
- narrow prod canary or game day proves only the observed target path and window, with guardrails
- Define pass or fail signals:
- burn rate
- p95 or p99
- fallback rate
- breaker transitions
- shed volume
- recovery time
- Record the injected fault, steady-state hypothesis, blast-radius limit, abort condition, observation window, recovery result, and dependencies simulated rather than observed.
Selective verification and hazard analysis
- Use formal methods when the question is
whether a recovery/failover protocol preserves an invariant under concurrent
faults, or when a counterexample is needed. Return the property, initial
states, transitions/environment assumptions, checked bound, result/trace, and
model-to-code limits. A finite invariant check does not establish liveness or
recovery time. Skip for routine timeout/retry configuration and named fault
tests whose behavior is already directly observable.
- Use safety engineering when
retries, rollback, failover, operators, or other functioning controls can
interact to cause a specified unacceptable loss, such as duplicate financial
effects or destructive recovery. Return loss → hazard → unsafe control action
and context → constraint → owner → verification evidence, including residual
unknowns. Skip for ordinary availability or latency tuning without a harmful
interaction question. Uptime and a hazard diagram do not prove safety.
Keep resilience policy, fault-injection execution, recovery evidence, and release
recommendations with this skill; use the foundation only for the named gap.
Pattern Rules
- retries happen at one layer only
- deadlines come before retries
- hedging is only for idempotent or cancellation-safe reads
- overload handling must shed early instead of collapsing late
- overload tests pass on goodput plateauing past capacity, not on "503s appeared"; shed responses are retried only at the retry-owning layer, within budget, after
Retry-After
- in regulated firms (EU DORA, UK FCA/PRA operational resilience), test each important business service against its impact tolerance, third parties included; counsel owns scoping — see disaster-recovery-testing.md
- readiness and liveness must stay bounded and shallow
- resilience behaviors belong in targeted checks; do not let rate-limit or degraded-mode coverage leak into unrelated happy-path suites
- prod experiments require blast-radius limits, abort criteria, dashboards, and owners
Failure Modes to Validate
- timeouts and deadline propagation
- retry storms and duplicate side effects
- partial dependency outages
- slow downstreams and long-tail latency
- queue buildup and connection-pool exhaustion
- one-bad-host behavior inside a pool
- degraded-mode responses and stale-data fallbacks
- rate-limit handling,
Retry-After, and client backoff expectations
- visible state convergence after recovery or backend resets
- failover and failback behavior
- metastable failure: a self-sustaining feedback loop (retry amplification, cache-miss stampede, queue backlog, connection-pool churn) that does not self-resolve after the original trigger clears — see references/cascading-failure-prevention.md
Testing Ladder
Deterministic first
- inject latency, errors, timeouts, malformed payloads, and unavailable endpoints in controlled tests
- verify the intended control activates and the wrong controls do not
Chaos second
- start in non-production
- use a small blast radius and a fixed time window
- stop immediately on error-budget or customer-impact breach
DR and game days last
- validate RTO and RPO claims explicitly
- rehearse recovery ownership, not only technical failover
Operational Guardrails
- every experiment needs a stated hypothesis and steady-state metric
- every run should capture timestamps, targets, blast radius, and dashboard links
- telemetry fields for retries, breaker transitions, hedging, shedding, and fallback are part of the resilience contract
- releases should be gated on reliability behavior, not only resource usage
Anti-Patterns
- no timeouts
- retries at every hop
- fixed-interval or unbounded retries
- fixed per-call retry count with no system-wide retry budget (does not tighten as error rate rises)
- retries without idempotency
- hedging unsafe writes
- no bulkheads or queue bounds
- deep readiness or liveness checks
- silent degraded mode
- untested failover plans
- happy-path-only load testing
Scripts
Typical usage:
python scripts/resilience_checker.py assess --input data/sample-service-profile.json
python scripts/resilience_checker.py gaps --input data/sample-service-profile.json
python scripts/resilience_checker.py report --input data/sample-service-profile.json --output resilience-report.md
Supply actual JSON booleans and explicit hedging N/A rationale. The heuristic score describes declared coverage; it does not establish fault behavior or recovery. See scripts/README.md for the input format and scoring logic.
ASCII Flow
Resilience request
-> Identify failure mode: slow, transient, sustained, overloaded, or degraded
-> Set deadlines, retry budgets, isolation, and fallback ownership
-> Add telemetry for saturation, errors, latency, and degraded behavior
-> Validate with deterministic fault injection before chaos experiments
-> Gate release on recovery evidence, SLO impact, and rollback path
-> Document runbook actions and residual risk
Navigation
Foundation applied recipes
Core references
Operational resources
Related Skills
Learnings Loop
When prior decisions or pitfalls are relevant, consult learnings.consolidated.md if present; use learnings.md only for needed history or as the available fallback. Otherwise skip both.
After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.