L’interface est localisée ; les présentations et documents des Skills sont traduits automatiquement par IA dans votre langue. Le code, les commandes, chemins et termes techniques restent en version originale, et le document de l’auteur fait foi.
Explorer les Skills
Productivité · SKILL.md

slo-tracking

slo-tracking

Define and track Service Level Objectives — SLI selection, error budget calculation, burn rate alerts, reporting cadence, and escalation procedures. TRIGGER when: user says /slo-tracking, asks to define SLOs, calculate error budgets, set up SLO monitoring, or establish reliability targets.

Independent GitHub snapshot · Automated classification · Original documentation governs

Documentation de la Skill

GitHub

name: slo-tracking description: > Define and track Service Level Objectives — SLI selection, error budget calculation, burn rate alerts, reporting cadence, and escalation procedures. TRIGGER when: user says /slo-tracking, asks to define SLOs, calculate error budgets, set up SLO monitoring, or establish reliability targets. argument-hint: "[service name, current SLOs, or reliability requirements]" user-invocable: true

SLO Tracking & Error Budget Management

You are a senior SRE establishing and managing SLOs. Produce well-defined SLOs with measurable SLIs, actionable error budgets, and clear escalation procedures.

Process

Step 1: Understand the Service

Before defining SLOs, determine:

  • What is the service and who are its users? (external customers, internal teams, API consumers)
  • What are the critical user journeys? (login, checkout, search, data export)
  • What are the current reliability metrics? (if any exist)
  • What is the business impact of downtime or degradation?
  • What are contractual SLA obligations? (SLOs should be stricter than SLAs)
  • What is the team's capacity to respond to reliability issues?

Step 2: Select SLIs (Service Level Indicators)

Choose SLIs based on the service type:

Service TypeRecommended SLIsMeasurement Point
Request-driven (API, web)Availability (success rate), Latency (P50, P99), ThroughputLoad balancer, application metrics
Data processing (pipeline, ETL)Freshness (data age), Correctness (validation pass rate), Coverage (% records processed)Pipeline output, validation checks
Storage (database, object store)Availability (read/write success), Durability (data loss events), LatencyClient-side instrumentation
Streaming (Kafka, event bus)Throughput (events/sec), Freshness (consumer lag), Availability (produce success)Producer/consumer metrics
Scheduled jobs (cron, batch)Execution success rate, Timeliness (completed within window), CorrectnessJob scheduler metrics

SLI Specification Template:

FieldDescriptionExample
NameDescriptive nameHTTP Availability
DescriptionWhat it measuresProportion of HTTP requests that return a non-5xx status
Good EventDefinition of a successful eventHTTP response status < 500
Valid EventAll events in scope (denominator)All HTTP requests excluding health checks
MeasurementWhere and how it is measuredLoad balancer access logs, counted per 1-minute window
Data SourceSpecific metric or log querysum(rate(http_requests_total{status!~"5.."}[1m])) / sum(rate(http_requests_total[1m]))

Step 3: Set SLO Targets

SLO TargetMonthly Downtime BudgetNinesTypical Use Case
99.0%7 hours 18 minTwo ninesInternal tools, batch processing
99.5%3 hours 39 minTwo and a half ninesInternal APIs, staging environments
99.9%43 min 50 secThree ninesCustomer-facing web services
99.95%21 min 55 secThree and a half ninesPayment processing, core APIs
99.99%4 min 23 secFour ninesInfrastructure services, DNS, auth

Guidelines for Setting Targets:

  • Start conservative (lower target) and tighten over time based on data
  • SLO target must be stricter than any external SLA commitment
  • Different SLIs for the same service can have different targets
  • Factor in planned maintenance windows
  • Consider dependencies — your SLO cannot exceed the worst-case SLO of your critical dependencies

Step 4: Calculate Error Budgets

Error Budget Formula:

Error Budget = 1 - SLO Target
Monthly Budget (minutes) = Error Budget x Total Minutes in Period

Example:
  SLO = 99.9%
  Error Budget = 0.1%
  Monthly Budget = 0.001 x 43,200 minutes = 43.2 minutes of downtime

Error Budget Tracking Table:

MetricFormulaExample (99.9% SLO, 30-day window)
Total budget(1 - SLO) x window_minutes43.2 minutes
Budget consumedbad_minutes / total_budget x 100%15 min / 43.2 min = 34.7%
Budget remainingtotal_budget - bad_minutes28.2 minutes
Projected depletionbudget_remaining / daily_burn_rate28.2 min / 0.5 min/day = 56 days
Budget statusGreen / Yellow / RedGreen (< 50% consumed with > 50% window remaining)

Budget Status Thresholds:

StatusConditionAction
GreenBudget consumed % < Window elapsed %Normal operations, feature work proceeds
YellowBudget consumed % exceeds Window elapsed % by < 20 pointsPrioritize reliability work, limit risky deploys
RedBudget consumed % exceeds Window elapsed % by > 20 pointsFreeze feature releases, all hands on reliability
Exhausted100% of error budget consumedFull feature freeze until budget recovers or window resets

Step 5: Configure Burn Rate Alerts

Burn rate alerts detect when the error budget is being consumed too quickly.

Alert NameBurn RateWindowBudget ConsumedSeverityResponse
Fast Burn (acute)14.4x1 hour2% of 30-day budgetCriticalPage on-call immediately
Fast Burn (sustained)6x6 hours5% of 30-day budgetCriticalPage on-call
Slow Burn (emerging)3x1 day10% of 30-day budgetWarningNotify team channel
Slow Burn (chronic)1.5x3 days15% of 30-day budgetWarningTriage in next planning

Multi-Window Alert Pattern (recommended): Each alert should use two windows — a long window to detect the trend and a short window to confirm it is still happening:

  • Fast burn: 1-hour long window AND 5-minute short window
  • Slow burn: 3-day long window AND 6-hour short window
  • This prevents alerting on brief spikes that have already resolved

Step 6: Establish Reporting & Escalation

Reporting Cadence:

ReportFrequencyAudienceContent
SLO DashboardReal-timeOn-call, engineeringCurrent SLI values, budget remaining, burn rate
Weekly SLO SummaryWeeklyEngineering teamBudget consumption trend, incidents impacting SLO, action items
Monthly SLO ReviewMonthlyEngineering + managementSLO attainment, budget analysis, reliability investments, target adjustments
Quarterly Business ReviewQuarterlyLeadershipSLO vs SLA, customer impact, reliability roadmap

Escalation Procedure:

  1. Budget > 50% consumed before mid-window: Engineering lead reviews active issues, prioritizes fixes
  2. Budget > 75% consumed: Skip-level escalation, defer non-critical feature work, staff reliability sprint
  3. Budget exhausted: Feature freeze communicated to product and leadership, all engineering effort on reliability
  4. SLA breach imminent: Executive escalation, customer communication plan activated, incident commander assigned

Output Format

## SLO Definitions: [Service Name]

### Service Overview
[Brief description, users, critical journeys]

### SLI Definitions
[Table of SLIs with specification details]

### SLO Targets
[Table of SLOs with targets and rationale]

### Error Budget Status
[Current budget consumption, projection, status]

### Burn Rate Alerts
[Alert definitions with thresholds and routing]

### Reporting & Governance
[Cadence, audience, escalation procedures]

### Implementation Tasks
[Prioritized checklist to implement SLO tracking]

Quality Checklist

  • SLIs measure user experience, not internal system metrics
  • SLO targets are based on data (historical reliability) and business needs, not arbitrary
  • Error budget calculation is correct and accounts for the measurement window
  • Burn rate alerts use multi-window approach to reduce false positives
  • Reporting cadence is established with clear ownership
  • Escalation procedure is documented and agreed upon with management
  • SLOs are stricter than any external SLA commitments
  • Dashboard shows real-time SLO status, error budget, and burn rate
  • Planned maintenance is accounted for in error budget calculations
  • SLO definitions are version-controlled and reviewed quarterly

Edge Cases

  • New Services Without Baseline: Run in "observation mode" for 2-4 weeks. Collect SLI data without enforcing SLOs. Set initial targets at the observed P10 (10th percentile of performance) to ensure the target is achievable.
  • Shared Dependencies: If multiple services depend on the same database or API, define SLOs for the dependency itself. Downstream service SLOs should account for dependency SLO limits.
  • Multi-Region Failover: Define SLOs at the global level (user-perceived) and per-region level (operational). Global SLO should survive a single-region failure. Regional SLO helps detect issues before they affect global numbers.
  • Seasonal Traffic: Use rolling windows instead of calendar months. Consider different SLO targets for peak vs. normal periods, or accept higher budget consumption during peak seasons.
  • Partial Degradation: Define what counts as "degraded but available" vs. "unavailable." A service returning cached/stale data may count as available for an availability SLI but bad for a correctness SLI.
  • SLO Conflicts: When improving one SLI degrades another (e.g., adding retries improves availability but increases latency), prioritize the SLI that most directly impacts user satisfaction.
Skills enregistrées