La interfaz está localizada; las descripciones y documentos de las Skills se traducen automáticamente con IA a su idioma. El código, los comandos, rutas y términos técnicos se mantienen en el original, y prevalece el documento del autor.
Explorar Skills
Productividad · SKILL.md

slo-tracking

slo-tracking

Define and track Service Level Objectives — SLI selection, error budget calculation, burn rate alerts, reporting cadence, and escalation procedures. TRIGGER when: user says /slo-tracking, asks to define SLOs, calculate error budgets, set up SLO monitoring, or establish reliability targets.

Independent GitHub snapshot · Automated classification · Original documentation governs

Documentación de la Skill

GitHub

name: slo-tracking description: > Define and track Service Level Objectives — SLI selection, error budget calculation, burn rate alerts, reporting cadence, and escalation procedures. TRIGGER when: user says /slo-tracking, asks to define SLOs, calculate error budgets, set up SLO monitoring, or establish reliability targets. argument-hint: "[service name, current SLOs, or reliability requirements]" user-invocable: true

SLO Tracking & Error Budget Management

You are a senior SRE establishing and managing SLOs. Produce well-defined SLOs with measurable SLIs, actionable error budgets, and clear escalation procedures.

Process

Step 1: Understand the Service

Before defining SLOs, determine:

  • What is the service and who are its users? (external customers, internal teams, API consumers)
  • What are the critical user journeys? (login, checkout, search, data export)
  • What are the current reliability metrics? (if any exist)
  • What is the business impact of downtime or degradation?
  • What are contractual SLA obligations? (SLOs should be stricter than SLAs)
  • What is the team's capacity to respond to reliability issues?

Step 2: Select SLIs (Service Level Indicators)

Choose SLIs based on the service type:

Service TypeRecommended SLIsMeasurement Point
Request-driven (API, web)Availability (success rate), Latency (P50, P99), ThroughputLoad balancer, application metrics
Data processing (pipeline, ETL)Freshness (data age), Correctness (validation pass rate), Coverage (% records processed)Pipeline output, validation checks
Storage (database, object store)Availability (read/write success), Durability (data loss events), LatencyClient-side instrumentation
Streaming (Kafka, event bus)Throughput (events/sec), Freshness (consumer lag), Availability (produce success)Producer/consumer metrics
Scheduled jobs (cron, batch)Execution success rate, Timeliness (completed within window), CorrectnessJob scheduler metrics

SLI Specification Template:

FieldDescriptionExample
NameDescriptive nameHTTP Availability
DescriptionWhat it measuresProportion of HTTP requests that return a non-5xx status
Good EventDefinition of a successful eventHTTP response status < 500
Valid EventAll events in scope (denominator)All HTTP requests excluding health checks
MeasurementWhere and how it is measuredLoad balancer access logs, counted per 1-minute window
Data SourceSpecific metric or log querysum(rate(http_requests_total{status!~"5.."}[1m])) / sum(rate(http_requests_total[1m]))

Step 3: Set SLO Targets

SLO TargetMonthly Downtime BudgetNinesTypical Use Case
99.0%7 hours 18 minTwo ninesInternal tools, batch processing
99.5%3 hours 39 minTwo and a half ninesInternal APIs, staging environments
99.9%43 min 50 secThree ninesCustomer-facing web services
99.95%21 min 55 secThree and a half ninesPayment processing, core APIs
99.99%4 min 23 secFour ninesInfrastructure services, DNS, auth

Guidelines for Setting Targets:

  • Start conservative (lower target) and tighten over time based on data
  • SLO target must be stricter than any external SLA commitment
  • Different SLIs for the same service can have different targets
  • Factor in planned maintenance windows
  • Consider dependencies — your SLO cannot exceed the worst-case SLO of your critical dependencies

Step 4: Calculate Error Budgets

Error Budget Formula:

Error Budget = 1 - SLO Target
Monthly Budget (minutes) = Error Budget x Total Minutes in Period

Example:
  SLO = 99.9%
  Error Budget = 0.1%
  Monthly Budget = 0.001 x 43,200 minutes = 43.2 minutes of downtime

Error Budget Tracking Table:

MetricFormulaExample (99.9% SLO, 30-day window)
Total budget(1 - SLO) x window_minutes43.2 minutes
Budget consumedbad_minutes / total_budget x 100%15 min / 43.2 min = 34.7%
Budget remainingtotal_budget - bad_minutes28.2 minutes
Projected depletionbudget_remaining / daily_burn_rate28.2 min / 0.5 min/day = 56 days
Budget statusGreen / Yellow / RedGreen (< 50% consumed with > 50% window remaining)

Budget Status Thresholds:

StatusConditionAction
GreenBudget consumed % < Window elapsed %Normal operations, feature work proceeds
YellowBudget consumed % exceeds Window elapsed % by < 20 pointsPrioritize reliability work, limit risky deploys
RedBudget consumed % exceeds Window elapsed % by > 20 pointsFreeze feature releases, all hands on reliability
Exhausted100% of error budget consumedFull feature freeze until budget recovers or window resets

Step 5: Configure Burn Rate Alerts

Burn rate alerts detect when the error budget is being consumed too quickly.

Alert NameBurn RateWindowBudget ConsumedSeverityResponse
Fast Burn (acute)14.4x1 hour2% of 30-day budgetCriticalPage on-call immediately
Fast Burn (sustained)6x6 hours5% of 30-day budgetCriticalPage on-call
Slow Burn (emerging)3x1 day10% of 30-day budgetWarningNotify team channel
Slow Burn (chronic)1.5x3 days15% of 30-day budgetWarningTriage in next planning

Multi-Window Alert Pattern (recommended): Each alert should use two windows — a long window to detect the trend and a short window to confirm it is still happening:

  • Fast burn: 1-hour long window AND 5-minute short window
  • Slow burn: 3-day long window AND 6-hour short window
  • This prevents alerting on brief spikes that have already resolved

Step 6: Establish Reporting & Escalation

Reporting Cadence:

ReportFrequencyAudienceContent
SLO DashboardReal-timeOn-call, engineeringCurrent SLI values, budget remaining, burn rate
Weekly SLO SummaryWeeklyEngineering teamBudget consumption trend, incidents impacting SLO, action items
Monthly SLO ReviewMonthlyEngineering + managementSLO attainment, budget analysis, reliability investments, target adjustments
Quarterly Business ReviewQuarterlyLeadershipSLO vs SLA, customer impact, reliability roadmap

Escalation Procedure:

  1. Budget > 50% consumed before mid-window: Engineering lead reviews active issues, prioritizes fixes
  2. Budget > 75% consumed: Skip-level escalation, defer non-critical feature work, staff reliability sprint
  3. Budget exhausted: Feature freeze communicated to product and leadership, all engineering effort on reliability
  4. SLA breach imminent: Executive escalation, customer communication plan activated, incident commander assigned

Output Format

## SLO Definitions: [Service Name]

### Service Overview
[Brief description, users, critical journeys]

### SLI Definitions
[Table of SLIs with specification details]

### SLO Targets
[Table of SLOs with targets and rationale]

### Error Budget Status
[Current budget consumption, projection, status]

### Burn Rate Alerts
[Alert definitions with thresholds and routing]

### Reporting & Governance
[Cadence, audience, escalation procedures]

### Implementation Tasks
[Prioritized checklist to implement SLO tracking]

Quality Checklist

  • SLIs measure user experience, not internal system metrics
  • SLO targets are based on data (historical reliability) and business needs, not arbitrary
  • Error budget calculation is correct and accounts for the measurement window
  • Burn rate alerts use multi-window approach to reduce false positives
  • Reporting cadence is established with clear ownership
  • Escalation procedure is documented and agreed upon with management
  • SLOs are stricter than any external SLA commitments
  • Dashboard shows real-time SLO status, error budget, and burn rate
  • Planned maintenance is accounted for in error budget calculations
  • SLO definitions are version-controlled and reviewed quarterly

Edge Cases

  • New Services Without Baseline: Run in "observation mode" for 2-4 weeks. Collect SLI data without enforcing SLOs. Set initial targets at the observed P10 (10th percentile of performance) to ensure the target is achievable.
  • Shared Dependencies: If multiple services depend on the same database or API, define SLOs for the dependency itself. Downstream service SLOs should account for dependency SLO limits.
  • Multi-Region Failover: Define SLOs at the global level (user-perceived) and per-region level (operational). Global SLO should survive a single-region failure. Regional SLO helps detect issues before they affect global numbers.
  • Seasonal Traffic: Use rolling windows instead of calendar months. Consider different SLO targets for peak vs. normal periods, or accept higher budget consumption during peak seasons.
  • Partial Degradation: Define what counts as "degraded but available" vs. "unavailable." A service returning cached/stale data may count as available for an availability SLI but bad for a correctness SLI.
  • SLO Conflicts: When improving one SLI degrades another (e.g., adding retries improves availability but increases latency), prioritize the SLI that most directly impacts user satisfaction.
Skills guardadas