---

name: define-slo-sli-sla locale: caveman source_locale: en source_commit: 82c77053 fence_basis_commit: 82c77053 translator: "Julius Brussee homage — caveman" translation_date: "2026-04-19" description: > Establish Service Level Objectives (SLO), Service Level Indicators (SLI), and Service Level Agreements (SLA) with error budget tracking, burn rate alerts, and automated reporting using Prometheus and tools like Sloth or Pyrra. Use when defining reliability targets for customer-facing services, balancing feature velocity against system reliability through error budgets, migrating from arbitrary uptime goals to data-driven metrics, or implementing Site Reliability Engineering practices. license: MIT allowed-tools: Read Write Edit Bash Grep Glob metadata: author: Philipp Thoss version: "1.0" domain: observability complexity: intermediate language: multi tags: slo, sli, sla, error-budget, burn-rate

Define SLO/SLI/SLA

Set measurable reliability targets with Service Level Objectives, track with indicators, manage error budgets.

When Use

• Define reliability targets for customer-facing services or APIs
• Set clear expectations between service providers and consumers
• Balance feature velocity with system reliability through error budgets
• Create objective criteria for incident severity and response
• Migrate from arbitrary uptime goals to data-driven reliability metrics
• Implement Site Reliability Engineering (SRE) practices
• Measure and improve service quality over time

Inputs

• Required: Service description and critical user journeys
• Required: Historical metrics data (request rates, latencies, error rates)
• Optional: Existing SLA commitments to customers
• Optional: Business requirements for service availability and performance
• Optional: Incident history and customer impact data

Steps

See Extended Examples for complete configuration files and templates.

Step 1: Grok SLI, SLO, SLA Hierarchy

Learn relationship and differences between three concepts.

Definitions:

SLI (Service Level Indicator)
- **What**: A quantitative measure of service behavior
- **Example**: Request success rate, request latency, system throughput
- **Measurement**: `successful_requests / total_requests * 100`

SLO (Service Level Objective)
- **What**: Target value or range for an SLI over a time window
- **Example**: 99.9% of requests succeed in 30-day window
- **Purpose**: Internal reliability target to guide operations

SLA (Service Level Agreement)
- **What**: Contractual commitment with consequences for missing SLO
- **Example**: 99.9% uptime SLA with refunds if breached
- **Purpose**: External promise to customers with penalties

Hierarchy:

SLA (99.9% uptime, customer refunds)
  ├─ SLO (99.95% success rate, internal target)
  │   └─ SLI (actual measured: 99.97% success rate)
  └─ Error Budget (0.05% failures allowed per month)

Key rule: SLO must be stricter than SLA. Gives buffer before customer impact.

Example:

• SLA: 99.9% availability (customer promise)
• SLO: 99.95% availability (internal target)
• Buffer: 0.05% cushion before SLA breach

Got: Team groks differences. Agreement on which metrics become SLIs. Alignment on SLO targets.

If fail:

• Review Google SRE book chapters on SLI/SLO/SLA
• Run workshop with stakeholders to align on definitions
• Start with simple success-rate SLI before complex latency SLOs

Step 2: Pick SLIs

Pick SLIs reflecting user experience and business impact.

Four Golden Signals (Google SRE):

1. Latency: Time to serve request

   # P95 latency
   histogram_quantile(0.95,
     sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
   )
   

2. Traffic: Demand on system

   # Requests per second
   sum(rate(http_requests_total[5m]))
   

3. Errors: Rate of failed requests

   # Error rate percentage
   sum(rate(http_requests_total{status=~"5.."}[5m]))
   / sum(rate(http_requests_total[5m])) * 100
   

4. Saturation: How "full" system is

   # CPU saturation
   avg(rate(node_cpu_seconds_total{mode!="idle"}[5m]))
   

Common SLI patterns:

# Availability SLI
availability:
  description: "Percentage of successful requests"
  query: |
    sum(rate(http_requests_total{status!~"5.."}[5m]))
    / sum(rate(http_requests_total[5m]))
  good_threshold: 0.999  # 99.9%

# Latency SLI
latency:
  description: "P99 request latency under 500ms"
  query: |
    histogram_quantile(0.99,
      sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
    ) < 0.5
  good_threshold: 0.95  # 95% of windows meet target

# Throughput SLI
throughput:
  description: "Requests processed per second"
  query: |
    sum(rate(http_requests_total[5m]))
  good_threshold: 1000  # Minimum 1000 req/s

# Data freshness SLI (for batch jobs)
freshness:
  description: "Data updated within last hour"
  query: |
    (time() - max(data_last_updated_timestamp)) < 3600
  good_threshold: 1  # Always fresh

SLI pick criteria:

• User-visible: Reflects actual user experience
• Measurable: Quantifiable from existing metrics
• Actionable: Team improves through engineering work
• Meaningful: Correlates with customer satisfaction
• Simple: Easy to grok and explain

Avoid:

• Internal system metrics not visible to users (CPU, memory)
• Vanity metrics not predicting customer impact
• Overly complex composite scores

Got: 2-4 SLIs picked per service. Covers availability and latency at minimum. Team agreement on measurement queries.

If fail:

• Map user journey. Find critical failure points
• Analyze incident history. Which metrics predicted customer impact?
• Validate SLI with A/B test. Degrade metric, measure customer complaints
• Start with simple availability SLI. Add complexity iteratively

Step 3: Set SLO Targets and Time Windows

Define realistic, achievable reliability targets.

SLO spec format:

service: user-api
slos:
  - name: availability
    objective: 99.9
    description: |
      99.9% of requests return non-5xx status codes
# ... (see EXAMPLES.md for complete configuration)

Time window selection:

Common windows:

• 30 days (monthly): Typical for external SLAs
• 7 days (weekly): Faster feedback for engineering teams
• 1 day (daily): High-frequency services needing rapid response

Example 30-day window error budget:

SLO: 99.9% availability over 30 days
Allowed failures: 0.1%
Total requests per month: 100M
Error budget: 100,000 failed requests
Daily budget: ~3,333 failed requests

Setting realistic targets:

1. Baseline current performance:

   # Check actual availability over past 90 days
   avg_over_time(
     (sum(rate(http_requests_total{status!~"5.."}[5m]))
     / sum(rate(http_requests_total[5m])))[90d:5m]
   )
   # Result: 99.95% → Set SLO at 99.9% (safer than current)
   

2. Calculate cost of nines:

   99%    → 7.2 hours downtime/month (low reliability)
   99.9%  → 43 minutes downtime/month (good)
   99.95% → 22 minutes downtime/month (very good)
   99.99% → 4.3 minutes downtime/month (expensive)
   99.999% → 26 seconds downtime/month (very expensive)
   

3. Balance user happiness vs engineering cost:

   • Too strict: Expensive. Slows feature development
   • Too loose: Poor user experience. Customer churn
   • Sweet spot: Slightly better than user expectations

Got: SLO targets set with business stakeholder buy-in. Documented with rationale. Error budget calculated.

If fail:

• Start with achievable target (e.g., 99% if current is 98.5%)
• Iterate SLO targets quarterly based on actual performance
• Get executive sponsorship for realistic targets vs "five nines" demands
• Document cost-benefit analysis for each additional nine

Step 4: Do SLO Monitoring with Sloth

Use Sloth to generate Prometheus recording rules and alerts from SLO specs.

Install Sloth:

# Binary installation
wget https://github.com/slok/sloth/releases/download/v0.11.0/sloth-linux-amd64
chmod +x sloth-linux-amd64
sudo mv sloth-linux-amd64 /usr/local/bin/sloth

# Or Docker
docker pull ghcr.io/slok/sloth:latest

Create Sloth SLO spec (slos/user-api.yml):

version: "prometheus/v1"
service: "user-api"
labels:
  team: "platform"
  tier: "1"
slos:
# ... (see EXAMPLES.md for complete configuration)

Generate Prometheus rules:

# Generate recording and alerting rules
sloth generate -i slos/user-api.yml -o prometheus/rules/user-api-slo.yml

# Validate generated rules
promtool check rules prometheus/rules/user-api-slo.yml

Generated recording rules (excerpt):

groups:
  - name: sloth-slo-sli-recordings-user-api-requests-availability
    interval: 30s
    rules:
      # SLI: Ratio of good events
      - record: slo:sli_error:ratio_rate5m
# ... (see EXAMPLES.md for complete configuration)

Generated alerts:

groups:
  - name: sloth-slo-alerts-user-api-requests-availability
    rules:
      # Fast burn: 2% budget consumed in 1 hour
      - alert: UserAPIHighErrorRate
        expr: |
# ... (see EXAMPLES.md for complete configuration)

Load rules into Prometheus:

# prometheus.yml
rule_files:
  - "rules/user-api-slo.yml"

Reload Prometheus:

curl -X POST http://localhost:9090/-/reload

Got: Sloth generates multi-window multi-burn-rate alerts. Recording rules evaluate successfully. Alerts fire during incidents.

If fail:

• Validate YAML syntax with yamllint slos/user-api.yml
• Check Sloth version compatibility (v0.11+ recommended)
• Verify Prometheus recording rule evaluation: curl http://localhost:9090/api/v1/rules
• Test with synthetic error injection to trigger alerts
• Check Sloth docs for SLI event query format

Step 5: Build Error Budget Dashboards

Visualize SLO compliance and error budget consumption in Grafana.

Grafana dashboard JSON (excerpt):

{
  "dashboard": {
    "title": "SLO Dashboard - User API",
    "panels": [
      {
        "type": "stat",
# ... (see EXAMPLES.md for complete configuration)

Key metrics to visualize:

• SLO target vs current SLI
• Error budget remaining (percentage and absolute)
• Burn rate (how fast budget depletes)
• Historical SLI trends (30-day rolling window)
• Time to exhaustion (if current burn rate continues)

Error budget policy dashboard (markdown panel):

## Error Budget Policy

**Current Status**: 78% budget remaining

### If Error Budget > 50%
- ✅ Full speed ahead on new features
# ... (see EXAMPLES.md for complete configuration)

Got: Dashboards show real-time SLO compliance. Error budget depletion visible. Team makes informed decisions about feature velocity.

If fail:

• Verify recording rules exist: curl http://localhost:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.name | contains("slo:"))'
• Check Prometheus datasource in Grafana has correct URL
• Validate query results in Explore view before adding to dashboard
• Set time range to appropriate window (e.g., 30d for monthly SLOs)

Step 6: Set Error Budget Policy

Define org process for managing error budgets.

Error budget policy template:

service: user-api
slo:
  availability: 99.9%
  latency_p99: 200ms
  window: 30 days

# ... (see EXAMPLES.md for complete configuration)

Automate policy enforcement:

# Example: Deployment gate script
import requests
import sys

def check_error_budget(service):
    # Query Prometheus for error budget
# ... (see EXAMPLES.md for complete configuration)

Integrate into CI/CD pipeline:

# .github/workflows/deploy.yml
jobs:
  check-error-budget:
    runs-on: ubuntu-latest
    steps:
      - name: Check SLO Error Budget
        run: |
          python scripts/check_error_budget.py user-api
      - name: Deploy
        if: success()
        run: |
          kubectl apply -f deploy/

Got: Clear policy documented. Automated gates prevent risky deployments during budget depletion. Team aligned on reliability priorities.

If fail:

• Start with manual policy enforcement (Slack reminders)
• Gradually automate with soft gates (warnings, not blocks)
• Get executive buy-in before hard gates (blocking deployments)
• Review policy effectiveness quarterly, adjust thresholds as needed

Checks

• SLIs picked reflect user experience and business impact
• SLO targets set with stakeholder agreement and documented rationale
• Prometheus recording rules generate SLI metrics successfully
• Multi-burn-rate alerts configured and tested with synthetic errors
• Grafana dashboards show real-time SLO compliance and error budget
• Error budget policy documented and communicated to team
• Automated gates prevent risky deployments during budget depletion
• Weekly/monthly SLO review meetings scheduled
• Incident retrospectives include SLO impact analysis
• SLO compliance reports shared with stakeholders

Pitfalls

• Overly strict SLOs: "Five nines" without cost analysis leads to burnout and slowed feature velocity. Start achievable, iterate up.
• Too many SLIs: Tracking 10+ indicators creates confusion. Focus on 2-4 critical user-facing metrics.
• SLO without SLA buffer: SLO equal to SLA leaves no margin before customer impact. Keep 0.05-0.1% buffer.
• Ignoring error budget: Tracking SLOs but not acting on budget depletion defeats purpose. Enforce error budget policy.
• Vanity metrics as SLIs: Internal metrics (CPU, memory) instead of user-visible metrics (latency, errors) misaligns priorities.
• No stakeholder buy-in: Engineering-only SLOs without product/business agreement leads to conflicts. Get executive sponsorship.
• Static SLOs: Never reviewing or adjusting targets as system evolves. Revisit quarterly based on actual performance and user feedback.

See Also

• setup-prometheus-monitoring - Configure Prometheus to collect metrics for SLI calculation
• configure-alerting-rules - Integrate SLO burn rate alerts with Alertmanager for on-call notifications
• build-grafana-dashboards - Visualize SLO compliance and error budget consumption
• write-incident-runbook - Include SLO impact in runbooks for prioritizing incident response