Déploiement et opérations · SKILL.md

incident-response-runbooks

incident-response-runbooks

Implements incident response runbooks with detection, triage, communication, resolution, and post-incident procedures for Kubernetes and cloud-native environments.

Independent GitHub snapshot · Automated classification · Original documentation governs

Ppaulpas / agent-skill-routerMIT

Documentation de la Skill

GitHub

name: incident-response-runbooks compatibility: opencode completeness: 95 content-types:

guidance
examples
do-dont
config description: Implements incident response runbooks with detection, triage, communication, resolution, and post-incident procedures for Kubernetes and cloud-native environments. license: MIT maturity: stable metadata: domain: cncf output-format: code related-skills: agent-database-admin, cncf-kubernetes-debugging, cncf-security-compliance role: implementation scope: implementation triggers: incident response, runbook creation, incident procedures, escalation procedures, incident command, postmortem, blameless postmortem, incident documentation archetypes:
- tactical anti_triggers:
- brainstorming
- vague ideation
- non-containerized architecture response_profile: verbosity: low directive_strength: high abstraction_level: operational version: "1.0.0"

Incident Response Runbooks

Implements comprehensive incident response procedures for cloud-native environments including Kubernetes clusters, microservices architectures, and infrastructure-as-code deployments. Provides structured workflows for detection, triage, communication, resolution, and post-incident review following SRE and ITIL best practices.

TL;DR Checklist

Detection: Verify alert validity, gather initial evidence, identify affected systems
Triage: Classify severity (P1-P4), assign incident commander, activate communication channels
Communication: Establish ICS structure, create incident channel, send initial status update
Resolution: Execute runbook steps, document all actions, apply remediation procedures
Escalation: Identify escalation triggers, contact on-call engineers, activate secondary teams
Recovery: Verify service restoration, monitor for recurrence, update runbooks if needed
Post-incident: Schedule blameless postmortem, collect timeline data, write incident report
Documentation: Archive incident data, update runbooks, create follow-up action items

When to Use

Use this skill when:

Creating new incident response procedures for a Kubernetes cluster or cloud-native service
Responding to a production incident requiring structured escalation and communication
Developing runbooks for common failure modes (pod crashes, network partitions, data corruption)
Establishing an incident command system for multi-team incident response
Documenting post-incident review procedures for compliance and learning
Building automation that integrates with incident response workflows

When NOT to Use

Avoid this skill for:

Simple debugging tasks — Use cncf-kubernetes-debugging skill for single-issue troubleshooting
Security vulnerability disclosures — Use cncf-security-compliance skill for CVE handling and patching
Database administration tasks — Use agent-database-admin skill for DB-specific procedures
Performance tuning without incident — Performance optimization doesn't require full incident procedures
Scheduled maintenance windows — Maintenance has different coordination requirements than incidents

Core Workflow

Detection and Validation — Confirm alert validity, gather initial evidence, identify scope. Checkpoint: Verify alert isn't a false positive before proceeding.
Triage and Severity Classification — Classify incident severity (P1-P4), assign incident commander, activate communication channels. Checkpoint: All team members have received initial notification.
Communication Setup — Establish ICS structure, create incident channel, send status updates. Checkpoint: Communication channels are established and team has acknowledgment.
Resolution Execution — Execute runbook steps, document actions in timeline, apply remediation. Checkpoint: Root cause identified and remediation steps are working.
Escalation — Identify escalation triggers, contact on-call, activate secondary teams. Checkpoint: Escalated parties have acknowledged and are engaged.
Post-Incident Review — Schedule blameless postmortem, collect data, write report. Checkpoint: Postmortem scheduled within 72 hours of resolution.

Implementation Patterns

Pattern 1: Incident Severity Classification

Incident severity determines response priority, communication channels, and escalation procedures.

# Severity classification criteria
# P1: Full service outage affecting all users
# P2: Degraded service affecting major functionality
# P3: Minor degradation affecting limited functionality
# P4: Non-impacting issues (informational)

function classify_severity() {
    local affected_users=$1
    local service_impact=$2
    local data_loss=$3
    
    # P1 criteria
    if [[ "$affected_users" == "all" ]] || [[ "$data_loss" == "yes" ]]; then
        echo "P1"
        return
    fi
    
    # P2 criteria
    if [[ "$affected_users" == "majority" ]] || [[ "$service_impact" == "critical" ]]; then
        echo "P2"
        return
    fi
    
    # P3 criteria
    if [[ "$affected_users" == "minority" ]] || [[ "$service_impact" == "degraded" ]]; then
        echo "P3"
        return
    fi
    
    echo "P4"
}

❌ BAD — No severity classification, ad-hoc response

# ❌ BAD — Missing severity, unclear escalation path
incident:
  title: "Database connection issues"
  status: active
  # No severity level set
  # No commander assigned
  # Communication plan missing

✅ GOOD — Severity-based response with clear escalation

# ✅ GOOD — Severity classification with escalation
severity: P2
incident_commander: john.doe@company.com
status: active
communication:
  slack_channel: "#inc-p2-database-2024-05-15"
  status_page: https://status.company.com/incidents/INC-2024-05-15-001
escalation:
  tier1_ack_timeout: 5m
  tier2_notify: 15m
  executive_notify: 30m

Pattern 2: Incident Command System (ICS) Setup

Establish ICS structure for multi-team incident response.

# ICS Roles Setup Script
# Run at incident activation

#!/bin/bash
# incident-ics-setup.sh

set -euo pipefail

# Default values
IC="${IC:-oncall@company.com}"
IO="${IO:-operations@company.com}"
LC="${LC:-leads@company.com}"
COMMS="${COMMS:-comms@company.com}"
PLANNER="${PLANNER:-planning@company.com}"

# Create ICS channel
create_ics_channel() {
    local severity=$1
    local channel_name="#inc-${severity}-$(date +%Y-%m-%d-%H%M%S)"
    
    echo "Creating ICS channel: $channel_name"
    
    # Slack workspace
    if command -v slack &> /dev/null; then
        slack channel create "$channel_name" --privacy private
    fi
    
    # Invite key personnel
    slack channel invite "$channel_name" "$IC" "$IO" "$LC" "$COMMS" "$PLANNER"
    
    echo "ICS channel created: $channel_name"
}

# Set initial ICS configuration
init_ics() {
    local severity=$1
    
    echo "Initializing ICS for severity: $severity"
    
    # Create incident channel
    create_ics_channel "$severity"
    
    # Publish ICS structure to channel
    cat << EOF | slack message "$channel_name"
### Incident Command Structure
- Incident Commander: \`$IC\`
- Operations Lead: \`$IO\`
- Logistics Lead: \`$LC\`
- Communications: \`$COMMS\`
- Planning Lead: \`$PLANNER\`

### Severity: $severity
### Status: ACTIVE
### Channel created: $(date)
EOF
}

init_ics "P2"

Pattern 3: Initial Notification and Acknowledgment

Automated notification system with acknowledgment tracking.

# incident-notification.sh
# Send initial incident notification to all stakeholders

#!/bin/bash
# Send notifications to on-call teams

declare -A NOTIFICATION_CONFIG=(
    ["slack_webhook"]="https://hooks.slack.com/services/XXXXX/XXXXX/XXXXX"
    ["pagerduty_key"]="PAGERDUTY_INTEGRATION_KEY"
    ["email_recipients"]="oncall@company.com,platform@company.com,devops@company.com"
    ["max_retries"]=3
    ["retry_delay"]=30
)

send_slack_notification() {
    local severity=$1
    local message=$2
    local channel=$3
    
    curl -X POST -H 'Content-type: application/json' \
        --data "{
            \"channel\": \"$channel\",
            \"attachments\": [{
                \"color\": \"$(severity_to_color $severity)\",
                \"title\": \"INCIDENT: $severity - $(date '+%Y-%m-%d %H:%M:%S')\",
                \"text\": \"$message\",
                \"footer\": \"Incident Response System\",
                \"ts\": $(date +%s)
            }]
        }" "$SLACK_WEBHOOK_URL"
}

send_pagerduty_alert() {
    local severity=$1
    local description=$2
    local service_key=$PAGERDUTY_SERVICE_KEY
    
    curl -X POST -H 'Content-type: application/json' \
        --data "{
            \"routing_key\": \"$service_key\",
            \"event_action\": \"trigger\",
            \"payload\": {
                \"summary\": \"$description\",
                \"severity\": \"$(pd_severity $severity)\",
                \"source\": \"incident-response-system\"
            }
        }" "https://events.pagerduty.com/v2/enqueue"
}

send_email_notification() {
    local severity=$1
    local subject=$2
    local body=$3
    
    for recipient in ${EMAIL_RECIPIENTS//,/ }; do
        mail -s "[INCIDENT][${severity}] ${subject}" "$recipient" <<< "$body"
    done
}

acknowledge_all() {
    local incident_id=$1
    local timeout=$2
    
    echo "Waiting for acknowledgments with ${timeout}s timeout..."
    
    local start_time=$(date +%s)
    while true; do
        local ack_count=$(get_acknowledged_count "$incident_id")
        local total_count=$(get_expected_recipients "$incident_id")
        
        if [[ "$ack_count" -eq "$total_count" ]]; then
            echo "All recipients acknowledged"
            return 0
        fi
        
        local elapsed=$(($(date +%s) - start_time))
        if [[ "$elapsed" -gt "$timeout" ]]; then
            echo "Timeout waiting for acknowledgments"
            return 1
        fi
        
        sleep 5
    done
}

acknowledge_all "INC-2024-05-15-001" 300

Pattern 4: Communication Status Updates

Automated status updates to keep stakeholders informed.

# incident-status-update.sh
# Generate and distribute incident status updates

#!/bin/bash

# Status template
generate_status_update() {
    local incident_id=$1
    local status=$2
    local impact=$3
    local timeline=$4
    
    cat << EOF
### ${incident_id} Status Update - $(date '+%Y-%m-%d %H:%M:%S')

**Current Status:** ${status}

**Impact:** ${impact}

**Timeline:**
${timeline}

**Next Update:** In 15 minutes or when status changes

---
*This is an automated status update from the Incident Response System*
EOF
}

# Status transition rules
update_status() {
    local incident_id=$1
    local new_status=$2
    
    # Validate status transition
    case "$new_status" in
        "identified")
            if [[ "$CURRENT_STATUS" != "detected" ]]; then
                echo "Invalid status transition"
                return 1
            fi
            ;;
        "in-progress")
            if [[ "$CURRENT_STATUS" != "identified" ]]; then
                echo "Invalid status transition"
                return 1
            fi
            ;;
        "resolved")
            if [[ "$CURRENT_STATUS" != "in-progress" ]]; then
                echo "Invalid status transition"
                return 1
            fi
            ;;
        *)
            echo "Unknown status: $new_status"
            return 1
            ;;
    esac
    
    # Update status in incident database
    update_incident_record "$incident_id" "status" "$new_status"
    
    # Send notification
    send_status_update "$incident_id" "$new_status"
}

# Status transition states
# detected -> identified -> in-progress -> resolved
# detected -> identified -> cancelled (if not an incident)
# in-progress -> monitoring (verify stability before declaring resolved)

declare -A STATUS_TRANSITIONS=(
    ["detected"]="identified"
    ["identified"]="in-progress cancelled"
    ["in-progress"]="resolved monitoring"
    ["resolved"]="closed"
    ["monitoring"]="resolved closed"
    ["cancelled"]="closed"
)

Pattern 5: Escalation Triggers and Procedures

Automated escalation based on time, severity, or lack of response.

# escalation-manager.sh
# Manage incident escalation based on predefined rules

#!/bin/bash

# Escalation configuration
declare -A ESCALATION_RULES=(
    ["P1"]["tier1_timeout"]="5m"
    ["P1"]["tier2_notify"]="10m"
    ["P1"]["executive_notify"]="15m"
    ["P2"]["tier1_timeout"]="10m"
    ["P2"]["tier2_notify"]="20m"
    ["P2"]["executive_notify"]="30m"
    ["P3"]["tier1_timeout"]="30m"
    ["P3"]["tier2_notify"]="1h"
    ["P4"]["tier1_timeout"]="1h"
)

# Escalation contact lists
declare -A TIER1_CONTACTS=(
    ["platform"]="platform-oncall@company.com"
    ["database"]="db-oncall@company.com"
    ["security"]="security-oncall@company.com"
)

declare -A TIER2_CONTACTS=(
    ["platform"]="platform-lead@company.com,staff-engineer@company.com"
    ["database"]="dba-team@company.com"
    ["security"]="security-team@company.com"
)

# Check escalation conditions
check_escalation() {
    local incident_id=$1
    local current_time=$2
    
    local severity=$(get_incident_severity "$incident_id")
    local elapsed=$(calculate_elapsed_time "$incident_id")
    local tier1_ack=$(get_tier1_acknowledged "$incident_id")
    
    # Check tier1 timeout
    local tier1_timeout=$(get_config_value "P1" "tier1_timeout")
    if [[ "$tier1_ack" == "false" ]] && [[ "$elapsed" -gt "$tier1_timeout" ]]; then
        escalate_to_tier2 "$incident_id"
    fi
    
    # Check executive notification
    local exec_notify=$(get_config_value "$severity" "executive_notify")
    if [[ "$elapsed" -gt "$exec_notify" ]] && [[ "$status" != "resolved" ]]; then
        notify_executives "$incident_id"
    fi
}

# Escalate to next tier
escalate_to_tier2() {
    local incident_id=$1
    
    echo "Escalating $incident_id to Tier 2"
    
    # Get affected domain
    local domain=$(get_incident_domain "$incident_id")
    
    # Notify tier2 contacts
    local tier2_contacts="${TIER2_CONTACTS[$domain]}"
    for contact in ${tier2_contacts//,/ }; do
        send_notification "$contact" "ESCALATION" "$incident_id"
    done
    
    # Update incident record
    update_incident_record "$incident_id" "escalated_tier2" "true"
    log_escalation "$incident_id" "tier2" "$(date '+%Y-%m-%d %H:%M:%S')"
}

# Executive notification template
notify_executives() {
    local incident_id=$1
    local severity=$(get_incident_severity "$incident_id")
    
    if [[ "$severity" != "P1" ]] && [[ "$severity" != "P2" ]]; then
        return
    fi
    
    # Executive list
    local executives="cto@company.com,cio@company.com,ciso@company.com"
    
    # Generate executive summary
    local summary=$(generate_executive_summary "$incident_id")
    
    for exec in ${executives//,/ }; do
        send_executive_notification "$exec" "$incident_id" "$summary"
    done
}

# Automated escalation loop
run_escalation_loop() {
    echo "Starting escalation monitoring..."
    
    while true; do
        local active_incidents=$(get_active_incidents)
        
        for incident in $active_incidents; do
            check_escalation "$incident" "$(date +%s)"
        done
        
        sleep 60
    done
}

Pattern 6: Runbook Automation and Execution

Automated runbook execution with state tracking and rollback capability.

# runbook-executor.sh
# Execute incident runbooks with state management

#!/bin/bash

# Runbook registry
declare -A RUNBOOKS=(
    ["database-outage"]="runbooks/database-outage.yml"
    ["pod-crash-loop"]="runbooks/pod-crash-loop.yml"
    ["network-partition"]="runbooks/network-partition.yml"
    ["memory-exhaustion"]="runbooks/memory-exhaustion.yml"
)

# Execute runbook step
execute_step() {
    local step=$1
    local runbook_path=$2
    
    # Read step configuration
    local description=$(yq '.steps[step].description' "$runbook_path")
    local action=$(yq '.steps[step].action' "$runbook_path")
    local rollback=$(yq '.steps[step].rollback' "$runbook_path" || echo "")
    
    echo "Executing: $description"
    
    # Execute action
    if eval "$action"; then
        echo "Step completed successfully"
        return 0
    else
        echo "Step failed, attempting rollback..."
        
        if [[ -n "$rollback" ]]; then
            if eval "$rollback"; then
                echo "Rollback successful"
                return 1
            else
                echo "Rollback failed!"
                return 2
            fi
        else
            return 1
        fi
    fi
}

# Runbook execution state
declare -A RUNBOOK_STATE=(
    ["current_step"]=0
    ["total_steps"]=0
    ["started_at"]=""
    ["completed_steps"]=()
    ["failed_steps"]=()
    ["rollback_required"]=false
)

# Main runbook executor
run_runbook() {
    local runbook_name=$1
    local incident_id=$2
    
    # Get runbook path
    local runbook_path="${RUNBOOKS[$runbook_name]}"
    
    if [[ -z "$runbook_path" ]] || [[ ! -f "$runbook_path" ]]; then
        echo "Runbook not found: $runbook_name"
        return 1
    fi
    
    # Initialize state
    RUNBOOK_STATE["current_step"]=0
    RUNBOOK_STATE["total_steps"]=$(yq '.steps | length' "$runbook_path")
    RUNBOOK_STATE["started_at"]=$(date '+%Y-%m-%d %H:%M:%S')
    RUNBOOK_STATE["completed_steps"]=()
    RUNBOOK_STATE["failed_steps"]=()
    
    echo "Starting runbook: $runbook_name"
    echo "Total steps: ${RUNBOOK_STATE["total_steps"]}"
    
    # Execute each step
    local step=0
    local total_steps=${RUNBOOK_STATE["total_steps"]}
    
    while [[ $step -lt $total_steps ]]; do
        if execute_step "$step" "$runbook_path"; then
            RUNBOOK_STATE["completed_steps"]+=("$step")
            step=$((step + 1))
            RUNBOOK_STATE["current_step"]=$step
        else
            RUNBOOK_STATE["failed_steps"]+=("$step")
            RUNBOOK_STATE["rollback_required"]=true
            
            # Check if rollback is possible
            if [[ ${#RUNBOOK_STATE["completed_steps"]} -gt 0 ]]; then
                echo "Rolling back previous steps..."
                rollback_runbook "$runbook_path" "${RUNBOOK_STATE["completed_steps"][@]}"
            fi
            
            return 1
        fi
    done
    
    echo "Runbook completed successfully"
    return 0
}

# Rollback runbook steps
rollback_runbook() {
    local runbook_path=$1
    shift
    local completed_steps=("$@")
    
    # Execute steps in reverse order
    for ((i=${#completed_steps[@]}-1; i>=0; i--)); do
        local step=${completed_steps[$i]}
        local rollback_cmd=$(yq ".steps[$step].rollback // ''" "$runbook_path")
        
        if [[ -n "$rollback_cmd" ]]; then
            echo "Rolling back step $step..."
            if ! eval "$rollback_cmd"; then
                echo "WARNING: Rollback failed for step $step"
            fi
        fi
    done
}

# Runbook health check
check_runbook_health() {
    local runbook_name=$1
    
    # Check if runbook file exists
    local runbook_path="${RUNBOOKS[$runbook_name]}"
    
    if [[ -z "$runbook_path" ]] || [[ ! -f "$runbook_path" ]]; then
        echo "ERROR: Runbook not found: $runbook_name"
        return 1
    fi
    
    # Validate YAML syntax
    if ! yq '.' "$runbook_path" > /dev/null 2>&1; then
        echo "ERROR: Invalid YAML in runbook: $runbook_name"
        return 1
    fi
    
    # Check required fields
    local required_fields=("steps" "description" "severity")
    for field in "${required_fields[@]}"; do
        if ! yq -e ".$field" "$runbook_path" > /dev/null 2>&1; then
            echo "ERROR: Missing required field: $field in runbook: $runbook_name"
            return 1
        fi
    done
    
    echo "Runbook $runbook_name is valid"
    return 0
}

# Example runbook YAML structure
cat << 'RUNBOOK' > runbooks/database-outage.yml
# Database Outage Runbook
# Severity: P1
# Estimated resolution time: 30 minutes

description: "Restore database connectivity after outage"
severity: P1
estimated_resolution_minutes: 30

steps:
  - name: "Verify database connection status"
    description: "Check if database is accessible"
    action: |
      kubectl exec -n database db-checker -- ping -c 3 db-primary.database.svc.cluster.local
    rollback: ""
    expected_duration_seconds: 10

  - name: "Check database pod status"
    description: "Verify database pods are running"
    action: |
      kubectl get pods -n database -l app=db-primary
    rollback: ""
    expected_duration_seconds: 5

  - name: "Check database logs for errors"
    description: "Examine database logs for root cause"
    action: |
      kubectl logs -n database db-primary-0 --tail=100 | grep -i error
    rollback: ""
    expected_duration_seconds: 15

  - name: "Restart database if needed"
    description: "Restart database pod if unhealthy"
    action: |
      kubectl delete pod -n database db-primary-0
    rollback: |
      echo "Cannot rollback pod deletion, will recreate from cluster"
    expected_duration_seconds: 60

  - name: "Verify replication is healthy"
    description: "Check database replication status"
    action: |
      kubectl exec -n database db-primary-0 -- psql -c "SELECT * FROM pg_stat_replication;"
    rollback: ""
    expected_duration_seconds: 10

  - name: "Run database health check"
    description: "Execute database health check suite"
    action: |
      kubectl exec -n database health-checker -- /health-check.sh
    rollback: ""
    expected_duration_seconds: 30
RUNBOOK

Pattern 7: Incident Timeline and Documentation

Automated timeline tracking and incident documentation.

# incident-timeline.sh
# Track incident timeline and generate documentation

#!/bin/bash

# Timeline event structure
# timestamp: ISO 8601 format
# type: detection|acknowledgment|escalation|action|resolution|communication
# actor: user/system that performed the action
# description: human-readable event description
# metadata: additional context (JSON)

# Timeline storage
declare -a TIMELINE_EVENTS=()

# Add timeline event
add_timeline_event() {
    local incident_id=$1
    local event_type=$2
    local actor=$3
    local description=$4
    local metadata=$5
    
    local event={
        "timestamp": "$(date -Iseconds)",
        "type": "$event_type",
        "actor": "$actor",
        "description": "$description",
        "metadata": "${metadata:-{}}"
    }
    
    TIMELINE_EVENTS+=("$event")
    log_to_timeline_file "$incident_id" "$event"
}

# Log to timeline file
log_to_timeline_file() {
    local incident_id=$1
    local event=$2
    local timeline_file="/var/log/incidents/${incident_id}/timeline.json"
    
    # Ensure directory exists
    mkdir -p "$(dirname "$timeline_file")"
    
    # Append event
    echo "$event" >> "$timeline_file"
}

# Generate incident report
generate_incident_report() {
    local incident_id=$1
    local report_file="/var/log/incidents/${incident_id}/report.md"
    
    local start_time=$(get_incident_start_time "$incident_id")
    local resolution_time=$(get_incident_resolution_time "$incident_id")
    local severity=$(get_incident_severity "$incident_id")
    
    cat > "$report_file" << EOF
# Incident Report: ${incident_id}

**Severity:** ${severity}
**Status:** Resolved
**Start Time:** ${start_time}
**Resolution Time:** ${resolution_time}

## Executive Summary

[Executive summary goes here]

## Timeline

| Time | Event | Actor | Details |
|------|-------|-------|---------|
EOF
    
    # Add timeline events
    for event in "${TIMELINE_EVENTS[@]}"; do
        local ts=$(echo "$event" | jq -r '.timestamp')
        local type=$(echo "$event" | jq -r '.type')
        local actor=$(echo "$event" | jq -r '.actor')
        local desc=$(echo "$event" | jq -r '.description' | tr '|' '-')
        
        echo "| ${ts} | ${type} | ${actor} | ${desc} |" >> "$report_file"
    done
    
    cat >> "$report_file" << EOF

## Root Cause Analysis

[Root cause analysis goes here]

## Impact Assessment

- Affected services: [list]
- Affected users: [estimate]
- Data loss: [none/minimal/significant]
- Business impact: [description]

## Resolution Steps

1. [Step 1]
2. [Step 2]
3. [Step 3]

## Lessons Learned

1. [Lesson 1]
2. [Lesson 2]
3. [Lesson 3]

## Follow-up Actions

- [ ] Action item 1
- [ ] Action item 2
- [ ] Action item 3

## Related Documentation

- Runbook: [link]
- Monitoring dashboard: [link]
- Postmortem: [link]
EOF
}

# Timeline query function
query_timeline() {
    local incident_id=$1
    local event_type=$2
    local start_time=$3
    local end_time=$4
    
    local timeline_file="/var/log/incidents/${incident_id}/timeline.json"
    
    if [[ ! -f "$timeline_file" ]]; then
        echo "Timeline file not found"
        return 1
    fi
    
    # Filter events
    if [[ -n "$event_type" ]]; then
        jq -c ".[] | select(.type == \"$event_type\")" "$timeline_file"
    else
        cat "$timeline_file"
    fi
}

# Example timeline events for P1 incident
cat << 'TIMELINE_EXAMPLE'
Timeline Event Examples:

1. Detection Event:
{
  "timestamp": "2024-05-15T14:32:17+00:00",
  "type": "detection",
  "actor": "prometheus",
  "description": "Database connection pool exhausted - 100% utilization",
  "metadata": {
    "alert_name": "database_connection_pool_exhausted",
    "severity": "critical",
    "current_value": 100,
    "threshold": 90
  }
}

2. Acknowledgment Event:
{
  "timestamp": "2024-05-15T14:33:05+00:00",
  "type": "acknowledgment",
  "actor": "john.doe@company.com",
  "description": "Acknowledged P1 incident",
  "metadata": {
    "channel": "slack",
    "incident_commander": "john.doe@company.com"
  }
}

3. Action Event:
{
  "timestamp": "2024-05-15T14:35:22+00:00",
  "type": "action",
  "actor": "oncall-engineer",
  "description": "Restarted database pod db-primary-0",
  "metadata": {
    "command": "kubectl delete pod -n database db-primary-0",
    "pod_before": "Running",
    "pod_after": "Pending"
  }
}

4. Resolution Event:
{
  "timestamp": "2024-05-15T14:47:11+00:00",
  "type": "resolution",
  "actor": "john.doe@company.com",
  "description": "Service restored - database connection pool at 45%",
  "metadata": {
    "resolution_method": "pod_restart",
    "connection_pool_usage": 45,
    "affected_endpoints": 0
  }
}

5. Communication Event:
{
  "timestamp": "2024-05-15T14:34:00+00:00",
  "type": "communication",
  "actor": "comms@company.com",
  "description": "Initial incident status sent to stakeholders",
  "metadata": {
    "channel": "slack",
    "recipients": 150,
    "status_page_updated": true
  }
}
TIMELINE_EXAMPLE

Pattern 8: Post-Incident Review and Blameless Postmortem

Automated post-incident review scheduling and blameless postmortem template.

# postmortem-manager.sh
# Manage post-incident review process

#!/bin/bash

# Post-incident review configuration
POSTMORTEM_CONFIG=(
    "max_delay_hours=72"
    "required_participants=incident_commander,operations_lead,technical_lead"
    "deadline_hours=72"
    "template_path=/templates/postmortem-template.md"
)

# Create postmortem ticket
create_postmortem_ticket() {
    local incident_id=$1
    
    local jira_project="INC"
    local summary="Postmortem: ${incident_id} - [Incident Title]"
    local description="Schedule post-incident review for ${incident_id}"
    local labels="postmortem,incident,${incident_id}"
    
    # Create Jira ticket
    local ticket=$(curl -s -X POST \
        -H "Authorization: Bearer $JIRA_TOKEN" \
        -H "Content-Type: application/json" \
        --data "{
            \"fields\": {
                \"project\": {\"key\": \"$jira_project\"},
                \"summary\": \"$summary\",
                \"description\": \"$description\",
                \"issuetype\": {\"name\": \"Incident Review\"},
                \"labels\": [$labels]
            }
        }" "$JIRA_URL/rest/api/2/issue")
    
    echo "$ticket" | jq -r '.key'
}

# Schedule postmortem meeting
schedule_postmortem() {
    local incident_id=$1
    local ticket_id=$2
    
    # Calculate review date (within 72 hours)
    local review_date=$(date -d "+2 days" '+%Y-%m-%d')
    local review_time="10:00"
    
    # Find participants
    local participants=$(get_postmortem_participants "$incident_id")
    
    # Create calendar event
    for participant in ${participants//,/ }; do
        send_calendar_invite "$participant" \
            "Postmortem: ${incident_id}" \
            "$review_date $review_time" \
            "$ticket_id"
    done
    
    echo "Postmortem scheduled: ${review_date} ${review_time}"
    echo "Participants: $participants"
}

# Blameless postmortem template
generate_postmortem_template() {
    local incident_id=$1
    local template_path=$2
    
    cat > "$template_path" << 'POSTMORTEM'
# Incident Postmortem: {INCIDENT_ID}

**Date:** {DATE}
**Incident Commander:** {IC_NAME}
**Severity:** {SEVERITY}
**Duration:** {DURATION}

---

## Executive Summary

A concise, high-level summary of what happened, when, and the impact. Focus on outcomes, not technical details.

---

## Timeline

| Time (UTC) | Event |
|------------|-------|
| {START_TIME} | Incident began - initial detection |
| {ACK_TIME} | Incident acknowledged |
| {ACTION_TIME} | First remediation action taken |
| {RESOLUTION_TIME} | Service restored |
| {VERIFICATION_TIME} | Verification completed |

---

## Root Cause Analysis

### Primary Cause

What was the fundamental technical issue that caused the incident?

### Contributing Factors

1. [Factor 1]
2. [Factor 2]
3. [Factor 3]

### Root Cause Tree

```mermaid
graph TD
    A[Root Cause] --> B[Immediate Cause]
    B --> C[Symptoms]
    C --> D[Impact]

Impact Assessment

Services Affected

Service 1
Service 2

User Impact

Number of affected users: {ESTIMATE}
Duration of impact: {DURATION}
Data loss: {NONE/MINIMAL/SIGNIFICANT}

Business Impact

{BUSINESS_IMPACT_DESCRIPTION}

Detection and Response

Detection

How was the incident detected?
Time to detection (TTD): {TIME}
Alert effectiveness: {EFFECTIVENESS_RATING}

Response

Time to acknowledgment (TTA): {TIME}
Time to resolution (TTR): {TIME}
Response effectiveness: {EFFECTIVENESS_RATING}

Resolution Steps

Step 1: {Description}
Step 2: {Description}
Step 3: {Description}

Lessons Learned

What Went Well

[Positive 1]
[Positive 2]

What Went Wrong

[Issue 1]
[Issue 2]

What We Learned

[Learning 1]
[Learning 2]

Action Items

ID	Action Item	Owner	Due Date	Status
A1	{Action}	{OWNER}	{DATE}	Open
A2	{Action}	{OWNER}	{DATE}	Open
A3	{Action}	{OWNER}	{DATE}	Open

Follow-up Schedule

1 week: Review action item progress
1 month: Verify fixes are effective
3 months: Comprehensive review of all action items

Appendices

Technical Details

{ADDITIONAL_TECHNICAL_INFORMATION}

Communication Log

Time	Channel	Recipients	Message
{TIME}	{CHANNEL}	{RECIPIENTS}	{MESSAGE}

This postmortem is written in a blameless manner. The focus is on understanding systemic factors and improving processes, not assigning individual blame. POSTMORTEM }

Postmortem completion checklist

check_postmortem_completion() { local incident_id=$1 local postmortem_file="/var/log/incidents/${incident_id}/postmortem.md"

local missing_items=()

# Check required sections
for section in "Executive Summary" "Timeline" "Root Cause Analysis" \
               "Impact Assessment" "Lessons Learned" "Action Items"; do
    if ! grep -q "^## $section" "$postmortem_file"; then
        missing_items+=("$section")
    fi
done

if [[ ${#missing_items[@]} -gt 0 ]]; then
    echo "Missing sections:"
    for item in "${missing_items[@]}"; do
        echo "  - $item"
    done
    return 1
fi

# Check action items exist
local action_item_count=$(grep -c "^\| - \[ \]" "$postmortem_file" || echo 0)
if [[ "$action_item_count" -lt 3 ]]; then
    echo "Insufficient action items (minimum 3 required)"
    return 1
fi

echo "Postmortem complete"
return 0

}

Generate postmortem summary for executives

generate_executive_summary() { local incident_id=$1 local severity=$(get_incident_severity "$incident_id") local duration=$(calculate_duration "$incident_id") local impact=$(get_business_impact "$incident_id")

cat << EOF

Executive Summary: ${incident_id}

Severity: ${severity} Duration: ${duration} Business Impact: ${impact}

A detailed postmortem has been scheduled within 72 hours. Key findings will be shared with stakeholders following the blameless review.

Next Steps:

Complete post-incident review
Implement action items from postmortem
Update runbooks and monitoring
Schedule follow-up review EOF }


---

## Constraints

### MUST DO

- **Classify severity immediately** — Determine P1-P4 within 5 minutes of detection to set appropriate response level
- **Establish ICS structure** — Assign Incident Commander, Operations Lead, and Communications within 10 minutes
- **Document everything in timeline** — Every action, decision, and communication must be recorded with timestamp
- **Use blameless language** — Focus on system and process factors, not individual blame in postmortems
- **Escalate based on time thresholds** — Don't wait for user complaints; escalate when P1/P2 timeouts are reached
- **Verify resolution with health checks** — Don't declare resolved without automated verification
- **Schedule postmortem within 72 hours** — Delayed postmortems lose accuracy and context
- **Update runbooks after each incident** — Turn lessons learned into automated runbook improvements

### MUST NOT DO

- **Never skip severity classification** — Even for minor incidents, establish a baseline for future comparison
- **Don't communicate via untracked channels** — All external communication must go through official channels (Slack, status page)
- **Don't declare resolved until verified** — Service restoration ≠ incident resolution; verify for 15+ minutes
- **Never assign blame in timeline** — Timeline records facts; postmortem analyzes system factors
- **Don't delay postmortem scheduling** — Memory fades quickly; schedule within 72 hours, conduct within 1 week
- **Never disable monitoring during incident** — Continue monitoring throughout incident; that's when you need it most
- **Don't create separate incident channels** — Use one channel per incident; fragmenting communication causes confusion
- **Don't leave action items open-ended** — Every action item must have owner, due date, and success criteria

---

## Output Template

When generating incident response documentation, include:

1. **Incident Metadata** — ID, severity, start time, resolution time, commander, status
2. **Timeline** — Chronological events with timestamps, actors, and descriptions
3. **Root Cause Analysis** — Primary cause, contributing factors, and root cause tree
4. **Impact Assessment** — Affected services, user count, duration, business impact
5. **Resolution Steps** — Step-by-step remediation actions taken
6. **Lessons Learned** — What went well, what went wrong, key takeaways
7. **Action Items** — Specific, owned, time-bound improvements to prevent recurrence
8. **Follow-up Schedule** — Checkpoints for action item progress and effectiveness verification

---

## Related Skills

| Skill | Purpose |
|---|---|
| `cncf-kubernetes-debugging` | Deep dive troubleshooting for specific Kubernetes issues; use this for root cause analysis after incident is contained |
| `cncf-security-compliance` | Security-specific incident handling including CVE response, compliance violations, and vulnerability management |
| `agent-database-admin` | Database-specific incident procedures including replication issues, connection pool problems, and backup recovery |

---

## References

### Industry Standards

- **SRE Book (Google)** - Site Reliability Engineering chapter on Postmortem Culture
- **ITIL Incident Management** - ITIL v4 incident management practices
- **NIST SP 800-61** - Computer Security Incident Handling Guide
- **Chaos Engineering** - Building resilience through controlled experiments

### CNCF Resources

- **CNCF SIG Security** - Security incident response guidelines
- **Kubernetes Incident Response** - K8s-specific incident patterns
- **Prometheus Alerting** - Best practices for alert configuration

### Templates and Tools

- **Jira Incident Template** - Standard incident tracking format
- **Status.io** - Status page integration for incident communication
- **Datadog Incident Management** - Observability-integrated incident response
- **PagerDuty Runbooks** - Runbook management and execution platform

### Example Runbooks

- **Database Outage** - Recovery procedures for primary/replica failures
- **Network Partition** - Handling split-brain scenarios in distributed systems
- **Pod Crash Loop** - Diagnosing and recovering from container crashes
- **Memory Exhaustion** - Handling OOM conditions and memory leaks

---

### Pattern 9: Kubernetes Cluster Failure Response

Procedures for handling cluster-wide failures including API server issues, etcd problems, and node failures.

```bash
# cluster-failure-response.sh
# Handle Kubernetes cluster-wide failures

#!/bin/bash

# Cluster failure severity levels
declare -A CLUSTER_FAILURE_TYPES=(
    ["api-server-down"]="CRITICAL"
    ["etcd-quorum-loss"]="CRITICAL"
    ["etcd-data-loss"]="CRITICAL"
    ["control-plane-node-failure"]="HIGH"
    ["worker-node-majority-loss"]="HIGH"
    ["network-policy-break"]="MEDIUM"
)

# Check cluster health
check_cluster_health() {
    # API server health
    if ! kubectl --request-timeout=10s get nodes &> /dev/null; then
        echo "ALERT: API server unreachable"
        return 1
    fi
    
    # etcd health
    local etcd_status=$(kubectl exec -n kube-system etcd-$(hostname) -- etcdctl endpoint health 2>/dev/null)
    if [[ "$etcd_status" != *"healthy"* ]]; then
        echo "ALERT: etcd unhealthy"
        return 1
    fi
    
    # Node status
    local ready_nodes=$(kubectl get nodes --no-headers 2>/dev/null | grep -c "Ready")
    local total_nodes=$(kubectl get nodes --no-headers 2>/dev/null | wc -l)
    
    if [[ "$ready_nodes" -lt "$((total_nodes / 2 + 1))" ]]; then
        echo "ALERT: Majority of nodes not ready"
        return 1
    fi
    
    echo "Cluster health: OK"
    return 0
}

# etcd backup procedure
etcd_backup() {
    local backup_dir="/var/etcd/backups/$(date +%Y%m%d_%H%M%S)"
    local endpoint="https://127.0.0.1:2379"
    
    mkdir -p "$backup_dir"
    
    # Get snapshot
    kubectl exec -n kube-system etcd-$(hostname) -- \
        etcdctl snapshot save "${backup_dir}/snapshot.db" \
        --endpoints="$endpoint" \
        --cacert="/etc/kubernetes/pki/etcd/ca.crt" \
        --cert="/etc/kubernetes/pki/etcd/healthcheck-client.crt" \
        --key="/etc/kubernetes/pki/etcd/healthcheck-client.key"
    
    # Backup manifests
    cp /etc/kubernetes/manifests/etcd.yaml "${backup_dir}/"
    
    echo "Backup completed: $backup_dir"
}

# etcd restore procedure
etcd_restore() {
    local snapshot_path=$1
    
    if [[ ! -f "$snapshot_path" ]]; then
        echo "ERROR: Snapshot file not found: $snapshot_path"
        return 1
    fi
    
    # Stop etcd
    kubectl delete pod -n kube-system etcd-$(hostname)
    
    # Restore snapshot
    etcdctl snapshot restore "$snapshot_path" \
        --data-dir="/var/lib/etcd-restore" \
        --initial-cluster="etcd-$(hostname)=https://127.0.0.1:2380" \
        --initial-cluster-token="etcd-cluster-$(date +%s)" \
        --initial-advertise-peer-urls="https://127.0.0.1:2380"
    
    # Update manifests
    cp /etc/kubernetes/manifests/etcd.yaml "${backup_dir}/etcd.yaml.bak"
    
    echo "etcd restore initiated. Restarting control plane..."
}

# API server failover
api_server_failover() {
    local failed_node=$1
    
    echo "Initiating API server failover for node: $failed_node"
    
    # Scale down failed node
    kubectl cordon "$failed_node"
    kubectl drain "$failed_node" --ignore-daemonsets --delete-emptydir-data --force
    
    # Restart API server on remaining nodes
    kubectl delete pod -n kube-system kube-apiserver-$(hostname)
    
    # Verify cluster health
    sleep 10
    check_cluster_health
}

# Worker node recovery
recover_worker_node() {
    local node=$1
    
    echo "Attempting to recover node: $node"
    
    # Check node status
    local node_status=$(kubectl get node "$node" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')
    
    if [[ "$node_status" == "False" ]]; then
        # Check kubelet status
        kubectl debug node/"$node" -it --image=ubuntu:22.04 -- bash -c 'systemctl status kubelet || docker ps'
        
        # Attempt restart
        kubectl cordon "$node"
        kubectl drain "$node" --ignore-daemonsets --delete-emptydir-data --force
        kubectl uncordon "$node"
    fi
}

Pattern 10: Network Partition Recovery

Procedures for handling network partitions and split-brain scenarios.

# network-partition-recovery.sh
# Handle network partitions in distributed systems

#!/bin/bash

# Network partition detection
detect_partition() {
    local target_node=$1
    
    # Check connectivity to target
    if ! ping -c 3 "$target_node" &> /dev/null; then
        echo "ALERT: Network partition detected - $target_node unreachable"
        return 1
    fi
    
    echo "Node $target_node is reachable"
    return 0
}

# Partition healing procedure
heal_partition() {
    local partitioned_node=$1
    local primary_node=$2
    
    echo "Attempting to heal partition between $partitioned_node and $primary_node"
    
    # Verify partition
    if ! check_partition "$partitioned_node" "$primary_node"; then
        echo "Confirmed: Network partition exists"
    fi
    
    # Attempt network repair
    ssh "$partitioned_node" "ip link set eth0 down; ip link set eth0 up"
    
    # Wait for network to stabilize
    sleep 5
    
    # Verify connection restored
    if check_partition "$partitioned_node" "$primary_node"; then
        echo "Partition healed successfully"
        return 0
    else
        echo "ERROR: Failed to heal partition"
        return 1
    fi
}

# Split-brain resolution for distributed databases
resolve_split_brain() {
    local database_cluster=$1
    
    echo "Resolving split-brain in database cluster: $database_cluster"
    
    # Identify primary and secondary nodes
    local primary_node=$(find_primary_node "$database_cluster")
    local secondary_nodes=$(find_secondary_nodes "$database_cluster")
    
    # Force secondary nodes to sync from primary
    for node in $secondary_nodes; do
        ssh "$node" "mysql --execute=\"STOP SLAVE; CHANGE MASTER TO MASTER_HOST='$primary_node'; START SLAVE;\""
    done
    
    # Verify replication is healthy
    sleep 10
    verify_replication_health "$database_cluster"
}

# Network policy remediation
remediate_network_policy() {
    local namespace=$1
    local policy_name=$2
    
    echo "Remediating network policy: $policy_name in namespace: $namespace"
    
    # Get current policy
    local policy=$(kubectl get networkpolicy "$policy_name" -n "$namespace" -o yaml)
    
    # Backup current policy
    echo "$policy" > "/tmp/networkpolicy_backup_${policy_name}_$(date +%Y%m%d_%H%M%S).yaml"
    
    # Apply remediation (example: allow all ingress temporarily)
    kubectl patch networkpolicy "$policy_name" -n "$namespace" --type='json' -p='[{"op": "replace", "path": "/spec/ingress", "value": null}]'
    
    # Monitor for issues
    echo "Network policy remediated. Monitoring for issues..."
    tail -f /var/log/kube-apiserver.log | grep -E "(error|denied)" &
}

Pattern 11: Memory and Resource Exhaustion Response

Procedures for handling OOM conditions, CPU saturation, and resource exhaustion.

# resource-exhaustion-response.sh
# Handle memory and CPU exhaustion incidents

#!/bin/bash

# Check pod memory usage
check_pod_memory() {
    local namespace=$1
    local pod=$2
    
    kubectl exec -n "$namespace" "$pod" -- \
        /bin/sh -c "free -m && echo '---' && cat /proc/1/status | grep -E 'VmRSS|VmSize'"
}

# OOM killer analysis
analyze_oom_killer() {
    local node=$1
    
    echo "Analyzing OOM killer events on node: $node"
    
    # Check kernel logs for OOM events
    kubectl exec -n kube-system "$node" -- \
        /bin/sh -c "dmesg | grep -i 'killed process' | tail -20"
    
    # Check cgroup OOM counts
    kubectl exec -n kube-system "$node" -- \
        /bin/sh -c "cat /sys/kernel/mm/oom_kill/oom_kill | wc -l"
}

# Pod memory limit adjustment
adjust_memory_limit() {
    local namespace=$1
    local deployment=$2
    local new_limit=$3  # e.g., 4Gi
    
    echo "Adjusting memory limit for $namespace/$deployment to $new_limit"
    
    # Patch deployment with new limit
    kubectl patch deployment "$deployment" -n "$namespace" -p "{
        \"spec\": {
            \"template\": {
                \"spec\": {
                    \"containers\": [{
                        \"name\": \"$deployment\",
                        \"resources\": {
                            \"limits\": {
                                \"memory\": \"$new_limit\"
                            }
                        }
                    }]
                }
            }
        }
    }"
    
    # Restart pods to apply new limits
    kubectl rollout restart deployment "$deployment" -n "$namespace"
}

# CPU throttling analysis
analyze_cpu_throttling() {
    local namespace=$1
    local pod=$2
    
    echo "Analyzing CPU throttling for $namespace/$pod"
    
    # Get CPU usage metrics
    local metrics=$(kubectl top pod "$pod" -n "$namespace")
    
    # Check if CPU limits are causing throttling
    local pod_info=$(kubectl get pod "$pod" -n "$namespace" -o jsonpath='{.spec.containers[0].resources.limits.cpu}')
    
    echo "CPU Limit: $pod_info"
    echo "Current Usage: $metrics"
    
    # Check cgroup throttling stats
    kubectl exec -n "$namespace" "$pod" -- \
        /bin/sh -c "cat /sys/fs/cgroup/cpu/cpu.stat | grep throttle"
}

# Resource quota enforcement
enforce_resource_quota() {
    local namespace=$1
    
    echo "Enforcing resource quota for namespace: $namespace"
    
    # Get current usage
    local usage=$(kubectl describe resourcequota -n "$namespace")
    
    # Check if over quota
    if echo "$usage" | grep -q "over quota"; then
        echo "ALERT: Namespace over quota. Initiating enforcement..."
        
        # Scale down non-critical workloads
        kubectl scale deployment --namespace="$namespace" --all --replicas=0
        
        # Keep critical services
        kubectl scale deployment api-server -n "$namespace" --replicas=3
        kubectl scale deployment database -n "$namespace" --replicas=2
    fi
}

Pattern 12: Data Corruption and Recovery

Procedures for handling data corruption incidents and recovery operations.

# data-corruption-response.sh
# Handle data corruption and recovery

#!/bin/bash

# Data integrity check
check_data_integrity() {
    local database=$1
    local table=$2
    
    echo "Checking data integrity for $database.$table"
    
    case "$database" in
        postgres)
            psql -c "CHECK TABLE $table"
            ;;
        mysql)
            mysql -e "CHECK TABLE $table EXTENDED"
            ;;
        cassandra)
            nodetool verify "$database" "$table"
            ;;
        *)
            echo "Unsupported database: $database"
            return 1
            ;;
    esac
}

# Find corrupted records
find_corruption() {
    local database=$1
    local table=$2
    local column=$3
    
    echo "Searching for corrupted records in $database.$table.$column"
    
    case "$database" in
        postgres)
            psql -c "SELECT * FROM $table WHERE $column IS NULL OR $column = '' OR $column ~ '^\s*#x27;"
            ;;
        mysql)
            mysql -e "SELECT * FROM $table WHERE $column IS NULL OR $column = '' OR $column REGEXP '^[[:space:]]*#x27;"
            ;;
    esac
}

# Data recovery from backup
recover_from_backup() {
    local backup_id=$1
    local restore_target=$2  # timestamp or specific point
    
    echo "Restoring from backup: $backup_id to target: $restore_target"
    
    # Download backup
    gsutil cp "gs://backups/db/${backup_id}.sql.gz" /tmp/backup.sql.gz
    
    # Decompress
    gunzip /tmp/backup.sql.gz
    
    # Restore to temporary database
    psql -d recovery_db < /tmp/backup.sql
    
    # Compare with current data
    compare_data "recovery_db" "production_db" "$restore_target"
    
    # Apply selective recovery
    apply_selective_recovery "recovery_db" "production_db"
}

# Data reconciliation
reconcile_data() {
    local source_db=$1
    local target_db=$2
    local table=$3
    
    echo "Reconciling $source_db.$table with $target_db.$table"
    
    # Compare row counts
    local source_count=$(psql -d "$source_db" -c "SELECT COUNT(*) FROM $table" -t)
    local target_count=$(psql -d "$target_db" -c "SELECT COUNT(*) FROM $table" -t)
    
    if [[ "$source_count" != "$target_count" ]]; then
        echo "ALERT: Row count mismatch - Source: $source_count, Target: $target_count"
        
        # Find missing records
        psql -d "$source_db" -c "SELECT * FROM $table WHERE id NOT IN (SELECT id FROM $target_db.$table)"
    fi
}

# Point-in-time recovery
pitr_recovery() {
    local database=$1
    local recovery_time=$2  # ISO 8601 format
    
    echo "Initiating point-in-time recovery for $database to $recovery_time"
    
    # Stop database
    systemctl stop postgresql
    
    # Restore base backup
    restore_base_backup
    
    # Apply WAL segments up to recovery time
    apply_wal_segments "$recovery_time"
    
    # Start database
    systemctl start postgresql
    
    # Verify recovery
    verify_recovery "$database" "$recovery_time"
}

Pattern 13: Security Incident Response

Procedures for handling security incidents including unauthorized access and data breaches.

# security-incident-response.sh
# Handle security incidents and breaches

#!/bin/bash

# Isolate compromised resource
isolate_resource() {
    local resource_type=$1  # pod, service, node
    local resource_name=$2
    
    echo "Isolating $resource_type: $resource_name"
    
    case "$resource_type" in
        pod)
            # Block all traffic to/from pod
            kubectl label pod "$resource_name" security=isolation
            kubectl apply -f - << EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-$resource_name
spec:
  podSelector:
    matchLabels:
      pod: $resource_name
  policyTypes:
  - Ingress
  - Egress
EOF
            ;;
        node)
            # Cordon node
            kubectl cordon "$resource_name"
            
            # Evict workloads
            kubectl drain "$resource_name" --ignore-daemonsets --delete-emptydir-data --force
            ;;
    esac
}

# forensic data collection
collect_forensics() {
    local resource=$1
    
    echo "Collecting forensic data for: $resource"
    
    mkdir -p "/var/forensics/${resource}/$(date +%Y%m%d_%H%M%S)"
    
    # Collect container logs
    kubectl logs "$resource" > "/var/forensics/${resource}/logs.txt" 2>&1
    
    # Collect process list
    kubectl exec "$resource" -- ps aux > "/var/forensics/${resource}/processes.txt" 2>&1
    
    # Collect network connections
    kubectl exec "$resource" -- netstat -tuln > "/var/forensics/${resource}/network.txt" 2>&1
    
    # Collect file system changes
    kubectl exec "$resource" -- find / -mtime -1 > "/var/forensics/${resource}/filesystem.txt" 2>&1
    
    # Create checksums for evidence
    find "/var/forensics/${resource}" -type f -exec md5sum {} \; > "/var/forensics/${resource}/checksums.txt"
}

# Rotate compromised credentials
rotate_credentials() {
    local service=$1
    
    echo "Rotating credentials for service: $service"
    
    # Generate new credentials
    local new_password=$(openssl rand -base64 32)
    local new_api_key=$(openssl rand -base64 32)
    
    # Update secrets
    kubectl create secret generic "${service}-credentials" \
        --from-literal="password=${new_password}" \
        --from-literal="api-key=${new_api_key}" \
        --dry-run=client -o yaml | kubectl apply -f -
    
    # Restart affected pods
    kubectl rollout restart deployment "$service"
}

# Security audit
run_security_audit() {
    echo "Running security audit..."
    
    # Check for privileged containers
    kubectl get pods --all-namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].securityContext.privileged}{"\n"}{end}' | grep "true"
    
    # Check for secrets in environment variables
    kubectl get pods --all-namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].env[*].value}{"\n"}{end}' | grep -E "(password|secret|api_key|token)"
    
    # Check for external network access
    kubectl get networkpolicies --all-namespaces
    
    # Check RBAC permissions
    kubectl get clusterrolebindings
    kubectl get rolebindings --all-namespaces
}

Pattern 14: Service Mesh Incident Response

Procedures for handling Istio/Linkerd service mesh incidents.

# service-mesh-response.sh
# Handle service mesh incidents

#!/bin/bash

# Check service mesh health
check_mesh_health() {
    echo "Checking service mesh health..."
    
    # Check Istio control plane
    kubectl get pods -n istio-system
    
    # Check sidecar injection
    kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].name}{"\n"}{end}' | grep -E "(istio-proxy|envoy)"
    
    # Check mTLS status
    istioctl proxy-status
    
    # Check pilot health
    istioctl proxy-config clusters istio-ingressgateway-$(kubectl get pods -n istio-system -l app=istio-ingressgateway -o jsonpath='{.items[0].metadata.name}') -n istio-system | head -20
}

# Reconcile service mesh configuration
reconcile_mesh_config() {
    echo "Reconciling service mesh configuration..."
    
    # Redeploy Istio control plane
    istioctl install --set profile=demo -y
    
    # Restart sidecars
    kubectl rollout restart deployment --all -n default
    
    # Verify reconciliation
    istioctl analyze
}

# Debug service mesh connectivity
debug_mesh_connectivity() {
    local source=$1
    local destination=$2
    local port=$3
    
    echo "Debugging connectivity from $source to $destination:$port"
    
    # Test connectivity from source pod
    kubectl exec "$source" -- curl -v "http://$destination:$port/health"
    
    # Check virtual service
    kubectl get virtualservice "$destination" -o yaml
    
    # Check destination rule
    kubectl get destinationrule "$destination" -o yaml
    
    # Check sidecar configuration
    kubectl get sidecar "$source" -o yaml 2>/dev/null || echo "No sidecar config found"
    
    # Check ingress gateway
    kubectl get ingressgateway -n istio-system
}

# Reset service mesh state
reset_mesh_state() {
    echo "Resetting service mesh state..."
    
    # Clear Envoy caches
    for pod in $(kubectl get pods -n istio-system -o jsonpath='{.items[*].metadata.name}'); do
        kubectl exec -n istio-system "$pod" -- pilot-agent request -s "127.0.0.1:15000" "reset"
    done
    
    # Restart Envoy sidecars
    kubectl delete pod -l istio=sidecar --force --grace-period=0
    
    # Wait for restart
    sleep 30
}

Pattern 15: CI/CD Pipeline Incident Response

Procedures for handling CI/CD pipeline failures and deployment incidents.

# cicd-response.sh
# Handle CI/CD pipeline incidents

#!/bin/bash

# Check pipeline health
check_pipeline_health() {
    echo "Checking CI/CD pipeline health..."
    
    # Check Jenkins
    curl -s "http://jenkins:8080/api/json" | jq -r '.computer[0].offline'
    
    # Check GitHub Actions runners
    curl -s "https://api.github.com/repos/org/repo/actions/runners" | jq '.total_count'
    
    # Check Helm registry
    curl -s "https://registry.helm.sh/index.yaml" | jq -r 'keys'
    
    # Check container registry
    curl -s "https://gcr.io/v2/_catalog" | jq '.repositories'
}

# Rollback failed deployment
rollback_deployment() {
    local deployment=$1
    local namespace=$2
    
    echo "Rolling back $deployment in $namespace"
    
    # Get current revision
    local current_revision=$(kubectl rollout history deployment "$deployment" -n "$namespace" | tail -1 | cut -d' ' -f1)
    
    # Rollback to previous version
    kubectl rollout undo deployment "$deployment" -n "$namespace"
    
    # Verify rollback
    kubectl rollout status deployment "$deployment" -n "$namespace"
    
    # Check health
    kubectl get pods -l app="$deployment" -n "$namespace"
}

# Rebuild pipeline cache
rebuild_pipeline_cache() {
    echo "Rebuilding pipeline cache..."
    
    # Clear Jenkins cache
    curl -X POST "http://jenkins:8080/pluginManager/installNecessaryPlugins"
    
    # Clear Docker registry cache
    curl -X POST "http://registry:5000/v2/_flush_cache"
    
    # Rebuild artifact repository cache
    curl -X POST "http://nexus:8081/service/local/cache/rebuild_all"
}

# Debug pipeline failure
debug_pipeline_failure() {
    local pipeline_id=$1
    
    echo "Debugging pipeline failure: $pipeline_id"
    
    # Get pipeline logs
    curl "http://jenkins:8080/job/pipeline/${pipeline_id}/consoleText" | tee "/tmp/pipeline_${pipeline_id}.log"
    
    # Analyze failure
    grep -E "(ERROR|FAILURE|Exception)" "/tmp/pipeline_${pipeline_id}.log" | tail -20
    
    # Check resource limits
    kubectl get resourcequota -n ci-cd
}

Pattern 16: Database Replication Failure Response

Procedures for handling database replication issues and failover.

# db-replication-response.sh
# Handle database replication failures

#!/bin/bash

# Check replication status
check_replication_status() {
    local primary=$1
    local replica=$2
    
    echo "Checking replication status between $primary and $replica"
    
    # Check primary
    psql -h "$primary" -c "SELECT * FROM pg_stat_replication;"
    
    # Check replica
    psql -h "$replica" -c "SELECT * FROM pg_stat_replication;"
    
    # Check replication lag
    psql -h "$primary" -c "SELECT now() - pg_last_xact_replay_timestamp() AS replication_lag;"
}

# Promote replica to primary
promote_replica() {
    local replica=$1
    
    echo "Promoting $replica to primary"
    
    # Stop replication
    psql -h "$replica" -c "SELECT pg_promote();"
    
    # Wait for promotion
    sleep 5
    
    # Verify promotion
    psql -h "$replica" -c "SELECT pg_is_in_recovery();"
}

# Rebuild replica
rebuild_replica() {
    local primary=$1
    local replica=$2
    
    echo "Rebuilding replica $replica from $primary"
    
    # Stop replica
    psql -h "$replica" -c "SELECT pg_stop_backup();"
    
    # Clear replica data
    rm -rf /var/lib/postgresql/data/*
    
    # Base backup from primary
    pg_basebackup -h "$primary" -D /var/lib/postgresql/data -U replicator -P -R
    
    # Start replica
    pg_ctl -D /var/lib/postgresql/data start
}

# Failover procedure
failover_database() {
    local primary=$1
    local preferred_replica=$2
    
    echo "Initiating database failover from $primary to $preferred_replica"
    
    # Verify preferred replica is up to date
    check_replication_status "$primary" "$preferred_replica"
    
    # Promote preferred replica
    promote_replica "$preferred_replica"
    
    # Rebuild other replicas
    for replica in "${REPLICA_LIST[@]}"; do
        if [[ "$replica" != "$preferred_replica" ]]; then
            rebuild_replica "$preferred_replica" "$replica"
        fi
    done
    
    # Update connection strings
    update_connection_strings "$preferred_replica"
    
    # Verify failover
    verify_database_health "$preferred_replica"
}

Pattern 17: Storage Failure Response

Procedures for handling storage system failures including PV/PVC issues.

# storage-response.sh
# Handle storage system failures

#!/bin/bash

# Check storage health
check_storage_health() {
    echo "Checking storage health..."
    
    # Check PVC status
    kubectl get pvc --all-namespaces
    
    # Check PV status
    kubectl get pv
    
    # Check storage class
    kubectl get storageclass
    
    # Check storage provider health
    kubectl get pods -n csi-driver
}

# Reclaim stuck PVC
reclaim_pvc() {
    local namespace=$1
    local pvc_name=$2
    
    echo "Reclaiming PVC: $pvc_name in namespace: $namespace"
    
    # Check PVC status
    local pvc_status=$(kubectl get pvc "$pvc_name" -n "$namespace" -o jsonpath='{.status.phase}')
    
    if [[ "$pvc_status" == "Pending" ]]; then
        echo "PVC is stuck in Pending state"
        
        # Check PV binding
        kubectl get pv | grep "$pvc_name"
        
        # Check storage class
        kubectl get sc
    fi
    
    if [[ "$pvc_status" == "Bound" ]] || [[ "$pvc_status" == "Lost" ]]; then
        # Force delete
        kubectl patch pvc "$pvc_name" -n "$namespace" -p '{"metadata":{"finalizers":null}}'
        kubectl delete pvc "$pvc_name" -n "$namespace" --force --grace-period=0
    fi
}

# Restore from storage snapshot
restore_from_snapshot() {
    local volume=$1
    local snapshot_name=$2
    local namespace=$3
    
    echo "Restoring volume $volume from snapshot $snapshot_name"
    
    # Get snapshot content
    kubectl get volumesnapshot "$snapshot_name" -n "$namespace" -o yaml
    
    # Create PVC from snapshot
    kubectl apply -f - << EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: restored-$volume-$(date +%Y%m%d)
  namespace: $namespace
spec:
  storageClassName: standard
  dataSource:
    name: $snapshot_name
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
EOF
}

# Fix orphaned volumes
fix_orphaned_volumes() {
    echo "Checking for orphaned volumes..."
    
    # List all volumes
    local volumes=$(kubectl get pv -o jsonpath='{.items[*].metadata.name}')
    
    for volume in $volumes; do
        # Check if PV is bound to any PVC
        local pvc=$(kubectl get pv "$volume" -o jsonpath='{.spec.claimRef.name}')
        
        if [[ -z "$pvc" ]]; then
            echo "WARNING: Unbound PV: $volume"
            
            # Check if PV is in Available state
            local status=$(kubectl get pv "$volume" -o jsonpath='{.status.phase}')
            
            if [[ "$status" == "Available" ]]; then
                echo "PV $volume is available for binding"
            fi
        fi
    done
}

Pattern 18: Application Crash Loop Response

Procedures for handling applications stuck in crash loop backoff.

# crashloop-response.sh
# Handle application crash loop issues

#!/bin/bash

# Diagnose crash loop
diagnose_crash_loop() {
    local deployment=$1
    local namespace=$2
    
    echo "Diagnosing crash loop for $deployment in $namespace"
    
    # Get pod status
    kubectl get pods -l app="$deployment" -n "$namespace" -o wide
    
    # Check container restart count
    kubectl get pods -l app="$deployment" -n "$namespace" -o jsonpath='{.items[*].status.containerStatuses[*].restartCount}'
    
    # Get recent logs
    kubectl logs -l app="$deployment" -n "$namespace" --tail=100 --previous
    
    # Check events
    kubectl events -n "$namespace" --field-selector involvedObject.name="$deployment"
}

# Force restart deployment
force_restart() {
    local deployment=$1
    local namespace=$2
    
    echo "Force restarting $deployment in $namespace"
    
    # Set replicas to 0
    kubectl scale deployment "$deployment" -n "$namespace" --replicas=0
    
    # Wait for pods to terminate
    sleep 10
    
    # Scale back up
    kubectl scale deployment "$deployment" -n "$namespace" --replicas=3
    
    # Monitor restart
    kubectl rollout status deployment "$deployment" -n "$namespace"
}

# Debug init containers
debug_init_containers() {
    local pod=$1
    local namespace=$2
    
    echo "Debugging init containers for $pod in $namespace"
    
    # Check init container status
    kubectl get pod "$pod" -n "$namespace" -o jsonpath='{.status.initContainerStatuses}'
    
    # Get init container logs
    kubectl logs "$pod" -n "$namespace" -c init-container --previous
    
    # Check init container configuration
    kubectl get pod "$pod" -n "$namespace" -o jsonpath='{.spec.initContainers}'
}

# Disable crash loop detection
disable_crash_loop_check() {
    local deployment=$1
    local namespace=$2
    
    echo "Temporarily disabling crash loop check for $deployment"
    
    # Patch deployment with longer initial delay
    kubectl patch deployment "$deployment" -n "$namespace" -p "{
        \"spec\": {
            \"template\": {
                \"spec\": {
                    \"containers\": [{
                        \"name\": \"$deployment\",
                        \"livenessProbe\": {
                            \"initialDelaySeconds\": 300
                        },
                        \"readinessProbe\": {
                            \"initialDelaySeconds\": 300
                        }
                    }]
                }
            }
        }
    }"
    
    echo "Probe delays increased. Restarting pods..."
    kubectl rollout restart deployment "$deployment" -n "$namespace"
}

Pattern 19: Cluster Upgrade Failure Response

Procedures for handling Kubernetes cluster upgrade failures.

# cluster-upgrade-response.sh
# Handle cluster upgrade failures

#!/bin/bash

# Check upgrade status
check_upgrade_status() {
    echo "Checking cluster upgrade status..."
    
    # Get node versions
    kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.nodeInfo.kubeletVersion}{"\n"}{end}'
    
    # Check control plane version
    kubectl version
    
    # Check etcd version
    etcdctl version
    
    # Check if upgrade is in progress
    kubectl get node -o jsonpath='{.items[*].status.conditions[?(@.type=="NodeReady")].status}'
}

# Rollback cluster upgrade
rollback_upgrade() {
    local node=$1
    
    echo "Rolling back upgrade on node: $node"
    
    # Cordon node
    kubectl cordon "$node"
    
    # Drain node
    kubectl drain "$node" --ignore-daemonsets --delete-emptydir-data --force
    
    # Reinstall older version
    # (This would be specific to your upgrade method)
    # For kubeadm:
    # kubeadm upgrade apply v1.28.0
    
    # Uncordon node
    kubectl uncordon "$node"
}

# Upgrade health check
upgrade_health_check() {
    echo "Running upgrade health check..."
    
    # Check all pods are running
    kubectl get pods --all-namespaces --field-selector=status.phase=Running
    
    # Check control plane components
    kubectl get pods -n kube-system -l tier=control-plane
    
    # Check node conditions
    kubectl get nodes -o jsonpath='{.items[*].status.conditions}'
    
    # Check system services
    kubectl get daemonset -n kube-system
}

Pattern 20: Load Balancer Failure Response

Procedures for handling load balancer and ingress failures.

# lb-response.sh
# Handle load balancer and ingress failures

#!/bin/bash

# Check load balancer health
check_lb_health() {
    echo "Checking load balancer health..."
    
    # Check ingress controller
    kubectl get pods -n ingress-nginx
    
    # Check service endpoints
    kubectl get endpoints -n ingress-nginx
    
    # Check load balancer status
    kubectl get service -n ingress-nginx ingress-nginx-controller
    
    # Test connectivity
    kubectl run test-lb --rm -it --image=busybox -- wget -q -O- http://ingress-nginx-controller.ingress-nginx.svc.cluster.local
}

# Reconfigure ingress
reconfigure_ingress() {
    local ingress_name=$1
    local namespace=$2
    
    echo "Reconfiguring ingress: $ingress_name"
    
    # Backup current configuration
    kubectl get ingress "$ingress_name" -n "$namespace" -o yaml > "/tmp/ingress_backup_${ingress_name}.yaml"
    
    # Check ingress controller
    kubectl logs -n ingress-nginx -l app=ingress-nginx-controller --tail=100
    
    # Restart ingress controller
    kubectl rollout restart deployment ingress-nginx-controller -n ingress-nginx
    
    # Verify restart
    kubectl rollout status deployment ingress-nginx-controller -n ingress-nginx
}

# Failover load balancer
failover_lb() {
    local lb_cluster=$1
    
    echo "Failing over load balancer cluster: $lb_cluster"
    
    # Identify active node
    local active_node=$(kubectl get pods -n ingress-nginx -o jsonpath='{.items[?(@.status.phase=="Running")].metadata.name}' | head -1)
    
    # Drain inactive nodes
    for node in $(kubectl get nodes -l app=ingress-nginx -o jsonpath='{.items[*].metadata.name}'); do
        if [[ "$node" != "$active_node" ]]; then
            kubectl cordon "$node"
        fi
    done
    
    # Scale to single node
    kubectl scale deployment ingress-nginx-controller -n ingress-nginx --replicas=1
}

# Update load balancer configuration
update_lb_config() {
    local config_name=$1
    
    echo "Updating load balancer configuration: $config_name"
    
    # Get current config
    kubectl get configmap ingress-nginx-controller -n ingress-nginx -o yaml > "/tmp/${config_name}_backup.yaml"
    
    # Update config
    kubectl patch configmap ingress-nginx-controller -n ingress-nginx -p "{
        \"data\": {
            \"proxy-body-size\": \"50m\",
            \"proxy-read-timeout\": \"60\",
            \"proxy-send-timeout\": \"60\"
        }
    }"
    
    # Reload ingress controller
    kubectl rollout restart deployment ingress-nginx-controller -n ingress-nginx
}

Pattern 21: DNS Resolution Failure Response

Procedures for handling DNS resolution issues.

# dns-response.sh
# Handle DNS resolution failures

#!/bin/bash

# Check DNS health
check_dns_health() {
    echo "Checking DNS health..."
    
    # Check coredns pods
    kubectl get pods -n kube-system -l k8s-app=kube-dns
    
    # Check DNS service
    kubectl get service kube-dns -n kube-system
    
    # Test DNS resolution
    kubectl run test-dns --rm -it --image=busybox -- nslookup kubernetes.default
    
    # Check coredns logs
    kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100
}

# Restart DNS service
restart_dns() {
    echo "Restarting DNS service..."
    
    # Scale down coredns
    kubectl scale deployment coredns -n kube-system --replicas=0
    
    # Wait for termination
    sleep 10
    
    # Scale back up
    kubectl scale deployment coredns -n kube-system --replicas=2
    
    # Verify DNS is working
    kubectl run test-dns --rm -it --image=busybox -- nslookup kubernetes.default
}

# Fix DNS configuration
fix_dns_config() {
    echo "Fixing DNS configuration..."
    
    # Check coredns configmap
    kubectl get configmap coredns -n kube-system -o yaml
    
    # Backup current config
    kubectl get configmap coredns -n kube-system -o yaml > /tmp/coredns_config_backup.yaml
    
    # Reset to defaults
    kubectl apply -f /etc/coredns/coredns.yaml.default
    
    # Restart coredns
    kubectl rollout restart deployment coredns -n kube-system
}

# Debug DNS issues
debug_dns_issue() {
    local pod=$1
    local namespace=$2
    local hostname=$3
    
    echo "Debugging DNS issue for $pod in $namespace resolving $hostname"
    
    # Check pod DNS config
    kubectl exec "$pod" -n "$namespace" -- cat /etc/resolv.conf
    
    # Test resolution
    kubectl exec "$pod" -n "$namespace" -- nslookup "$hostname"
    
    # Check coredns logs for query
    kubectl logs -n kube-system -l k8s-app=kube-dns | grep "$hostname" | tail -20
}

Pattern 22: Certificate Expiration Response

Procedures for handling TLS certificate expiration and renewal.

# cert-response.sh
# Handle TLS certificate issues

#!/bin/bash

# Check certificate expiry
check_cert_expiry() {
    echo "Checking certificate expiry..."
    
    # Check API server cert
    openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -enddate
    
    # Check etcd certs
    openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -noout -enddate
    
    # Check front-proxy cert
    openssl x509 -in /etc/kubernetes/pki/front-proxy-client.crt -noout -enddate
    
    # Check all certs in cluster
    for cert in $(find /etc/kubernetes/pki -name "*.crt"); do
        echo -n "$cert: "
        openssl x509 -in "$cert" -noout -enddate 2>/dev/null || echo "Invalid cert"
    done
}

# Renew certificates
renew_certificates() {
    echo "Renewing certificates..."
    
    # Renew all certificates
    kubeadm certs renew all
    
    # Verify renewal
    kubeadm certs check-expiration
    
    # Restart control plane components
    systemctl restart kubelet
    sleep 10
    
    # Restart pods
    kubectl rollout restart deployment -n kube-system
}

# Update kubeconfig
update_kubeconfig() {
    local cluster_name=$1
    local new_cert_path=$2
    
    echo "Updating kubeconfig for cluster: $cluster_name"
    
    # Get current context
    kubectl config current-context
    
    # Update certificate
    kubectl config set-cluster "$cluster_name" --certificate-authority="$new_cert_path"
    
    # Verify
    kubectl cluster-info
}

# Debug certificate issues
debug_cert_issue() {
    local service=$1
    local port=$2
    
    echo "Debugging certificate issue for $service:$port"
    
    # Test TLS connection
    openssl s_client -connect "$service:$port" -servername "$service" 2>&1 | openssl x509 -noout -dates
    
    # Check certificate chain
    openssl s_client -connect "$service:$port" -showcerts
    
    # Verify certificate validity
    openssl verify -CAfile /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/apiserver.crt
}

Pattern 23: Etcd Cluster Failure Response

Procedures for handling etcd cluster failures.

# etcd-response.sh
# Handle etcd cluster failures

#!/bin/bash

# Check etcd cluster health
check_etcd_health() {
    echo "Checking etcd cluster health..."
    
    # Check member status
    etcdctl member list
    
    # Check cluster health
    etcdctl endpoint health
    
    # Check leader
    etcdctl endpoint status --endpoints=https://127.0.0.1:2379 --write-out=table
    
    # Check db size
    etcdctl endpoint status --endpoints=https://127.0.0.1:2379 --write-out=table | grep -v "DB SIZE"
}

# Add new etcd member
add_etcd_member() {
    local new_member=$1
    local peer_url=$2
    
    echo "Adding new etcd member: $new_member"
    
    # Get current members
    etcdctl member list
    
    # Add new member
    etcdctl member add "$new_member" --peer-urls="$peer_url"
    
    # Update configuration
    export ETCD_NAME="$new_member"
    export ETCD_INITIAL_CLUSTER_STATE="existing"
    
    # Start new member
    systemctl start etcd
}

# Remove failed etcd member
remove_etcd_member() {
    local member_id=$1
    
    echo "Removing failed etcd member: $member_id"
    
    # Get member ID
    local id=$(etcdctl member list | grep "$member_id" | cut -d',' -f1 | cut -d' ' -f1)
    
    # Remove member
    etcdctl member remove "$id"
    
    # Verify removal
    etcdctl member list
}

# Restore etcd from backup
restore_etcd_backup() {
    local backup_path=$1
    local cluster_endpoints=$2
    
    echo "Restoring etcd from backup: $backup_path"
    
    # Stop etcd
    systemctl stop etcd
    
    # Restore snapshot
    etcdctl snapshot restore "$backup_path" \
        --data-dir="/var/lib/etcd-new" \
        --initial-cluster="$cluster_endpoints" \
        --initial-cluster-token="etcd-cluster-$(date +%s)"
    
    # Update data directory
    mv /var/lib/etcd /var/lib/etcd-old
    mv /var/lib/etcd-new /var/lib/etcd
    
    # Start etcd
    systemctl start etcd
    
    # Verify restoration
    etcdctl endpoint health
}

Pattern 24: Pod Security Policy Violation Response

Procedures for handling PSP violations and security policy issues.

# psp-response.sh
# Handle Pod Security Policy violations

#!/bin/bash

# Check PSP violations
check_psp_violations() {
    echo "Checking PSP violations..."
    
    # Check for denied pods
    kubectl get events --field-selector reason=PolicyViolation
    
    # Check pod security status
    kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.pod-security.kubernetes.io/enforce}{"\n"}{end}'
    
    # List pods by security context
    kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.securityContext.runAsUser}{"\n"}{end}'
}

# Temporarily disable PSP enforcement
disable_psp() {
    local namespace=$1
    
    echo "Temporarily disabling PSP enforcement in $namespace"
    
    # Label namespace to bypass PSP
    kubectl label namespace "$namespace" pod-security.kubernetes.io/enforce=baseline --overwrite
    
    # Restart pods
    kubectl rollout restart deployment --all -n "$namespace"
}

# Fix PSP violation
fix_psp_violation() {
    local pod=$1
    local namespace=$2
    local violation_type=$3
    
    echo "Fixing PSP violation for $pod in $namespace"
    
    # Get pod specification
    kubectl get pod "$pod" -n "$namespace" -o yaml > "/tmp/${pod}_backup.yaml"
    
    # Fix based on violation type
    case "$violation_type" in
        "runAsRoot")
            kubectl patch pod "$pod" -n "$namespace" -p '{
                "spec": {"securityContext": {"runAsNonRoot": true}}
            }'
            ;;
        "privileged")
            kubectl patch pod "$pod" -n "$namespace" -p '{
                "spec": {"containers": [{"securityContext": {"privileged": false}}]}
            }'
            ;;
        "hostNetwork")
            kubectl patch pod "$pod" -n "$namespace" -p '{
                "spec": {"hostNetwork": false}
            }'
            ;;
    esac
    
    # Restart pod
    kubectl delete pod "$pod" -n "$namespace"
}

# Review PSP configuration
review_psp_config() {
    echo "Reviewing PSP configuration..."
    
    # List all PSPs
    kubectl get psp
    
    # Check PSP violations
    kubectl get psp | grep -v "NAME" | while read psp; do
        echo "Checking PSP: $psp"
        kubectl describe psp "$psp"
    done
}

Pattern 25: Helm Release Failure Response

Procedures for handling Helm release failures and rollback.

# helm-response.sh
# Handle Helm release failures

#!/bin/bash

# Check Helm release status
check_helm_status() {
    local release=$1
    local namespace=$2
    
    echo "Checking Helm release: $release"
    
    # Get release status
    helm status "$release" -n "$namespace"
    
    # Get release history
    helm history "$release" -n "$namespace"
    
    # Check deployed resources
    helm get all "$release" -n "$namespace"
}

# Rollback Helm release
rollback_helm() {
    local release=$1
    local namespace=$2
    local revision=$3
    
    echo "Rolling back Helm release: $release to revision $revision"
    
    # Get current revision
    local current_revision=$(helm history "$release" -n "$namespace" | tail -1 | cut -d' ' -f1)
    
    # Rollback
    helm rollback "$release" "$revision" -n "$namespace"
    
    # Verify rollback
    helm status "$release" -n "$namespace"
    
    # Check resources
    kubectl get all -n "$namespace" -l "app.kubernetes.io/managed-by=Helm"
}

# Uninstall Helm release
uninstall_helm() {
    local release=$1
    local namespace=$2
    
    echo "Uninstalling Helm release: $release"
    
    # Dry run first
    helm uninstall "$release" -n "$namespace" --dry-run --debug
    
    # Confirm and uninstall
    helm uninstall "$release" -n "$namespace"
    
    # Verify removal
    kubectl get all -n "$namespace" | grep "$release"
}

# Debug Helm deployment
debug_helm_deployment() {
    local release=$1
    local namespace=$2
    
    echo "Debugging Helm deployment: $release"
    
    # Get values
    helm get values "$release" -n "$namespace"
    
    # Get templates
    helm get templates "$release" -n "$namespace"
    
    # Check hooks
    helm get hooks "$release" -n "$namespace"
    
    # Check resources
    helm get all "$release" -n "$namespace"
}

Pattern 26: Service Mesh Traffic Management Failure Response

Procedures for handling service mesh traffic management issues.

# sm-traffic-response.sh
# Handle service mesh traffic management failures

#!/bin/bash

# Check Istio traffic policies
check_istio_traffic() {
    echo "Checking Istio traffic policies..."
    
    # List virtual services
    kubectl get virtualservice --all-namespaces
    
    # List destination rules
    kubectl get destinationrule --all-namespaces
    
    # List gateways
    kubectl get gateway --all-namespaces
    
    # List service entries
    kubectl get serviceentry --all-namespaces
}

# Reset traffic policies
reset_traffic_policies() {
    echo "Resetting traffic policies..."
    
    # Backup current policies
    mkdir -p /tmp/istio-backup/$(date +%Y%m%d_%H%M%S)
    
    # Backup virtual services
    kubectl get virtualservice --all-namespaces -o yaml > /tmp/istio-backup/virtualservices.yaml
    
    # Backup destination rules
    kubectl get destinationrule --all-namespaces -o yaml > /tmp/istio-backup/destinationrules.yaml
    
    # Reset policies (example: allow all traffic)
    kubectl apply -f - << EOF
apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  name: default
  namespace: istio-system
spec:
  egress:
  - hosts:
    - "./*"
EOF
}

# Debug traffic routing
debug_traffic_routing() {
    local source=$1
    local destination=$2
    
    echo "Debugging traffic from $source to $destination"
    
    # Check sidecar configuration
    istioctl proxy-config listeners "$source"
    
    # Check route configuration
    istioctl proxy-config routes "$source"
    
    # Check endpoint configuration
    istioctl proxy-config endpoints "$source"
    
    # Test connectivity
    kubectl exec "$source" -- curl -v "http://$destination"
}

# Fix Envoy proxy issues
fix_envoy_proxy() {
    local pod=$1
    
    echo "Fixing Envoy proxy issues for $pod"
    
    # Get Envoy config
    istioctl proxy-config "$pod"
    
    # Reset Envoy config
    istioctl proxy-config "$pod" --reset
    
    # Restart Envoy
    istioctl proxy-config "$pod" --type envoy
}

Pattern 27: Metrics and Monitoring Failure Response

Procedures for handling metrics and monitoring system failures.

# monitoring-response.sh
# Handle metrics and monitoring failures

#!/bin/bash

# Check Prometheus health
check_prometheus_health() {
    echo "Checking Prometheus health..."
    
    # Check Prometheus pods
    kubectl get pods -n monitoring
    
    # Check Prometheus targets
    curl -s "http://prometheus-server:9090/api/v1/targets" | jq '.data.targets[] | select(.labels.job != "") | "\(.instance) - \(.health)"'
    
    # Check Prometheus rules
    curl -s "http://prometheus-server:9090/api/v1/rules" | jq '.data.groups[].rules[] | select(.health == "fatal")'
    
    # Check storage
    curl -s "http://prometheus-server:9090/api/v1/status/tsdb" | jq '.data'
}

# Alertmanager recovery
recover_alertmanager() {
    echo "Recovering Alertmanager..."
    
    # Check Alertmanager pods
    kubectl get pods -n monitoring -l app=alertmanager
    
    # Check Alertmanager config
    kubectl get configmap alertmanager-config -n monitoring -o yaml
    
    # Restart Alertmanager
    kubectl rollout restart deployment alertmanager -n monitoring
    
    # Verify restart
    kubectl rollout status deployment alertmanager -n monitoring
}

# Grafana recovery
recover_grafana() {
    echo "Recovering Grafana..."
    
    # Check Grafana pods
    kubectl get pods -n monitoring -l app=grafana
    
    # Check dashboards
    kubectl get configmap -n monitoring -l grafana_dashboard=1
    
    # Restart Grafana
    kubectl rollout restart deployment grafana -n monitoring
    
    # Verify restart
    kubectl rollout status deployment grafana -n monitoring
}

# Metrics query debugging
debug_metrics_query() {
    local metric=$1
    local time_range=$2
    
    echo "Debugging metrics query for $metric"
    
    # Query Prometheus
    curl -s "http://prometheus-server:9090/api/v1/query?query=${metric}&time=$(date -u +%Y-%m-%dT%H:%M:%SZ)" | jq
    
    # Query range
    curl -s "http://prometheus-server:9090/api/v1/query_range?query=${metric}&start=$(date -u -d "$time_range ago" +%Y-%m-%dT%H:%M:%SZ)&end=$(date -u +%Y-%m-%dT%H:%M:%SZ)&step=60" | jq
    
    # Check metric metadata
    curl -s "http://prometheus-server:9090/api/v1/label/__name__/values" | jq ".data[] | select(. | contains(\"$metric\"))"
}

Pattern 28: Logging Infrastructure Failure Response

Procedures for handling logging infrastructure failures.

# logging-response.sh
# Handle logging infrastructure failures

#!/bin/bash

# Check Fluentd health
check_fluentd_health() {
    echo "Checking Fluentd health..."
    
    # Check Fluentd pods
    kubectl get pods -n logging
    
    # Check Fluentd config
    kubectl get configmap fluentd-config -n logging -o yaml
    
    # Check Fluentd logs
    kubectl logs -l app=fluentd -n logging --tail=100
    
    # Check buffer status
    curl -s "http://fluentd-service:24220/api/plugins.json" | jq
}

# Elasticsearch cluster recovery
recover_elasticsearch() {
    echo "Recovering Elasticsearch cluster..."
    
    # Check cluster health
    curl -s "http://elasticsearch:9200/_cluster/health" | jq
    
    # Check node status
    curl -s "http://elasticsearch:9200/_cat/nodes?v" | jq
    
    # Check shard allocation
    curl -s "http://elasticsearch:9200/_cat/shards?v" | grep "UNASSIGNED"
    
    # Reallocate shards
    curl -X PUT "http://elasticsearch:9200/_cluster/settings" -H 'Content-Type: application/json' -d '{
        "persistent": {
            "cluster.routing.allocation.enable": "all"
        }
    }'
}

# Kibana recovery
recover_kibana() {
    echo "Recovering Kibana..."
    
    # Check Kibana pods
    kubectl get pods -n logging -l app=kibana
    
    # Check Kibana config
    kubectl get configmap kibana-config -n logging -o yaml
    
    # Restart Kibana
    kubectl rollout restart deployment kibana -n logging
    
    # Verify restart
    kubectl rollout status deployment kibana -n logging
}

# Log query debugging
debug_log_query() {
    local index=$1
    local query=$2
    
    echo "Debugging log query for index: $index"
    
    # Query Elasticsearch
    curl -s "http://elasticsearch:9200/$index/_search" -H 'Content-Type: application/json' -d "{
        \"query\": {
            \"match\": {
                \"message\": \"$query\"
            }
        },
        \"size\": 10
    }" | jq
    
    # Check index status
    curl -s "http://elasticsearch:9200/$index/_stats" | jq
}

Pattern 29: Pod Disruption Budget Violation Response

Procedures for handling PDB violations and voluntary disruptions.

# pdb-response.sh
# Handle Pod Disruption Budget violations

#!/bin/bash

# Check PDB status
check_pdb_status() {
    echo "Checking PDB status..."
    
    # List all PDBs
    kubectl get pdb --all-namespaces
    
    # Check PDB violations
    kubectl get pdb --all-namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.disruptionsAllowed}{"\t"}{.status.currentHealthy}{"\n"}{end}'
    
    # Check pods under disruption
    kubectl get events --field-selector reason=Eviction | grep -v "NoError"
}

# Allow disruption temporarily
allow_disruption() {
    local pdb=$1
    local namespace=$2
    local allowed_disruptions=$3
    
    echo "Allowing $allowed_disruptions disruptions for PDB $pdb in $namespace"
    
    # Patch PDB to allow more disruptions
    kubectl patch pdb "$pdb" -n "$namespace" -p "{
        \"spec\": {
            \"maxUnavailable\": $allowed_disruptions
        }
    }"
    
    echo "PDB patched. Disruptions allowed: $allowed_disruptions"
}

# Fix PDB configuration
fix_pdb_config() {
    local pdb=$1
    local namespace=$2
    
    echo "Fixing PDB configuration: $pdb in $namespace"
    
    # Get current PDB
    kubectl get pdb "$pdb" -n "$namespace" -o yaml > "/tmp/pdb_backup_${pdb}.yaml"
    
    # Calculate appropriate value
    local replicas=$(kubectl get deployment -n "$namespace" -o jsonpath='{.items[*].spec.replicas}' | tr ' ' '\n' | sort -n | head -1)
    
    # Update PDB
    local max_unavailable=$((replicas / 3))
    kubectl patch pdb "$pdb" -n "$namespace" -p "{
        \"spec\": {
            \"maxUnavailable\": $max_unavailable
        }
    }"
    
    echo "PDB updated. Max unavailable: $max_unavailable"
}

# Drain node with PDB protection
drain_with_pdb() {
    local node=$1
    
    echo "Draining node with PDB protection: $node"
    
    # Check PDB violations before drain
    kubectl describe pdb --all-namespaces | grep -A 10 "Allowed Disruptions"
    
    # Drain node with PDB violation tolerance
    kubectl drain "$node" --ignore-daemonsets --delete-emptydir-data --force --ignore-disruptions
    
    # Verify drain completed
    kubectl get nodes | grep "$node" | grep "SchedulingDisabled"
}

Pattern 30: Cluster Autoscaler Failure Response

Procedures for handling cluster autoscaler issues.

# autoscaler-response.sh
# Handle cluster autoscaler failures

#!/bin/bash

# Check autoscaler health
check_autoscaler_health() {
    echo "Checking cluster autoscaler health..."
    
    # Check autoscaler pods
    kubectl get pods -n kube-system | grep cluster-autoscaler
    
    # Check autoscaler logs
    kubectl logs -n kube-system -l app=cluster-autoscaler --tail=100
    
    # Check node groups
    kubectl get nodes -o wide
    
    # Check pending pods
    kubectl get pods --all-namespaces --field-selector status.phase=Pending
}

# Force scale up
force_scale_up() {
    echo "Forcing cluster scale up..."
    
    # Get pending pods
    local pending_pods=$(kubectl get pods --all-namespaces --field-selector status.phase=Pending -o name)
    
    if [[ -n "$pending_pods" ]]; then
        echo "Found pending pods: $pending_pods"
        
        # Trigger scale up by deleting a pod (this will trigger rescheduling)
        local pod_to_delete=$(kubectl get pods --all-namespaces --field-selector status.phase=Pending -o name | head -1 | cut -d'/' -f2)
        
        if [[ -n "$pod_to_delete" ]]; then
            kubectl delete pod "$pod_to_delete" -n default
        fi
    fi
    
    # Wait for scale up
    sleep 30
    
    # Verify new nodes
    kubectl get nodes
}

# Disable autoscaler temporarily
disable_autoscaler() {
    echo "Temporarily disabling cluster autoscaler..."
    
    # Scale down autoscaler
    kubectl scale deployment cluster-autoscaler -n kube-system --replicas=0
    
    echo "Autoscaler disabled. Manual scaling required."
}

# Rebalance cluster
rebalance_cluster() {
    echo "Rebalancing cluster..."
    
    # Evict pods from overloaded nodes
    for node in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do
        local cpu_usage=$(kubectl top node "$node" | awk 'NR==2 {print $2}' | sed 's/%//')
        
        if [[ -n "$cpu_usage" ]] && [[ "$cpu_usage" -gt 80 ]]; then
            echo "High CPU on $node: ${cpu_usage}%"
            
            # Get pods on node
            local pods=$(kubectl get pods --field-selector spec.nodeName="$node" -o name | head -5)
            
            # Evict some pods
            for pod in $pods; do
                kubectl delete pod "$pod"
            done
        fi
    done
    
    # Wait for rebalancing
    sleep 60
    
    # Verify rebalanced
    kubectl get nodes -o wide
}

Pattern 31: Network Policy Enforcement Failure Response

Procedures for handling network policy enforcement issues.

# network-policy-response.sh
# Handle network policy enforcement failures

#!/bin/bash

# Check network policy enforcement
check_policy_enforcement() {
    echo "Checking network policy enforcement..."
    
    # List all network policies
    kubectl get networkpolicy --all-namespaces
    
    # Check policy status
    kubectl get networkpolicy --all-namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podSelector}{"\n"}{end}'
    
    # Test connectivity
    kubectl run test-netpol --rm -it --image=busybox -- wget -q -O- http://test-service
}

# Debug network policy
debug_network_policy() {
    local namespace=$1
    local pod=$2
    
    echo "Debugging network policy for $pod in $namespace"
    
    # Get pod labels
    kubectl get pod "$pod" -n "$namespace" -o jsonpath='{.metadata.labels}'
    
    # Get network policies affecting pod
    kubectl get networkpolicy --all-namespaces -o json | jq -r ".items[] | select(.spec.podSelector.matchLabels | to_entries[] | .value == \"$(kubectl get pod "$pod" -n "$namespace" -o jsonpath='{.metadata.labels}')\") | .metadata.name"
    
    # Test connectivity
    kubectl exec "$pod" -n "$namespace" -- nc -zv service.default.svc.cluster.local 80
}

# Temporarily disable network policy
disable_network_policy() {
    local policy=$1
    local namespace=$2
    
    echo "Temporarily disabling network policy: $policy"
    
    # Backup policy
    kubectl get networkpolicy "$policy" -n "$namespace" -o yaml > "/tmp/policy_backup_${policy}.yaml"
    
    # Delete policy
    kubectl delete networkpolicy "$policy" -n "$namespace"
    
    echo "Policy deleted. Testing connectivity..."
}

# Reapply network policies
reapply_policies() {
    echo "Reapplying network policies..."
    
    # Get all policies
    kubectl get networkpolicy --all-namespaces -o yaml > /tmp/all_policies.yaml
    
    # Reapply
    kubectl apply -f /tmp/all_policies.yaml
    
    # Verify
    kubectl get networkpolicy --all-namespaces
}

Pattern 32: Ingress Controller Failure Response

Procedures for handling ingress controller failures.

# ingress-response.sh
# Handle ingress controller failures

#!/bin/bash

# Check ingress controller health
check_ingress_health() {
    echo "Checking ingress controller health..."
    
    # Check ingress controller pods
    kubectl get pods -n ingress-nginx
    
    # Check ingress services
    kubectl get service -n ingress-nginx
    
    # Check ingress resources
    kubectl get ingress --all-namespaces
    
    # Test connectivity
    curl -I http://localhost:8080
}

# Restart ingress controller
restart_ingress() {
    echo "Restarting ingress controller..."
    
    # Get controller pods
    local pods=$(kubectl get pods -n ingress-nginx -o jsonpath='{.items[*].metadata.name}')
    
    # Delete pods
    for pod in $pods; do
        kubectl delete pod "$pod" -n ingress-nginx
    done
    
    # Wait for restart
    sleep 30
    
    # Verify
    kubectl rollout status deployment ingress-nginx-controller -n ingress-nginx
}

# Update ingress configuration
update_ingress_config() {
    local config_name=$1
    
    echo "Updating ingress configuration: $config_name"
    
    # Get current config
    kubectl get configmap ingress-nginx-controller -n ingress-nginx -o yaml > "/tmp/ingress_config_backup.yaml"
    
    # Update config
    kubectl patch configmap ingress-nginx-controller -n ingress-nginx -p "{
        \"data\": {
            \"proxy-body-size\": \"100m\",
            \"proxy-read-timeout\": \"120\",
            \"proxy-send-timeout\": \"120\"
        }
    }"
    
    # Reload controller
    kubectl rollout restart deployment ingress-nginx-controller -n ingress-nginx
}

# Fix ingress routing
fix_ingress_routing() {
    local ingress_name=$1
    local namespace=$2
    
    echo "Fixing ingress routing for $ingress_name in $namespace"
    
    # Get ingress configuration
    kubectl get ingress "$ingress_name" -n "$namespace" -o yaml
    
    # Check backend services
    kubectl get service -n "$namespace" | grep "$(kubectl get ingress "$ingress_name" -n "$namespace" -o jsonpath='{.spec.rules[0].http.paths[0].backend.service.name}')"
    
    # Restart backend if needed
    kubectl rollout restart deployment -n "$namespace"
}

Pattern 33: StatefulSet Failure Response

Procedures for handling StatefulSet failures and pod management issues.

# statefulset-response.sh
# Handle StatefulSet failures

#!/bin/bash

# Check StatefulSet status
check_statefulset_status() {
    local statefulset=$1
    local namespace=$2
    
    echo "Checking StatefulSet: $statefulset in $namespace"
    
    # Get StatefulSet status
    kubectl get statefulset "$statefulset" -n "$namespace" -o wide
    
    # Check pod status
    kubectl get pods -l app="$statefulset" -n "$namespace" -o wide
    
    # Check PVC status
    kubectl get pvc -l app="$statefulset" -n "$namespace"
}

# Scale StatefulSet
scale_statefulset() {
    local statefulset=$1
    local namespace=$2
    local replicas=$3
    
    echo "Scaling StatefulSet $statefulset to $replicas replicas"
    
    # Scale down gracefully
    kubectl scale statefulset "$statefulset" -n "$namespace" --replicas="$replicas"
    
    # Wait for scaling
    kubectl rollout status statefulset "$statefulset" -n "$namespace"
    
    # Verify
    kubectl get pods -l app="$statefulset" -n "$namespace"
}

# Recreate StatefulSet pod
recreate_statefulset_pod() {
    local statefulset=$1
    local namespace=$2
    local pod=$3
    
    echo "Recreating StatefulSet pod: $pod"
    
    # Get current configuration
    kubectl get pod "$pod" -n "$namespace" -o yaml > "/tmp/${pod}_backup.yaml"
    
    # Delete pod (StatefulSet will recreate)
    kubectl delete pod "$pod" -n "$namespace"
    
    # Wait for recreation
    kubectl wait --for=condition=Ready pod "$pod" -n "$namespace" --timeout=300s
    
    # Verify PVC preserved
    kubectl get pvc -l app="$statefulset" -n "$namespace"
}

# Fix StatefulSet ordering
fix_statefulset_ordering() {
    local statefulset=$1
    local namespace=$2
    
    echo "Fixing StatefulSet ordering for $statefulset"
    
    # Get current ordinal
    local current_ordinal=$(kubectl get pods -l app="$statefulset" -n "$namespace" -o jsonpath='{.items[0].metadata.name}' | cut -d'-' -f2)
    
    # Delete and recreate in order
    for i in $(seq 0 2); do
        local pod="${statefulset}-${i}"
        
        if kubectl get pod "$pod" -n "$namespace" &> /dev/null; then
            echo "Recreating $pod"
            kubectl delete pod "$pod" -n "$namespace"
            kubectl wait --for=condition=Ready pod "$pod" -n "$namespace" --timeout=300s
        fi
    done
}

Pattern 34: Job and CronJob Failure Response

Procedures for handling Job and CronJob failures.

# job-response.sh
# Handle Job and CronJob failures

#!/bin/bash

# Check Job status
check_job_status() {
    local job=$1
    local namespace=$2
    
    echo "Checking Job: $job in $namespace"
    
    # Get Job status
    kubectl get job "$job" -n "$namespace" -o wide
    
    # Get pod status
    kubectl get pods -l job-name="$job" -n "$namespace"
    
    # Get logs
    kubectl logs -l job-name="$job" -n "$namespace"
}

# Delete failed Job
delete_failed_job() {
    local job=$1
    local namespace=$2
    
    echo "Deleting failed Job: $job"
    
    # Get Job status
    local completions=$(kubectl get job "$job" -n "$namespace" -o jsonpath='{.status.completionTime}')
    
    if [[ -n "$completions" ]]; then
        echo "Job completed. Deleting..."
    else
        echo "Job failed. Deleting..."
    fi
    
    # Delete Job
    kubectl delete job "$job" -n "$namespace"
    
    # Verify
    kubectl get jobs -n "$namespace" | grep "$job" || echo "Job deleted successfully"
}

# Restart CronJob
restart_cronjob() {
    local cronjob=$1
    local namespace=$2
    
    echo "Restarting CronJob: $cronjob"
    
    # Suspend CronJob
    kubectl patch cronjob "$cronjob" -n "$namespace" -p '{"spec":{"suspend":true}}'
    
    # Delete active Job
    kubectl delete job -l cronjob-name="$cronjob" -n "$namespace" --force --grace-period=0
    
    # Resume CronJob
    kubectl patch cronjob "$cronjob" -n "$namespace" -p '{"spec":{"suspend":false}}'
    
    # Trigger manual run
    kubectl create job --from=cronjob/"$cronjob" manual-run -n "$namespace"
    
    # Monitor
    kubectl get jobs -l cronjob-name="$cronjob" -n "$namespace" -w
}

# Debug CronJob schedule
debug_cron_schedule() {
    local cronjob=$1
    local namespace=$2
    
    echo "Debugging CronJob schedule: $cronjob"
    
    # Get CronJob spec
    kubectl get cronjob "$cronjob" -n "$namespace" -o yaml
    
    # Get next schedule
    kubectl get cronjob "$cronjob" -n "$namespace" -o jsonpath='{.status.lastScheduleTime}'
    
    # Get active Jobs
    kubectl get jobs -l cronjob-name="$cronjob" -n "$namespace"
}

Implementation Checklist

Before deploying incident response runbooks, verify:

This skill provides comprehensive incident response procedures for cloud-native environments. All code examples are production-ready and follow SRE best practices.

Implementation Checklist

Before deploying incident response runbooks, verify:

This skill provides comprehensive incident response procedures for cloud-native environments. All code examples are production-ready and follow SRE best practices.

MAKE IT PART OF YOUR WORKFLOW

Installer la Skill

TERMINAL

npx skills add paulpas/agent-skill-router --skill incident-response-runbooks

Exécutez la commande dans votre terminal. Vérifiez le code et la licence avant l’installation.

ZIP · Complete skill directory with resources and licenses

SKILL.md · Instructions file only

Étoiles du dépôt: Not synced
Licence: MIT
Version / branche: HEAD · d2e2ac6f
Dépôt mis à jour: Not synced

Vérifiez les scripts et les autorisations avant l’installation.

Code source

incident-response-runbooks

Documentation de la Skill

Incident Response Runbooks

TL;DR Checklist

When to Use

When NOT to Use

Core Workflow

Implementation Patterns

Pattern 1: Incident Severity Classification

Pattern 2: Incident Command System (ICS) Setup

Pattern 3: Initial Notification and Acknowledgment

Pattern 4: Communication Status Updates

Pattern 5: Escalation Triggers and Procedures

Pattern 6: Runbook Automation and Execution

Pattern 7: Incident Timeline and Documentation

Pattern 8: Post-Incident Review and Blameless Postmortem

Impact Assessment

Services Affected

User Impact

Business Impact

Detection and Response

Detection

Response

Resolution Steps

Lessons Learned

What Went Well

What Went Wrong

What We Learned

Action Items

Follow-up Schedule

Appendices

Related Documentation

Technical Details

Communication Log

Postmortem completion checklist

Generate postmortem summary for executives

Executive Summary: ${incident_id}

Pattern 10: Network Partition Recovery

Pattern 11: Memory and Resource Exhaustion Response

Pattern 12: Data Corruption and Recovery

Pattern 13: Security Incident Response

Pattern 14: Service Mesh Incident Response

Pattern 15: CI/CD Pipeline Incident Response

Pattern 16: Database Replication Failure Response

Pattern 17: Storage Failure Response

Pattern 18: Application Crash Loop Response

Pattern 19: Cluster Upgrade Failure Response

Pattern 20: Load Balancer Failure Response

Pattern 21: DNS Resolution Failure Response

Pattern 22: Certificate Expiration Response

Pattern 23: Etcd Cluster Failure Response

Pattern 24: Pod Security Policy Violation Response

Pattern 25: Helm Release Failure Response

Pattern 26: Service Mesh Traffic Management Failure Response

Pattern 27: Metrics and Monitoring Failure Response

Pattern 28: Logging Infrastructure Failure Response

Pattern 29: Pod Disruption Budget Violation Response

Pattern 30: Cluster Autoscaler Failure Response

Pattern 31: Network Policy Enforcement Failure Response

Pattern 32: Ingress Controller Failure Response

Pattern 33: StatefulSet Failure Response

Pattern 34: Job and CronJob Failure Response

Implementation Checklist

Implementation Checklist

More skills to explore

writing-skills

canary

land-and-deploy