Productivité · SKILL.md

litmus

litmus

"Litmus in Chaos Engineering - cloud native architecture, patterns, pits" and best practices

Independent GitHub snapshot · Automated classification · Original documentation governs

Ppaulpas / agent-skill-routerMIT

Documentation de la Skill

GitHub

name: litmus compatibility: opencode completeness: 95 content-types:

guidance
examples
do-dont
config description: '"Litmus in Chaos Engineering - cloud native architecture, patterns, pits" and best practices' license: MIT maturity: stable metadata: domain: cncf output-format: manifests related-skills: null role: reference scope: infrastructure triggers: cdn, chaos, engineering, infrastructure as code, litmus, monitoring, cloudformation, cloudfront archetypes:
- educational
- strategic anti_triggers:
- brainstorming
- vague ideation
- non-containerized architecture response_profile: verbosity: medium directive_strength: low abstraction_level: strategic version: "1.0.0"

Litmus in Cloud-Native Engineering

Purpose and Use Cases

What Problem Does It Solve?

Chaos engineering at scale: Enables systematic testing of distributed systems resilience through automated chaos experiments
Proactive failure detection: Identify system weaknesses before they cause production incidents
Self-healing verification: Test and validate automated recovery mechanisms
Resilience culturing: Build engineering teams' confidence in system reliability
Compliance and audit: Document system behavior under failure conditions for regulatory compliance

When to Use

Critical production systems: Financial, healthcare, or infrastructure systems where failures are costly
Complex distributed architectures: Microservices, multi-cloud, or hybrid deployments
After major deployments: Validate new deployments don't introduce resilience gaps
Regular resilience testing: Scheduled chaos experiments as part of SRE practices
Pre-production validation: Test staging environments before production releases

Key Use Cases

Pod failure testing: Simulate pod crashes, node failures, or network partitions
Resource exhaustion: Test system behavior under CPU, memory, or disk pressure
Network latency and loss: Simulate network partitions, high latency, or packet loss
Dependency failures: Test failure modes when dependent services fail
Self-healing verification: Confirm Kubernetes self-healing mechanisms work correctly

Architecture Design Patterns

Core Components

Litmus Chaos Operator

litmus
├── chaos-operator (main orchestrator)
├── chaos-exporter (metrics collection)
├── chaos-dashboard (UI and API)
├── experiment-images (chaos experiments)
└── workflow-images (chaos workflows)

Chaos Operator: Manages chaos experiments and resources
Chaos Exporter: Exposes experiment metrics to Prometheus
Chaos Dashboard: Web UI for experiment management
Experiment Images: Docker containers running chaos scenarios
Workflow Images: Custom chaos experiment definitions

Chaos Experiment Structure

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: pod-delete
spec:
  definition:
    execution:
      - name: chaos
        image: litmuschaos/go-runner:latest
        command:
          - /bin/bash
          - -c
          - |
            # Chaos experiment logic
            kubectl delete pod ${CHAOSENGINE} -n ${TARGET_NAMESPACE}
        env:
          - name: TARGET_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: CHAOSENGINE
            value: ${CHAOSENGINE}

Experiment definitions: YAML configurations for chaos scenarios
Execution containers: Docker images running chaos logic
Environment variables: Configuration passed to experiment
RBAC permissions: Required Kubernetes permissions

Chaos Engine

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: nginx-chaos
  namespace: default
spec:
  appinfo:
    appns: default
    applabel: "app=nginx"
    appkind: deployment
  jobCleanUpPolicy: delete
  engineState: active
  chaosServiceAccount: litmus
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "30"
            - name: CHAOS_INTERVAL
              value: "10"
            - name: Pod_Delete
              value: "true"

Engine configuration: Main orchestration object
Target application: What to test and how
Experiment definitions: List of chaos experiments to run
Service account: RBAC permissions for experiments
Job cleanup: Cleanup behavior after experiments

Component Interactions

Chaos Engine
    ↓ (creates job)
Kubernetes Job
    ↓ (runs experiment)
Chaos Experiment Container
    ↓ (executes chaos)
Target Application
    ↓ (monitors results)
Chaos Exporter
    ↓ (collects metrics)
Prometheus

Data Flow Patterns

Experiment Execution Flow

1. ChaosEngine resource created
2. Chaos operator detects new engine
3. Creates Kubernetes Job for experiment
4. Job runs chaos experiment container
5. Experiment applies chaos to target
6. Results collected and reported
7. Job completed, results exported

Self-Healing Test Flow

1. Chaos experiment triggers failure
2. Kubernetes self-healing detects failure
3. Kubernetes recreates failed resources
4. Chaos experiment monitors recovery
5. Success/failure reported
6. Results stored for analysis

Design Principles

Idempotency

Experiments can be run multiple times safely
No state dependencies between runs
Clean rollback after each experiment

Declarative Configuration

All chaos defined as Kubernetes resources
Version control friendly YAML configurations
GitOps compatible chaos workflows

Minimal Instrumentation

No application code changes required
Works with any Kubernetes application
Label-based target selection

Integration Approaches

Integration with Other CNCF Projects

Prometheus Integration

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: litmus-exporter
  namespace: litmus
spec:
  selector:
    matchLabels:
      name: litmus-exporter
  namespaceSelector:
    matchNames:
      - litmus
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s

Metrics collection: Prometheus scrapes chaos exporter
Alerting: Prometheus alerts on chaos failure rates
Dashboard integration: Grafana dashboards for chaos metrics

Grafana Integration

{
  "annotations": {
    "list": [{
      "name": "Chaos Experiments",
      "datasource": "Prometheus",
      "enable": true,
      "expr": "chaos_experiment_total"
    }]
  },
  "panels": [
    {
      "title": "Experiment Success Rate",
      "targets": [{
        "expr": "chaos_experiment_success_ratio"
      }]
    }
  ]
}

Pre-built dashboards: Chaos experiment visualization
Real-time monitoring: Live experiment status
Historical analysis: Experiment trends over time

ArgoCD Integration

apiVersion: apps/v1
kind: ConfigMap
metadata:
  name: litmus-gitops
data:
  chaos-engine.yaml: |
    apiVersion: litmuschaos.io/v1alpha1
    kind: ChaosEngine
    metadata:
      name: gitops-chaos
    spec:
      appinfo: ...
      experiments: []

GitOps chaos: Chaos experiments version controlled
Automated execution: ArgoCD triggers chaos tests
Sync verification: GitOps sync status validated

Tekton Integration

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: chaos-test-pipeline
spec:
  pipelineRef:
    name: chaos-test-pipeline
  tasks:
    - name: chaos-test
      taskRef:
        name: litmus-chaos-test
      params:
        - name: engine
          value: "my-chaos-engine"

CI/CD integration: Chaos tests in CI/CD pipelines
Automated chaos: Trigger chaos on code changes
Test results: Pipeline results include chaos outcomes

Kyverno Integration

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-chaos-tests
spec:
  validationFailureAction: Enforce
  rules:
    - name: chaos-tested
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Production deployments must have chaos test coverage"
        deny: {}

Policy enforcement: Ensure chaos testing in CI/CD
Compliance checking: Verify chaos coverage
Automated gates: Block deployments without chaos tests

API Patterns

Experiment API

ChaosExperiment: Defines chaos scenario
ChaosEngine: Orchestrates chaos execution
ChaosResult: Stores experiment results
ChaosScheduler: Schedule recurring experiments

Result API

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosResult
metadata:
  name: nginx-chaos-pod-delete
  namespace: default
spec:
  engine: nginx-chaos
  experiment: pod-delete
  scenario: Chaos
  status:
    verdict: Pass
    failStep: ""
    probeSuccessPercentage: "100"
    lastUpdateTime: "2023-01-01T00:00:00Z"

Configuration Patterns

Chaos Engine Configuration

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: web-app-chaos
spec:
  appinfo:
    appns: default
    applabel: "app=web-app"
    appkind: deployment
  engineState: active
  jobCleanUpPolicy: delete
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TARGET_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: TOTAL_CHAOS_DURATION
              value: "60"
            - name: CHAOS_INTERVAL
              value: "15"
            - name: FORCE
              value: "true"

RBAC Configuration

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: litmus-role
rules:
  - apiGroups: [""]
    resources: ["pods", "services", "configmaps", "events", "pods/log"]
    verbs: ["get", "list", "delete"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "delete"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["get", "list", "create", "delete"]
  related-skills: null
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: litmus-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: litmus-role
subjects:
  - kind: ServiceAccount
    name: litmus
    namespace: litmus

Extension Mechanisms

Custom Experiment Templates

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: custom-experiment
spec:
  definition:
    execution:
      - name: chaos
        image: my-registry/custom-chaos:latest
        command:
          - /bin/bash
          - -c
          - ./run-custom-chaos.sh
        env:
          - name: CUSTOM_PARAM
            value: "value"

Workflow Templates

apiVersion: litmuschaos.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: resilience-test
spec:
  sequence:
    - experiment: pod-delete
    - experiment: pod-cpu-hog
    - experiment: pod-memory-hog
  parallel:
    - experiment: pod-network-latency
    - experiment: pod-disk-fill

Common Pitfalls and How to Avoid Them

Configuration Issues

Insufficient RBAC

Problem: Chaos experiments fail due to missing permissions.

Solution:

# Ensure proper RBAC
kubectl create clusterrolebinding litmus-binding \
  --clusterrole=litmus-role \
  --serviceaccount=litmus:default

Target Application Misconfiguration

Problem: Experiments don't target correct resources.

Solutions:

# Verify app labels
kubectl get pods --show-labels

# Correct ChaosEngine configuration
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
spec:
  appinfo:
    appns: default
    applabel: "app=nginx"  # Must match actual labels
    appkind: deployment

Resource Limits Not Set

Problem: Chaos experiments consume too many resources.

Solution:

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: chaos-runner
      resources:
        requests:
          memory: "256Mi"
          cpu: "100m"
        limits:
          memory: "512Mi"
          cpu: "500m"

Performance Issues

Experiment Overhead

Problem: Chaos experiments impact production performance.

Solutions:

Run experiments in staging first
Use non-production namespaces for initial testing
Schedule experiments during low-traffic periods
Limit experiment duration and frequency

Metrics Collection Impact

Problem: Chaos exporter affects application performance.

Solutions:

Deploy exporter in separate namespace
Configure appropriate scrape intervals
Limit metrics collection scope
Use remote storage for large-scale deployments

Operational Challenges

Experiment Failures

Problem: Experiments fail with unclear error messages.

Solutions:

# Check experiment logs
kubectl logs -f pod/litmus-chaos-runner-xxxxx -n litmus

# Check chaos result
kubectl get chaosresult nginx-chaos-pod-delete -o yaml

# Verify chaos operator logs
kubectl logs -n litmus deploy/chaos-operator

Cleanup Issues

Problem: Experiments leave resources behind.

Solutions:

Use jobCleanUpPolicy: delete
Implement proper cleanup in experiment scripts
Monitor for orphaned resources
Regular cleanup audits

Version Compatibility

Problem: Litmus versions incompatible with Kubernetes.

Solutions:

Check Litmus compatibility matrix
Test in staging before production
Use specific Litmus versions
Follow upgrade guides carefully

Security Pitfalls

Privilege Escalation

Problem: Chaos experiments gain excessive permissions.

Solutions:

Use least-privilege RBAC
Avoid cluster-admin service accounts
Audit experiment permissions regularly
Use dedicated chaos service accounts

Secret Exposure

Problem: Chaos configurations expose sensitive data.

Solutions:

Use Kubernetes secrets for sensitive data
Avoid secrets in experiment YAMLs
Encrypt secrets at rest
Audit secret access

Network Policies

Problem: Chaos experiments blocked by network policies.

Solutions:

Exclude chaos components from network policies
Create dedicated network policy exceptions
Test network chaos separately
Document network requirements

Coding Practices

Idiomatic Configuration

Standard Experiment YAML

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: pod-delete
  labels:
    name: pod-delete
    app.kubernetes.io/component: chaos-experiment
spec:
  definition:
    execution:
      - name: chaos
        image: litmuschaos/go-runner:latest
        command:
          - /bin/bash
          - -c
          - |
            # chaos logic here
            echo "Starting pod deletion chaos"
        env:
          - name: TARGET_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        resources:
          requests:
            memory: 128Mi
            cpu: 100m
          limits:
            memory: 256Mi
            cpu: 200m

API Usage Patterns

Running Experiments

# Create chaos engine
kubectl apply -f chaos-engine.yaml

# Monitor experiment
kubectl get chaosresult -w

# Check experiment status
kubectl get chaosengine nginx-chaos

# View experiment logs
kubectl logs -f deploy/chaos-operator -n litmus

Updating Experiments

# Update chaos engine
kubectl patch chaosengine nginx-chaos \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/experiments/0/spec/components/env/0/value", "value": "90"}]'

# Delete chaos engine
kubectl delete chaosengine nginx-chaos

Observability Best Practices

Metrics Collection

Experiment duration: Track experiment execution time
Success/failure rates: Monitor experiment outcomes
Resource usage: Track chaos component resource consumption
Failure trends: Identify recurring failure patterns

Logging Strategy

Experiment logs: Detailed chaos execution logs
Application logs: Monitor application response to chaos
Audit logs: Track who ran what experiment and when
Error logs: Capture and alert on failures

Development Workflow

Experiment Development

Create experiment YAML definition
Build chaos image with experiment logic
Test in isolated namespace
Verify cleanup behavior
Document experiment parameters
Version control configuration

CI/CD Integration

# Example GitLab CI
chaos-test:
  stage: test
  script:
    - kubectl apply -f chaos-engine.yaml
    - kubectl wait --for=condition=Complete job/litmus-chaos-runner --timeout=5m
    - kubectl get chaosresult -o yaml
  rules:
    - if: $CI_COMMIT_BRANCH == 'main'

Fundamentals

Essential Concepts

Chaos Experiment Lifecycle

Definition: YAML file defining chaos scenario
Execution: Kubernetes Job running chaos container
Monitoring: Real-time experiment status
Result: ChaosResult storing experiment outcome
Cleanup: Resource cleanup after completion

Litmus Architecture

Chaos Operator: Main orchestrator
Chaos Exporter: Metrics collection
Chaos Dashboard: Web UI
Experiment Images: Docker containers for chaos
Workflow Images: Custom chaos definitions

Terminology Glossary

Data Models and Types

ChaosExperiment Schema

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: string
spec:
  definition:
    execution:
      - name: string
        image: string
        command: []
        env: []
        resources: {}

ChaosEngine Schema

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: string
spec:
  appinfo: {}
  engineState: active|stop
  jobCleanUpPolicy: delete|keep
  chaosServiceAccount: string
  experiments: []

Lifecycle Management

Installation Lifecycle

Prerequisites: Kubernetes cluster, kubectl
Install Litmus: kubectl apply -f litmus-operator.yaml
Create Service Account: RBAC configuration
Deploy Experiments: Apply experiment YAMLs
Verify Installation: Run litmus check
Configure Experiments: Create ChaosEngine resources

Experiment Lifecycle

Definition: Create ChaosExperiment YAML
Execution: Create ChaosEngine referencing experiment
Monitoring: Watch experiment progress
Result: Collect ChaosResult
Cleanup: Remove ChaosEngine

State Management

Experiment State

Pending: Waiting to start
Running: Currently executing
Completed: Finished execution
Failed: Failed to complete
Stopped: Manually stopped

Application State During Chaos

Normal: No chaos active
Degraded: Chaos causing issues
Recovered: Self-healing occurred
Failed: Recovery failed

Scaling and Deployment Patterns

Horizontal Scaling

Multiple Chaos Engines

# Run chaos in multiple namespaces
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: chaos-web-namespace-1
  namespace: namespace-1
spec: ...
  related-skills: null
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: chaos-web-namespace-2
  namespace: namespace-2
spec: ...

Parallel Experiments

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
spec:
  experiments:
    - name: pod-delete
    - name: pod-cpu-hog
    - name: pod-network-latency

High Availability

Chaos Operator HA

# Deploy multiple operator replicas
kubectl scale -n litmus deploy/chaos-operator --replicas=3

Dashboard HA

# Deploy dashboard with replicas
kubectl scale -n litmus deploy/litmus-dashboard --replicas=3

Production Deployments

Multi-Cluster Chaos

# Export chaos results from all clusters
kubectl get chaosresult --all-namespaces -o json > results.json

Scheduled Experiments

apiVersion: batch/v1
kind: CronJob
metadata:
  name: weekly-chaos
spec:
  schedule: "0 0 * * 0"  # Sunday at midnight
  jobTemplate:
    spec:
      template:
        spec:
          # Create chaos engine
          restartPolicy: Never

Upgrade Strategies

Rolling Updates

Test new version in staging
Update chaos operator
Update experiment images
Update chaos engines
Verify functionality

Blue-Green Updates

Deploy new version alongside old
Run parallel experiments
Compare results
Switch traffic to new version
Decommission old version

Resource Management

Resource Sizing

resources:
  requests:
    memory: "128Mi"
    cpu: "100m"
  limits:
    memory: "256Mi"
    cpu: "500m"

Storage Management

Result storage: ChaosResult storage requirements
Log rotation: Exponential backoff for retries
Cleanup: Automatic resource cleanup
Export: Export results for analysis

Additional Resources

Official Documentation

Litmus Documentation: https://litmuschaos.io/
Litmus GitHub: https://github.com/litmuschaos/litmus
Chaos Hub: https://hub.litmuschaos.io/
Getting Started: https://docs.litmuschaos.io/docs/getstarted

CNCF References

Litmus in CNCF Landscape: https://landscape.cncf.io/?group=projects&filter=chaos-engineering
CNCF Chaos Engineering: https://github.com/cncf/sig-chaos-engineering

Tools and Libraries

litmusctl: CLI tool for Litmus
pylitmus: Python client library
litmus-go: Go SDK for chaos experiments
chaos-mesh: Alternative chaos platform

Tutorials and Guides

Basic Chaos Experiment: https://docs.litmuschaos.io/docs/basic-chaos-experiment
Custom Experiment: https://docs.litmuschaos.io/docs/create-custom-experiment
Self-Healing: https://docs.litmuschaos.io/docs/self-healing
Multicluster: https://docs.litmuschaos.io/docs/multicluster-chaos

Community Resources

Litmus Slack: https://kubernetes.slack.com/archives/CMR557BAG
Litmus Twitter: https://twitter.com/litmus_chaos
Blog: https://litmuschaos.io/blog/
Community Calls: Weekly community meetings

Related Standards

SRE Book: Google SRE practices
Chaos Engineering: Principles and practices
Resilience Patterns: System resilience patterns

OpenTelemetry Integration

Tracing: Chaos experiment tracing
Metrics: Prometheus metrics collection
Logs: Chaos-related logging
Events: Kubernetes event integration

Troubleshooting

Common Issues

Deployment Failures
- Check pod logs for errors
- Verify configuration values
- Ensure network connectivity
Performance Issues
- Monitor resource usage
- Adjust resource limits
- Check for bottlenecks
Configuration Errors
- Validate YAML syntax
- Check required fields
- Verify environment-specific settings
Integration Problems
- Verify API compatibility
- Check dependency versions
- Review integration documentation

Getting Help

Check official documentation
Search GitHub issues
Join community channels
Review logs and metrics

Examples

Basic Configuration

# Basic configuration example
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{project_name}}-config
  namespace: default
data:
  # Configuration goes here
  config.yaml: |
    # Base configuration
    # Add your settings here

Kubernetes Deployment

# Kubernetes deployment for {{project_name}}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{project_name}}
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: {{project_name}}
  template:
    metadata:
      labels:
        app: {{project_name}}
    spec:
      containers:
      - name: {{project_name}}
        image: {{project_name}}:latest
        ports:
        - containerPort: 8080
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"

Kubernetes Service

# Kubernetes service for {{project_name}}
apiVersion: v1
kind: Service
metadata:
  name: {{project_name}}
  namespace: default
spec:
  selector:
    app: {{project_name}}
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: ClusterIP

Constraints

MUST DO

Cite authoritative primary sources (official documentation, RFCs, standards bodies) — avoid secondary or blog references
Include version-specific guidance when the reference topic has significant version-dependent behavior
Structure reference content with clear navigation: overview first, then detailed subsections organized by use case
Keep examples minimal and self-contained so readers can copy-paste without needing external context

MUST NOT DO

Do not present opinionated practices as facts — distinguish between standards, recommendations, and personal preferences
Avoid outdated API references or deprecated patterns; explicitly note version requirements for each code example
Never include incomplete or pseudocode examples in reference materials — all examples should be runnable
Do not conflate different product versions when documenting features that vary across releases

MAKE IT PART OF YOUR WORKFLOW

Installer la Skill

TERMINAL

npx skills add paulpas/agent-skill-router --skill litmus

Exécutez la commande dans votre terminal. Vérifiez le code et la licence avant l’installation.

ZIP · Complete skill directory with resources and licenses

SKILL.md · Instructions file only

Étoiles du dépôt: Not synced
Licence: MIT
Version / branche: HEAD · d2e2ac6f
Dépôt mis à jour: Not synced

Vérifiez les scripts et les autorisations avant l’installation.

Code source