fluentd

---

name: fluentd compatibility: opencode completeness: 95 content-types:

• guidance
• examples
• do-dont
• config description: '"Fluentd unified logging layer for collecting, transforming, and routing" log data in cloud-native environments' license: MIT maturity: stable metadata: domain: cncf output-format: manifests role: reference scope: infrastructure triggers: fluentd, log collection, log routing, logging, unified logging, cloudwatch, log forwarding, monitoring archetypes:
  • educational
  • strategic anti_triggers:
  • brainstorming
  • vague ideation
  • non-containerized architecture response_profile: verbosity: medium directive_strength: low abstraction_level: strategic version: "1.0.0"

---

related-skills: cncf-aws-cloudwatch, cncf-azure-monitor, cncf-cortex, cncf-gcp-autoscaling

Tutorial

This tutorial will guide you through installing, configuring, and using Fluentd for centralized logging collection.

Prerequisites

Before beginning, ensure you have:

• A running Kubernetes cluster (minikube, kind, EKS, GKE, AKS)
• kubectl configured to access your cluster
• Basic understanding of logging concepts and Kubernetes architecture
• A logging destination (Elasticsearch, Fluentd, S3, etc.)

Verify your setup:

# Check cluster connectivity
kubectl cluster-info

# Verify kubectl configuration
kubectl get nodes

# Check cluster version
kubectl version --client --short

---

1. Installation

Fluentd can be installed in various ways depending on your environment and requirements.

Method 1: Using Helm (Recommended for Kubernetes)

# Add the Fluentd Helm repository
helm repo add fluent-stable https://fluent.github.io/helm-charts
helm repo add fluentd https://fluent.github.io/helm-charts

# Update repositories
helm repo update

# Install Fluentd with default configuration
helm install fluentd fluent-stable/fluentd \
  --namespace logging \
  --create-namespace \
  --wait

# Install with custom values
helm install fluentd fluent-stable/fluentd \
  --namespace logging \
  --create-namespace \
  --set fluentd.conf.type="s3" \
  --set output.host="elasticsearch.logging.svc.cluster.local"

# Verify installation
helm list -n logging
kubectl -n logging get pods

Method 2: Using kubectl apply (Direct Manifests)

# Download the Fluentd manifest
curl -O https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-kubernetes-1.16.yaml

# Or use the simplified version
kubectl apply -f https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-elasticsearch.yaml

# Verify deployment
kubectl get daemonset fluentd -n kube-system
kubectl get pods -n kube-system -l app=fluentd

Method 3: Installing Fluentd on Bare Metal

# Install Fluentd using RubyGems
gem install fluentd --no-document

# Or use td-agent (Fluentd's distribution)
# For Ubuntu/Debian
curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-focal-td-agent4.sh | sh

# For CentOS/RHEL
curl -fsSL https://toolbelt.treasuredata.com/sh/install-redhat-centos-td-agent4.sh | sh

# Verify installation
td-agent --version
fluentd --version

Method 4: Using Docker

# Pull the Fluentd Docker image
docker pull fluent/fluentd:v1.16

# Run Fluentd with custom configuration
docker run -d \
  -p 24224:24224 \
  -v /path/to/fluent.conf:/fluentd/etc/fluent.conf \
  -v /var/log:/fluentd/log \
  fluent/fluentd:v1.16

# Verify container is running
docker ps | grep fluentd

---

2. Basic Configuration

Fluentd Configuration File Structure

# fluent.conf - Basic configuration
<system>
  log_level info
  suppress_config_dump true
</system>

# Source: Kubernetes container logs
<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  read_from_head true
  <parse>
    @type json
    time_key time
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </parse>
</source>

# Filter: Enrich logs with Kubernetes metadata
<filter kubernetes.**>
  @type kubernetes_metadata
  kubernetes_url https://kubernetes.default.svc
  verify_ssl false
</filter>

# Match: Send logs to Elasticsearch
<match **>
  @type elasticsearch
  host elasticsearch.logging.svc.cluster.local
  port 9200
  logstash_format true
  logstash_prefix kubernetes
  flatten_hashes true
  <buffer>
    @type file
    path /var/log/fluentd/buffer
    flush_interval 5s
    flush_thread_count 2
    retry_type exponential_backoff
  </buffer>
</match>

Environment Variables for Configuration

# Set environment variables for runtime configuration
export FLUENTD_CONF=fluent.conf
export FLUENTD_ARGS=--no-config-autoreload

# Kubernetes environment variables (for kubernetes_metadata filter)
export KUBERNETES_SERVICE_HOST=kubernetes.default.svc
export KUBERNETES_SERVICE_PORT=443

ConfigMap for Kubernetes

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: logging
data:
  fluent.conf: |
    <system>
      log_level info
    </system>
    
    <source>
      @type forward
      port 24224
      bind 0.0.0.0
    </source>
    
    <match **>
      @type file
      path /var/log/fluentd/output
      append true
      <format>
        @type json
      </format>
      <buffer>
        @type file
        path /var/log/fluentd/buffer
        flush_interval 1s
      </buffer>
    </match>
  
  parsers.conf: |
    <parse>
      @type json
      time_key time
      time_format %Y-%m-%dT%H:%M:%S.%NZ
    </parse>

---

related-skills: cncf-aws-cloudwatch, cncf-azure-monitor, cncf-cortex, cncf-gcp-autoscaling

3. Usage Examples

Viewing Fluentd Logs

# View Fluentd logs in Kubernetes
kubectl -n logging logs -f deployment/fluentd

# Follow Fluentd logs with timestamp
kubectl -n logging logs -f deployment/fluentd --timestamps

# View logs from a specific pod
kubectl -n logging logs fluentd-xyz123 -f

# Check Fluentd pod status
kubectl -n logging get pods -l app=fluentd

Testing Log Collection

# Create a test pod to generate logs
kubectl run test-logger --image=busybox --restart=Never -- tail -f /dev/null

# In another terminal, send a test log
kubectl exec test-logger -- sh -c 'echo '{"level":"info","message":"Test log message"}' | fluent-cat test.tag'

# Verify the log was collected
kubectl -n logging logs -l app=fluentd | grep "Test log"

Loading Configuration from ConfigMap

# Create ConfigMap from fluent.conf
kubectl create configmap fluentd-config \
  --from-file=fluent.conf \
  --namespace=logging

# Or update existing ConfigMap
kubectl create configmap fluentd-config \
  --from-file=fluent.conf \
  --namespace=logging \
  --dry-run=client \
  -o yaml | kubectl apply -f -

# Verify ConfigMap
kubectl -n logging get configmap fluentd-config -o yaml

Setting up Multiple Outputs

# Multiple output configuration
<match **>
  @type copy
  <store>
    @type elasticsearch
    host elasticsearch.logging.svc.cluster.local
    port 9200
    logstash_format true
  </store>
  <store>
    @type s3
    s3_bucket my-logs-bucket
    s3_region us-east-1
    path logs/%Y/%m/%d/
    <format>
      @type json
    </format>
    <buffer>
      @type file
      path /var/log/fluentd/s3
    </buffer>
  </store>
  <store>
    @type stdout
  </store>
</match>

---

4. Common Operations

Monitoring Fluentd

# Check Fluentd metrics (if Prometheus plugin is enabled)
kubectl -n logging get pods -l app=fluentd -o wide

# View Fluentd pod resources
kubectl -n logging describe pod -l app=fluentd

# Check Fluentd configuration reload status
kubectl -n logging logs deployment/fluentd | grep "config file"

# View Fluentd buffer status
kubectl -n logging exec -it deployment/fluentd -- ls -la /var/log/fluentd/buffer

Restarting Fluentd DaemonSet

# Rolling restart of Fluentd pods
kubectl -n logging delete pods -l app=fluentd

# Wait for pods to restart
kubectl -n logging rollout status daemonset/fluentd

# Verify all pods are running
kubectl -n logging get pods -l app=fluentd

Updating Configuration

# Update ConfigMap
kubectl create configmap fluentd-config \
  --from-file=fluent.conf \
  --namespace=logging \
  --dry-run=client \
  -o yaml | kubectl apply -f -

# Delete pods to trigger reload
kubectl -n logging delete pods -l app=fluentd

Scaling Fluentd

# Fluentd typically runs as a DaemonSet (one per node)
# To scale, update the DaemonSet
kubectl -n logging patch daemonset fluentd -p '{"spec":{"template":{"spec":{"containers":[{"name":"fluentd","resources":{"limits":{"cpu":"500m","memory":"512Mi"}}}]}}}}'

# Or edit the daemonset
kubectl -n logging edit daemonset fluentd

Backup and Restore

# Backup Fluentd configuration
kubectl -n logging get configmap fluentd-config -o yaml > fluentd-config-backup.yaml

# Export buffer state (if needed)
kubectl -n logging exec deployment/fluentd -- tar czf /tmp/buffer-backup.tar.gz /var/log/fluentd/buffer

# Restore configuration
kubectl -n logging apply -f fluentd-config-backup.yaml

---

5. Best Practices

Configuration Best Practices

1. Use Separate Sources: Define separate <source> blocks for different log types
2. Filter Early: Apply filters as close to the source as possible
3. Buffer Properly: Configure buffer to prevent data loss during outages
4. Use Tags: Use meaningful tags for route matching
5. Separate Concerns: Keep configuration modular with @include directives

Performance Optimization

1. Optimize Buffer: Adjust buffer settings based on log volume

   <buffer>
     @type file
     flush_interval 5s
     flush_thread_count 2
     chunk_limit_size 8MB
     queued_chunks_limit_size 32
   </buffer>
   

2. Use Multiple Threads: Configure flush_thread_count for parallel processing

3. Reduce Parsing: Use forward input to receive pre-parsed logs when possible

4. Compress Buffers: Enable compression for large buffers

Security Best Practices

1. RBAC: Configure proper RBAC for Fluentd service account
2. Network Policies: Restrict Fluentd network access to required destinations
3. Secrets Management: Use Kubernetes Secrets for sensitive configuration
4. Log Redaction: Implement filters to redact sensitive data
5. TLS Encryption: Use TLS for log transmission to external systems

Monitoring and Alerting

1. Prometheus Metrics: Enable Fluentd's Prometheus plugin

   <source>
     @type prometheus
     bind 0.0.0.0
     port 24231
   </source>
   

2. Health Checks: Configure readiness/liveness probes

3. Alert on Backlog: Monitor buffer size and alert on growth

4. Log Volume Monitoring: Track log volume trends

Kubernetes-Specific Tips

1. Use DaemonSet: Deploy Fluentd as a DaemonSet for node-level log collection
2. Namespace Isolation: Use separate Fluentd instances per namespace for isolation
3. Label Selectors: Use label selectors to filter logs by application
4. Resource Limits: Set appropriate CPU/memory limits based on cluster size

---

Examples

Basic Configuration

# Basic configuration example
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: logging
data:
  fluent.conf: |
    <system>
      log_level info
    </system>
    
    <source>
      @type forward
      port 24224
      bind 0.0.0.0
    </source>
    
    <match **>
      @type file
      path /var/log/fluentd/output
      append true
      <format>
        @type json
      </format>
      <buffer>
        @type file
        path /var/log/fluentd/buffer
        flush_interval 1s
      </buffer>
    </match>

Kubernetes Deployment

# Fluentd DaemonSet for Kubernetes
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging
  labels:
    app: fluentd
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd:v1.16
        env:
        - name: FLUENT_UID
          value: "0"
        resources:
          limits:
            memory: "512Mi"
            cpu: "500m"
        volumeMounts:
        - name: config-volume
          mountPath: /fluentd/etc
        - name: varlog
          mountPath: /var/log
      volumes:
      - name: config-volume
        configMap:
          name: fluentd-config
      - name: varlog
        hostPath:
          path: /var/log

Kubernetes Service

# Kubernetes service for Fluentd
apiVersion: v1
kind: Service
metadata:
  name: fluentd
  namespace: logging
spec:
  selector:
    app: fluentd
  ports:
  - protocol: TCP
    port: 24224
    targetPort: 24224
    name: fluentd
  - protocol: TCP
    port: 24231
    targetPort: 24231
    name: prometheus
  type: ClusterIP

Kubernetes Container Log Collection

# Collect logs from Kubernetes containers
<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  read_from_head true
  <parse>
    @type json
    time_key time
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </parse>
</source>

# Enrich with Kubernetes metadata
<filter kubernetes.**>
  @type kubernetes_metadata
  kubernetes_url https://kubernetes.default.svc
  verify_ssl false
</filter>

# Send to Elasticsearch
<match kubernetes.**>
  @type elasticsearch
  host elasticsearch.logging.svc.cluster.local
  port 9200
  logstash_format true
  logstash_prefix kubernetes
  <buffer>
    @type file
    path /var/log/fluentd/buffer
    flush_interval 5s
  </buffer>
</match>

Multi-Output Configuration

# Copy logs to multiple destinations
<match **>
  @type copy
  <store>
    @type elasticsearch
    host es1.example.com
    port 9200
  </store>
  <store>
    @type elasticsearch
    host es2.example.com
    port 9200
  </store>
  <store>
    @type s3
    s3_bucket my-logs
    s3_region us-east-1
  </store>
</match>

---

When to Use

Use this skill when:

• Integrating a CNCF project into Kubernetes infrastructure — You need to configure, deploy, or troubleshoot a cloud-native tool within a cluster
• Designing cloud-native architecture — You are selecting and integrating CNCF tools to solve specific infrastructure challenges
• Resolving operational issues — A CNCF component is misbehaving, underperforming, or needs configuration changes

---

Core Workflow

1. Assess Requirements — Understand the use case, scale, integration needs, and existing infrastructure. Checkpoint: Document requirements, constraints, and success criteria.

2. Design Architecture — Plan component interactions, data flow, and deployment strategy using cloud-native best practices. Checkpoint: Verify the architecture addresses all requirements and follows CNCF conventions.

3. Implement & Configure — Create manifests, configurations, and deployment scripts. Include resource limits, health checks, and observability hooks. Checkpoint: Validate all YAML against schema and test in a staging environment.

4. Deploy & Monitor — Apply manifests to the cluster, verify component health, and confirm observability is working. Checkpoint: Confirm all pods/services are running, probes passing, and metrics/alerts configured.

---

Constraints

MUST DO

• Include at least one complete working YAML manifest example
• Note when content is auto-generated vs. manually verified
• Reference relevant CNCF project documentation

MUST NOT DO

• Deploy manifests without testing in a staging environment first
• Use deprecated API versions (e.g., apps/v1beta1)
• Omit resource limits and requests in Kubernetes manifests