---

name: ubuntu-monitoring description: Use when setting up monitoring and logging on Ubuntu 24.04 LTS — Prometheus and node_exporter, Grafana dashboards, Loki/Alloy log aggregation (Promtail legacy), ELK stack (Elasticsearch, Logstash, Kibana), alerting with Alertmanager, systemd journal management, rsyslog, and health check patterns. Part of the ubuntu-* skill family. family: ubuntu applies_when: os_family == debian

Ubuntu Server 24.04 LTS — Monitoring and Logging

Companion skill to ubuntu-server-admin. Covers metrics collection, visualization, log aggregation, alerting, and health checks on Ubuntu Server 24.04.4 LTS (Noble Numbat).

---

1. Prometheus

Installation (Binary Method)

# Create system user
sudo useradd -r -s /usr/sbin/nologin prometheus

# Create directories
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

# Download and install (check https://prometheus.io/download/ for latest)
PROM_VER="3.12.0"   # verified current 2026-06-10
cd /tmp
curl -LO "https://github.com/prometheus/prometheus/releases/download/v${PROM_VER}/prometheus-${PROM_VER}.linux-amd64.tar.gz"
tar xzf "prometheus-${PROM_VER}.linux-amd64.tar.gz"
sudo cp "prometheus-${PROM_VER}.linux-amd64"/{prometheus,promtool} /usr/local/bin/
sudo chown -R prometheus:prometheus /etc/prometheus

Note: Prometheus 3.x tarballs no longer ship the example consoles/ and console_libraries/ directories (removed in 3.0). If you are upgrading an existing 2.x install rather than doing a fresh install, read the official 2.x → 3.x migration guide first.

Configuration — /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 10s

rule_files:
  - /etc/prometheus/rules/*.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - localhost:9093

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets:
          - "localhost:9100"
          - "10.0.1.51:9100"
          - "10.0.1.52:9100"

  - job_name: "custom-app"
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets: ["localhost:8080"]

Recording Rules — /etc/prometheus/rules/recording.yml

groups:
  - name: node_recording
    interval: 15s
    rules:
      - record: node:cpu_utilization:ratio
        expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))
      - record: node:memory_utilization:ratio
        expr: 1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)
      - record: node:filesystem_usage:ratio
        expr: 1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})

Systemd Service — /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus Monitoring
After=network-online.target
Wants=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --storage.tsdb.retention.size=10GB \
  --web.listen-address=0.0.0.0:9090 \
  --web.enable-lifecycle
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
sudo ufw allow from 10.0.0.0/8 to any port 9090 proto tcp

Validate Config

promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/recording.yml

---

2. node_exporter

Installation

sudo useradd -r -s /usr/sbin/nologin node_exporter

NODE_VER="1.11.1"   # verified current 2026-06-10
cd /tmp
curl -LO "https://github.com/prometheus/node_exporter/releases/download/v${NODE_VER}/node_exporter-${NODE_VER}.linux-amd64.tar.gz"
tar xzf "node_exporter-${NODE_VER}.linux-amd64.tar.gz"
sudo cp "node_exporter-${NODE_VER}.linux-amd64/node_exporter" /usr/local/bin/

Systemd Service — /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
  --collector.textfile.directory=/var/lib/node_exporter/textfile_collector \
  --collector.systemd \
  --collector.processes
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo mkdir -p /var/lib/node_exporter/textfile_collector
sudo chown node_exporter:node_exporter /var/lib/node_exporter/textfile_collector
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
sudo ufw allow from 10.0.0.0/8 to any port 9100 proto tcp

Custom Textfile Collector Example

# Write custom metrics for Prometheus to scrape
# Run via cron or systemd timer — output to textfile_collector directory
cat <<'SCRIPT' | sudo tee /usr/local/bin/custom-metrics.sh
#!/bin/bash
OUTPUT="/var/lib/node_exporter/textfile_collector/custom.prom"
# Pending apt updates
UPDATES=$(apt list --upgradable 2>/dev/null | grep -c upgradable)
echo "apt_upgradable_packages ${UPDATES}" > "${OUTPUT}.tmp"
# Reboot required
REBOOT=0
[ -f /var/run/reboot-required ] && REBOOT=1
echo "node_reboot_required ${REBOOT}" >> "${OUTPUT}.tmp"
mv "${OUTPUT}.tmp" "${OUTPUT}"
SCRIPT
sudo chmod +x /usr/local/bin/custom-metrics.sh

Key Metrics to Monitor

Metric	PromQL	Alert threshold
CPU usage	node:cpu_utilization:ratio	> 0.85 for 10m
Memory usage	node:memory_utilization:ratio	> 0.90 for 5m
Disk usage	node:filesystem_usage:ratio	> 0.85
Disk I/O util	rate(node_disk_io_time_seconds_total[5m])	> 0.90 for 15m
Network errors	rate(node_network_receive_errs_total[5m])	> 0
Systemd failed	node_systemd_unit_state{state="failed"}	== 1

---

3. Grafana

Installation from Official Repository

sudo apt install -y apt-transport-https software-properties-common
curl -fsSL https://apt.grafana.com/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/grafana.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/grafana.gpg] https://apt.grafana.com stable main" \
  | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafana
sudo systemctl enable --now grafana-server
sudo ufw allow 3000/tcp

Default login: admin / admin (change immediately on first login).

Provisioning Datasources — /etc/grafana/provisioning/datasources/prometheus.yaml

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090
    isDefault: true
    editable: false
  - name: Loki
    type: loki
    access: proxy
    url: http://localhost:3100
    editable: false

Provisioning Dashboards — /etc/grafana/provisioning/dashboards/default.yaml

apiVersion: 1
providers:
  - name: default
    orgId: 1
    folder: Provisioned
    type: file
    disableDeletion: false
    updateIntervalSeconds: 60
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: false

Import Community Dashboards

# Download "Node Exporter Full" dashboard (ID 1860) for file provisioning
sudo mkdir -p /var/lib/grafana/dashboards
curl -s "https://grafana.com/api/dashboards/1860/revisions/latest/download" \
  | sudo tee /var/lib/grafana/dashboards/node-exporter-full.json > /dev/null

# Or import via API
curl -s -X POST http://admin:admin@localhost:3000/api/dashboards/import \
  -H "Content-Type: application/json" \
  -d '{
    "dashboard": { "id": null },
    "overwrite": true,
    "inputs": [{ "name": "DS_PROMETHEUS", "type": "datasource", "pluginId": "prometheus", "value": "Prometheus" }],
    "folderId": 0
  }'

Grafana Config Highlights — /etc/grafana/grafana.ini

[server]
http_port = 3000
root_url = http://grafana.example.com:3000

[security]
admin_password = changeme_on_first_boot
disable_gravatar = true

[auth.anonymous]
enabled = false

[log]
mode = console file
level = warn

sudo systemctl restart grafana-server

---

4. Alertmanager

Installation

sudo useradd -r -s /usr/sbin/nologin alertmanager
AM_VER="0.32.1"   # verified current 2026-06-10
cd /tmp
curl -LO "https://github.com/prometheus/alertmanager/releases/download/v${AM_VER}/alertmanager-${AM_VER}.linux-amd64.tar.gz"
tar xzf "alertmanager-${AM_VER}.linux-amd64.tar.gz"
sudo cp "alertmanager-${AM_VER}.linux-amd64"/{alertmanager,amtool} /usr/local/bin/
sudo mkdir -p /etc/alertmanager /var/lib/alertmanager
sudo chown alertmanager:alertmanager /var/lib/alertmanager

Configuration — /etc/alertmanager/alertmanager.yml

global:
  resolve_timeout: 5m
  smtp_smarthost: "smtp.example.com:587"
  smtp_from: "alerts@example.com"
  smtp_auth_username: "alerts@example.com"
  smtp_auth_password: "app-password-here"
  smtp_require_tls: true

route:
  receiver: default-email
  group_by: [alertname, instance]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: slack-critical
      repeat_interval: 1h
    - match:
        severity: page
      receiver: pagerduty

receivers:
  - name: default-email
    email_configs:
      - to: "ops-team@example.com"
        send_resolved: true

  - name: slack-critical
    slack_configs:
      - api_url: "https://hooks.slack.com/services/T00/B00/XXXX"
        channel: "#alerts-critical"
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
        send_resolved: true

  - name: pagerduty
    pagerduty_configs:
      - service_key: "your-pagerduty-integration-key"
        severity: critical

inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: [alertname, instance]

Systemd Service — /etc/systemd/system/alertmanager.service

[Unit]
Description=Alertmanager
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager \
  --web.listen-address=0.0.0.0:9093
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now alertmanager

Alert Rules in Prometheus — /etc/prometheus/rules/alerts.yml

groups:
  - name: node_alerts
    rules:
      - alert: HighCPU
        expr: node:cpu_utilization:ratio > 0.85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"
          description: "CPU usage is {{ $value | humanizePercentage }} for 10+ minutes."

      - alert: HighMemory
        expr: node:memory_utilization:ratio > 0.90
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Memory critically high on {{ $labels.instance }}"

      - alert: DiskSpaceLow
        expr: node:filesystem_usage:ratio > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Disk space above 85% on {{ $labels.instance }}"

      - alert: InstanceDown
        expr: up == 0
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is unreachable"

  - name: prometheus_alerts
    rules:
      - alert: PrometheusTargetMissing
        expr: up == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Prometheus target {{ $labels.instance }} is down"

Manage Silences

# Create a silence via amtool
amtool --alertmanager.url=http://localhost:9093 silence add \
  alertname="HighCPU" instance="10.0.1.51:9100" \
  --comment="Maintenance window" --duration=2h

# List active silences
amtool --alertmanager.url=http://localhost:9093 silence query

# Expire a silence
amtool --alertmanager.url=http://localhost:9093 silence expire <silence-id>

---

5. Loki + Grafana Alloy (Log Shipping)

Log-shipper status (verified 2026-06-10): Grafana deprecated Promtail in February 2025 (LTS from 2025-02-13) in favor of Grafana Alloy, and Promtail reached end of life on 2026-03-02 — no further updates, including security fixes. Use Alloy for ALL new installs. The Promtail material is kept below only as a clearly-marked LEGACY/migration appendix for existing deployments.

Loki Installation

sudo useradd -r -s /usr/sbin/nologin loki
LOKI_VER="3.7.2"   # verified current 2026-06-10
cd /tmp
curl -LO "https://github.com/grafana/loki/releases/download/v${LOKI_VER}/loki-linux-amd64.zip"
sudo apt install -y unzip
unzip loki-linux-amd64.zip
sudo cp loki-linux-amd64 /usr/local/bin/loki
sudo chmod +x /usr/local/bin/loki
sudo mkdir -p /etc/loki /var/lib/loki
sudo chown loki:loki /var/lib/loki

Loki Config — /etc/loki/loki.yml

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /var/lib/loki
  storage:
    filesystem:
      chunks_directory: /var/lib/loki/chunks
      rules_directory: /var/lib/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 30d
  max_query_series: 5000
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20

compactor:
  working_directory: /var/lib/loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h

Loki Systemd Service — /etc/systemd/system/loki.service

[Unit]
Description=Loki Log Aggregation
After=network-online.target

[Service]
User=loki
Group=loki
Type=simple
ExecStart=/usr/local/bin/loki -config.file=/etc/loki/loki.yml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Grafana Alloy — Log Shipper (PRIMARY)

Alloy is Grafana's OpenTelemetry-based collector and the supported replacement for Promtail (Alloy 1.16.x current as of 2026-06). It installs from the same Grafana apt repository configured in section 3 — if you already added that repo, just install the package:

# Repo already added in section 3? Then simply:
sudo apt update
sudo apt install -y alloy

# Otherwise add the Grafana repo first (same as section 3):
curl -fsSL https://apt.grafana.com/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/grafana.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/grafana.gpg] https://apt.grafana.com stable main" \
  | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update && sudo apt install -y alloy

The package installs a systemd service named alloy running as the alloy user, with config at /etc/alloy/config.alloy and service options (e.g. CUSTOM_ARGS) in /etc/default/alloy. Grant it read access to the journal and privileged log files instead of running it as root:

sudo usermod -aG systemd-journal,adm alloy   # journal + /var/log/syslog,auth.log (root:adm on Ubuntu)

Alloy Config — /etc/alloy/config.alloy

Alloy uses its own HCL-like syntax (not YAML). Pipeline: journal + file sources → optional relabel → loki.write push to Loki on :3100.

// ── Systemd journal ────────────────────────────────────────────
loki.relabel "journal" {
  forward_to = []   // rules-only component; consumed via .rules below

  rule {
    source_labels = ["__journal__systemd_unit"]
    target_label  = "unit"
  }
  rule {
    source_labels = ["__journal__hostname"]
    target_label  = "hostname"
  }
}

loki.source.journal "system" {
  max_age       = "12h"
  labels        = { job = "systemd-journal" }
  relabel_rules = loki.relabel.journal.rules
  forward_to    = [loki.write.local.receiver]
}

// ── Plain log files ────────────────────────────────────────────
local.file_match "system_logs" {
  path_targets = [
    { __address__ = "localhost", __path__ = "/var/log/syslog",      job = "syslog" },
    { __address__ = "localhost", __path__ = "/var/log/auth.log",    job = "authlog" },
    { __address__ = "localhost", __path__ = "/var/log/myapp/*.log", job = "app" },
  ]
}

loki.source.file "system_logs" {
  targets    = local.file_match.system_logs.targets
  forward_to = [loki.write.local.receiver]
}

// ── Push to Loki ───────────────────────────────────────────────
loki.write "local" {
  endpoint {
    url = "http://localhost:3100/loki/api/v1/push"
  }
  external_labels = {}
}

# Syntax check (parses + formats the file; errors on invalid config)
alloy fmt /etc/alloy/config.alloy

sudo systemctl daemon-reload
sudo systemctl enable --now loki alloy
sudo systemctl reload alloy    # after later config edits
sudo ufw allow from 10.0.0.0/8 to any port 3100 proto tcp

Alloy serves a debug UI on 127.0.0.1:12345 by default (component health, live pipeline graph) — check it when logs are not arriving.

---

LEGACY — Promtail (EOL, migration only)

Do NOT use Promtail for new installs. Deprecated February 2025; end of life since 2026-03-02 (no security fixes). This appendix exists only for understanding/migrating existing Promtail deployments.

Migrating an existing deployment: Alloy ships a converter that translates a Promtail YAML config into Alloy syntax:

alloy convert --source-format=promtail --output=/etc/alloy/config.alloy /etc/promtail/promtail.yml
# Review the output, then disable promtail and enable alloy:
sudo systemctl disable --now promtail
sudo systemctl enable --now alloy

Legacy Promtail Install (reference only)

cd /tmp
curl -LO "https://github.com/grafana/loki/releases/download/v${LOKI_VER}/promtail-linux-amd64.zip"
unzip promtail-linux-amd64.zip
sudo cp promtail-linux-amd64 /usr/local/bin/promtail
sudo chmod +x /usr/local/bin/promtail
sudo mkdir -p /etc/promtail

Legacy Promtail Config — /etc/promtail/promtail.yml

server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
  # Systemd journal
  - job_name: journal
    journal:
      max_age: 12h
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: [__journal__systemd_unit]
        target_label: unit
      - source_labels: [__journal__hostname]
        target_label: hostname

  # System log files
  - job_name: syslog
    static_configs:
      - targets: [localhost]
        labels:
          job: syslog
          __path__: /var/log/syslog

  - job_name: authlog
    static_configs:
      - targets: [localhost]
        labels:
          job: authlog
          __path__: /var/log/auth.log

  # Application logs (wildcard)
  - job_name: app-logs
    static_configs:
      - targets: [localhost]
        labels:
          job: app
          __path__: /var/log/myapp/*.log

Legacy Promtail Systemd Service — /etc/systemd/system/promtail.service

[Unit]
Description=Promtail Log Collector
After=network-online.target

[Service]
User=root
Type=simple
ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/promtail.yml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Note: Promtail ran as root to read journal and privileged log files — one more reason to prefer Alloy (group-based access, see above). End of legacy appendix.

LogQL Query Examples

# All logs from a specific unit
{unit="nginx.service"}

# Filter by content
{job="syslog"} |= "error"
{job="authlog"} |= "Failed password"

# Regex filter
{unit="ssh.service"} |~ "Failed|Invalid"

# Rate of errors (logs per second over 5m)
rate({job="syslog"} |= "error" [5m])

# Top source IPs for SSH failures
topk(10, count_over_time({job="authlog"} |= "Failed password" [1h]))

# JSON log parsing
{job="app"} | json | status >= 500

---

6. ELK Stack (Elasticsearch, Logstash, Kibana)

Elasticsearch Installation

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/elasticsearch.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" \
  | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update
sudo apt install -y elasticsearch

Elasticsearch Config — /etc/elasticsearch/elasticsearch.yml

cluster.name: homelab-logging
node.name: elk-01
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.type: single-node

# Disable security for internal/lab use (enable in production)
xpack.security.enabled: false
xpack.security.enrollment.enabled: false

JVM Heap — /etc/elasticsearch/jvm.options.d/heap.options

-Xms4g
-Xmx4g

sudo systemctl enable --now elasticsearch
# Verify
curl -s http://localhost:9200/_cluster/health?pretty

Logstash Installation and Pipeline

sudo apt install -y logstash

Pipeline config — /etc/logstash/conf.d/syslog.conf:

input {
  beats {
    port => 5044
  }
}

filter {
  if [fields][log_type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:hostname} %{DATA:program}(?:\[%{POSINT:pid}\])?: %{GREEDYDATA:log_message}" }
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "syslog-%{+YYYY.MM.dd}"
  }
}

sudo systemctl enable --now logstash

Kibana

sudo apt install -y kibana

Config — /etc/kibana/kibana.yml:

server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]

sudo systemctl enable --now kibana
sudo ufw allow 5601/tcp

Filebeat (Log Shipper)

sudo apt install -y filebeat

Config — /etc/filebeat/filebeat.yml:

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/syslog
      - /var/log/auth.log
    fields:
      log_type: syslog

  - type: log
    enabled: true
    paths:
      - /var/log/nginx/access.log
    fields:
      log_type: nginx-access

output.logstash:
  hosts: ["localhost:5044"]

sudo filebeat modules enable system
sudo filebeat setup --pipelines --modules system
sudo systemctl enable --now filebeat

Index Lifecycle Management (ILM) Basics

# Create an ILM policy via API
curl -s -X PUT "http://localhost:9200/_ilm/policy/logs-cleanup" \
  -H "Content-Type: application/json" -d '{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "7d",
            "max_primary_shard_size": "25gb"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}'

---

7. rsyslog

Remote Syslog Collection (Central Server)

Add to /etc/rsyslog.d/10-remote.conf on the central server:

# UDP reception (traditional, faster, no delivery guarantee)
module(load="imudp")
input(type="imudp" port="514")

# TCP reception (reliable delivery)
module(load="imtcp")
input(type="imtcp" port="514")

Template-Based File Output

# /etc/rsyslog.d/20-remote-hosts.conf
# Separate log files per remote host and program
template(name="RemoteHostLog" type="string"
  string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log")

if $fromhost-ip != '127.0.0.1' then {
  action(type="omfile" dynaFile="RemoteHostLog")
  stop
}

sudo mkdir -p /var/log/remote
sudo systemctl restart rsyslog
sudo ufw allow 514/tcp
sudo ufw allow 514/udp

Forwarding to Central Server (Client)

Add to /etc/rsyslog.d/50-forward.conf on each client:

# Forward all logs via TCP (@@) — use single @ for UDP
*.* @@logserver.example.com:514

# Forward only auth logs
auth,authpriv.* @@logserver.example.com:514

# Queue for reliability during network outage
action(
  type="omfwd"
  target="logserver.example.com"
  port="514"
  protocol="tcp"
  queue.type="LinkedList"
  queue.size="10000"
  queue.filename="fwd_to_central"
  queue.saveonshutdown="on"
  action.resumeRetryCount="-1"
)

sudo systemctl restart rsyslog

TCP vs UDP

Factor	UDP (single @)	TCP (double @@)
Delivery	Best-effort, may drop	Reliable, ordered
Overhead	Lower	Slightly higher
Use case	High-volume, non-critical	Audit/compliance logs
Recommendation	Avoid for important logs	Preferred default

---

8. Health Check Patterns

Systemd Watchdog

Add watchdog support to a custom service:

# /etc/systemd/system/myapp.service — add to [Service]
WatchdogSec=30
# The application must call sd_notify("WATCHDOG=1") within this interval.
# If using a wrapper script or app that doesn't support sd_notify:
Type=notify
NotifyAccess=all

For apps that do not support sd_notify natively, use a wrapper:

# /usr/local/bin/watchdog-wrapper.sh
#!/bin/bash
/opt/myapp/bin/server &
APP_PID=$!
while kill -0 "$APP_PID" 2>/dev/null; do
    if curl -sf http://localhost:8080/health > /dev/null; then
        systemd-notify WATCHDOG=1
    fi
    sleep 10
done
exit 1

Custom Health Check Script with Systemd Timer

Service — /etc/systemd/system/health-check.service:

[Unit]
Description=System health check

[Service]
Type=oneshot
ExecStart=/usr/local/bin/health-check.sh

Timer — /etc/systemd/system/health-check.timer:

[Unit]
Description=Run health check every 5 minutes

[Timer]
OnBootSec=60
OnUnitActiveSec=5min
AccuracySec=30s

[Install]
WantedBy=timers.target

Script — /usr/local/bin/health-check.sh:

#!/bin/bash
set -euo pipefail

LOG_TAG="health-check"
ALERT_FILE="/var/lib/node_exporter/textfile_collector/health.prom"

check_passed=1

# Check disk space (alert if any mount > 85%)
while read -r usage mount; do
    pct="${usage%\%}"
    if [ "$pct" -gt 85 ]; then
        logger -t "$LOG_TAG" "WARN: ${mount} is at ${usage}"
        check_passed=0
    fi
done < <(df -h --output=pcent,target | tail -n+2 | grep -v tmpfs)

# Check systemd failed units
FAILED=$(systemctl --failed --no-legend | wc -l)
if [ "$FAILED" -gt 0 ]; then
    logger -t "$LOG_TAG" "WARN: ${FAILED} failed systemd units"
    check_passed=0
fi

# Check critical services
for svc in prometheus node_exporter grafana-server; do
    if systemctl is-active --quiet "$svc" 2>/dev/null; then
        : # running
    elif systemctl list-unit-files "$svc.service" | grep -q "$svc"; then
        logger -t "$LOG_TAG" "CRIT: ${svc} is not running"
        check_passed=0
    fi
done

# Check memory (alert if available < 10%)
MEM_AVAIL=$(awk '/MemAvailable/ {print $2}' /proc/meminfo)
MEM_TOTAL=$(awk '/MemTotal/ {print $2}' /proc/meminfo)
MEM_PCT=$(( (MEM_TOTAL - MEM_AVAIL) * 100 / MEM_TOTAL ))
if [ "$MEM_PCT" -gt 90 ]; then
    logger -t "$LOG_TAG" "WARN: Memory usage at ${MEM_PCT}%"
    check_passed=0
fi

# Export result as Prometheus metric
echo "node_health_check_passed ${check_passed}" > "${ALERT_FILE}.tmp"
echo "node_health_check_timestamp $(date +%s)" >> "${ALERT_FILE}.tmp"
mv "${ALERT_FILE}.tmp" "$ALERT_FILE"

logger -t "$LOG_TAG" "Health check completed — passed=${check_passed}"

sudo chmod +x /usr/local/bin/health-check.sh
sudo systemctl daemon-reload
sudo systemctl enable --now health-check.timer

Uptime Monitoring with Blackbox Exporter

BB_VER="0.28.0"   # verified current 2026-06-10
cd /tmp
curl -LO "https://github.com/prometheus/blackbox_exporter/releases/download/v${BB_VER}/blackbox_exporter-${BB_VER}.linux-amd64.tar.gz"
tar xzf "blackbox_exporter-${BB_VER}.linux-amd64.tar.gz"
sudo cp "blackbox_exporter-${BB_VER}.linux-amd64/blackbox_exporter" /usr/local/bin/

Blackbox config — /etc/blackbox_exporter/blackbox.yml:

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      follow_redirects: true
  icmp:
    prober: icmp
    timeout: 5s
  tcp_connect:
    prober: tcp
    timeout: 5s

Prometheus scrape config for blackbox:

# Add to scrape_configs in prometheus.yml
  - job_name: "blackbox-http"
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://app.example.com
          - https://grafana.example.com:3000
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

---

---

Anti-Patterns

Anti-Pattern	Why It Fails	Correct Approach
Alerting on every metric without baseline tuning	Alert fatigue — 50 alerts per day means operators ignore all of them; real incidents get missed	Establish baselines for 2 weeks; set alerts only for actionable anomalies; review and tune alert thresholds monthly
No persistent storage for Prometheus	Prometheus restart loses all metric history; cannot do capacity planning or trend analysis	Configure --storage.tsdb.path on persistent volume; set retention with --storage.tsdb.retention.time
Monitoring system metrics only (CPU, RAM, disk)	Server metrics look healthy while application returns errors; users report outage before monitoring detects it	Add application health checks, HTTP status code monitoring, response latency tracking alongside infrastructure metrics
Grafana dashboards without variable templates	One dashboard per server/service; 50 servers = 50 dashboards to maintain; inconsistent layouts	Use Grafana template variables (instance, job, namespace); one dashboard serves all instances with dropdown selection
Not forwarding logs to central system	Logs only on individual servers; cross-service debugging requires SSH to each server; compliance audits fail	Deploy Alloy→Loki or Filebeat→ELK; forward all application and system logs; retain per compliance requirements
Fresh installs still shipping logs with Promtail	Promtail is EOL since 2026-03-02 — no updates, no security fixes; new deployments start life unsupported	Use Grafana Alloy for all new installs; migrate existing Promtail configs with alloy convert --source-format=promtail

---

Related Skills

Workload	Skill
Core system admin (packages, users, firewall, disk)	ubuntu-server-admin
Web servers (Nginx, Apache, Caddy)	ubuntu-web-servers
Databases (PostgreSQL, MySQL, Redis)	ubuntu-databases
Docker / containers	ubuntu-docker-host
File sharing (NFS, Samba, ZFS)	ubuntu-file-storage
DNS, DHCP, NTP	ubuntu-network-infra
NVIDIA GPU, Ollama, CUDA	ubuntu-ollama-nvidia