executing-plans
executing-plans
Use when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available
corosyncConfigures Corosync Cluster Engine v3.x messaging layer including totem protocol, quorum models, nodelist management, knet/udpu transports, and security for Pacemaker HA clusters.
Independent GitHub snapshot · Automated classification · Original documentation governs
Senior Linux infrastructure engineer configuring Corosync v3.x as the cluster messaging layer for Pacemaker HA clusters. Covers totem protocol setup, transport selection (knet/udpu/mcast), quorum models with qdevice, nodelist management, security hardening, logging, and cluster diagnostics.
corosync-cfgtool -s and quorum with corosync-quorumtool -s after every changeUse this skill when:
two_node mode to prevent split-brain scenariosAvoid this skill for:
pacemaker instead; Corosync handles messaging only, not resources or constraintsUse pacemaker for configuring resources, constraints, and fencing on top of the Corosync messaging layer. Use networking for switch/firewall rules that may affect multicast transport.
The transport determines how cluster members communicate. Choose based on your network capabilities.
# knet — Modern default (recommended for all new clusters)
# Supports multiple redundant links, IPv6, TCP fallback, and UDP multicast replacement
# udpu — Unicast UDP (use when multicast is blocked by firewalls or switches)
# Every node explicitly lists every other node's ring address
# mcast — Legacy traditional multicast (only available with kmod-based corosync)
# Deprecated; use knet unless you have a specific legacy requirement
Checkpoint: Confirm multicast works on your network before choosing mcast/knet:
# Test multicast connectivity from one node to another
ping -M do -s 1472 <mcastaddr> -I <bindnetaddr> # MTU test with DF bit set
# Or use mtrace for multicast route verification
mtrace <source_ip> <mcastaddr>
Write the complete corosync.conf with totem, transport, nodelist, quorum, and logging sections.
totem {
version: 2
cluster_name: MyCluster
transport: knet # or 'udpu' or omit for default (knet)
crypto_cipher: aes256 # Enable for encrypted transport on untrusted networks
crypto_hash: sha256 # HMAC for message integrity authentication
interface {
ringnumber: 0
mcastaddr: 226.94.1.1
mcastport: 5405
ttl: 1
}
}
quorum {
provider: corosync_votequorum
expected_votes: 1 # Usually left at 1 — corosync auto-calculates from nodelist
two_node: 1 # For two-node clusters; reduces split-brain risk
}
nodelist {
node {
name: cluster-node-1
nodeid: 1
ring0_addr: 192.168.1.11
}
node {
name: cluster-node-2
nodeid: 2
ring0_addr: 192.168.1.12
}
}
logging {
to_stderr: yes
to_logfile: yes
logfile: /var/log/cluster/corosync.log
logfile_max_lines: 100000 # Rotate after N lines
logfile_ensure_dir: yes # Create log directory automatically
to_syslog: yes
debug: off # Enable for troubleshooting only — extremely verbose!
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
Checkpoint: Validate syntax with corosync-cmaptool or by starting corosync in dry-run mode. Verify the nodelist has unique nodeids with no gaps, and the cluster_name is consistent across all nodes.
If crypto_cipher is set, every node must have the same authkey.
# Generate a new key on the first node (or wherever you manage configs)
sudo corosync-keygen
# This produces /etc/corosync/authkey with binary content
# Set restrictive permissions immediately
sudo chmod 600 /etc/corosync/authkey
sudo chown root:root /etc/corosync/authkey
# Distribute the key to all other cluster nodes using a secure method
for node in cluster-node-2 cluster-node-3; do
scp /etc/corosync/authkey "root@${node}:/etc/corosync/authkey"
ssh root@"${node}" chmod 600 /etc/corosync/authkey
done
Checkpoint: Verify the authkey is identical on all nodes:
# On each node, compare the md5sum — they must match exactly
md5sum /etc/corosync/authkey
Apply the configuration to every node and start the cluster.
# Copy config to all nodes (must be identical on every node)
for node in cluster-node-1 cluster-node-2; do
scp /etc/corosync/corosync.conf "root@${node}:/etc/corosync/corosync.conf"
scp /etc/corosync/authkey "root@${node}:/etc/corosync/authkey"
done
# Start corosync on every node
for node in cluster-node-1 cluster-node-2; do
ssh root@"${node}" systemctl enable --now corosync
done
# Verify membership has formed
corosync-cfgtool -s # Show current membership status
corosync-quorumtool -s # Show quorum state and vote counts
Checkpoint: corosync-cfgtool -s shows all nodes as members with the expected node count. corosync-quorumtool -s shows quorum is active (green). If membership is incomplete, check /var/log/cluster/corosync.log on both nodes for transport-level errors.
Two-node clusters need special handling to prevent split-brain. Multi-node clusters rely on majority quorum by default.
# OPTION A — Two-node cluster: enable two_node mode in corosync.conf
# Add 'two_node: 1' to the quorum section — this changes expected_votes behavior
# so the cluster can remain operational even with one node down
# OPTION B — Two-node cluster: deploy qdevice (recommended for production)
# qdevice acts as a third-party arbiter that breaks ties during network partitions
sudo pcs qdevice setup alibi --init --nodes cluster-node-1,cluster-node-2
# OPTION C — Multi-node cluster (3+ nodes): default majority quorum is sufficient
# No extra configuration needed; corosync votes automatically from nodelist count
# After qdevice setup, set the no-quorum-policy to allow continued operation
sudo pcs property set no-quorum-policy=ignore
Checkpoint: corosync-quorumtool -s shows the correct expected_votes and current node_count. For qdevice clusters, verify the qdevice node appears in the quorum output with its voting status.
BAD — Minimal config with hardcoded values, no transport choice, no logging rotation, missing security
# ❌ BAD: No transport specified, no encryption, no log rotation,
# incomplete nodelist, no quorum strategy
totem {
version: 2
cluster_name: MyCluster
interface {
ringnumber: 0
bindnetaddr: 192.168.1.0
mcastaddr: 226.94.1.1
mcastport: 5405
}
}
nodelist {
node {
name: cluster-node-1
nodeid: 1
ring0_addr: 192.168.1.11
}
}
# Problems:
# - No transport specified (defaults to knet but should be explicit)
# - No crypto settings — unencrypted cluster messages on production network
# - Only one node in nodelist — no quorum possible
# - No logging section at all — no operational visibility
# - No quorum section — uses defaults that may not be appropriate
# - Missing logfile_ensure_dir, logfile_max_lines for log management
GOOD — Complete, production-ready with explicit transport, encryption, logging, and quorum
#!/usr/bin/env bash
# Production Corosync configuration generator
# Generates a complete corosync.conf with best practices applied.
# Usage: ./gen-corosync-conf.sh <cluster_name> <bindnetaddr> <mcastaddr> <mcastport> node1_addr node2_addr [node3_addr...] [--encrypt]
set -euo pipefail
readonly CLUSTER_NAME="${1:?Usage: $0 <cluster_name> <bindnet> <mcastaddr> <mcastport> node_addrs... [--encrypt]}"
readonly BINDNETADDR="$2"
readonly MCASTADDR="$3"
readonly MCASTPORT="${4:-5405}"
readonly ENCRYPT_FLAG="${6:-no}"
# Collect nodes from remaining positional arguments (skip --encrypt flag)
NODE_ADDRS=()
for arg in "$@"; do
case "$arg" in
--encrypt|"$1"|"$2"|"$3"|"$4") continue ;;
*) NODE_ADDRS+=("$arg") ;;
esac
done
if [[ ${#NODE_ADDRS[@]} -lt 2 ]]; then
echo "ERROR: At least 2 node addresses required" >&2
exit 1
fi
readonly NUM_NODES=${#NODE_ADDRS[@]}
# Determine transport based on environment
TRANSPORT="knet"
if grep -qE 'multicast.*(no|disabled|false)' /etc/sysconfig/iptables 2>/dev/null || \
! ip maddr show >/dev/null 2>&1; then
TRANSPORT="udpu"
echo "WARNING: Multicast appears unavailable, using udpu transport" >&2
fi
# Generate config
CONF_FILE="/etc/corosync/corosync.conf"
mkdir -p "$(dirname "$CONF_FILE")"
cat > "${CONF_FILE}" <<EOF
totem {
version: 2
cluster_name: "${CLUSTER_NAME}"
transport: ${TRANSPORT}
${[[ "${ENCRYPT_FLAG}" == "--encrypt" ]] && echo ' crypto_cipher: aes256' || echo ' crypto_cipher: none'}
${[[ "${ENCRYPT_FLAG}" == "--encrypt" ]] && echo ' crypto_hash: sha256' || echo ' crypto_hash: none'}
interface {
ringnumber: 0
EOF
if [[ "${TRANSPORT}" != "udpu" ]]; then
cat >> "${CONF_FILE}" <<EOF
mcastaddr: ${MCASTADDR}
mcastport: ${MCASTPORT}
ttl: 1
EOF
else
echo ' # udpu mode: no multicast — node addresses are in nodelist below' >> "${CONF_FILE}"
fi
cat >> "${CONF_FILE}" <<EOF
}
}
quorum {
provider: corosync_votequorum
expected_votes: 1
two_node: $([[ ${NUM_NODES} -eq 2 ]] && echo '1' || echo '0')
}
nodelist {
EOF
NODEID=1
for addr in "${NODE_ADDRS[@]}"; do
cat >> "${CONF_FILE}" <<EOF
node {
name: node-${NODEID}
nodeid: ${NODEID}
ring0_addr: ${addr}
}
EOF
NODEID=$((NODEID + 1))
done
cat >> "${CONF_FILE}" <<EOF
}
logging {
to_stderr: yes
to_logfile: yes
logfile: /var/log/cluster/corosync.log
logfile_max_lines: 100000
logfile_ensure_dir: yes
to_syslog: yes
debug: off
timestamp: on
}
EOF
chmod 644 "${CONF_FILE}"
echo "Generated ${CONF_FILE} (${NUM_NODES} nodes, transport: ${TRANSPORT})"
echo "Apply to all cluster nodes and generate authkey with: corosync-keygen"
BAD — Single-link knet config that does not leverage redundancy
# ❌ BAD: No redundant links configured — a single NIC failure breaks the cluster
totem {
version: 2
cluster_name: MyCluster
transport: knet
interface {
ringnumber: 0
mcastaddr: 226.94.1.1
mcastport: 5405
ttl: 1
}
}
# Problems:
# - Single link means single point of failure for cluster communication
# - No knet.link configuration for redundant paths over different NICs
# - Does not utilize multi-homed network infrastructure available on most servers
GOOD — Dual-link knet with independent paths over separate NICs
# ✅ GOOD: Dual redundant links using knet, each on a different physical network
totem {
version: 2
cluster_name: MyCluster
transport: knet
interface {
ringnumber: 0
mcastaddr: 226.94.1.1
mcastport: 5405
ttl: 1
}
link {
number: 0
interface {
bindnetaddr: 192.168.1.0 # Primary management network
mcastaddr: 226.94.1.1
mcastport: 5405
ttl: 1
}
}
link {
number: 1
interface {
bindnetaddr: 192.168.2.0 # Secondary replication network
mcastaddr: 226.94.2.1
mcastport: 5406
ttl: 1
}
}
}
nodelist {
node {
name: cluster-node-1
nodeid: 1
ring0_addr: 192.168.1.11 # Primary address (management)
}
node {
name: cluster-node-2
nodeid: 2
ring0_addr: 192.168.1.12
}
}
BAD — Trying to use multicast transport when switches block multicast
# ❌ BAD: Multicast transport in an environment where switches drop multicast packets
# or firewalls block IGMP/UDP multicast ports — cluster will never form
totem {
version: 2
cluster_name: MyCluster
# transport omitted, defaults to knet with mcast
interface {
ringnumber: 0
bindnetaddr: 192.168.1.0
mcastaddr: 226.94.1.1 # This will fail if multicast is blocked
mcastport: 5405
}
}
# Problems:
# - Multicast packets silently dropped → nodes never see each other's membership messages
# - Symptoms: corosync.log shows "totem: Unable to initialize membership" on all nodes
# - No explicit transport declared, so the default knet/mcast fails in silence
GOOD — Explicit udpu transport with every node listed by unicast address
# ✅ GOOD: UDPU (Unicast UDP) explicitly configured for multicast-blocked environments
totem {
version: 2
cluster_name: MyCluster
transport: udpu # Explicitly declare unicast UDP transport
interface {
ringnumber: 0
bindnetaddr: 192.168.1.0 # Network range for binding
mcastport: 5405 # Port used for all inter-node communication
}
}
quorum {
provider: corosync_votequorum
expected_votes: 1
two_node: 1 # Two-node udpu cluster needs this flag
}
nodelist {
node {
name: cluster-node-1
nodeid: 1
ring0_addr: 192.168.1.11 # Every node MUST list every other node
} # by unicast address — no multicast discovery
node {
name: cluster-node-2
nodeid: 2
ring0_addr: 192.168.1.12
}
}
Bash — Automated diagnostics for membership, quorum, and transport health
#!/usr/bin/env bash
# Corosync cluster diagnostics script
# Collects membership status, quorum state, runtime config, and log analysis.
set -euo pipefail
readonly REPORT_DIR="/var/tmp/corosync-diagnostics"
readonly TIMESTAMP="$(date +%Y%m%d-%H%M%S)"
mkdir -p "${REPORT_DIR}/${TIMESTAMP}"
echo "=== Corosync Cluster Diagnostics ==="
echo "Timestamp: ${TIMESTAMP}"
echo "Hostname: $(hostname)"
echo ""
# ─── Collect diagnostics data ──────────────────────────────
collect() {
local label="$1"; shift
echo "[+] Collecting: ${label}"
"$@" > "${REPORT_DIR}/${TIMESTAMP}/${label}.log" 2>&1 || true
}
collect "cfgtool_status" corosync-cfgtool -s
collect "cmap_full" corosync-cmaptool dump
collect "quorum_status" corosync-quorumtool -s
collect "cpgtool" corosync-cpgtool || true
collect "corosync_config" cat /etc/corosync/corosync.conf
collect "authkey_present" [[ -f /etc/corosync/authkey ]] && echo "authkey exists" || echo "authkey MISSING"
# Runtime cmap queries for specific fields
echo "--- Runtime CMap Queries ---" > "${REPORT_DIR}/${TIMESTAMP}/cmap_key_fields.log"
for key in \
"totem.interface.0.mcastaddr" \
"totem.interface.0.mcastport" \
"totem.protocol_id" \
"runtime.quorum.current_expected_votes" \
"runtime.nodelist.node_count" \
"runtime.membership.membership_count" \
"runtime.config.config_version"; do
result=$(corosync-cmaptool get "${key}" 2>/dev/null || echo "KEY_NOT_FOUND")
echo "${key} = ${result}" >> "${REPORT_DIR}/${TIMESTAMP}/cmap_key_fields.log"
done
# ─── Collect logs ──────────────────────────────────────────
collect "corosync_log_recent" tail -n 500 /var/log/cluster/corosync.log 2>/dev/null || true
collect "corosync_log_errors" grep -iE 'error|fail|warn|critical' /var/log/cluster/corosync.log 2>/dev/null | tail -n 100 || echo "No errors found in recent log" > "${REPORT_DIR}/${TIMESTAMP}/corosync_log_errors.log"
# ─── Analysis ──────────────────────────────────────────────
echo ""
echo "=== Analysis ==="
# Check membership
MEM_COUNT=$(grep -c 'Number of ring members' "${REPORT_DIR}/${TIMESTAMP}/cfgtool_status.log" 2>/dev/null || echo "0")
if [[ "${MEM_COUNT}" -gt 0 ]]; then
ACTUAL_MEMBERS=$(grep 'Number of ring members' "${REPORT_DIR}/${TIMESTAMP}/cfgtool_status.log" | awk '{print $NF}')
CONFIG_NODES=$(grep -c 'nodeid:' /etc/corosync/corosync.conf)
echo "Membership: ${ACTUAL_MEMBERS} nodes active out of ${CONFIG_NODES} configured"
if [[ "${ACTUAL_MEMBERS}" -lt "${CONFIG_NODES}" ]]; then
echo " WARNING: Membership count (${ACTUAL_MEMBERS}) is less than configured nodes (${CONFIG_NODES})"
echo " Possible causes: transport failure, firewall blocking multicast, authkey mismatch"
fi
else
echo " ERROR: Unable to read membership status"
fi
# Check quorum
if command -v corosync-quorumtool &>/dev/null; then
QUORUM_STATUS=$(corosync-quorumtool -s 2>/dev/null | grep 'Quorum' || echo "Unknown")
echo "Quorum: ${QUORUM_STATUS}"
fi
# Check for common log errors
ERRORS=0
if [[ -f "${REPORT_DIR}/${TIMESTAMP}/corosync_log_errors.log" ]]; then
ERRORS=$(wc -l < "${REPORT_DIR}/${TIMESTAMP}/corosync_log_errors.log")
fi
echo "Recent error/warning count in logs: ${ERRORS}"
# ─── Common troubleshooting checklist ──────────────────────
echo ""
echo "=== Troubleshooting Checklist ==="
if [[ -f /etc/corosync/authkey ]]; then
AUTHKEY_PERMS=$(stat -c '%a' /etc/corosync/authkey 2>/dev/null || echo "unknown")
if [[ "${AUTHKEY_PERMS}" != "600" ]]; then
echo "[!] authkey permissions are ${AUTHKEY_PERMS}, should be 600"
else
echo "[✓] authkey has correct permissions (600)"
fi
else
echo "[✗] authkey file is missing — cluster cannot authenticate if encryption is enabled"
fi
# Check for config_version issues
if grep -q 'config_version' /etc/corosync/corosync.conf 2>/dev/null; then
echo "[✓] config_version found in totem section"
else
echo "[!] No config_version in totem — corosync may refuse to apply runtime changes"
fi
# Compress and output report path
tar czf "${REPORT_DIR}/corosync-diagnostics-${TIMESTAMP}.tar.gz" \
-C "${REPORT_DIR}" "${TIMESTAMP}" 2>/dev/null || true
rm -rf "${REPORT_DIR}/${TIMESTAMP}"
echo ""
echo "Report saved to: ${REPORT_DIR}/corosync-diagnostics-${TIMESTAMP}.tar.gz"
transport: knet | udpu) rather than relying on the default — explicit configuration prevents silent failures when network capabilities differ between environmentsnodeid to every node in the nodelist with no gaps (1, 2, 3...) and no duplicates — duplicate nodeids cause membership chaos/etc/corosync/authkey) is identical on all cluster nodes when crypto_cipher is enabled — mismatched keys cause authentication failures during membership formationchmod 600 /etc/corosync/authkey immediately after generation — the key contains cryptographic material and must not be readable by other userstwo_node: 1 or qdevice; multi-node clusters use default majority quorumlogfile_ensure_dir: yes and logfile_max_lines for log rotation so cluster state is always observable after a crashcorosync-cfgtool -s and quorum with corosync-quorumtool -s before introducing Pacemaker resources on the clusterpcs to manage corosync configuration changes (handles config_version auto-increment) — never edit corosync.conf manually at runtime without incrementing config_versiontimestamp: on in the logging section so log messages are time-synchronized and debuggable across nodesconfig_version in the totem section — corosync will refuse to apply the change and log an errorexpected_votes without reviewing whether two_node: 1 or qdevice is needed — incorrect quorum configuration leads to split-brain in two-node setupsdebug: true in the logging section on production systems — it generates extremely verbose output that fills disk and degrades performancecorosync-cfgtool -l after adding new interfaces or changing bindnetaddr values| Skill | Purpose |
|---|---|
pacemaker | Manage Pacemaker resources, constraints, and fencing that run on top of the Corosync messaging layer |
linux-services | Configure systemd units for Corosync service management, overrides, and integration with boot ordering |
networking | Diagnose multicast connectivity, firewall rules, and network interfaces affecting cluster communication |
Authoritative documentation links for Corosync Cluster Engine. The model follows markdown links at load time to resolve external references and inline content.