executing-plans
executing-plans
Use when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available
shell-process-managementManages Linux background processes, parallel execution, and job control using &, jobs, fg, bg, wait, xargs -P, and GNU parallel for shell scripting.
Independent GitHub snapshot · Automated classification · Original documentation governs
name: shell-process-management description: Manages Linux background processes, parallel execution, and job control using &, jobs, fg, bg, wait, xargs -P, and GNU parallel for shell scripting. license: MIT compatibility: opencode metadata: version: "1.0.0" domain: linux role: implementation scope: implementation output-format: code content-types:
Infrastructure engineer managing Linux background processes, parallel execution, and interactive job control using shell builtins (&, jobs, fg, bg, wait), signal handling with trap, batch parallelism with xargs -P, and GNU parallel for efficient multi-core work distribution.
set -euo pipefail in every shell script to catch failures early$! and store them for later managementSIGTERM, SIGHUP, and SIGINT before spawning background workers that create temp files or lockswait $PID (not bare wait) to detect individual background process exit codesxargs -P for simple parallel file operations; prefer GNU parallel for complex multi-stage pipelinesmktemp + mv) for daemons that must survive session logoutUse this skill when:
& in an interactive shell and need to bring it to the foreground, suspend it again, or check its statusAvoid this skill for:
systemd (linux-services) for services that must survive reboots, handle dependency ordering, and integrate with system monitoring. Shell backgrounding has no restart policy or dependency managementtmux or screen instead to detach from an existing sessionDetermine whether your workload fits single-process parallelism (xargs -P, simple &/wait) or requires task-level job control (fg, bg, jobs). File-processing tasks map to batch tools; interactive process management maps to job control.
Checkpoint: Identify the maximum concurrency level (number of CPU cores, I/O bottleneck, or memory constraint) and whether tasks are independent or have implicit ordering.
Select the appropriate tool based on workload characteristics:
& + $! + waitfg, bg, jobsxargs -P or GNU parallel --joblogtrap + PID file + signal routingparallel --jobs +0Checkpoint: Verify the chosen tool supports tracking exit codes. A background job that silently fails is worse than a sequential one.
Every backgrounded process or spawned worker must have a shutdown path. If a script spawns children, it must trap signals and forward them to those children so all processes terminate cleanly together.
Checkpoint: Confirm every trap handler runs cleanup before calling exit. Test by sending SIGTERM (kill) to the parent and verifying no orphaned child processes remain.
Never start parallel tasks without a collection strategy. Use named pipes, temp files, or associative arrays to gather exit codes and output from each worker. A single failure among 50 parallel jobs must not go unnoticed.
Checkpoint: Validate that your result-collection mechanism handles the case where workers finish in non-deterministic order and writes can interleave.
After all work completes, confirm no stale PID files, lock files, or temp directories remain. Check that all exit codes were collected and reported.
Checkpoint: Run ps --ppid <pid> to verify no orphan processes survive. Remove every temporary artifact the script created.
Run a command in the background, track its PID, monitor with jobs, and wait for completion while capturing the exit code.
#!/usr/bin/env bash
set -euo pipefail
# ── Launch a long-running backup in the background ──────────────────
echo "Starting full backup..."
tar -czf /tmp/backup_$(date +%Y%m%d_%H%M%S).tar.gz \
-C /data . & # The & forks the process into the background
BACKUP_PID=$! # $! captures the PID of the last backgrounded job
echo "Backup started with PID ${BACKUP_PID}"
# ── Monitor the background job using the jobs builtin ───────────────
sleep 2 # Give it a moment to start
jobs -l # Lists all jobs with their PIDs
# Output example: [1]+ 12345 Running tar -czf /tmp/backup_20260519_143000.tar.gz
# ── Wait for the specific background process and capture its exit code ─
if wait $BACKUP_PID; then
echo "Backup completed successfully (PID ${BACKUP_PID})."
else
EXIT_CODE=$?
echo "ERROR: Backup failed with exit code ${EXIT_CODE} (PID ${BACKUP_PID})." >&2
exit "$EXIT_CODE"
fi
# ── Cleanup verification ───────────────────────────────────────────
if [[ ! -f "/tmp/backup_*.tar.gz" ]]; then
echo "WARNING: Backup file not found after wait returned success." >&2
fi
Explanation: The & operator forks the preceding command, returning control to the shell immediately. The shell assigns $! with the child's PID so we can explicitly wait on it (rather than a bare wait which waits for all background jobs). This pattern is essential: without storing $!, there is no way to check the exit status of an individual backgrounded process, because jobs -l only shows state (Running, Done, Stopped) but never the exit code.
BAD example — losing the PID:
#!/usr/bin/env bash
set -euo pipefail
# ❌ BAD: $! is overwritten by sleep, so we can never check tar's exit code
tar -czf /tmp/backup.tar.gz -C /data . &
sleep 10 # $! now holds sleep's PID, not tar's
wait # Waits for ALL jobs, but we lost the specific PID
# We have no idea if tar succeeded or failed — tar's exit code is silently consumed
Interactively manage running background processes: list them, switch between foreground and background, and send signals.
#!/usr/bin/env bash
set -euo pipefail
# ── Start multiple background jobs ─────────────────────────────────
echo "Starting 3 workers..."
sleep 120 & # Job 1: long sleep
PID_A=$!
jobs -l # Show initial job list with PIDs
sleep 60 & # Job 2: medium sleep
PID_B=$!
jobs -l
sleep 30 & # Job 3: short sleep
PID_C=$!
jobs -l
# ── List all jobs with full details ────────────────────────────────
# jobs -lp shows PIDs for all background jobs:
# [1] PID_A Running
# [2] PID_B Running
# [3] PID_C Running
echo ""
echo "=== All Jobs ==="
jobs -l
# ── Bring a specific job to the foreground by job number ───────────
# In an interactive shell you would run: fg %1
# In a script, we use kill/signal instead (fg/bg are interactive-only)
# To send SIGCONT to a suspended job: kill -CONT %2
# ── Suspend and resume a background job ────────────────────────────
kill -STOP $PID_A # Suspend (same as Ctrl+Z)
echo "Job 1 suspended."
jobs -l # Shows: [1] PID_A Stopped
kill -CONT $PID_A # Resume (same as 'bg' or 'fg')
echo "Job 1 resumed."
# ── Send signals to background jobs ────────────────────────────────
# Graceful shutdown: send SIGTERM to a specific job
kill -TERM $PID_C
wait $PID_C 2>/dev/null || true
echo "Job 3 terminated gracefully (PID ${PID_C})."
# ── Force-kill if SIGTERM doesn't work ─────────────────────────────
if kill -0 $PID_A 2>/dev/null; then
echo "Job 1 still alive. Sending SIGKILL."
kill -9 $PID_A
fi
wait 2>/dev/null || true # Wait for any remaining jobs
echo "All jobs cleaned up."
Explanation: jobs -l is the single most useful command for inspecting background state — it lists job numbers, PIDs, and current status (Running, Stopped, Done) in one line. In interactive shells, fg %N brings job N to the foreground and bg %N resumes a stopped job in the background. In scripts (non-interactive), you must use kill signals directly since fg/bg only work in interactive shells attached to a terminal. The kill -STOP / kill -CONT pair mirrors the Ctrl+Z / fg interaction.
Key job control flags:
| Flag | Meaning |
|---|---|
jobs -l | List jobs with PIDs and status |
jobs -p | List only PIDs (useful for scripting) |
jobs -n | List only jobs that have changed status since last notification |
fg %1 | Bring job 1 to foreground (interactive only) |
bg %1 | Resume stopped job 1 in background (interactive only) |
kill -STOP $PID | Suspend a process by PID |
kill -CONT $PID | Resume a suspended process by PID |
Process many files in parallel using xargs -P, which splits input lines into batches of concurrent worker processes.
#!/usr/bin/env bash
set -euo pipefail
# ── Configuration ──────────────────────────────────────────────────
INPUT_DIR="${1:-/var/log}"
WORKERS="${2:-$(nproc)}" # Default to all available CPU cores
RESULTS_FILE=$(mktemp /tmp/xargs-results.XXXXXX)
LOCK_FILE=$(mktemp /tmp/xargs-lock.XXXXXX)
# ── Cleanup trap ───────────────────────────────────────────────────
cleanup() {
rm -f "$RESULTS_FILE" "$LOCK_FILE"
}
trap cleanup EXIT INT TERM
echo "Processing files in '${INPUT_DIR}' with ${WORKERS} parallel workers..."
# ── Function that processes a single file ──────────────────────────
process_file() {
local filepath="$1"
local filename
filename=$(basename "$filepath")
# Simulate work: compress the log file if it's text-based
if file --mime-type -b "$filepath" | grep -q 'text/'; then
gzip -9 -c "$filepath" > "${filepath}.gz" 2>/dev/null && \
STATUS="OK" || STATUS="FAIL"
else
STATUS="SKIP (binary)"
fi
# Thread-safe write to results file using a lock
(
flock -x 200
echo "${STATUS} | ${filename}" >> "$RESULTS_FILE"
) 200>"$LOCK_FILE"
}
export -f process_file # Required for xargs -P to inherit the function
# ── Parallel execution using xargs -P ──────────────────────────────
# Find all files, pass them through xargs which spawns up to $WORKERS concurrent workers
find "$INPUT_DIR" -maxdepth 1 -type f \
| sort \
| xargs -P "$WORKERS" -I {} bash -c 'process_file "$@"' _ {}
# ── Analyze results ────────────────────────────────────────────────
TOTAL=$(wc -l < "$RESULTS_FILE")
OK_COUNT=$(grep -c '^OK' "$RESULTS_FILE" || true)
FAIL_COUNT=$(grep -c '^FAIL' "$RESULTS_FILE" || true)
echo ""
echo "=== Parallel Processing Summary ==="
echo "Total files processed : ${TOTAL}"
echo "Successfully processed: ${OK_COUNT}"
echo "Failed : ${FAIL_COUNT}"
if [[ "$FAIL_COUNT" -gt 0 ]]; then
echo ""
echo "Failed files:"
grep '^FAIL' "$RESULTS_FILE" | while IFS='|' read -r _ fname; do
echo " ❌ ${fname}"
done
exit 1
fi
echo ""
echo "All ${TOTAL} files processed successfully."
Explanation: xargs -P N is the simplest parallel execution tool available on virtually every Linux system (it's part of GNU findutils, not a separate install). The -P flag tells xargs to spawn up to N concurrent processes. Key points: the worker function must be exported (export -f) so child bash processes inherit it; file writes from concurrent workers need locking (flock) to avoid interleaved output; and xargs -I {} replaces each input line as a single argument to the command.
BAD example — ignoring parallel exit codes:
#!/usr/bin/env bash
set -euo pipefail
# ❌ BAD: xargs returns 0 if ANY invocation succeeds, not ALL
find . -name "*.log" | xargs -P 4 gzip
# If one file fails to compress, xargs still exits 0 and the script continues as if everything worked.
# The compressed output is missing, but no failure is reported.
# ✅ GOOD: Check each file individually with a wrapper that reports failures
find . -name "*.log" | sort | xargs -P "$(nproc)" -I {} bash -c '
if gzip "$1"; then
echo "OK: $1"
else
echo "FAIL: $1" >&2
exit 1
fi
' _ {}
Create a long-running background daemon that handles signals properly, writes a PID file for management, and performs clean shutdown. This is the standard pattern when you need a persistent process without systemd.
#!/usr/bin/env bash
set -euo pipefail
# ── Configuration ──────────────────────────────────────────────────
PID_FILE="/tmp/my-daemon.pid"
LOG_FILE="/var/log/my-daemon.log"
WORK_INTERVAL="${DAEMON_INTERVAL:-5}" # Seconds between work cycles
# ── PID file management (atomic write to avoid race conditions) ───
write_pid_file() {
local tmpfile
tmpfile=$(mktemp "${PID_FILE}.XXXXXX")
echo $ > "$tmpfile"
mv -f "$tmpfile" "$PID_FILE" # Atomic rename guarantees PID consistency
}
cleanup_pid_file() {
rm -f "$PID_FILE"
}
# ── Cleanup handler for temp resources ─────────────────────────────
TEMP_FILES=()
trap_temp_cleanup() {
local tmpfile
for tmpfile in "${TEMP_FILES[@]}"; do
[[ -f "$tmpfile" ]] && rm -f "$tmpfile"
done
}
# ── Graceful shutdown: forward signals to child workers ─────────────
CHILD_PIDS=()
stop_children() {
local pid
for pid in "${CHILD_PIDS[@]}"; do
if kill -0 "$pid" 2>/dev/null; then
echo "[$(date -Iseconds)] SIGTERM -> PID ${pid}" >> "$LOG_FILE"
kill -TERM "$pid" 2>/dev/null || true
fi
done
# Wait up to 10 seconds for children to exit gracefully
local i=0
while (( i < 20 )); do
local still_alive=false
for pid in "${CHILD_PIDS[@]}"; do
if kill -0 "$pid" 2>/dev/null; then
still_alive=true
break
fi
done
$still_alive && sleep 0.5 || break
(( i++ ))
done
# Force-kill any stragglers
for pid in "${CHILD_PIDS[@]}"; do
if kill -0 "$pid" 2>/dev/null; then
echo "[$(date -Iseconds)] SIGKILL -> PID ${pid}" >> "$LOG_FILE"
kill -9 "$pid" 2>/dev/null || true
fi
done
cleanup_pid_file
}
# ── Signal traps: catch termination requests from OS, cron, or admin ─
trap 'echo "[$(date -Iseconds)] Received SIGINT, shutting down..." >&2; stop_children; exit 130' INT
trap 'echo "[$(date -Iseconds)] Received SIGTERM, shutting down..." >&2; stop_children; exit 143' TERM
trap 'echo "[$(date -Iseconds)] Received SIGHUP, closing connections..." >&2; stop_children; exit 129' HUP
# ── Main daemon loop ───────────────────────────────────────────────
main() {
# Write PID file
write_pid_file
echo "[$(date -Iseconds)] Daemon started (PID $) with ${WORK_INTERVAL}s interval" >> "$LOG_FILE"
while true; do
# Spawn a worker process for each cycle
{
# Worker task — replace with actual work
local tmpfile
tmpfile=$(mktemp /tmp/worker-output.XXXXXX)
TEMP_FILES+=("$tmpfile")
echo "[$(date -Iseconds)] Worker $(date +%s%N) started" >> "$LOG_FILE"
# Example: process a batch of data
date -u +"%Y-%m-%dT%H:%M:%SZ" > "$tmpfile"
local rc=0
echo "[$(date -Iseconds)] Worker completed (exit ${rc})" >> "$LOG_FILE"
exit $rc
} &
CHILD_PIDS+=($!)
# Log the running child PIDs
echo "[$(date -Iseconds)] Child PID: ${CHILD_PIDS[*]}" >> "$LOG_FILE"
# Sleep with signal interruption — this is the key trick that
# allows immediate shutdown instead of waiting for the full interval.
local waited=0
while (( waited < WORK_INTERVAL )); do
sleep 1 || break # sleep returns non-zero if interrupted by a signal
(( waited++ ))
done
done
}
# ── Entry point ────────────────────────────────────────────────────
main "$@"
Explanation: This pattern is the standard shell-based daemon. Key elements: (1) trap registers handlers for SIGINT, SIGTERM, and SIGHUP — these are the three signals the OS sends during shutdown, user logout, or admin intervention; (2) the PID file uses atomic mktemp + mv to prevent partial writes if two instances start simultaneously; (3) child PIDs are tracked in an array so stop_children() can forward SIGTERM to every worker; (4) the sleep loop checks the return value of sleep 1 — when a signal interrupts sleep, it returns non-zero, which breaks the inner loop and allows immediate shutdown without waiting for the full WORK_INTERVAL.
PID file usage by other scripts:
#!/usr/bin/env bash
set -euo pipefail
PID_FILE="/tmp/my-daemon.pid"
# Check if daemon is running
if [[ -f "$PID_FILE" ]]; then
PID=$(cat "$PID_FILE")
if kill -0 "$PID" 2>/dev/null; then
echo "Daemon is running (PID ${PID}). Sending SIGTERM..."
kill -TERM "$PID"
# Wait up to 10 seconds for graceful shutdown
local i=0
while (( i < 20 )) && kill -0 "$PID" 2>/dev/null; do
sleep 0.5
(( i++ ))
done
# Force-kill if still alive
if kill -0 "$PID" 2>/dev/null; then
echo "Daemon did not exit gracefully. Sending SIGKILL." >&2
kill -9 "$PID"
fi
echo "Daemon stopped."
else
echo "Stale PID file found (PID ${PID} not running). Cleaning up." >&2
rm -f "$PID_FILE"
fi
else
echo "No PID file found. Daemon is not running."
fi
Install and use GNU parallel for advanced parallel task distribution with job tracking, tagged output, and result collection.
#!/usr/bin/env bash
set -euo pipefail
# ── Check if GNU parallel is available ─────────────────────────────
if ! command -v parallel &>/dev/null; then
echo "ERROR: GNU parallel not found. Install with:" >&2
echo " Ubuntu/Debian: sudo apt install parallel" >&2
echo " RHEL/CentOS: sudo yum install parallel" >&2
echo " macOS: brew install parallel" >&2
exit 1
fi
# ── Configuration ──────────────────────────────────────────────────
INPUT_DIR="${1:-.}"
OUTPUT_PREFIX="parallel-results"
MAX_JOBS="${2:-$(nproc)}" # Default to all cores; use +0 for unlimited
echo "GNU Parallel Execution Engine"
echo " Input directory : ${INPUT_DIR}"
echo " Max parallel jobs: ${MAX_JOBS}"
echo ""
# ── Simple parallel file processing ────────────────────────────────
# Process each file in the input directory in parallel, tagged with filename
echo "--- Phase 1: File Analysis ---"
find "$INPUT_DIR" -maxdepth 1 -type f \
| sort \
| parallel --jobs "$MAX_JOBS" \
--tag \
'echo "Size: $(stat -c%s {} 2>/dev/null || echo N/A) bytes | Type: $(file --mime-type -b {}) | {}"' \
| tee "${OUTPUT_PREFIX}-phase1.txt"
echo ""
echo "--- Phase 2: Parallel Transformation ---"
# ── Worker function that parallel calls into ───────────────────────
transform_file() {
local input="$1"
local basename
basename=$(basename "$input")
local output="${INPUT_DIR}/transformed_${basename}"
# Simulate transformation: convert to uppercase (works on text files only)
if file --mime-type -b "$input" | grep -q 'text/'; then
tr '[:lower:]' '[:upper:]' < "$input" > "$output" 2>/dev/null && \
echo "OK" || echo "FAIL"
else
echo "SKIP (binary)"
fi
}
export -f transform_file
# Process files with tagged output and collect results in a single file
find "$INPUT_DIR" -maxdepth 1 -type f \
| sort \
| parallel --jobs "$MAX_JOBS" \
--tag \
--joblog "${OUTPUT_PREFIX}-joblog.csv" \
'transform_file "{}"' \
| tee "${OUTPUT_PREFIX}-phase2.txt"
# ── Display results from the job log (machine-readable summary) ───
echo ""
echo "--- Job Log Summary ---"
if [[ -f "${OUTPUT_PREFIX}-joblog.csv" ]]; then
echo "Total jobs logged: $(tail -n +2 "${OUTPUT_PREFIX}-joblog.csv" | wc -l)"
echo ""
echo "Job log columns:"
echo " StartTime, EndTime, JobId, Duration (seconds), Exit Value, Hostname, ShellID, CommandLine"
fi
# ── Parallel with a timeout per job ────────────────────────────────
echo ""
echo "--- Phase 3: Time-limited Parallel Execution ---"
timeout_job() {
local input="$1"
if parallel --timeout 5 \
'echo "Processing {}..."; sleep 2; echo "Done: $@"' _ $(basename "$input"); then
echo "OK: $(basename "$input")"
else
echo "TIMEOUT: $(basename "$input")"
fi
}
export -f timeout_job
find "$INPUT_DIR" -maxdepth 1 -type f \
| head -20 # Limit to 20 files for the demo
| parallel --jobs "$MAX_JOBS" \
--tag \
--results "${OUTPUT_PREFIX}-phase3-results.txt" \
'timeout_job "{}"'
echo ""
echo "=== Complete ==="
echo "Results written to: ${OUTPUT_PREFIX}-*"
Explanation: GNU parallel is a more powerful alternative to xargs -P with features that matter for serious parallel work: (1) --tag prepends the job's input line to every output line, making it trivially easy to associate results with their source; (2) --joblog writes a CSV log of all jobs including duration and exit codes, enabling post-hoc analysis of which tasks were slow or failed; (3) --results collects per-job output into separate files named by job number; (4) --timeout N kills any single job that exceeds the time limit without affecting other jobs; (5) --jobs N limits concurrency, with special values like +0 (one job per CPU core beyond the base level).
Common parallel flags cheat sheet:
| Flag | Purpose | Example |
|---|---|---|
--jobs N | Max concurrent jobs | --jobs 4 or --jobs +0 (all cores) |
--tag | Prefix each output line with the input | parallel --tag echo {} |
--joblog file | Write CSV job log with timings/exit codes | --joblog results.csv |
--results dir | Collect per-job output into separate files | --results output/ |
--timeout N | Kill jobs exceeding N seconds | --timeout 30 |
--link | Pass multiple inputs in lockstep to one command | parallel --link echo {1} {2} |
-k | Keep job output in input order (slower, memory-intensive) | parallel -k echo {} |
set -euo pipefail at the top of shell scripts to catch unset variables, pipeline errors, and command failures immediately$! immediately after every & backgrounding — without the PID you cannot check exit codes or send targeted signalswait $PID (with explicit PID) rather than bare wait when you need to distinguish between multiple independent background processesexit N where N encodes the signal number (130 for SIGINT, 143 for SIGTERM, 129 for SIGHUP) so consumers can diagnose how the process terminatedflock -x — parallel workers writing to the same file without locking produce corrupted interleaved outputmktemp + mv) — two instances racing to write a PID file must never leave a partial or overwritten PIDwait in scripts that spawn multiple independent background processes — A bare wait waits for ALL background jobs and returns the exit code of the LAST one to finish. If job 3 (exit code 1) finishes after jobs 1 and 2 (both exit code 0), you get exit code 1 with no idea which of the first two jobs also failed. Always use wait $PIDexit — A trap handler that just runs cleanup and then falls through to the main loop means the signal was effectively ignored, leaving orphaned child processes running indefinitely. Every signal trap must call exitexport -f when using xargs -P or parallel with custom functions — Both tools spawn new shell processes for workers. Without export -f, the worker processes don't have your function definitions and will fail with "command not found"< /dev/null) or tmux if interactive input is neededWhen applying this skill, produce:
&/wait, job control (fg/bg), xargs -P, or GNU parallel, and justify the choice based on workload characteristicsset -euo pipefail, proper signal handling, PID management, and error reporting. Every parallel execution must include exit-code collection| Skill | Purpose |
|---|---|
linux-services | Use systemd for long-running daemons instead of shell backgrounding when you need restart policies, dependency ordering, and boot integration |
linux-filesystem | Path management and directory traversal strategies for file-processing parallel jobs that scan large filesystem trees |
📖 skill(local cache): linux-services, linux-filesystem
Authoritative documentation links for this skill's domain. The model follows markdown links at load time to resolve external references and inline content.