协作头脑风暴
brainstorming
在实现前澄清需求、探索设计选择,并形成清晰的开发方向。
plan-capacity【开发与工程】Perform capacity planning using historical metrics and growth models. Use predict_linear for forecasting, identify resource constraints, calculate headroom, and recommend scaling actions before saturation. Use before seasonal traffic spikes or product launches, during quarterly capacity reviews, when resource utilization trends upward, or before budg…
独立 GitHub 快照 · 自动分类 · 以原作者文档为准
依資料驅動之容量規劃,預測資源需求並防止飽和。
向 Prometheus 查詢關鍵資源指標:
# CPU usage trend over 8 weeks
avg(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (instance)
# Memory usage trend
avg(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) by (instance)
# Disk usage growth
avg(node_filesystem_size_bytes - node_filesystem_free_bytes) by (instance, device)
# Request rate growth
sum(rate(http_requests_total[5m])) by (service)
# Database connection pool usage
avg(db_connection_pool_used / db_connection_pool_max) by (instance)
匯出以分析:
# Export 8 weeks of CPU data
curl -G 'http://prometheus:9090/api/v1/query_range' \
--data-urlencode 'query=avg(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (instance)' \
--data-urlencode 'start=2024-12-15T00:00:00Z' \
--data-urlencode 'end=2025-02-09T00:00:00Z' \
--data-urlencode 'step=1h' | jq '.data.result' > cpu_8weeks.json
預期: 各資源之乾淨時間序列資料,無大幅缺漏。
失敗時: 資料缺漏將降低預測準確度。檢查指標保留期與抓取間隔。
以 Prometheus 之 predict_linear() 預測飽和:
# Predict when CPU will hit 80% (4 weeks ahead)
predict_linear(
avg(rate(node_cpu_seconds_total{mode!="idle"}[5m]))[8w:],
4*7*24*3600 # 4 weeks in seconds
) > 0.80
# Predict disk full date (8 weeks ahead)
predict_linear(
avg(node_filesystem_size_bytes - node_filesystem_free_bytes)[8w:],
8*7*24*3600
) > 0.95 * avg(node_filesystem_size_bytes)
# Predict memory pressure (2 weeks ahead)
predict_linear(
avg(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes)[8w:],
2*7*24*3600
) / avg(node_memory_MemTotal_bytes) > 0.90
# Predict request rate capacity breach (4 weeks ahead)
predict_linear(
sum(rate(http_requests_total[5m]))[8w:],
4*7*24*3600
) > 10000 # known capacity limit
建立預測儀表板:
{
"dashboard": {
"title": "Capacity Forecast",
"panels": [
{
"title": "CPU Saturation Forecast (4 weeks)",
"targets": [
{
"expr": "predict_linear(avg(rate(node_cpu_seconds_total{mode!=\"idle\"}[5m]))[8w:], 4*7*24*3600)",
"legendFormat": "Predicted CPU"
},
{
"expr": "0.80",
"legendFormat": "Target Threshold (80%)"
}
]
},
{
"title": "Disk Full Date",
"targets": [
{
"expr": "(avg(node_filesystem_size_bytes) - predict_linear(avg(node_filesystem_free_bytes)[8w:], 8*7*24*3600)) / avg(node_filesystem_size_bytes)",
"legendFormat": "Predicted Usage %"
}
]
}
]
}
}
預期: 清晰之視覺化,顯示資源何時將觸及門檻。
失敗時: 若預測結果有異(負值、劇烈波動),檢查:
定飽和前之安全邊際:
# CPU headroom (percentage remaining before 80% threshold)
(0.80 - avg(rate(node_cpu_seconds_total{mode!="idle"}[5m]))) / 0.80 * 100
# Memory headroom (bytes remaining before 90% usage)
avg(node_memory_MemAvailable_bytes) - (avg(node_memory_MemTotal_bytes) * 0.10)
# Request rate headroom (requests/sec before saturation)
10000 - sum(rate(http_requests_total[5m]))
# Time until saturation (weeks until CPU hits 80%)
(0.80 - avg(rate(node_cpu_seconds_total{mode!="idle"}[5m]))) /
deriv(avg(rate(node_cpu_seconds_total{mode!="idle"}[5m]))[8w:]) /
(7*24*3600)
製作餘裕摘要報告:
cat > capacity_headroom.md <<'EOF'
# Capacity Headroom Report (2025-02-09)
## Current Utilization
- **CPU**: 45% average (target: <80%)
- **Memory**: 62% (target: <90%)
- **Disk**: 71% (target: <95%)
- **Request Rate**: 4,200 req/s (capacity: 10,000)
## Headroom Analysis
- **CPU**: 35% headroom → ~12 weeks until saturation
- **Memory**: 28% headroom → ~16 weeks until saturation
- **Disk**: 24% headroom → ~8 weeks until full
- **Request Rate**: 5,800 req/s headroom → ~20 weeks until capacity
## Priority Actions
1. **Disk**: Implement log rotation or expand volume within 4 weeks
2. **CPU**: Plan horizontal scaling in next quarter
3. **Memory**: Monitor but no immediate action needed
EOF
預期: 各資源之量化餘裕,附飽和時間估計。
失敗時: 若餘裕已為負,已陷被動模式。需立即擴容。
納入業務預測:
# Example Python script for scenario modeling
import pandas as pd
import numpy as np
# Load historical data
df = pd.read_json('cpu_8weeks.json')
# Calculate weekly growth rate
growth_rate_weekly = df['value'].pct_change(periods=7).mean()
# Scenario 1: Current trend
weeks_ahead = 12
current_trend = df['value'].iloc[-1] * (1 + growth_rate_weekly) ** weeks_ahead
# Scenario 2: 2x user growth (marketing campaign)
accelerated_trend = df['value'].iloc[-1] * (1 + growth_rate_weekly * 2) ** weeks_ahead
# Scenario 3: New feature launch (+30% baseline)
feature_launch = (df['value'].iloc[-1] * 1.30) * (1 + growth_rate_weekly) ** weeks_ahead
print(f"Current Trend (12 weeks): {current_trend:.1%} CPU")
print(f"2x Growth Scenario: {accelerated_trend:.1%} CPU")
print(f"Feature Launch Scenario: {feature_launch:.1%} CPU")
print(f"Threshold: 80%")
預期: 多種情境,呈現業務變動對容量之影響。
失敗時: 若情境超出容量,於事件前優先擴容。
擬定可執行之建議:
## Capacity Scaling Plan
### Immediate Actions (Next 4 Weeks)
1. **Disk Expansion** [Priority: HIGH]
- Current: 500GB, 71% used
- Projected full date: 2025-04-01 (8 weeks)
- Action: Expand to 1TB by 2025-03-15
- Cost: $50/month additional
- Justification: 5 weeks lead time needed
2. **Log Rotation Policy** [Priority: MEDIUM]
- Current: Logs retained 90 days
- Action: Reduce to 30 days, archive to S3
- Savings: ~150GB disk space
- Cost: $5/month S3 storage
### Near-Term Actions (Next Quarter)
3. **Horizontal Scaling - API Tier** [Priority: MEDIUM]
- Current: 4 instances, 45% CPU
- Projected: 65% CPU by 2025-05-01
- Action: Add 2 instances (to 6 total)
- Cost: $400/month
- Trigger: When CPU avg exceeds 60% for 7 days
4. **Database Connection Pool** [Priority: LOW]
- Current: 50 max connections, 40% used
- Projected: 55% by Q3
- Action: Increase to 75 in Q2
- Cost: None (configuration change)
### Long-Term Planning (Next 6 Months)
5. **Migration to Auto-Scaling** [Priority: MEDIUM]
- Current: Manual scaling
- Action: Implement Kubernetes HPA (Horizontal Pod Autoscaler)
- Timeline: Q3 2025
- Benefit: Automatic response to load spikes
預期: 已排序之清單,附成本、時程與觸發條件。
失敗時: 若建議因成本被否,重新檢視門檻或接受風險。
對低餘裕設置告警:
# capacity_alerts.yml
groups:
- name: capacity
interval: 1h
rules:
- alert: CPUCapacityLow
expr: |
(0.80 - avg(rate(node_cpu_seconds_total{mode!="idle"}[5m]))) / 0.80 < 0.20
for: 24h
labels:
severity: warning
annotations:
summary: "CPU headroom below 20%"
description: "Current CPU headroom: {{ $value | humanizePercentage }}. Scaling needed within 4 weeks."
- alert: DiskFillForecast
expr: |
predict_linear(avg(node_filesystem_free_bytes)[8w:], 4*7*24*3600) < 0.10 * avg(node_filesystem_size_bytes)
for: 1h
labels:
severity: warning
annotations:
summary: "Disk projected to fill within 4 weeks"
description: "Expand disk volume soon."
- alert: MemoryCapacityLow
expr: |
avg(node_memory_MemAvailable_bytes) < 0.15 * avg(node_memory_MemTotal_bytes)
for: 6h
labels:
severity: warning
annotations:
summary: "Memory headroom below 15%"
預期: 告警於飽和前觸發,留下主動擴容之時間。
失敗時: 若告警過頻(告警疲勞)或過遲(被動應急),調整門檻。
predict_linear() 查詢回傳合理預測(無負值)setup-prometheus-monitoring —— 蒐集容量規劃所用之指標build-grafana-dashboards —— 視覺化預測與餘裕optimize-cloud-costs —— 在容量規劃與成本最佳化間取得平衡