算法艺术创作
algorithmic-art
通过代码和可调参数创作生成艺术与交互式视觉效果。
enterprise-business-continuity【设计与创意】Use this skill when designing Business Continuity Plans (BCP) and Disaster Recovery (DR) at the business-service level: ranking services by criticality, mapping RPO/RTO per service, planning for regional outage, vendor failure, ransomware, key-person loss, supply-chain attack, and pandemic/site loss. This skill enforces: BIA (Business Impact Analysis…
独立 GitHub 快照 · 自动分类 · 以原作者文档为准
Define the BCP that keeps the business running through any plausible disruption -- regional outage, vendor collapse, ransomware, key-person loss, pandemic, supply-chain compromise, data-center loss. Maps technical DR to business outcomes via BIA and tier classification.
A six-phase methodology for building organizational resilience:
Phase 1 - Recognize: Identify critical business services, their revenue impact, regulatory obligations, and stakeholder expectations. Conduct Business Impact Analysis for every service. Classify by criticality tier.
Phase 2 - Evaluate: Assess risks across all threat categories. Map dependencies between services, infrastructure, vendors, and people. Identify single points of failure and concentration risks.
Phase 3 - Strategize: Define recovery objectives (RTO, RPO, MAO) per tier. Select recovery strategies (active-active, active-passive, backup-restore, manual workaround). Design vendor fallback plans.
Phase 4 - Implement: Build DR runbooks for every Tier-1 service. Establish crisis communication tree. Set up war-room infrastructure. Deploy backup and replication per strategy. Document holding statements.
Phase 5 - Learn: Conduct drills at scheduled cadence. Tabletop exercises quarterly. Technical failover semi-annually. Full BCP exercise annually. Incorporate lessons learned into plan updates.
Phase 6 - Nurture: Maintain the BCP through regular reviews, audits, and training. Align with insurance coverage and regulatory requirements. Update as architecture changes.
The Business Impact Analysis quantifies the cost of downtime for each service:
Revenue Impact: Lost revenue per hour of downtime. Include direct (transaction-based) and indirect (reputation-driven churn).
Productivity Impact: Internal user hours lost. If 500 employees cannot work at $100/hour blended cost, that is $50,000/hour.
Regulatory Impact: Fines, penalties, legal costs. GDPR fines up to 4% of global revenue. PCI fines per card number exposed.
Recovery Cost: Overtime, vendor emergency fees, expedited shipping, cloud spike costs.
Customer Impact: Churn rate acceleration. Post-outage support volume. SLA credit payouts.
Compute MAO (Maximum Acceptable Outage) as the intersection of financial survivability and stakeholder tolerance.
| Strategy | RPO | RTO | Cost | Complexity | Best For |
|---|---|---|---|---|---|
| Active-Active | <1s | <1min | 2x-3x | High | Tier-1 revenue-critical |
| Active-Passive | <5min | <15min | 1.5x-2x | Medium | Tier-1 and Tier-2 |
| Backup-Restore | 1-24h | 4-48h | 1x-1.2x | Low | Tier-2 and Tier-3 |
| Manual Workaround | N/A | 24h+ | Minimal | N/A | Tier-4 |
Exact user phrases: "business continuity", "BCP", "BIA", "business impact analysis", "disaster recovery plan", "DR plan", "crisis management", "ransomware response", "vendor lock-in", "vendor failure", "key person risk", "site loss", "pandemic plan", "regional outage", "tabletop exercise", "continuity drill", "RPO", "RTO" (when at business level).
BCP document with BIA, tier matrix, DR runbooks per tier, comms tree, drill schedule.
Tier-1 services: {list, RTO, RPO, MAO, dependencies}
Tier-2 services: {...}
Tier-3 services: {...}
Crisis comms: {chain, channels, holding statement template}
Drill cadence: {tabletop quarterly, full failover annual}
# Runbook entries
No preamble. No postamble. No explanations. No filler/hedging/transitions. Compress output.
350 lines.
For each business service, compute:
Service Revenue/hr Customers Regulatory MAO Tier
Checkout $50,000 100k PCI 15m 1
Auth/SSO $0 500k - 30m 1
Reporting $0 5k - 4h 2
Internal CMS $0 200 - 24h 3
BIA interview questions per service owner:
Tier-1 (revenue / regulatory critical)
RTO <= 15m, RPO <= 1m, 24x7 on-call, multi-region active-active, automated failover
Tier-2 (important, recoverable)
RTO <= 4h, RPO <= 15m, business-hour on-call, multi-AZ, semi-auto promotion
Tier-3 (best effort)
RTO <= 24h, RPO <= 4h, business-hour, single-region, backup-restore
Tier-4 (deferrable)
RTO <= 1 week, RPO <= 24h, manual recovery from cold backup
Tier granularity: some services have sub-components that need different tiers. Auth service authentication path is Tier-1, profile management is Tier-2. Document per-path RTO if needed.
Build a graph: Business Service -> App -> DB -> Infra -> Vendor. Find the long pole -- your tier is capped by your weakest dependency.
Checkout (Tier-1)
+- payment-service
| +- stripe (vendor: SLA 99.95% -> caps Checkout at 99.95% unless redundant PSP)
| +- db-payments (multi-AZ sync)
+- inventory-service -> db-inventory
+- notification -> SES (vendor)
Action: every Tier-1 vendor needs a documented fallback (secondary PSP, secondary email, etc.).
Map both direct and transitive dependencies. A Tier-1 service may depend on a Tier-3 logging service. The logging service failure should not bring down the Tier-1 service. Design for graceful degradation: the Tier-1 service should function without logs during an outage.
Mandatory scenarios to cover:
Each playbook: detection -> decision authority -> first 30min actions -> escalation -> comms -> recovery -> postmortem.
Playbook structure standard:
Severity 1 (full outage of Tier-1)
T+0 on-call engineer pages
T+5min incident commander declared, war room opened (Slack #inc-YYYYMMDD-NNN)
T+10min exec on-call notified (CTO + COO)
T+15min status page updated with holding statement
T+30min customer success team briefed; key accounts contacted
T+1h external comms (twitter/email) if user-visible
T+2h executive update; legal/PR engaged if data involved
Hourly status page updates until resolved
Holding statement template stored in references/holding-statements.md.
Define communication channels per audience: internal (Slack, email), customers (status page, email), partners (direct call), regulatory (registered letter with timeline), press (PR-approved statement).
Tier-1 vendor Primary Fallback Switchover time
Payments Stripe Adyen (pre-integrated) manual flip, 30m
Email transactional SendGrid SES (pre-configured) DNS swap, 15m
DNS Route53 Cloudflare DNS 15m TTL
CDN Cloudflare Fastly DNS swap, 30m
Auth Auth0 in-house SAML fallback 4h (degraded)
Observability DataDog Grafana Cloud manual cutover, 4h
Rule: if a vendor outage would breach Tier-1 MAO, you MUST have a tested fallback.
Fallback testing: switch traffic to fallback in staging quarterly. Measure switchover time and degradation. Document any manual steps that cannot be automated.
Quarterly: tabletop exercise (1 scenario, 2h, exec + eng + comms attend)
Semi-annual: technical failover drill (region failover with real traffic)
Annual: full BCP exercise (multi-team, multi-hour, includes vendor fallback)
On change: re-test affected runbook within 30 days of architectural change
Drill documentation: each drill produces an after-action report with timeline, decisions made, gaps identified, and action items. Track action items to closure before next drill.
Insurance review cadence: annually during renewal. Verify coverage limits match current MAO. Document any coverage gaps and present to risk committee.
Pitfall 1: RTO/RPO that match technical capability rather than business need. Setting RTO to 1 hour because you can, when the business needs 4 hours, over-engineers the solution and wastes budget. Set RTO/RPO from BIA, not from infrastructure limitations.
Pitfall 2: Untested backups. A backup that has never been restored is not a backup. Test full restore quarterly. Test partial restore (single file/DB) monthly. Measure restore time against RTO.
Pitfall 3: Single-region dependency in a multi-region architecture. Your app may be multi-region, but if all instances connect to a single-region database or queue, you have no real redundancy.
Pitfall 4: No executive buy-in for drill participation. Tabletop exercises fail when executives are too busy to attend. Schedule drills on the executive calendar 6 months in advance. Make attendance mandatory.
Pitfall 5: Plans that are not updated. A BCP written and never revisited becomes stale within 6 months. Architecture changes, vendor changes, team changes all invalidate the plan.
Pitfall 6: Assuming vendor SLA equals availability. An SLA is a contractual commitment with credits, not a guarantee. Your DR plan must cover vendor outages regardless of SLA.
Pitfall 7: Ignoring people dependencies. Key-person risk is real. Every critical process must have at least two trained operators. Document runbooks so they are not dependent on memory.
Pitfall 8: Holding statements not pre-approved. Writing crisis communications under pressure with legal review creates delays. Pre-approve templates for common scenarios.
Practice 1: Start with the BIA, not the solution. Understand what the business needs before designing technical DR. The BIA drives RTO/RPO, which drives architecture decisions.
Practice 2: Design for graceful degradation. When a dependency fails, degrade functionality rather than going completely dark. Show cached data. Accept orders but process later. Display a waiting page.
Practice 3: Automate failover testing. Manual failover drills are good. Automated chaos engineering (GameDays) is better. Introduce real failures (kill a region, throttle a vendor) and verify the system recovers.
Practice 4: Keep the BCP as a living document. Store in a wiki or shared drive, not a PDF on one person's laptop. Anyone in the on-call rotation should be able to find and execute the runbook.
Practice 5: Pre-declare war-room channels and tools. Slack channel naming convention (#inc-YYYYMMDD-NNN). Zoom bridge always available. Status page provider configured. Do not configure tools during an incident.
Practice 6: Cross-train for key roles. Incident commander, scribe, comms lead, and technical lead should all have backups. Rotate role assignments in drills so everyone is familiar.
| Standard | Requirement | BCP Mapping |
|---|---|---|
| ISO 22301 | Business continuity management system | Full BCP framework |
| ISO 27031 | ICT readiness for business continuity | IT/DR component |
| NIST SP 800-34 | Contingency planning | DR runbooks, testing |
| SOC 2 | Business continuity / disaster recovery | BCP, DR tests, evidence |
| HIPAA | 164.308(a)(7) - Contingency Plan | Emergency mode, backup, DR |
| PCI DSS | Requirement 12 - Information Security Policy | BCP including CDE recovery |
1. Executive Summary
2. Plan Scope and Assumptions
3. Business Impact Analysis
4. Service Tier Classification
5. Recovery Strategies by Tier
6. Dependency Maps
7. Crisis Communication Plan
8. Failure Scenario Playbooks (per scenario)
9. Vendor Fallback Plans
10. Drill Schedule and After-Action Reports
11. Insurance and Regulatory Alignment
12. Plan Maintenance and Review Process
13. Appendices (runbooks, contact lists, system diagrams)
Revenue Impact = (Avg Revenue per Hour × Outage Duration) + (Churn Rate × CLV × Affected Users)
Productivity Impact = (Affected Employees × Blended Hourly Rate × Outage Duration)
Regulatory Impact = Base Fine + (Affected Records × Per-Record Penalty)
Recovery Cost = (Overtime Hours × OT Rate) + Emergency Vendor Fees + Cloud Spike Costs
Total Cost of Outage = Revenue + Productivity + Regulatory + Recovery
class BusinessImpactAnalysis:
def __init__(self, service_name, revenue_per_hour, affected_users, churn_rate=0.05):
self.service_name = service_name
self.revenue_per_hour = revenue_per_hour
self.affected_users = affected_users
self.churn_rate = churn_rate
def revenue_impact(self, hours):
return self.revenue_per_hour * hours
def churn_impact(self, clv=500):
return self.affected_users * self.churn_rate * clv
def regulatory_fine(self, records_exposed=0, per_record_penalty=0):
return records_exposed * per_record_penalty
def mao_hours(self, max_financial_loss=100000):
return max_financial_loss / (self.revenue_per_hour + (self.affected_users * self.churn_rate * 500 / 8760))
def tier_classify(self):
rph = self.revenue_per_hour
if rph > 100000 or self.affected_users > 1000000:
return "Tier-1"
elif rph > 10000 or self.affected_users > 100000:
return "Tier-2"
elif rph > 1000:
return "Tier-3"
return "Tier-4"
checkout = BusinessImpactAnalysis("Checkout", 50000, 100000)
print(checkout.mao_hours()) # Max hours before $100K loss
services:
checkout:
tier: 1
rto: 15m
rpo: 1m
dependencies:
- payment-service
- inventory-service
- auth-service
vendors:
stripe: { sla: "99.95%", fallback: adyen }
dr_runbook: runbooks/checkout-dr.md
payment-service:
tier: 1
rto: 15m
rpo: 1m
dependencies:
- db-payments
vendors:
stripe: { sla: "99.95%", fallback: adyen }
# DR Runbook: {Service Name}
## Tier: {1-4} | RTO: {time} | RPO: {time}
### Detection
- Alert: {metric} crosses {threshold}
- Monitoring: {dashboard URL}
### Decision
- Declare incident when: {criteria}
- Decision authority: {role}
### First 30 Minutes
1. {immediate action}
2. {parallel workstream}
3. {containment step}
### Recovery Procedure
1. {step 1}
2. {step 2}
3. {verification step}
### Rollback
- Trigger if: {condition}
- Procedure: {steps}
### Communication
- Internal: {Slack channel}
- Status page: {URL}
- Executive: {contact}
- Regulatory: {notification timeline}
# Example: Kill a region to test failover
scenario = {
"name": "us-east-1-region-failure",
"target": "checkout-service",
"attack": {
"type": "blackhole",
"target": "us-east-1",
"duration": 3600
},
"hypothesis": "Traffic routes to us-west-2 within 30s with <1% error rate",
"probes": [
"http_get https://checkout.example.com/health",
"metric_query 'aws_alb_target_response_time'",
"metric_query 'aws_alb_error_rate_5xx'"
],
"rollback": "dns_failover_to_us_west_2"
}
Skipping the Business Impact Analysis and guessing RTO/RPO leads to either over-engineered (waste) or inadequate (risk) DR. Teams set RTO=1h because "that sounds right" without knowing the actual MAO. This results in spending $50K/month on multi-region replication for a service that could tolerate 24h of downtime.
One person writes and maintains all runbooks. When that person leaves or is unavailable, no one knows how to execute DR. Runbooks become stale because the single owner is a bottleneck. Every runbook must have a secondary owner and be reviewed quarterly.
If executing the runbook requires SSH access to production, database credentials, and intimate knowledge of the architecture, the plan will fail when the primary engineers are unavailable. Runbooks must be executable by any on-call engineer with standard access.
Every drill succeeds because everyone knows the scenario in advance and prepares. The real value is in testing failure modes: partial failure, cascading failure, network partition, corrupted data. Test with chaos engineering in production.
Plans reflect the last outage rather than anticipating future threats. If the last disaster was a cloud region failure, the plan focuses on multi-region. Meanwhile, a ransomware or supply-chain attack would bypass those defenses. Use threat modeling to identify current risks, not historical ones.
A mid-stage SaaS company with active-active multi-region architecture survived a 6-hour AWS us-east-1 outage with zero revenue impact. Key factors: all Tier-1 services deployed in us-west-2 and eu-west-1 with traffic routing via Route53 health checks. Database was multi-region with asynchronous replication. Failover was fully automated. The BCP called for a quarterly failover drill, which had been executed 3 weeks before the actual outage.
A healthcare company suffered a ransomware attack that encrypted their primary database and all online backups. Because immutable backups (write-once-read-many storage) were part of their BCP, they recovered from a clean snapshot taken 4 hours before the attack. Data loss was limited to 4 hours of transactions (within RPO). The DR runbook, tested semi-annually, guided recovery in 6 hours (within RTO). Total cost: $2M in recovery and security hardening vs. potential $50M ransom demand.
A fintech startup lost their only infrastructure engineer with root access to all production systems. Root credentials were in a password manager with multi-person access (pre-configured per BCP), so the remaining team could access systems. Runbooks documented deployment, monitoring, and incident response procedures. Within 48 hours, two contractors were onboarded and productive. The BCP was updated to require minimum two operators per critical function.
enterprise-high-availability for technical replication / failover design.enterprise-sla-management for customer SLA structure and credit calculations.devops-backup-dr for backup tooling, retention, immutability config.devops-incident-response for paging, on-call rotation, postmortem template.management-risk-management for enterprise risk register integration.security-* for ransomware forensics and breach response coordination.