quality-regression-testingUse when the user asks about regression testing, test selection, regression suite design, automation maintenance, flaky tests, or regression metrics. Do NOT use for: smoke/BVT testing (quality-smoke-testing), acceptance testing (quality-acceptance-testing), or exploratory testing (quality-exploratory-testing).
Independent GitHub snapshot · Automated classification · Original documentation governs
Ensure that new code changes do not break existing functionality through effective test selection, suite design, automation maintenance, and metric-driven optimization. This skill covers impact analysis, risk-based selection, prioritization, flaky test management, and CI integration.
User mentions regression testing, test selection, regression suite, flaky tests, automation maintenance, or asks what to re-test after a change.
Structured recommendation with:
How many tests are in the suite?
├── < 100 → Run all (no selection needed)
├── 100-1000 → Dependency-graph selection
│ ├── Dependency graph available? → Use graph
│ └── No graph → Coverage-based selection
├── 1000-10000 → Risk-based + dependency selection
│ ├── Historical data available? → Add failure rate to risk model
│ └── No history → Coverage + module risk only
└── 10000+ → ML-based selection
├── Data available? → Train selection model
└── No data → Start with risk-based, collect data
Layer 1 (Smoke): < 5 min, always run on every deployment
Critical path tests, auth, main workflow
Layer 2 (Core): < 30 min, run on every merge
All core business functionality, all integration tests
Layer 3 (Full): < 2 hours, run daily
Complete suite including edge cases
Layer 4 (Extended): 2-8 hours, run pre-release
Full suite + performance + chaos + security
| Aspect | Regression Testing | Smoke Testing | Acceptance Testing |
|---|---|---|---|
| Scope | Existing functionality | Critical path | Business requirements |
| Frequency | Every change + scheduled | Every deployment | Per release/story |
| Suite size | 100-10000+ tests | 5-20 tests | 10-100 scenarios |
| Execution time | Minutes to hours | < 5 minutes | Hours to days |
| Selection | Risk-based + selection | Fixed suite | Per-feature planning |
| Focus | Regression detection | Stability check | Requirement validation |
| Automation | Fully automated | Fully automated | Mixed (automated + manual) |
# .regression/dependency-graph.yml
services:
- name: "payment-service"
modules:
- "src/payment/**"
- "src/common/transactions/**"
affected_tests:
- "tests/payment/**"
- "tests/integration/checkout/**"
risk_score: 0.9
business_criticality: "P0"
- name: "user-service"
modules:
- "src/user/**"
affected_tests:
- "tests/user/**"
- "tests/integration/auth/**"
risk_score: 0.7
business_criticality: "P1"
risk_scoring:
factors:
change_proximity:
weight: 0.4
rules:
- "directly modified file: score 1.0"
- "imported by modified file: score 0.5"
- "transitively imported: score 0.2"
failure_history:
weight: 0.3
rules:
- "failed in last 10 runs: score 1.0"
- "failed in last 50 runs: score 0.5"
- "never failed: score 0.1"
business_criticality:
weight: 0.3
rules:
- "P0 (revenue-critical): score 1.0"
- "P1 (important): score 0.7"
- "P2 (standard): score 0.4"
- "P3 (nice-to-have): score 0.1"
thresholds:
run_always: ">= 0.8"
run_if_budget: ">= 0.4"
skip: "< 0.4"
name: Regression Tests
on:
pull_request:
branches: [main]
schedule:
- cron: "0 2 * * *" # Daily full regression
jobs:
smoke-tier:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npx playwright test --grep @smoke --timeout 30000
env:
CI: true
- if: failure()
run: echo "Smoke tests failed - blocking merge"
# Pipeline stops here on failure
core-tier:
needs: smoke-tier
runs-on: ubuntu-latest
strategy:
matrix:
shard: [1, 2, 3]
steps:
- uses: actions/checkout@v4
- name: Compute affected tests
run: |
CHANGED=$(git diff --name-only origin/main...HEAD)
node .regression/select-tests.js --changes="$CHANGED" --tier=core
- run: npx vitest run --shard=${{ matrix.shard }}/3 $(cat test-list.txt)
env:
CI: true
full-regression:
if: github.event_name == 'schedule'
runs-on: ubuntu-latest
strategy:
matrix:
shard: [1, 2, 3, 4, 5, 6, 7, 8]
steps:
- uses: actions/checkout@v4
- run: npx vitest run --shard=${{ matrix.shard }}/8
Running the complete regression suite on every commit wastes CI resources and slows feedback. Once a suite exceeds 100 tests, implement test selection. Start with dependency-graph-based selection, add risk scoring as historical data accumulates.
Failing to quarantine flaky tests erodes trust in the entire test suite. Developers start ignoring failures, including real regressions. Quarantine flaky tests within 24 hours. Fix within one sprint or delete the test.
Running regression tests manually is slow, error-prone, and doesn't scale. Every regression test must be automated. Manual testing has its place (exploratory, UAT) but regression is repetitive by definition.
Regression suites grow without bound as new features are added without removing obsolete tests. A 10,000-test suite that's never pruned has significant dead weight. Audit quarterly: remove tests for removed features, consolidate overlapping tests, retire flaky tests.
Without time budgets, regression suites grow unbounded. Each sprint adds tests but never removes them. Set a total suite time budget. When the budget is exceeded, prioritize: run highest risk tests first, defer low-risk tests to nightly.
Using test selection without measuring recall means you don't know what you're missing (false negatives). Track bugs that escaped to production and should have been caught by regression. If recall drops below 75%, refine the selection model.
| Level | Characteristics | Practices |
|---|---|---|
| 1: Initial | Manual regression | Testers manually re-test critical paths, no automation, no tracking |
| 2: Defined | Automated regression suite | Full suite automated, run on every merge, basic CI integration |
| 3: Managed | Tiered regression | Smoke/core/full tiers, risk-based selection, test selection by dependency graph |
| 4: Measured | Optimized regression | ML-based selection, flaky test auto-quarantine, regression metrics dashboard, < 60 min full suite |
| 5: Optimized | Predictive regression | AI-driven test selection, mutation-based selection validation, automatic test generation from production failures |
metrics:
suite_health:
total_tests: 4250
active: 4120
quarantined: 45
flaky_rate: 1.1%
execution_time_avg: "42 minutes"
selection:
avg_tests_per_pr: 340
selection_recall: 87%
false_negative_rate: 2.3%
missed_bugs_last_quarter: 3
quality_gates:
smoke_pass_rate: 99.8%
core_pass_rate: 97.5%
full_pass_rate: 96.2%
blocking_failures_per_week: 2.1
flaky_test_workflow:
detection:
- "Auto-detect: test fails in CI but passes re-run (max 3 retries)"
- "Mark as flaky: quarantine from main suite, notify owner"
- "Track: flaky rate per test, per suite, per team"
quarantine:
- "Move to quarantined suite: excluded from CI gates"
- "Assign owner: last modifier of test file"
- "Set SLA: fix within 5 business days or delete"
investigation:
steps:
- "Check test isolation: shared state, timing dependencies"
- "Check environment: resource contention, network latency"
- "Check data: non-deterministic fixtures, stale mocks"
- "Check assertions: race conditions, async timing"
resolution:
- "Fix root cause: improve isolation, add retry logic, stabilize data"
- "Verify: 10 consecutive runs in CI with zero failures"
- "Re-integrate: move from quarantined back to main suite"
metrics:
flaky_rate_target: "< 1% of total suite"
quarantine_age_target: "< 5 days"
auto_fix_rate: "> 50% (common patterns: wait strategies, data isolation)"
test_data_principles:
isolation:
- "Each test run gets its own dataset — no sharing across runs"
- "Use unique identifiers (timestamp + run ID) to avoid collisions"
- "Clean up after test: delete-by-run-id in afterAll"
factories:
- "Use factory functions with sensible defaults and overrides"
- "Generate unique values for constrained fields (email, username)"
- "Never hardcode IDs — they may not exist in test environment"
seeding:
- "API-level seeding: POST /api/test/seed with required entities"
- "Database-level seeding: SQL INSERT for bulk data needs"
- "Seed in beforeAll, clean in afterAll"
environments:
- "CI: fresh seed per run, cleanup guaranteed"
- "Staging: seeded data with known state, refreshed daily"
- "Production: never use in regression tests except read-only"
Running regression tests only before release and not monitoring production after deployment. The pre-release suite may pass while production has problems due to configuration differences, data volume, or real user behavior. Implement production health checks and canary analysis alongside pre-release regression.
Security regression tests are skipped to save time in the deployment pipeline. Security vulnerabilities re-emerge when changes inadvertently reintroduce previously fixed issues. Security regression tests must run on every deployment regardless of change scope.
Relying on manual testing for regression sign-off. Manual regression is slow, error-prone, and doesn't scale beyond a few tests. Every regression test must be automated. Reserve manual testing for exploratory testing and UAT.
After regression testing, hand off to:
quality-smoke-testing — for BVT smoke suite updates on new critical pathsquality-acceptance-testing — if acceptance criteria gaps were uncoveredquality-e2e-testing — for end-to-end coverage of newly identified risk areasquality-integration-testing — for deeper investigation of regression failuresfrom typing import List, Dict, Optional, Callable
from dataclasses import dataclass
from datetime import datetime
import json
import time
@dataclass
class RegressionTest:
name: str
run_fn: Callable
tier: int # 1=critical, 2=important, 3=nice-to-have
module: str
timeout: int = 30
retries: int = 0
@dataclass
class TestResult:
name: str
passed: bool
duration_ms: float
error: Optional[str] = None
retry_count: int = 0
class RegressionSuite:
def __init__(self, name: str):
self.name = name
self.tests: List[RegressionTest] = []
def add_test(self, test: RegressionTest):
self.tests.append(test)
def run_tier(self, tier: int, parallel: bool = False) -> Dict:
tests = [t for t in self.tests if t.tier == tier]
results = []
start = time.time()
if parallel:
import concurrent.futures
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
futures = {executor.submit(self._run_single, t): t for t in tests}
for future in concurrent.futures.as_completed(futures):
results.append(future.result())
else:
for test in tests:
results.append(self._run_single(test))
duration = time.time() - start
passed = [r for r in results if r.passed]
failed = [r for r in results if not r.passed]
return {
"suite": self.name,
"tier": tier,
"total": len(results),
"passed": len(passed),
"failed": len(failed),
"duration_seconds": round(duration, 2),
"results": results,
}
def _run_single(self, test: RegressionTest) -> TestResult:
for attempt in range(test.retries + 1):
start = time.time()
try:
test.run_fn()
duration = (time.time() - start) * 1000
return TestResult(
name=test.name,
passed=True,
duration_ms=round(duration, 2),
retry_count=attempt,
)
except Exception as e:
if attempt < test.retries:
continue
duration = (time.time() - start) * 1000
return TestResult(
name=test.name,
passed=False,
duration_ms=round(duration, 2),
error=str(e),
retry_count=attempt,
)
return TestResult(name=test.name, passed=False, duration_ms=0)
def run_all(self, parallel: bool = False) -> Dict:
tiers = sorted(set(t.tier for t in self.tests))
all_results = {}
for tier in tiers:
all_results[tier] = self.run_tier(tier, parallel)
return all_results
class FlakyTestDetector:
def __init__(self, window_size: int = 10):
self.history: Dict[str, List[bool]] = {}
self.window_size = window_size
def record_result(self, test_name: str, passed: bool):
if test_name not in self.history:
self.history[test_name] = []
self.history[test_name].append(passed)
if len(self.history[test_name]) > self.window_size:
self.history[test_name].pop(0)
def is_flaky(self, test_name: str, threshold: float = 0.2) -> bool:
if test_name not in self.history:
return False
results = self.history[test_name]
if len(results) < 5:
return False
pass_rate = sum(results) / len(results)
return pass_rate < (1 - threshold) and pass_rate > threshold
What changed?
├── New feature added
│ ├── Add tests for new feature
│ ├── Run full regression for affected module
│ └── Run smoke tests for entire system
│
├── Bug fix
│ ├── Add regression test for the fixed bug
│ ├── Run all tests in the affected module
│ └── Run integration tests for dependent modules
│
├── Refactoring (no behavior change)
│ ├── Run full test suite for refactored module
│ ├── Contract tests for API changes
│ └── Compare before/after test coverage
│
├── Dependency update
│ ├── Run full regression suite
│ ├── Run security-focused tests
│ └── Test edge cases in updated dependency
│
└── Configuration / infrastructure change
├── Smoke tests for deployment
├── Health check tests
└── Performance benchmark comparison
| Anti-Pattern | Why It Fails | Correct Approach |
|---|---|---|
| No tiered regression | Full suite too slow for CI feedback | Tier 1 (critical) in 5 min, Tier 2 in 15 min, Tier 3 nightly |
| Ignoring flaky tests | Wasted CI time, distrust in suite | Quarantine flaky tests immediately, fix within 5 days |
| Only adding, never removing | Suite grows unbounded, slow | Remove tests for deleted features, review suite quarterly |
| Determinism not enforced | Random failures from shared state | Isolate test data per run, no shared state |
| No baseline for comparison | Don't know if performance regressed | Store baseline metrics, compare on each run |
git diff to identify changed files. Map files to test suites. Only run affected suites. Cache unaffected suite results from previous CI run.