---

name: quality-engineering-automation-health description: Measures whether an automation suite builds release confidence via feedback-loop length, suite reliability, release cadence, and production escape rate, and emits a release_confidence verdict. Use when judging suite value, ROI, or pre-release trust; not for writing or healing tests. metadata: triggers: keywords: - automation health - release confidence - suite reliability - flaky rate - feedback loop - escape rate - automation roi - suite value

Quality Engineering: Automation Health

Priority: P1 (HIGH)

Mission

Automation exists to reduce ambiguity at release time, not to catch every bug. Finding bugs is testing's job; building confidence is automation's. Judge a suite by whether the team can deploy on Friday afternoon without fear.

Three Questions

A high-value suite answers all three with evidence:

1. Core workflows intact: the flows that create revenue and user value still run end to end.
2. No serious regression: the latest change did not break what was already stable.
3. Fast feedback: a developer learns what they broke in minutes, not hours.

Four Metrics

Metric	Key	Question it answers
Feedback loop	feedback_loop_minutes	How long from push to a trusted green or red?
Suite reliability	suite_reliability_pct	When a run is red, does the team investigate or just re-run?
Release cadence	release_cadence	Did automation let the team ship more often without more production risk?
Production escape rate	prod_escape_rate	How many serious defects passed the whole pipeline and reached real users?

Formulas and data sources per CI provider live in Metrics Definitions.

Verdict

release_confidence: high | medium | low

• high: all three questions answered yes with evidence; reliability at or above the team threshold; escape rate at or below baseline; feedback loop within target.
• medium: one question lacks evidence, or reliability or feedback loop misses target while escapes stay at baseline.
• low: any question answered no, a red run is not trusted, or escape rate rose after the last release.

Report with Confidence Report Template; feed release_confidence into test-loop, uat-signoff, and deploy-release handoffs.

Anti-Patterns

• No ranking by bug count: a suite that catches few bugs on CI usually means developers catch them locally first; that is success, not waste.
• No deleting never-failing tests: a test that never fails is a safety net, not dead weight. It is what lets engineers refactor, bump dependencies, and change config without silent breakage.
• No coverage % as the goal: coverage measures lines touched, not confidence earned.
• No pass rate as health: a 99% pass rate with untrusted reds is worse than 95% the team believes.

Red Flags

"this test never fails, delete it" · "just re-run it, it's probably flaky" · "we found zero bugs so automation isn't paying off" · "coverage is 90%, we're safe". Each swaps a confidence question for a vanity number; re-frame with the Three Questions before acting.

References

• Metrics Definitions
• Confidence Report Template