verify-web-app-runtimeVerify that a frontend or visual change actually works by running the app in headless Chromium and observing pixels plus console output — runtime verification for WebGL, GPGPU, Canvas, and WebAudio surfaces that static review and unit tests cannot cover. Launches with the SwiftShader/ANGLE flags WebGL2 needs, probes EXT_color_buffer_half_float before trusti…
Independent GitHub snapshot · Automated classification · Original documentation governs
Prove that a web app's visual runtime actually works — not that its elements exist, not that its unit tests pass, but that real pixels land on a real canvas with the expected GPU capabilities, in a headless browser with a CI-friendly exit code. This skill encodes the gotchas that bite every cold start of headless WebGL verification, each as an explicit procedure step.
http://localhost:5173/myapp/ — a missing base path 404s silently)click / wait / assert_attr step objects (clicks double as
the WebAudio user gesture)./verify-runtime-out)gpgpu, webgl error, nan, fallback)EXT_color_buffer_half_float is required
(default: yes; pass --skip-half-float for apps without a GPGPU path)The packaged verifier at scripts/verify_runtime.py implements every step below; the procedure explains what it asserts and why, so each check can also be reproduced or adapted standalone.
HMR does not reset GPGPU textures: a hot-reloaded tab keeps the previous simulation state (e.g. particle position textures survive the module swap), so anything observed through HMR proves nothing about a cold start. Only a fresh page load in a fresh browser is a valid test surface.
npm run dev & # or: npm run build && npm run preview
curl -sf http://localhost:5173/myapp/ >/dev/null && echo "server up"
Expected: The app answers on a stable URL. All verification below happens
via fresh browser.launch() + page.goto() — never by observing a tab that
hot-reloaded.
On failure: If the URL 404s, check the base path — Vite apps often serve
under /<repo-name>/, and the bare origin silently serves nothing. If only an
HMR-updated tab shows the change, restart the dev server or build a preview
before verifying.
Headless Chromium has no GPU; without software rendering flags,
getContext('webgl2') returns null and every downstream check fails for
the wrong reason.
CHROMIUM_ARGS = [
"--use-gl=angle",
"--use-angle=swiftshader",
"--enable-unsafe-swiftshader",
"--ignore-gpu-blocklist",
]
browser = playwright.chromium.launch(headless=True, args=CHROMIUM_ARGS)
Expected: A WebGL2 context is obtainable in the headless page (verified by the probe in Step 3).
On failure: Copy the four flags exactly — a misspelled flag is silently
ignored by Chromium and the symptom is identical (webgl2: false). If the
flags are correct and WebGL2 is still unavailable, update the browser:
python -m playwright install chromium.
GPGPU pipelines render to half-float framebuffers. If the extension is missing, well-built apps fall back silently to a non-GPGPU mode — the page looks alive, but the screenshots verify the wrong code path.
webgl = page.evaluate("""() => {
const gl = document.createElement('canvas').getContext('webgl2');
return gl ? {webgl2: true, halfFloat: !!gl.getExtension('EXT_color_buffer_half_float')}
: {webgl2: false};
}""")
Expected: {"webgl2": true, "halfFloat": true}. SwiftShader supports the
extension, so a headless pass here is representative.
On failure: webgl2: false means Step 2's flags did not take effect.
halfFloat: false means any GPGPU verification is invalid — the app is
exercising its fallback. Fail the run (the packaged script does) rather than
screenshot the fallback; relax with --skip-half-float only for apps that
have no GPGPU path at all.
Well-behaved apps pause their requestAnimationFrame loop when
document.hidden is true. A hidden page produces a black canvas that looks
exactly like a rendering bug.
info = page.evaluate("() => ({visibility: document.visibilityState})")
assert info["visibility"] == "visible"
Expected: visibilityState === 'visible' in every context, checked before
trusting any pixel assertion.
On failure: A black canvas with a hidden page is a test-harness artifact,
not an app bug. Call page.bring_to_front() and re-check; do not run other
foreground automation against the same headless browser while verifying.
Reach the surface under test through the UI, exactly as a user would.
Browsers refuse to start an AudioContext without a user gesture — real
Playwright clicks count, programmatic dispatchEvent calls do not.
[
{"action": "click", "role": "button", "name": "Switch to 3D view", "settle": 2.5},
{"action": "click", "role": "button", "name": "Sand", "exact": true, "settle": 4},
{"action": "assert_attr", "role": "button", "name": "Sand", "exact": true,
"attr": "aria-pressed", "equals": "true"}
]
The "Switch to 3D view" / "Sand" names above are examples from the origin
project — replace them with your app's controls. settle (seconds) lets a
simulation reach a representative state before pixels are judged.
python3 scripts/verify_runtime.py --url http://localhost:5173/myapp/ \
--steps steps.json --out /tmp/verify-out
Expected: Every click resolves its target (ARIA role + accessible name
preferred, CSS selector as fallback), and every assert_attr step passes —
e.g. the mode button reports aria-pressed="true", proving the app accepted
the mode switch rather than ignoring the click.
On failure: A click timeout usually means a wrong accessible name — dump
candidates with page.get_by_role("button").all_inner_texts(). An
assert_attr mismatch means the UI ignored the interaction: check the
console log for errors thrown by the click handler.
A present canvas can still be empty. Element presence, canvas dimensions, and even a running rAF loop all pass while the app draws nothing. Only sampled pixels prove rendering.
from PIL import Image
grayscale = Image.open("norm_canvas.png").convert("L")
histogram = grayscale.histogram()
total_pixels = grayscale.size[0] * grayscale.size[1]
lit = sum(histogram[26:]) / total_pixels * 100 # pixels brighter than 25
assert lit >= 1.0, f"canvas effectively black (lit={lit:.1f}%)"
Expected: More than 1% of canvas pixels brighter than grayscale 25. The origin run measured ~9% lit for a healthy particle scene — comfortably above threshold without being tuned to the content.
On failure: Screenshot the full page too and compare: if the page shows
content but the canvas crop is black, the draw loop is not producing pixels
(check Steps 3 and 4 first). If a legitimately sparse scene sits under 1%,
raise settle so more of the scene accumulates, or lower --lit-percent-min
deliberately and note why.
Apps that honor prefers-reduced-motion must show a calm, settled state.
Verify it in a separate fresh context — never by toggling emulation on the
already-running page.
context = browser.new_context(viewport={"width": 1280, "height": 900},
reduced_motion="reduce")
The packaged script re-runs all previous assertions in this context, then takes two canvas screenshots one second apart and requires them near-identical (<= 2% of pixels changed) — a still pose, not a paused-by-accident one.
Expected: Reduced-motion context passes the same luminance/visibility checks and the two screenshots differ by <= 2% of pixels.
On failure: A large diff means animation continues despite
prefers-reduced-motion — check that the app queries the media feature and
that the settled pose is reachable without animation. If the app deliberately
keeps subtle motion, raise --still-max-changed-percent and document the
decision.
Pixels can look right while the console reports a degraded path. Collect
console and pageerror events for the whole run and match them against
error keywords.
gpu_errors = [line for line in logs
if any(keyword in line.lower()
for keyword in ["gpgpu", "webgl error", "nan", "fallback"])]
Expected: Zero matching lines. The packaged script appends any hits to the
failure list and writes the full log to <out>/console.log.
On failure: Read the matched lines in console.log. nan in a shader or
simulation log means numeric blow-up even if this frame looked fine;
fallback means Step 3's guarantee was violated at app level. Substring
matches on innocent words (e.g. "nan" inside a longer token) are tuned away
with explicit --console-error-keyword flags.
pip install playwright pillow
python -m playwright install chromium
python3 skills/verify-web-app-runtime/scripts/verify_runtime.py \
--url http://localhost:5173/myapp/ \
--steps steps.json \
--out /tmp/verify-out
Expected: Per-context summary lines, FAILURES: none, exit code 0.
/tmp/verify-out/ contains norm_page.png, norm_canvas.png, rm_page.png,
rm_canvas.png, rm_canvas_still.png, console.log, and report.json. The
origin run against a healthy build reported: half-float present, mode active,
visibility visible, ~9% pixels lit, zero GPGPU errors.
On failure: The exit code is 1 and every failed assertion is listed —
work through them in procedure order (flags before probe, probe before
pixels), since early failures cause misleading later ones. report.json
holds the raw evidence for each context.
report.json shows "webgl2": true and "halfFloat": true
(or the run explicitly used --skip-half-float)visibilityState is visible in both contextsconsole.log contains no lines matching the error keywordsexpect(canvas).toBeVisible() passes on a
pitch-black canvas. Only the luminance assertion (Step 6) proves rendering.
Know its converse limit too: an undrawn canvas composites as transparent,
so over a bright page background it screenshots as fully lit — which is why
Validation requires demonstrating the verifier can also fail.null — and the resulting black canvas is
indistinguishable from an app bug. Rule the harness out first.settle values, not fixed global sleeps.AudioContext stays suspended until
a user gesture; drive the UI with real Playwright clicks (Step 5) before
asserting anything audio-dependent.nan keyword substring-matches
innocent tokens. Tune with --console-error-keyword instead of ignoring
console failures wholesale.This skill is a core skill of the frontend-runtime-verifier agent and is
exercised by the visual-pr-review team. See
references/EXAMPLES.md for the full origin recipe,
the steps-DSL reference, a no-GPGPU variant, and a CI integration example.