Visual Qa

Perform pixel-level visual regression testing on web apps by capturing, comparing screenshots, and gating deployments based on configurable similarity thresh...

Security Scan

The skill's code and instructions match its stated purpose (visual regression testing); no unexplained credentials, hidden endpoints, or obfuscated code were found, though it will run local processes and user-supplied server commands which you should treat as untrusted input.

Purpose & Capability

Name/description match the included scripts (capture.py, diff.py, gate.py). Required capabilities (Playwright, Pillow) and behavior (start local server, capture screenshots, pixel diffs) are consistent with a visual QA tool.

Instruction Scope

Instructions stay within the visual testing domain (reading config, capturing pages, comparing images). The scripts will start local servers, open network connections to localhost ports, read/write project files (baseline/current/diff dirs), and execute the user-supplied 'server' command. Running arbitrary server commands is required for the use case but is a potential danger if the config/command comes from an untrusted source.

Install Mechanism

No automated install spec; this is instruction-only and asks the user to pip install Playwright and Pillow and to run Playwright's browser install. No downloads or archive extraction are performed by the skill itself.

Credentials

Skill requests no environment variables or credentials. It does set PORT in the environment when launching a dev server (expected), but it does not read secrets or external credentials. The only notable point: the 'server' argument is executed via shell (capture.start_server uses shell=True), so untrusted server strings could run arbitrary commands—this is functional but a security consideration.

Persistence & Privilege

Skill does not request permanent presence, does not modify other skills, and uses only per-run temporary directories (gate.py uses TemporaryDirectory). It writes baseline/current/diff files into project paths when requested (expected for this tool).

Assessment

This skill appears to do what it claims: capture screenshots with Playwright, compare them with Pillow, and gate on similarity. Before installing or running it: (1) ensure Playwright/pillow are installed and you have Chromium available; (2) review any .visual-qa.json or CLI 'server' command you pass—the tool will execute that command on your machine (capture.start_server uses shell execution), so do not run untrusted configurations; (3) run in an isolated environment or CI runner with limited privileges if you introduce server commands from third parties; (4) note minor platform differences (uses 'cp' for copying diffs) and that the tool will bind/connect to localhost ports to start/verify the app.

A visual regression testing pipeline for web applications. Capture baseline screenshots of your app, compare new screenshots against baselines using pixel-level diffing, and pass/fail based on configurable similarity thresholds.

When to Use

• Pre-deploy visual verification — gate merges/deploys until UI changes are approved
• Regression testing — catch unintended visual changes in CI
• UI review workflow — generate diff images for design/code review
• Cross-browser/viewport testing — verify responsive layouts
• Component library QA — ensure design system changes don't break consumers

Quick Start

1. Capture Baselines

# Capture single URL
python scripts/capture.py http://localhost:3000 --output .visual-qa/baselines

# Capture multiple viewports
python scripts/capture.py http://localhost:3000 --viewports desktop mobile tablet --output .visual-qa/baselines

# Capture multiple pages from config
python scripts/capture.py --config .visual-qa.json --output .visual-qa/baselines

2. Compare Against Baselines

# Compare new screenshots
python scripts/diff.py --baseline .visual-qa/baselines --current .visual-qa/current --threshold 99

3. All-in-One Gate

# Capture + diff in one command
python scripts/gate.py --baseline .visual-qa/baselines --url http://localhost:3000 --threshold 99

# With local dev server
python scripts/gate.py --baseline .visual-qa/baselines --server "npm run dev" --port 3000 --threshold 99

# Using config file
python scripts/gate.py --config .visual-qa.json

Config File Pattern

Create .visual-qa.json in your project root:

{
  "urls": ["/", "/about", "/pricing"],
  "baseUrl": "http://localhost:3000",
  "viewports": ["desktop", "mobile"],
  "threshold": 99,
  "server": "npm run dev",
  "port": 3000,
  "baselineDir": ".visual-qa/baselines",
  "ignore": [".visual-qa/diffs", ".visual-qa/current"]
}

Scripts

All scripts support --help for detailed usage.

capture.py

Capture screenshots using Playwright (headless Chromium).

Features:

• Multiple viewport sizes: desktop (1280x800), tablet (768x1024), mobile (375x812)
• Waits for networkidle before capture
• Optional local server start/stop
• Configurable output directory
• Descriptive filenames: {url-slug}_{viewport}.png

Usage:

python scripts/capture.py <url> --output <dir> [options]
python scripts/capture.py --config <config.json> --output <dir>

diff.py

Compare screenshots using pixel-level diffing (Pillow).

Features:

• Pixel-by-pixel comparison
• Diff images with red/magenta overlay highlighting changes
• Similarity percentage per image pair
• Pass/fail based on threshold (default 99%)
• Summary report with pass/fail status
• Saves diff images to output directory

Usage:

python scripts/diff.py --baseline <dir> --current <dir> --output <diff-dir> --threshold <percent>

gate.py

All-in-one gate: capture + diff in a single command.

Features:

• Combines capture and diff steps
• Starts/stops local server automatically if needed
• Returns exit code 0 (pass) or 1 (fail)
• Human-readable summary output
• Can use config file or CLI args

Usage:

python scripts/gate.py --baseline <dir> --url <url> --threshold <percent>
python scripts/gate.py --baseline <dir> --server <command> --port <port> --threshold <percent>
python scripts/gate.py --config <config.json>

Workflow Examples

Initial Baseline Capture

# Start your app
npm run dev

# Capture baselines (desktop + mobile)
python scripts/capture.py http://localhost:3000 --viewports desktop mobile --output .visual-qa/baselines

CI/CD Integration

# In your CI pipeline after build
python scripts/gate.py --baseline .visual-qa/baselines --server "npm start" --port 3000 --threshold 99

# Exit code 0 = pass, 1 = fail
if [ $? -eq 0 ]; then
  echo "Visual QA passed ✓"
else
  echo "Visual QA failed ✗"
  exit 1
fi

Review Workflow

# 1. Developer makes UI changes
# 2. Capture new screenshots
python scripts/capture.py http://localhost:3000 --output .visual-qa/current

# 3. Generate diff images
python scripts/diff.py --baseline .visual-qa/baselines --current .visual-qa/current --output .visual-qa/diffs

# 4. Review diff images in .visual-qa/diffs/
# 5. If changes are intentional, update baselines:
rm -rf .visual-qa/baselines
mv .visual-qa/current .visual-qa/baselines

Multi-Page Testing

Create .visual-qa.json:

{
  "urls": ["/", "/products", "/about", "/contact"],
  "baseUrl": "http://localhost:3000",
  "viewports": ["desktop", "mobile"],
  "threshold": 99,
  "baselineDir": ".visual-qa/baselines"
}

# Capture all pages
python scripts/capture.py --config .visual-qa.json --output .visual-qa/baselines

# Gate all pages
python scripts/gate.py --config .visual-qa.json

Dependencies

Scripts require Playwright and Pillow:

pip install playwright pillow
python -m playwright install chromium

Scripts will check for dependencies and print install instructions if missing.

Thresholds

The --threshold parameter controls similarity percentage (0-100):

• 99% (default) — strict, catches most visual changes
• 95% — moderate, allows minor rendering differences (anti-aliasing, fonts)
• 90% — loose, allows more variation (use for dynamic content)

Experiment to find the right threshold for your app. Start strict (99%) and loosen if you get false positives.

Ignoring Dynamic Content

For pages with dynamic content (dates, user-specific data):

1. Use data attributes to hide dynamic elements during testing:

   [data-test-hide] { visibility: hidden !important; }
   

2. Capture specific viewport regions (future enhancement)

3. Loosen threshold for pages with acceptable dynamic content

Troubleshooting

"Command not found: python"

• Use python3 instead of python

"Playwright not installed"

• Run: pip install playwright && python -m playwright install chromium

"Similarity below threshold but images look the same"

• Font rendering, anti-aliasing, or sub-pixel differences. Lower threshold to 98-95%.

"Server not starting"

• Check that --port matches your server's port
• Ensure server command is correct (npm run dev, npm start, etc.)
• Increase wait time in gate.py (default 5s)

"Images not found"

• Check that baseline directory exists and contains PNGs
• Ensure current screenshots were captured to the correct directory
• Verify filenames match pattern: {url-slug}_{viewport}.png

Tips

• Commit baselines to Git so your team shares the same reference
• Add .visual-qa/diffs and .visual-qa/current to .gitignore
• Run in CI as a required check before merge
• Update baselines when intentional UI changes are made
• Use multiple viewports to catch responsive layout issues
• Test empty/error/loading states by capturing those URLs explicitly

Integration with Other Skills

• ux-qa-gate — Use visual-qa as part of the UX QA checklist
• webapp-testing — Combine with Playwright functional tests
• coding-agent — Sub-agents building UI must pass visual-qa before completion

---

For detailed script options, run:

python scripts/capture.py --help
python scripts/diff.py --help
python scripts/gate.py --help