GitHub Actions Step Flake Audit

Detect flaky GitHub Actions job steps by finding mixed success/failure conclusions across runs.

Security Scan

The skill is internally coherent for analyzing local GitHub Actions run JSON files; it processes local artifacts with no hidden network calls or credential requests, though the README example references the gh CLI even though the skill doesn't require it.

Purpose & Capability

The name/description match the implementation: the script aggregates step outcomes from GitHub Actions run JSON files and scores flaky steps. Small mismatch: SKILL.md shows how to obtain JSON with `gh run view`, but the skill does not declare `gh` as a required binary or try to call `gh` itself — it only reads pre-collected JSON files. This is a minor documentation vs. requirement inconsistency, not a functional problem.

Instruction Scope

Runtime instructions and the script operate on local JSON files (RUN_GLOB) and only inspect fields in those files. The script reads, parses, filters, aggregates, and prints results or exits non-zero when FAIL_ON_CRITICAL is set. It does not attempt to read other system files, access network endpoints, or export data externally.

Install Mechanism

There is no install spec (instruction-only plus a bundled script), so nothing is downloaded or installed by the skill. Required binaries are minimal (bash, python3) and match the provided script.

Credentials

The skill requests no environment variables or credentials. It accepts many optional runtime variables (regex filters, thresholds) which are reasonable for its purpose. There are no secret-like env vars declared or accessed.

Persistence & Privilege

The skill does not request persistent presence (always:false) and does not modify agent system configuration. It runs as a one-off script operating on files the user supplies.

Assessment

This skill analyzes local GitHub Actions run JSON exports and does not send data out or require credentials. Before running: 1) ensure RUN_GLOB points only to intended artifact files (avoid broad globs that might match sensitive JSON), 2) if you want to collect runs using the `gh` example in SKILL.md, run that yourself — the skill does not call `gh` and does not declare it as a dependency, and 3) inspect any real run JSONs to confirm they don't contain sensitive secrets you don't want processed or stored. Otherwise the tool appears coherent and appropriate for its stated purpose.

Use this skill to catch flaky CI steps that alternate between passing and failing across workflow runs.

What this skill does

• Reads GitHub Actions run JSON exports (gh run view --json ...)
• Groups step outcomes by repository + workflow + job + step name
• Scores each step for flake risk when both success and failure outcomes are present
• Ranks the most unstable steps by failure rate and failed-run volume
• Supports text/json outputs and optional fail gate for CI enforcement

Inputs

Optional:

• RUN_GLOB (default: artifacts/github-actions/*.json)
• TOP_N (default: 20)
• OUTPUT_FORMAT (text or json, default: text)
• MIN_OCCURRENCES (default: 3) — minimum observed step runs before scoring
• WARN_FAILURE_RATE (default: 0.20) — flaky failure-rate threshold
• CRITICAL_FAILURE_RATE (default: 0.40) — critical flaky failure-rate threshold
• FAIL_ON_CRITICAL (0 or 1, default: 0)
• REPO_MATCH / REPO_EXCLUDE (regex, optional)
• WORKFLOW_MATCH / WORKFLOW_EXCLUDE (regex, optional)
• JOB_MATCH / JOB_EXCLUDE (regex, optional)
• STEP_MATCH / STEP_EXCLUDE (regex, optional)

Collect run JSON

gh run view <run-id> --json databaseId,workflowName,headBranch,headSha,url,repository,jobs \
  > artifacts/github-actions/run-<run-id>.json

Run

Text report:

RUN_GLOB='artifacts/github-actions/*.json' \
MIN_OCCURRENCES=5 \
WARN_FAILURE_RATE=0.15 \
CRITICAL_FAILURE_RATE=0.35 \
bash skills/github-actions-step-flake-audit/scripts/step-flake-audit.sh

JSON output + fail gate:

RUN_GLOB='artifacts/github-actions/*.json' \
OUTPUT_FORMAT=json \
FAIL_ON_CRITICAL=1 \
bash skills/github-actions-step-flake-audit/scripts/step-flake-audit.sh

Run against bundled fixtures:

RUN_GLOB='skills/github-actions-step-flake-audit/fixtures/*.json' \
bash skills/github-actions-step-flake-audit/scripts/step-flake-audit.sh

Output contract

• Exit 0 in reporting mode (default)
• Exit 1 when FAIL_ON_CRITICAL=1 and one or more flaky step groups are critical
• Text mode prints summary + top flaky steps
• JSON mode prints summary + ranked groups + critical groups