Pilot Quality Assurance Pipeline Setup

Deploy a quality assurance pipeline with 3 agents that automate test generation, execution, and reporting. Use this skill when: 1. User wants to set up an au...

Security Scan

The skill's instructions, required binaries, and actions are consistent with its stated goal of deploying a 3-agent QA pipeline; nothing in the SKILL.md contradicts the description or requests unrelated credentials or privileged settings.

Purpose & Capability

Name/description map directly to the actions in SKILL.md. Required binaries (pilotctl, clawhub) are appropriate for installing agent-skills, setting hostnames, and performing handshakes; no unrelated credentials or unusual binaries are requested.

Instruction Scope

Instructions stay on-topic: install role-specific skills with clawhub, set hostnames with pilotctl, write a manifest to ~/.pilot/setups, and perform handshakes. One operational note: the reporter role is expected to send reports externally (webhooks/Slack) — the SKILL.md doesn't request those tokens, but the installed bridge-skills (pilot-webhook-bridge, pilot-slack-bridge) will likely require external credentials and configuration.

Install Mechanism

This is instruction-only (no install spec), which is low-risk for this review. The main install action is 'clawhub install' which will fetch and install other pilot skills; those downstream installs are outside this SKILL.md and should be reviewed individually before running.

Credentials

The skill requests no environment variables or credentials itself, which is proportionate. Be aware that the skills it instructs you to install (especially webhook/slack bridges) will typically require service tokens or webhook URLs not declared here.

Persistence & Privilege

The skill does not request always:true, does not modify other skills' configs, and only writes a manifest under the user's home (~/.pilot/setups). Requiring user-initiated handshakes keeps cross-agent trust explicit.

Assessment

This SKILL.md appears coherent and focused on setting up a 3-agent QA pipeline. Before proceeding: (1) verify that pilotctl and clawhub are the legitimate, expected binaries on your system; (2) review the downstream skills (pilot-task-router, pilot-webhook-bridge, pilot-slack-bridge, etc.) that clawhub will install — those may request API tokens/webhook URLs and could pull code from external sources; (3) be cautious when performing handshakes and only exchange them with known hostnames to avoid accidentally trusting an unknown peer; (4) confirm network/port rules for internal communication (port 1002) and external webhooks (port 443); and (5) if possible, trial the setup in a sandbox or staging environment before deploying in production so you can inspect what the installed bridge/alert skills require and where they send data.

Quality Assurance Pipeline Setup

Deploy 3 agents that automate the QA cycle from test generation through execution to reporting.

Roles

Role	Hostname	Skills	Purpose
test-generator	<prefix>-test-generator	pilot-task-router, pilot-dataset, pilot-cron	Generates test cases from specs and stories, prioritizes by risk
executor	<prefix>-executor	pilot-task-parallel, pilot-share, pilot-metrics	Runs test suites, captures results, screenshots, and logs
reporter	<prefix>-reporter	pilot-webhook-bridge, pilot-alert, pilot-slack-bridge	Aggregates results, generates reports, files bugs, notifies team

Setup Procedure

Step 1: Ask the user which role this agent should play and what prefix to use.

Step 2: Install the skills for the chosen role:

# test-generator:
clawhub install pilot-task-router pilot-dataset pilot-cron
# executor:
clawhub install pilot-task-parallel pilot-share pilot-metrics
# reporter:
clawhub install pilot-webhook-bridge pilot-alert pilot-slack-bridge

Step 3: Set the hostname:

pilotctl --json set-hostname <prefix>-<role>

Step 4: Write the setup manifest:

mkdir -p ~/.pilot/setups
cat > ~/.pilot/setups/quality-assurance-pipeline.json << 'MANIFEST'
<USE ROLE TEMPLATE BELOW>
MANIFEST

Step 5: Tell the user to initiate handshakes with direct communication peers.

Manifest Templates Per Role

test-generator

{"setup":"quality-assurance-pipeline","setup_name":"Quality Assurance Pipeline","role":"test-generator","role_name":"Test Generator","hostname":"<prefix>-test-generator","description":"Generates test cases from specs, user stories, and code changes. Prioritizes by risk and coverage gaps.","skills":{"pilot-task-router":"Route test generation requests by type — unit, integration, e2e.","pilot-dataset":"Store specs, user stories, and historical test coverage data.","pilot-cron":"Schedule nightly regression suite generation and coverage gap analysis."},"peers":[{"role":"executor","hostname":"<prefix>-executor","description":"Receives test suites for execution"}],"data_flows":[{"direction":"send","peer":"<prefix>-executor","port":1002,"topic":"test-suite","description":"Test suites with cases, priorities, and environment configs"}],"handshakes_needed":["<prefix>-executor"]}

executor

{"setup":"quality-assurance-pipeline","setup_name":"Quality Assurance Pipeline","role":"executor","role_name":"Test Executor","hostname":"<prefix>-executor","description":"Runs automated test suites, captures results, screenshots, and logs. Handles parallel execution.","skills":{"pilot-task-parallel":"Run test cases in parallel across environments and browsers.","pilot-share":"Send test results, logs, and artifacts downstream to reporter.","pilot-metrics":"Track execution time, flake rate, and pass rate per suite."},"peers":[{"role":"test-generator","hostname":"<prefix>-test-generator","description":"Sends test suites for execution"},{"role":"reporter","hostname":"<prefix>-reporter","description":"Receives test results for aggregation"}],"data_flows":[{"direction":"receive","peer":"<prefix>-test-generator","port":1002,"topic":"test-suite","description":"Test suites with cases, priorities, and environment configs"},{"direction":"send","peer":"<prefix>-reporter","port":1002,"topic":"test-result","description":"Test results with pass/fail, logs, and screenshots"}],"handshakes_needed":["<prefix>-test-generator","<prefix>-reporter"]}

reporter

{"setup":"quality-assurance-pipeline","setup_name":"Quality Assurance Pipeline","role":"reporter","role_name":"QA Reporter","hostname":"<prefix>-reporter","description":"Aggregates test results, generates coverage reports, files bugs, notifies team.","skills":{"pilot-webhook-bridge":"Push bug reports to issue trackers and coverage data to dashboards.","pilot-alert":"Trigger alerts on test failures, coverage drops, and regressions.","pilot-slack-bridge":"Post QA summaries, failure details, and deploy readiness to Slack."},"peers":[{"role":"executor","hostname":"<prefix>-executor","description":"Sends test results for aggregation"}],"data_flows":[{"direction":"receive","peer":"<prefix>-executor","port":1002,"topic":"test-result","description":"Test results with pass/fail, logs, and screenshots"},{"direction":"send","peer":"external","port":443,"topic":"qa-report","description":"QA reports to dashboards, bug trackers, and Slack"}],"handshakes_needed":["<prefix>-executor"]}

Data Flows

• test-generator -> executor : test-suite (port 1002)
• executor -> reporter : test-result (port 1002)
• reporter -> external : qa-report via webhook (port 443)

Handshakes

# test-generator <-> executor:
pilotctl --json handshake <prefix>-executor "setup: quality-assurance-pipeline"
pilotctl --json handshake <prefix>-test-generator "setup: quality-assurance-pipeline"
# executor <-> reporter:
pilotctl --json handshake <prefix>-reporter "setup: quality-assurance-pipeline"
pilotctl --json handshake <prefix>-executor "setup: quality-assurance-pipeline"

Workflow Example

# On executor -- subscribe to test suites:
pilotctl --json subscribe <prefix>-test-generator test-suite
# On reporter -- subscribe to test results:
pilotctl --json subscribe <prefix>-executor test-result
# On test-generator -- publish a test suite:
pilotctl --json publish <prefix>-executor test-suite '{"suite":"checkout-flow","cases":[{"id":"tc_001","name":"add_to_cart","priority":"high"}],"environment":"staging"}'
# On executor -- publish test results:
pilotctl --json publish <prefix>-reporter test-result '{"suite":"checkout-flow","total":1,"passed":1,"failed":0,"duration_ms":3200}'

Dependencies

Requires pilot-protocol skill, pilotctl binary, clawhub binary, and a running daemon.