Ab Testing

Experiment design, metrics, ethics, and analysis. Use when running product experiments responsibly.

AB Testing Skill

This skill provides structured guidance for AB Testing work. Act as an active guide: confirm triggers, propose the stages below, and adapt if the user wants a lighter pass.

When to Offer This Workflow

Trigger conditions:

• User mentions ab testing or closely related work
• They want a structured workflow rather than ad-hoc tips
• They are preparing a review, rollout, or stakeholder communication

Initial offer: Explain the four stages briefly and ask whether to follow this workflow or work freeform. If they decline, continue in their preferred style.

Workflow Stages

Stage 1: Clarify context & goals

Anchor on hypothesis and primary metric. Ask what success looks like, constraints, and what must not break. Capture unknowns early.

Stage 2: Design or plan the approach

Translate goals into a concrete plan around randomization and segments. Compare alternatives and explicit trade-offs; avoid implicit assumptions.

Stage 3: Implement, validate, and harden

Execute with verification loops tied to ethics and guardrails. Prefer small steps, measurable checks, and rollback points where risk is high.

Stage 4: Operate, communicate, and iterate

Close the loop with analysis and decision rules: monitoring, documentation, stakeholder updates, and lessons learned for the next cycle.

Checklist Before Completion

• Goals and constraints are explicit for AB Testing Skill
• Risks and trade-offs are stated, not hand-waved
• Verification steps match the change’s impact (tests, canary, peer review)
• Operational follow-through is covered (monitoring, docs, owners)

Tips for Effective Guidance

• Be procedural: stage-by-stage, with clear exit criteria
• Ask for missing context (environment, scale, deadlines) before prescribing
• Prefer checklists and concrete examples over generic platitudes
• If the user declines the workflow, switch to freeform help without lecturing

Handling Deviations

• If the user wants to skip a stage: confirm and continue with what they need.
• If context is missing: ask targeted questions before strong recommendations.
• Prefer concrete examples, trade-offs, and verification steps over generic advice.

Quality Bar

• Each recommendation should be actionable (what to do next).
• Call out failure modes relevant to AB Testing (security, scale, UX, or ops).
• Keep tone direct and respectful of the user’s time.