ClawHub
Back to skill

Security audit

Ab Test Planner

Security checks for vulnerabilities and agentic risk

Overview

This skill is a planning checklist for A/B tests and does not request code execution, credentials, persistence, or system-changing authority.

This appears safe to install as a planning aid. Users should still treat its sample-size table as rough guidance and verify calculations with a statistics tool before making business decisions, especially for pricing or revenue-impacting experiments.

A.I.G

Vulnerability Patterns
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs

SkillSpector

Vulnerability Patterns
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
Findings (1)

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
75% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · SKILL.md (reported line 125)May include surrounding context.

md
| Dimension | 0 | 5 | 10 |
|---|---|---|---|
| Statistical rigour | No sample size, or a number with no stated baseline/MDE behind it | Sample size present but MDE is guessed or copied from the lookup table without checking the actual baseline; power/significance unstated | Sample size derived from the stated baseline and MDE at 80% power / 95% confidence, duration checked against real daily traffic and the 2–4 week window, and the low-traffic escape hatch invoked if it doesn't fit |
| Hypothesis discipline | "Let's see what happens" — no direction, no magnitude, or multiple changes bundled into one variant | Directional hypothesis but missing magnitude, segment, or the evidence-based *because*; variant purity not confirmed | Full template filled (change, metric, direction, magnitude, segment, rationale citing data), and the treatment isolates exactly one change with excluded ideas named as follow-up tests |
| Guardrails & rollback | No guardrail metrics, or a rollback line with no threshold | Guardrails named but denominators/definitions ambiguous; rollback trigger vague ("if things look bad") | 1–2 guardrails protecting revenue or core engagement with pre-agreed definitions, concrete rollback thresholds, and the peeking-vs-harm-monitoring distinction handled explicitly |
| Decision readiness | No interpretation guide; results will be argued about after the fact | Ship/iterate/reject listed but thresholds fuzzy; inconclusive outcome missing or treated as a soft win | All four outcomes (ship / iterate / reject / inconclusive) mapped to pre-committed thresholds, including what an inconclusive result costs and what each outcome changes next |

Static analysis

No suspicious patterns detected.