Back to skill

Security audit

Runbook Automator

Security checks for vulnerabilities and agentic risk

Overview

This skill has a legitimate runbook-automation purpose but needs Review because it can turn operational documentation into live Bash playbooks without clear approval, validation, or containment.

Use this only with qualified operator review. Treat source runbooks as untrusted, inspect every generated command and rollback step, default to DRY_RUN=true, run with least-privileged credentials in a controlled environment, and verify any Slack webhook before sending incident details.

Vulnerability Patterns
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (1)

T09 · Insecure Skill Coding Practices

Error
Location
SKILL.md:18
Finding

Untrusted Runbook Content Can Be Converted into Executable Shell Commands

Content
View full analysis

Vulnerability Details

File Location: SKILL.md, lines 18-76 and 138-143
Vulnerability Type: Untrusted command generation and potential command injection
Risk Level: High

Vulnerable Code

markdown
### 1. `convert` — Parse Runbook and Generate Automation

#### Step 1: Identify Runbook Format

Read the input runbook (markdown, Confluence wiki, Google Doc, plain text) and extract:
- **Title and scope** — what incident does this address
- **Prerequisites** — access, tools, permissions needed
- **Steps** — ordered actions (distinguish manual vs automatable)
- **Decision points** — if/then branches
- **Verification steps** — how to confirm each step worked
- **Rollback steps** — how to undo if things go wrong
- **Escalation criteria** — when to page someone

#### Step 2: Classify Each Step

For each step in the runbook, classify as:

| Type | Example | Automation |
|------|---------|------------|
| **Command** | "Run `kubectl rollout restart`" | Direct script execution |
| **Check** | "Verify pods are running" | Script with assertion |
| **Decision** | "If error rate > 5%, proceed to step 4" | Conditional branch |
| **Manual** | "Call the database team" | Notification + pause |
| **Observation** | "Watch the dashboard for 10 minutes" | Timed wait + metric check |

#### Step 3: Generate Executable Playbook

```bash
# --- Step 1: [Name] ---
step_1() {
  log "Step 1: [description]"
  if [[ "$DRY_RUN" == "true" ]]; then
    log "DRY RUN: would execute [command]"
    return 0
  fi
  # [actual command]
  [command] || fail 1 "[error description]"
  # Verify
  [verification command] || fail 1 "Verification failed"
  log "Step 1: ✅ Complete"
}

3. test — Dry-Run a Generated Playbook

Execute the generated script with DRY_RUN=true:

  • Validate all commands exist in PATH
  • Check prerequisite access (can reach hosts, have credentials)
  • Verify notification h ...[truncated 3658 chars]
Remediation
View remediation

Remediation Suggestions

  1. Treat all source documents as untrusted input

    • Explicitly state that commands copied from runbooks must never be executed or emitted as trusted code without validation.
    • Record the source, owner, revision, and trust status of each imported runbook.
  2. Require command-by-command human approval

    • Present every generated command, verification command, and rollback command for review.
    • Require explicit confirmation before producing an executable artifact or performing a non-dry-run execution.
    • Highlight commands involving privilege changes, credentials, network transfers, destructive operations, or persistence.
  3. Use structured command representations

    • Represent each operation as an executable plus a separate argument array rather than interpolated shell text.
    • Avoid eval, bash -c, command substitution, and direct insertion of document content into shell scripts.
    • Generate code using safely quoted arguments and fixed templates.
  4. Enforce restrictive validation and allowlists

    • Allow only approved executables and documented argument patterns.
    • Reject or require elevated review for shell control operators, redirections, pipes, command substitutions, multiline commands, encoded payloads, and unexpected network destinations.
    • Validate resource names, hostnames, file paths, URLs, and metric values against strict schemas.
  5. Add mandatory policy checks

    • Scan generated playbooks for destructive commands, credential access, external uploads, persistence mechanisms, privilege escalation, and shell-injection patterns.
    • Fail closed when a command cannot be classified or safely parsed.
    • Require rollback and verification steps to pass the same security checks as primary commands.
  6. Execute in a least-privileged sandbox

    • Use an isolated container or dedicated runner with a read-only filesystem where possible.
    • Pro ...[truncated 756 chars]
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
Findings (4)

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The skill markets automated conversion of incident runbooks into executable playbooks but does not warn that generated scripts may restart services, fail over systems, or perform rollback actions. In this context, lack of safety framing increases the chance that users treat generated code as ready-to-run, which can cause outages or unintended destructive changes.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
90% confidence
Finding

The trigger phrases are broad enough that normal requests about scripts, wiki pages, or incident automation could invoke this skill unintentionally. Because the skill is designed to generate executable operational playbooks, accidental activation can lead to unsafe automation suggestions for sensitive infrastructure tasks.

Content

No source excerpt is available for this finding.

External Transmission

Medium
Category
Data Exfiltration
Confidence
88% confidence
Finding

The generated script includes a notification function that transmits runbook status messages to an external Slack webhook via curl. Even though notification hooks are a legitimate feature, sending operational context to an external endpoint can leak incident details, system names, or error information if webhook destinations are misconfigured, attacker-controlled, or insufficiently trusted.

Content

Scanner excerpt · SKILL.md (reported line 59)May include surrounding context.

md
notify() {
  log "NOTIFY: $1"
  if [[ -n "$SLACK_WEBHOOK" ]]; then
    curl -s -X POST "$SLACK_WEBHOOK" -H 'Content-Type: application/json' \
      -d "{\"text\": \"🔧 Runbook: $1\"}" > /dev/null
  fi
}

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
75% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · SKILL.md (reported line 124)May include surrounding context.

md
For each runbook, check:
- **Missing rollback steps** — what happens if step 3 fails?
- **No verification** — steps that say "do X" but never check if X worked
- **Stale commands** — references to deprecated tools, old hostnames, removed services
- **Missing decision criteria** — "if it's bad, escalate" (how bad? what metric?)
- **No estimated time** — SLA-critical runbooks need time bounds per step

Static analysis

No suspicious patterns detected.