Flaky Test Detective

• Outcome: Pass
• Latest audit: Sep 16, 2026 · 7:50 AM UTC
• Version: 1.0.1

Overview

This skill is a straightforward flaky-test troubleshooting guide with one limited shell-script hygiene issue users should avoid copying unchanged.

Installers should treat this as a normal testing-assistance skill. Before running the multi-run shell snippet on a shared machine, replace the fixed /tmp paths with a private mktemp directory and avoid running the workflow with elevated privileges.

A.I.G

Vulnerability Patterns

• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally

Findings (1)

T09 · Insecure Skill Coding Practices

Warning

• Location: SKILL.md:42

Finding

Predictable Temporary Files Permit Symlink-Based File Overwrite

Content

View full analysis

Vulnerability Details

File Location: SKILL.md, lines 42–65
Vulnerability Type: Predictable temporary files and unsafe file creation
Risk Level: Medium

Vulnerable Code

bash

# Run tests N times and collect results
RESULTS_FILE="/tmp/flaky-results.json"
echo '[]' > "$RESULTS_FILE"

for i in $(seq 1 5); do
  echo "=== Run $i/5 ==="
  # Capture per-test results (adjust for your framework)
  npm test -- --json 2>/dev/null | python3 -c "
import json, sys
try:
    data = json.load(sys.stdin)
    for suite in data.get('testResults', []):
        for test in suite.get('testResults', []):
            print(f'{test[\"status\"]}\t{test[\"fullName\"]}')
except: pass
" >> "/tmp/run-$i.txt"
done

# Find tests with inconsistent results across runs
python3 -c "
import os, collections
results = collections.defaultdict(list)
for i in range(1, 6):
    path = f'/tmp/run-{i}.txt'

Technical Analysis

The Skill instructs the user or Agent to create and append to predictable filenames in the shared /tmp directory. Shell redirections such as > and >> follow symbolic links. The commands neither create files exclusively nor verify file ownership, type, or permissions before writing.

A local attacker who can write to /tmp can pre-create /tmp/flaky-results.json or any /tmp/run-N.txt path as a symbolic link to another file writable by the victim. When the Skill runs, echo '[]' > "$RESULTS_FILE" truncates the symlink target, while the test-result command appends output through /tmp/run-N.txt.

The RESULTS_FILE variable is initialized but not used afterward, making that particular shared temporary-file write unnecessary for the declared workflow. The test result files are necessary for analysis, but they do not need globally predictable paths.

Attack Path

1. The attacker has local access to the same system and permission to create entries in the shared /tmp directory. ...[truncated 1537 chars]

Remediation

View remediation

Remediation Suggestions

Create a private, randomly named temporary directory and store all result files within it:

bash

RESULTS_DIR="$(mktemp -d)" || exit 1
chmod 700 "$RESULTS_DIR"
trap 'rm -rf -- "$RESULTS_DIR"' EXIT

for i in $(seq 1 5); do
  RUN_FILE="$RESULTS_DIR/run-$i.txt"
  npm test -- --json 2>/dev/null | python3 -c '
import json, sys
try:
    data = json.load(sys.stdin)
    for suite in data.get("testResults", []):
        for test in suite.get("testResults", []):
            print(f"{test[\"status\"]}\t{test[\"fullName\"]}")
except Exception:
    pass
' >> "$RUN_FILE"
done

Update the analysis code to read from RESULTS_DIR, preferably by passing the directory as a command-line argument or environment variable rather than embedding a fixed path.

Additional hardening measures:

• Remove the unused /tmp/flaky-results.json initialization.
• Set a restrictive umask, such as umask 077, before creating temporary artifacts.
• Do not use fixed filenames directly under shared temporary directories.
• Ensure cleanup is registered immediately after successful temporary-directory creation.
• Avoid executing the workflow with elevated privileges.
• If individual files must be created outside a private directory, use exclusive creation and reject symbolic links rather than relying on ordinary shell redirection.

SkillSpector

Vulnerability Patterns

• Memory Poisoning: Persistent Context Injection, Context Window Stuffing, Memory Manipulation
• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands
• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access
• Supply Chain: Unpinned Dependencies, External Script Fetching, Obfuscated Code

Findings (1)

Memory Manipulation

High

• Category: Memory Poisoning
• Confidence: 80% confidence

Finding

Skill manipulates agent memory, state, or stored context. Memory corruption can alter personality, override safety rules, or cause unpredictable behavior.

Content

Scanner excerpt · SKILL.md (reported line 153)May include surrounding context.

md

- **Timing:** Replace fixed delays with polling/retry assertions
- **Shared state:** Add setup/teardown, use test isolation
- **Order dependency:** Make tests independent, reset state
- **Network:** Mock external calls, use test fixtures
- **Environment:** Pin timezone, use deterministic dates, avoid temp paths