Flaky Test Detective

• Outcome: Pass
• Latest audit: Sep 16, 2026 · 7:50 AM UTC
• Version: 1.0.1

Overview

This skill is a straightforward flaky-test troubleshooting guide with one limited shell-script hygiene issue users should avoid copying unchanged.

Installers should treat this as a normal testing-assistance skill. Before running the multi-run shell snippet on a shared machine, replace the fixed /tmp paths with a private mktemp directory and avoid running the workflow with elevated privileges.

A.I.G

Vulnerability Patterns

• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally

Findings (1)

T09 · Insecure Skill Coding Practices

Warning

• Location: SKILL.md:42
• Finding: Predictable Temporary Files Permit Symlink-Based File Overwrite## Vulnerability Details **File Location**: `SKILL.md`, lines 42–65 **Vulnerability Type**: Predictable temporary files and unsafe file creation **Risk Level**: Medium ### Vulnerable Code ```bash # Run tests N times and collect results RESULTS_FILE="/tmp/flaky-results.json" echo '[]' > "$RESULTS_FILE" for i in $(seq 1 5); do echo "=== Run $i/5 ===" # Capture per-test results (adjust for your framework) npm test -- --json 2>/dev/null | python3 -c " import json, sys try: data = json.load(sys.stdin) for suite in data.get('testResults', []): for test in suite.get('testResults', []): print(f'{test[\"status\"]}\t{test[\"fullName\"]}') except: pass " >> "/tmp/run-$i.txt" done # Find tests with inconsistent results across runs python3 -c " import os, collections results = collections.defaultdict(list) for i in range(1, 6): path = f'/tmp/run-{i}.txt' ``` ### Technical Analysis The Skill instructs the user or Agent to create and append to predictable filenames in the shared `/tmp` directory. Shell redirections such as `>` and `>>` follow symbolic links. The commands neither create files exclusively nor verify file ownership, type, or permissions before writing. A local attacker who can write to `/tmp` can pre-create `/tmp/flaky-results.json` or any `/tmp/run-N.txt` path as a symbolic link to another file writable by the victim. When the Skill runs, `echo '[]' > "$RESULTS_FILE"` truncates the symlink target, while the test-result command appends output through `/tmp/run-N.txt`. The `RESULTS_FILE` variable is initialized but not used afterward, making that particular shared temporary-file write unnecessary for the declared workflow. The test result files are necessary for analysis, but they do not need globally predictable paths. ### Attack Path 1. The attacker has local access to the same system and permission to create entries in the shared `/tmp` directory. ...[truncated 1537 chars]
• Remediation: ## Remediation Suggestions Create a private, randomly named temporary directory and store all result files within it: ```bash RESULTS_DIR="$(mktemp -d)" || exit 1 chmod 700 "$RESULTS_DIR" trap 'rm -rf -- "$RESULTS_DIR"' EXIT for i in $(seq 1 5); do RUN_FILE="$RESULTS_DIR/run-$i.txt" npm test -- --json 2>/dev/null | python3 -c ' import json, sys try: data = json.load(sys.stdin) for suite in data.get("testResults", []): for test in suite.get("testResults", []): print(f"{test[\"status\"]}\t{test[\"fullName\"]}") except Exception: pass ' >> "$RUN_FILE" done ``` Update the analysis code to read from `RESULTS_DIR`, preferably by passing the directory as a command-line argument or environment variable rather than embedding a fixed path. Additional hardening measures: - Remove the unused `/tmp/flaky-results.json` initialization. - Set a restrictive umask, such as `umask 077`, before creating temporary artifacts. - Do not use fixed filenames directly under shared temporary directories. - Ensure cleanup is registered immediately after successful temporary-directory creation. - Avoid executing the workflow with elevated privileges. - If individual files must be created outside a private directory, use exclusive creation and reject symbolic links rather than relying on ordinary shell redirection.

SkillSpector

Vulnerability Patterns

• Memory Poisoning: Persistent Context Injection, Context Window Stuffing, Memory Manipulation
• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands
• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access
• Supply Chain: Unpinned Dependencies, External Script Fetching, Obfuscated Code

Findings (1)

Memory Manipulation

High

• Category: Memory Poisoning

Content

- **Timing:** Replace fixed delays with polling/retry assertions
- **Shared state:** Add setup/teardown, use test isolation
- **Order dependency:** Make tests independent, reset state
- **Network:** Mock external calls, use test fixtures
- **Environment:** Pin timezone, use deterministic dates, avoid temp paths

• Confidence: 80% confidence
• Finding: Skill manipulates agent memory, state, or stored context. Memory corruption can alter personality, override safety rules, or cause unpredictable behavior.

Static analysis

No suspicious patterns detected.