Back to skill

Security audit

OpenClaw Skill Tester

Security checks for vulnerabilities and agentic risk

Overview

This skill is presented as a testing framework, but its bundled tests are mostly simulated or fixed and can produce misleading reports.

Treat this as a demo or prototype, not a trustworthy testing or release gate. Before installing or using it in CI, require real skill execution, failure propagation, report totals derived from actual results, sanitized skill names, and a private report directory.

Vulnerability Patterns
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (2)

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/test_runner.sh:127
Finding

Path Traversal and Symlink-Based File Overwrite in Report Generation

Content
View full analysis

Vulnerability Details

File Location: scripts/test_runner.sh, lines 127-134
Vulnerability Type: Untrusted path construction and unsafe temporary-file handling
Risk Level: Medium

Vulnerable Code

bash
generate_report() {
    local skill=$1
    local timestamp=$(date -u +"%Y-%m-%dT%H:%M:%S")
    local report_dir="/tmp/skill-tests"
    local report_file="$report_dir/${skill}_test_report_$(date +%Y%m%d_%H%M%S).json"

    mkdir -p "$report_dir"

    cat > "$report_file" <<EOF

The skill value originates from the user-controlled --skill argument:

bash
--skill)
    SKILL_NAME="$2"
    shift 2
    ;;

Technical Analysis

The script directly embeds the user-controlled Skill name in report_file without validating it as a simple directory name. Shell quoting prevents command injection, but it does not prevent path traversal. A value containing / or .. can cause the normalized report path to escape /tmp/skill-tests.

The report directory is also a predictable, shared location under /tmp. The generated filename uses a timestamp with one-second precision and is opened with a normal shell redirection. No exclusive file creation, ownership check, or symlink protection is performed. A local attacker able to write to the report directory may therefore pre-create a symbolic link at a predicted report path.

Attack Path

Path traversal scenario:

  1. An attacker invokes the runner with a crafted Skill value containing traversal components, such as ../../target.
  2. The test commands may fail because the Skill does not exist, but their failures are suppressed with || true.
  3. generate_report constructs a path similar to: /tmp/skill-tests/../../target_test_report_TIMESTAMP.json.
  4. The operating system resolves the traversal components and creates or truncates the resulting file outside the intended report directory.
  5. The report content is written wit ...[truncated 1049 chars]
Remediation
View remediation

Remediation Suggestions

  1. Validate the Skill name before using it in a path. Permit only a conservative set of characters:

    bash
    if [[ ! "$skill" =~ ^[A-Za-z0-9._-]+$ ]] || [[ "$skill" == "." || "$skill" == ".." ]]; then
        print_error "Invalid skill name"
        return 1
    fi
    
  2. Create a private report directory owned by the current user and restrict its permissions:

    bash
    report_dir="${SKILL_TESTER_REPORT_DIR:-$HOME/.local/state/skill-tests}"
    install -d -m 0700 -- "$report_dir"
    
  3. Resolve the final path and verify that it remains beneath the expected report directory before writing.

  4. Create report files atomically and without following existing symbolic links. Use a securely created temporary file in the destination directory, set restrictive permissions, and atomically rename it after writing.

  5. Refuse to overwrite existing files and verify that the destination is a regular file owned by the expected user.

  6. Avoid predictable shared /tmp paths. If temporary storage is required, use mktemp -d, register a cleanup trap, and restrict the directory to mode 0700.

other

Warning
Location
scripts/test_runner.sh:76
Finding

Suppressed Test Failures and Fabricated Security-Relevant Reports

Content
View full analysis

Vulnerability Details

File Location: scripts/test_runner.sh, lines 76-153
Vulnerability Type: Test result integrity failure
Risk Level: Medium

Vulnerable Code

Every test failure is explicitly converted into success:

bash
python3 "$SKILL_TESTER_DIR/test_trigger.py" \
    --skill "$skill" \
    --input "监控股票价格" \
    --expected true \
    ${VERBOSE:+--verbose} || true

python3 "$SKILL_TESTER_DIR/test_trigger.py" \
    --skill "$skill" \
    --input "查看A股" \
    --expected true \
    ${VERBOSE:+--verbose} || true

python3 "$SKILL_TESTER_DIR/test_trigger.py" \
    --skill "$skill" \
    --input "随便聊聊" \
    --expected false \
    ${VERBOSE:+--verbose} || true

python3 "$SKILL_TESTER_DIR/test_trigger.py" \
    --skill "$skill" \
    --input "今天天气" \
    --expected false \
    ${VERBOSE:+--verbose} || true

The same suppression is used for functionality and comparison tests:

bash
python3 "$SKILL_TESTER_DIR/test_functionality.py" \
    --skill "$skill" \
    --all-cases \
    ${VERBOSE:+--verbose} || true
bash
python3 "$SKILL_TESTER_DIR/test_comparison.py" \
    --skill "$skill" \
    --baseline "no-skill" \
    --metric all \
    --iterations "$ITERATIONS" \
    ${VERBOSE:+--verbose} || true

The generated report is composed of fixed values unrelated to the preceding test results:

bash
cat > "$report_file" <<EOF
{
  "skill_name": "$skill",
  "timestamp": "$timestamp",
  "test_summary": {
    "total": 20,
    "passed": 18,
    "failed": 2,
    "skipped": 0
  },
  "trigger_tests": {
    "passed": 10,
    "failed": 0
  },
  "functionality_tests": {
    "passed": 8,
    "failed": 2
  },
  "comparison_metrics": {
    "tool_calls_reduction": "45%",
    "token_savings": "32%",
    "response_time_improvement": "28%"
  }
}
EOF

The underlying functionality test also validates simulat ...[truncated 3076 chars]

Remediation
View remediation

Remediation Suggestions

  1. Remove || true from required tests and preserve each command's exit status.

  2. Collect structured results from every test, then calculate totals from those results instead of using fixed values.

  3. Return a nonzero exit status when any required test fails, cannot run, or lacks required fixtures.

  4. Replace generate_simulated_output with an isolated invocation of the selected Skill. Apply explicit timeouts and capture standard output, standard error, exit status, and resource metrics.

  5. Clearly distinguish mock or demonstration mode from real validation. Mock results must be labeled as simulated and must not be accepted as release evidence.

  6. Generate reports from an in-memory result model populated during the current run. Include individual test names, actual outcomes, errors, and skipped-test reasons.

  7. Treat missing test cases as an error when tests are required rather than exiting successfully:

    python
    if not test_cases:
        print("No test cases found", file=sys.stderr)
        sys.exit(2)
    
  8. Add regression tests confirming that:

    • a failed child test makes the runner fail;
    • a missing Skill cannot produce a passing report;
    • report totals exactly match executed tests;
    • simulated metrics cannot appear in production-mode reports.
Vulnerability Patterns
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Findings (12)

Tp4

High
Category
MCP Tool Poisoning
Confidence
94% confidence
Finding

The skill is presented as an automated testing framework that validates trigger behavior, functionality, and performance, but the finding indicates it does not actually perform those checks and may rely on simulated or hardcoded output. This is dangerous because users could trust false test results, deploy unverified skills, and miss real security or reliability defects due to a misleading assurance mechanism.

Content

No source excerpt is available for this finding.

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
70% confidence
Finding

Without declared permissions the skill's intent is opaque and cannot be validated.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The manifest description and nearly all user-facing documentation are written in Chinese, with no indication that users may choose another language or that the skill is intentionally limited to a Chinese-speaking or region-specific context. Under the policy, language or locale constraints should be opt-in or clearly justified.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

The script advertises a comparison of skill performance but only reports hard-coded simulated numbers, so users may make decisions based on fabricated benchmark results. In a testing framework context, this is dangerous because it can silently misrepresent quality, regressions, or efficiency gains and undermine trust in validation outputs.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
98% confidence
Finding

The CLI accepts --skill and --baseline parameters but never uses them during execution, creating a deceptive interface that implies meaningful comparison while ignoring user input. This can mislead operators, mask testing gaps, and enable false claims about a skill's performance in CI or review workflows.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
98% confidence
Finding

The docstring and file header describe this as a functionality testing script that verifies skill correctness, and execute_functionality_test() says it executes functional tests. However, the implementation explicitly uses mocked behavior via generate_simulated_output() and comments state that actual skill invocation is not performed, so the documented intent contradicts the real behavior.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
90% confidence
Finding

The file includes a natural-language title and user-facing messaging in Chinese only, beginning with the description '统一测试执行器'. Under the policy, forcing a specific language without offering a user choice or documenting a justified locale constraint is a language/locale policy violation.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
98% confidence
Finding

The report generator writes hard-coded success/failure counts and performance metrics instead of using actual test outcomes. This can mislead users into trusting a skill that was never properly validated, weakening security review and deployment decisions; the risk is increased because earlier test commands are suffixed with '|| true', so failures are explicitly ignored while the report still presents favorable results.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

This Python file contains natural-language strings in Chinese for the module description and user-visible error/output messages, while the CLI argument descriptions are in English. That creates a locale-specific experience without any opt-in, fallback, or explanation that the tool is intentionally region-specific, which matches the language/locale policy violation criteria.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The inline comments at L23-L24 state that trigger words should be read from skill configuration and that the current logic is only an example. However, the function is used as the actual implementation for testing and returns hardcoded patterns for only two skill names, which contradicts the stated intent of loading a skill's trigger patterns.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
94% confidence
Finding

This code includes multiple user-facing strings in Chinese, including the module description and printed CLI output, without any opt-in or locale selection. That can violate a language/locale policy when skills are expected to avoid forcing a specific language unless documented or user-selectable.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
91% confidence
Finding

This code file contains user-facing messages and documentation in Chinese, such as the module description and printed status text, with no indication that the user can opt into another language. That can violate a language/locale policy when a skill implicitly forces one language for interaction.

Content

No source excerpt is available for this finding.