Reliability Evidence Pack

• Outcome: Review
• Latest audit: Sep 11, 2026 · 3:43 AM UTC
• Version: 2.0.2

Overview

The skill has a coherent reliability-logging purpose, but several shipped paths can mislead validation results or expose sensitive artifacts and logs beyond what the documentation clearly discloses.

Install only after reviewing the scripts you plan to use. Prefer direct local scripts over the packaged CLI and GitHub Action until validation actually fails closed and package installation is pinned. Do not run the serve command on sensitive bundles unless it is bound to loopback and protected. Keep REP artifacts in an isolated directory, redact sensitive context/log lines, and define retention limits for cron-generated records.

A.I.G

Vulnerability Patterns

• Unauthorized Access and Privilege Escalation: Obtains permissions beyond the task's legitimate needs
• Tool Hijacking and Spoofing: Modifies or replaces tools so legitimate-looking calls execute attacker logic
• Insecure Dependencies: Introduces malicious components through unsafe dependency sources
• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads

Findings (5)

T07 · Tool Hijacking and Spoofing

Error

• Location: ', 'Output file', 'rep-report.html') .option('-f, --format <format>', 'Format: html, json, markdown', 'html') .option('-s, --summary', 'Show summary only') .action((options) => { console.log(chalk.blue('Generating report...')); console.log(chalk.green(`Format: ${options.format}`)); console.log(chalk.green(`Output: ${options.output}`)); if (options.summary) { console.log(chalk.gray('Mode: Summary only')); } console.log(chalk.bold.green('\n✓ Report generated successfully!') ...[truncated 1599 chars]:14
• Finding: CLI Spoofs Successful Operations Without Performing Them<![CDATA[ ## Vulnerability Details **File Location**: `cli/bin/cli.js:14-83` **Vulnerability Type**: Tool spoofing and false validation results **Risk Level**: Critical ### Vulnerable Code ```js program .command('init') .description('Initialize a new REP project') .option('-n, --name <name>', 'Project name', 'rep-project') .option('-t, --template <template>', 'Template to use', 'default') .action((options) => { console.log(chalk.blue('Initializing REP project...')); console.log(chalk.green(`Project name: ${options.name}`)); console.log(chalk.green(`Template: ${options.template}`)); console.log(chalk.bold.green('\n✓ Project initialized successfully!')); }); program .command('validate') .description('Validate REP configuration and resources') .option('-c, --config <path>', 'Config file path', './rep.config.js') .option('-v, --verbose', 'Verbose output') .action((options) => { console.log(chalk.blue('Validating REP configuration...')); console.log(chalk.gray(`Config path: ${options.config}`)); if (options.verbose) { console.log(chalk.gray('Running in verbose mode...')); } console.log(chalk.bold.green('\n✓ Validation passed!')); }); program .command('report') .description('Generate REP evaluation report') .option('-o, --output <file>', 'Output file', 'rep-report.html') .option('-f, --format <format>', 'Format: html, json, markdown', 'html') .option('-s, --summary', 'Show summary only') .action((options) => { console.log(chalk.blue('Generating report...')); console.log(chalk.green(`Format: ${options.format}`)); console.log(chalk.green(`Output: ${options.output}`)); if (options.summary) { console.log(chalk.gray('Mode: Summary only')); } console.log(chalk.bold.green('\n✓ Report generated successfully!')); }); ``` ### Technical Analysis The packaged `rep` CLI presents operational commands but only prints success messages. The `init` command does not crea ...[truncated 1328 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions - Replace placeholder handlers with calls to the bundled implementations under `scripts/`. - Ensure `validate` reads the specified input, performs actual schema and integrity checks, and returns a nonzero exit code on failure. - Verify that `init`, `report`, and `emit` produce their documented filesystem or event side effects before reporting success. - Remove hard-coded statistics or clearly label them as demonstration data. - Add automated tests using valid and invalid fixtures. - Test both output text and process exit status. - Avoid publishing the CLI as production-ready until all advertised commands are functional. ]]>

T05 · Unauthorized Access and Privilege Escalation

Error

• Location: scripts/rep.mjs:1147
• Finding: Unauthenticated Artifact API Permits JSONL Path Traversal and Network Disclosure<![CDATA[ ## Vulnerability Details **File Location**: `scripts/rep.mjs:1147-1222` **Vulnerability Type**: Path traversal, missing authentication, and unsafe network exposure **Risk Level**: High ### Vulnerable Code ```js function getArtifacts(type) { const filePath = path.join(artifactsDir, `${type}.jsonl`); if (!fs.existsSync(filePath)) { return { error: `Artifact type not found: ${type}` }; } const content = fs.readFileSync(filePath, 'utf-8'); const lines = content.trim().split('\n').filter(Boolean); return lines.map(line => { try { return JSON.parse(line); } catch { return { error: 'invalid json' }; } }); } ``` ```js const server = createServer((req, res) => { const url = new URL(req.url, `http://localhost:${port}`); res.setHeader('Content-Type', 'application/json'); res.setHeader('Access-Control-Allow-Origin', '*'); try { if (url.pathname === '/stats' || url.pathname === '/') { res.end(JSON.stringify(getStats(), null, 2)); } else if (url.pathname === '/artifacts') { const type = url.searchParams.get('type'); if (type) { res.end(JSON.stringify(getArtifacts(type), null, 2)); } else { res.end(JSON.stringify(Object.keys(getStats().artifacts), null, 2)); } } else if (url.pathname === '/validate') { res.end(JSON.stringify(validateBundle(), null, 2)); } else if (url.pathname === '/health') { res.end(JSON.stringify({ status: 'ok', timestamp: new Date().toISOString() })); } else { res.statusCode = 404; res.end(JSON.stringify({ error: 'Not found. Endpoints: /stats, /artifacts, /validate, /health' })); } } catch (err) { res.statusCode = 500; res.end(JSON.stringify({ error: err.message })); } }); server.listen(port, () => { console.log(`✅ REP API server running at http://localhost:${port}`); console.log(` Endpoints:`); console.log(` - GET /stats Bundle statistics`); console.log(` - GET /artifacts List artifact ...[truncated 2037 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions - Allow only known artifact type identifiers, such as values matching a strict allowlist or `^[A-Za-z0-9_-]+$`. - Resolve the requested path with `path.resolve()` and verify it remains inside the canonical artifacts directory. - Reject path separators, `..`, absolute paths, encoded traversal sequences, and unexpected extensions. - Bind explicitly to `127.0.0.1` by default: ```js server.listen(port, '127.0.0.1', callback); ``` - Require an authentication mechanism before supporting non-loopback access. - Replace wildcard CORS with an explicit trusted-origin allowlist or disable CORS. - Apply restrictive filesystem permissions to artifact directories. - Avoid returning raw internal exception messages to clients. - Add tests for plain, encoded, and platform-specific traversal payloads. ]]>

T05 · Unauthorized Access and Privilege Escalation

Warning

• Location: scripts/rep-near-miss-cron.mjs:66
• Finding: Near-Miss Collector Reads and Replicates Workspace Log Content Beyond the Documented Boundary<![CDATA[ ## Vulnerability Details **File Location**: `scripts/rep-near-miss-cron.mjs:66-143` **Vulnerability Type**: Excessive file access and sensitive log replication **Risk Level**: Medium ### Vulnerable Code ```js async function checkNearMisses() { const findings = []; const workspace = process.env.OPENCLAW_WORKSPACE || path.join(__dirname, '..'); // 1. Check subagent health log const subagentLog = path.join(workspace, 'memory', 'subagent-health.log'); if (fs.existsSync(subagentLog)) { const content = fs.readFileSync(subagentLog, 'utf-8'); const lines = content.trim().split('\n'); const recentLines = lines.slice(-20); for (const line of recentLines) { if (line.includes('failed') || line.includes('error') || line.includes('timeout')) { findings.push({ source: 'subagent-health', issue: line.substring(0, 200), severity: 'medium' }); } } } // 2. Check openclaw healthchecks log const healthLog = path.join(workspace, 'openclaw-healthchecks.log'); if (fs.existsSync(healthLog)) { const content = fs.readFileSync(healthLog, 'utf-8'); const lines = content.trim().split('\n').filter(l => l.includes('FAIL') || l.includes('ERROR')); const recentFails = lines.slice(-10); for (const line of recentFails) { findings.push({ source: 'healthcheck', issue: line.substring(0, 200), severity: 'high' }); } } // 4. Check for recent cron failures const cronLog = path.join(workspace, 'memory', 'cron-failures.log'); if (fs.existsSync(cronLog)) { const content = fs.readFileSync(cronLog, 'utf-8'); const lines = content.trim().split('\n').slice(-5); for (const line of lines) { findings.push({ source: 'cron', issue: line.substring(0, 200), severity: 'medium' }); } } return findings; } ``` ### Technical Analysis The collector reads OpenClaw workspace files outside th ...[truncated 1611 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions - Require explicit opt-in configuration for each monitored log path. - Update documentation to disclose every file and directory read by the collector. - Run the collector under a dedicated unprivileged account. - Parse structured error fields instead of copying raw log lines. - Redact credentials, tokens, authorization headers, URLs with embedded credentials, private paths, and personal identifiers before persistence. - Make raw-line capture disabled by default. - Apply restrictive permissions and retention limits to generated artifacts. - Provide a dry-run mode that reports intended input paths without reading them. - Add tests confirming that representative secrets are removed from findings. ]]>

T08 · Insecure Dependencies

Error

• Location: github-action/entrypoint.sh:22
• Finding: GitHub Action Installs and Executes Unpinned External Packages<![CDATA[ ## Vulnerability Details **File Location**: `github-action/entrypoint.sh:22-31` **Vulnerability Type**: Unpinned dependency installation and package-name ambiguity **Risk Level**: High ### Vulnerable Code ```bash # Check if rep CLI is available if ! command -v rep &> /dev/null; then echo "::warning:: REP CLI not found in PATH. Installing rep..." # Try to install rep using npm or cargo if command -v npm &> /dev/null; then npm install -g @anthropic/rep-cli || npm install -g rep elif command -v cargo &> /dev/null; then cargo install rep else echo "::error:: Cannot install REP CLI: neither npm nor cargo is available" echo '{"valid": false, "errors": ["REP CLI not installed"], "warnings": []}' > "$REPORT_FILE" echo "result=fail" >> "$GITHUB_OUTPUT" echo "errors=1" >> "$GITHUB_OUTPUT" echo "warnings=0" >> "$GITHUB_OUTPUT" echo "json-output=$(cat "$REPORT_FILE" | jq -Rs .)" >> "$GITHUB_OUTPUT" exit 1 fi fi ``` ### Technical Analysis When `rep` is absent, the composite Action globally installs the latest package associated with one of several registry names. No exact version, lockfile, checksum, provenance constraint, or integrity verification is used. The npm and Cargo package identities are also not demonstrably tied to the local implementation included in this project. Package installation can execute lifecycle or build scripts, while the installed binary is subsequently executed in the CI runner. Therefore, a compromised package release, dependency-confusion event, transferred package name, or unrelated package can become code execution in CI. The `rep-version` Action input does not pin the installed dependency; it is only added later as a validator command argument. ### Attack Path 1. A repository uses the provided GitHub Action. 2. The runner does not already contain a `rep` executable. 3. The entrypoint installs the latest available `@anthropic/rep ...[truncated 797 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions - Execute the repository’s bundled validator directly instead of installing a registry package. - If an external package is unavoidable, pin an exact reviewed version and verify package integrity and provenance. - Remove fallback installation of ambiguous npm and Cargo packages. - Install dependencies locally rather than globally. - Use a committed lockfile with reproducible installation. - Disable unnecessary package lifecycle scripts where feasible. - Run CI with the minimum `GITHUB_TOKEN` permissions and avoid exposing deployment secrets to validation jobs. - Document the precise package publisher and release-verification process. ]]>

T09 · Insecure Skill Coding Practices

Error

• Location: github-action/entrypoint.sh:68
• Finding: GitHub Action Discards Validator Failure Exit Codes<![CDATA[ ## Vulnerability Details **File Location**: `github-action/entrypoint.sh:68-69` **Vulnerability Type**: Fail-open validation and incorrect shell status handling **Risk Level**: High ### Vulnerable Code ```bash # Run validation and capture output VALIDATION_OUTPUT=$($REVALIDATE_CMD 2>&1) || true VALIDATION_EXIT_CODE=$? ``` The affected status is later used as follows: ```bash if [ "$VALIDATION_EXIT_CODE" -ne 0 ] && [ "$RESULT" = "fail" ]; then echo "::error:: Validation failed with exit code $VALIDATION_EXIT_CODE" exit "$VALIDATION_EXIT_CODE" fi ``` ### Technical Analysis When the validator exits with a nonzero status, `|| true` executes. The immediately following `$?` therefore records the successful status of `true`, normally zero, rather than the validator’s status. The later exit-code condition cannot detect the original validator failure. The script then relies on output parsing to infer success. If no valid JSON object is extracted, the fallback marks failure only when output contains the word `error`. A validator that crashes, is terminated, or returns nonzero output without that word can consequently be reported as passing. ### Attack Path 1. A malformed bundle, runtime condition, or hostile input causes `rep validate` to exit nonzero. 2. The `|| true` branch runs and resets the shell status to zero. 3. `VALIDATION_EXIT_CODE` is assigned zero. 4. Output does not contain parseable JSON or the specific fallback keyword expected by the script. 5. The default `RESULT="pass"` remains in effect. 6. The Action publishes a passing result and exits successfully despite validator failure. ### Impact Assessment This issue bypasses the intended CI reliability and integrity gate. It does not grant direct operating-system privileges, but it allows invalid or unvalidated artifacts to progress through workflows. If later jobs deploy, publish, or sign based on this result, the effective scope includes those downstream operations. ]]>
• Remediation: <![CDATA[ ## Remediation Suggestions Capture the command status without overwriting it: ```bash set +e VALIDATION_OUTPUT=$("${CMD[@]}" 2>&1) VALIDATION_EXIT_CODE=$? set -e ``` Additional hardening should include: - Build the validator invocation as a Bash array instead of a command string. - Treat malformed, missing, or ambiguous validator output as failure. - Make any nonzero validator exit status fail the Action independently of text parsing. - Validate the complete JSON output rather than extracting the first brace-delimited substring with `grep`. - Add CI tests for ordinary validation failures, crashes, signals, malformed JSON, and output that does not contain the word `error`. - Set Action outputs before exiting so failed runs still expose reliable diagnostics. ]]>

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access
• Supply Chain: Unpinned Dependencies, External Script Fetching, Obfuscated Code
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• MCP Tool Poisoning: Hidden Instructions, Unicode Deception, Parameter Description Injection

Findings (29)

Ae1

High

• Category: analysis-evasion

Content

- `rep.mjs` - Main CLI for initialization, validation, and bundle management

• Confidence: 100% confidence
• Finding: Referenced artifact was not completely inspected

Ae1

High

• Category: analysis-evasion

Content

- `rep.mjs` - Main CLI for initialization, validation, and bundle management

• Confidence: 100% confidence
• Finding: Referenced artifact was not completely inspected

Ae1

High

• Category: analysis-evasion

Content

- `rep.mjs` - Main CLI for initialization, validation, and bundle management

• Confidence: 100% confidence
• Finding: Referenced artifact was not completely inspected

Ae1

High

• Category: analysis-evasion

Content

- `rep-validate.mjs` - Schema validation engine for REP artifacts

• Confidence: 100% confidence
• Finding: Referenced artifact was not completely inspected

Ae1

High

• Category: analysis-evasion

Content

- `rep-validate.mjs` - Schema validation engine for REP artifacts

• Confidence: 100% confidence
• Finding: Referenced artifact was not completely inspected

Ae1

High

• Category: analysis-evasion

Content

- `rep-validate.mjs` - Schema validation engine for REP artifacts

• Confidence: 100% confidence
• Finding: Referenced artifact was not completely inspected

Ae1

High

• Category: analysis-evasion

Content

- `rep-heartbeat-cron.mjs` - Records agent heartbeats on a schedule

• Confidence: 100% confidence
• Finding: Referenced artifact was not completely inspected

Ae1

High

• Category: analysis-evasion

Content

- `rep-heartbeat-cron.mjs` - Records agent heartbeats on a schedule

• Confidence: 100% confidence
• Finding: Referenced artifact was not completely inspected

Ae1

High

• Category: analysis-evasion

Content

- `rep-heartbeat-cron.mjs` - Records agent heartbeats on a schedule

• Confidence: 100% confidence
• Finding: Referenced artifact was not completely inspected

Ae1

High

• Category: analysis-evasion

Content

- `rep-near-miss-cron.mjs` - Tracks near-miss reliability events

• Confidence: 100% confidence
• Finding: Referenced artifact was not completely inspected

Ae1

High

• Category: analysis-evasion

Content

- `rep-performance-baseline.mjs` - Captures performance metrics

• Confidence: 100% confidence
• Finding: Referenced artifact was not completely inspected

Ae1

High

• Category: analysis-evasion

Content

- `rep-generate-bundle.mjs` - Generates REP bundles from artifacts

• Confidence: 100% confidence
• Finding: Referenced artifact was not completely inspected

Sudo/Root Execution

Medium

• Category: Privilege Escalation

Content

Ensure the artifacts directory is writable:
```bash
mkdir -p /path/to/artifacts
chmod 755 /path/to/artifacts
```

• Confidence: 80% confidence
• Finding: Commands invoke sudo or root privileges. Verify this elevated access is necessary and justified.

Autonomous Decision Making

Medium

• Category: Excessive Agency

Content

"mitigations": ["maintain rollback capability", "gradual rollout", "monitoring_dashboard"]
    },
    "requested_by": "agent:capability-evolver",
    "approved_by": "system:auto-approved",
    "priority": "high",
    "estimated_implementation_hours": 4
  }

• Confidence: 85% confidence
• Finding: The example artifact includes `approved_by: system:auto-approved` for an evolution recommendation tied to model or parameter changes, which normalizes autonomous approval of operational modifications. In the context of an agent reliability/evolution system, auto-approval can bypass human review and separation-of-duties controls, increasing the risk of unsafe self-modification, policy drift, or deployment of harmful changes.

Missing User Warnings

Medium

• Confidence: 90% confidence
• Finding: The log shows blocked external or data-affecting actions, including sending a Telegram message and writing files, with 'User Notification: None' for at least some rejected operations. Suppressing user-visible warnings during integrity failures can hide important security or reliability events, delaying detection of corruption, preventing informed user response, and obscuring whether attempted side effects occurred or were safely blocked.

Missing User Warnings

Medium

• Confidence: 91% confidence
• Finding: The document describes a rollback and quarantine process that discards 2.5 minutes of session state and marks actions for replay, but it contains no user-facing notification, consent, or recovery acknowledgment step. In an agent-memory context, silent restoration can cause the agent and user to operate on divergent assumptions about what actions were completed, leading to confusion, duplicate actions, or missed work.

Missing User Warnings

Medium

• Confidence: 98% confidence
• Finding: The script constructs a shell command string from untrusted GitHub Action inputs such as INPUT_REP_PATH and INPUT_REP_VERSION, then executes it via command substitution. Because quoting is assembled inside the string rather than enforced through an argument array, an attacker controlling workflow inputs can inject shell metacharacters or command substitutions and achieve arbitrary command execution on the runner. In CI, this can expose repository secrets, modify artifacts, or tamper with validation results.

Intent-Code Divergence

Medium

• Confidence: 97% confidence
• Finding: The CLI advertises `--verify-hash` as if it performs real SHA256 content verification, but the implementation explicitly admits it only does a limited comparison and cannot reliably verify integrity. This can cause users or downstream automation to trust tampered or malformed artifacts as having been cryptographically validated when they have not, undermining integrity controls.

Intent-Code Divergence

Medium

• Confidence: 94% confidence
• Finding: The integrity-check command is presented as verifying bundle hashes, but for legacy artifacts it increments the valid count even when the recomputed hash does not match the stored hash. That creates a trust-boundary failure: tampered or corrupted legacy records can be reported as valid, undermining auditability and any downstream process that relies on integrity-check output for security or compliance decisions.

Missing User Warnings

Medium

• Confidence: 88% confidence
• Finding: The serve command exposes bundle statistics and full artifact contents over HTTP with permissive CORS and no authentication, access control, or user-facing warning about the sensitivity of local data. In the REP context, artifacts may contain incident, policy, operational, or reliability evidence, so accidental exposure to other local users, browser contexts, or network-reachable interfaces can leak sensitive information.

Intent-Code Divergence

Medium

• Confidence: 97% confidence
• Finding: The generate command fabricates a random content_hash instead of computing it from artifact contents, which breaks the semantic meaning of the field and produces artifacts that appear integrity-protected when they are not. Consumers may trust these generated artifacts as authentic or untampered, but the hash provides no binding to content, enabling silent modification or invalid test data to propagate into validation workflows.

Missing User Warnings

Low

• Confidence: 82% confidence
• Finding: The best-practices section recommends automated generation via cron jobs, and later sections describe cron jobs generating heartbeat and incident artifacts. Because this behavior can continuously write monitoring data over time, a user-facing warning about storage growth and automated persistence is warranted but absent.

Missing User Warnings

Low

• Confidence: 88% confidence
• Finding: This markdown instructs subagents to emit artifacts by appending JSONL records to files via `fs.appendFileSync`, which changes local data on disk. The surrounding documentation presents this as an integration pattern but does not explicitly warn users that enabling or using the workflow will write to artifact files and may create or modify audit records automatically.

Unpinned Dependencies

Low

• Category: Supply Chain

Content

"author": "",
  "license": "MIT",
  "dependencies": {
    "commander": "^11.1.0",
    "chalk": "^4.1.2"
  },
  "engines": {

• Confidence: 40% confidence
• Finding: Dependencies lack version pinning, allowing potential malicious package updates. Consider pinning versions.

Unpinned Dependencies

Low

• Category: Supply Chain

Content

"license": "MIT",
  "dependencies": {
    "commander": "^11.1.0",
    "chalk": "^4.1.2"
  },
  "engines": {
    "node": ">=14.0.0"

• Confidence: 40% confidence
• Finding: Dependencies lack version pinning, allowing potential malicious package updates. Consider pinning versions.

Static analysis

Detected: suspicious.dangerous_exec

Shell command execution detected (child_process).

Critical

• Code: suspicious.dangerous_exec
• Location: scripts/rep.mjs:805