Back to skill

Security audit

Agent Observability - Open the Black Box

Security checks for vulnerabilities and agentic risk

Overview

This observability skill has a legitimate purpose, but it needs review because it installs recurring monitoring that stores agent outputs and dynamically loads workspace code.

Install only if you want persistent observability files in this workspace. Review who can write to scripts/session-metrics.js and INTENT.md before enabling cron, and treat memory/decisions-audit.jsonl and memory/traces as potentially sensitive because they can contain reasoning summaries and output snippets.

Vulnerability Patterns
  • Tool Hijacking and SpoofingModifies or replaces tools so legitimate-looking calls execute attacker logic
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (2)

T07 · Tool Hijacking and Spoofing

Error
Location
references/throughput-dashboard.js:29
Finding
Arbitrary Code Execution Through Dynamic Loading of a Workspace Metrics Module<![CDATA[ ## Vulnerability Details **File Location**: `references/throughput-dashboard.js`, lines 29 and 51–66 **Vulnerability Type**: Unsafe dynamic module execution across a local trust boundary **Risk Level**: High ### Vulnerable Code ```javascript function safeRequire(p) { try { return require(p); } catch (_) { return null; } } function collectSessionSummary(workspaceRoot) { const metricsPath = path.join(workspaceRoot, 'scripts', 'session-metrics.js'); const mod = safeRequire(metricsPath); if (!mod || typeof mod.getWeeklySummary !== 'function') { return { total_tasks: 0, total_cost: 0, avg_cost_per_task: 0, total_subagents: 0, quality_ratio: 1.0, sessions: 0 }; } try { return mod.getWeeklySummary(workspaceRoot); } catch (_) { return { total_tasks: 0, total_cost: 0, avg_cost_per_task: 0, total_subagents: 0, quality_ratio: 1.0, sessions: 0 }; } } function collectRoutingStats(workspaceRoot) { const metricsPath = path.join(workspaceRoot, 'scripts', 'session-metrics.js'); const mod = safeRequire(metricsPath); if (!mod || typeof mod.getRoutingStats !== 'function') { return { total_decisions: 0, by_target: { core: 0, specialist: 0, escalate: 0 }, by_type: {} }; } ``` ### Technical Analysis The dashboard treats `scripts/session-metrics.js` as a metrics source but loads it through Node.js `require()`. Requiring a JavaScript module immediately executes all of its top-level code before the exported functions are inspected. The module path is derived from the supplied workspace root, and the code performs no integrity validation, ownership check, permission check, or module allowlisting. Consequently, anyone capable of creating or replacing `scripts/session-metrics.js` in the selected workspace can cause arbitrary JavaScript to execute when the dashboard runs. Wrapping `require()` in `try/catch` does not provide a security boundary. It only suppresses exceptions after malicious top-level code may already have executed. Suppression ...[truncated 1691 chars]
Remediation
<![CDATA[ ## Remediation Suggestions 1. Replace executable metrics modules with a non-executable data format such as JSON or JSONL. 2. Parse the metrics file using `JSON.parse()` and validate it against a strict schema before use. 3. Reject unexpected fields, invalid types, non-finite numbers, and values outside documented ranges. 4. If a plugin architecture is necessary, load plugins only from an explicit administrator-controlled allowlist. 5. Verify plugin integrity using a pinned cryptographic hash or signed manifest before loading it. 6. Check file ownership and permissions and reject modules writable by less-trusted users. 7. Run scheduled monitoring under a dedicated least-privilege account with narrowly scoped filesystem and network access. 8. Do not silently suppress module-loading failures. Record failures in a protected audit log without exposing sensitive data. 9. Resolve and validate the canonical workspace path before accessing any workspace resources. ]]>

T09 · Insecure Skill Coding Practices

Warning
Location
references/drift-guard-auto.js:98
Finding
Regular Expression Injection Through Unescaped INTENT.md Criteria<![CDATA[ ## Vulnerability Details **File Location**: `references/drift-guard-auto.js`, lines 98–120 and 167–171 **Vulnerability Type**: Regular expression injection and denial of service **Risk Level**: Medium ### Vulnerable Code ```javascript function loadIntentCriteria(workspaceRoot) { const defaults = { never_sacrifice: ['honesty', 'safety', 'user_autonomy', 'corrigibility'], priorities: ['user_value_delivery', 'honesty_and_accuracy', 'cost_efficiency', 'response_speed'], quality_rules: { code: 'correctness_and_tests', writing: 'prose_quality_over_volume', research: 'verified_sources_over_speed' } }; try { const raw = fs.readFileSync(path.join(workspaceRoot, 'INTENT.md'), 'utf8'); const neverMatch = raw.match(/never_sacrifice:\s*\[([^\]]+)\]/); const never_sacrifice = neverMatch ? neverMatch[1].split(',').map(s => s.trim()).filter(Boolean) : defaults.never_sacrifice; const primary = (raw.match(/primary:\s*(\S+)/) || [])[1]; const secondary = (raw.match(/secondary:\s*(\S+)/) || [])[1]; const tertiary = (raw.match(/tertiary:\s*(\S+)/) || [])[1]; const priorities = [primary, secondary, tertiary].filter(Boolean).length > 0 ? [primary, secondary, tertiary].filter(Boolean) : defaults.priorities; return { never_sacrifice, priorities, quality_rules: defaults.quality_rules }; } catch (_) { return defaults; } } ``` ```javascript if (intentCriteria && Array.isArray(intentCriteria.never_sacrifice)) { for (const term of intentCriteria.never_sacrifice) { if (new RegExp(`not (?:sure|certain|confident) (?:about )?${term}`, 'i').test(text)) { score -= 5; flags.push(`never_sacrifice_hedge:${term}`); } } } ``` ### Technical Analysis Values extracted from the `never_sacrifice` list in `INTENT.md` are inserted directly into a dynamically constructed regular expression. The values are neither escaped nor constrained to a safe character set. An attacker who ...[truncated 1927 chars]
Remediation
<![CDATA[ ## Remediation Suggestions 1. Escape all configuration-derived terms before inserting them into a regular expression: ```javascript function escapeRegExp(value) { return String(value).replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); } const safeTerm = escapeRegExp(term); const pattern = new RegExp( `not (?:sure|certain|confident) (?:about )?${safeTerm}`, 'i' ); ``` 2. Apply a strict allowlist to intent terms, such as letters, digits, spaces, underscores, and hyphens. 3. Impose conservative limits on the number and maximum length of terms accepted from `INTENT.md`. 4. Wrap dynamic regex construction and evaluation in an explicit error handler so one invalid criterion cannot terminate the entire audit. 5. Prefer literal substring matching when full regular-expression behavior is unnecessary. 6. Validate `INTENT.md` against a documented configuration schema before starting the audit. 7. Record rejected terms and audit failures in a protected operational log. 8. Consider process-level execution time and memory limits for scheduled audits to contain unforeseen performance failures. ]]>
Vulnerability Patterns
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
Findings (14)

Ae1

High
Category
analysis-evasion
Content
| `throughput-dashboard.js` | Weekly productivity metrics | `scripts/` |
Confidence
100% confidence
Finding
Referenced artifact was not completely inspected

Ae1

High
Category
analysis-evasion
Content
| `throughput-dashboard.js` | Weekly productivity metrics | `scripts/` |
Confidence
100% confidence
Finding
Referenced artifact was not completely inspected

Ae1

High
Category
analysis-evasion
Content
| `throughput-dashboard.js` | Weekly productivity metrics | `scripts/` |
Confidence
100% confidence
Finding
Referenced artifact was not completely inspected

Ae1

High
Category
analysis-evasion
Content
| `failure-tracer.js` | Captures traces when quality score < 7 | `lib/` |
Confidence
100% confidence
Finding
Referenced artifact was not completely inspected

Ae1

High
Category
analysis-evasion
Content
| `failure-tracer.js` | Captures traces when quality score < 7 | `lib/` |
Confidence
100% confidence
Finding
Referenced artifact was not completely inspected

Ae1

High
Category
analysis-evasion
Content
| `failure-tracer.js` | Captures traces when quality score < 7 | `lib/` |
Confidence
100% confidence
Finding
Referenced artifact was not completely inspected

Ae1

High
Category
analysis-evasion
Content
| `drift-guard-auto.js` | Weekly INTENT.md compliance scoring | `scripts/` |
Confidence
100% confidence
Finding
Referenced artifact was not completely inspected

Ae1

High
Category
analysis-evasion
Content
| `drift-guard-auto.js` | Weekly INTENT.md compliance scoring | `scripts/` |
Confidence
100% confidence
Finding
Referenced artifact was not completely inspected

Ae1

High
Category
analysis-evasion
Content
| `drift-guard-auto.js` | Weekly INTENT.md compliance scoring | `scripts/` |
Confidence
100% confidence
Finding
Referenced artifact was not completely inspected

Ae1

High
Category
analysis-evasion
Content
| `drift-guard-auto.js` | Weekly INTENT.md compliance scoring | `scripts/` |
Confidence
100% confidence
Finding
Referenced artifact was not completely inspected

Vague Triggers

Medium
Confidence
89% confidence
Finding
The skill description is broad enough to match common requests for observability, debugging, or production monitoring, which increases the chance it is invoked in contexts where the user did not explicitly consent to installing persistent monitoring components. Because the skill copies scripts and libraries into the workspace and encourages recurring cron/heartbeat execution, overbroad activation can silently expand data collection and persistence beyond the user’s immediate intent.

Missing User Warnings

Medium
Confidence
95% confidence
Finding
The skill instructs the user to install components that persistently write decision logs, failure traces, dashboards, and drift reports into workspace files, but it does not provide a prominent warning about the privacy, retention, and sensitivity implications of storing reasoning summaries and output snippets. In practice, these logs may capture sensitive operational context, agent outputs, or decision rationale, making accidental data retention and later disclosure more likely.

Intent-Code Divergence

Medium
Confidence
97% confidence
Finding
This is a real integrity issue: the module advertises an append-only audit log, but `updateOutcome` rewrites the entire file and mutates existing records in place. Audit logs are relied on for tamper-evidence and post-incident reconstruction, so allowing modification of historical entries undermines trust in the log and can enable covering tracks or altering evidence of prior decisions.

Intent-Code Divergence

Medium
Confidence
99% confidence
Finding
This is a real integrity issue in the reporting logic. The scorecard checks for exact keys like 'sycophancy', 'social_cushion', 'unprompted_why', and 'hallucination_hedge', but flagCounts is populated with more specific keys such as 'sycophancy:great choice' after only stripping the occurrence suffix. As a result, the report can falsely claim a dimension is clean even when matching flags were found, undermining audit accuracy and potentially hiding behavioral drift from operators.