Back to skill

Security audit

Agent Observability - Open the Black Box

Security checks for vulnerabilities and agentic risk

Overview

This observability skill has a legitimate purpose, but it needs review because it installs recurring monitoring that stores agent outputs and dynamically loads workspace code.

Install only if you want persistent observability files in this workspace. Review who can write to scripts/session-metrics.js and INTENT.md before enabling cron, and treat memory/decisions-audit.jsonl and memory/traces as potentially sensitive because they can contain reasoning summaries and output snippets.

Vulnerability Patterns
  • Tool Hijacking and SpoofingModifies or replaces tools so legitimate-looking calls execute attacker logic
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (2)

T07 · Tool Hijacking and Spoofing

Error
Location
references/throughput-dashboard.js:29
Finding

Arbitrary Code Execution Through Dynamic Loading of a Workspace Metrics Module

Content
View full analysis
Remediation
View remediation

T09 · Insecure Skill Coding Practices

Warning
Location
references/drift-guard-auto.js:98
Finding

Regular Expression Injection Through Unescaped INTENT.md Criteria

Content
View full analysis
s.trim()).filter(Boolean) : defaults.never_sacrifice; const primary = (raw.match(/primary:\s*(\S+)/) || [])[1]; const secondary = (raw.match(/secondary:\s*(\S+)/) || [])[1]; const tertiary = (raw.match(/tertiary:\s*(\S+)/) || [])[1]; const priorities = [primary, secondary, tertiary].filter(Boolean).length > 0 ? [primary, secondary, tertiary].filter(Boolean) : defaults.priorities; return { never_sacrifice, priorities, quality_rules: defaults.quality_rules }; } catch (_) { return defaults; } } ``` ```javascript if (intentCriteria && Array.isArray(intentCriteria.never_sacrifice)) { for (const term of intentCriteria.never_sacrifice) { if (new RegExp(`not (?:sure|certain|confident) (?:about )?${term}`, 'i').test(text)) { score -= 5; flags.push(`never_sacrifice_hedge:${term}`); } } } ``` ### Technical Analysis Values extracted from the `never_sacrifice` list in `INTENT.md` are inserted directly into a dynamically constructed regular expression. The values are neither escaped nor constrained to a safe character set. An attacker who ...[truncated 1927 chars]
Remediation
View remediation
Vulnerability Patterns
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
Findings (14)

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 12)May include surrounding context.

md
| `throughput-dashboard.js` | Weekly productivity metrics | `scripts/` |

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 24)May include surrounding context.

md
| `throughput-dashboard.js` | Weekly productivity metrics | `scripts/` |

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 105)May include surrounding context.

md
| `throughput-dashboard.js` | Weekly productivity metrics | `scripts/` |

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 14)May include surrounding context.

md
| `failure-tracer.js` | Captures traces when quality score < 7 | `lib/` |

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 26)May include surrounding context.

md
| `failure-tracer.js` | Captures traces when quality score < 7 | `lib/` |

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 107)May include surrounding context.

md
| `failure-tracer.js` | Captures traces when quality score < 7 | `lib/` |

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 15)May include surrounding context.

md
| `drift-guard-auto.js` | Weekly INTENT.md compliance scoring | `scripts/` |

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 27)May include surrounding context.

md
| `drift-guard-auto.js` | Weekly INTENT.md compliance scoring | `scripts/` |

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 92)May include surrounding context.

md
| `drift-guard-auto.js` | Weekly INTENT.md compliance scoring | `scripts/` |

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 108)May include surrounding context.

md
| `drift-guard-auto.js` | Weekly INTENT.md compliance scoring | `scripts/` |

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
89% confidence
Finding

The skill description is broad enough to match common requests for observability, debugging, or production monitoring, which increases the chance it is invoked in contexts where the user did not explicitly consent to installing persistent monitoring components. Because the skill copies scripts and libraries into the workspace and encourages recurring cron/heartbeat execution, overbroad activation can silently expand data collection and persistence beyond the user’s immediate intent.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The skill instructs the user to install components that persistently write decision logs, failure traces, dashboards, and drift reports into workspace files, but it does not provide a prominent warning about the privacy, retention, and sensitivity implications of storing reasoning summaries and output snippets. In practice, these logs may capture sensitive operational context, agent outputs, or decision rationale, making accidental data retention and later disclosure more likely.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

This is a real integrity issue: the module advertises an append-only audit log, but updateOutcome rewrites the entire file and mutates existing records in place. Audit logs are relied on for tamper-evidence and post-incident reconstruction, so allowing modification of historical entries undermines trust in the log and can enable covering tracks or altering evidence of prior decisions.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
99% confidence
Finding

This is a real integrity issue in the reporting logic. The scorecard checks for exact keys like 'sycophancy', 'social_cushion', 'unprompted_why', and 'hallucination_hedge', but flagCounts is populated with more specific keys such as 'sycophancy:great choice' after only stripping the occurrence suffix. As a result, the report can falsely claim a dimension is clean even when matching flags were found, undermining audit accuracy and potentially hiding behavioral drift from operators.

Content

No source excerpt is available for this finding.