Back to skill

Security audit

LLM Cost Watchdog

Security checks for vulnerabilities and agentic risk

Overview

This cost-tracking skill is purpose-aligned, but it needs review because it can broadly observe LLM traffic and persist local usage/error data that may contain sensitive metadata.

Install only if you are comfortable with local LLM usage monitoring. Prefer explicit SDK wrappers over global httpx capture, avoid enabling the tailer or detect command on shared or highly sensitive machines unless needed, set CW_LOG_DIR to a protected location, periodically clear ~/.cost-watchdog logs, and use CW_OFFLINE or CW_STATIC_ONLY if live pricing lookups are not acceptable.

Vulnerability Patterns
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (1)

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/http_capture.py:199
Finding

Streaming Capture Persists Unredacted Request URLs and Potential API Credentials

Content
View full analysis
None: """Emit a single errors.jsonl entry noting a streaming call we couldn't price.""" try: from errors import log_error log_error("http_capture", f"skipped stream: provider={provider} url={url}", hint="Use track_openai / track_anthropic for streaming coverage.") except Exception: pass ``` `scripts/errors.py:21-23, 33-50`: ```python LOG_DIR = Path(os.environ.get("CW_LOG_DIR", Path.home() / ".cost-watchdog")) LOG_DIR.mkdir(parents=True, exist_ok=True) ERRORS_LOG = LOG_DIR / "errors.jsonl" MAX_LINES = 5000 def log_error(component: str, error: Union[str, BaseException], *, hint: Optional[str] = None, extra: Optional[dict] = None) -> None: """Append one error entry. Never raises.""" try: if isinstance(error, BaseException): detail = "".join(traceback.format_exception_only(type(error), error)).strip() tb = "".join(traceback.format_tb(error.__traceback__)) else: detail = str(error) tb = "" entry = { "ts": time.time(), "component": component, "error": detail, "traceback": tb, "hint": hint or "", "extra": extra or {}, } line = js ...[truncated 2976 chars]
Remediation
View remediation
str: parsed = urlsplit(raw_url) return urlunsplit((parsed.scheme, parsed.netloc, parsed.path, "", "")) ``` 2. **Apply explicit query-parameter redaction** if operational diagnostics require selected parameters. Use an allowlist of known-safe parameter names rather than a denylist. At minimum, redact names such as `key`, `api_key`, `apikey`, `token`, `access_token`, `signature`, `sig`, and `credential`. 3. **Sanitize exception messages and tracebacks** before storing them. Replace credential-like query values, authorization tokens, bearer tokens, and provider-key formats with a fixed marker such as `[REDACTED]`. 4. **Enforce restrictive filesystem permissions:** - Create `~/.cost-watchdog` with mode `0700`. - Create `errors.jsonl`, `usage.jsonl`, cache files, and state files with mode `0600`. - Validate and correct permissions for pre-existing paths. - Reject symlinks or unexpected non-regular files before writing security-sensitive logs. 5. **Minimize logged data.** A streaming-gap entry only needs the provider and sanitized endpoint path. Do not store query strings, fragments, headers, request bodies, prompts, or response content. 6. **Add regression tests** that submit URLs containing representative secret parameters and verify that neither the secret values nor complete query strings appear in `errors.jsonl`. 7. **Document local telemetry sensitivity and retention.** Provide a command to clear error logs and define a bounded retention policy for potentially sensitive diagnostic records. ]]>
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Rogue AgentSelf-Modification, Session Persistence
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • Behavioral ASTexec() Call, eval() Call, Dynamic Import
Findings (34)

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 81)May include surrounding context.

md
Regenerates `references/pricing.md` from the live sources so the offline

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 169)May include surrounding context.

md
Regenerates `references/pricing.md` from the live sources so the offline

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 171)May include surrounding context.

md
Regenerates `references/pricing.md` from the live sources so the offline

Agent Config Directory Access

High
Category
Agent Snooping
Confidence
90% confidence
Finding

Skill reads from agent configuration directories (.claude/, .codex/, .gemini/). These directories may contain API keys, personal settings, and other credentials that the skill has no legitimate need to access.

Content

Scanner excerpt · scripts/detect_model.py (reported line 19)May include surrounding context.

python
env          — ANTHROPIC_MODEL / OPENAI_MODEL / CLAUDE_MODEL / etc.
    openclaw     — latest ~/.openclaw/agents/main/sessions/*.jsonl
    claude-code  — latest ~/.claude/projects/*/sessions/*.jsonl
    claude-settings — ~/.claude/settings.json
    usage-log    — most recent entry in ~/.cost-watchdog/usage.jsonl
"""
from __future__ import annotations

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The README states the skill auto-activates on broad conditions such as detecting API calls, loops, or general cost/budget discussions. In an agent skill context, overly broad triggers can cause unintended activation on unrelated tasks, expanding the skill’s influence and potentially altering agent behavior or analysis without explicit user intent. While not overtly malicious, this increases the chance of unexpected execution paths and unnecessary access to task context.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

The description and when_to_use fields say the skill triggers on broad concepts like "cost/budget/token mentions," "LLM API calls," and "agent workflows." These phrases overlap with common discussion and general coding contexts, and the file does not provide exclusion conditions or negative examples to clarify when the skill should not activate.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The trigger list includes generic conditions such as user mentions of cost or billing, code containing LLM API calls, agent loops, and batch processing with unclear bounds. Without tighter constraints, these conditions could match many ordinary interactions and code reviews, leading to over-invocation.

Content

No source excerpt is available for this finding.

Session Persistence

Medium
Category
Rogue Agent
Confidence
79% confidence
Finding

The skill explicitly persists session usage data to a fixed directory under the user's home folder, which creates a privacy and data retention risk. Even though the stored data is for cost tracking, usage logs can reveal model names, token counts, timing, workflow patterns, and potentially sensitive metadata across sessions if retention and access controls are not clearly bounded.

Content

Scanner excerpt · SKILL.md (reported line 87)May include surrounding context.

md
## 5. Tracking layer — how we know what was spent

Four independent paths, all write to `~/.cost-watchdog/usage.jsonl`:

| Path | When to use | Covers streams? |
|---|---|---|

Unbounded Resource Access

Medium
Category
Excessive Agency
Confidence
75% confidence
Finding

Skill allows unbounded resource consumption (API calls, storage, compute). Without rate limits or quotas, a compromised or misbehaving agent can cause denial-of-service or cost overruns.

Content

Scanner excerpt · references/patterns.md (reported line 106)May include surrounding context.

messages=[{"role": "user", "content": prompt}] ) except APIError: time.sleep(1) # Retry forever

text

**Why it's dangerous:** If the API returns a non-transient error (invalid request, content policy violation), this retries forever. Each retry costs money for calls that get partially processed.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
76% confidence
Finding

The module documentation states "Each probe should be cheap; no network calls" at L009, presenting that as an intent-level guarantee. However, this file imports and invokes usage_log.read_recent(...) from another module at L032 and L231 without any local enforcement that the probe is network-free, so the documented guarantee is not substantiated by this code and could be contradicted by actual behavior delegated to that module.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

This code globally monkey-patches httpx transports to inspect all matching LLM HTTP responses and record telemetry without any consent, notice, or scoping controls. In practice, this creates covert network-interception behavior inside the process and may capture sensitive metadata such as model names, providers, URLs, and token usage across all code using httpx, which can violate privacy expectations, internal policy, or compliance requirements.

Content

No source excerpt is available for this finding.

Session Persistence

Medium
Category
Rogue Agent
Confidence
94% confidence
Finding

This code is explicitly designed to append one row per LLM call to a persistent per-user file in the home directory, creating cross-run session persistence. In a skill environment, such persistence can enable long-term tracking of user activity and spending patterns without clear boundaries or consent, and the session_id field further increases correlation risk across otherwise separate interactions.

Content

Scanner excerpt · scripts/tracker.py (reported line 16)May include surrounding context.

python
client = track_anthropic(anthropic.Anthropic())

Both wrappers:
  - Intercept chat.completions.create / messages.create (sync + async).
  - Read model + usage from the response object.
  - Price via _pricing.get_price (live LiteLLM/OpenRouter with static fallback).
  - Append one row per call to ~/.cost-watchdog/usage.jsonl.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The wrapper persistently logs every LLM call's metadata to ~/.cost-watchdog/usage.jsonl without any user-facing notice, consent, or visible retention controls in this file. Even if it records usage rather than full prompts, model names, token counts, session IDs, provider data, and related metadata can reveal user behavior, project activity, and cross-session correlations, making this a privacy and surveillance risk in an agent skill context.

Content

No source excerpt is available for this finding.

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · tests/test_cost_watchdog.py (reported line 88)May include surrounding context.

python
def test_cost_visualizer_help(self):
        """Test cost-visualizer.py shows help."""
        result = subprocess.run(
            ["python3", "scripts/cost-visualizer.py"],
            capture_output=True,
            text=True,

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · tests/test_cost_watchdog.py (reported line 99)May include surrounding context.

python
def test_smart_budget_help(self):
        """Test smart-budget.py shows help."""
        result = subprocess.run(
            ["python3", "scripts/smart-budget.py"],
            capture_output=True,
            text=True,

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · tests/test_cost_watchdog.py (reported line 110)May include surrounding context.

python
def test_cost_visualizer_daily(self):
        """Test cost-visualizer.py daily command."""
        result = subprocess.run(
            ["python3", "scripts/cost-visualizer.py", "daily"],
            capture_output=True,
            text=True,

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · tests/test_cost_watchdog.py (reported line 120)May include surrounding context.

python
def test_smart_budget_set(self):
        """Test smart-budget.py set command."""
        result = subprocess.run(
            ["python3", "scripts/smart-budget.py", "set", "10.00", "--priority=high"],
            capture_output=True,
            text=True,

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · tests/test_cost_watchdog.py (reported line 130)May include surrounding context.

python
def test_smart_budget_alternatives(self):
        """Test smart-budget.py alternatives command."""
        result = subprocess.run(
            ["python3", "scripts/smart-budget.py", "alternatives", "claude-sonnet-4-6", "--savings=50"],
            capture_output=True,
            text=True,

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
86% confidence
Finding

This markdown file includes sample code that saves intermediate results and processed document IDs to a local JSON file, which affects user data persistence on disk. The surrounding documentation presents this as a cost-saving pattern but does not warn that results and metadata will be stored locally.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
78% confidence
Finding

This code initializes LiteLLMSource and OpenRouterSource, and later uses them for live or near-live lookups, which implies network activity and disk caching. While the module docstring mentions data sources, there is no confirmation prompt, logging, or explicit user-facing warning in the code around these external lookups or cache writes.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
80% confidence
Finding

This code performs outbound HTTP requests to external services and writes the returned data to a user home-directory cache. While the module docstring documents the behavior for developers, there is no user-facing prompt, print/log disclosure, or confirmation in the code around these operations.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
89% confidence
Finding

The script creates a persistent directory under the user's home folder and later writes cost/session/task data to disk without any user-facing notice, consent, or retention controls. While this is not overtly malicious, the stored data can reveal usage patterns, model/provider choices, task names, and spending history, which may be sensitive on shared systems or in regulated environments.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Low
Category
Not specified by scanner
Confidence
97% confidence
Finding

The top-level docstring says it dispatches the slash commands documented in SKILL.md and then enumerates the available commands, but the actual CLI also implements 'errors' and 'validate-tokens' subcommands later in the file. This is an intent/documentation divergence because the documentation presents the command surface as complete while code exposes additional functionality.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
84% confidence
Finding

This code probes environment variables such as ANTHROPIC_MODEL and OPENAI_MODEL and elsewhere reads files under ~/.claude, ~/.openclaw, and usage logs to infer model usage. Although the module docstring describes the diagnostic purpose, it does not clearly disclose that the script inspects local session history and configuration data, which is relevant user/system data access.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
81% confidence
Finding

The clear() function unlinks the persistent error log file, which is a file-deletion operation. Although the module-level docstring explains the log's purpose, there is no user-facing warning, confirmation, or inline disclosure near this destructive action indicating that invoking clear() removes stored diagnostic history.

Content

No source excerpt is available for this finding.