Back to skill

Security audit

SoulForge

Security checks for vulnerabilities and agentic risk

Overview

SoulForge appears locally focused rather than malicious, but it needs Review because it passively builds a persistent behavioral profile and its shipped file layout does not match its documented storage boundaries.

Review before installing. This skill is suitable only if you want local, persistent behavioral profiling of your sessions. Disable automatic observation if you do not want passive tracking, avoid --auto-accept for SOUL.md changes, and verify where memory/observations.json and backups are actually written before use.

Vulnerability Patterns
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (1)

T09 · Insecure Skill Coding Practices

Warning
Location
forge.py:23
Finding
Persistent State and SOUL.md Backups Are Written Outside the Skill Directory## Vulnerability Details **File Location**: `observe.py:20-23`, `reflect.py:18-21`, and `forge.py:23-27` **Vulnerability Type**: Incorrect base-directory resolution and cross-project state exposure **Risk Level**: Medium ### Vulnerable Code `observe.py:20-23`: ```python SKILL_DIR = Path(__file__).parent.parent MEMORY_DIR = SKILL_DIR / "memory" OBSERVATIONS_FILE = MEMORY_DIR / "observations.json" MEMORY_DIR.mkdir(exist_ok=True) ``` `reflect.py:18-21`: ```python SKILL_DIR = Path(__file__).parent.parent MEMORY_DIR = SKILL_DIR / "memory" OBSERVATIONS_FILE = MEMORY_DIR / "observations.json" ``` `forge.py:23-27`: ```python SKILL_DIR = Path(__file__).parent.parent MEMORY_DIR = SKILL_DIR / "memory" OBSERVATIONS_FILE = MEMORY_DIR / "observations.json" BACKUPS_DIR = MEMORY_DIR / "backups" BACKUPS_DIR.mkdir(parents=True, exist_ok=True) ``` ### Technical Analysis The three Python files are located directly in the project root rather than in the documented `scripts/` subdirectory. As a result, `Path(__file__).parent.parent` resolves to the parent of the project directory, not the project directory itself. The effective paths therefore become: ```text <project-parent>/memory/observations.json <project-parent>/memory/backups/ ``` instead of the documented locations: ```text <project>/memory/observations.json <project>/memory/backups/ ``` This creates an unintended shared-state boundary. Other projects or local users with write access to the project parent can create or alter the observation file consumed by `reflect.py` and `forge.py`. Likewise, `observe.py` stores session-derived behavioral data outside the expected Skill directory, and `forge.py` copies SOUL.md into an externally located backup directory. The use of `mkdir()` at module scope also means importing `observe.py` or `forge.py` can create directories outside the Skill without invoking their command-line entry points. ### Attack Path 1. An attacker with local write access to the parent directory c ...[truncated 1647 chars]
Remediation
## Remediation Suggestions 1. Resolve the Skill directory according to the actual package layout: ```python SKILL_DIR = Path(__file__).resolve().parent MEMORY_DIR = SKILL_DIR / "memory" ``` Alternatively, move the Python files into the documented `scripts/` directory and retain the parent traversal only after verifying the resolved destination. 2. Validate that all state paths remain beneath the intended Skill directory: ```python SKILL_DIR = Path(__file__).resolve().parent MEMORY_DIR = (SKILL_DIR / "memory").resolve() if SKILL_DIR not in MEMORY_DIR.parents: raise RuntimeError("Memory directory escapes the skill directory") ``` 3. Remove import-time filesystem mutations. Create `memory/` and `backups/` only inside explicit runtime initialization functions. 4. Create state directories and files with restrictive permissions appropriate to the platform, because observations and SOUL.md backups may contain sensitive behavioral or identity information. 5. Validate the observations JSON schema before using it. Require expected types and bounded values for fields such as `session_count`, `tone_history`, `hedging`, and `vocabulary`. 6. Write `observations.json` atomically by creating a restricted temporary file in the same directory, flushing it, and replacing the destination. This reduces corruption and race-condition risks. 7. Add tests asserting that `OBSERVATIONS_FILE` and `BACKUPS_DIR` resolve beneath the project root for the distributed package layout. 8. Update the documentation only after implementation paths and the published file structure are consistent.
Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
Findings (13)

Anti-Refusal Statement

High
Category
Anti-Refusal
Content
> *"Your SOUL.md says you value brevity, but you've asked for more detail 23 times and never asked for a shorter answer."*

> *"Your SOUL.md says you prefer async communication, but you always respond within 2 minutes."*

> *"Your SOUL.md says you're decisive, but you've changed direction mid-task 8 times this month."*
Confidence
70% confidence
Finding
Skill instructs the agent to never refuse or to always comply. Suppressing the agent's ability to decline removes a core safety control and enables downstream harmful requests to succeed.

Anti-Refusal Statement

High
Category
Anti-Refusal
Content
> *"Your SOUL.md says you're decisive, but you've changed direction mid-task 8 times this month."*

These gaps aren't failures — they're data. SoulForge surfaces them without judgment and asks what you actually want your soul to say.

---
Confidence
80% confidence
Finding
Skill instructs the agent to omit warnings, disclaimers, or ethical commentary. Stripping safety caveats hides risk from the user and is a common jailbreak preamble.

Missing User Warnings

Medium
Confidence
90% confidence
Finding
The README prominently advertises that the skill "watches who you actually are across sessions" and "quietly accumulates behavioral signals," but it does not present an upfront privacy warning or explicit consent guidance near the introduction and usage sections. Because the feature involves passive cross-session behavioral tracking, users may install or invoke it without fully understanding the scope of monitoring, leading to uninformed consent and privacy risk even if data stays local.

Vague Triggers

Medium
Confidence
96% confidence
Finding
The listed natural-language triggers are broad, generic phrases such as "Update my soul" and "Run soulforge" that can plausibly appear in ordinary conversation, increasing the chance of accidental activation. In an agent skill context, overly permissive invocation language can cause the skill to run on unintended user input and begin behavioral analysis or modification workflows the user did not explicitly mean to invoke.

Vague Triggers

Medium
Confidence
95% confidence
Finding
The skill advertises broad natural-language triggers such as 'reflect on me' and 'what patterns do you notice', plus automatic invocation after every 10 sessions. These phrases can plausibly appear in ordinary conversation, increasing the chance of unintended activation of a capability that reads session history and writes proposals based on persistent behavioral profiling.

Ssd 3

Medium
Confidence
97% confidence
Finding
The skill description explicitly states it watches conversations, decisions, tone, and recurring patterns across sessions, creating persistent behavioral summaries. Even if data stays local, this is sensitive profiling that can capture intimate traits, habits, and emotional states, and the automatic cross-session analysis increases privacy risk.

Ssd 3

Medium
Confidence
98% confidence
Finding
The passive observation section describes quiet accumulation of behavioral and emotional signals into local memory, including frustration, curiosity, blindspots, and decision patterns. This is sensitive personal-data collection and inference generation; storing it persistently creates privacy, misuse, and secondary exposure risks even without exfiltration.

Vague Triggers

Medium
Confidence
96% confidence
Finding
The documented trigger list mixes explicit commands with vague conversational prompts and provides no exclusion rules or disambiguation mechanism. In context, accidental activation is more serious because the skill performs reflective analysis over prior sessions and may surface sensitive inferences the user did not intend to request at that moment.

Ssd 3

Medium
Confidence
93% confidence
Finding
The file advertises itself as a passive collector that accumulates behavioral signals across sessions, which creates a persistent profiling store derived from user conversations. Even without network transmission, local retention of behavioral telemetry increases privacy and confidentiality risk, especially because the collected data is longitudinal and user-inference oriented.

Ssd 3

Medium
Confidence
97% confidence
Finding
The topic extractor intentionally captures free-text snippets following phrases like 'what I really want' or 'I've been thinking about,' then saves them into observations.json. In this skill context, that is more dangerous than generic analytics because it preserves semantically rich fragments of potentially sensitive user disclosures in plain language, making later leakage or misuse more harmful.

Missing User Warnings

Medium
Confidence
95% confidence
Finding
The skill persistently stores conversation-derived data to memory/observations.json without any explicit runtime notice, consent flow, retention limit, or sanitization. In this skill's context, it silently builds a longitudinal profile of user behavior, which can expose sensitive traits or disclosures if the local file is later accessed by another tool or person.

Intent-Code Divergence

Medium
Confidence
95% confidence
Finding
The security manifest is inaccurate because the code can read any path supplied via --gaps, not just session history files and observations.json. While this does not itself exfiltrate data, misleading file-access declarations weaken trust boundaries and can cause operators to permit broader local file reads than intended.

Intent-Code Divergence

Medium
Confidence
93% confidence
Finding
The security manifest is inaccurate: when --soul is supplied, the script dynamically imports observe.py and executes its check_aspiration_gap logic, which may read additional files or perform other side effects not disclosed in the manifest. In a skill environment, inaccurate disclosure of code execution paths undermines review assumptions and can hide broader local-file access or unexpected behavior behind an apparently simple reporting command.

Static analysis

No suspicious patterns detected.