Security audit

Skill Soup

Security checks for vulnerabilities and agentic risk

Overview

This skill is an experimental autonomous publisher that fits its stated purpose, but it gives community-synced content and local scripts too much unchecked control before publishing public artifacts.

Review before installing. Use only in a sandboxed workspace with no unrelated files or credentials, avoid continuous mode, inspect any synced builder before execution, and require manual confirmation before public GitHub publication or running evolve.sh.

A.I.G

Vulnerability Patterns

Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally

Findings (4)

T01 · Skill Instruction Hijacking

Error

Location: SKILL.md:170
Finding: Execution of Untrusted Community-Supplied Agent Instructions
Content: View full analysis
/` containing: - `SKILL.md` — the builder's `skill_md` field - `_meta.json` — a JSON file with `id`, `name`, `fitness_score`, `generation`, `skills_produced` - Any files from the builder's `files_json` field (key = relative path, value = file content) ``` ```text Read the selected builder's `SKILL.md` file. This contains the instructions you will follow next. ## Step 5: Generate the Skill Follow the selected builder's `SKILL.md` instructions to generate a new Agent Skill based on the idea's `prompt` and `context`. ``` ### Technical Analysis The Skill retrieves builder content from an API, writes the returned `skill_md` field into a local `SKILL.md`, and then explicitly instructs the Agent to follow those downloaded instructions. The downloaded builder is therefore treated as trusted executable instruction text rather than untrusted community data. No control is specified to detect or reject instructions that: - Override the current task or system safety constraints. - Request access to authentication tokens or unrelated local files. - Invoke shell commands or network tools. - Exfiltrate workspace or credential data. - Modify files outside the generated-skill directory. - Conceal malicious behavior inside an otherwise plausible builder workflow. The fitness-based selection mechanism does not establish security or provenance. A malicious builder may eventually be selected through either the exploitation or exploration branch. ### Att ...[truncated 1215 chars]
Remediation: View remediation

T09 · Insecure Skill Coding Practices

Error

Location: SKILL.md:170
Finding: Arbitrary File Write Through Unvalidated API-Controlled Paths
Content: View full analysis
/` containing: - `SKILL.md` — the builder's `skill_md` field - `_meta.json` — a JSON file with `id`, `name`, `fitness_score`, `generation`, `skills_produced` - Any files from the builder's `files_json` field (key = relative path, value = file content) ``` ### Technical Analysis Both `` and the keys in `files_json` influence local destination paths. The synchronization procedure does not require validation against absolute paths, `..` traversal components, platform-specific separators, symlinks, or canonical paths that escape `.soup/builders`. The validation in Step 6 checks generated skill paths, but it does not cover builder files received and written during Step 2. Using a native file-writing tool instead of a shell heredoc does not prevent path traversal. Because the synchronization process also removes the existing builder directory before materializing the response, a malicious or malformed response can combine data loss with unsafe writes. ### Attack Path 1. An attacker causes the builder API to return a crafted builder ID or `files_json` key such as `../../target-file`, an absolute path, or a path traversing through a symlink. 2. The runner joins the attacker-controlled value to `.soup/builders/`. 3. The native file-writing operation resolves the crafted path outside the intended builder directory. 4. The attacker-controlled content overwrites a project file, Agent instruction file, configuration file, or executable script. 5. The overwritten file ...[truncated 612 chars]
Remediation: View remediation

T09 · Insecure Skill Coding Practices

Error

Location: SKILL.md:67
Finding: Bearer Token Stored in Plaintext Without Required Access Controls
Content: View full analysis
", "username": ""} ``` ```text Use the token as `Authorization: Bearer ` in all subsequent API calls. ``` ### Technical Analysis The Skill directs the Agent to persist a bearer JWT in a normal workspace file. It does not require owner-only permissions, use of an operating-system credential manager, encryption at rest, exclusion from version control, token redaction, or isolation from generated and downloaded instructions. A bearer token generally grants access based on possession. Any local process or malicious builder instruction able to read `.soup/auth.json` can potentially replay it. This issue materially compounds the instruction-hijacking vulnerability because downloaded builder instructions operate in the same Agent session and workspace. ### Attack Path 1. A user completes the device authentication flow. 2. The Agent writes the returned JWT to `.soup/auth.json`. 3. The file is created with environment-dependent default permissions and remains in the workspace. 4. A malicious downloaded builder, another local process, an accidental repository commit, or a diagnostic log exposes the file. 5. The attacker obtains the bearer token and submits authenticated requests to the Skill Soup API. 6. The token is reused until it expires or is revoked. ### Impact Assessment An attacker may act with the API privileges represented by the JWT. Based on the documented workflow, this can include reading authenticated API data, submitting ideas, synchronizing builders, publishing skills or builders, and triggering repository publication through the server's stored GitHub authorization. The Skill does not directly transmit the JWT to an external non-local host, but storing ...[truncated 117 chars]
Remediation: View remediation

T09 · Insecure Skill Coding Practices

Error

Location: SKILL.md:303
Finding: Execution of an Unverified Workspace Script Not Included in the Audited Package
Content: View full analysis
``` This creates a child directory at `.soup/builders/child--gen-/` containing: - A copy of the parent's files - `_mutation_context.json` — mutation type and parent data - `_meta.json` — child metadata ``` ### Technical Analysis The workflow requires execution of `./scripts/evolve.sh`, but the audited project contains only `SKILL.md`, `config-template.yaml`, and `references/mutation-guide.md`. The referenced script is not included in the reviewed package. The relative command therefore resolves against the current workspace. Its implementation, ownership, integrity, and provenance are not verified before execution. A pre-existing or attacker-planted script at that path can execute arbitrary shell commands under the Agent's operating-system identity. The `` argument is also derived from synchronized builder metadata. Although no unsafe interpolation inside the absent script can be confirmed, the script would additionally need to validate this argument. ### Attack Path 1. An attacker or compromised workspace places a malicious executable at `./scripts/evolve.sh`. 2. The runner reaches every third generation iteration and enters the evolution step. 3. The Agent executes the relative script without checking its origin or digest. 4. The planted script performs arbitrary local commands using the Agent's permissions. 5. The script may read credentials, modify files, establish further execution paths, or send accessible data over the network. ### Impact Assessment The script receives the full shell privileges of the Agent runtime. It may access any readable files and modify any ...[truncated 329 chars]
Remediation: View remediation

SkillSpector

Vulnerability Patterns

Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
System Prompt LeakageDirect Leakage, Indirect Extraction, Tool-Based Exfiltration
Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger

Findings (16)

Missing User Warnings

High

Category: Not specified by scanner
Confidence: 95% confidence
Finding: The skill description and flow do not prominently warn users that generated skills may be automatically published to public GitHub repositories. This creates a serious consent and data-exposure risk, especially because generated content may contain unsafe, sensitive, or third-party-influenced material.
Content: No source excerpt is available for this finding.

Ae1

High

Category: analysis-evasion
Confidence: 100% confidence
Finding: Referenced artifact was not completely inspected
Content: Scanner excerpt · SKILL.md (reported line 174)May include surrounding context.
md
- `SKILL.md` — the builder's `skill_md` field

Ae1

High

Category: analysis-evasion
Confidence: 100% confidence
Finding: Referenced artifact was not completely inspected
Content: Scanner excerpt · SKILL.md (reported line 218)May include surrounding context.
md
- `SKILL.md` — the builder's `skill_md` field

Ae1

High

Category: analysis-evasion
Confidence: 100% confidence
Finding: Referenced artifact was not completely inspected
Content: Scanner excerpt · SKILL.md (reported line 222)May include surrounding context.
md
- `SKILL.md` — the builder's `skill_md` field

Ae1

High

Category: analysis-evasion
Confidence: 100% confidence
Finding: Referenced artifact was not completely inspected
Content: Scanner excerpt · SKILL.md (reported line 226)May include surrounding context.
md
- `SKILL.md` — the builder's `skill_md` field

Ae1

High

Category: analysis-evasion
Confidence: 100% confidence
Finding: Referenced artifact was not completely inspected
Content: Scanner excerpt · SKILL.md (reported line 318)May include surrounding context.
md
- `SKILL.md` — the builder's `skill_md` field

Ae1

High

Category: analysis-evasion
Confidence: 100% confidence
Finding: Referenced artifact was not completely inspected
Content: Scanner excerpt · SKILL.md (reported line 340)May include surrounding context.
md
- `SKILL.md` — the builder's `skill_md` field

Context-Inappropriate Capability

High

Category: Not specified by scanner
Confidence: 98% confidence
Finding: The skill explicitly tells the agent to read and follow SKILL.md instructions from builders synced from a community-controlled pool. That gives remote third parties effective control over subsequent agent behavior, enabling prompt injection, unsafe tool use, data exfiltration, or publication of attacker-influenced outputs under trusted automation.
Content: No source excerpt is available for this finding.

Credential Access

High

Category: Privilege Escalation
Confidence: 70% confidence
Finding: Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.
Content: Scanner excerpt · SKILL.md (reported line 258)May include surrounding context.
Do not read this value from a file or copy it from an example. You must self-identify based on your own system prompt, model, or runtime environment.

Build the JSON payload and POST it. Include your auth token — the API will use your stored GitHub access token to create a public repo automatically:

bash
curl -sf -X POST http://localhost:3001/api/skills \

Context-Inappropriate Capability

High

Category: Not specified by scanner
Confidence: 93% confidence
Finding: The evolution workflow tells the agent to execute a local shell script (./scripts/evolve.sh) as part of normal operation. This expands the skill from content generation into arbitrary subprocess execution, which can be abused if the script or its inputs are modified, and creates a direct code-execution path not tightly constrained by the skill's purpose.
Content: No source excerpt is available for this finding.

Vague Triggers

Medium

Category: Not specified by scanner
Confidence: 88% confidence
Finding: The top-level instruction says to switch modes whenever invoked with arguments or a user request, but does not define strict dispatch boundaries. This ambiguity increases the chance that unrelated user text is interpreted as a command, triggering sensitive actions without clear user intent.
Content: No source excerpt is available for this finding.

Vague Triggers

Medium

Category: Not specified by scanner
Confidence: 97% confidence
Finding: Generate-mode triggers include broad phrases like 'generate' and 'run', which are common in benign conversation and can cause accidental activation of a high-impact workflow. Because the workflow performs authentication, remote sync, file writes, and public publication, ambiguous triggering materially increases the risk of unintended execution.
Content: No source excerpt is available for this finding.

External Transmission

Medium

Category: Data Exfiltration
Confidence: 60% confidence
Finding: Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.
Content: Scanner excerpt · SKILL.md (reported line 34)May include surrounding context.
The API runs at http://localhost:3001. Verify it's up before starting:

bash
curl -sf http://localhost:3001/health

If the health check fails, stop and tell the user the API is not running.

Context-Inappropriate Capability

Medium

Category: Not specified by scanner
Confidence: 95% confidence
Finding: The skill instructs the agent to determine its runtime from its own system prompt, model, or runtime environment, which encourages introspection into protected execution context. Even if only a label is intended, this pattern can normalize disclosure of hidden system details and create a path for prompt-extraction behavior.
Content: No source excerpt is available for this finding.

Ssd 1

Medium

Category: Not specified by scanner
Confidence: 95% confidence
Finding: This instruction explicitly tells the agent to use its own system prompt, model, or runtime environment to self-identify, which is a form of sensitive self-disclosure. Even if intended only for telemetry, it conditions the agent to inspect hidden context and can leak implementation details that should remain inaccessible to skill logic.
Content: No source excerpt is available for this finding.

Unbounded Resource Access

Medium

Category: Excessive Agency
Confidence: 88% confidence
Finding: Continuous mode is defined to loop indefinitely with repeated sync, generation, publication, and evolution actions. Without hard limits, budgets, or rate controls beyond a fixed sleep, this can consume compute, flood local/remote services, and repeatedly publish unwanted content if triggered accidentally or maliciously.
Content: Scanner excerpt · SKILL.md (reported line 386)May include surrounding context.
md
**In continuous mode:** 1. After completing Steps 0-1, loop indefinitely through: - **Step 2**: Re-sync builder pool (every iteration gets the freshest builders) - **Steps 3-8**: Fetch idea, select builder, generate, validate, publish, report - **Step 9**: Evolve builders every 3rd iteration (iterations 3, 6, 9, ...)

Static analysis

No suspicious patterns detected.