Back to skill

Security audit

Skill Forge

Security checks for vulnerabilities and agentic risk

Overview

This is a plausible skill and test-suite generator, but it under-documents how untrusted specs become agent instructions and claims safeguards the code does not implement.

Review generated SKILL.md files before installing them, especially when the spec comes from another person or includes critical constraints. Use private output directories, be cautious with --auto-improve because it can mutate skill instructions through another local tool, and do not rely on the documented calibration/atomic-write safeguards until the implementation actually provides them.

Vulnerability Patterns
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (2)

T01 · Skill Instruction Hijacking

Warning
Location
scripts/skill_generator.py:88
Finding

Untrusted Specification Content Is Amplified into Agent Instructions

Content
View full analysis
"SkillSpec": """Load a SkillSpec from a YAML file.""" data = yaml.safe_load(path.read_text()) return cls( **{k: v for k, v in data.items() if k in cls.__dataclass_fields__} ) def validate(self) -> list[str]: """Return a list of validation errors (empty if valid).""" errors = [] if not self.name: errors.append("name is required") if not self.purpose: errors.append("purpose is required") if len(self.name) > 50: errors.append("name too long (max 50 chars)") if not self.name.replace("-", "").replace("_", "").isalnum(): errors.append( "name must contain only alphanumeric, hyphens, underscores" ) return errors ``` ### Technical Analysis The `critical_constraints` field is loaded directly from the input YAML and interpolated into the generated `SKILL.md`. Its values receive no type, length, content, or policy validation. The generator then strengthens each value with imperative language such as `MUST`, `non-negotiable`, and `NEVER skip or rationalize around them`. Because `SKILL.md` is intended to ...[truncated 2021 chars]
Remediation
View remediation

T09 · Insecure Skill Coding Practices

Note
Location
scripts/task_suite_generator.py:356
Finding

Output Files Are Written Non-Atomically and Existing Symbolic Links Are Followed

Content
View full analysis
Remediation
View remediation
Vulnerability Patterns
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • Behavioral ASTexec() Call, eval() Call, Dynamic Import
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Findings (18)

Tp2

High
Category
MCP Tool Poisoning
Confidence
85% confidence
Finding

Mixing characters from multiple Unicode scripts in a single identifier is a common technique to create visually ambiguous tool names.

Content

No source excerpt is available for this finding.

Tp2

High
Category
MCP Tool Poisoning
Confidence
85% confidence
Finding

Mixing characters from multiple Unicode scripts in a single identifier is a common technique to create visually ambiguous tool names.

Content

No source excerpt is available for this finding.

Tp2

High
Category
MCP Tool Poisoning
Confidence
85% confidence
Finding

Mixing characters from multiple Unicode scripts in a single identifier is a common technique to create visually ambiguous tool names.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding

Regardless of wording variation, the core issue is the same: the skill's declared primary capability does not match the observed implementation. For a tool that is supposed to produce runnable evaluation suites, this mismatch undermines trust in outputs and can silently degrade governance, testing coverage, and release decisions.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

Regardless of wording variation, the core issue is the same: the skill's declared primary capability does not match the observed implementation. For a tool that is supposed to produce runnable evaluation suites, this mismatch undermines trust in outputs and can silently degrade governance, testing coverage, and release decisions.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
92% confidence
Finding

Regardless of wording variation, the core issue is the same: the skill's declared primary capability does not match the observed implementation. For a tool that is supposed to produce runnable evaluation suites, this mismatch undermines trust in outputs and can silently degrade governance, testing coverage, and release decisions.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

Regardless of wording variation, the core issue is the same: the skill's declared primary capability does not match the observed implementation. For a tool that is supposed to produce runnable evaluation suites, this mismatch undermines trust in outputs and can silently degrade governance, testing coverage, and release decisions.

Content

No source excerpt is available for this finding.

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 160)May include surrounding context.

md
| `--from-spec` | `SKILL.md` + `task_suite.yaml` | `<output>/<name>/SKILL.md`, `<output>/<name>/task_suite.yaml` |

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 162)May include surrounding context.

md
| `--from-spec` | `SKILL.md` + `task_suite.yaml` | `<output>/<name>/SKILL.md`, `<output>/<name>/task_suite.yaml` |

Description-Behavior Mismatch

High
Category
Not specified by scanner
Confidence
97% confidence
Finding

The manifest explicitly says this skill is not for driving the improvement loop and directs that use case to improvement-orchestrator. However, the code exposes an --auto-improve mode and later executes the improvement-orchestrator script, making end-to-end improvement-loop execution part of this skill's actual behavior.

Content

No source excerpt is available for this finding.

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
89% confidence
Finding

The skill advertises file read, file write, and shell-oriented workflows but does not declare any explicit tool scope such as permissions or allowed-tools. In an agent environment, this creates an authorization ambiguity where the model may invoke broader filesystem or shell capabilities than intended, increasing the risk of overreach or unsafe command execution.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
93% confidence
Finding

Natural-language policy review applies to all file types. The frontmatter description is written in Chinese while the rest of the document is in English, and there is no indication that the skill is region-specific or that users may choose their preferred language/locale.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
87% confidence
Finding

Several triggers such as 'generate skill' and 'generate task suite' are broad enough to match ordinary user requests outside the intended narrow context. Over-broad activation can cause the wrong skill to be invoked, leading to unintended file operations, shell use, or automation behavior in contexts where the user did not explicitly request this tool.

Content

No source excerpt is available for this finding.

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/forge.py (reported line 180)May include surrounding context.

python
import subprocess

    result = subprocess.run(
        [
            sys.executable,
            str(evaluator_script),

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
86% confidence
Finding

This skill's stated purpose centers on generating SKILL.md and task suites, with the manifest specifically excluding driving the improvement loop. The code nonetheless launches a separate orchestrator program through subprocess, which is a stronger control capability than is justified by a generator-focused skill.

Content

No source excerpt is available for this finding.

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/forge.py (reported line 214)May include surrounding context.

python
import subprocess

    result = subprocess.run(
        [
            sys.executable,
            str(orchestrator_script),

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
84% confidence
Finding

The trigger generator derives activation phrases directly from free-form purpose text using minimal filtering, which can produce overly broad or policy-insensitive triggers. In a skill-routing system, this can cause unintended invocation of the skill for unrelated or sensitive requests, increasing the chance of prompt/behavior misrouting and misuse at runtime.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
80% confidence
Finding

This code's stated purpose is to generate SKILL.md content from a spec, which implies downstream creation or overwrite of a markdown artifact, but the module-level and function docstrings do not warn about possible file creation or replacement effects. For code-file warning checks, there is no visible confirmation prompt or explicit user-facing disclosure about that data-affecting behavior in the file.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.