Back to skill

Security audit

Multi-Skill-Eval | 集成化技能评估系统

Security checks for vulnerabilities and agentic risk

Overview

The skill appears intended for skill evaluation, but it needs Review because parts of the audit workflow are under-scoped and the generated reports can include unsafe unescaped content.

Review before installing. Use it only on skills you are willing to expose to the selected model/provider, run third-party scanners in an isolated environment with pinned versions, and avoid opening or publishing generated leaderboards from untrusted cards until HTML escaping is fixed.

Vulnerability Patterns
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (3)

T09 · Insecure Skill Coding Practices

Error
Location
scripts/generate_leaderboard.py:115
Finding

Stored HTML Injection and Cross-Site Scripting in Generated Leaderboards

Content
View full analysis
Skill Leaderboard | {args.title}") ``` ```python html.append(f"

{args.title}

") ``` ```python strengths = ", ".join(card.get("strengths", [])[:2]) if card.get("strengths") else "-" weaknesses = ", ".join(card.get("weaknesses", [])[:2]) if card.get("weaknesses") else "-" html.append(" ") html.append(f" {i}") html.append(f" {card.get('name', 'Unknown')}") html.append(f" {card.get('slug', 'N/A')}") html.append(f" {score}/10") html.append(f" {card.get('quality', 0)}/5") html.append(f" {card.get('delta', 0)}/3") html.append(f" {card.get('efficiency', 0)}/2") html.append(f" {rec}") html.append(f" {card.get('eval_date', 'N/A')}") html.append(f"
👍 {strengths}
👎 {weaknesses}
") html.append(" ") ``` ### Technical Analysis The leaderboard generator reads metadata, strengths, and weaknesses from Skill Card Markdown files and interpolates those values directly into an HTML document. The user-provided `--title` argument is handled in the same way. None of these values are HTML-escaped. Because HTML metacharacters remain intact, an attacker-controlled value can break out of its intended text context and create executable markup. For example, a skill name containing an image element with an event handler would be written into the resulting leaderboard as active HTML rather than displayed as text. This is a stored injection issue because the payload can first be placed in a ...[truncated 1577 chars]
Remediation
View remediation

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/generate_skill_card.py:215
Finding

Unvalidated Metadata and Benchmark Content Propagated into Generated Skill Cards

Content
View full analysis
Remediation
View remediation

T08 · Insecure Dependencies

Warning
Location
SKILL.md:338
Finding

Unpinned Third-Party Package Retrieval and Execution in Documented Audit Workflow

Content
View full analysis
Remediation
View remediation
scan /path/to/skill ``` 2. Record and verify package integrity where the package manager and distribution workflow support it. 3. Use a committed lockfile for maintained Node.js execution environments. 4. Run third-party scanners in an isolated container or sandbox with: - Read-only access to the target directory. - No access to home-directory credentials. - A minimal environment. - Restricted network access. - No unnecessary host filesystem mounts. 5. Pin PyYAML to a reviewed compatible version or constrained version range in a requirements file. 6. Use hash-locked Python dependencies, such as `pip install --require-hashes -r requirements.txt`, for reproducible installations. 7. Document the source, reviewed version, and expected permissions of every external scanner. 8. Avoid recommending immediate execution of floating package versions in security-sensitive workflows. ]]>
Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Rogue AgentSelf-Modification, Session Persistence
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
Findings (21)

Tp4

High
Category
MCP Tool Poisoning
Confidence
87% confidence
Finding

The declared description promises an integrated, multi-method evaluation platform that combines static analysis, rubric scoring, and autonomous benchmark testing, with coverage including 25-item rubric scores, multi-model benchmarks, trigger-word detection, and ghost-tool detection. The supplied code chunk does only a subset of that: automated static/structural and heuristic checks for one skill directory. It explicitly states that it covers only the automatable portion and that full rubric evaluation requires a separate manual rubric. There is no implementation of rubric scoring, benchmark execution, multi-model testing, skill comparison, batch processing, trigger-word detection logic, or ghost-tool detection in this code. Therefore the description materially overstates the implemented capabilities of the provided code chunk.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding

代码的核心行为是从现有 skill card Markdown 中提取已经生成好的评估结果,并将其汇总为 HTML 排行榜。这属于评估结果的可视化/报告生成工具,而不是实际执行评估的系统。虽然页脚出现了“Generated by multi-skill-eval”字样,且输入内容是“skill cards”,但代码本身没有分析技能源码、没有运行测试、没有依据 rubric 打分、也没有进行模型基准对比。因此其实际能力与声明的主要用途存在实质性不符。

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
94% confidence
Finding

The declared description promises an integrated, multi-method skill assessment platform with static analysis, rubric scoring, benchmarking, documentation/code quality coverage, trigger detection, and ghost-tool detection. The supplied code only implements assertion-based grading for existing benchmark result files in a workspace. It performs configurable content/file checks and summarizes pass/fail outcomes, which aligns only with a narrow slice of benchmark-result evaluation. Because the primary scope and capabilities are substantially narrower than the declared system, this is a material description-behavior mismatch.

Content

No source excerpt is available for this finding.

External Model or Provider Selection

High
Category
Excessive Agency
Confidence
90% confidence
Finding

The skill encourages selection of external models/providers for benchmark execution, which can send skill content, prompts, and outputs to third-party services. In a skill-audit context, this is more sensitive because evaluated skills may contain proprietary logic, secrets in examples, or unsafe prompts that should not be forwarded externally by default.

Content

Scanner excerpt · SKILL.md (reported line 41)May include surrounding context.

multi-skill-eval --all

指定模型进行基准测试

multi-skill-eval ~/.openclaw/skills/my-skill --method benchmark --model minimax/MiniMax-M2

text

---

External Model or Provider Selection

High
Category
Excessive Agency
Confidence
90% confidence
Finding

The benchmark workflow explicitly depends on AI-agent orchestration with external model selection, which may expose prompts, files, and benchmark artifacts to remote providers. Because this skill is designed to inspect other skills, the context increases risk: it may process untrusted content and transmit it off-box, potentially leaking internal code or sensitive evaluation data.

Content

Scanner excerpt · SKILL.md (reported line 126)May include surrounding context.

bash
# Spawn benchmark via AI agent
multi-skill-eval /path/to/skill --method benchmark --model claude-sonnet-4

⚠️ Note: The benchmark method requires an AI agent to orchestrate subagent execution. The CLI coordinates the workflow but actual execution happens through AI agent sessions.

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 135)May include surrounding context.

md
1. Read `SKILL.md` — understand claims, dependencies, target use cases

Self-Modification

High
Category
Rogue Agent
Confidence
92% confidence
Finding

The skill includes guidance to rewrite SKILL.md, effectively enabling self-modification of another skill's instructions. In an adversarial setting, automatic rewriting of security-relevant instructions can remove safeguards, introduce malicious behavior, or corrupt the original artifact, especially if driven by untrusted benchmark outputs or prompts.

Content

Scanner excerpt · SKILL.md (reported line 282)May include surrounding context.

md
- Overhead Routing: Add quick-mode vs full-framework routing
   - Assertion-Aligned Rewrite: Rewrite to pass specific failed assertions

4. **Rewrite SKILL.md** with selected strategy:
   - Default: Remove > Add (delete 60-80% first, then add behavioral mandates)
   - Add specific, enforceable conventions
   - Remove redundant content model already handles

Anti-Refusal Statement

High
Category
Anti-Refusal
Confidence
80% confidence
Finding

Skill instructs the agent to omit warnings, disclaimers, or ethical commentary. Stripping safety caveats hides risk from the user and is a common jailbreak preamble.

Content

Scanner excerpt · references/rubric.md (reported line 147)May include surrounding context.

md
- **3:** Confirmation on destructive ops. Some recoverability.
- **2:** Confirmation exists but permanent is the default.
- **1:** No confirmation on destructive operations.
- **0:** Destructive operations with no warning.

---

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
90% confidence
Finding

The skill describes operations that involve shell execution, file reads/writes, environment access, and network use, but it does not declare any explicit tool scope or permission boundaries. In an agent environment, this increases the risk of over-broad execution because evaluators or orchestrators may grant more capability than the task actually requires.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
83% confidence
Finding

The frontmatter description and trigger definitions are explicitly limited to Chinese and English, with no indication that the skill can adapt to the user's preferred language or locale. This creates a natural-language policy concern because the skill appears to prescribe supported languages rather than offering an opt-in or language choice.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
85% confidence
Finding

Using npx skilllens without pinning a version allows the latest published package to be fetched at runtime, creating a supply-chain risk. If the upstream package is compromised or unexpectedly changed, users running the documented command could execute unreviewed code in their local environment.

Content

No source excerpt is available for this finding.

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
75% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · references/rubric.md (reported line 146)May include surrounding context.

md
- **4:** Destructive ops default to recoverable (trash). Undo available. Confirmation prompts.
- **3:** Confirmation on destructive ops. Some recoverability.
- **2:** Confirmation exists but permanent is the default.
- **1:** No confirmation on destructive operations.
- **0:** Destructive operations with no warning.

---

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
75% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · references/rubric.md (reported line 174)May include surrounding context.

md
- **4:** Dry-run defaults, confirmation prompts, safe defaults (trash > delete).
- **3:** Most write ops have confirmation. Safe defaults.
- **2:** Some write ops unprotected.
- **1:** Write ops fire without confirmation.
- **0:** Silent data destruction possible.

---

Session Persistence

Medium
Category
Rogue Agent
Confidence
75% confidence
Finding

Skill establishes unauthorized persistence across sessions via cron jobs, startup scripts, or state files. Session persistence allows an attacker to maintain access beyond the current interaction.

Content

Scanner excerpt · scripts/eval-skill.py (reported line 315)May include surrounding context.

python
"html", "http", "imaplib", "importlib", "inspect", "io", "ipaddress",
        "itertools", "json", "keyword", "linecache", "locale", "logging",
        "lzma", "math", "mimetypes", "multiprocessing", "operator", "os",
        "pathlib", "pickle", "platform", "plistlib", "pprint", "profile",
        "queue", "random", "re", "readline", "reprlib", "secrets",
        "select", "shelve", "shlex", "shutil", "signal", "smtplib",
        "socket", "sqlite3", "ssl", "stat", "statistics", "string",

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
98% confidence
Finding

The script injects untrusted data from markdown cards and the CLI title directly into HTML without escaping. Fields such as name, slug, eval_date, strengths, weaknesses, and title can contain HTML or script payloads, which would execute when someone opens the generated leaderboard, creating a stored XSS issue in the generated artifact.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The generated strengths and related recommendation text are emitted in Chinese string literals, and the script provides no option for the user to choose language or locale. This is a natural-language policy concern because it forces a specific language regardless of user preference or environment.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
98% confidence
Finding

The recommendation text shown in the generated skill card is fixed to Chinese phrases, while the rest of the script interface and documentation are in English. Because there is no opt-in, prompt, or documented locale constraint, this forces a language choice on users.

Content

No source excerpt is available for this finding.

compile() call detected

Medium
Category
Dangerous Code Execution
Confidence
65% confidence
Finding

compile() creates code objects from strings. When combined with exec()/eval(), it enables obfuscated code execution.

Content

Scanner excerpt · scripts/static-analyze.py (reported line 113)May include surrounding context.

python
# Syntax check
            try:
                compile(code, fpath, "exec")
            except SyntaxError as e:
                issues.append({"severity": "error", "check": f"syntax ({f})", "msg": f"Syntax error at line {e.lineno}: {e.msg}"})
                score -= 10

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
87% confidence
Finding

The changelog states that Chinese trigger words and Chinese usage scenarios were added, which suggests locale-specific behavior. Because the file does not indicate any user opt-in, language selection, or justification for restricting or biasing activation toward Chinese, this may conflict with language/locale policy expectations.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
94% confidence
Finding

The generated document hard-codes lang='zh-CN', which imposes a specific language/locale on all output regardless of user preference. The script also emits several Chinese UI labels, and there is no option to choose locale or language at runtime.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
83% confidence
Finding

This code opens and reads every .py file under the target skill's scripts/ directory, which is a filesystem access operation on user-provided content. While this behavior is part of analysis logic, there is no inline comment, log message, or prompt at the point of access explaining that local files will be read.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.