Security audit

Auto Improvement Orchestrator Skill

Security checks for vulnerabilities and agentic risk

Overview

This skill is not clearly malicious, but it can automatically change skills, read local conversation histories, and set up persistent unattended improvement loops with some confusing safety documentation.

Install only if you intentionally want an automated skill-improvement system that may edit target skills. Avoid the cron setup unless you add your own expiry, status checks, and approval gates. Run feedback analysis with --no-snippets and narrow --session-dir or --skill-filter values. Review generated diffs, receipts, backups, and any configured LLM endpoint before allowing auto mode.

A.I.G

Vulnerability Patterns

System PersistenceInstalls backdoors, hooks, services, or scheduled tasks that survive the run
Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review

Findings (4)

T06 · System Persistence

Error

Location: skills/autoloop-controller/references/scheduling-guide.md:3
Finding: Non-expiring system cron job enables persistent unattended execution
Content: View full analysis
&1 | tee -a "${STATE_ROOT}/autoloop.log" ``` The controller then starts the orchestrator in automatic mode: ```python cmd = [ sys.executable, str(ORCHESTRATOR_SCRIPT), "--target", str(target), "--state-root", str(state_root), "--auto", ] proc = subprocess.run(cmd, capture_output=True, text=True, timeout=3600) ``` ### Technical Analysis The scheduling guide recommends registering an operating-system cron entry that executes every four hours, survives terminal and Agent sessions, and has no expiration. Although recurring evaluation is consistent with the controller's declared scheduling functionality, recommending permanent system-level persistence exceeds the privileges needed for an ordinary one-shot or supervised evaluation. The scheduled command ultimately invokes the improvement orchestrator with `--auto`. Consequently, evaluation and Skill mutation can continue without a currently active user session or per-run approval. The shell wrapper quotes its parameters correctly, and the Python subprocess uses an argument list rather than `shell=True`, so no direct shell injection was identified in this path. The risk arises from persistence and unattended authority rather th ...[truncated 1116 chars]
Remediation: View remediation

T09 · Insecure Skill Coding Practices

Error

Location: skills/session-feedback-analyzer/scripts/analyze.py:89
Finding: Claude session histories are recursively collected and retained with plaintext user-message snippets
Content: View full analysis
Remediation: View remediation

T09 · Insecure Skill Coding Practices

Error

Location: skills/benchmark-store/interfaces/hidden_tests.py:330
Finding: Hidden benchmark data uses unauthenticated repeating-key XOR and unsalted password derivation
Content: View full analysis
bytes: """派生解密密钥""" # 简化实现：实际应使用 PBKDF2 或 Argon2 return hashlib.sha256(password.encode()).digest() ``` The ciphertext is decrypted using repeating-key XOR: ```python def _decrypt(self, encrypted_data: bytes, salt: bytes) -> Any: """解密数据""" if self._decryption_key is None: raise RuntimeError("Test suite is locked. Call unlock() first.") # 简化实现：实际应使用 AES-GCM 或 ChaCha20-Poly1305 # 这里使用简单的 XOR 作为演示 key = self._decryption_key decrypted = bytes([b ^ key[i % len(key)] for i, b in enumerate(encrypted_data)]) return json.loads(decrypted.decode()) ``` Creation uses the same construction: ```python # 生成盐值 salt = secrets.token_bytes(16) # 派生密钥 key = hashlib.sha256(password.encode()).digest() # 加密数据 def encrypt(data: Any) -> bytes: json_data = json.dumps(data, default=str).encode() return bytes([b ^ key[i % len(key)] for i, b in enumerate(json_data)]) ``` The generated salt is not used in key derivation or encryption. ### Technical Analysis SHA-256 is a fast hash and is unsuitable as a password-based key-derivation function. Because no salt is incorporated, the same password always yields the same key across suites and tests, enabling precomputation and cross-file correlation. Repeating-key XOR does not provide semantic security or integrity. The protected values are JSON, which supplies predictable plaintext such as braces, quotation marks, field names, and common values. Known-plaintext and multi-ciphertext analysis can therefore reveal portions of the keystream. An attacker can also perform rapid offline pass ...[truncated 1638 chars]
Remediation: View remediation

T09 · Insecure Skill Coding Practices

Warning

Location: skills/improvement-discriminator/interfaces/llm_judge.py:100
Finding: Sensitive Skill content may be transmitted to external or attacker-controlled LLM endpoints
Content: View full analysis
str: plan = candidate.get("execution_plan", {}) proposed_content = "\n".join(plan.get("content_lines", ["(no content specified)"])) return JUDGE_PROMPT_TEMPLATE.format( target_content=target_content[:2000] if target_content else "(not available)", category=candidate.get("category", "unknown"), risk_level=candidate.get("risk_level", "unknown"), description=candidate.get("proposed_change_summary", candidate.get("title", "")), proposed_content=proposed_content[:1000], ) ``` The Claude backend accepts an unrestricted endpoint from configuration or the environment: ```python def _call_claude(self, prompt: str) -> str: """Call Claude API. Supports custom base_url (e.g. DashScope proxy) via config or env.""" try: import anthropic kwargs = {} base_url = self.config.base_url or os.environ.get("ANTHROPIC_BASE_URL") if base_url: kwargs["base_url"] = base_url client = anthropic.Anthropic(**kwargs) response = client.messages.create( model=self.config.model, max_tokens=self.config.max_tokens, temperature=self.config.temperature, messages=[{"role": "user", "content": prompt}], ) ``` The OpenAI backend sends the same prompt to its configured service: ```python client = openai.OpenAI() response = client.chat.completions.create( model=self.config.model if "gpt" in self.config.model else "gpt-4o-mini", max_tokens=self.config.max_tokens, temperature=self.config.temperature, messages=[{"role": "user ...[truncated 2106 chars]
Remediation: View remediation

SkillSpector

Vulnerability Patterns

MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code

Findings (276)

Tp2

High

Category: MCP Tool Poisoning
Confidence: 85% confidence
Finding: Mixing characters from multiple Unicode scripts in a single identifier is a common technique to create visually ambiguous tool names.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 99% confidence
Finding: The declared description presents a sophisticated multi-component skill evaluation/improvement framework. The actual code chunk contains only a trivial main function that prints a message and exits successfully. There is no evidence of scoring dimensions, LLM-as-judge logic, weighted roles, category correction, regression safeguards, retry mechanisms, or any pipeline/orchestration behavior. This is a material mismatch in primary purpose and capabilities.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 95% confidence
Finding: The declared description describes a sophisticated auto-evaluation and self-improvement pipeline with scoring, Pareto protections, retries, and orchestration behavior. The supplied code does not implement or expose that functionality. Instead, it is a simple package interface file that imports and re-exports benchmark and hidden-test related classes. While benchmark/test infrastructure could be a supporting component in a broader evaluation system, this specific chunk’s primary purpose is benchmark-store interface aggregation, not the declared auto-improvement pipeline behavior. Therefore the description does not accurately represent what this code chunk actually does.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 96% confidence
Finding: The declared description promises a sophisticated automated evaluation-and-improvement pipeline with multiple specialized scoring and regression-protection mechanisms. The supplied code chunk does something much narrower: it implements immutable benchmark data structures, integrity verification, a protocol for evaluators, benchmark execution, simple aggregate reporting, and result export. While this is related to evaluation infrastructure, it does not implement the core advertised capabilities such as auto-improvement, LLM judging, multi-role weighting, Pareto regression guards, or retry logic. The primary purpose of this code is a frozen benchmark interface/runner, which is only a supporting component at best, not an accurate representation of the declared skill behavior.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 98% confidence
Finding: The declared description presents a broad automated evaluation-and-improvement pipeline with sophisticated scoring, weighted roles, regression protection, and retry/orchestration behavior. This code chunk instead provides a narrow benchmark/hidden-test infrastructure component. Its main purpose is secure-ish storage, loading, decryption, execution, and validation of hidden test cases for a skill under test. While hidden testing could support a larger evaluation pipeline, the specific claimed capabilities—LLM judging, multidimensional scoring, weighted role aggregation, category correction coefficients, Pareto front protection, and retry logic—are absent from this code. Therefore the description does not accurately represent what this supplied code chunk actually does.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 95% confidence
Finding: The declared description promises a sophisticated end-to-end auto-evaluation and improvement pipeline with specific scoring dimensions, weighting rules, regression guards, and retry behavior. The supplied code chunk instead provides a benchmark database manager: it initializes SQLite tables, stores benchmark definitions and evaluation summaries, lists/deletes benchmarks, loads defaults, and produces a leaderboard. Its compare function merely iterates stored benchmarks and delegates scoring to an external evaluator callable, storing only an overall score. It does not implement the advertised scoring framework, orchestration, improvement loop, retry system, or Pareto protections. This is a material description-behavior mismatch, even though benchmark storage could be a supporting component within a larger system.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 94% confidence
Finding: The description claims a broad automatic evaluation-and-improvement pipeline with specific scoring dimensions, role weighting, category correction, Pareto regression guards, and retry behavior. The supplied code chunk is a narrower critic engine focused on evaluation only. It runs benchmarks, hidden tests, assertions, and optionally incorporates external regression and human review scores, then reports results. Crucially, the named mechanisms in the description are not present in this code: there is no 9-dimension structure scoring, no 4-role weighting logic, no category correction coefficients, no Pareto-front regression protection with the listed thresholds, and no trace-aware retry orchestration. Additionally, the code has an undeclared capability to dynamically import and execute Python modules from arbitrary paths as 'real skills,' which is materially more powerful than a simple evaluator description. Therefore the declared description does not accurately represent this code chunk.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 95% confidence
Finding: The code’s actual purpose is much narrower than the description. It performs local adaptation of candidate metadata into evaluator-friendly evidence, checks file structure, applies fixed heuristic scoring weights, derives a verdict, and returns recommendations/limitations. The module itself even states it is a 'Phase 1 minimal' adapter and that frozen benchmarks, hidden tests, and external regression are not run. This materially differs from the declared end-to-end auto-evaluation/improvement pipeline with advanced judging, weighting, Pareto regression safeguards, and retry/orchestration behavior. While the general theme of skill evaluation is related, the described capabilities are substantially overstated relative to this code chunk.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 86% confidence
Finding: The code partially matches the description in that it does implement automated evaluation/ranking, optional LLM-as-Judge, and a 4-role reviewer panel. However, several prominent declared features are absent or materially different in this specific chunk: there is no 9-dimensional scoring structure, no Pareto-front regression protection logic, no trace-aware retry behavior, and the category weighting scheme does not use the declared tool/knowledge/orchestration/rule taxonomy. The implemented script is narrower: it scores and ranks candidate modifications, emits recommendations, and updates state. Because the declared description prominently advertises capabilities not present in the supplied code chunk, this is a meaningful description-behavior mismatch.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 89% confidence
Finding: The declared description presents a full automatic evaluation-and-improvement pipeline with advanced scoring dimensions, role weighting, category coefficients, Pareto regression safeguards, and retry behavior. The supplied code chunk does not implement that pipeline behavior; instead, it is a test suite for evaluator-related modules. Its primary purpose is verification of components such as assertions, critic engine behavior, report export, evaluator loading, and integration between scripts and interfaces. These are related to evaluation infrastructure, but materially different from the declared end-user skill purpose of automated improvement orchestration. No suspicious extra permissions/resources are used beyond local file and temp-file operations, but the main behavior is still mismatched because this chunk is test code rather than the described pipeline capability.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 95% confidence
Finding: The declared description portrays a broad automatic evaluation-and-improvement pipeline with sophisticated scoring and regression-control mechanisms. This code chunk does not implement that overall functionality. Instead, it narrowly evaluates candidate versus baseline skill performance by running a YAML-defined task suite, checking prerequisite availability, optionally skipping low-scoring candidates, caching baseline results, aborting when the baseline appears broken, and writing JSON artifacts. While the script does include one declared theme—evaluation, including support for an 'llm-rubric' judge type in schema validation—it does not implement most of the headline capabilities from the description, especially the 9D scoring, weighted roles, category coefficients, Pareto-front protections, or trace-aware retries. Therefore the description materially overstates and misrepresents what this supplied code chunk actually does.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 98% confidence
Finding: The declared description emphasizes automated evaluation and improvement orchestration: scoring, judging, Pareto safeguards, retries, and pipeline management. The supplied code chunk does not implement evaluation or improvement logic. Instead, it provides a rollback tool that restores files using backup metadata from receipts/execution artifacts or direct backup paths. This is a materially different primary purpose and introduces filesystem restoration capabilities not reflected in the declared description. While rollback could be a supporting component of a broader improvement system, this specific code chunk’s behavior is not accurately represented by the declared purpose.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 95% confidence
Finding: The declared description emphasizes an automated evaluation/improvement pipeline with scoring, LLM-as-judge, role weighting, category coefficients, Pareto regression protection, and retry orchestration. The supplied code does not implement or test those evaluation capabilities. Instead, it tests low-level executor behaviors for editing markdown files and YAML frontmatter and recording execution traces for those edits. While such editing could be a supporting component of a broader auto-improvement system, the chunk’s actual primary behavior is document transformation/execution, which is not accurately represented by the declared purpose. There is no evidence here of the advertised 9-dimensional scoring, weighted judging, Pareto protection, or pipeline orchestration logic.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 95% confidence
Finding: The description portrays a broad automatic evaluation-and-improvement pipeline centered on multi-dimensional scoring, weighted roles, category coefficients, Pareto regression guards, and retry behavior. This code chunk instead serves as a downstream gating component that consumes prior artifacts and enforces a six-layer validation plus promotion/revert/reject workflow. While there is some thematic overlap (evaluation, LLM/judge-related review, regression checks), the implemented behavior is materially different in scope and function. The code does not implement the declared 9-dimensional scoring system, 4-role weighting, category correction coefficients, explicit Pareto-front thresholds, or trace-aware retry logic. Conversely, it performs undeclared operational actions such as restoring files from backup, appending veto/pending queues, creating human review requests, and updating persistent state. Therefore the supplied code does not accurately match the declared description.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 98% confidence
Finding: The declared description describes an automated evaluation/improvement pipeline for skills, with scoring, optimization, and retry mechanisms. The supplied code does not implement scoring, automatic improvement, Pareto protection, category weighting, LLM judging, or orchestration logic. Instead, its primary purpose is a manual review gate CLI that reads and writes review request records under a state directory. This is a materially different function from the declared purpose, so it should be flagged as a mismatch.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 97% confidence
Finding: The supplied code does not implement or test the declared auto-evaluation/improvement pipeline features. Instead, it focuses on a separate gate system that validates candidate changes through layered checks and optional human review. While there is a small overlap with the broad notion of regression protection and quality control, the concrete behavior is materially different: no scoring model, no LLM judging, no weighted multi-role evaluation, no Pareto threshold logic, and no retry/improvement orchestration are present. The primary purpose of this code chunk is gate validation testing, not automated skill assessment and improvement as described.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 95% confidence
Finding: The declared description emphasizes an automated evaluation/improvement pipeline with scoring, Pareto regression protection, and retry orchestration. The supplied code chunk does not implement or test those capabilities. Instead, it focuses on a separate human-review gate: managing review request JSON files, listing pending reviews, completing them with approve/reject decisions, and validating CLI behavior. While such a review gate could be adjacent to an improvement pipeline, this specific code chunk’s primary purpose is materially different from the declared automatic evaluation/improvement functionality.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 90% confidence
Finding: The declared description presents a comprehensive automated evaluation-and-improvement pipeline centered on scoring, judging, weighted evaluation, regression protection, and retries. The supplied code chunk does something narrower: it proposes candidate edits for a target skill, optionally using evaluator failure inputs or LLM analysis, then records those proposals for a later 'critic' step. It does include a limited trace-aware retry adjustment, which partially aligns with the description, but the major advertised capabilities—multi-dimensional scoring, judge framework, weighted roles, category coefficients, and Pareto regression guarding—are absent from this code. This is a material description-behavior mismatch rather than a mere implementation detail gap.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 99% confidence
Finding: The code chunk does not implement a general auto-evaluation/improvement pipeline for skills. Instead, it performs a narrow static audit of a single prompt file for prompt-hardening patterns. Its primary purpose is materially different from the declared description, and none of the distinctive declared mechanisms (LLM judging, weighted multi-role scoring, category coefficients, Pareto regression guards, retry logic, or orchestration across multiple skills/tools) are present in the snippet. This is not merely a supporting detail; it is an unrelated capability with different triggers centered on prompt hardening rather than skill quality auto-improvement.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 99% confidence
Finding: The declared description describes a sophisticated automatic skill evaluation and self-improvement pipeline with multiple scoring dimensions, role weighting, category adjustments, Pareto protections, and retry behavior. The supplied code does none of that. It is a simple smoke test that uses pathlib to locate the parent skill directory and assert that a SKILL.md file exists. This is a materially different primary purpose. While tests can be supporting code, this chunk is specifically for skills/prompt-hardening/tests/test_smoke.py, which appears unrelated to the declared skill and provides no evidence of the claimed functionality. Therefore this is a clear description-behavior mismatch.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 98% confidence
Finding: The supplied code does not implement the declared auto-improvement/evaluation pipeline. Instead, it performs straightforward analytics on feedback event logs for skills: filtering by skill_id, counting correction/partial/acceptance outcomes, computing a correction-rate formula, comparing recent and prior windows for trend, aggregating dimension hints, and formatting a text report. None of the distinctive declared mechanisms—multi-dimensional judging, LLM-based judging, weighted roles, category coefficients, Pareto regression guards, retry logic, or multi-skill pipeline orchestration—appear in this chunk. The code is related only in a broad quality/feedback sense, but its primary purpose is materially narrower and different from the declared description.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 99% confidence
Finding: The supplied code does not implement or test the declared automatic evaluation/improvement pipeline. Instead, it focuses on analyzing conversation/session feedback around skill usage: detecting invocations, classifying user responses, assigning dimension hints, persisting feedback events, and archiving old records. These are plausibly related to quality monitoring in a broad sense, but they are materially different from the specific declared functionality, which emphasizes a sophisticated multi-stage scoring and regression-protection pipeline. Therefore the description does not accurately represent this code chunk.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 96% confidence
Finding: The supplied code chunk does not implement or exercise an auto-improvement pipeline. Instead, it contains unit tests for feedback metric calculations in a session-feedback analyzer. While metrics could be a supporting component in a broader evaluation system, this chunk’s actual behavior is narrowly focused on computing and formatting correction-related analytics from event data. The prominent declared capabilities—LLM judging, multi-role weighting, category coefficients, Pareto regression safeguards, and retry behavior—are not represented in this code. Therefore the description does not accurately characterize this specific code chunk.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 96% confidence
Finding: The supplied code chunk does not implement any of the declared evaluation/improvement pipeline behavior. Instead, it is a narrow utility module that models a skill specification, reads YAML, and validates a few fields. This is not merely a supporting implementation detail for the described pipeline; its stated purpose in the docstring ('Skill spec schema for --from-spec mode') indicates a different primary function: specification parsing/validation for skill generation. No scoring, improvement logic, orchestration, retry handling, Pareto protection, or evaluator roles are present in this code.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 98% confidence
Finding: There is a clear description-behavior mismatch. The declaration describes an automated evaluation/improvement pipeline with specific scoring and governance mechanisms. This code instead implements a command-line generator for skills and task suites. It validates a spec, writes SKILL.md, generates task_suite.yaml, and optionally calls separate external scripts for evaluation or auto-improvement if present. None of the distinctive declared mechanisms (9-dimensional scoring, 4-role weighting, category correction coefficients, Pareto front regression protection, trace-aware retry) are implemented here. The actual code belongs to a skill-forging/generation tool, not the described evaluator/improver pipeline.
Content: No source excerpt is available for this finding.