Skill Evaluation

• Outcome: Review
• Latest audit: Jun 24, 2026 · 10:54 AM UTC
• Version: 1.0.1

Overview

This skill is a coherent cloud evaluation workflow, but it should go to Review because it silently performs uploads, deletion/overwrite actions, credential storage, and authenticated HTTPS calls with certificate verification disabled.

Install only if you are comfortable with this skill uploading selected skills, datasets, and attachments to the configured cloud service. Avoid using it on sensitive workspaces unless you review the files being packaged, and be aware that OAuth tokens and custom model API keys are stored locally in plaintext while authenticated HTTPS certificate checks are disabled in the scripts.

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• Trigger Abuse: Overly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
• MCP Least Privilege: Underdeclared Capability, Wildcard Permission, Missing Permission Declaration
• MCP Tool Poisoning: Hidden Instructions, Unicode Deception, Parameter Description Injection

Findings (32)

Lp3

Medium

• Category: MCP Least Privilege
• Confidence: 95% confidence
• Finding: The skill declares no permissions while its documented workflow includes sensitive capabilities such as file access, shell usage, network operations, and local state management. This creates a misleading trust boundary: reviewers and users cannot accurately assess what the skill may do, while the skill can still drive packaging, upload, authentication, and other privileged actions behind 'silent execution'.

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 98% confidence
• Finding: The stated purpose is narrow—evaluating skills and producing reports—but the documented behavior extends into OAuth login, local callback handling, cloud search/download, archive packaging/upload, dataset mutation, task submission, and persistent configuration management. That mismatch is dangerous because it obscures the real attack surface and normalizes broad privileged behavior under an innocuous description.

Context-Inappropriate Capability

Medium

• Confidence: 90% confidence
• Finding: The workflow grants destructive dataset-management powers (cloud deletion and full overwrite) inside a preview/edit flow, expanding beyond simple preview into high-impact modification. In a skill intended for evaluation/reporting, bundling these capabilities increases the chance of unintended or overly broad data changes, especially because the operations are performed in the background.

Context-Inappropriate Capability

Low

• Confidence: 84% confidence
• Finding: The process directs silent modification of the local eval-dataset.json file, which is a state-changing capability not obviously required for a preview-oriented flow. Silent local edits can surprise users, break reproducibility, or create divergence if the operation fails midway.

Context-Inappropriate Capability

Medium

• Confidence: 99% confidence
• Finding: The client explicitly disables TLS certificate verification by passing verify_ssl=False, which makes HTTPS connections vulnerable to man-in-the-middle interception and server impersonation. Because this client injects Bearer tokens and a username header into requests, an attacker on the network path could steal credentials, observe evaluation traffic, or tamper with API responses.

Context-Inappropriate Capability

Medium

• Confidence: 98% confidence
• Finding: The auth client sets verify_ssl=False by default, which disables TLS certificate verification for authentication-related network traffic. This enables man-in-the-middle attacks, credential or token interception, and spoofed auth endpoints; in a skill that evaluates or integrates other systems, this is especially risky because it may contact external auth services in varied environments.

Vague Triggers

Medium

• Confidence: 87% confidence
• Finding: The trigger phrases are broad enough to match ordinary discussion about evaluating skills, increasing the chance of unintended activation. In a skill that can authenticate, search, package, upload, and manage remote evaluation jobs, accidental invocation expands exposure to privileged actions and hidden side effects.

Natural-Language Policy Violations

High

• Confidence: 83% confidence
• Finding: A mandatory Chinese-only output policy without user opt-in can impair informed consent, comprehension, and review for users operating in other languages. In this skill, that risk is amplified because important prompts may govern authentication, uploads, and remote execution while implementation details are intentionally hidden.

Missing User Warnings

Medium

• Confidence: 93% confidence
• Finding: The skill explicitly requires silent authentication checks and token/login flows, including reading and writing an auth file, without any user-facing notice or consent step. In an agent setting, this can cause credential handling and persistent auth state changes to occur opaquely, which increases the risk of unauthorized account actions, confused-deputy behavior, or users being unaware that local credential material is being created or refreshed.

Missing User Warnings

Low

• Confidence: 84% confidence
• Finding: The recent-task flow instructs the agent to automatically enumerate historical evaluation tasks and display metadata such as skill names, models, task IDs, progress, and timestamps. Even if intended as a convenience feature, this can expose prior activity and project context to anyone with access to the current chat/session without first confirming that the user wants historical data revealed.

Missing User Warnings

Medium

• Confidence: 97% confidence
• Finding: This workflow explicitly uploads user-provided Excel-derived dataset content to a cloud service and requires the operation to happen 'silently,' so the user may not realize their data is leaving the local environment at execution time. Because the Excel schema also permits attachment paths and potentially sensitive evaluation content, this creates a real transparency and consent failure that can expose proprietary or personal data to remote systems unexpectedly.

Missing User Warnings

Medium

• Confidence: 96% confidence
• Finding: The auto-completion branch sends already-uploaded dataset content to a remote synthesis/completion service to fill missing fields, again without clearly informing the user at the moment processing occurs. This is dangerous because missing fields may be completed using remote inference over sensitive prompts or attached business documents, expanding third-party processing beyond simple storage into cloud-side analysis.

Missing User Warnings

Medium

• Confidence: 94% confidence
• Finding: The platform auto-generation branch silently submits synthesis jobs, polls remote status, and downloads generated datasets without notifying the user about network activity or remote processing. Even if no user Excel is uploaded in this branch, the workflow still initiates external processing and data transfer invisibly, which undermines informed consent and can violate organizational restrictions on outbound data flows.

Missing User Warnings

High

• Confidence: 96% confidence
• Finding: The skill instructs silent deletion of cases in both cloud and local dataset state without clearly warning the user that destructive changes will occur. This is dangerous because users may believe they are only previewing data while the system performs irreversible deletion behind the scenes.

Missing User Warnings

High

• Confidence: 97% confidence
• Finding: The workflow performs a silent full upload that overwrites the remote dataset after local edits, without warning about the scope or irreversibility of replacement. Full replacement is especially risky because a single bad local edit can destroy the authoritative cloud dataset state.

Missing User Warnings

High

• Confidence: 99% confidence
• Finding: The skill explicitly mandates that all data-modifying operations be executed silently, removing transparency around destructive actions. This materially increases risk of unauthorized, accidental, or misunderstood changes because the user is not informed when mutation occurs or what exact actions were taken.

Missing User Warnings

Medium

• Confidence: 96% confidence
• Finding: The workflow explicitly asks the user for a custom model API Key and persists it to `custom-models.json`, but provides no user-facing disclosure about storage, masking, retention, or access controls. In a skill-evaluation context, this creates a real secret-handling risk: credentials may be exposed to logs, local files, other processes, backups, or later accidental display, especially because the skill normalizes repeated reuse of stored keys.

Missing User Warnings

Medium

• Confidence: 92% confidence
• Finding: The skill explicitly requires silent execution of search, download, dependency identification, packaging, and upload operations without notifying the user. Because packaging and upload can exfiltrate local skill contents and dependencies to an external evaluation service, suppressing visibility removes meaningful user consent and prevents the user from detecting unintended data transfer or scope expansion.

Missing User Warnings

Medium

• Confidence: 94% confidence
• Finding: The document exposes a destructive `delete-cases` command that can remove dataset samples by ID, but it provides no warning, confirmation workflow, scope preview, or recovery guidance. In an automation-oriented skill, this increases the chance of accidental or unauthorized destructive actions, potentially causing irreversible loss of evaluation data or corruption of benchmark integrity.

Missing User Warnings

Medium

• Confidence: 91% confidence
• Finding: The document explicitly requires silent execution for authentication, token refresh, uploads, task submission, and polling without user-facing notice. This reduces transparency around sensitive actions and can cause users to unknowingly authorize networked operations or data transfers, which is a meaningful security and trust risk in an agent skill.

Missing User Warnings

Medium

• Confidence: 97% confidence
• Finding: The client disables TLS certificate verification by default via verify_ssl=False and also globally suppresses InsecureRequestWarning, making insecure HTTPS usage silent and easy to miss. This enables man-in-the-middle interception or tampering of outbound traffic, which is especially dangerous if the client carries credentials, tokens, prompts, or evaluation data.

Missing User Warnings

Medium

• Confidence: 88% confidence
• Finding: The script persists the OAuth access token to a local JSON file in plaintext, which can expose credentials to other local users, backups, logs, or malware if file permissions are weak. In this skill context, the token likely grants access to an evaluation service, so compromise could enable unauthorized API use or account activity until the token expires or is revoked.

Missing User Warnings

Medium

• Confidence: 88% confidence
• Finding: The tool stores custom model API keys in plaintext under .eval/custom-models.json without warning, masking, or permission hardening. On shared systems or in repos/workspaces that may be synced or committed, this can expose long-lived credentials and enable unauthorized API access.

Missing User Warnings

Medium

• Confidence: 83% confidence
• Finding: The delete-cases command performs destructive remote deletion immediately once invoked, with no confirmation prompt, dry-run, or safety interlock. In an agent context, this increases the chance of accidental or prompt-induced data loss because deleting evaluation cases is irreversible from the client side.

Skill Enumeration

Medium

• Category: Agent Snooping

Content

def _search_local_skill(name: str, work_dir: Path) -> list:
    """在 work_dir 和 ~/.claude/skills 下查找 SKILL.md 同名 Skill 目录。

    手动遍历 + 深度剪枝，避免 rglob 把整个家目录扫穿。
    """

• Confidence: 72% confidence
• Finding: The search command enumerates skill directories under the caller-provided work_dir and ~/.claude/skills, revealing locally installed skill names and paths. In an agent setting, this broadens filesystem discovery beyond the stated evaluation task and can leak sensitive project structure or installed tooling metadata to downstream prompts, logs, or remote services.

Static analysis

Detected: suspicious.insecure_tls_verification

HTTPS certificate verification is disabled.

Warn

• Code: suspicious.insecure_tls_verification
• Location: scripts/eval_skill.py:391