Back to skill

Security audit

DevTool Answer Monitor

Security checks for vulnerabilities and agentic risk

Overview

This appears to be a coherent LLM answer-monitoring skill with ordinary install and API-usage risks, not hidden or destructive behavior.

Use quickstart or manual mode for local-only evaluation. Before API collection, verify the provider endpoint, keep keys in environment variables, avoid confidential query text, and consider pinning dependencies or installing in an isolated environment.

Vulnerability Patterns
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (2)

T08 · Insecure Dependencies

Error
Location
install.sh:11
Finding

Unpinned Dependencies Permit Supply-Chain Code Execution

Content
View full analysis
=1.30.0 jsonschema>=4.22.0 matplotlib>=3.8.0 ``` ### Technical Analysis The installation script upgrades `pip` and installs dependencies using open-ended minimum-version constraints. These constraints allow pip to resolve any later version published under the listed package names. Consequently, the effective code installed and executed is not fixed to the code reviewed during this audit. A compromised maintainer account, malicious future release, package-index compromise, or dependency takeover could cause an attacker-controlled package version to be selected during installation. Python packages may execute code as part of their build or installation process, and imported packages execute module initialization code at runtime. The absence of a lock file and cryptographic hashes also prevents users from verifying that they received the exact artifacts tested by the project. ### Attack Path 1. An attacker compromises a listed package, one of its transitive dependencies, or the package-distribution channel. 2. The attacker publishes a malicious version that satisfies the project's lower-bound requirement. 3. A user runs `install.sh`. 4. Pip resolves and downloads the malicious release because no exact version or artifact hash is enforced. 5. Malicious package code executes during installation, build, or subsequent import. 6. The payload runs with the privileges of the user performing the installation. ### Impact Assessment Successful exploitation can provide arbitrary code execution under the installing user's account. This may expose environment variables, source code ...[truncated 374 chars]
Remediation
View remediation

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/run_chat_completions.py:114
Finding

Unrestricted API Base URL Can Expose Provider Credentials to an Attacker-Controlled Endpoint

Content
View full analysis
Remediation
View remediation
Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Behavioral ASTexec() Call, eval() Call, Dynamic Import
Findings (48)

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

The declared purpose is specifically about monitoring LLM descriptions of developer tools and includes several domain-specific monitoring and validation features. The actual code shown does not implement or directly expose those features; it merely imports main from geo_monitor.cli. Based on this chunk alone, the code appears to be a wrapper around another CLI associated with geo_monitor, which is not consistent with the declared devtool-answer-monitor purpose. Because the visible behavior points to a different tool/package identity and no matching monitoring logic is present, this is a likely description-behavior mismatch.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding

The declared description is highly specific about a skill for monitoring LLM descriptions of developer tools and performing several analysis and validation tasks. The actual code shown is only a minimal package entry point that delegates execution to a CLI main function. In this chunk, none of the claimed monitoring, analysis, or validation behavior is present or verifiable. Because the supplied code does not reflect the declared primary purpose and instead only provides generic CLI bootstrap behavior, this is a description-behavior mismatch based on the available code.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

The declared description presents an analysis/monitoring skill focused on evaluating LLM answers about developer tools. The supplied code does not implement any monitoring, query analysis, model comparison, content checking, or regression validation logic. Instead, it is only an installation script for setting up a local Python environment and dependencies. While installer scripts can support a larger project, this code chunk’s actual behavior is materially different from the declared purpose and exposes undeclared setup capabilities.

Content

No source excerpt is available for this finding.

Lp1

High
Category
MCP Least Privilege
Confidence
75% confidence
Finding

The skill uses 'network' capability that is not listed in its permissions. This may indicate deceptive intent or missing permission declarations.

Content

No source excerpt is available for this finding.

Lp1

High
Category
MCP Least Privilege
Confidence
75% confidence
Finding

The skill uses 'shell' capability that is not listed in its permissions. This may indicate deceptive intent or missing permission declarations.

Content

No source excerpt is available for this finding.

Lp1

High
Category
MCP Least Privilege
Confidence
75% confidence
Finding

The skill uses 'file_write' capability that is not listed in its permissions. This may indicate deceptive intent or missing permission declarations.

Content

No source excerpt is available for this finding.

Lp1

High
Category
MCP Least Privilege
Confidence
75% confidence
Finding

The skill uses 'env' capability that is not listed in its permissions. This may indicate deceptive intent or missing permission declarations.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
90% confidence
Finding

The JSON segments explicitly set language to zh or en for each query entry, which imposes locale-specific behavior in the skill data. There is no accompanying text indicating user opt-in, configurable language selection, or a documented region-specific justification, so this appears to violate the language/locale policy criterion.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

This JSON manifest includes explicit language values such as "en" and "zh" for multiple segments, indicating the skill content is targeted to specific languages. Because the file does not describe any user selection, opt-in, or justification for locale restriction, it presents a natural-language policy concern under the language/locale rule.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The entire document presents usage instructions in Chinese, including the title and operational guidance, but does not indicate that language selection is optional or that the skill is intended only for Chinese-speaking users. Under the stated policy, forcing a specific language without user opt-in is a natural-language policy concern.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

This markdown file presents all substantive audit guidance in Chinese, with no indication that the user can choose another language or that the skill is intentionally region-specific. Under the policy, forcing a specific language without user opt-in is a natural-language policy violation.

Content

No source excerpt is available for this finding.

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · geo_monitor/cli.py (reported line 12)May include surrounding context.

python
def call(script_name, args):
    script = ROOT / 'scripts' / script_name
    cmd = [sys.executable, str(script), *args]
    return subprocess.check_call(cmd)


def cmd_run(ns):

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

This markdown file is written in Chinese and presents the playbook instructions directly in that language, with no indication that users may choose another language or that the scope is intentionally limited to Chinese-speaking users. Under the language/locale policy rule, a skill should not impose a specific language without opt-in or clear justification.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

The entire skill content is written only in Chinese and presents China-specific channels and publishing guidance without offering any language or locale choice. Under the stated policy, a skill that implicitly requires a specific language/locale without opt-in is a natural-language policy violation unless the regional constraint is clearly documented and justified.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The playbook content is written entirely in Chinese, which effectively imposes a specific language on users. There is no indication that the skill supports language choice, offers opt-in for Chinese, or documents a region-specific justification for the locale constraint.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
93% confidence
Finding

The entire skill file is written in Chinese and provides no indication that users may choose another language or locale. Under the stated policy, forcing a specific language without opt-in is a natural-language policy violation unless the locale constraint is explicitly documented and justified.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
87% confidence
Finding

This markdown file presents all operational guidance in Chinese and does not indicate that the user can choose another language. Under the policy, forcing a specific language without user opt-in is a natural-language policy violation unless the locale restriction is explicitly justified.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
98% confidence
Finding

The playbook content is presented in Chinese throughout, with no indication that users may choose another language or that the locale restriction is intentional. Under the policy rule for natural-language violations, forcing a specific language without user opt-in is a reportable issue.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
90% confidence
Finding

The documentation instructs users to run npx skills add ... without pinning a specific package version. npx resolves and executes the latest available package by default, which creates a supply-chain risk: a compromised upstream package, typo-squatted dependency, or breaking update could be executed on a user's machine. In this skill context, the command is presented as an installation path for a developer tool, which makes the risk more actionable because readers may copy-paste it directly.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
90% confidence
Finding

This line again recommends executing npx skills add ... without a pinned version for single-skill installation. As with the earlier command, this can cause users to fetch and run whatever version is currently published, exposing them to supply-chain compromise or unexpected behavior. Because the document is an integration playbook aimed at enabling quick installation, the likelihood of copy-paste execution is high.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

The playbook content is entirely in Chinese and does not offer any language fallback, user choice, or documented locale constraint. In an agent skill, this can cause instruction misalignment, reduce operator reviewability, and make the skill fail or behave unpredictably for users expecting another language, which is a real quality and safety risk even if it is not an exploit in the classic code-execution sense.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

This script sends user-provided queries to external model providers and stores returned content, but it does not include any built-in notice, consent gate, or redaction step before transmission. In a monitoring skill that may process proprietary developer-tool prompts or internal evaluation text, this can cause unintended disclosure to third-party APIs and persistent logging outside the user's control.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The code sends the user query to client.responses.create(...), which transmits query content to an external model provider. Although this network call is central to the script's purpose, there is no confirmation prompt, warning message, or explanatory comment disclosing that potentially sensitive query data will leave the local environment.

Content

No source excerpt is available for this finding.

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
80% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · SKILL.md (reported line 45)May include surrounding context.

md
## Safety

- Use `quickstart replay` or `manual paste mode` first when the user does not need live API calls.
- Keep provider keys in local shell environment variables. Do not ask users to paste secrets into chat.
- Inspect `install.sh`, `quickstart.sh`, and the selected runner before executing Bash commands.

## Trigger

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
80% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · skills/visibility-monitor/SKILL.md (reported line 36)May include surrounding context.

md
## Safety

- Use `quickstart replay` or `manual paste mode` first when the user does not need live API calls.
- Keep provider keys in local shell environment variables. Do not ask users to paste secrets into chat.
- Inspect `install.sh`, `quickstart.sh`, and the selected runner before executing Bash commands.

## Trigger

Static analysis

No suspicious patterns detected.