Back to skill

Security audit

LLM Tester

Security checks for vulnerabilities and agentic risk

Overview

This LLM benchmarking skill matches its stated purpose, but needs Review because it can send your API key and sample text to any endpoint configured in the environment.

Before installing or running, use only non-sensitive benchmark samples, verify LLM_API_BASE is the intended HTTPS provider endpoint, use a narrowly scoped API key with spending limits, and consider pinning dependencies. Do not run it in an environment where LLM_API_BASE could be set by untrusted shell, CI, or wrapper configuration.

Vulnerability Patterns
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (2)

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/llm_benchmark.py:28
Finding

API credentials and sample contents can be transmitted to an arbitrary endpoint

Content
View full analysis
dict[str, Any]: """调用单个模型""" if not API_KEY: return {"status": "error", "time": 0, "error": "未设置 DASHSCOPE_API_KEY 环境变量"} start = time.time() try: resp = requests.post( API_BASE, headers={ "Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json", }, json={ "model": model, "messages": [{"role": "user", "content": prompt}], "temperature": 0.3, "response_format": {"type": "json_object"}, }, timeout=timeout, ) ``` ### Technical Analysis Sending prompts to a remote model provider is necessary for the declared benchmarking functionality. The security issue is that `LLM_API_BASE` may specify any URL while the program always attaches the `DASHSCOPE_API_KEY` bearer credential and the complete generated prompt. The destination is not restricted to the documented DashScope service, and the code does not require HTTPS. Consequently, a malicious or mistakenly configured environment can redirect requests to an attacker-controlled server or a plaintext HTTP endpoint. The transmitted prompt contains the prompt template and up to 2,000 characters from each sample file. These files may contain proprietary text, source code, personal data, or other confidential benchmark material. The program provides no destination confirmation, sensitivity warning, or redaction mechani ...[truncated 1634 chars]
Remediation
View remediation

T08 · Insecure Dependencies

Note
Location
scripts/requirements.txt:1
Finding

Third-party dependency is not version-pinned or integrity-verified

Content
View full analysis
Remediation
View remediation
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
  • System Prompt LeakageDirect Leakage, Indirect Extraction, Tool-Based Exfiltration
  • Taint TrackingDirect Taint Flow, Variable-Mediated Taint Flow, Credential Exfiltration Chain
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
Findings (11)

Tainted flow: 'API_BASE' from os.environ.get (line 28, credential/environment) → requests.post (network output)

Critical
Category
Data Flow
Confidence
91% confidence
Finding

The request destination is taken directly from the LLM_API_BASE environment variable and is used together with the Authorization header containing the API key. If that environment variable is altered, the script will send the bearer token and all prompts/sample data to an attacker-controlled endpoint, creating a clear SSRF-style exfiltration path for credentials and benchmark contents.

Content

Scanner excerpt · scripts/llm_benchmark.py (reported line 89)May include surrounding context.

python
start = time.time()
    try:
        resp = requests.post(
            API_BASE,
            headers={
                "Authorization": f"Bearer {API_KEY}",

Direct Prompt Extraction

High
Category
System Prompt Leakage
Confidence
85% confidence
Finding

Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Content

Scanner excerpt · scripts/llm_benchmark.py (reported line 75)May include surrounding context.

python
if not prompts:
        raise ValueError(f"Prompt 目录中没有找到 .txt 文件:{prompts_dir}")
    
    return prompts


def call_model(

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
93% confidence
Finding

The skill documents capabilities that involve reading files, using environment variables, writing reports, and sending data over the network, but it does not declare any explicit tool scope or permission boundary. This increases the risk of overbroad execution in agents that rely on metadata for sandboxing or user awareness, especially because benchmark inputs and outputs may include sensitive local content and API credentials.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

The skill is designed to send sample contents and prompt templates to an external LLM API, but the description does not warn users that local data will leave the environment. In benchmarking contexts, samples may contain proprietary text, customer data, or unreleased content, so the missing disclosure can lead to unintended exfiltration and compliance violations.

Content

No source excerpt is available for this finding.

External Transmission

Medium
Category
Data Exfiltration
Confidence
80% confidence
Finding

Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.

Content

Scanner excerpt · scripts/llm_benchmark.py (reported line 89)May include surrounding context.

python
start = time.time()
    try:
        resp = requests.post(
            API_BASE,
            headers={
                "Authorization": f"Bearer {API_KEY}",

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The tool reads local sample files and embeds their contents into prompts that are transmitted to a remote LLM service, but the CLI/help text does not clearly warn users that local data will leave the machine. In a benchmarking context, users may test proprietary datasets, so lack of disclosure increases the risk of accidental sensitive-data exposure.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
78% confidence
Finding

The natural-language instructions and examples are entirely in Chinese, and the file does not indicate that this is optional, configurable, or tied to a justified region-specific use case. Under the language/locale policy, forcing a specific language without user opt-in can be a policy concern.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
90% confidence
Finding

This is a code file, so natural-language policy checks apply to docstrings and help text. The description, usage instructions, and later CLI/help strings are presented only in Chinese, which can violate a language/locale policy when no user opt-in or region-specific justification is provided.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
80% confidence
Finding

Access to sensitive environment variables is in scope for missing-warning checks on code files. The script consumes DASHSCOPE_API_KEY but does not disclose in its main description or help text that it relies on a credential to contact an external provider.

Content

No source excerpt is available for this finding.

Unpinned Dependencies

Low
Category
Supply Chain
Confidence
96% confidence
Finding

The dependency manifest specifies requests without a version pin, which makes builds non-reproducible and can silently pull in different releases over time. That increases supply-chain risk and makes it impossible to verify whether a deployed version includes security fixes or is affected by known vulnerabilities.

Content

Scanner excerpt · scripts/requirements.txt (reported line 1)May include surrounding context.

text
requests

Unverifiable Dependency: requests has 16 known advisory(ies) (CVE-2014-1830 (Exposure of Sensitive Information to an Unauthorized Actor in Requests); CVE-2024-47081 (Requests vulnerable to .netrc credentials leak via malicious URLs); CVE-2024-35195 (Requests `Session` object does not verify requests after making first request wi) +13 more), but the manifest does not pin a version, so it is unknown whether the installed release is affected

Low
Category
Supply Chain
Confidence
90% confidence
Finding

requests has multiple published advisories, and because no version is pinned, the manifest provides no assurance that installation will avoid affected releases. In a tool that performs LLM testing and may make outbound HTTP requests to model APIs, a vulnerable HTTP client can expose credentials, mishandle TLS/session behavior, or leak sensitive request data depending on the installed version.

Content

No source excerpt is available for this finding.