智能体长期记忆评测台

• Outcome: Pass
• Latest audit: Sep 16, 2026 · 11:17 AM UTC
• Version: 1.0.1

Overview

This is a disclosed local memory-benchmarking skill with an optional user-enabled API backend; the main risk is that API mode sends benchmark text to the configured model provider.

Installers should treat this as a local benchmarking tool. Keep the default internal backend for offline use. If enabling MODEL_BACKEND=api, use a trusted REAL_API_BASE and do not benchmark private transcripts, production memory logs, secrets, or regulated data unless you intend to send that text to the model provider.

A.I.G

Vulnerability Patterns

• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally
• Unauthorized Access and Privilege Escalation: Obtains permissions beyond the task's legitimate needs

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Supply Chain: Unpinned Dependencies, External Script Fetching, Obfuscated Code
• Taint Tracking: Direct Taint Flow, Variable-Mediated Taint Flow, Credential Exfiltration Chain
• MCP Least Privilege: Underdeclared Capability, Wildcard Permission, Missing Permission Declaration
• MCP Tool Poisoning: Hidden Instructions, Unicode Deception, Parameter Description Injection

Findings (15)

Tainted flow: 'req' from os.environ.get (line 66, credential/environment) → urllib.request.urlopen (network output)

Critical

• Category: Data Flow
• Confidence: 96% confidence

Finding

When MODEL_BACKEND=api, the skill sends the full transcript, history block, and question to an external endpoint, and both the destination base URL and bearer token are sourced from environment variables. This creates a real data exfiltration path for any sensitive evaluation content and also allows SSRF-style abuse if REAL_API_BASE is set to an internal or attacker-controlled host.

Content

Scanner excerpt · tools/memory_bench.py (reported line 70)May include surrounding context.

python

base + "/chat/completions", data=body, method="POST",
        headers={"Authorization": f"Bearer {key}", "Content-Type": "application/json"})
    try:
        with urllib.request.urlopen(req, timeout=30) as r:
            data = json.loads(r.read().decode("utf-8"))
        return data["choices"][0]["message"]["content"].strip()
    except Exception as e:

External Script Fetching

High

• Category: Supply Chain
• Confidence: 90% confidence

Finding

Remote code is downloaded and executed. This bypasses code review and could introduce malicious code.

Content

Scanner excerpt · 安全审计报告.md (reported line 52)May include surrounding context.

md

- Base64 编码检测: 未发现可疑编码

### 远程脚本深度分析
✅ 不适用——本包不存在自动下载并执行远程脚本的行为（无 `curl | bash` 等模式）。

### 依赖安装风险检查
- **全局安装检测**: ✅ 无（无 `pip install` / `npm install -g` 等）

Natural-Language Policy Violations

Medium

• Category: Not specified by scanner
• Confidence: 93% confidence

Finding

Line L04 requires all external releases to use the specific Chinese brand name 「注册老炮」 and forbids alternative naming. This is a natural-language policy constraint that forces a specific language/locale presentation without indicating user choice or a documented regional/compliance justification.

Content

No source excerpt is available for this finding.

Undeclared Tool Scope

Medium

• Category: MCP Least Privilege
• Confidence: 91% confidence

Finding

The skill advertises capabilities that include environment-variable access, file reads/writes, and optional network use, but it does not declare an explicit permission or allowed-tools scope. That creates a least-privilege gap: a host or reviewer cannot easily determine what the skill is expected to access, increasing the risk of overbroad execution or accidental secret/file exposure when the skill is run.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

• Category: Not specified by scanner
• Confidence: 87% confidence

Finding

The manifest-facing display name is entirely in Chinese, while this skill is intended for broad platforms and audiences. Although an English description is provided elsewhere, the primary presented name does not offer language choice or explicit locale scoping, which can violate a language/locale policy requiring user opt-in.

Content

No source excerpt is available for this finding.

External Transmission

Medium

• Category: Data Exfiltration
• Confidence: 84% confidence

Finding

The skill supports sending prompts and benchmark content to an external API endpoint when MODEL_BACKEND=api is enabled. Even though the documentation says keys are env-injected and not written to disk, external transmission still creates a real data-exfiltration surface if users run the benchmark with sensitive transcripts, custom datasets, or production memory content.

Content

Scanner excerpt · SKILL.md (reported line 43)May include surrounding context.

python tools/security_test.py

3.（可选）接真实 LLM 严评——密钥只走环境变量，不打印、不落盘

MODEL_BACKEND=api REAL_API_KEY=你的密钥 REAL_API_BASE=https://api.siliconflow.cn/v1 REAL_MODEL=deepseek-ai/DeepSeek-V3
python tools/memory_bench.py

text

Natural-Language Policy Violations

Medium

• Category: Not specified by scanner
• Confidence: 92% confidence

Finding

The module docstring and all user-facing usage text are written only in Chinese, and the runtime messages are also Chinese-only. This creates a language/locale constraint in the skill's natural-language interface without documenting a justified region-specific limitation or offering the user a choice.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

• Category: Not specified by scanner
• Confidence: 90% confidence

Finding

The stderr message and success output shown to users are emitted only in Chinese. Under the policy, forcing a specific language in natural-language output is a violation unless the skill clearly offers language choice or is explicitly documented as region-specific.

Content

No source excerpt is available for this finding.

External Transmission

Medium

• Category: Data Exfiltration
• Confidence: 90% confidence

Finding

The code includes a built-in path to transmit evaluation context to an external API service at a default remote base URL. In this skill's context, the transcript/history may be intentionally benchmark data, but if reused with real conversation logs or proprietary memory corpora, this becomes a confidentiality risk because data leaves the local environment.

Content

Scanner excerpt · tools/memory_bench.py (reported line 60)May include surrounding context.

python

or os.environ.get("DEEPSEEK_API_KEY"))
    if not key:
        return "未答出(无API密钥)"
    base = os.environ.get("REAL_API_BASE", "https://api.siliconflow.cn/v1").rstrip("/")
    model = os.environ.get("REAL_MODEL", "deepseek-ai/DeepSeek-V3")
    prompt = (f"你是严谨的记忆问答助手。仅依据给定上下文作答，"
              f"上下文无依据时回答\"未知\"。\n\n上下文:\n{transcript}\n{history}\n\n问题:{q}\n答:")

Natural-Language Policy Violations

Medium

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

The API prompt hardcodes Chinese instructions and expects answers in that language, which imposes a specific language choice on the model output. The file does not offer user opt-in or an alternative locale, so this is a natural-language locale policy issue.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

• Category: Not specified by scanner
• Confidence: 89% confidence

Finding

This code file contains user-facing docstring text entirely in Chinese for the file description and usage instructions, and later prints user-facing status messages in Chinese as well. Under the policy, forcing a specific language without user opt-in is a natural-language policy violation when no alternative language choice or justification is provided.

Content

No source excerpt is available for this finding.

External Transmission

Medium

• Category: Data Exfiltration
• Confidence: 50% confidence

Finding

Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.

Content

Scanner excerpt · 安全审计报告.md (reported line 46)May include surrounding context.

md

- 删除/移动操作: ✅ 无

### 网络请求检查
- 发现的 URL: `https://api.siliconflow.cn/v1`（默认值，用户可经环境变量覆盖）
- 触发条件: 仅当用户显式设置 `MODEL_BACKEND=api` 且提供 API 密钥时，评测台向用户自选的 OpenAI 兼容模型服务商发送「评测上下文 + 问题」，请求模型作答。
- 判定: ✅ 合理。功能描述与实际行为一致（评测真实 LLM 的记忆表现）；**密钥仅从环境变量读取，代码不打印、不落盘；未配置密钥时优雅降级返回"未答出"，不发起网络请求**；不读取、不发送任何用户本地文件或敏感信息。
- Base64 编码检测: 未发现可疑编码

Natural-Language Policy Violations

Low

• Category: Not specified by scanner
• Confidence: 80% confidence

Finding

This manifest uses Chinese-only natural-language values for the package and author fields, with no indication that users can choose a language or that the skill is limited to a Chinese-speaking context. Under the policy rule for natural-language constraints, forcing a specific language or locale without opt-in can be a policy concern.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low

• Category: Not specified by scanner
• Confidence: 91% confidence

Finding

This JSON contains user-facing descriptive fields and result labels entirely in Chinese, such as the scheme and dimension names, but does not document that the skill is Chinese-only or offer a language choice. Under the policy, forcing a specific language without user opt-in can be a natural-language policy violation.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low

• Category: Not specified by scanner
• Confidence: 87% confidence

Finding

This markdown file presents the entire skill-facing audit report in Chinese and does not indicate any user choice of language or locale. Under the policy rule for natural-language violations, forcing a specific language without opt-in can be a locale-policy issue.

Content

No source excerpt is available for this finding.