Back to skill

Security audit

AI Question

Security checks for vulnerabilities and agentic risk

Overview

The skill is mostly a disclosed QuizAI question-bank helper, but its service scripts can install software and force-stop unrelated local processes without enough safeguards.

Install only if you intend to use it inside the QuizAI project and are comfortable with local file processing, database writes, and a local FastAPI service. Review or modify the start and stop PowerShell scripts before using them: prepare Python/dependencies yourself, avoid automatic winget or pip installs during startup, and ensure the stop script only terminates the intended QuizAI process.

Vulnerability Patterns
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (2)

T08 · Insecure Dependencies

Warning
Location
scripts/start_question_service.ps1:32
Finding
Automatic Installation of Unreviewed Runtime and Project Dependencies<![CDATA[ ## Vulnerability Details **File Location**: `scripts/start_question_service.ps1`, lines 32–34 and 52–54 **Vulnerability Type**: Unsafe automatic dependency installation **Risk Level**: Medium ### Vulnerable Code ```powershell if (-not (Test-Python)) { Write-Host "Python not found, trying winget install Python 3.12..." winget install -e --id Python.Python.3.12 --accept-package-agreements --accept-source-agreements $env:Path = [System.Environment]::GetEnvironmentVariable("Path", "Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path", "User") if (-not (Test-Python)) { Write-Error "Python still unavailable after install" } } ``` ```powershell Write-Host "Installing dependencies..." & $pythonExe -m pip install -q --upgrade pip & $pythonExe -m pip install -q -r (Join-Path $Root "requirements.txt") ``` ### Technical Analysis The service-start operation performs software installation as a side effect. If Python is unavailable, the script installs Python through Winget while automatically accepting package and source agreements. It then upgrades pip and installs every dependency listed in the enclosing QuizAI project's `requirements.txt`. The relevant `requirements.txt` is not included in the audited Skill package. Its package names, versions, hashes, index sources, and transitive dependencies therefore cannot be verified as part of this audit. Python package installation may execute package build backends, setup hooks, and other package-controlled code. The risk is amplified if dependencies are not pinned and hash-verified, or if the environment uses a compromised or attacker-controlled package index. A modified host-project `requirements.txt` can also introduce arbitrary packages without requiring changes to the reviewed Skill. ### Attack Path 1. An attacker modifies the enclosing project's `requirements.txt`, compromises a listed package, publishes a dependency-confusion package, or influences pip ...[truncated 942 chars]
Remediation
<![CDATA[ ## Remediation Suggestions 1. Separate dependency installation from service startup. The start script should only launch an already prepared environment. 2. Require explicit, informed user confirmation before installing Python, upgrading pip, or installing project dependencies. 3. Use a reviewed lock file containing exact dependency versions and cryptographic hashes. 4. Install dependencies with hash enforcement, such as: ```powershell & $pythonExe -m pip install --require-hashes -r requirements.lock ``` 5. Pin the package index to an approved HTTPS repository and prevent fallback to unintended extra indexes. 6. Do not automatically upgrade pip during every service start. Manage pip versions as part of a separate, reviewed setup process. 7. Validate the integrity and expected location of the host project's dependency file before using it. 8. Run installation and the application under a dedicated, least-privileged account or sandbox where practical. 9. Record dependency installation results and fail closed if package authenticity or integrity validation fails. ]]>

T09 · Insecure Skill Coding Practices

Error
Location
scripts/stop_question_service.ps1:24
Finding
Shutdown Script Can Forcefully Terminate Unrelated Processes<![CDATA[ ## Vulnerability Details **File Location**: `scripts/stop_question_service.ps1`, lines 24–52 **Vulnerability Type**: Insufficient process identity validation **Risk Level**: High ### Vulnerable Code ```powershell function Stop-ProcessTree($processId) { if (-not $processId) { return $false } try { taskkill /PID $processId /T /F 2>$null | Out-Null if ($LASTEXITCODE -eq 0) { Write-Host "Stopped process tree PID $processId" return $true } Stop-Process -Id $processId -Force -ErrorAction Stop Write-Host "Stopped process PID $processId" return $true } catch { return $false } } if (Test-Path $servicePidFile) { $savedPid = (Get-Content $servicePidFile -Raw).Trim() if ($savedPid -match '^\d+$') { if (Stop-ProcessTree ([int]$savedPid)) { $stopped = $true } } Remove-Item $servicePidFile -Force -ErrorAction SilentlyContinue } $conns = @(Get-NetTCPConnection -LocalPort 8000 -State Listen -ErrorAction SilentlyContinue) foreach ($c in $conns) { if (Stop-ProcessTree $c.OwningProcess) { $stopped = $true } } ``` ### Technical Analysis The shutdown script forcefully terminates a process tree based only on a numeric PID from `data/quizai.pid`. It does not verify the process executable, command line, creation time, owner, project path, or relationship to the expected QuizAI instance. A PID is not a durable process identity. If QuizAI exits and Windows later reuses its PID, the stale PID file may identify an unrelated process. If an attacker or another local process can write to the project data directory, it can also place a chosen numeric PID in the file. The fallback logic is broader still: it enumerates every process listening on local port 8000 and forcefully terminates each process tree. There is no validation that the listener is the QuizAI application. The use of `taskkill /T /F` may also terminate child processes of the selected ta ...[truncated 1375 chars]
Remediation
<![CDATA[ ## Remediation Suggestions 1. Store a stronger process identity when starting the service, including: - PID - Process creation timestamp - Expected executable path - Expected project root - A random per-launch identifier where feasible 2. Before termination, retrieve the target process and verify that: - Its executable is the expected virtual-environment Python binary. - Its command line contains the expected absolute `run.py` path. - Its creation time matches the recorded value. - Its executable and working context belong to the expected QuizAI project. 3. Treat a missing, malformed, stale, or mismatched PID record as an error and do not terminate the process. 4. Remove the behavior that terminates every listener on port 8000. A port number does not establish application identity. 5. If port-based discovery is retained as diagnostic information, require explicit user confirmation and display the executable path and command line before taking action. 6. Attempt graceful shutdown before using forced termination. 7. Restrict write access to the PID file and its parent directory. 8. Avoid terminating an entire process tree unless all relevant descendants have been validated as belonging to the application. 9. Report identity-validation failures rather than silently proceeding to broader process discovery. ]]>
Vulnerability Patterns
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • Behavioral ASTexec() Call, eval() Call, Dynamic Import
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Findings (15)

Tp4

High
Category
MCP Tool Poisoning
Confidence
96% confidence
Finding
声明描述的是一个较完整的题库处理与服务控制技能,但提供的代码片段实际只是“资料文本抽取”工具,用于把上传文件转成纯文本供后续模型阅读。它没有任何生成题目、规范题库、输出 Excel、调用模型、或管理 Python 服务的逻辑。虽然“从上传资料出题”流程中,文本提取可以算支持性步骤,但该片段的主要功能与声明的主要用途存在明显差异,因此应判定为不匹配。

Tp4

High
Category
MCP Tool Poisoning
Confidence
93% confidence
Finding
声明描述涵盖三类能力:1) 从上传资料生成题库 Excel,2) 规范/格式化题库,3) 启动/停止 Python 刷题服务。该代码块的实际职责较窄:它要么把已有 JSON 题库交给另一个脚本导出为 Excel,要么先提取资料文本,然后提示用户去按指南用对话模型生成 JSON,再手动执行导出命令。因此,代码并未直接完成“模型出题生成题库”的核心过程,只是为该流程做前置提取和后置导出。更明显的是,声明中的启动/停止服务能力在此代码中完全不存在。基于给定代码块,声明相对实际行为明显过宽,属于描述与行为不一致。

Tp4

High
Category
MCP Tool Poisoning
Confidence
94% confidence
Finding
声明的核心功能是生成/规范题库以及启动或停止服务;但提供的代码仅执行本地文件题库导入。它支持 JSON 和 Excel 文件输入,必要时对 JSON 题目调用 prepare_export_questions 做整理/打乱选项,然后初始化数据库并将题库存入 SQLite。代码中没有对话式模型交互、没有从上传资料生成新题目、没有导出或规范 Excel 文件、也没有任何启动/停止 Python 服务的逻辑。因此其实际主要用途与声明描述存在实质性偏差。

Lp3

Medium
Category
MCP Least Privilege
Confidence
97% confidence
Finding
The skill instructs the agent to read files, write generated JSON/XLSX outputs, and execute shell/PowerShell commands, but it does not declare any explicit tool scope such as permissions or allowed-tools. That creates an authorization gap: an agent or host may permit broader file-system or shell access than users expect, increasing the risk of unintended command execution or modification of local project data.

Vague Triggers

Medium
Confidence
93% confidence
Finding
The activation description is broad and includes generic exam-related phrases, which can cause the skill to trigger for loosely related requests. Because the skill can read user-supplied files, write outputs, and run local scripts or services, over-broad invocation raises the chance of unintended execution in contexts where the user did not mean to authorize those actions.

Vague Triggers

Medium
Confidence
88% confidence
Finding
The instruction that the agent should execute the corresponding chapter whenever the user mentions the listed functions is ambiguous and permissive. In this skill's context, ambiguity is more dangerous because execution paths include shell commands, local service control, and database writes, so a mistaken activation can have real side effects on the host environment.

Natural-Language Policy Violations

Medium
Confidence
95% confidence
Finding
The document title and all operational guidance are written as a mandatory Chinese-only generation guide, and nowhere offers the user a language or locale choice. Under the policy, forcing a specific language without user opt-in is a natural-language policy violation unless the locale restriction is explicitly justified.

subprocess module call

Medium
Category
Dangerous Code Execution
Content
"-o",
            args.output,
        ]
        return subprocess.call(cmd)

    extract_script = SCRIPTS / "extract_material.py"
    extract_out = Path(args.extract_out)
Confidence
70% confidence
Finding
subprocess module calls execute external commands. Without careful input validation, this enables command injection.

subprocess module call

Medium
Category
Dangerous Code Execution
Content
extract_out = ROOT / extract_out

    cmd = [sys.executable, str(extract_script), *args.inputs, "-o", str(extract_out)]
    code = subprocess.call(cmd)
    if code != 0:
        return code
Confidence
70% confidence
Finding
subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Context-Inappropriate Capability

Medium
Confidence
89% confidence
Finding
The script automatically performs a system-wide `winget install` of Python when `python` is not found, which goes beyond the minimal action of starting an application service. In an agent/skill context, this can cause unexpected host modification, trigger privileged package installation flows, and expand the blast radius if the skill is invoked in sensitive environments where users did not explicitly consent to software installation.

Missing User Warnings

Medium
Confidence
91% confidence
Finding
The manifest explicitly advertises capabilities to start and stop a local FastAPI service and even kill the QuizAI process, but it provides no user-facing warning, confirmation requirement, or scope limitation in the manifest metadata. In agent-driven environments, this can lead to unintended process control actions, service interruption, or changes to the local development environment when a user only loosely matches a trigger phrase.

Vague Triggers

Medium
Confidence
88% confidence
Finding
The keyword "考试题目" is generic natural language that could appear in ordinary user requests about exams, not specifically this skill's workflow. In a manifest trigger list, such a broad phrase increases the chance of unintended invocation when the user is merely discussing exam questions rather than asking to run this skill.

Natural-Language Policy Violations

Low
Confidence
95% confidence
Finding
The RuntimeError string is written only in Chinese, which imposes a specific language on users regardless of their locale or preferences. This is a natural-language policy concern because the file does not provide an alternative language or indicate that Chinese is a documented, region-specific requirement.

Natural-Language Policy Violations

Low
Confidence
95% confidence
Finding
The module docstring is written only in Chinese, and the script's CLI description and status/error messages are also Chinese-only. This creates a locale/language policy concern because the skill imposes a specific language on users without any opt-in or documented region-specific justification.

Natural-Language Policy Violations

Low
Confidence
97% confidence
Finding
The docstring, argparse descriptions, error messages, and success output are all hard-coded in Chinese throughout the script. This can violate a language/locale policy when a skill imposes a specific language on users without opt-in or justification.

Static analysis

Detected: suspicious.install_untrusted_source

Install source points to URL shortener or raw IP.

Warn
Code
suspicious.install_untrusted_source
Location
skill.manifest.json:100