Perf Test Flagos

• Outcome: Review
• Latest audit: Sep 16, 2026 · 8:27 AM UTC
• Version: 1.0.0

Overview

The skill matches a model-benchmarking purpose, but it needs review because it can run untrusted model repository code and broad Docker shell commands without enough scoping or warning.

Review before installing or running. Use only a dedicated, unprivileged benchmark container; avoid sensitive mounts, credentials, Docker socket access, and privileged mode. Do not use --trust-remote-code unless the model repository and exact revision are vetted, and validate or quote all user-supplied command parameters before execution.

A.I.G

Vulnerability Patterns

• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally

Findings (2)

T03 · Remote Payload Retrieval and Execution

Error

• Location: SKILL.md:47
• Finding: Unconditional Execution of Model-Supplied Remote Code<![CDATA[ ## Vulnerability Details **File Location**: `SKILL.md:47-56`; additional occurrences at `scripts/run_benchmark.py:71-85` and `references/benchmark-profiles.md:30-45` **Vulnerability Type**: Untrusted model code execution **Risk Level**: High ### Vulnerable Code ```bash vllm serve <MODEL_PATH> \ --tensor-parallel-size <TP_SIZE> \ --max-num-batched-tokens 4096 \ --max-num-seqs 256 \ --trust-remote-code \ --port 8000 \ <EXTRA_ARGS> ``` The benchmark client also enables the same behavior: ```python cmd = [ "vllm", "bench", "serve", "--host", "127.0.0.1", "--port", str(args.port), "--backend", "openai-chat", "--model", args.model, "--tokenizer", args.tokenizer, "--dataset-name", "random", "--endpoint", "/v1/chat/completions", "--ignore-eos", "--trust-remote-code", "--random-input-len", str(args.input_len), "--random-output-len", str(args.output_len), "--num-prompts", str(args.num_prompts), ] ``` ### Technical Analysis The Skill unconditionally supplies `--trust-remote-code` when starting vLLM and when loading the tokenizer through the benchmark client. This option permits model repositories to provide and execute custom Python implementations during model or tokenizer initialization. The model path or identifier originates from the user or preceding workflow context. If it refers to a malicious or compromised remote repository, repository-controlled Python can execute inside the container. The executable content may also change after the Skill itself has been reviewed unless the model repository and revision are pinned. This capability is unnecessary for models natively supported by vLLM and therefore exceeds the minimum privileges required for general performance benchmarking. ### Attack Path 1. An attacker supplies a malicious model repository identifier, or compromises a repository expected by the operator. 2. The operator or Agent passes that identifier as `MODEL_PATH` ...[truncated 1230 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions 1. Remove `--trust-remote-code` from both server and benchmark commands by default. 2. Require explicit, informed user approval before enabling it for a model that cannot run without custom code. 3. Allow only reviewed model repositories and pin each model to an immutable commit or revision. 4. Download and inspect required custom model code before execution rather than trusting mutable repository content at runtime. 5. Run custom model code in a dedicated, unprivileged container with: - No Docker socket. - No privileged mode. - No host-sensitive mounts. - No credentials or unrelated secrets. - A read-only root filesystem where practical. - Dropped Linux capabilities and a non-root user. - Restricted outbound network access. 6. Separate the model-serving environment from containers containing sensitive benchmark data. 7. Record the repository, immutable revision, and whether remote code was authorized in the generated report. ]]>

T09 · Insecure Skill Coding Practices

Warning

• Location: SKILL.md:43
• Finding: Shell Command Injection Through Unquoted Workflow Parameters<![CDATA[ ## Vulnerability Details **File Location**: `SKILL.md:43-56`; related command templates at `SKILL.md:61-68`, `SKILL.md:76-81`, and `SKILL.md:100-106` **Vulnerability Type**: Shell command injection **Risk Level**: Medium ### Vulnerable Code ```bash docker exec -d <CONTAINER> bash -c ' export USE_FLAGGEMS=<0|1> export FLAGCX_PATH=<path_or_unset> export VLLM_PLUGINS=<fl_or_unset> vllm serve <MODEL_PATH> \ --tensor-parallel-size <TP_SIZE> \ --max-num-batched-tokens 4096 \ --max-num-seqs 256 \ --trust-remote-code \ --port 8000 \ <EXTRA_ARGS> ' ``` Other affected templates include: ```bash docker exec <CONTAINER> python3 /tmp/run_all_benchmarks.py \ --model <MODEL_NAME> \ --tokenizer <MODEL_PATH> \ --port 8000 \ --output-dir /data/results/perf ``` ### Technical Analysis The Skill directs the Agent to obtain the container name, model path, tensor-parallel size, stack configuration, and optional arguments from the user or workflow context. These values are then represented as direct substitutions in shell command templates without quoting, strict validation, or an allowlist. There are two parsing boundaries: 1. The host shell parses arguments such as `<CONTAINER>`, `<MODEL_NAME>`, and `<MODEL_PATH>`. 2. The container-side `bash -c` parses environment values, the model path, numeric parameters, and `<EXTRA_ARGS>`. Shell metacharacters, whitespace, command substitutions, redirections, or quote characters in substituted values can alter command structure. The unrestricted `<EXTRA_ARGS>` field is particularly dangerous because it is intentionally inserted as shell syntax inside `bash -c`. The Python scripts use argument-list subprocess calls without `shell=True`, which is safer. The vulnerability is in the documented Agent-driven Docker shell workflow rather than those Python subprocess calls. ### Attack Path 1. The Skill is invoked standalone or receives values from an upstream workflow. 2. An attacker controls ...[truncated 1666 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions 1. Do not construct commands by directly interpolating user-controlled values into `bash -c`. 2. Use argument arrays or a dedicated wrapper script that receives each value as a separate argument. 3. Pass environment variables with structured Docker options such as `docker exec --env NAME=value` rather than generating shell `export` statements. 4. Apply strict validation: - Container names: allow only the Docker-supported identifier character set. - Tensor-parallel size and port: require bounded integers. - Model and tokenizer paths: require approved paths or reviewed repository identifiers. - Plugin and library paths: require canonical absolute paths under approved directories. 5. Replace unrestricted `EXTRA_ARGS` with an allowlist of supported options. Parse each option structurally and reject shell metacharacters or unknown flags. 6. If a shell is unavoidable, escape every dynamic value with a reliable shell-quoting implementation; do not rely on manual quote placement. 7. Display the resolved command and request confirmation before executing commands that use externally supplied model or stack parameters. 8. Run Docker operations with the least privileges possible and prohibit privileged containers, Docker socket mounts, and unnecessary host filesystem mounts. ]]>

SkillSpector

Vulnerability Patterns

• Supply Chain: Unpinned Dependencies, External Script Fetching, Obfuscated Code
• Behavioral AST: exec() Call, eval() Call, Dynamic Import
• MCP Least Privilege: Underdeclared Capability, Wildcard Permission, Missing Permission Declaration
• MCP Tool Poisoning: Hidden Instructions, Unicode Deception, Parameter Description Injection
• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands

Findings (7)

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 95% confidence
• Finding: The code is clearly related to the declared benchmarking domain, but it only implements a subset of the stated behavior. It runs one performance benchmark invocation of `vllm bench serve`, parses latency/throughput/TTFT/TPOT-style metrics, and returns structured JSON. However, the declared description says the skill runs both accuracy benchmarks (FlagEval, when available) and performance benchmarks across five workload profiles. This code contains no FlagEval or accuracy-evaluation logic, and no orchestration for five profiles—just one profile passed by arguments. There are no suspicious undeclared resource accesses or unrelated triggers, but the declared description materially overstates the implemented capabilities in this chunk.

External Script Fetching

High

• Category: Supply Chain

Content

```bash
docker exec <CONTAINER> bash -c '
curl -s http://localhost:8000/v1/models | python3 -c "
import json, sys; print(json.load(sys.stdin)[\"data\"][0][\"id\"])"
'
```

• Confidence: 90% confidence
• Finding: Remote code is downloaded and executed. This bypasses code review and could introduce malicious code.

Lp1

High

• Category: MCP Least Privilege
• Confidence: 75% confidence
• Finding: The skill uses 'shell' capability that is not listed in its permissions. This may indicate deceptive intent or missing permission declarations.

Missing User Warnings

Medium

• Confidence: 95% confidence
• Finding: The documented benchmark command includes `--trust-remote-code`, which permits execution of code supplied by the model repository during tokenizer/model loading. In a benchmarking skill, users may copy-paste this command directly, so omitting any warning or safer alternative can lead to unintended arbitrary code execution on the host running the benchmark.

subprocess module call

Medium

• Category: Dangerous Code Execution

Content

cmd.extend(["--extra-args", extra_args])

    try:
        proc = subprocess.run(cmd, capture_output=True, text=True, timeout=660)
        result = json.loads(proc.stdout)
    except subprocess.TimeoutExpired:
        result = {"name": profile["name"], "status": "FAIL", "error": "Timed out"}

• Confidence: 70% confidence
• Finding: subprocess module calls execute external commands. Without careful input validation, this enables command injection.

subprocess module call

Medium

• Category: Dangerous Code Execution

Content

cmd.extend(args.extra_args.split())

    try:
        proc = subprocess.run(
            cmd, capture_output=True, text=True, timeout=600
        )
        output = proc.stdout + proc.stderr

• Confidence: 70% confidence
• Finding: subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Missing User Warnings

Low

• Confidence: 87% confidence
• Finding: The skill copies files into the container, starts a model-serving process, and later kills it, but does not explicitly warn the user that it will modify the container filesystem and service state. In an operational environment this can unintentionally disrupt existing workloads, overwrite temporary files, or interfere with other processes if the container is shared.

Static analysis

No suspicious patterns detected.