Back to skill

Security audit

Perf Test Flagos

Security checks for vulnerabilities and agentic risk

Overview

The skill matches a model-benchmarking purpose, but it needs review because it can run untrusted model repository code and broad Docker shell commands without enough scoping or warning.

Review before installing or running. Use only a dedicated, unprivileged benchmark container; avoid sensitive mounts, credentials, Docker socket access, and privileged mode. Do not use --trust-remote-code unless the model repository and exact revision are vetted, and validate or quote all user-supplied command parameters before execution.

Vulnerability Patterns
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (2)

T03 · Remote Payload Retrieval and Execution

Error
Location
SKILL.md:47
Finding

Unconditional Execution of Model-Supplied Remote Code

Content
View full analysis
\ --tensor-parallel-size \ --max-num-batched-tokens 4096 \ --max-num-seqs 256 \ --trust-remote-code \ --port 8000 \ ``` The benchmark client also enables the same behavior: ```python cmd = [ "vllm", "bench", "serve", "--host", "127.0.0.1", "--port", str(args.port), "--backend", "openai-chat", "--model", args.model, "--tokenizer", args.tokenizer, "--dataset-name", "random", "--endpoint", "/v1/chat/completions", "--ignore-eos", "--trust-remote-code", "--random-input-len", str(args.input_len), "--random-output-len", str(args.output_len), "--num-prompts", str(args.num_prompts), ] ``` ### Technical Analysis The Skill unconditionally supplies `--trust-remote-code` when starting vLLM and when loading the tokenizer through the benchmark client. This option permits model repositories to provide and execute custom Python implementations during model or tokenizer initialization. The model path or identifier originates from the user or preceding workflow context. If it refers to a malicious or compromised remote repository, repository-controlled Python can execute inside the container. The executable content may also change after the Skill itself has been reviewed unless the model repository and revision are pinned. This capability is unnecessary for models natively supported by vLLM and therefore exceeds the minimum privileges required for general performance benchmarking. ### Attack Path 1. An attacker supplies a malicious model repository identifier, or compromises a repository expected by the operator. 2. The operator or Agent passes that identifier as `MODEL_PATH` ...[truncated 1230 chars]
Remediation
View remediation

T09 · Insecure Skill Coding Practices

Warning
Location
SKILL.md:43
Finding

Shell Command Injection Through Unquoted Workflow Parameters

Content
View full analysis
bash -c ' export USE_FLAGGEMS=<0|1> export FLAGCX_PATH= export VLLM_PLUGINS= vllm serve \ --tensor-parallel-size \ --max-num-batched-tokens 4096 \ --max-num-seqs 256 \ --trust-remote-code \ --port 8000 \ ' ``` Other affected templates include: ```bash docker exec python3 /tmp/run_all_benchmarks.py \ --model \ --tokenizer \ --port 8000 \ --output-dir /data/results/perf ``` ### Technical Analysis The Skill directs the Agent to obtain the container name, model path, tensor-parallel size, stack configuration, and optional arguments from the user or workflow context. These values are then represented as direct substitutions in shell command templates without quoting, strict validation, or an allowlist. There are two parsing boundaries: 1. The host shell parses arguments such as ``, ``, and ``. 2. The container-side `bash -c` parses environment values, the model path, numeric parameters, and ``. Shell metacharacters, whitespace, command substitutions, redirections, or quote characters in substituted values can alter command structure. The unrestricted `` field is particularly dangerous because it is intentionally inserted as shell syntax inside `bash -c`. The Python scripts use argument-list subprocess calls without `shell=True`, which is safer. The vulnerability is in the documented Agent-driven Docker shell workflow rather than those Python subprocess calls. ### Attack Path 1. The Skill is invoked standalone or receives values from an upstream workflow. 2. An attacker controls ...[truncated 1666 chars]
Remediation
View remediation
Vulnerability Patterns
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
  • Behavioral ASTexec() Call, eval() Call, Dynamic Import
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
Findings (7)

Tp4

High
Category
MCP Tool Poisoning
Confidence
95% confidence
Finding

The code is clearly related to the declared benchmarking domain, but it only implements a subset of the stated behavior. It runs one performance benchmark invocation of vllm bench serve, parses latency/throughput/TTFT/TPOT-style metrics, and returns structured JSON. However, the declared description says the skill runs both accuracy benchmarks (FlagEval, when available) and performance benchmarks across five workload profiles. This code contains no FlagEval or accuracy-evaluation logic, and no orchestration for five profiles—just one profile passed by arguments. There are no suspicious undeclared resource accesses or unrelated triggers, but the declared description materially overstates the implemented capabilities in this chunk.

Content

No source excerpt is available for this finding.

External Script Fetching

High
Category
Supply Chain
Confidence
90% confidence
Finding

Remote code is downloaded and executed. This bypasses code review and could introduce malicious code.

Content

Scanner excerpt · SKILL.md (reported line 82)May include surrounding context.

bash
docker exec <CONTAINER> bash -c '
curl -s http://localhost:8000/v1/models | python3 -c "
import json, sys; print(json.load(sys.stdin)[\"data\"][0][\"id\"])"
'

Lp1

High
Category
MCP Least Privilege
Confidence
75% confidence
Finding

The skill uses 'shell' capability that is not listed in its permissions. This may indicate deceptive intent or missing permission declarations.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The documented benchmark command includes --trust-remote-code, which permits execution of code supplied by the model repository during tokenizer/model loading. In a benchmarking skill, users may copy-paste this command directly, so omitting any warning or safer alternative can lead to unintended arbitrary code execution on the host running the benchmark.

Content

No source excerpt is available for this finding.

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/run_all_benchmarks.py (reported line 44)May include surrounding context.

python
cmd.extend(["--extra-args", extra_args])

    try:
        proc = subprocess.run(cmd, capture_output=True, text=True, timeout=660)
        result = json.loads(proc.stdout)
    except subprocess.TimeoutExpired:
        result = {"name": profile["name"], "status": "FAIL", "error": "Timed out"}

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/run_benchmark.py (reported line 93)May include surrounding context.

python
cmd.extend(args.extra_args.split())

    try:
        proc = subprocess.run(
            cmd, capture_output=True, text=True, timeout=600
        )
        output = proc.stdout + proc.stderr

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
87% confidence
Finding

The skill copies files into the container, starts a model-serving process, and later kills it, but does not explicitly warn the user that it will modify the container filesystem and service state. In an operational environment this can unintentionally disrupt existing workloads, overwrite temporary files, or interfere with other processes if the container is shared.

Content

No source excerpt is available for this finding.