Back to skill

Security audit

llama.cpp Benchmark

Security checks for vulnerabilities and agentic risk

Overview

The skill is a real llama.cpp benchmarking helper, but it can automatically run a discovered local binary and build unpinned upstream code, so users should review it before installing.

Install only if you are comfortable with this skill running a locally discovered llama-bench executable and, when using the build script, cloning or pulling llama.cpp from GitHub and compiling it. Prefer providing a known trusted llama-bench path, reviewing the build directory and git remote before updating, and avoiding use with elevated privileges or sensitive environment variables.

Vulnerability Patterns
  • Tool Hijacking and SpoofingModifies or replaces tools so legitimate-looking calls execute attacker logic
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (2)

T07 · Tool Hijacking and Spoofing

Error
Location
scripts/benchmark_models.sh:39
Finding

Untrusted llama-bench Auto-Discovery Can Execute an Attacker-Controlled Binary

Content
View full analysis
/dev/null | grep -E "llama-bench$" | head -1) if [ -n "$found" ] && [ -x "$found" ]; then echo "$found" return 0 fi return 1 ``` The discovered executable is subsequently invoked: ```bash if $LLAMA_BENCH -m "$model" -p "$PROMPT_SIZES" -n "$GEN_SIZES" -ngl "$GPU_LAYERS" 2>&1 >> "$RESULTS_FILE"; then echo -e "${GREEN}✓ Completed: $model_name${NC}" else echo -e "${RED}✗ Failed: $model_name${NC}" echo "ERROR: Benchmark failed" >> "$RESULTS_FILE" fi ``` ### Technical Analysis When none of the preferred `llama-bench` paths exists, `find_llama_bench` recursively searches the user's home directory and `/DATA`. It selects the first path whose basename is `llama-bench` and checks only whether that file is executable. The script does not verify: - The discovered file's owner or permissions. - Whether its parent directory is writable by untrusted users. - Its canonical path. - Its checksum, signature, or provenance. - Whether it originates from an approved llama.cpp installation. Consequently, an executable merely named `llama-bench` can be treated as a trusted tool. The `$LLAMA_BENCH` command variable is also invoked without double quotes, which can produce incorrect command parsing when the selected path contains whitespace, although the primary risk is trusting an unverified executable. ### Attack Path 1. The expected binaries at the script's preferred locations are absent. 2. An attacker who can write into a searched location under the victim's home directory or `/DATA` creates an executable named `llama-bench`. 3. The attacker arranges for that file to be the first matching result returned by `find`. 4. The victim invokes `scripts/benchmark_models.sh` with a GGU ...[truncated 856 chars]
Remediation
View remediation
> "$RESULTS_FILE" 2>&1; then echo -e "${GREEN}✓ Completed: $model_name${NC}" else echo -e "${RED}✗ Failed: $model_name${NC}" echo "ERROR: Benchmark failed" >> "$RESULTS_FILE" fi ``` 7. If automatic lookup remains necessary, use a trusted `PATH`, resolve the result with `command -v`, and then apply ownership, permission, and path validation before execution. ]]>

T08 · Insecure Dependencies

Warning
Location
scripts/build_llamacpp.sh:101
Finding

Mutable and Unverified Upstream Source Is Pulled and Built

Content
View full analysis
Remediation
View remediation
&2 exit 1 fi ``` 5. Fetch the intended revision explicitly and check it out in detached-HEAD mode instead of pulling and merging a mutable branch: ```bash git fetch --depth 1 origin "$APPROVED_COMMIT" git checkout --detach "$APPROVED_COMMIT" test "$(git rev-parse HEAD)" = "$APPROVED_COMMIT" || exit 1 ``` 6. Display the exact revision and request explicit confirmation before executing build logic. 7. Build in an isolated, non-privileged environment with minimal filesystem and credential access. 8. Avoid running the build script as root or with unnecessary secrets in environment variables. ]]>
Vulnerability Patterns
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Tool MisuseTool Parameter Abuse, Chaining Abuse, Unsafe Defaults
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Findings (5)

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

The skill is presented as a benchmarking utility, but its documented behavior also includes updating from GitHub, rebuilding llama.cpp, cleaning build directories, and inspecting git state. That mismatch can lead an agent or user to approve and invoke the skill expecting low-risk local benchmarking while it performs higher-risk supply-chain and system-modifying operations instead.

Content

No source excerpt is available for this finding.

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
95% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · scripts/build_llamacpp.sh (reported line 118)May include surrounding context.

sh
clean_build() {
    if [ -d "$BUILD_DIR/build" ]; then
        echo -e "${YELLOW}Cleaning build directory...${NC}"
        rm -rf "$BUILD_DIR/build"
        echo -e "${GREEN}✓ Build directory cleaned${NC}"
    fi
}

Description-Behavior Mismatch

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

This script does more than benchmark models: it clones and updates llama.cpp from GitHub, which introduces code-fetching and trust-on-first-use behavior into a skill described as benchmarking GGUF models. Pulling and building the latest upstream code can unexpectedly execute unreviewed build logic and changes, expanding the attack surface beyond local benchmarking.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
93% confidence
Finding

The benchmarking-focused skill performs network access to GitHub to clone code, which is not necessary for benchmarking local GGUF models. This increases exposure to supply-chain risk, remote content changes, and unexpected outbound connectivity in environments where a benchmark tool would normally be local-only.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium
Category
Not specified by scanner
Confidence
90% confidence
Finding

The script includes destructive build cleanup and build-system orchestration that are broader than the stated benchmarking purpose. Even though the deletion is scoped to the build directory, combining cleanup, reconfiguration, and compilation in a benchmark skill increases operational risk and may surprise users who expect read-only performance testing.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.