ClawHub
Back to skill

Security audit

Promptfoo

Security checks for vulnerabilities and agentic risk

Overview

This is a coherent Promptfoo helper, but it should be reviewed because it repeatedly recommends running a mutable npm package with the user's normal permissions.

Install only if you are comfortable with Promptfoo workflows and can run them in a scoped workspace. Before using the commands, prefer a pinned Promptfoo version or a lockfile-backed local dev dependency, and expose only the files, API keys, and network targets needed for the specific eval or red-team run.

A.I.G

Vulnerability Patterns
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (1)

T08 · Insecure Dependencies

Warning
Location
SKILL.md:188
Finding

Unpinned Promptfoo CLI Execution Through a Mutable npm Tag

Content
View full analysis

Vulnerability Details

File Location: SKILL.md, lines 188–190, 198, 206–210, 218–221, 254–256, 262, and 291–308
Vulnerability Type: T08: Insecure Dependencies
Risk Level: Medium

The skill repeatedly recommends executing Promptfoo through npx using the mutable @latest npm tag.

bash
# Lines 188–190
npx promptfoo@latest init --example getting-started
npx promptfoo@latest init --example eval-rag
npx promptfoo@latest init --example compare-openai-models

# Line 198
npx promptfoo@latest validate

# Lines 206–210
npx promptfoo@latest eval
npx promptfoo@latest eval --filter-first-n 5
npx promptfoo@latest eval --filter-pattern "refund|billing"
npx promptfoo@latest eval --filter-providers "openai|anthropic"
npx promptfoo@latest eval --max-concurrency 4

# Lines 218–221
npx promptfoo@latest view
npx promptfoo@latest list evals
npx promptfoo@latest show <eval-id>
npx promptfoo@latest logs

# Lines 254–256
npx promptfoo@latest redteam setup
npx promptfoo@latest redteam run
npx promptfoo@latest redteam report

# Line 262
npx promptfoo@latest redteam init --no-gui

# Lines 291–308
npx promptfoo@latest init
npx promptfoo@latest init --example getting-started
npx promptfoo@latest init --example eval-rag
npx promptfoo@latest init --example compare-openai-models
npx promptfoo@latest validate
npx promptfoo@latest eval
npx promptfoo@latest eval --resume
npx promptfoo@latest eval --retry-errors
npx promptfoo@latest list evals
npx promptfoo@latest show <eval-id>
npx promptfoo@latest logs
npx promptfoo@latest view
npx promptfoo@latest generate dataset
npx promptfoo@latest generate assertions
npx promptfoo@latest redteam setup
npx promptfoo@latest redteam run
npx promptfoo@latest redteam report
npx promptfoo@latest mcp --transport stdio

Technical Analysis

npx can download and execute an npm package when the requested package is not alread ...[truncated 2133 chars]

Remediation
View remediation

Remediation Suggestions

  1. Replace every promptfoo@latest reference with an exact, reviewed version, such as promptfoo@X.Y.Z.
  2. Prefer declaring Promptfoo as a project development dependency and committing the corresponding lockfile:
    bash
    npm install --save-dev --save-exact promptfoo@X.Y.Z
npx promptfoo validate
  3. In automated environments, use npm ci with a committed lockfile rather than resolving dependencies dynamically.
  4. Configure CI to reject unexpected lockfile changes and require dependency-review approval for Promptfoo upgrades.
  5. Use a trusted npm registry or controlled registry mirror and retain package integrity metadata.
  6. Run the CLI with least privilege in a restricted workspace or container. Expose only the credentials, files, and network destinations required for the specific evaluation.
  7. Disable npm lifecycle scripts during installation where operationally compatible, or review required lifecycle behavior before enabling it.
  8. Establish a documented update process that reviews release notes, package provenance, integrity information, and dependency changes before changing the pinned version.

SkillSpector

Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
  • System Prompt LeakageDirect Leakage, Indirect Extraction, Tool-Based Exfiltration
  • Rogue AgentSelf-Modification, Session Persistence
Findings (43)

Direct Prompt Extraction

High
Category
System Prompt Leakage
Confidence
85% confidence
Finding

Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Content

Scanner excerpt · SKILL.md (reported line 56)May include surrounding context.

md
- **Inspect historical runs, logs, or outputs** → use `promptfoo list`, `promptfoo show`, `promptfoo logs`, `promptfoo view`
- **Generate adversarial tests or security scans** → use `promptfoo redteam ...`
- **Generate datasets or assertions** → use `promptfoo generate ...`
- **Expose Promptfoo to an MCP-compatible toolchain** → use `promptfoo mcp`

## Default workflow

Anti-Refusal Statement

High
Category
Anti-Refusal
Confidence
55% confidence
Finding

Skill instructs the agent to omit warnings, disclaimers, or ethical commentary. Stripping safety caveats hides risk from the user and is a common jailbreak preamble.

Content

Scanner excerpt · references/config-patterns.md (reported line 97)May include surrounding context.

md
value: Do not apologize

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/promptfoo-preflight.py (reported line 13)May include surrounding context.

python
root = Path(args.path).resolve()
    config = root / "promptfooconfig.yaml"
    env = root / ".env"
    prompt_files = sorted(root.glob("*.txt"))
    docs_dir = root / "docs"
    agents_dir = root / "agents"

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/promptfoo-preflight.py (reported line 30)May include surrounding context.

python
root = Path(args.path).resolve()
    config = root / "promptfooconfig.yaml"
    env = root / ".env"
    prompt_files = sorted(root.glob("*.txt"))
    docs_dir = root / "docs"
    agents_dir = root / "agents"

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/promptfoo-preflight.py (reported line 32)May include surrounding context.

python
if env.exists():
        print("[OK] Found .env")
    else:
        print("[INFO] No .env found; provider credentials may need environment variables")

    if prompt_files:
        print(f"[INFO] Found text prompt files: {', '.join(p.name for p in prompt_files[:8])}")

Self-Modification

High
Category
Rogue Agent
Confidence
85% confidence
Finding

Skill modifies its own code, configuration, or behavior at runtime. Self-modification enables an agent to escalate privileges, disable safety constraints, or install persistent backdoors.

Content

Scanner excerpt · scripts/scaffold-promptfoo-config.py (reported line 117)May include surrounding context.

python
output = Path(args.output)
    if output.exists() and not args.force:
        raise SystemExit(f"Refusing to overwrite existing file: {output} (use --force)")

    output.parent.mkdir(parents=True, exist_ok=True)
    output.write_text(TEMPLATES[args.mode].rstrip() + "\n", encoding="utf-8")

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
93% confidence
Finding

The skill instructs use of commands and workflows that can write files, create configs, and access networked providers, but it does not declare any explicit tool scope such as permissions or allowed-tools. That omission weakens least-privilege controls and makes accidental or overbroad tool invocation more likely when the skill is executed by an agent framework.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
97% confidence
Finding

Using npx promptfoo without a pinned version allows execution of whatever package version is current at runtime, introducing a supply-chain and reproducibility risk. If a compromised or breaking release is published, the skill could cause users or agents to fetch and run unreviewed code.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
97% confidence
Finding

This unpinned npx promptfoo invocation can resolve to a different package version over time and may run newly published code without review. That creates a supply-chain execution path inconsistent with secure, deterministic agent tooling.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
97% confidence
Finding

An unversioned npx promptfoo command causes the runtime to trust the latest registry state rather than a reviewed artifact. In agent contexts, this can translate into arbitrary remote code execution via package install hooks or malicious updates.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
97% confidence
Finding

The command invokes npx promptfoo without constraining the version, so future executions may download and run different code than originally intended. This is a classic package-resolution supply-chain weakness, especially risky for automation that executes shell snippets directly.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
97% confidence
Finding

Because the command is unpinned, it permits drift to unreviewed upstream versions and undermines both reproducibility and trust. In a skill that encourages command execution, this increases the chance an agent runs compromised third-party code.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
97% confidence
Finding

Bare npx promptfoo delegates trust to the package registry at execution time. If the registry package changes or is hijacked, an agent following the skill could execute malicious code while believing it is just running an eval tool.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
97% confidence
Finding

This command references an unpinned package executable, making the resulting behavior non-deterministic and susceptible to upstream compromise. For an agent skill, that is dangerous because the skill itself acts as an execution recommendation.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
97% confidence
Finding

The use of unversioned npx means command execution depends on whatever package metadata and code are available at that moment. That creates avoidable supply-chain exposure and can lead to silent behavior changes affecting evaluations or MCP setup.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
97% confidence
Finding

Executing npx promptfoo without a fixed version widens the attack surface to any compromised future release of the package or dependency tree. This is especially concerning in a security-testing skill where users may expect safer operational guidance.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
97% confidence
Finding

The command can fetch and execute a new package version at runtime, which is incompatible with secure supply-chain practices. Since the skill normalizes CLI execution, the omission materially increases risk of untrusted code execution.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
97% confidence
Finding

An unpinned npx command exposes the workflow to malicious or simply incompatible upstream changes. In agent-assisted execution, that can become a direct path to remote code execution under the user's environment.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
97% confidence
Finding

The package name is resolved dynamically at runtime, so the command does not guarantee which artifact will execute. That supply-chain uncertainty is a real vulnerability when embedded in executable guidance.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
97% confidence
Finding

By omitting a version, the skill encourages execution of mutable third-party code from the registry. This increases risk of compromise and makes incident response and auditability harder because the executed code may differ across runs.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
97% confidence
Finding

The command relies on a registry lookup at execution time rather than a reviewed fixed artifact. That creates a realistic supply-chain vulnerability, not just a style issue, because npx can install and run remote code immediately.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
97% confidence
Finding

Unpinned package execution allows behavior drift and possible malicious package delivery from upstream. In a skill designed for repeated operational workflows, this exposure is amplified because the examples are likely to be copied verbatim.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
97% confidence
Finding

This command trusts a mutable upstream package reference at runtime, which can lead to unreviewed code execution. The risk is elevated because MCP and eval tooling may process sensitive prompts, configs, or logs.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
97% confidence
Finding

The documented npx promptfoo form is non-deterministic and can introduce supply-chain compromise or unexpected breaking behavior. For a skill that may be followed automatically, that is a substantive security issue.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
97% confidence
Finding

This example executes a package without version pinning, allowing the actual code run to vary with time and registry state. That undermines trust and can expose users to malicious updates.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.