Back to skill

Security audit

RAG Evaluation Toolkit

Security checks for vulnerabilities and agentic risk

Overview

This is a disclosed RAG evaluation skill that uses local or approved cloud LLM judges, with sensitive raw-content storage kept opt-in.

Use a virtual environment, prefer RAGAS_LLM=ollama/... for confidential data, and set RAGAS_ALLOW_CLOUD=1 only when the question, answer, and contexts are allowed to leave the machine. Avoid --save, --include-content, and --save-individual unless local retention of evaluation data is intended, and consider pinning dependencies before enterprise use.

Vulnerability Patterns
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (1)

T08 · Insecure Dependencies

Warning
Location
scripts/setup.sh:47
Finding

Unpinned Third-Party Dependencies Create a Supply-Chain Risk

Content
View full analysis

Vulnerability Details

File Location: scripts/setup.sh, lines 47–66
Vulnerability Type: Unpinned and unverified third-party dependency installation
Risk Level: Medium

Vulnerable Code

bash
# ── Upgrade pip silently ──────────────────────────────────────────────────────
info "Upgrading pip..."
"$PYTHON" -m pip install --quiet --upgrade pip
ok "pip up to date"

# ── Core ragas packages ───────────────────────────────────────────────────────
info "Installing ragas + datasets..."
"$PYTHON" -m pip install --quiet "ragas>=0.2.0" "datasets>=2.0"
ok "ragas + datasets"

# ── LangChain integrations (required for LLM-as-judge) ───────────────────────
info "Installing LangChain core + integrations..."
"$PYTHON" -m pip install --quiet \
    "langchain-core>=0.3" \
    "langchain-community>=0.3" \
    "langchain-openai>=0.2" \
    "langchain-anthropic>=0.2"
ok "langchain-core, langchain-community, langchain-openai, langchain-anthropic"

# ── Optional: HuggingFace offline embeddings ─────────────────────────────────
info "Installing optional offline embeddings (HuggingFace)..."
if "$PYTHON" -m pip install --quiet langchain-huggingface sentence-transformers 2>/dev/null; then

Technical Analysis

The setup process resolves and installs mutable package versions from pip without exact version pins, integrity hashes, a committed lockfile, or an explicitly enforced trusted package index. Constraints such as ragas>=0.2.0 permit any later release, while several optional packages have no version constraint at all. The script also dynamically upgrades pip.

Consequently, the code reviewed in this project does not fully determine the code that users execute. A compromised future release, compromised dependency account, malicious transitive dependency, or attacker-controlled package index configured in the user's pip environment could introduce arbitrary code into the virtual environment.

This is a supply-chain weakness rather than evidence that any cur ...[truncated 1719 chars]

Remediation
View remediation

Remediation Suggestions

  1. Replace broad constraints with exact, reviewed versions for direct dependencies.
  2. Generate and commit a reproducible lockfile containing all transitive dependencies.
  3. Include SHA-256 hashes and install with pip --require-hashes.
  4. Install from a dedicated requirements file rather than resolving packages independently in multiple commands.
  5. Explicitly enforce the intended HTTPS package index and disable unintended supplemental indexes where operationally appropriate.
  6. Avoid automatically upgrading pip during routine setup; pin and review the installer version separately.
  7. Use an automated dependency-update process that runs security scanning and tests before lockfile changes are merged.
  8. Consider the following installation pattern:
bash
"$PYTHON" -m pip install \
    --require-hashes \
    --only-binary=:all: \
    -r requirements.lock
  1. Continue requiring an activated virtual environment, as the existing check usefully reduces the scope of package modifications.
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • Behavioral ASTexec() Call, eval() Call, Dynamic Import
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
Findings (17)

Missing User Warnings

High
Category
Not specified by scanner
Confidence
96% confidence
Finding

The dependency section states that OpenAI or Anthropic keys are used because Ragas relies on an LLM-as-judge, but it omits an explicit warning that submitted questions, answers, and contexts may be transmitted to third-party providers. For enterprise RAG evaluation, these inputs often contain proprietary or regulated data, so undisclosed external transmission materially raises privacy, compliance, and data exfiltration risk.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
95% confidence
Finding

The core behavior is related to RAG evaluation with Ragas metrics, so the general domain matches. However, the declared description is narrower than the code's actual behavior in two material ways. First, the description emphasizes evaluating a specifically provided RAG question/answer/contexts, while this script is explicitly a batch evaluator over a JSONL golden dataset with aggregation and reporting. Second, the description says not to persist raw content unless requested, but the script always writes a JSON batch report to disk, and it can also persist per-sample raw content when --save-individual is used. Even though raw content inclusion in the main report is gated by --include-content, persistence behavior exists and the default disk output is not mentioned. Therefore this is a meaningful description-behavior mismatch.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding

The declared description says the skill evaluates RAG outputs with Ragas metrics when explicitly requested by the user. The supplied code does not evaluate any RAG question, answer, or contexts, does not compute Ragas metrics, and does not perform hallucination analysis. Its primary purpose is materially different: it is a unittest script for selection logic in another module, using mocked modules, environment-variable setup, and assertions about Ollama and embedding choice. While this may support the broader skill implementation, this specific code chunk's behavior is testing infrastructure rather than the declared end-user functionality, so it is a mismatch.

Content

No source excerpt is available for this finding.

Env Variable Harvesting

High
Category
Data Exfiltration
Confidence
60% confidence
Finding

Code enumerates, copies, or searches environment variables for secrets. Bulk environment access can collect credentials unrelated to the skill's stated purpose.

Content

Scanner excerpt · scripts/test_selection.py (reported line 38)May include surrounding context.

python
class SelectionTests(unittest.TestCase):
    def setUp(self):
        self.original_env = os.environ.copy()
        self.original_modules = dict(sys.modules)

        community = types.ModuleType("langchain_community")

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
91% confidence
Finding

The PRD documents writing evaluation results to disk but does not include a user warning or consent mechanism around local storage. In this skill's context, stored eval artifacts can include sensitive prompts, answers, and retrieved contexts from private corpora, increasing confidentiality and retention risk.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium
Category
Not specified by scanner
Confidence
93% confidence
Finding

The PRD explicitly says evaluation results will be saved to local disk even though the skill metadata says raw content should not be persisted unless requested. In a RAG evaluation workflow, the question, answer, contexts, and derived scores may contain sensitive enterprise data, so default persistence creates a real data handling and privacy risk.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The trigger phrases are broad and conversational, making accidental invocation more likely in normal agent interactions. Because this skill may install packages, call subprocesses, persist evaluation outputs, and send data to external LLM judges, unintended activation can expose sensitive content or perform side effects without clear user intent.

Content

No source excerpt is available for this finding.

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
90% confidence
Finding

The skill instructs the agent to access environment variables and read/write temporary files, but it does not declare any explicit tool scope or permissions boundary. That mismatch can lead to overbroad execution in hosts that rely on manifest-declared capabilities, increasing the risk of unintended secret exposure or file access beyond what users expect.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

The manifest frames the skill as evaluating a specifically provided RAG question, answer, and retrieved contexts. This script instead accepts a JSONL dataset and iterates across many samples, performing aggregation and producing a batch report, which is a broader capability than single-example evaluation.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
93% confidence
Finding

The --include-content option can embed raw questions and references in the batch report without any explicit disclosure warning, increasing the chance that sensitive evaluation data is copied into shareable artifacts. In a RAG setting, questions and references may contain confidential prompts, customer data, or gold answers, so accidental disclosure is plausible.

Content

No source excerpt is available for this finding.

Ssd 3

Medium
Category
Not specified by scanner
Confidence
90% confidence
Finding

The optional flags intentionally enable retention of user-provided evaluation material, including report inclusion and per-sample saved records. In this skill context, that is more dangerous because RAG inputs commonly contain retrieved enterprise content and model outputs that users may expect not to be persisted unless explicitly and clearly requested.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

When --save-individual is enabled, the script writes full question, answer, and context payloads to disk, which may contain sensitive user or proprietary retrieval data. Because this is a RAG-evaluation tool, those contexts often include private documents, making local persistence a meaningful data leakage and retention risk if users enable the flag without strong warning or controls.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The manifest describes a RAG evaluation skill that should not persist raw content unless requested. This script always writes a JSON report to disk, and the report includes the input file path and per-sample results; combined with the optional content flags and batch-processing behavior, the script is oriented around persistent reporting rather than purely transient evaluation of a specifically provided question/answer/context tuple.

Content

No source excerpt is available for this finding.

Dynamic import via __import__()

Medium
Category
Dangerous Code Execution
Confidence
75% confidence
Finding

Dynamic import() can load arbitrary modules at runtime, bypassing static analysis and potentially importing malicious code.

Content

Scanner excerpt · scripts/run_eval.py (reported line 51)May include surrounding context.

python
missing = []
    for pkg in ("ragas", "datasets"):
        try:
            __import__(pkg)
        except ImportError:
            missing.append(pkg)
    if missing:

Ssd 3

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

When --save is used, the script writes the raw question, answer, and retrieved contexts to local JSONL files under memory/eval-results. In this skill, those fields are likely to contain sensitive RAG inputs, internal documents, prompts, or user data, so plaintext persistence increases the risk of local data exposure, unintended retention, and later misuse by other tools or users on the same system.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Low
Category
Not specified by scanner
Confidence
87% confidence
Finding

The manifest describes a skill for evaluating a provided RAG question, answer, and contexts with Ragas metrics. The PRD broadens the capability to 'Ragas + DeepEval' and later plans additional DeepEval-based checks, which are not justified by the narrower stated purpose in the manifest.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Low
Category
Not specified by scanner
Confidence
89% confidence
Finding

The module docstring states that the optional reference is 'not used by metrics, kept for records,' which minimizes its role, but later code conditionally stores the reference in result entries when include-content is enabled. This documentation does not accurately reflect that reference content may be persisted into generated reports.

Content

No source excerpt is available for this finding.

Static analysis

Detected: suspicious.dynamic_code_execution

Dynamic code execution detected.

Critical
Code
suspicious.dynamic_code_execution
Location
scripts/test_selection.py:18