Security audit

RAG Evaluation Toolkit

Security checks across malware telemetry and agentic risk

Overview

This skill is a coherent RAG evaluation tool that discloses its cloud-judge and optional local raw-data storage behavior.

Install only if you are comfortable evaluating RAG content with either a local judge or an explicitly approved cloud judge. Do not set RAGAS_ALLOW_CLOUD=1 for confidential data unless provider transmission is acceptable, and avoid --save, --include-content, or --save-individual unless you intentionally want local plaintext evaluation records.

SkillSpector

By NVIDIA

Vulnerability Patterns

Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access

Findings (9)

Description-Behavior Mismatch

Medium

Confidence: 96% confidence
Finding: The PRD explicitly specifies saving evaluation results to `memory/eval-results/YYYY-MM-DD.jsonl`, which conflicts with the skill metadata stating raw content should not be persisted unless requested. In this skill context, the evaluated question, answer, and retrieved contexts may contain sensitive prompts, proprietary documents, or customer data, so default persistence creates an avoidable data-retention and privacy risk.

Description-Behavior Mismatch

Medium

Confidence: 95% confidence
Finding: The script always writes a JSON report containing per-sample results and metadata to disk, even though the skill description says raw content should not be persisted unless requested. While the current default report does not include full question/answer/context fields unless flags are set, it still persists evaluation artifacts by default and offers easy opt-in paths to store raw content, creating a privacy and policy mismatch for potentially sensitive RAG evaluation data.

Vague Triggers

Medium

Confidence: 87% confidence
Finding: The trigger phrases such as 'evaluate this response', 'check hallucination', and 'quality check' are broad enough to match ordinary conversation, increasing the chance the skill runs unintentionally. In this skill, accidental invocation could send user-supplied content to external judge models or trigger storage/processing of sensitive RAG data without clear user intent.

Missing User Warnings

Medium

Confidence: 95% confidence
Finding: The PRD describes both persistent storage and use of OpenAI or Anthropic as judge models, but does not warn users that their prompts, answers, and retrieved contexts may be retained locally or transmitted to third parties. In a RAG evaluation workflow, those inputs often include confidential enterprise knowledge, making undisclosed storage and third-party handling a meaningful privacy and compliance risk.

Missing User Warnings

Medium

Confidence: 93% confidence
Finding: The --save-individual option persists each raw question, answer, and context record to disk without any built-in warning, consent check, or redaction. In a RAG evaluation workflow, these fields may contain proprietary documents, personal data, or regulated content, so silent storage materially increases disclosure risk if logs or result directories are accessed later.

Missing User Warnings

Medium

Confidence: 91% confidence
Finding: The --include-content flag allows raw questions and references to be embedded into the batch report without any warning that the report may contain sensitive or proprietary material. Because reports are then written to disk, this increases the chance of unintended retention, sharing, or backup of confidential evaluation inputs.

Missing User Warnings

Medium

Confidence: 90% confidence
Finding: When --save is used, the script persists the full raw question, answer, and retrieved contexts to disk in memory/eval-results without any redaction, minimization, or explicit sensitivity warning at the point of use. In a RAG-evaluation workflow, these fields can easily contain proprietary documents, PII, credentials, or regulated data, creating a local data-retention and disclosure risk.

Ssd 3

Medium

Confidence: 84% confidence
Finding: The file-level documented behavior advertises optional raw persistence of evaluation inputs and results, which is a real data-retention risk for a tool designed to process arbitrary user-provided questions and retrieved contexts. Because RAG contexts often originate from sensitive corpora, even optional plaintext storage can expose confidential material if users enable it casually or if the host is shared or later compromised.

Ssd 3

Medium

Confidence: 96% confidence
Finding: The saved record includes the full input payload—question, answer, and all retrieved contexts—written as plaintext JSONL. If those contexts contain internal documents or sensitive user content, anyone with filesystem access, backups, logs, or incident-response artifacts could recover the raw data long after evaluation is complete.

VirusTotal

65/65 vendors flagged this skill as clean.

View on VirusTotal

Static analysis

Detected: suspicious.dynamic_code_execution

Dynamic code execution detected.

Critical

Code: suspicious.dynamic_code_execution
Location: scripts/test_selection.py:18