Security audit

Azure Ai Evaluation Py

Security checks across malware telemetry and agentic risk

Overview

This is a coherent Azure AI evaluation skill that uses Azure credentials and may send evaluation data to Azure services when the user runs those workflows.

Install this only if you intend to use Azure AI Evaluation. Use least-privilege Azure credentials, prefer a virtual environment, and review or redact datasets before AI-assisted evaluators, safety evaluators, prompt-based evaluators, or Foundry logging because evaluation content and results may be processed or stored by Azure services.

SkillSpector

By NVIDIA

Vulnerability Patterns

Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access

Findings (7)

Lp3

Medium

Category: MCP Least Privilege
Confidence: 88% confidence
Finding: The skill demonstrates capabilities to read environment variables and potentially write files or logs, but it does not declare corresponding permissions or clearly scope those operations. This creates a transparency and consent problem: a caller may invoke the skill without realizing it can access secrets or persist evaluation outputs containing sensitive prompts and responses.

Vague Triggers

Medium

Confidence: 80% confidence
Finding: The trigger phrases include generic terms such as 'evaluate' and 'evaluators', which may cause the skill to activate in unrelated contexts. Over-broad activation can increase the chance that sensitive user data is routed into this skill unexpectedly, especially because the skill supports remote evaluation and project logging.

Missing User Warnings

Medium

Confidence: 94% confidence
Finding: The documentation promotes remote evaluation support and Foundry logging without warning that prompts, model responses, context, and evaluation datasets may be transmitted to Azure services. In a security-sensitive environment, this omission can lead users to send proprietary or regulated data off-box without informed consent.

Missing User Warnings

Medium

Confidence: 92% confidence
Finding: The examples show sending `query`, `context`, and `response` content to Azure-hosted evaluators but do not disclose that this data leaves the local environment and is transmitted to an external service. In an evaluation skill, users may paste sensitive prompts, documents, or model outputs, so omission of a data-transfer warning can lead to unintended disclosure of proprietary or regulated data.

Missing User Warnings

Medium

Confidence: 95% confidence
Finding: The safety evaluator section instructs users to configure Azure project scope and pass user content into managed safety services without warning that the content is transmitted to Azure. Because safety evaluations often involve highly sensitive or harmful text, failing to disclose external processing increases the risk of privacy, compliance, and data-governance violations.

Missing User Warnings

Medium

Confidence: 91% confidence
Finding: The prompt-based evaluator example transmits raw query and response content to Azure OpenAI, but the documentation does not warn users that application data may leave the local process and be processed by an external service. In an evaluation SDK context, those fields can contain sensitive prompts, proprietary context, or user data, so omission of a privacy/data-handling warning can lead to unintentional exposure.

Missing User Warnings

Medium

Confidence: 94% confidence
Finding: The multi-criteria evaluator loops over criteria and sends query, response, and context to an external model multiple times, increasing both exposure surface and volume of transmitted data. Because the documentation lacks any disclosure about external transmission, privacy implications, or cost/amplification effects, users may unknowingly process sensitive content through repeated third-party API calls.

VirusTotal

65/65 vendors flagged this skill as clean.

View on VirusTotal

Static analysis

Detected: suspicious.exposed_secret_literal, suspicious.prompt_injection_instructions

File appears to expose a hardcoded API secret or token.

Critical

Code: suspicious.exposed_secret_literal
Location: references/custom-evaluators.md:151

Prompt-injection style instruction pattern detected.

Warn

Code: suspicious.prompt_injection_instructions
Location: references/built-in-evaluators.md:322