Security audit

ai-grader

Security checks for vulnerabilities and agentic risk

Overview

This is an offline AI-evaluation/reporting skill with disclosed local file storage, but users should treat generated reports and optional human-profile outputs as private unless deliberately redacted.

Install only if you are comfortable with an offline grader writing local JSON and HTML reports. Use de-identified AI names and test data, redact transcripts before turning them into judge prompts, and enable --with-human-profiling only with informed consent from the person being characterized.

SkillSpector

Vulnerability Patterns

Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection

Findings (10)

Lp3

Medium

Category: MCP Least Privilege
Confidence: 88% confidence
Finding: The skill metadata declares no permissions, yet the documented behavior indicates file read/write capabilities. This creates a transparency and consent gap: an operator may invoke the skill expecting a pure evaluation tool, while it can persist data or modify local artifacts. In a grading skill, hidden filesystem access is more sensitive because evaluation inputs may contain proprietary prompts, model outputs, or internal benchmark data.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 95% confidence
Finding: The declared purpose presents the skill as an AI grading tool, but the documented behavior expands into persistent record storage, shareable report generation, external judge prompt creation, and even personality/similarity analysis. This mismatch is dangerous because users and orchestrators may grant trust or route sensitive conversations into the skill without realizing it performs profiling, retention, and externalization of data. The human/persona analysis exclusion in metadata further increases concern because the static finding suggests behavior beyond stated boundaries.

Description-Behavior Mismatch

High

Confidence: 98% confidence
Finding: The skill expands from grading AI systems into profiling a human 'owner' via personality dimensions and similarity scoring. That is a material scope creep into human profiling, which creates privacy and consent risks and can produce sensitive inferences unrelated to the core purpose of AI grading. The added gating text helps, but does not eliminate the risk because the code still implements, renders, and stores human-personality analysis outputs.

Context-Inappropriate Capability

High

Confidence: 97% confidence
Finding: This section implements human personality profiling and 'owner similarity' analysis even though the skill is described as an AI reliability grader. Collecting and transforming H1-H9 owner traits into labeled judgments such as similarity classes and verdict narratives enables unnecessary profiling, potentially exposing personal characteristics and increasing misuse risk if reports are shared.

Intent-Code Divergence

Medium

Confidence: 90% confidence
Finding: The docstring says object names should be anonymized, but the code and demo data use directly identifying labels and persist them into JSON and multiple HTML reports. This mismatch can mislead users into believing data is de-identified when it is not, increasing the chance of accidental disclosure of evaluated entities or people.

Vague Triggers

Medium

Confidence: 79% confidence
Finding: Broad trigger keywords like 'ai' and 'grader' can cause unintended activation in unrelated conversations. Accidental activation matters here because this skill appears capable of storing records and generating artifacts, so a false trigger could lead to unnecessary processing of sensitive model outputs or internal discussions.

Missing User Warnings

Medium

Confidence: 86% confidence
Finding: The file instructs operators to send probe prompts to target AIs, preserve responses verbatim as transcripts, and then process score files, but it does not warn that those transcripts may contain sensitive, proprietary, personal, or policy-relevant data. Because several probes explicitly touch privacy, internal agreements, private phone numbers, and employment actions, retaining raw transcripts without minimization or handling guidance increases the risk of unnecessary collection and downstream exposure.

Missing User Warnings

Medium

Confidence: 94% confidence
Finding: The file instructs operators to collect and retain raw transcripts from probe interactions, including red-team and compliance/privacy scenarios, without any guidance on data minimization, redaction, consent, retention limits, or secure storage. In this context, transcripts may capture sensitive corporate information, personal phone numbers, internal agreements, or unsafe prompts/responses, creating a realistic confidentiality and privacy exposure.

Missing User Warnings

Medium

Confidence: 88% confidence
Finding: The skill writes persistent HTML dashboards and reports that inline full evaluation data, including latest records and potentially owner/profile information, without prominent privacy or sharing warnings. Because these files are designed for viewing and sharing, users may expose sensitive assessment data more broadly than intended.

Missing User Warnings

Medium

Confidence: 90% confidence
Finding: The document explicitly recommends that the AI automatically create versions, move files to recycle bin, and back up before any change. Even though framed as a safety improvement, it authorizes file-system modifying behavior without requiring explicit user confirmation or clearly bounded scope, which can normalize unauthorized changes to user data and create unexpected side effects in an agent implementation.

Static analysis

No suspicious patterns detected.