Back to skill

Security audit

SOTA Agent

Security checks for vulnerabilities and agentic risk

Overview

This skill is a local SOTA research-planning helper with disclosed campaign file generation, but users should not rely on its rendered summaries as complete secret-redaction tooling.

Install only if you want local SOTA campaign planning files and review artifacts. Keep outputs in a dedicated workspace, do not put secrets or private notebook/session data into candidate or program JSON, and manually review generated Markdown before publishing because the report renderer is not a complete public-release scrubber.

Vulnerability Patterns
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (1)

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/sota_public_safety.py:307
Finding

Incomplete Secret Redaction and Markdown Injection in Generated Audit Summaries

Content
View full analysis
str: value = value.strip() if not value: return "" if "://" in value: return sanitize_url_for_display(value, allow_raw=allow_raw) if is_absolute_like(value): return sanitize_path(value, allow_absolute_paths=allow_raw) return value ``` Both summary renderers recursively process string values without retaining the associated field name. They then place the resulting content directly into Markdown: ```python def sanitize_value(value: Any) -> Any: if isinstance(value, str): return sanitize_ref(value) if isinstance(value, list): cleaned = [] for item in value: sanitized = sanitize_value(item) if sanitized in ("", [], {}): continue cleaned.append(sanitized) return cleaned if isinstance(value, dict): cleaned = {} for key, item in value.items(): sanitized = sanitize_value(item) if sanitized in ("", [], {}): continue cleaned[key] = sanitized return cleaned return value def add_field(lines: list[str], label: str, value: Any) -> None: value = sanitize_value(value) if value in (None, "", [], {}): return rendered = json.dumps(value, ensure_ascii=True, sort_keys=True) if isinstance(value, (list, dict)) else str(value) lines.append(f"- {label}: `{rendered}`") ``` The generated content is written as a supposedly sanit ...[truncated 3155 chars]
Remediation
View remediation
Vulnerability Patterns
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Findings (10)

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

The declared description is about an interactive SOTA research assistant for computer vision and data-science workflows, including benchmark planning and paper/claim analysis. The actual code does none of that. Instead, it performs filesystem-oriented export support: parsing CLI arguments, resolving paths, sanitizing them, checking file existence/type, and emitting a machine-readable JSON manifest. This is a materially different primary purpose and capability set from the declared skill behavior, so it should be flagged as a mismatch.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding

The declared description presents an analysis-oriented SOTA agent for CV/DS research tasks such as benchmark scouting, paper triage, ablation design, and claim review. The actual code does not perform any of those research or planning functions. Instead, it is an initialization script for generating a sanitized evidence-card JSON template for an external browsing/review workflow. This is a materially different primary purpose: operational metadata/file generation rather than SOTA analysis assistance. No harmful extra permissions are evident, but the behavior is still mismatched because the code's implemented capability is infrastructure/support tooling unrelated to the declared end-user skill behavior.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
96% confidence
Finding

The declared description presents an operational skill for CV/DS SOTA work: benchmark planning, paper triage, ablation design, and claim review. The supplied code does not carry out any of those substantive tasks. Instead, it initializes a campaign folder structure and populates it with template documents and a CSV file. While this scaffold supports the stated workflow, the actual primary behavior is project setup on the local filesystem, a materially narrower and different function than the declared agent-like research capability. Therefore this is a description-behavior mismatch.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

The declared description presents a high-level SOTA research support skill for computer vision and data-science campaigns, including benchmark scouting, paper triage, ablation design, and claim review. The supplied code instead implements a narrow utility script that parses command-line arguments and writes a machine-readable validation scorecard JSON file to local storage. While such a script could be a supporting component in a larger campaign workflow, this code chunk itself does not perform the described research-planning or paper-analysis functions. Its primary purpose is materially different from the declared purpose, so this is a mismatch.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
96% confidence
Finding

The declared description presents an analysis-oriented SOTA agent for CV/data-science campaign support: benchmark planning, paper triage, ablation design, and claim review. The supplied code does none of that. It is an operational bootstrap/metadata utility that parses command-line inputs and writes a sanitized JSON manifest for a long external run. While the filename and comments mention 'SOTA run,' that only relates to experiment bookkeeping, not the declared research-support behavior. This is a materially different primary purpose, so it should be flagged as a mismatch.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

The code’s primary function is deterministic report generation: it reads JSON files, sanitizes string/list/dict contents, and writes a markdown summary. It does not search literature, scout benchmarks, analyze papers, plan experiments, design ablations, or evaluate claims beyond displaying already-supplied metadata. This is a materially different purpose from the declared description of an interactive SOTA agent for CV/DS campaign work. No suspicious extra permissions or unrelated external resource access are present, but the core behavior is substantially narrower and different from the declared skill purpose.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

The supplied code does not perform benchmark scouting, paper triage, ablation design, claim review, or any CV/DS campaign logic. Its primary function is privacy and safety sanitization of artifacts: detecting sensitive keys, credential-bearing URLs, private hosts, and absolute paths, then redacting or aliasing them for safe display. That is a materially different purpose from the declared SOTA-agent description, so this is a clear description-behavior mismatch.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
96% confidence
Finding

The declared description presents a general-purpose 'SOTA Agent' for CV/DS campaign work, including higher-level analytical tasks such as benchmark scouting, paper triage, ablation design, and claim review. The supplied code does not implement those behaviors. Instead, it only parses CLI entries, sorts candidates by score on a fixed metric, computes ranks and deltas, and emits a JSON scoreboard file. While this is loosely related to SOTA tracking, the actual code is much narrower and materially different in primary purpose from the declared agent-style analytical functionality.

Content

No source excerpt is available for this finding.

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
87% confidence
Finding

The skill explicitly instructs use of local Python scripts that read and write campaign artifacts, and it also references public URLs, yet it declares no explicit tool scope or permission boundaries. Missing scope metadata can cause the runtime to grant broader-than-necessary file or network access, increasing the chance of unintended data exposure or misuse if the skill is invoked in a sensitive workspace.

Content

No source excerpt is available for this finding.

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
75% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · SKILL.md (reported line 119)May include surrounding context.

md
- Read only the papers or repos that change the candidate plan.
- Extract the minimum useful fields: task, metric, split, data, compute, architecture, augmentations, training tricks, and caveats.
- Prefer a reproduced strong baseline over copying five tricks from five papers without control.
- Do not treat leaderboard rows as ground truth without checking task definition and split rules.

### Ablation rules

Static analysis

No suspicious patterns detected.