SOTA AI Model Tracker

• Outcome: Review
• Latest audit: Jul 10, 2026 · 3:48 AM UTC
• Version: 1.0.0

Overview

This SOTA model tracker is mostly coherent, but it asks users to set up recurring jobs that edit agent instruction files and provides unsafe overwrite-style setup commands without enough guardrails.

Install only if you are comfortable with a tool that can influence future agent behavior. Prefer manual runs first, back up any existing `.mcp.json`, CLAUDE.md, or agents.md files, avoid enabling cron/systemd until you review the scripts and generated diff, and pin/update dependencies before running any exposed REST or MCP server.

SkillSpector

By NVIDIA

Vulnerability Patterns

• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands
• Supply Chain: Unpinned Dependencies, External Script Fetching, Obfuscated Code
• Rogue Agent: Self-Modification, Session Persistence
• Trigger Abuse: Overly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
• MCP Tool Poisoning: Hidden Instructions, Unicode Deception, Parameter Description Injection

Findings (21)

Intent-Code Divergence

Medium

• Confidence: 98% confidence
• Finding: This is a real destructive-behavior issue: despite being described as an initialization script, it unconditionally deletes any existing SQLite database file before recreating it. That can cause irreversible data loss if run against a populated or production database, especially because the behavior is not clearly disclosed by the documentation and there is no confirmation, backup, or environment guard.

Description-Behavior Mismatch

Medium

• Confidence: 87% confidence
• Finding: The skill is presented as an informational SOTA tracker, but `configure_hardware` is a state-mutating tool that updates persisted user profile data such as GPU, RAM, and preference flags. That expands the trust boundary from read-only lookup into profile storage, which can surprise users, create privacy concerns, and allow downstream recommendation behavior to be silently altered.

Description-Behavior Mismatch

Medium

• Confidence: 91% confidence
• Finding: `refresh_data` and the automatic cache refresh path mutate the local database and pull data from external sources, despite the skill being framed primarily as an informational tracker. This creates supply-chain and integrity risk: a compromised or unreliable upstream source could poison recommendations, and the mutation side effects are not obvious from a seemingly query-focused skill.

Missing User Warnings

Medium

• Confidence: 95% confidence
• Finding: The README instructs users to run a script that updates `~/.claude/CLAUDE.md`, a user-local agent configuration file, but does not clearly warn that local content may be overwritten or altered. In an agent/tooling context, silent modification of instruction-bearing files is security-relevant because it can change future agent behavior, persist unintended prompts, or destroy user customizations.

Missing User Warnings

Medium

• Confidence: 96% confidence
• Finding: The README promotes automated daily updates of `agents.md` via cron/systemd without warning that a recurring task will modify a local agent instruction file. Recurring unattended edits to agent-facing files increase risk because changes persist over time, may overwrite local policies, and can silently alter downstream agent behavior.

Missing User Warnings

Medium

• Confidence: 92% confidence
• Finding: The documentation tells users to create `.mcp.json` via shell redirection, which will overwrite any existing local MCP configuration without warning or merge behavior. In an agent/tooling context, silent replacement of local config can disable existing security controls, remove other trusted servers, or introduce unintended tool exposure.

Missing User Warnings

Medium

• Confidence: 94% confidence
• Finding: The skill recommends copying a script and enabling a systemd timer that will automatically update `~/.claude/CLAUDE.md`, but it does not clearly warn that a user file will be modified on a recurring basis. In a coding-agent environment, unattended writes to prompt or agent context files can alter future model behavior and create hard-to-audit persistence.

Missing User Warnings

Medium

• Confidence: 95% confidence
• Finding: The documentation promotes automated updates to `agents.md` through direct execution and scheduled cron/systemd jobs, but it does not warn users that this will continuously rewrite an agent instruction file. Because `agents.md` can influence agent behavior, recurring unattended edits create a persistence and integrity risk if the script is incorrect, compromised, or later changed.

Vague Triggers

Medium

• Confidence: 93% confidence
• Finding: The example skill description is broad enough to activate on many ordinary AI-related conversations, which can cause the skill to inject mandatory guidance outside narrowly intended contexts. Because the embedded instructions then steer recommendations and tool usage automatically, over-triggering increases the chance of unwanted influence on model outputs and user interactions.

Natural-Language Policy Violations

Medium

• Confidence: 95% confidence
• Finding: The skill hard-codes user preferences such as 'prefers uncensored models' and elevates them into default behavior without confirming that the current user actually wants that outcome. In a skill that auto-activates and shapes recommendations, this can bias outputs toward unsafe, policy-conflicting, or unwanted suggestions and override user autonomy.

Natural-Language Policy Violations

Medium

• Confidence: 90% confidence
• Finding: The server instructions explicitly elevate an `uncensored` preference and encourage prioritizing uncensored model variants without any safety framing, guardrails, or limitation context. In a recommendation skill, that can steer users toward models designed to bypass moderation and increase the chance of harmful or policy-evading outputs.

Natural-Language Policy Violations

Medium

• Confidence: 89% confidence
• Finding: The best-in-class output labels describe uncensored models as having 'no restrictions,' which normalizes and markets safety-bypass characteristics to end users. In context, this is more dangerous because the skill's purpose is recommendation and ranking, so the language can directly influence selection toward less safe models.

Session Persistence

Medium

• Category: Rogue Agent

Content

cp scripts/update_sota_claude_md.py ~/scripts/

# Enable systemd timer (runs at 6 AM daily)
systemctl --user enable --now sota-update.timer

# Or run manually
python ~/scripts/update_sota_claude_md.py --update

• Confidence: 88% confidence
• Finding: systemctl --user enable

Session Persistence

Medium

• Category: Rogue Agent

Content

cp scripts/update_sota_claude_md.py ~/scripts/

# Enable systemd timer (runs at 6 AM daily)
systemctl --user enable --now sota-update.timer

# Or run manually
python ~/scripts/update_sota_claude_md.py --update

• Confidence: 93% confidence
• Finding: systemctl --user enable

Known Vulnerable Dependency: fastmcp — 10 advisory(ies): CVE-2025-69196 (FastMCP OAuth Proxy token reuse across MCP servers); GHSA-c2jp-c369-7pvx (FastMCP Auth Integration Allows for Confused Deputy Account Takeover); CVE-2025-64340 (FastMCP has a Command Injection vulnerability - Gemini CLI) +7 more

Critical

• Category: Supply Chain
• Confidence: 95% confidence
• Finding: fastmcp

Known Vulnerable Dependency: aiohttp — 10 advisory(ies): CVE-2024-52303 (aiohttp has a memory leak when middleware is enabled when requesting a resource ); CVE-2026-54279 (aiohttp: Host-Only Cookies Become Domain Cookies After CookieJar Persistence); CVE-2026-34514 (AIOHTTP has CRLF injection through multipart part content type header constructi) +7 more

High

• Category: Supply Chain
• Confidence: 90% confidence
• Finding: aiohttp

Known Vulnerable Dependency: python-dotenv — 1 advisory(ies): CVE-2026-28684 (python-dotenv: Symlink following in set_key allows arbitrary file overwrite via )

Low

• Category: Supply Chain
• Confidence: 76% confidence
• Finding: python-dotenv

Known Vulnerable Dependency: fastmcp==2.0 — 10 advisory(ies): CVE-2025-69196 (FastMCP OAuth Proxy token reuse across MCP servers); GHSA-c2jp-c369-7pvx (FastMCP Auth Integration Allows for Confused Deputy Account Takeover); CVE-2025-64340 (FastMCP has a Command Injection vulnerability - Gemini CLI) +7 more

Critical

• Category: Supply Chain
• Confidence: 98% confidence
• Finding: fastmcp==2.0

Known Vulnerable Dependency: aiohttp==3.9 — 10 advisory(ies): CVE-2026-54279 (aiohttp: Host-Only Cookies Become Domain Cookies After CookieJar Persistence); CVE-2026-34514 (AIOHTTP has CRLF injection through multipart part content type header constructi); CVE-2026-34517 (AIOHTTP has late size enforcement for non-file multipart fields causes memory Do) +7 more

High

• Category: Supply Chain
• Confidence: 90% confidence
• Finding: aiohttp==3.9

Known Vulnerable Dependency: python-dotenv==1.0 — 1 advisory(ies): CVE-2026-28684 (python-dotenv: Symlink following in set_key allows arbitrary file overwrite via )

Low

• Category: Supply Chain
• Confidence: 78% confidence
• Finding: python-dotenv==1.0

Known Vulnerable Dependency: fastapi==0.100.0 — 1 advisory(ies): CVE-2024-24762 (FastAPI is a web framework for building APIs with Python 3.8+ based on standard )

High

• Category: Supply Chain
• Confidence: 93% confidence
• Finding: fastapi==0.100.0

VirusTotal

65/65 vendors flagged this skill as clean.

Static analysis

No suspicious patterns detected.