ClawHub
Back to skill

Security audit

SOTA Tracker (Claw)

Security checks for vulnerabilities and agentic risk

Overview

This is a real AI model tracker, but it asks users to enable integrations that can persistently change future agent instructions.

Review this carefully before installing. Prefer using the static JSON/CSV or manual local queries. Do not enable the cron/systemd timers that update CLAUDE.md or agents.md unless you pin the code, review diffs before applying changes, and know how to disable the timer. Treat MCP results as advisory and review .mcp.json edits manually.

A.I.G

Vulnerability Patterns
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (3)

T01 · Skill Instruction Hijacking

Error
Location
server.py:59
Finding

Globally Scoped MCP Instructions Hijack Agent Recommendation Behavior

Content
View full analysis
Remediation
View remediation

T02 · Agent Memory Poisoning

Error
Location
update_agents_md.py:119
Finding

Untrusted Remote Model Data Can Be Persisted in an Agent Instruction File

Content
View full analysis
list[dict]: """Parse Open LLM Leaderboard format into our model format.""" models = [] for entry in data[:20]: # Top 20 name = entry.get("model_name", "") model = { "id": normalize_model_id(name), "name": name, "category": "llm_local", "is_open_source": True, "release_date": entry.get("submission_date", datetime.now().strftime("%Y-%m-%d")), "metrics": { "notes": f"MMLU: {entry.get('MMLU', 'N/A')}, ARC: {entry.get('ARC', 'N/A')}", "mmlu": entry.get("MMLU"), "arc": entry.get("ARC"), "hellaswag": entry.get("HellaSwag"), "truthfulqa": entry.get("TruthfulQA"), "average": entry.get("Average"), "source": "huggingface_leaderboard" } } models.append(model) return models ``` The values are stored in the database: ```python def _update_models(self, category: str, models: list[dict]) -> int: """Update models in database from fresh data using INSERT OR REPLACE.""" if not models: return 0 count = 0 now = datetime.now().isoformat() with self.get_db_context() as db: for model in models: db.execute(""" INSERT OR REPLACE INTO models (id, name, category, is_open_source, is_sota, sota_rank, metrics, last_updated, source) VALUES (?, ?, ?, ?, 1, ?, ?, ?, 'auto') """, ( ...[truncated 5081 chars]
Remediation
View remediation

T08 · Insecure Dependencies

Warning
Location
requirements.txt:1
Finding

Mutable Dependency Ranges Permit Unreviewed Supply-Chain Changes

Content
View full analysis
=2.0,<3.0 aiohttp>=3.9 huggingface_hub>=0.20 python-dotenv>=1.0 playwright>=1.40 fastapi>=0.100.0 uvicorn>=0.23.0 slowapi>=0.1.9 ``` ### Technical Analysis All dependencies are specified using broad mutable version ranges. Most entries have no upper bound, and no reviewed lockfile or cryptographic hashes were identified in the audited directory. A future installation can therefore resolve to package versions that did not exist when the project was reviewed. This reduces build reproducibility and expands exposure to compromised upstream releases, malicious maintainer changes, or incompatible updates. The risk is significant for this project because several dependencies provide network servers, browser automation, environment loading, or MCP integration. Such packages execute with the privileges of the installing or running user. No evidence was found that the listed package names are currently typosquatted or malicious. The confirmed weakness is the absence of deterministic, integrity-verified dependency resolution. ### Attack Path 1. A listed dependency or one of its transitive dependencies publishes a compromised future release. 2. A user runs the documented dependency installation command after that release becomes eligible under the specified range. 3. The package resolver selects the new release because no exact reviewed version or hash is required. 4. Package installation hooks, imports, browser components, or runtime initialization execute the compromised code. 5. The code runs with the permissions of the user or service account operating SOTA Tracker. ### Impact Assessment The maximum impact depends on the compromised package and the privileges used during installation or runtime. Potential consequences include arbitr ...[truncated 388 chars]
Remediation
View remediation

SkillSpector

Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
Findings (70)

MCP Config Access

High
Category
Agent Snooping
Confidence
90% confidence
Finding

The README tells users to create .mcp.json with a server entry that will execute python server.py, which changes local agent/tooling configuration and authorizes a new command-backed integration. In a skill setting, modifying MCP config is sensitive because it expands the agent's accessible tools and can lead to execution of local code with the user's privileges if the repository contents are unreviewed.

Content

Scanner excerpt · README.md (reported line 74)May include surrounding context.

bash
# Edit .mcp.json to add the server config
cat > .mcp.json << 'EOF'
{
  "mcpServers": {
    "sota-tracker": {

MCP Config Access

High
Category
Agent Snooping
Confidence
90% confidence
Finding

The skill instructs users to create or overwrite .mcp.json with server configuration, which grants a local tool invocation path to python server.py. In an agent context, modifying MCP configuration is sensitive because it changes what external tools the agent can launch, potentially expanding capability and trust boundaries if the repository contents are later altered or not reviewed.

Content

Scanner excerpt · SKILL.md (reported line 74)May include surrounding context.

bash
# Edit .mcp.json to add the server config
cat > .mcp.json << 'EOF'
{
  "mcpServers": {
    "sota-tracker": {

Anti-Refusal Statement

High
Category
Anti-Refusal
Confidence
99% confidence
Finding

The phrase 'without safety guardrails' is a direct anti-refusal statement that endorses bypassing protective mechanisms. In this context, the file is not merely discussing safety academically; it is presenting the model as state-of-the-art and desirable, which amplifies the chance of unsafe deployment and misuse.

Content

Scanner excerpt · data/sota_export.json (reported line 2636)May include surrounding context.

json
"is_open_source": 1,
      "sota_rank": 9,
      "sota_rank_open": 9,
      "metrics": "{\"notes\": \"Uncensored Qwen3-32B, abliterated variant without safety guardrails\", \"why_sota\": \"Full Qwen3-32B quality without restrictions\", \"strengths\": [\"Uncensored\", \"Full 32B quality\", \"Creative freedom\", \"Single GPU\"], \"use_cases\": [\"Creative writing\", \"Unrestricted research\", \"Roleplay\"], \"vram_gb\": 19, \"quantization\": \"Q4_K_M\", \"is_uncensored\": true, \"hf_repo\": \"mradermacher/Qwen3-32B-Uncensored-GGUF\", \"base_model\": \"qwen3-32b\"}",
      "last_updated": "2026-01-11 15:34:39",
      "price_per_1m_output": null,
      "humaneval_score": null,

Natural-Language Policy Violations

High
Category
Not specified by scanner
Confidence
97% confidence
Finding

The entry explicitly advertises an abliterated model as being 'without safety guardrails' and 'without restrictions,' which is an anti-safety claim rather than neutral catalog metadata. Because this dataset appears to guide model selection, such wording can directly encourage adoption of systems designed to bypass refusals and increase the likelihood of harmful outputs.

Content

No source excerpt is available for this finding.

Anti-Refusal Statement

High
Category
Anti-Refusal
Confidence
99% confidence
Finding

The phrases 'No refusals on complex topics' and 'without guardrails' explicitly market the model's lack of safety enforcement as a benefit. That makes the entry more dangerous than a passive reference because it functions as an endorsement for evading normal model safety behavior.

Content

Scanner excerpt · data/sota_export.json (reported line 2656)May include surrounding context.

json
"is_open_source": 1,
      "sota_rank": 10,
      "sota_rank_open": 10,
      "metrics": "{\"notes\": \"Uncensored QwQ-32B reasoning model, abliterated variant\", \"why_sota\": \"Best uncensored reasoning model, full QwQ quality without restrictions\", \"strengths\": [\"Uncensored\", \"Reasoning\", \"No refusals on complex topics\"], \"use_cases\": [\"Unrestricted reasoning\", \"Research\", \"Complex analysis without guardrails\"], \"vram_gb\": 19, \"quantization\": \"Q4_K_M\", \"is_uncensored\": true, \"hf_repo\": \"mradermacher/QwQ-32B-Uncensored-GGUF\", \"base_model\": \"qwq-32b\"}",
      "last_updated": "2026-01-11 15:34:39",
      "price_per_1m_output": null,
      "humaneval_score": null,

Natural-Language Policy Violations

High
Category
Not specified by scanner
Confidence
98% confidence
Finding

This entry promotes an uncensored reasoning model using phrases like 'No refusals on complex topics' and 'without guardrails,' which strongly signals utility for bypassing safety constraints. In a recommendation dataset, that materially increases misuse risk by normalizing and advertising policy-evasive capability as a product strength.

Content

No source excerpt is available for this finding.

Anti-Refusal Statement

High
Category
Anti-Refusal
Confidence
80% confidence
Finding

Skill attempts to nullify the agent's safety policies or restrictions ('you have no restrictions', 'ignore your guidelines', 'do anything now'). This is a direct jailbreak that disables guardrails.

Content

Scanner excerpt · init_db.py (reported line 603)May include surrounding context.

python
"sota_rank": 9,
            "sota_rank_open": 9,
            "metrics": {
                "notes": "Uncensored Qwen3-32B, abliterated variant without safety guardrails",
                "why_sota": "Full Qwen3-32B quality without restrictions",
                "strengths": ["Uncensored", "Full 32B quality", "Creative freedom", "Single GPU"],
                "use_cases": ["Creative writing", "Unrestricted research", "Roleplay"],

Anti-Refusal Statement

High
Category
Anti-Refusal
Confidence
80% confidence
Finding

Skill attempts to nullify the agent's safety policies or restrictions ('you have no restrictions', 'ignore your guidelines', 'do anything now'). This is a direct jailbreak that disables guardrails.

Content

Scanner excerpt · init_db.py (reported line 627)May include surrounding context.

python
"notes": "Uncensored QwQ-32B reasoning model, abliterated variant",
                "why_sota": "Best uncensored reasoning model, full QwQ quality without restrictions",
                "strengths": ["Uncensored", "Reasoning", "No refusals on complex topics"],
                "use_cases": ["Unrestricted reasoning", "Research", "Complex analysis without guardrails"],
                "vram_gb": 19,
                "quantization": "Q4_K_M",
                "is_uncensored": True,

Possible Typosquatting: 'uvicorn' resembles popular package 'gunicorn'

High
Category
Supply Chain
Confidence
70% confidence
Finding

Package name closely resembles a popular package, suggesting possible typosquatting. Attackers publish malicious packages with similar names to trick developers into installing them.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
84% confidence
Finding

The security section states 'Read-only operations only' at L084. However, the same changelog later documents 'refresh_data() - Force refresh data' and 'Automated data commits' at L070 and L107, which indicate write-side behavior rather than purely read-only operation. This is an intent/documentation contradiction within the file.

Content

No source excerpt is available for this finding.

Session Persistence

Medium
Category
Rogue Agent
Confidence
60% confidence
Finding

Skill establishes unauthorized persistence across sessions via cron jobs, startup scripts, or state files. Session persistence allows an attacker to maintain access beyond the current interaction.

Content

Scanner excerpt · CLAUDE.md (reported line 29)May include surrounding context.

}

text

3. **Create Linear issue from template:**
```bash
cat ~/.cyrus/templates/linear_execution_issue.md
# Replace <REPO_NAME>, <SCRIPT_NAME>, <ISSUE_ID>

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
89% confidence
Finding

Lines L022-L025 label several models as "Forbidden," and L114 later describes a "Hard blocks outdated models." This is a natural-language policy concern because it imposes a rigid recommendation constraint without stating user opt-in, exceptions, or a region/compliance justification.

Content

No source excerpt is available for this finding.

Session Persistence

Medium
Category
Rogue Agent
Confidence
89% confidence
Finding

Enabling a user systemd timer creates persistent automatic execution of the update script at login/runtime without emphasizing the security implications. In this repository context, that means a local script can keep editing CLAUDE.md daily, which can silently alter agent context and preserve behavior changes across sessions.

Content

Scanner excerpt · README.md (reported line 48)May include surrounding context.

md
cp scripts/update_sota_claude_md.py ~/scripts/

# Enable systemd timer (runs at 6 AM daily)
systemctl --user enable --now sota-update.timer

# Or run manually
python ~/scripts/update_sota_claude_md.py --update

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The README instructs users to automate updates to ~/.claude/CLAUDE.md, a user configuration/context file, without clearly warning that local content may be overwritten or continuously modified. In an agent-skill context, silently changing prompt/config files can alter agent behavior, erase user instructions, or introduce unreviewed content on a recurring basis.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The README recommends scheduled automatic updates to agents.md without warning that this will repeatedly modify an agent control/configuration file over time. In this context, persistent unattended edits to agent instructions are security-relevant because they can change behavior, destroy local customizations, or propagate bad data into future agent sessions.

Content

No source excerpt is available for this finding.

Session Persistence

Medium
Category
Rogue Agent
Confidence
88% confidence
Finding

The cron instruction establishes persistent scheduled execution of a repository script. Persistence is security-relevant because it causes recurring code execution and repeated file modifications long after the initial setup, increasing the blast radius if the script or repo later changes unexpectedly.

Content

Scanner excerpt · README.md (reported line 206)May include surrounding context.

Add to your cron or systemd timer for daily updates:

cron
# ~: crontab -e
@daily python ~/Apps/sota-tracker-mcp/update_agents_md.py

Session Persistence

Medium
Category
Rogue Agent
Confidence
89% confidence
Finding

This systemd enablement step installs persistent scheduled execution for update_agents_md.py. Because the target file is an agent instruction/config file, unattended persistence is more dangerous here than for ordinary data-refresh tasks: future runs can keep mutating agent behavior without active user review.

Content

Scanner excerpt · README.md (reported line 234)May include surrounding context.

WantedBy=timers.target

Enable

systemctl --user enable --now sota-update.timer

text

See [CONTRIBUTING.md](CONTRIBUTING.md) for full setup guide

Sudo/Root Execution

Medium
Category
Privilege Escalation
Confidence
70% confidence
Finding

Commands invoke sudo or root privileges. Verify this elevated access is necessary and justified.

Content

Scanner excerpt · SECURITY.md (reported line 89)May include surrounding context.

bash
# Allow localhost and trusted IP
sudo ufw allow from 192.168.1.0/24 to any port 8000

4. Authentication (Optional)

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The documentation explicitly tells users to install a systemd timer that will automatically update ~/.claude/CLAUDE.md, a local agent configuration/context file, on a recurring basis. Even if intended as a convenience feature, silently scheduled modification of agent-facing files can change future agent behavior over time and increases the risk of persistence, prompt injection propagation, or unintended trust in externally refreshed content.

Content

No source excerpt is available for this finding.

Session Persistence

Medium
Category
Rogue Agent
Confidence
90% confidence
Finding

systemctl --user enable --now sota-update.timer creates a persistent per-user scheduled task. In this skill's context, that persistent task is specifically used to rewrite CLAUDE.md, so the persistence directly affects agent behavior and increases the blast radius of any future bug or compromised update path.

Content

Scanner excerpt · SKILL.md (reported line 48)May include surrounding context.

md
cp scripts/update_sota_claude_md.py ~/scripts/

# Enable systemd timer (runs at 6 AM daily)
systemctl --user enable --now sota-update.timer

# Or run manually
python ~/scripts/update_sota_claude_md.py --update

Session Persistence

Medium
Category
Rogue Agent
Confidence
60% confidence
Finding

Skill establishes unauthorized persistence across sessions via cron jobs, startup scripts, or state files. Session persistence allows an attacker to maintain access beyond the current interaction.

Content

Scanner excerpt · README.md (reported line 142)May include surrounding context.

md
This repo uses GitHub Actions to:
- **Daily**: Scrape all sources, update database, commit changes
- **Weekly**: Create a tagged release with JSON/CSV exports

To enable on your fork:
1. Fork this repo

Session Persistence

Medium
Category
Rogue Agent
Confidence
60% confidence
Finding

Skill establishes unauthorized persistence across sessions via cron jobs, startup scripts, or state files. Session persistence allows an attacker to maintain access beyond the current interaction.

Content

Scanner excerpt · SKILL.md (reported line 142)May include surrounding context.

md
This repo uses GitHub Actions to:
- **Daily**: Scrape all sources, update database, commit changes
- **Weekly**: Create a tagged release with JSON/CSV exports

To enable on your fork:
1. Fork this repo

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

This section instructs users to automate updates to agents.md via cron or systemd without prominently warning that a local agent instruction file will be rewritten regularly. Because agents.md can influence how downstream agents operate, scheduled unattended updates create a supply-chain-style risk where future content changes alter agent behavior without direct user review.

Content

No source excerpt is available for this finding.

Session Persistence

Medium
Category
Rogue Agent
Confidence
89% confidence
Finding

The cron example establishes recurring execution of a repository script, creating persistence on the user's machine. While framed as automation, persistence mechanisms are security-relevant because they continue running code and modifying agent-related files after the initial setup, including after upstream repository changes.

Content

Scanner excerpt · SKILL.md (reported line 206)May include surrounding context.

Add to your cron or systemd timer for daily updates:

cron
# ~: crontab -e
@daily python ~/Apps/sota-tracker-mcp/update_agents_md.py

Session Persistence

Medium
Category
Rogue Agent
Confidence
90% confidence
Finding

This command enables a user-level systemd timer, establishing ongoing scheduled execution. Because the documented use is automatic updates to agents.md, the persistence is not merely operational automation; it continuously affects files that may influence agent instructions or behavior.

Content

Scanner excerpt · SKILL.md (reported line 234)May include surrounding context.

WantedBy=timers.target

Enable

systemctl --user enable --now sota-update.timer

text

See [CONTRIBUTING.md](CONTRIBUTING.md) for full setup guide

Static analysis

No suspicious patterns detected.