Model Watch

• Outcome: Review
• Latest audit: Sep 16, 2026 · 9:41 AM UTC
• Version: 1.0.0

Overview

The tool is a coherent local benchmark CLI, but its install instructions use mutable remote GitHub code that can differ from the reviewed artifact.

Review the source you will actually install. Prefer a pinned commit or signed/versioned release instead of the GitHub default branch install command, and avoid submitting sensitive model outputs because they are saved locally in `~/.hermes/model-watch-history.json`.

A.I.G

Vulnerability Patterns

• Insecure Dependencies: Introduces malicious components through unsafe dependency sources
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally

Findings (1)

T08 · Insecure Dependencies

Warning

• Location: SKILL.md:24

Finding

Unpinned Installation from a Mutable Remote Git Repository

Content

View full analysis

Vulnerability Details

File Location: SKILL.md:24
Vulnerability Type: Supply-chain risk from an unpinned remote dependency
Risk Level: Medium

Vulnerable Code

bash

pip install git+https://github.com/minirr890112-byte/model-watch.git

The same instruction also appears in README.md:11.

Technical Analysis

The documented installation command retrieves and installs the current contents of a remote Git repository without pinning an immutable commit SHA, verifying a package hash, or requiring a signed release. Consequently, the code installed by users can differ from the version audited here.

During installation, pip can invoke the remotely supplied Python build backend and package build logic. If the repository or its maintainer account is compromised—or if a later revision becomes malicious—the installation process may execute attacker-controlled code under the installing user's account.

The audited local implementation itself does not retrieve or execute remote payloads at runtime. The risk specifically arises from the mutable installation source documented by the project.

Attack Path

1. An attacker compromises the referenced GitHub repository, its maintainer account, or another mechanism capable of changing its default branch.
2. The attacker adds malicious package code or build-time behavior to that branch.
3. A user follows the documented pip install git+https://... command.
4. Pip downloads the attacker-controlled revision and invokes its package build or installation process.
5. Malicious code executes with the privileges of the user running pip and may persist as part of the installed package.

Impact Assessment

Successful exploitation can provide arbitrary code execution with the installing user's privileges. The attacker could access or modify files available to that user, steal credentials present in the user's environment, alter the installed package, or establish user-level persistence. If installation i ...[truncated 118 chars]

Remediation

View remediation

Remediation Suggestions

• Replace the mutable branch installation command with a versioned package from a trusted package registry.
• Publish signed, reproducible releases and document installation of an explicit version.
• If Git installation is necessary, pin the dependency to a reviewed immutable commit SHA, for example:

  bash

  pip install "git+https://github.com/minirr890112-byte/model-watch.git@<reviewed-full-commit-sha>"
  

• Prefer locked dependencies with verified cryptographic hashes, such as pip requirements using --require-hashes.
• Protect the source repository with multi-factor authentication, branch protection, mandatory review, and signed release tags.
• Keep SKILL.md, README.md, and package metadata synchronized so all installation instructions refer to the same reviewed release.

SkillSpector

Vulnerability Patterns

• Supply Chain: Unpinned Dependencies, External Script Fetching, Obfuscated Code
• YARA Signatures: Malware Match, Webshell Match, Cryptominer Match
• MCP Least Privilege: Underdeclared Capability, Wildcard Permission, Missing Permission Declaration
• MCP Tool Poisoning: Hidden Instructions, Unicode Deception, Parameter Description Injection
• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands

Findings (10)

YARA rule 'agent_skill_remote_bootstrap_execution': Remote script or code download followed by execution/bootstrap installation [agent_skills]

High

• Category: YARA Match
• Confidence: 89% confidence

Finding

The README instructs users to install directly from a remote GitHub repository via pip, which causes code from the referenced repository to be fetched and executed during package installation. In the context of an agent skill, this is dangerous because users may run unpinned, unaudited code from a mutable remote source, enabling supply-chain compromise if the repo is malicious, hijacked, or later changed.

Content

Scanner excerpt · README.md (reported line 11)May include surrounding context.

model-watch

Benchmark AI API models over time. Detect when they get dumber.

"Opus 4.7 was hallucinating a lot today... shocking to see such degradation" — r/ClaudeAI (49↑) "Anthropic admits to have made hosted models more stupid" — r/LocalLLaMA (281↑)

Install

bash

pip install git+https://github.com/minirr890112-byte/model-watch.git

Or from local:

bash

cd model-watch && pip install .

Usage

bash

# View benchmark questions
model-watch demo

# Submit model outputs (from your API calls)
model-watch submit '{"reasoning_1":"...", "coding_1":"...", ...}'

# View score history
model-watch history

# Check for degradation alerts
model-watch alert

Benchmarks

7 standardized tests covering:

• Reasoning (2 tests)
• Coding (2 tests)
• Writing (1 test)
• Instruction following (1 test)
• Hallucination detection (2 tests)

How it works

YARA rule 'agent_skill_remote_bootstrap_execution': Remote script or code download followed by execution/bootstrap installation [agent_skills]

High

• Category: YARA Match
• Confidence: 88% confidence

Finding

The Quick Start instructs users to install directly from a GitHub repository using pip install git+https://..., which performs remote code retrieval and package installation without pinning to a vetted release or immutable commit. This creates a software supply-chain risk: if the repository, dependency chain, or referenced branch is compromised, users may execute attacker-controlled code during installation.

Content

Scanner excerpt · SKILL.md (reported line 24)May include surrounding context.

(49↑). "Anthropic admits to have made hosted models more stupid" — r/LocalLLaMA (281↑). You're paying the same price for a dumber model and you don't even know it.

The solution: Standardized benchmark suite you run yourself. 7 tests across 5 categories. Scores stored locally. Alerts when recent scores drop >10% vs your historical average. Hard data, not vibes.

Quick Start

bash

pip install git+https://github.com/minirr890112-byte/model-watch.git

model-watch demo              # View benchmark questions
model-watch submit '{"reasoning_1":"...","coding_1":"...",...}'  # Submit outputs
model-watch history           # View score history
model-watch alert             # Check for degradation

Benchmarks (7 tests)

Category	Tests	What it measures
Reasoning	2	Logic, multi-step deduction
Coding	2	Code generation, debugging
Writing	1	Quality, coherence
Instruction-following	1	Precision, constra

Undeclared Tool Scope

Medium

• Category: MCP Least Privilege
• Confidence: 78% confidence

Finding

The skill documents local persistence to ~/.hermes/model-watch-history.json, which implies file write capability, but it does not declare any permissions or allowed-tools scope. In an agent ecosystem, undeclared write behavior weakens policy enforcement and transparency, making it easier for a skill to modify local state without explicit review.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

The manifest description defines a specific degradation condition: alert when scores drop more than 10% versus the historical average. In code, check_alert() only evaluates after 3 runs and uses the average of the last three scores versus older scores, which is materially different from the described behavior and can suppress or delay alerts that the manifest implies should fire.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low

• Category: Not specified by scanner
• Confidence: 86% confidence

Finding

This markdown file states that scores are stored locally in ~/.hermes/model-watch-history.json, which is a data-writing behavior affecting user data on disk. The README explains what happens, but it does not present this as a warning or disclosure about persistence, retention, or overwrite implications.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low

• Category: Not specified by scanner
• Confidence: 78% confidence

Finding

The heading ## 生态系统 introduces non-English content context in an otherwise English README, which can create an implicit language/locale inconsistency without user opt-in. The file does not explain that bilingual presentation is intentional or offer a language choice.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Low

• Category: Not specified by scanner
• Confidence: 99% confidence

Finding

The top-level docstring says the skill 'Runs 7 standardized tests,' which communicates the intended benchmark scope. However, BENCHMARKS contains 8 entries, so the inline documentation contradicts the actual implemented test set.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low

• Category: Not specified by scanner
• Confidence: 89% confidence

Finding

The submit flow appends user-provided model outputs and scores to ~/.hermes/model-watch-history.json, which is a file write affecting user data persistence. Although the path is printed after completion, there is no pre-action warning, confirmation, or explanatory docstring/comment near the write operation to disclose that submitted data will be stored locally.

Content

No source excerpt is available for this finding.

Unverifiable Dependency: setuptools has 10 known advisory(ies) (CVE-2013-1633 (Setuptools vulnerable to Man-in-the-middle attacks); CVE-2025-47273 (setuptools has a path traversal vulnerability in PackageIndex.download that lead); CVE-2024-6345 (setuptools vulnerable to Command Injection via package URL) +7 more), but the manifest does not pin a version, so it is unknown whether the installed release is affected

Low

• Category: Supply Chain
• Confidence: 40% confidence

Finding

Dependency has known vulnerabilities (CVEs). Using packages with unpatched security flaws exposes the environment to known exploits.

Content

No source excerpt is available for this finding.

Unverifiable Dependency: wheel has 4 known advisory(ies) (CVE-2026-24049 (Wheel Affected by Arbitrary File Permission Modification via Path Traversal in w); CVE-2022-40898 (pypa/wheel vulnerable to Regular Expression denial of service (ReDoS)); CVE-2022-40898 (An issue discovered in Python Packaging Authority (PyPA) Wheel 0.37.1 and earlie) +1 more), but the manifest does not pin a version, so it is unknown whether the installed release is affected

Low

• Category: Supply Chain
• Confidence: 40% confidence

Finding

Dependency has known vulnerabilities (CVEs). Using packages with unpatched security flaws exposes the environment to known exploits.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.