Semantic Consistency Auditor

• Outcome: Review
• Latest audit: Sep 16, 2026 · 4:31 AM UTC
• Version: 1.0.0

Overview

The skill is a clinical-note evaluation tool, but its package framing understates that sensitive medical text may be processed and stored.

Review this skill carefully before installing. Use it only in an isolated, non-privileged environment, avoid identifiable patient data unless your compliance controls allow it, pin and verify dependencies/models, and do not save or print raw clinical notes to shared logs or insecure output paths.

A.I.G

Vulnerability Patterns

• Insecure Dependencies: Introduces malicious components through unsafe dependency sources
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally

Findings (1)

T08 · Insecure Dependencies

Error

• Location: requirements.txt:1

Finding

Unpinned and Ambiguous Third-Party Dependencies Enable Supply-Chain Compromise

Content

View full analysis

Vulnerability Details

File Location: requirements.txt, lines 1–6
Vulnerability Type: Unpinned and potentially mismatched package dependencies
Risk Level: High

Vulnerable Code

text

bert_score
comet
dataclasses
numpy
torch
yaml

Technical Analysis

Every dependency is specified without an exact version or integrity hash. Consequently, installation resolves whatever package version is available from the configured package index at installation time. This makes builds non-reproducible and exposes users to compromised future releases, dependency takeover, and package-index substitution.

Several declarations are also ambiguous or inconsistent with the documented and imported modules:

• yaml is declared, while the commonly intended distribution providing the yaml module is PyYAML.
• comet is declared, while SKILL.md instructs users to install comet-ml, and scripts/main.py imports download_model and load_from_checkpoint from comet. These names refer to potentially different package ecosystems and do not clearly identify the intended COMET evaluation implementation.
• dataclasses is unnecessary under the documented Python 3.10+ baseline because it is part of the standard library.

The application imports these dependencies at startup in scripts/main.py, including bert_score, comet, numpy, and torch. Python packages may execute arbitrary code during installation and module import. Therefore, resolving an unintended or compromised distribution can lead directly to code execution under the account installing or running the skill.

No evidence in the audited files establishes that a currently published dependency is malicious. The confirmed weakness is the unsafe, unpinned, and inconsistent dependency specification.

Attack Path

1. A user follows the documented installation command:

   bash

   pip install -r requirements.txt
   

2. The package installer resolves unpinned names from its configure ...[truncated 1045 chars]

Remediation

View remediation

Remediation Suggestions

1. Identify and document the exact distributions that provide every imported module. Resolve the current inconsistency among comet, comet-ml, and the COMET evaluation API before release.

2. Replace ambiguous package names with the authoritative distribution names. For example, use PyYAML rather than yaml if PyYAML is intended.

3. Remove dataclasses when Python 3.10+ is required.

4. Pin all direct and transitive dependencies to reviewed versions using a lock file or generated constraints file.

5. Require cryptographic hashes during deployment, for example with a hash-locked requirements file and:

   bash

   pip install --require-hashes -r requirements.lock
   

6. Configure installers to use a trusted package index explicitly and disable untrusted extra indexes.

7. Scan locked dependencies for known vulnerabilities and review updates before regenerating the lock file.

8. Install dependencies in an isolated, non-privileged virtual environment or container without production secrets.

9. Add automated tests verifying that each declared distribution supplies the expected import and API, particularly download_model and load_from_checkpoint.

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Supply Chain: Unpinned Dependencies, External Script Fetching, Obfuscated Code
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• Rogue Agent: Self-Modification, Session Persistence
• MCP Least Privilege: Underdeclared Capability, Wildcard Permission, Missing Permission Declaration

Findings (28)

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 97% confidence

Finding

The manifest markets the skill as an academic-writing auditor, while the body describes a medical clinical-note evaluation tool that downloads and runs large external ML models. This kind of description-behavior mismatch is dangerous because it can cause users or policy systems to approve the skill under a lower-risk classification while it actually processes sensitive medical text and may trigger network/model-download behavior.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

High

• Category: Not specified by scanner
• Confidence: 97% confidence

Finding

The manifest presents a generic semantic auditor for academic writing, but the implementation narrative is specialized medical semantic scoring. Mislabeling specialized medical processing as generic writing assistance obscures privacy, compliance, and validation requirements, making unsafe adoption more likely.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

High

• Category: Not specified by scanner
• Confidence: 98% confidence

Finding

The top-level framing says the skill is for academic writing workflows, but the substantive function is evaluation of clinical notes against expert gold standards. This contradiction can mislead reviewers and users about both the data sensitivity and operational behavior, increasing the chance of inappropriate deployment in contexts with weaker safeguards.

Content

No source excerpt is available for this finding.

Ae1

High

• Category: analysis-evasion
• Confidence: 100% confidence

Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 31)May include surrounding context.

md

- `bert_score`: `unspecified`. Declared in `requirements.txt`.

Ae1

High

• Category: analysis-evasion
• Confidence: 100% confidence

Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 32)May include surrounding context.

md

- `bert_score`: `unspecified`. Declared in `requirements.txt`.

Ae1

High

• Category: analysis-evasion
• Confidence: 100% confidence

Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 33)May include surrounding context.

md

- `bert_score`: `unspecified`. Declared in `requirements.txt`.

Ae1

High

• Category: analysis-evasion
• Confidence: 100% confidence

Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 34)May include surrounding context.

md

- `bert_score`: `unspecified`. Declared in `requirements.txt`.

Ae1

High

• Category: analysis-evasion
• Confidence: 100% confidence

Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 35)May include surrounding context.

md

- `bert_score`: `unspecified`. Declared in `requirements.txt`.

Ae1

High

• Category: analysis-evasion
• Confidence: 100% confidence

Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 36)May include surrounding context.

md

- `bert_score`: `unspecified`. Declared in `requirements.txt`.

Undeclared Tool Scope

Medium

• Category: MCP Least Privilege
• Confidence: 93% confidence

Finding

The skill documents executable behavior and implies file input/output via scripts and result generation, but it does not declare any explicit tool scope such as permissions or allowed-tools. That mismatch weakens reviewability and can lead operators to grant broader file access than intended, especially because the workflow includes reading batch files and writing result artifacts.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

The skill encourages evaluating and batch-processing clinical notes, but it does not warn about patient privacy, de-identification, retention, or transmission risks. Because the workflow also references external ML dependencies and likely model downloads, sensitive medical text could be handled in ways users do not anticipate.

Content

No source excerpt is available for this finding.

Session Persistence

Medium

• Category: Rogue Agent
• Confidence: 60% confidence

Finding

Skill establishes unauthorized persistence across sessions via cron jobs, startup scripts, or state files. Session persistence allows an attacker to maintain access beyond the current interaction.

Content

Scanner excerpt · SKILL.md (reported line 112)May include surrounding context.

text

# Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # Linux/Mac

Natural-Language Policy Violations

Medium

• Category: Not specified by scanner
• Confidence: 88% confidence

Finding

The configuration sets lang: "zh" and the examples also use lang="zh", while the skill description does not state that it is intentionally limited to Chinese-only workflows. This can violate language/locale policy because it defaults users into a specific language without explicit opt-in or a documented rationale.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium

• Category: Not specified by scanner
• Confidence: 97% confidence

Finding

The manifest says this skill is for academic writing workflows with structured execution and clear output boundaries, but the module docstring and CLI help repeatedly frame it as a semantic evaluator for medical records. That is a meaningful purpose-level mismatch rather than an implementation detail, because the code’s inputs, examples, and evaluation framing target a different domain and use case than the manifest claims.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium

• Category: Not specified by scanner
• Confidence: 92% confidence

Finding

The code downloads a COMET model at runtime, which violates the claimed bounded/offline-like auditing behavior and creates a supply-chain and data-egress risk. In security-sensitive environments, implicit network access can undermine containment guarantees and may fetch unpinned remote artifacts into the execution environment.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium

• Category: Not specified by scanner
• Confidence: 98% confidence

Finding

The comment states that gold is used as reference and AI text as the translation, but the actual payload sets both "src" and "mt" to sources[0]. This is an active contradiction between documentation and behavior, and it materially affects the meaning of the computed COMET score.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

The tool writes full evaluation output, including raw AI-generated and gold-standard medical-record text, to an output file without redaction or an explicit sensitivity control. If used on real medical data, this can expose protected health information through insecure file storage, accidental sharing, backups, or downstream processing.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 96% confidence

Finding

Printing full results to stdout can leak sensitive medical-record content into terminal scrollback, shell history capture, job logs, notebook outputs, CI/CD logs, or centralized logging systems. In the medical-record context, this is especially dangerous because the exposed text may contain PHI and clinical details far beyond what is necessary for routine evaluation feedback.

Content

No source excerpt is available for this finding.

Scope Creep

Low

• Category: Excessive Agency
• Confidence: 70% confidence

Finding

Skill's behavior or capabilities extend beyond its stated purpose. Scope creep allows an agent to perform actions unrelated to its documented functionality, increasing the attack surface.

Content

Scanner excerpt · SKILL.md (reported line 92)May include surrounding context.

md

## Overview

Semantic Consistency Auditor is a medical AI evaluation tool used to assess the semantic consistency between AI-generated clinical notes and expert-written gold standards from a semantic level. This tool is not limited to traditional string matching or bag-of-words models, but uses deep learning models to understand semantic entailment relationships, capable of identifying expressions with different wording but similar meaning.

## Algorithms

Unpinned Dependencies

Low

• Category: Supply Chain
• Confidence: 97% confidence

Finding

The dependency list leaves bert_score unpinned, so installations may resolve to different versions over time, including releases with breaking changes or newly introduced vulnerabilities. This weakens build reproducibility and software supply-chain integrity, even though the file alone does not prove active compromise.

Content

Scanner excerpt · requirements.txt (reported line 1)May include surrounding context.

text

bert_score
comet
dataclasses
numpy

Unpinned Dependencies

Low

• Category: Supply Chain
• Confidence: 97% confidence

Finding

The comet package is unpinned, which allows dependency resolution to drift across environments and time. That increases exposure to malicious, vulnerable, or incompatible upstream releases and makes the skill harder to audit reliably.

Content

Scanner excerpt · requirements.txt (reported line 2)May include surrounding context.

text

bert_score
comet
dataclasses
numpy
torch

Unpinned Dependencies

Low

• Category: Supply Chain
• Confidence: 95% confidence

Finding

The dataclasses entry is unpinned, so the installed package version is not controlled. Unpinned dependencies create reproducibility and supply-chain risk, though this specific entry may also be unnecessary on modern Python versions where dataclasses is part of the standard library.

Content

Scanner excerpt · requirements.txt (reported line 3)May include surrounding context.

text

bert_score
comet
dataclasses
numpy
torch
yaml

Unpinned Dependencies

Low

• Category: Supply Chain
• Confidence: 99% confidence

Finding

numpy is unpinned, which is especially risky because it is widely used and has had multiple advisories across versions. Without version pinning, consumers cannot determine whether installs are affected by known flaws or whether future installs will unexpectedly pull a vulnerable release.

Content

Scanner excerpt · requirements.txt (reported line 4)May include surrounding context.

text

bert_score
comet
dataclasses
numpy
torch
yaml

Unverifiable Dependency: numpy has 16 known advisory(ies) (CVE-2014-1859 (Numpy arbitrary file write via symlink attack); CVE-2021-41495 (NumPy NULL Pointer Dereference); CVE-2021-33430 (NumPy Buffer Overflow (Disputed)) +13 more), but the manifest does not pin a version, so it is unknown whether the installed release is affected

Low

• Category: Supply Chain
• Confidence: 98% confidence

Finding

numpy has known advisories across some versions, but the manifest does not specify which version will be installed. Because the dependency is unpinned, it is impossible to verify whether deployments are safe, and future installs could resolve to a vulnerable version depending on repository state and transitive constraints.

Content

No source excerpt is available for this finding.

Unpinned Dependencies

Low

• Category: Supply Chain
• Confidence: 99% confidence

Finding

torch is unpinned, creating supply-chain and reproducibility risk for a large, security-relevant package with a history of serious vulnerabilities. Different installations may obtain materially different code, complicating auditability and potentially exposing users to vulnerable releases.

Content

Scanner excerpt · requirements.txt (reported line 5)May include surrounding context.

text

comet
dataclasses
numpy
torch
yaml

Static analysis

No suspicious patterns detected.