Security audit

research-harness

Security checks for vulnerabilities and agentic risk

Overview

This is a documentation-only research workflow skill with proportionate guidance and no hidden code, install hooks, persistence, or credential handling.

Install only if you want an agent to help structure an AI research experiment. Review generated prompts before printing or sharing them, and keep the human-review gates for claims, rubrics, schemas, and baselines in place.

A.I.G

Vulnerability Patterns

Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs

SkillSpector

Vulnerability Patterns

Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
System Prompt LeakageDirect Leakage, Indirect Extraction, Tool-Based Exfiltration
Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands

Findings (9)

Direct Prompt Extraction

High

Category: System Prompt Leakage
Confidence: 85% confidence
Finding: The instruction to 'print prompt' in a dry run can expose full experiment prompts, including embedded system instructions, proprietary evaluation rubrics, hidden test content, API placeholders, or sensitive context assembled from workspace files. In a research harness, prompt inspection is operationally useful, but without redaction rules it creates a direct prompt-disclosure surface that could leak protected artifacts or contaminate evaluations.
Content: Scanner excerpt · SKILL.md (reported line 119)May include surrounding context.
md
**Goal**: Run experiments, score, compute statistics, analyze errors. Preflight gate: local validators must pass. Then: 1. **Dry run** — print prompt, no API call 2. **Smoke run** — 1 task × 2 groups, verify output parsing 3. **Wave 1** — small set × all groups, minimum viable data 4. **Scoring**: Track A (rule-based) + Track B (semantic) cross-validation. Details in `references/scoring-statistics.md`.

Direct Prompt Extraction

High

Category: System Prompt Leakage
Confidence: 85% confidence
Finding: Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.
Content: Scanner excerpt · references/experiment-design.md (reported line 90)May include surrounding context.
md
## Execution Sequence 1. **Dry run**: print prompt, no API call — verify prompt construction 2. **Smoke run**: 1 task × 2 groups — verify output parsing and schema compliance 3. **Wave 1**: small task set × all groups — minimum viable data for signal detection 4. **Wave 2**: expanded task set — coverage and generalization

Description-Behavior Mismatch

Medium

Category: Not specified by scanner
Confidence: 80% confidence
Finding: L16 says the skill is a set of guardrails 'enforced by repository structure and validation rules,' which implies operational mechanisms beyond pure guidance. Later, L35 explicitly states the skill does not contain automation scripts and only teaches agents how to think. Those claims together suggest a description-level mismatch about whether the skill is purely advisory or also includes executable/enforcing components.
Content: No source excerpt is available for this finding.

Intent-Code Divergence

Medium

Category: Not specified by scanner
Confidence: 86% confidence
Finding: At L16, the README says the skill provides guardrails 'enforced' by repository structure and validation rules, implying active enforcement. At L35, it says the skill contains no automation scripts and teaches how to think rather than what commands to run. This is not merely incomplete documentation; it is an active contradiction about whether the skill has enforcement logic versus being guidance-only.
Content: No source excerpt is available for this finding.

Autonomous Decision Making

Medium

Category: Excessive Agency
Confidence: 75% confidence
Finding: Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.
Content: Scanner excerpt · SKILL.md (reported line 133)May include surrounding context.
md
- **Claim-safe memo**: After any closed loop that produces strong signals, write a memo separating: supported findings, positive signals, not-yet-supported claims, and required next evidence. This prevents overclaiming and directly drives the next phase. Never jump to expansion without calibrating the claim boundary first. - **Alignment doc**: ~1 page with state, entry files, new surfaces, preflight commands, protected surfaces. Never pass chat history. - **Upstream proposals**: Any insight affecting direction goes to `sync/upstream_proposals/` first. Template in `references/agent-collaboration.md`. - **Writing markers**: `[REF-MISSING]`, `[CRITICAL-CHECK]`, `[TODO]`. Never use AI numbers without verification. ---

Natural-Language Policy Violations

Low

Category: Not specified by scanner
Confidence: 77% confidence
Finding: This README is written entirely in Chinese for its main instructions and description, while the policy requires flagging language or locale constraints that do not provide user opt-in or choice. Although L05 links to an English version, this file itself does not state that users may choose their preferred language, so the content effectively assumes a fixed language for readers of this document.
Content: No source excerpt is available for this finding.

Vague Triggers

Low

Category: Not specified by scanner
Confidence: 77% confidence
Finding: The skill lists several situations and then says to trigger on phrases 'anything like' a mixed set of examples, but it does not define where the trigger scope stops or provide negative examples. Terms such as 'experiment framework' and '评分体系' can arise in ordinary discussion about research or evaluation and may cause over-broad activation in contexts not intended for this specific skill.
Content: No source excerpt is available for this finding.

Natural-Language Policy Violations

Low

Category: Not specified by scanner
Confidence: 90% confidence
Finding: The entire skill description is written only in Chinese and does not indicate that other languages are supported or that Chinese is required for a justified region-specific use case. Under the policy, forcing a specific language without user opt-in is a natural-language locale violation.
Content: No source excerpt is available for this finding.

Natural-Language Policy Violations

Low

Category: Not specified by scanner
Confidence: 81% confidence
Finding: Line L121 singles out "Chinese quotation marks" as a special case to watch for, which introduces a language-specific handling rule in natural-language guidance. The document does not explain why this locale-specific constraint is necessary or offer any broader multilingual framing, so it can be read as an unjustified language/locale policy preference.
Content: No source excerpt is available for this finding.