ClawHub
Back to skill

Security audit

long-run-harness

Security checks for vulnerabilities and agentic risk

Overview

The skill is a coherent multi-agent harness builder, but it gives autonomous agents broad file, shell, environment, and git authority without enough enforceable containment.

Install only if you intend to build and run an autonomous local coding harness. Use it in a disposable clone or container, avoid production credentials in the environment, pin dependencies, disable deepcode unless you trust the executable path, and disable or strictly path-scope git commits before using it on an existing repository.

A.I.G

Vulnerability Patterns
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
Findings (5)

T05 · Unauthorized Access and Privilege Escalation

Error
Location
instructions/agent-patterns.md:222
Finding

Agent file and shell tools operate without enforceable workspace boundaries

Content
View full analysis
Remediation
View remediation

T09 · Insecure Skill Coding Practices

Error
Location
instructions/iteration-loop.md:301
Finding

Generic Git checkpoint helper stages unrelated repository content

Content
View full analysis
None: logger = log.get() try: subprocess.run( ["git", "add", "-A"], cwd=project_dir, check=True, capture_output=True, ) result = subprocess.run( ["git", "diff", "--cached", "--quiet"], cwd=project_dir, ) if result.returncode == 0: return # nothing staged — skip subprocess.run( ["git", "commit", "-m", message], cwd=project_dir, check=True, capture_output=True, ) logger.info(f" [Git] Committed: {message}") except subprocess.CalledProcessError as e: logger.info(f" [Git] Commit skipped ({e})") ``` The helper is called after a successful sprint at `instructions/iteration-loop.md:231`: ```python git_commit(project_dir, f"feat: sprint {contract.sprint_number} complete") return result, handoff ``` ### Technical Analysis `git add -A` stages every tracked modification, deletion, and untracked file not excluded by Git ignore rules. The helper does not inspect `cfg.workspace.git.strategy`, does not use the write allowlist, and does not exclude protected paths. Although separate documentation recommends disabled or path-scoped commits for existing repositories, the supplied generic orchestration helper does not enforce that policy. Documentation is not a substitute for checking the configured strategy at the commit boundary. ### Attack Path 1. A user starts the harness in an existing repository that contains unrelated uncommitted work. 2. The generator changes files for a sprint, or an attacker causes an untracked sensitive file to be created in the repository. 3. The sprint reaches its successful completion branch. 4. `gi ...[truncated 759 chars]
Remediation
View remediation

T05 · Unauthorized Access and Privilege Escalation

Error
Location
instructions/agent-patterns.md:700
Finding

Configurable Deepcode executable inherits the complete process environment

Content
View full analysis
Remediation
View remediation

T09 · Insecure Skill Coding Practices

Warning
Location
instructions/agent-patterns.md:634
Finding

Evaluator verdict logic can pass incomplete contract-result sets

Content
View full analysis
EvalResult: scores = g.get("rubric_scores", {}) avg = sum(scores.values()) / len(scores) if scores else 0.0 all_pass = all( r["status"] == "pass" for r in g.get("contract_results", []) ) pt = cfg.verdict.pass_threshold if cfg else 3.0 cpt = cfg.verdict.conditional_pass_threshold if cfg else 2.0 if not all_pass: verdict = "fail" elif avg >= pt: verdict = "pass" elif avg >= cpt: verdict = "conditional_pass" else: verdict = "fail" contract_map = { sc.id: sc.description for sc in contract.success_criteria } criteria = [ SuccessCriterion( id=r["id"], description=contract_map.get(r["id"], ""), status=r["status"], evidence=r.get("evidence", ""), ) for r in g.get("contract_results", []) ] return EvalResult( verdict=verdict, rubric_average=round(avg, 2), contract_results=criteria, feedback=g.get("feedback", ""), ) ``` ### Technical Analysis The code determines `all_pass` from only the results supplied by the evaluator. It does not verify that every success criterion in `contract.success_criteria` appears exactly once. Python’s `all()` returns `True` for an empty iterable. Although an entirely empty result set would still receive an average of zero, an empty or partial `contract_results` array paired with sufficiently high rubric scores can receive `pass` or `conditional_pass`. The implementation also does not reject: - Missing contract criterion IDs. - Duplicate result IDs. - Unknown result IDs. - Invalid or non ...[truncated 1228 chars]
Remediation
View remediation

T08 · Insecure Dependencies

Warning
Location
SKILL.md:24
Finding

Third-party Python dependencies are installed without version or integrity pinning

Content
View full analysis
Remediation
View remediation

SkillSpector

Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Output HandlingUnvalidated Output Injection, Cross-Context Output, Unbounded Output
Findings (19)

Unvalidated Output Injection

High
Category
Output Handling
Confidence
91% confidence
Finding

The harness treats Codex stdout as trusted structured input and feeds it into _parse_codex_eval() to influence pass/fail decisions without strong validation of schema completeness, field bounds, or correspondence to expected criteria. A compromised or manipulated agent output can forge evaluation results, causing the orchestrator to accept incomplete or unsafe implementations and corrupt downstream decision-making.

Content

Scanner excerpt · instructions/agent-patterns.md (reported line 539)May include surrounding context.

md
f"```"
    )
    output = _run_codex(prompt, agent_cfg, cwd=str(app_dir) if app_dir else None, timeout=600)
    return _parse_codex_eval(output, contract, cfg)

def _parse_codex_eval(output: str, contract: SprintContract, cfg: HarnessConfig | None) -> EvalResult:
    """Extract the JSON block from Codex output and build an EvalResult."""

Unvalidated Output Injection

High
Category
Output Handling
Confidence
94% confidence
Finding

_parse_codex_eval() uses regex extraction over arbitrary model output and then deserializes the first matching JSON-like block, which is brittle and susceptible to malformed, partial, or adversarially crafted content. Because the parsed object directly drives grading logic, an attacker controlling prompt inputs or agent behavior could inject fabricated contract results and rubric scores that mislead the harness.

Content

Scanner excerpt · instructions/agent-patterns.md (reported line 541)May include surrounding context.

md
output = _run_codex(prompt, agent_cfg, cwd=str(app_dir) if app_dir else None, timeout=600)
    return _parse_codex_eval(output, contract, cfg)

def _parse_codex_eval(output: str, contract: SprintContract, cfg: HarnessConfig | None) -> EvalResult:
    """Extract the JSON block from Codex output and build an EvalResult."""
    match = re.search(r"```json\s*(\{.*?\})\s*```", output, re.DOTALL)
    if not match:

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · instructions/config.md (reported line 148)May include surrounding context.

md
keep_last_evidence_per_sprint: int = 3
    write_allowlist: list[str] = field(default_factory=lambda: ["src/**", "package.json", "package-lock.json"])
    protected_paths: list[str] = field(default_factory=lambda: [
        ".git/**", ".env", ".env.*", "node_modules/**", ".next/**",
        "dist/**", "build/**", "harness-state/**", "harness-logs/**",
    ])
    git: WorkspaceGitConfig = field(default_factory=WorkspaceGitConfig)

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · instructions/config.md (reported line 168)May include surrounding context.

md
keep_last_evidence_per_sprint: int = 3
    write_allowlist: list[str] = field(default_factory=lambda: ["src/**", "package.json", "package-lock.json"])
    protected_paths: list[str] = field(default_factory=lambda: [
        ".git/**", ".env", ".env.*", "node_modules/**", ".next/**",
        "dist/**", "build/**", "harness-state/**", "harness-logs/**",
    ])
    git: WorkspaceGitConfig = field(default_factory=WorkspaceGitConfig)

Unvalidated Output Injection

High
Category
Output Handling
Confidence
80% confidence
Finding

Model output is used without validation or sanitization. Unvalidated output injected into downstream contexts (SQL, shell, HTML) enables injection attacks and arbitrary code execution.

Content

Scanner excerpt · instructions/evaluator-evidence.md (reported line 58)May include surrounding context.

harness-state/evidence/sprint-N/screenshots/

text

Store axe/Lighthouse raw output under:

```text
harness-state/evidence/sprint-N/axe/

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · SKILL.md (reported line 142)May include surrounding context.

yaml
protected_paths:
  - .git/**
  - .env
  - .env.*
  - node_modules/**
  - .next/**

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · instructions/config.md (reported line 252)May include surrounding context.

yaml
protected_paths:
  - .git/**
  - .env
  - .env.*
  - node_modules/**
  - .next/**

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · instructions/config.md (reported line 323)May include surrounding context.

yaml
protected_paths:
  - .git/**
  - .env
  - .env.*
  - node_modules/**
  - .next/**

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · instructions/git-safety.md (reported line 73)May include surrounding context.

yaml
protected_paths:
  - .git/**
  - .env
  - .env.*
  - node_modules/**
  - .next/**

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
89% confidence
Finding

Lines L013-L015 state that the skill 'produces a runnable harness' and that 'You are writing the harness, not running inside it.' However, later sections instruct runtime behaviors such as prompting users interactively, running generation/evaluation loops, collecting artifacts, and committing to git. That is an active contradiction in intent documentation, not merely omitted detail.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
79% confidence
Finding

The instruction to confirm /Users/xzhao/.local/bin/deepcode and the repeated reference to that exact path later in the file prescribe a specific user/home-directory layout. This is a natural-language policy concern because it forces a particular local environment assumption rather than offering a configurable or user-chosen path.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
86% confidence
Finding

The setup example includes a realistic secret format (export ANTHROPIC_API_KEY=sk-...) without adjacent guidance on secure credential handling. In practice, users may paste real keys into shell history, checked-in scripts, CI logs, or shared transcripts, increasing the chance of credential exposure.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
81% confidence
Finding

This line tells the user to use cli_path=/Users/xzhao/.local/bin/deepcode, which bakes in a user-specific filesystem location. The language does not present this as optional or environment-specific guidance, so it effectively enforces a particular setup without opt-in.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
90% confidence
Finding

This markdown file documents that the generator's Codex path runs with --approval-mode full-auto and manages file reads/writes and shell commands internally. Although the behavior is described, there is no explicit caution or user-facing warning about the safety impact of granting autonomous write and command execution capabilities, which could affect user data or system integrity.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
89% confidence
Finding

The manifest frames this skill as building a Planner→Generator→Evaluator harness or long-running orchestrator, but these lines explicitly describe the generator backend as allowing Codex to perform file reads/writes and shell commands autonomously. While code generation is in scope for such a harness, granting an agent broad shell-command capability is a materially stronger capability than the manifest describes and is not justified in the manifest text itself.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
87% confidence
Finding

The skill is described as a multi-agent harness/orchestrator, and evaluator behavior would reasonably include assessing outputs, but these lines explicitly state that Codex may run shell commands during evaluation. Shell execution is a broad operational capability not declared in the manifest and is not an obvious requirement of merely orchestrating or evaluating agent work.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
88% confidence
Finding

This guidance explicitly instructs the harness to run repeated file-writing helpers and automatic git add/git commit operations, but it does not require an explicit user opt-in or warning before modifying project files and repository history. In a long-running autonomous agent context, silent persistence and history mutation increase the risk of unintended changes, committing sensitive files, or polluting an existing repository in ways the user did not authorize.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
80% confidence
Finding

The evaluator markdown explains that the Codex path can read files and run shell commands such as curl against the app, which may involve network activity and system interaction. The behavior is operationally described but not accompanied by a clear warning about possible privacy or integrity implications.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
84% confidence
Finding

This markdown file instructs implementers to run git diff via subprocess.run and to save multiple state artifacts to disk, but it does not include any warning that the skill reads repository state and writes persistent files under the project directory. Because SQP-2 applies to markdown files when descriptions omit warnings about behaviors affecting user data or system integrity, this is a qualifying missing-warning issue.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.