Structured Multi-Agent Deliberation

• Outcome: Pass
• Latest audit: Sep 16, 2026 · 7:27 AM UTC
• Version: 0.1.0

Overview

This skill is a disclosed structured-deliberation aid with local reference scripts; it has implementation quality issues but no artifact-backed malicious behavior.

Reasonable to install as a methodology/reference skill, but treat the Python scripts as draft utilities. Before using it for consequential decisions, tighten JSONL parse failures, restrict evidence references to approved project paths and real anchors, and review the stopping-detector math.

A.I.G

Vulnerability Patterns

• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally

Findings (3)

T09 · Insecure Skill Coding Practices

Warning

• Location: scripts/claims-validator.py:35

Finding

Malformed JSONL Records Are Silently Discarded

Content

View full analysis

Vulnerability Details

File Locations:

• scripts/claims-validator.py:35-42
• scripts/round-controller.py:66-73
• scripts/stopping-detector.py:34-41

Vulnerability Type: Fail-open parsing of security-relevant state
Risk Level: Medium

Vulnerable Code

From scripts/claims-validator.py:

python

with path.open('r', encoding='utf-8') as f:
    for line_num, line in enumerate(f, 1):
        line = line.strip()
        if not line:
            continue
        try:
            entries.append(json.loads(line))
        except json.JSONDecodeError as e:
            print(f"ERROR {path}:{line_num} — invalid JSON: {e}", file=sys.stderr)
return entries

From scripts/round-controller.py:

python

with path.open('r', encoding='utf-8') as f:
    for line in f:
        line = line.strip()
        if line:
            try:
                entries.append(json.loads(line))
            except json.JSONDecodeError:
                pass
return entries

From scripts/stopping-detector.py:

python

with path.open('r', encoding='utf-8') as f:
    for line in f:
        line = line.strip()
        if line:
            try:
                entries.append(json.loads(line))
            except json.JSONDecodeError:
                pass
return entries

Technical Analysis

The three JSONL loaders continue after encountering malformed records. The controller and stopping detector discard malformed records without even reporting them. Although the validator prints an error, it does not append the parse failure to the list returned by validate(). The validator can therefore exit successfully after processing an incomplete representation of the state.

This is a fail-open integrity defect because claims, verifications, decisions, and lead history are security-relevant inputs to validation and stopping decisions. A malformed or deliberately corrupted record is treated as absent rather than causing the operation to fail.

The append-only design al ...[truncated 1514 chars]

Remediation

View remediation

Remediation Suggestions

1. Make all JSON parsing errors fatal for validation and decision-making.
2. Return both parsed entries and parse errors from a shared strict loader.
3. Add every malformed line to the validator's error list and force a nonzero exit.
4. Include the file path and line number in every parsing error.
5. Do not run stopping calculations when any required state file is malformed.
6. Consider safely recovering only a demonstrably truncated final line, and require explicit operator confirmation before ignoring it.
7. Add tests covering malformed records at the beginning, middle, and end of every JSONL file.

A strict loader could follow this pattern:

python

def load_jsonl_strict(path: Path) -> List[Dict]:
    entries = []
    errors = []

    if not path.exists():
        return entries

    with path.open("r", encoding="utf-8") as stream:
        for line_number, line in enumerate(stream, 1):
            if not line.strip():
                continue
            try:
                value = json.loads(line)
            except json.JSONDecodeError as exc:
                errors.append(f"{path}:{line_number}: {exc}")
                continue

            if not isinstance(value, dict):
                errors.append(f"{path}:{line_number}: expected a JSON object")
                continue

            entries.append(value)

    if errors:
        raise ValueError("Invalid JSONL state:\n" + "\n".join(errors))

    return entries

T09 · Insecure Skill Coding Practices

Warning

• Location: scripts/claims-validator.py:76

Finding

Evidence References Permit Repository Escape and Fabricated Anchors

Content

View full analysis

Vulnerability Details

File Location: scripts/claims-validator.py:76-82
Vulnerability Type: Path traversal and incomplete evidence validation
Risk Level: Medium

Vulnerable Code

python

for ref in refs:
    # ref format: "artifacts/round_N/role.md#section" or "state/decisions.jsonl#L5"
    file_part = ref.split("#")[0]
    full_path = repo_root / file_part
    if not full_path.exists():
        errors.append(f"verification {vid} evidence_ref points to missing file: {file_part}")

Technical Analysis

The validator treats the path portion of an evidence_ref as trusted input. It joins that value to repo_root but does not resolve and verify that the resulting path remains inside an approved project directory.

A reference containing ../ can therefore escape the repository. In Python path composition, an absolute file_part can also replace the intended base path. The code only checks exists(), so directories and arbitrary existing host paths can satisfy the check.

The documented format includes a section after #, but the validator discards that section and never verifies that the named anchor or line exists. A reference such as artifacts/round_1/action.md#invented-section passes whenever the file exists, even if the purported evidence does not.

Attack Path

1. An attacker or untrusted verification generator creates a verification with at least two crafted evidence_refs.
2. The references point to existing paths using traversal, absolute paths, or legitimate artifact files with nonexistent anchors.
3. The validator removes everything after # and checks only path existence.
4. Existing unrelated files satisfy the validation requirement.
5. The unsupported verification is accepted and may confirm or refute a claim.
6. Subsequent decisions can cite the verification, giving fabricated evidence the appearance of validated provenance.

For example, a crafted reference could use a form equivalent to:

text

../../some
...[truncated 641 chars]

Remediation

View remediation

Remediation Suggestions

1. Reject absolute paths in evidence references.
2. Resolve the candidate path with Path.resolve(strict=True).
3. Enforce containment under explicitly approved roots, such as artifacts/ and selected files under state/.
4. Require the target to be a regular file rather than merely an existing path.
5. Define a strict grammar for evidence references.
6. Require an anchor or line range and verify that it exists in the target file.
7. Reject symlinks that resolve outside approved roots.
8. Avoid exposing detailed existence results for paths outside the approved namespace.

Example containment logic:

python

allowed_root = (repo_root / "artifacts").resolve()
candidate = (repo_root / file_part).resolve(strict=True)

if candidate != allowed_root and allowed_root not in candidate.parents:
    errors.append(
        f"verification {vid} evidence_ref escapes the allowed artifact root"
    )
elif not candidate.is_file():
    errors.append(
        f"verification {vid} evidence_ref does not identify a regular file"
    )

For references to state/decisions.jsonl, use a separate explicit allowlist rather than broadening access to the entire repository. Parse the fragment after # and validate the requested heading or line number against the target file.

T09 · Insecure Skill Coding Practices

Warning

• Location: scripts/stopping-detector.py:75

Finding

Disagreement Trend Calculation Omits Zero-Disagreement Rounds

Content

View full analysis

Vulnerability Details

File Location: scripts/stopping-detector.py:75-101
Related Specification: references/stopping-criteria.md:27-34
Vulnerability Type: Incorrect security-relevant convergence calculation
Risk Level: Medium

Vulnerable Code

python

def signal_2_disagreement_slope(verifications: List[Dict]) -> Tuple[bool, str]:
    """OLS slope of disagreement_count over rounds; ≥0 healthy, ≥-0.2 stable."""
    if not verifications:
        return False, "no verifications"

    disagreement_per_round = Counter()
    for v in verifications:
        if v.get("result") in ("BROKEN", "UNCLEAR"):
            disagreement_per_round[v["round"]] += 1

    rounds = sorted(disagreement_per_round.keys())
    if len(rounds) < 3:
        return False, f"only {len(rounds)} rounds with disagreement data"

    xs = rounds
    ys = [disagreement_per_round[r] for r in rounds]
    n = len(xs)
    mean_x = sum(xs) / n
    mean_y = sum(ys) / n
    num = sum((x - mean_x) * (y - mean_y) for x, y in zip(xs, ys))
    den = sum((x - mean_x) ** 2 for x in xs)
    slope = num / den if den else 0

    if slope >= 0:
        return True, f"slope = {slope:.3f} ≥ 0 (anti-sycophantic)"
    if slope >= -0.2:
        return True, f"slope = {slope:.3f} ∈ [-0.2, 0) (stable)"
    return False, f"slope = {slope:.3f} < -0.2 (sycophancy risk)"

Technical Analysis

The disagreement_per_round counter receives entries only for rounds containing a BROKEN or UNCLEAR verification. Consequently, rounds with zero disagreements are entirely omitted from rounds, rather than represented with a count of zero.

This creates survivorship bias in the regression. If early rounds contain disagreements and later rounds contain only ROBUST results, the later rounds disappear from the calculation. That is precisely the pattern the detector is intended to identify as a potential sycophancy cascade.

The implementation also treats a slope between -0.2 and z ...[truncated 1705 chars]

Remediation

View remediation

Remediation Suggestions

1. Derive the round set from all verification records, not only disagreement records.
2. Insert a disagreement count of zero for every observed round without BROKEN or UNCLEAR results.
3. Enforce the documented whole-experiment requirement of slope >= 0, or update the specification and justify any tolerance.
4. Separately verify that the last three rounds maintain the required trend.
5. Decide explicitly how NOT_APPLICABLE stress checks affect the denominator and document that behavior.
6. Add tests for late all-ROBUST collapse, missing round numbers, stress rounds, and mixed verdicts.

A corrected basis for the regression could begin as follows:

python

all_rounds = sorted({
    v["round"] for v in verifications
    if isinstance(v.get("round"), int)
})

if len(all_rounds) < 3:
    return False, f"only {len(all_rounds)} observed rounds"

disagreement_per_round = Counter({round_number: 0 for round_number in all_rounds})

for verification in verifications:
    if verification.get("result") in ("BROKEN", "UNCLEAR"):
        disagreement_per_round[verification["round"]] += 1

xs = all_rounds
ys = [disagreement_per_round[round_number] for round_number in all_rounds]

After calculating the whole-run slope, calculate a separate slope or monotonicity condition for the final three observed rounds and require both checks to pass.

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• Behavioral AST: exec() Call, eval() Call, Dynamic Import
• MCP Least Privilege: Underdeclared Capability, Wildcard Permission, Missing Permission Declaration
• MCP Tool Poisoning: Hidden Instructions, Unicode Deception, Parameter Description Injection

Findings (9)

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 99% confidence

Finding

The declared description suggests a framework for running or structuring multi-agent deliberation, including agent roles, protocols, and stopping criteria. The supplied code does not implement any such framework. Instead, it validates data integrity across JSONL artifacts for a deliberation run: checking references, evidence files, claim statuses, supersession links, citations, and required fields. While this is loosely related to deliberation artifacts and verification, it is a narrow validation utility rather than the described multi-agent deliberation framework. Therefore the code’s primary purpose is materially different from the declared purpose.

Content

No source excerpt is available for this finding.

Undeclared Tool Scope

Medium

• Category: MCP Least Privilege
• Confidence: 70% confidence

Finding

Without declared permissions the skill's intent is opaque and cannot be validated.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium

• Category: Not specified by scanner
• Confidence: 91% confidence

Finding

The manifest describes a framework for multi-agent deliberation with role schemas, verification protocols, and stopping criteria. Spawning a separate process via subprocess is not an obvious requirement of that purpose and introduces an execution capability beyond ordinary orchestration and file/state management.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 90% confidence

Finding

The code invokes claims-validator.py via subprocess.run, which is a safety-relevant operation because it executes another program in the local environment. There is no confirmation prompt and no user-facing log or warning before this execution, so users are not explicitly informed that the skill will spawn a subprocess.

Content

No source excerpt is available for this finding.

subprocess module call

Medium

• Category: Dangerous Code Execution
• Confidence: 70% confidence

Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/round-controller.py (reported line 214)May include surrounding context.

python

# Run validator
    import subprocess
    script_path = Path(__file__).parent / "claims-validator.py"
    result = subprocess.run(
        ["python", str(script_path), "--state-dir", str(state_dir)],
        capture_output=True,
        text=True,

Autonomous Decision Making

Medium

• Category: Excessive Agency
• Confidence: 75% confidence

Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · references/stopping-criteria.md (reported line 80)May include surrounding context.

md

def signal_1_claim_refutation_stable(verifications: List[Dict]) -> Tuple[bool, str]:
    """Rolling 3-round avg of status transitions drops below 50% of mid-experiment peak."""
    if not verifications:
        return False, "no verifications"

    rounds = sorted({v["round"] for v in verifications})
    if len(rounds) < 6:

Autonomous Decision Making

Medium

• Category: Excessive Agency
• Confidence: 75% confidence

Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · scripts/claims-validator.py (reported line 95)May include surrounding context.

python

def signal_1_claim_refutation_stable(verifications: List[Dict]) -> Tuple[bool, str]:
    """Rolling 3-round avg of status transitions drops below 50% of mid-experiment peak."""
    if not verifications:
        return False, "no verifications"

    rounds = sorted({v["round"] for v in verifications})
    if len(rounds) < 6:

Autonomous Decision Making

Medium

• Category: Excessive Agency
• Confidence: 75% confidence

Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · scripts/stopping-detector.py (reported line 48)May include surrounding context.

python

def signal_1_claim_refutation_stable(verifications: List[Dict]) -> Tuple[bool, str]:
    """Rolling 3-round avg of status transitions drops below 50% of mid-experiment peak."""
    if not verifications:
        return False, "no verifications"

    rounds = sorted({v["round"] for v in verifications})
    if len(rounds) < 6:

Autonomous Decision Making

Medium

• Category: Excessive Agency
• Confidence: 75% confidence

Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · scripts/stopping-detector.py (reported line 76)May include surrounding context.

python

def signal_1_claim_refutation_stable(verifications: List[Dict]) -> Tuple[bool, str]:
    """Rolling 3-round avg of status transitions drops below 50% of mid-experiment peak."""
    if not verifications:
        return False, "no verifications"

    rounds = sorted({v["round"] for v in verifications})
    if len(rounds) < 6:

Static analysis

No suspicious patterns detected.