ClawHub
Back to skill

Security audit

AI Output Verification Trust Tiers

Security checks for vulnerabilities and agentic risk

Overview

This skill helps users set appropriate review rules for AI output and explicitly reserves irreversible actions for humans.

Use this as a planning framework and keep named human reviewers responsible for consequential outputs. Its tier system is the author's synthesis, not a compliance standard.

A.I.G

Vulnerability Patterns
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs

SkillSpector

Vulnerability Patterns
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
Findings (3)

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
75% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · SKILL.md (reported line 12)May include surrounding context.

md
## Overview

Teams usually make one of two mistakes with AI output. They trust all of it — and fabricated figures, citations or terms reach a client — or they review all of it, and the AI saves no time. Both are wrong, and the second quietly becomes the first: reviewers who check everything start approving without checking.

The core mechanism: **tier every type of output by error cost × verifiability → give each tier a verification method and an evidence requirement → set escalation rules → move tiers up or down based on spot-check results.** Trust is set by what an error costs, not by how good the model usually is.

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
75% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · references/sources.md (reported line 13)May include surrounding context.

md
## Human reviewers

- **Parasuraman, R. & Manzey, D. H.** (2010). "Complacency and Bias in Human Use of Automation: An Attentional Integration." *Human Factors*, 52(3), 381–410. **Primary source for automation complacency and automation bias — why reviewers of reliable-seeming systems start approving without checking** (Applying It Well #4; the "review everything" rationalization). https://doi.org/10.1177/0018720810376055

## Worked example

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
75% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · examples/mata-v-avianca-fabricated-citations-2023.md (reported line 5)May include surrounding context.

md
> *Example for the [ai-output-verification-trust-tiers](../SKILL.md) skill.*

A worked example of a high-cost, easily verifiable AI output released with no verification at all — documented in Judge P. Kevin Castel's *Opinion and Order on Sanctions*, U.S. District Court for the Southern District of New York, June 22, 2023 (22-cv-1461 (PKC), ECF 54). Every fact below is from the court's findings. The case is useful because nearly every fake move in this skill's rationalization table appears in the record.

**The setup.** Roberto Mata sued Avianca over a knee injury from a serving cart on a flight. Avianca moved to dismiss the claims as time-barred under the Montreal Convention. On March 1, 2023, plaintiff's counsel filed an "Affirmation in Opposition" that "cited and quoted from purported judicial decisions that were said to be published in the Federal Reporter, the Federal Supplement and Westlaw." The research had been done by an attorney who turned to ChatGPT because the firm's resear

Static analysis

No suspicious patterns detected.