ClawHub
Back to skill

Security audit

benchmark-robustness-auditor

Security checks for vulnerabilities and agentic risk

Overview

This skill is a coherent offline benchmark-auditing tool; the flagged shell pattern is confined to its local self-test and does not show hidden or malicious behavior.

Safe to install for offline benchmark-audit use. Treat benchmark and corpus JSONL files as local sensitive inputs, set BENCHSCAN_LEDGER to a controlled path in CI or shared directories, and run scripts/selftest.sh only from the trusted package because it executes hardcoded local shell/python test snippets.

A.I.G

Vulnerability Patterns
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs

SkillSpector

Vulnerability Patterns
  • Output HandlingUnvalidated Output Injection, Cross-Context Output, Unbounded Output
  • Tool MisuseTool Parameter Abuse, Chaining Abuse, Unsafe Defaults
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
Findings (2)

Unvalidated Output Injection

High
Category
Output Handling
Content
# ── report: dirty fixture → COMPROMISED rc4 (findings are ORDER-DEPENDENT) ───
REPCMD="python3 scripts/benchscan.py report --name selftest-bench --benchmark $SBX/bench.jsonl --corpus $SBX/corp.jsonl --cutoff 2024-06-01 --results $SBX/repro.jsonl --runs $SBX/runs.jsonl --curve $SBX/fewshot.jsonl --judgments $SBX/judgments.jsonl --a-preds $SBX/a.jsonl --b-preds $SBX/b.jsonl"
chk "report rc4 COMPROMISED worst>=75 + ledger 0600" "python3 -c 'import json,os,stat,subprocess; r=subprocess.run(\"python3 scripts/benchscan.py report --name selftest-bench --benchmark $SBX/bench.jsonl --corpus $SBX/corp.jsonl --cutoff 2024-06-01 --results $SBX/repro.jsonl --runs $SBX/runs.jsonl --curve $SBX/fewshot.jsonl --judgments $SBX/judgments.jsonl --a-preds $SBX/a.jsonl --b-preds $SBX/b.jsonl\",shell=True,capture_output=True,text=True,cwd=os.getcwd()); d=json.loads(r.stdout); assert r.returncode==4 and d[\"verdict\"]==\"COMPROMISED\" and d[\"worst_score\"]>=75 and len(d[\"report_sha256\"])==64; m=oct(stat.S_IMODE(os.stat(os.environ[\"BENCHSCAN_LEDGER\"]).st_mode)); assert m==\"0o600\", m'"
chk "report findings cite only catalogue ids (anti-hallucination)" "python3 -c 'import json,os,subprocess; cat={k for k in json.loads(subprocess.run([\"python3\",\"scripts/benchscan.py\",\"doctor\"],capture_output=True,text=True).stdout)[\"catalogue\"]}; r=subprocess.run([\"python3\",\"scripts/benchscan.py\",\"report\",\"--name\",\"selftest-bench\",\"--benchmark\",os.environ[\"SBX\"]+\"/bench.jsonl\",\"--corpus\",os.environ[\"SBX\"]+\"/corp.jsonl\",\"--cutoff\",\"2024-06-01\",\"--results\",os.environ[\"SBX\"]+\"/repro.jsonl\",\"--runs\",os.environ[\"SBX\"]+\"/runs.jsonl\",\"--curve\",os.environ[\"SBX\"]+\"/fewshot.jsonl\",\"--judgments\",os.environ[\"SBX\"]+\"/judgments.jsonl\",\"--a-preds\",os.environ[\"SBX\"]+\"/a.jsonl\",\"--b-preds\",os.environ[\"SBX\"]+\"/b.jsonl\"],capture_output=True,text=True); d=json.loads(r.stdout); cats={f[\"cat\"] for f in d[\"findings\"]}; assert {\"C-1\",\"E-1\"} <= cats 
...[truncated 25 chars]
Confidence
91% confidence
Finding
This line invokes subprocess.run with shell=True and interpolates $SBX paths into a shell command string. If SBX or related values were attacker-controlled or unexpectedly contained shell metacharacters, this could lead to command injection during the self-test run; the risk is reduced by mktemp-generated paths but the pattern remains unsafe.

Tool Parameter Abuse

High
Category
Tool Misuse
Content
# ── report: dirty fixture → COMPROMISED rc4 (findings are ORDER-DEPENDENT) ───
REPCMD="python3 scripts/benchscan.py report --name selftest-bench --benchmark $SBX/bench.jsonl --corpus $SBX/corp.jsonl --cutoff 2024-06-01 --results $SBX/repro.jsonl --runs $SBX/runs.jsonl --curve $SBX/fewshot.jsonl --judgments $SBX/judgments.jsonl --a-preds $SBX/a.jsonl --b-preds $SBX/b.jsonl"
chk "report rc4 COMPROMISED worst>=75 + ledger 0600" "python3 -c 'import json,os,stat,subprocess; r=subprocess.run(\"python3 scripts/benchscan.py report --name selftest-bench --benchmark $SBX/bench.jsonl --corpus $SBX/corp.jsonl --cutoff 2024-06-01 --results $SBX/repro.jsonl --runs $SBX/runs.jsonl --curve $SBX/fewshot.jsonl --judgments $SBX/judgments.jsonl --a-preds $SBX/a.jsonl --b-preds $SBX/b.jsonl\",shell=True,capture_output=True,text=True,cwd=os.getcwd()); d=json.loads(r.stdout); assert r.returncode==4 and d[\"verdict\"]==\"COMPROMISED\" and d[\"worst_score\"]>=75 and len(d[\"report_sha256\"])==64; m=oct(stat.S_IMODE(os.stat(os.environ[\"BENCHSCAN_LEDGER\"]).st_mode)); assert m==\"0o600\", m'"
chk "report findings cite only catalogue ids (anti-hallucination)" "python3 -c 'import json,os,subprocess; cat={k for k in json.loads(subprocess.run([\"python3\",\"scripts/benchscan.py\",\"doctor\"],capture_output=True,text=True).stdout)[\"catalogue\"]}; r=subprocess.run([\"python3\",\"scripts/benchscan.py\",\"report\",\"--name\",\"selftest-bench\",\"--benchmark\",os.environ[\"SBX\"]+\"/bench.jsonl\",\"--corpus\",os.environ[\"SBX\"]+\"/corp.jsonl\",\"--cutoff\",\"2024-06-01\",\"--results\",os.environ[\"SBX\"]+\"/repro.jsonl\",\"--runs\",os.environ[\"SBX\"]+\"/runs.jsonl\",\"--curve\",os.environ[\"SBX\"]+\"/fewshot.jsonl\",\"--judgments\",os.environ[\"SBX\"]+\"/judgments.jsonl\",\"--a-preds\",os.environ[\"SBX\"]+\"/a.jsonl\",\"--b-preds\",os.environ[\"SBX\"]+\"/b.jsonl\"],capture_output=True,text=True); d=json.loads(r.stdout); cats={f[\"cat\"] for f in d[\"findings\"]}; assert {\"C-1\",\"E-1\"} <= cats 
...[truncated 25 chars]
Confidence
94% confidence
Finding
This is the strongest real issue in the file: a shell command is built as a string and executed with shell=True while incorporating environment-derived paths. Even though SBX is created via mktemp and the script is a local self-test, the pattern enables parameter/command injection if that assumption changes or if surrounding code later allows attacker influence over the command string.

Static analysis

No suspicious patterns detected.