Back to skill

Security audit

inference-expert-agents

Security checks for vulnerabilities and agentic risk

Overview

The skill is mostly coherent, but needs review because its privacy documentation contradicts real outbound citation checks and its published verification hash does not match the inspected skill file.

Review before installing. Do not rely on the README's offline/privacy claims or its printed SKILL.md hash; treat citecheck.py as an online tool that sends citation IDs, DOIs, and title queries to public registries, and avoid using it with confidential or unpublished research metadata unless that egress is acceptable.

Vulnerability Patterns
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (2)

T09 · Insecure Skill Coding Practices

Warning
Location
tools/citecheck.py:54
Finding
Undisclosed External Transmission of Citation Data<![CDATA[ ## Vulnerability Details **File Location**: `tools/citecheck.py:54-55, 72-80, 89-90, 135-136, 151-152`; contradictory privacy claims at `README.md:117-125` **Vulnerability Type**: Undisclosed network communication and potential sensitive metadata disclosure **Risk Level**: Medium ### Complete Code Snippet ```python ARXIV_API = "https://export.arxiv.org/api/query" OPENALEX_API = "https://api.openalex.org/works" NS = {"a": "http://www.w3.org/2005/Atom", "ar": "http://arxiv.org/schemas/atom"} UA = {"User-Agent": "inference-expert-agents/citecheck 3.0"} TIMEOUT = 45 class Unreachable(Exception): """Network/registry failure — distinct from 'the citation is wrong'.""" def _get(url: str, timeout: int = TIMEOUT) -> bytes: try: req = urllib.request.Request(url, headers=UA) with urllib.request.urlopen(req, timeout=timeout) as r: return r.read() except urllib.error.HTTPError as e: raise Unreachable(f"HTTP {e.code} from {url.split('?')[0]}") from e except (urllib.error.URLError, TimeoutError, OSError) as e: raise Unreachable(f"{type(e).__name__} reaching {url.split('?')[0]}") from e ``` The values transmitted through these requests are constructed as follows: ```python q = urllib.parse.urlencode({"id_list": ",".join(ids), "max_results": 100}) root = ET.fromstring(_get(f"{ARXIV_API}?{q}")) ``` ```python q = urllib.parse.urlencode({"filter": f"doi:{doi}", "per-page": 1}) data = json.loads(_get(f"{OPENALEX_API}?{q}")) ``` ```python q = urllib.parse.urlencode({"filter": f"title.search:{query}", "per-page": 3}) data = json.loads(_get(f"{OPENALEX_API}?{q}")) ``` This behavior conflicts with the following claims in `README.md`: ```text - Network: none. All processing is local. - Data read/sent: only the text you pass to the tools; nothing is transmitted anywhere. ``` ### Technical Analysis `citecheck.py` performs HTTPS requests to `export.arxiv.org` and `api.openalex.org`. Depending on the selec ...[truncated 2434 chars]
Remediation
<![CDATA[ ## Remediation Suggestions 1. Correct the README permissions and privacy sections to state clearly that `citecheck.py` communicates with: - `https://export.arxiv.org` - `https://api.openalex.org` 2. Document exactly what is transmitted for each subcommand: arXiv IDs, DOI values, and title-query text. 3. Require explicit authorization through an option such as `--allow-network`; refuse network requests unless it is supplied. 4. Emit a concise warning before network use, particularly for `scan`, `check`, and `title`. 5. Provide a local-only mode that validates syntax or checks against a caller-supplied registry snapshot without making external requests. 6. Advise users not to submit confidential, unpublished, or personally identifying title queries. 7. Consider using POST requests where supported to reduce query leakage through URL logs, while noting that this does not prevent disclosure to the destination service. 8. Add tests that ensure the network consent gate is enforced and that documentation accurately reflects current behavior. ]]>

T08 · Insecure Dependencies

Warning
Location
README.md:90
Finding
Installation Command Executes an Unverified Third-Party npm CLI<![CDATA[ ## Vulnerability Details **File Location**: `README.md:90-94` **Vulnerability Type**: Supply-chain exposure through installation-time third-party code execution **Risk Level**: Medium ### Complete Code Snippet ```bash # pin the CLI to a reviewed version — never @latest in security-sensitive installs npx clawhub@0.23.3 install @orionshaowswmw/inference-expert-agents ``` ### Technical Analysis The documented installation procedure uses `npx` to resolve, download, and execute the `clawhub` npm package. Pinning the package to version `0.23.3` reduces uncontrolled version drift, but a version pin alone does not provide cryptographic integrity or prove that the resolved registry artifact is the one reviewed by the user. The README does not provide an expected package checksum, signature, immutable artifact identity, or a download-and-review procedure. Consequently, users following the recommended command place installation-time trust in the npm registry, package publisher, dependency graph, and the executable behavior of the `clawhub` package. The audited Python tools otherwise claim a minimal runtime dependency on `python3` and the standard library. Executing a separately distributed npm CLI introduces a broader code-execution trust boundary than the Skill's own runtime functionality requires. This finding does not establish that `clawhub@0.23.3` is malicious. It identifies the absence of integrity controls around a command that executes externally distributed code. ### Attack Path 1. A user copies the installation command from the README. 2. `npx` resolves `clawhub@0.23.3` and its dependency graph through npm infrastructure. 3. The downloaded CLI executes with the privileges of the installing user. 4. If the package artifact, publisher account, registry response, or dependency chain has been compromised, attacker-controlled code executes before or during Skill installation. 5. That code could access any files, environment variables, network resou ...[truncated 962 chars]
Remediation
<![CDATA[ ## Remediation Suggestions 1. Publish a cryptographic checksum or signed provenance record for the exact installer and Skill artifact. 2. Provide a download-first workflow that allows users to inspect and verify the package before executing it. 3. Document the expected npm registry and recommend lockfile or integrity verification where supported. 4. State explicitly that `npx` downloads and executes third-party code with the invoking user's permissions. 5. Recommend running installation in a restricted environment without sensitive environment variables or unnecessary filesystem access. 6. Avoid advising users to run the installer with `sudo` or another elevated account. 7. Where possible, offer a non-executing installation option based on a signed, immutable archive. 8. Maintain the version pin and add reproducible-build or package-provenance information to strengthen assurance beyond version naming alone. ]]>
Vulnerability Patterns
  • Memory PoisoningPersistent Context Injection, Context Window Stuffing, Memory Manipulation
  • Rogue AgentSelf-Modification, Session Persistence
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Findings (34)

Intent-Code Divergence

High
Confidence
99% confidence
Finding
The security/privacy documentation contradicts itself about whether the skill performs network calls, which undermines trust in the safety posture and can cause policy violations. In security-sensitive pipelines, incorrect documentation about egress is itself dangerous because users may expose sensitive prompts or content to external services unintentionally.

Memory Manipulation

High
Category
Memory Poisoning
Content
- **Files written:** nothing by default. Stateful tools (`cache_kit`, `memory_kit`,
  `ladder_trace`, `eval_regression`, `retrieval`, `selfimprove`) are strictly opt-in: they refuse stateful
  operations unless you pass `--db PATH` (0600 enforced on every write and load — loose
  files are repaired). State may contain prompt text; keep the path private. Corrupt state files are refused with an error and left untouched — never silently reset. Selftest uses
  `mktemp -d`, cleaned on exit.
- **Network:** none. All processing is local.
- **Binaries:** `python3` (standard library only — no pip installs).
Confidence
90% confidence
Finding
Skill manipulates agent memory, state, or stored context. Memory corruption can alter personality, override safety rules, or cause unpredictable behavior.

Description-Behavior Mismatch

High
Confidence
98% confidence
Finding
The README makes materially inconsistent statements about network behavior: earlier it says `citecheck.py` queries public scholarly registries, while the permissions/privacy section says `Network: none. All processing is local.` This can mislead operators into deploying the skill in environments that prohibit egress or handling data under a false assumption that no text ever leaves the host.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding
There is a clear description-behavior mismatch. The declared purpose describes a sophisticated system for managing AI inference quality, verification, and cost. The actual code only performs token-efficient JSON re-encoding: parsing stdin JSON, optionally compacting formatting, optionally replacing keys with short aliases plus metadata, and expanding that representation. It does not interface with an AI agent, make decisions about inference compute, verify claims, manage citations, or implement any of the listed reliability/control mechanisms. This is not merely a supporting detail; it is an entirely different primary purpose.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding
The declared description promises a broad inference-control and verification framework for AI reasoning quality, claim checking, citation enforcement, abstention, memory, MCP exposure, eval regression gates, and answer certificates. The supplied code does none of that. Its primary purpose is narrowly to shorten text outputs using regex-based sentence/phrase removal and provide summary metrics. While the description mentions controlling inference cost, this code only reduces output verbosity after generation; it does not decide inference budgets, verify claims, orchestrate deliberation, manage memory, or expose MCP capabilities. This is a materially different primary purpose and includes undeclared text-condensation functionality.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding
The declared description presents a broad inference-governance/meta-reasoning skill focused on deciding compute budgets, verifying claims, preventing hallucinations, orchestrating ensembles, abstention, evaluation gates, and related agent-control behaviors. The supplied code does not implement any of those core functions. Instead, it is a narrow cache utility: it stores and retrieves prompt/answer pairs from disk, performs exact or approximate lookup via token overlap, prunes by age and capacity, and reports cache statistics. While 'semantic caching' is one phrase mentioned in the description, this code only covers that single supporting component and none of the larger declared behaviors. Because the actual primary purpose is materially different and introduces concrete filesystem-backed caching behavior not reflected as the main function of the description, this is a clear mismatch.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding
The declared description presents a broad inference-governance and reliability framework for an AI agent. The supplied code instead is a standalone command-line compression toolkit. Its concrete behaviors are: splitting text into sentences, estimating token count via a chars/4 heuristic, scoring sentences lexically, producing an extractive compressed version of text, reporting token-budget stats, and printing a static Chain-of-Draft instruction snippet. While token efficiency and a draft-output contract are loosely related to inference cost control, the vast majority of the declared capabilities are absent, and the primary purpose is materially different. Therefore the description does not accurately represent the code.

Tp4

High
Category
MCP Tool Poisoning
Confidence
96% confidence
Finding
The description presents a comprehensive system for controlling inference compute and agent reliability across many mechanisms. The supplied code implements only one small subset: claim extraction, independent verification-question generation, heuristic verdicting of verification answers, and answer revision. While claim-level verification is indeed part of the declared description, the actual code lacks most of the prominently declared capabilities and has a much narrower primary purpose. There are no extra risky permissions or hidden resource accesses, but the description materially overstates the skill’s scope and functionality.

Tp4

High
Category
MCP Tool Poisoning
Confidence
96% confidence
Finding
The description promises a comprehensive system for deciding how much inference compute to spend, verifying claims before assertion, controlling hallucination/overthinking, and managing many advanced reliability mechanisms. The supplied code does only two things: (1) build a protocol/schedule for multi-agent debate rounds, and (2) calculate a rough consensus metric from final positions by normalizing answer strings and checking agreement rate. While multi-agent debate could be considered one small component related to judge/debate ensembles, the actual code does not implement most of the declared functionality, and its primary purpose is materially narrower and different. There are no undeclared sensitive permissions or resource accesses, but the description substantially overstates the code’s behavior, so this is a mismatch.

Tp4

High
Category
MCP Tool Poisoning
Confidence
94% confidence
Finding
This is a description-behavior mismatch because the declaration presents a comprehensive inference orchestration and verification framework, while the code chunk only covers one sub-area explicitly mentioned in the description: eval regression gates. The implementation reads golden cases and outputs, computes pass/fail rates, stores run metadata, queues failures, compares runs, and suggests cron scheduling. It does not implement the many other advertised mechanisms for adaptive reasoning, verification, abstention, caching, memory, or MCP exposure. There is no evidence of unrelated dangerous resource access beyond optional local state-file reads/writes, but the declared purpose materially overstates the functionality of this code chunk.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding
The declared description is about turning an agent into an expert at inference governance and verification, including compute allocation, claim checking, citation controls, abstention, caching, memory, and evaluation gates. The supplied code does none of that. It is a narrow utility for best-effort JSON repair of model outputs, operating on input text from arguments, file, or stdin and emitting repaired JSON. This is a materially different primary purpose, not just a supporting detail of the declared skill. No sensitive or undeclared resource access is evident beyond normal local input handling, but the behavior is plainly unrelated to the declared capability set.

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding
The description presents a comprehensive inference-control and verification framework, emphasizing compute budgeting, self-consistency, claim verification, citation gating, abstention, cost cascades, evaluation gates, and related reasoning controls. The supplied code does not implement those behaviors. Instead, it only implements one narrow component mentioned in passing in the description: agent memory. Specifically, it stores reflections/rules/skills in a JSON file, searches them via simple token-overlap scoring, forgets old entries, and reports counts. This is a materially different primary purpose from the declared end-to-end inference triage/verification system. The code also performs concrete persistent filesystem state management that is not reflected as the main declared behavior. Because the implemented functionality is only a small subsystem and lacks the claimed inference-budgeting/verification/citation/judging/cost-control mechanisms, this is a clear description-behavior mismatch.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding
The declared description promises a broad AI inference-governance/verification skill, but the supplied code is a standalone retrieval tool for building and querying a local text index. While retrieval can support anti-hallucination workflows in a broader system, this code does not implement the declared features such as deliberation budgets, self-consistency, citation gates, calibrated abstention, judge ensembles, cost cascades, semantic caching, agent memory, MCP exposure, eval regression gates, or answer certificates. Its primary purpose is materially different: local BM25 search over a private corpus.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding
The declared description is about controlling and validating an AI agent's inference behavior and verification process. The supplied code does something materially different: it provides a command-line utility for sandbox debugging drills and transcript-based environment hygiene checks. Its logic is centered on predefined packaging/runtime failure scenarios and simple keyword-based scoring/inspection, not on inference compute allocation, factual verification, calibration, citation enforcement, research loops, or agent memory. Although both may loosely relate to agent reliability, the primary purpose and concrete capabilities do not match.

Tp4

High
Category
MCP Tool Poisoning
Confidence
95% confidence
Finding
The code does not implement the declared inference-control behaviors such as difficulty triage, deliberation budgets, self-consistency checks, claim-level verification, citation gating, judge ensembles, calibrated abstention, semantic caching, deep-research loops, MCP exposure, or answer certificates. Instead, it is a standalone CLI utility for maintaining and self-improving a rule set using recorded outcomes and eval regression results. While there is a loose thematic connection to eval regression gates and memory, the primary purpose is materially different: self-modification governance rather than runtime inference orchestration or verification. Therefore the description significantly overstates and misrepresents what the code chunk actually does.

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding
The declared description presents a comprehensive skill for deciding how much reasoning compute to spend and for validating outputs before asserting them. The supplied code does not implement those behaviors. Instead, it is a stdlib-only read-only command-line utility focused on generation speed: measuring tok/s, summarizing sample throughput, listing serving optimization levers (speculative decoding, batching, quantization, prompt caching, etc.), and advising parallel execution of independent samples. While there is a small thematic overlap with cost control, semantic/prompt caching, and self-consistency sampling, the actual code’s primary purpose is operational performance tuning rather than inference triage, verification, abstention, or answer certification. Therefore the description materially overstates and mischaracterizes the code's actual behavior.

Tp4

High
Category
MCP Tool Poisoning
Confidence
95% confidence
Finding
The description promises a comprehensive system for controlling and auditing inference behavior across many dimensions of agent reasoning and cost management. The supplied code does not implement those capabilities. Instead, it provides three specific utilities: (1) checking that a proposed tool call matches a declared tool specification, including required args, types, enums, and undeclared arguments; (2) validating JSON data against a small JSON-schema-like subset; and (3) returning simple retry decisions for 'assert' vs 'suggest' semantics based on attempt counts. These are related to agent robustness and structured output, but they are only a small supporting subset of the declared functionality. The primary purpose is materially different in scope and focus, so this is a mismatch.

Ae1

High
Category
analysis-evasion
Content
read `manifest.json` first (tool registry + progressive-disclosure loading plan); pipe any
Confidence
100% confidence
Finding
Referenced artifact was not completely inspected

Ae1

High
Category
analysis-evasion
Content
read `manifest.json` first (tool registry + progressive-disclosure loading plan); pipe any
Confidence
100% confidence
Finding
Referenced artifact was not completely inspected

Ae1

High
Category
analysis-evasion
Content
read `manifest.json` first (tool registry + progressive-disclosure loading plan); pipe any
Confidence
100% confidence
Finding
Referenced artifact was not completely inspected

Memory Manipulation

High
Category
Memory Poisoning
Content
" && ok "_common: fail hint additive, default unchanged" || bad "_common hint"


# T49 corrupt state is refused (rc=2) and left untouched; fresh db still works
echo '{corrupt!!' > "$SBX/corrupt.json"
out="$(python3 "$HERE/tools/cache_kit.py" get --db "$SBX/corrupt.json" --prompt "p" 2>&1)"; rc=$?
echo "$out" | grep -q 'corrupt/unreadable' && [ "$rc" -eq 2 ] && [ "$(cat "$SBX/corrupt.json")" = '{corrupt!!' ] \
Confidence
90% confidence
Finding
Skill manipulates agent memory, state, or stored context. Memory corruption can alter personality, override safety rules, or cause unpredictable behavior.

Description-Behavior Mismatch

High
Confidence
98% confidence
Finding
The manifest describes a skill for inference control features such as deliberation budgets, verification, abstention, caching, research loops, and answer certificates. This file instead provides canned sandbox failure drills and heuristic checks for package/install/interpreter hygiene, which is a different operational domain and not an obvious implementation detail of inference expertise.

Self-Modification

High
Category
Rogue Agent
Content
decision = "ADOPT" if delta >= a.min_delta else "REJECT"
    frozen = cand["pass_rate"] < 0.5
    if frozen:
        decision = "FREEZE (candidate pass rate below 0.5 — fix basics before self-modifying)"
    ev.setdefault("adoptions", []).insert(0, {
        "ts": time.time(), "baseline": a.baseline_run, "candidate": a.candidate_run,
        "delta": delta, "decision": decision})
Confidence
90% confidence
Finding
Skill modifies its own code, configuration, or behavior at runtime. Self-modification enables an agent to escalate privileges, disable safety constraints, or install persistent backdoors.

Natural-Language Policy Violations

Medium
Confidence
81% confidence
Finding
The README describes the skill as turning an agent into an expert, which is a strong qualitative claim presented as fact rather than a bounded or clearly qualified description. This can be read as overstating the system's capabilities and conflicts with organizational expectations against unsupported natural-language performance claims.

Skill Enumeration

Medium
Category
Agent Snooping
Content
```bash
# integrity check on THIS skill's own installed copy (no other skill is read)
sha256sum skills/inference-expert-agents/SKILL.md
```

## Self-test
Confidence
80% confidence
Finding
Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Static analysis

No suspicious patterns detected.