Back to skill

Security audit

Agent Testing & QA Toolkit: Integration, Chaos, and Contract Testing for Multi-Agent Systems

Security checks for vulnerabilities and agentic risk

Overview

This is a non-executing QA guide, but its copy-ready examples handle real API keys, financial workflows, and saved API recordings in ways that need review before use.

Review and harden the examples before installing or using this guide in CI. Use sandbox-only, capped credentials for tests; do not use production keys in the harness; validate and allowlist API hosts before adding bearer headers; disable cross-origin redirects; redact cassette contents and response headers; and never commit unreviewed recordings that may include account, escrow, balance, or transaction data.

Vulnerability Patterns
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (2)

T09 · Insecure Skill Coding Practices

Error
Location
SKILL.md:237
Finding

Bearer API Key Can Be Sent to an Arbitrary Configurable Host

Content
View full analysis
dict: """Make a real API call to GreenHelix.""" start = time.monotonic() resp = self.session.post( f"{self.base_url}/v1", json={"tool": tool, "input": input_data}, ) ``` ### Technical Analysis The `AgentTestHarness` accepts an unrestricted `base_url` and stores the API key in the default headers of a persistent `requests.Session`. No validation ensures that the destination uses HTTPS or belongs to an approved GreenHelix hostname. When the harness operates in `record` or `passthrough` mode, `_live_call()` sends the bearer credential directly to the configured destination. A malicious or accidentally modified configuration can therefore redirect an authenticated request to an attacker-controlled server. Authentication is necessary for live GreenHelix integration tests, but allowing the credential to be transmitted to an arbitrary host exceeds the minimum privileges required by the declared testing functionality. The issue appears in illustrative, non-executable guide content; however, the guide explicitly pre ...[truncated 1373 chars]
Remediation
View remediation
str: parsed = urlparse(value) if ( parsed.scheme != "https" or parsed.hostname not in ALLOWED_HOSTS or parsed.username is not None or parsed.password is not None or parsed.port not in (None, 443) ): raise ValueError("Unapproved GreenHelix API endpoint") return value.rstrip("/") def _live_call(self, tool: str, input_data: dict) -> dict: endpoint = validate_base_url(self.base_url) return requests.post( f"{endpoint}/v1", headers={ "Content-Type": "application/json", "Authorization": f"Bearer {self.api_key}", }, ...[truncated 108 chars]

T09 · Insecure Skill Coding Practices

Warning
Location
SKILL.md:199
Finding

Sensitive API Interactions Are Written to Plaintext Cassette Files

Content
View full analysis
None: data = { "name": self.name, "metadata": self.metadata, "entries": [ { "request_tool": e.request_tool, "request_input": e.request_input, "response_status": e.response_status, "response_body": e.response_body, "response_headers": e.response_headers, "timestamp": e.timestamp, "latency_ms": e.latency_ms, } for e in self.entries ], } path.parent.mkdir(parents=True, exist_ok=True) path.write_text(json.dumps(data, indent=2, default=str)) ``` ```python if self._current_cassette: entry = CassetteEntry( request_tool=tool, request_input=input_data, response_status=result["status_code"], response_body=result["body"], response_headers=result["headers"], timestamp=time.time(), latency_ms=result["latency_ms"], ) self._current_cassette.entries.append(entry) ``` ### Technical Analysis Record mode captures complete tool inputs, response bodies, and response headers and serializes them as readable JSON under the cassette directory. The implementation provides no field redaction, safe-header allowlist, encryption, restrictive file permissions, retention policy, or warning against committing recordings. Commerce and agent API interactions may contain agent identifiers, escrow identifiers, transaction values, balances, marketplace details, operational metadata, or server-returned cookies and tokens. Saving all fields indiscriminately violates data minimization and creates an additional persistent copy of sensitive information. Although the request `Authorization` header is ...[truncated 1606 chars]
Remediation
View remediation
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
Findings (9)

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

Lines L0024-L0030 present the material as non-executing illustrative examples using the GreenHelix sandbox with no API key required to get started. Later sections include GitHub Actions deploy steps and monitoring code that explicitly use staging and production API keys and production endpoints, which directly contradicts the earlier framing of the examples' environment and credential assumptions.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
89% confidence
Finding

The guide encourages recording full request/response cassettes to disk before clearly foregrounding the risk that those artifacts can contain sensitive API data, identifiers, balances, or other transaction details. In real CI or shared repositories, this can lead to inadvertent retention and disclosure of sensitive response content even if API keys themselves are not stored.

Content

No source excerpt is available for this finding.

External Transmission

Medium
Category
Data Exfiltration
Confidence
60% confidence
Finding

Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.

Content

Scanner excerpt · SKILL.md (reported line 2133)May include surrounding context.

md
def configure_test_budget(api_key: str, agent_id: str) -> None:
    """Set conservative budget caps for test environments."""
    import requests
    resp = requests.post(
        "https://sandbox.greenhelix.netthe REST API",
        headers={
            "Authorization": f"Bearer {api_key}",

External Transmission

Medium
Category
Data Exfiltration
Confidence
70% confidence
Finding

Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.

Content

Scanner excerpt · SKILL.md (reported line 2133)May include surrounding context.

md
def configure_test_budget(api_key: str, agent_id: str) -> None:
    """Set conservative budget caps for test environments."""
    import requests
    resp = requests.post(
        "https://sandbox.greenhelix.netthe REST API",
        headers={
            "Authorization": f"Bearer {api_key}",

External Transmission

Medium
Category
Data Exfiltration
Confidence
50% confidence
Finding

Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.

Content

Scanner excerpt · SKILL.md (reported line 2165)May include surrounding context.

md
def check_rollback_criteria(
    api_key: str,
    agent_id: str,
    base_url: str = "https://api.greenhelix.net/v1",
) -> bool:
    """Return True if a rollback is needed."""
    session = requests.Session()

External Transmission

Medium
Category
Data Exfiltration
Confidence
50% confidence
Finding

Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.

Content

Scanner excerpt · SKILL.md (reported line 2656)May include surrounding context.

md
def check_rollback_criteria(
    api_key: str,
    agent_id: str,
    base_url: str = "https://api.greenhelix.net/v1",
) -> bool:
    """Return True if a rollback is needed."""
    session = requests.Session()

External Transmission

Medium
Category
Data Exfiltration
Confidence
50% confidence
Finding

Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.

Content

Scanner excerpt · SKILL.md (reported line 2707)May include surrounding context.

md
def check_rollback_criteria(
    api_key: str,
    agent_id: str,
    base_url: str = "https://api.greenhelix.net/v1",
) -> bool:
    """Return True if a rollback is needed."""
    session = requests.Session()

Intent-Code Divergence

Low
Category
Not specified by scanner
Confidence
99% confidence
Finding

L0521 states the REST API endpoint is POST /v1/{tool} with a JSON body containing tool and input, but the harness implementation posts to a single /v1-style endpoint while passing tool in the body. This is an active documentation contradiction that can mislead users building tests or mocks from the guide.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Low
Category
Not specified by scanner
Confidence
83% confidence
Finding

The example comment at L1715-L1717 says adding created_at as a required output is fine, which asserts a compatibility policy. However, the validator implementation does not actually check for newly required output fields at all, so the explanatory comment overstates what compatibility means and may mislead users about what the code verifies.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.