Scrape

• Outcome: Review
• Latest audit: Sep 10, 2026 · 5:54 PM UTC
• Version: 1.0.0

Overview

The skill is mostly a disclosed web-scraping guide, but its sample compliance code can allow scraping when robots.txt checks fail.

Review before installing or using the sample code. The guidance is not deceptive and does not install anything, but users should change the robots.txt check to fail closed or ask before proceeding on errors, and add URL/redirect/IP validation before using the fetcher in any shared tool, server, or workflow that accepts user-supplied URLs.

A.I.G

Vulnerability Patterns

• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally

Findings (2)

T09 · Insecure Skill Coding Practices

Warning

• Location: code.md:15
• Finding: Fail-Open robots.txt Validation Permits Scraping Without Confirmed Authorization<![CDATA[ ## Vulnerability Details **File Location**: `code.md`, lines 15-18 **Vulnerability Type**: Fail-open authorization and compliance check **Risk Level**: Medium ### Complete Code Snippet ```python try: rp.read() except Exception: return True # No robots.txt = allowed ``` ### Technical Analysis The `can_scrape` implementation treats every exception raised while retrieving or parsing `robots.txt` as permission to scrape. The exception handler does not distinguish a confirmed absence of a robots file from DNS errors, TLS failures, connection timeouts, access-denied responses, malformed content, or parser failures. This is a fail-open control: when the mechanism responsible for determining whether access is permitted cannot reach a reliable decision, it authorizes the operation. It also conflicts with the Skill's stated policy that robots.txt must be checked before scraping. ### Attack Path 1. The scraper is given a target URL. 2. The target's `robots.txt` request fails because of a network error, TLS problem, malformed response, or deliberate disruption. 3. `RobotFileParser.read()` raises an exception. 4. The broad `except Exception` handler returns `True`. 5. The caller interprets the result as authorization and proceeds to scrape the target without successfully evaluating its robots.txt policy. ### Impact Assessment An affected scraper can access paths without confirming that automated access is allowed. This does not directly grant operating-system privileges or establish code execution, but it bypasses the Skill's intended access-policy gate and may cause unauthorized requests, terms-of-service violations, blocking of the scraper's network identity, or legal and compliance exposure. ]]>
• Remediation: <![CDATA[ ## Remediation Suggestions - Fail closed when robots.txt cannot be reliably retrieved or parsed. - Return a tri-state result such as `ALLOWED`, `DISALLOWED`, or `INDETERMINATE` rather than conflating errors with authorization. - Permit scraping only when the result is explicitly `ALLOWED`. - Distinguish a confirmed HTTP 404 response from network, TLS, authorization, server, and parsing failures. - Set explicit connection and read timeouts. - Log the failure reason without exposing credentials or sensitive query parameters. - Require explicit operator approval before proceeding after an indeterminate result. Example hardened behavior: ```python from enum import Enum from urllib.robotparser import RobotFileParser from urllib.parse import urlparse class RobotsDecision(Enum): ALLOWED = "allowed" DISALLOWED = "disallowed" INDETERMINATE = "indeterminate" def can_scrape(url: str, user_agent: str = "*") -> RobotsDecision: parsed = urlparse(url) robots_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt" rp = RobotFileParser() rp.set_url(robots_url) try: rp.read() except Exception: return RobotsDecision.INDETERMINATE return ( RobotsDecision.ALLOWED if rp.can_fetch(user_agent, url) else RobotsDecision.DISALLOWED ) ``` ]]>

T09 · Insecure Skill Coding Practices

Error

• Location: code.md:48
• Finding: Unrestricted URL Fetching Enables Server-Side Request Forgery<![CDATA[ ## Vulnerability Details **File Location**: `code.md`, lines 48-84 **Vulnerability Type**: Server-Side Request Forgery (SSRF) **Risk Level**: High ### Complete Code Snippet ```python def fetch_politely(session, url, min_delay=2.0, max_retries=5): """Fetch with rate limiting, backoff, and audit logging.""" for attempt in range(max_retries): # Polite delay with jitter delay = min_delay + random.uniform(0, 0.5) time.sleep(delay) response = session.get(url) # Audit trail logger.info(f"SCRAPE url={url} status={response.status_code}") # Check rate limit headers remaining = response.headers.get("X-RateLimit-Remaining") if remaining and int(remaining) < 5: logger.warning(f"Rate limit low: {remaining} remaining") time.sleep(10) # Proactive slowdown # Handle 429 if response.status_code == 429: retry_after = response.headers.get("Retry-After", 60) wait = int(retry_after) if str(retry_after).isdigit() else 60 logger.warning(f"429 received, waiting {wait}s") time.sleep(wait) continue # Success or client error (don't retry 4xx except 429) if response.status_code < 500: return response # Server error: exponential backoff wait = min(2 ** attempt + random.uniform(0, 1), 60) logger.warning(f"5xx error, retry in {wait:.1f}s") time.sleep(wait) raise Exception(f"Failed after {max_retries} retries: {url}") ``` ### Technical Analysis The fetcher passes the supplied `url` directly to `session.get()` without validating its scheme, hostname, resolved IP address, port, or redirect destination. No allowlist is enforced, and no checks reject loopback, private, link-local, multicast, reserved, or cloud metadata address ranges. The `requests` library follows redirects by def ...[truncated 1864 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions - Allow only explicitly supported schemes, normally `https` and, where necessary, `http`. - Reject URLs containing embedded credentials. - Use an explicit hostname allowlist whenever the intended scraping scope is known. - Resolve the hostname before connecting and reject every loopback, private, link-local, multicast, unspecified, and reserved IP address. - Validate all IPv4 and IPv6 results rather than checking only one resolved address. - Disable automatic redirects and validate each redirect target before following it. - Defend against DNS rebinding by connecting to a validated resolved address while preserving the intended hostname for TLS and HTTP handling, or route requests through an SSRF-resistant proxy. - Restrict destination ports to those required by the task. - Apply outbound firewall or proxy rules so application-level validation is not the only protection. - Set explicit connection and read timeouts and a maximum response size. - Avoid logging credentials, tokens, or sensitive query parameters contained in URLs. A hardened request flow should follow this sequence: 1. Parse and normalize the URL. 2. Verify the scheme, hostname, and port against policy. 3. Resolve all destination addresses. 4. Reject prohibited address ranges. 5. Perform the request without automatic redirects. 6. Validate and resolve each redirect destination before issuing another request. 7. Enforce the same policy on every retry. ]]>

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• MCP Tool Poisoning: Hidden Instructions, Unicode Deception, Parameter Description Injection
• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands
• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access

Findings (4)

Autonomous Decision Making

Medium

• Category: Excessive Agency

Content

- **Public data, no login** — Generally legal (hiQ v. LinkedIn 2022)
- **Bypassing barriers** — CFAA violation risk (Van Buren v. US 2021)
- **Ignoring robots.txt** — Gray area, often breaches ToS (Meta v. Bright Data 2024)
- **Personal data without consent** — GDPR/CCPA violation
- **Republishing copyrighted content** — Copyright infringement

## Request Discipline

• Confidence: 75% confidence
• Finding: Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Description-Behavior Mismatch

Medium

• Confidence: 97% confidence
• Finding: The robots.txt compliance check fails open: any exception while fetching or parsing robots.txt causes the function to return True and allow scraping. In a scraping skill whose stated purpose is legal, robots.txt-compliant operation, this makes policy bypass likely during transient network errors, DNS/TLS failures, parser issues, or deliberate blocking of robots.txt retrieval.

Intent-Code Divergence

Medium

• Confidence: 98% confidence
• Finding: The comment says 'No robots.txt = allowed', but the code actually allows scraping on any exception from rp.read(), not just when robots.txt is absent. This mismatch is dangerous because maintainers may believe the logic is narrowly permissive while it actually bypasses robots enforcement under many failure conditions.

Natural-Language Policy Violations

Low

• Confidence: 95% confidence
• Finding: The example session setup forces `Accept-Language: en-US,en;q=0.9` for all requests. This is a natural-language/locale policy concern because it imposes a specific language preference without offering user opt-in or documenting why an English-only locale is required.

Static analysis

No suspicious patterns detected.