T09 · Insecure Skill Coding Practices
Error
- Location
- scripts/robots_txt_gen.py:313
- Finding
- Server-Side Request Forgery Through Unrestricted Remote robots.txt Validation<![CDATA[ ## Vulnerability Details **File Location**: `scripts/robots_txt_gen.py`, lines 313-318 **Vulnerability Type**: Server-Side Request Forgery (SSRF) and unbounded response consumption **Risk Level**: High ### Vulnerable Code ```python def _load_content_url(url): try: req = urllib.request.Request(url, headers={"User-Agent": "robots-txt-gen/1.0"}) with urllib.request.urlopen(req, timeout=10) as resp: return resp.read().decode("utf-8", errors="replace") except Exception as e: print(f"Error fetching {url}: {e}", file=sys.stderr) sys.exit(1) ``` ### Technical Analysis The `validate --url` command passes a user-controlled URL directly to `urllib.request.urlopen()` without validating its scheme, destination hostname, resolved IP address, or redirect targets. Consequently, a user who can control the command arguments can cause the process to issue requests using the host system's network access. The implementation does not prevent requests to loopback, private, link-local, reserved, or cloud metadata addresses. Automatic redirects can also allow a public URL to redirect the request to an internal destination unless every redirect target is independently validated. The response is loaded in full through `resp.read()` without a maximum size. A remote server can therefore return an excessively large body and cause substantial memory consumption or process termination. Fetched content is subsequently passed to the robots.txt validator. Validation messages can reproduce portions of malformed input, creating a potential channel through which text returned by an internal endpoint may be disclosed to the caller. ### Attack Path 1. An attacker obtains the ability to influence the URL supplied to the Skill's validation command. 2. The attacker invokes a command such as: ```bash python3 scripts/robots_txt_gen.py validate --url http://127.0.0.1:PORT/internal ``` Alternatively, the attacker supplies a pu ...[truncated 1463 chars]
- Remediation
- <![CDATA[ ## Remediation Suggestions Apply layered URL and response validation: 1. Parse the URL before opening it and allow only explicitly required schemes, normally `https` and optionally `http`. 2. Reject URLs containing embedded credentials. 3. Resolve the hostname and reject every resolved address belonging to loopback, private, link-local, multicast, reserved, or unspecified address ranges for both IPv4 and IPv6. 4. Prevent DNS rebinding by ensuring that the validated destination is the destination used for the connection. 5. Disable automatic redirects or implement a controlled redirect handler that repeats scheme, hostname, and resolved-address validation for every redirect target. 6. Prefer an explicit allowlist of trusted hosts when remote validation is used in an automated or multi-user environment. 7. Stream the response in bounded chunks rather than calling an unrestricted `resp.read()`. 8. Enforce a conservative maximum response size appropriate for robots.txt files and abort when the declared or observed size exceeds it. 9. Restrict accepted content types where practical and retain connection and read timeouts. 10. Avoid reflecting arbitrary fetched lines in diagnostic output, or sanitize and truncate such values before displaying them. 11. Run the Skill in an environment with outbound network restrictions that blocks access to internal networks and cloud metadata services. A hardened implementation should validate the initial URL and all redirects, verify resolved addresses, and read no more than a fixed maximum number of bytes. ]]>
