T09 · Insecure Skill Coding Practices
Error
- Location
- scripts/crawl.py:46
- Finding
- Unrestricted URL Fetching Enables Server-Side Request Forgery<![CDATA[ ## Vulnerability Details **File Location**: `scripts/crawl.py:46-50`, `scripts/crawl.py:236-239`, `scripts/crawl.py:325-326`, `scripts/crawl.py:379-394` **Vulnerability Type**: Server-Side Request Forgery (SSRF) **Risk Level**: High ### Vulnerable Code ```python def fetch(url: str): """Fetch URL, return (html_text, final_url) or (None, None) on failure.""" try: r = httpx.get(url, headers=HEADERS, timeout=TIMEOUT, follow_redirects=True) if r.status_code == 200 and "text/html" in r.headers.get("content-type", ""): return r.text, str(r.url) ``` ```python # ── Level 1 ────────────────────────────────────────────────────────────── level1_urls = [root_url] + extra_urls level1_pages = {} for url in level1_urls: html, final_url = fetch(url) ``` ```python # ── Check for existing llms.txt ─────────────────────────────────────────── llms_txt_url = f"https://{domain}/llms.txt" existing_llms_txt = fetch_text_file(llms_txt_url) ``` ```python if __name__ == "__main__": if len(sys.argv) < 2: print("Usage: crawl.py <url> [extra_url1 extra_url2 ...]", file=sys.stderr) sys.exit(1) args = sys.argv[1:] deep = "--deep" in args args = [a for a in args if a != "--deep"] root_url = args[0] extra_urls = args[1:] if len(args) > 1 else [] if not root_url.startswith("http"): root_url = "https://" + root_url result = crawl(root_url, extra_urls, deep=deep) print(json.dumps(result, indent=2)) ``` ### Technical Analysis The crawler sends requests to the user-controlled root URL and every user-provided extra URL without validating the URL scheme, destination address, port, or DNS resolution. It also enables `follow_redirects=True` without checking the destination of each redirect. Consequently, the crawler can be instructed to contact: - Loopback services such as `127.0.0.1` or `::1` - RFC1918 private networks - Link-local services and cloud metadata endpoints - Internal DN ...[truncated 1985 chars]
- Remediation
- <![CDATA[ ## Remediation Suggestions 1. Parse every URL with a strict URL parser and permit only exact `http` and `https` schemes. 2. Reject URLs containing credentials, malformed hostnames, or disallowed ports. 3. Resolve the hostname before connecting and reject all loopback, private, link-local, multicast, unspecified, reserved, and documentation address ranges for both IPv4 and IPv6. 4. Validate every redirect destination before following it. Prefer disabling automatic redirects and processing each redirect manually. 5. Re-resolve the hostname on each request and redirect to reduce DNS-rebinding exposure. 6. Block cloud metadata destinations, including link-local metadata addresses, even when reached through DNS aliases. 7. Require optional extra URLs to use the approved registrable domain unless the user explicitly authorizes another public domain. 8. Apply outbound network controls at the container or firewall level so the crawler cannot reach private networks. 9. Maintain a strict port allowlist, normally ports 80 and 443. 10. Add tests covering direct private addresses, IPv6 loopback, alternative numeric IP notation, DNS rebinding, and public-to-private redirects. ]]>
