T05 · Unauthorized Access and Privilege Escalation
Error
- Location
- scripts/scrape_to_markdown.py:163
- Finding
- Unrestricted URL Fetching Enables Server-Side Request Forgery<![CDATA[ ## Vulnerability Details **File Location**: `scripts/scrape_to_markdown.py`, lines 163–165; request sink at lines 184–209 **Vulnerability Type**: Server-Side Request Forgery through insufficient destination validation **Risk Level**: High ### Vulnerable Code ```python def validate_url(url): p = urlparse(url) return p.scheme in ("http", "https") and bool(p.netloc) ``` The validation result is subsequently trusted when fetching the URL: ```python for u in urls: if not validate_url(u): print(json.dumps({"ok": False, "error": f"Invalid URL: {u}"}, ensure_ascii=False)) sys.exit(1) out_dir = Path(args.output_dir) out_dir.mkdir(parents=True, exist_ok=True) results = [] for url in urls: item = { "url": url, "ok": False, "title": "", "status": None, "selector_used": None, "backend": None, "markdown": "", "preview": "", "output_markdown_file": None, "error": None, } try: page, backend = fetch_page( url=url, js=args.js, wait_selector=args.wait_selector or None, timeout=args.timeout, automatch_domain=args.automatch_domain or None, ) ``` ### Technical Analysis The URL validator only verifies that the scheme is HTTP or HTTPS and that a network location is present. It does not verify whether the destination is a public Internet host. Consequently, the validator accepts destinations such as: - Loopback addresses, including `127.0.0.1` and `::1` - Private IPv4 and IPv6 network ranges - Link-local addresses - Cloud instance metadata addresses - Reserved or unspecified addresses - Hostnames that resolve to internal addresses This behavior contradicts the public-web-only restriction documented in `SKILL.md`. It also exceeds the minimum network privileges required for public webpage scraping. The fetched response is converted to Markdown, written to the output direct ...[truncated 1900 chars]
- Remediation
- <![CDATA[ ## Remediation Suggestions 1. Resolve each destination hostname before making a request. 2. Reject every resolved IPv4 and IPv6 address classified as loopback, private, link-local, multicast, reserved, or unspecified. 3. Apply the same validation to every redirect target before following it. 4. Defend against DNS rebinding by ensuring that the connection uses an already validated address or by verifying the actual connected peer address. 5. Consider using an explicit domain allowlist when the expected scraping targets are known. 6. Reject URLs containing ambiguous host representations, malformed ports, or user-information components unless explicitly required. 7. Add tests covering at least: - `127.0.0.1` - `::1` - RFC1918 IPv4 ranges - IPv6 unique-local ranges - `169.254.169.254` - Hostnames resolving to internal addresses - Redirects from public hosts to internal addresses 8. Enforce outbound network restrictions at the container or firewall layer as defense in depth. ]]>
