T09 · Insecure Skill Coding Practices
Error
- Location
- scripts/scan.py:253
- Finding
- Server-Side Request Forgery Through Unvalidated Sitemap URLs<![CDATA[ ## Vulnerability Details **File Location**: `scripts/scan.py:253-261` and `scripts/scan.py:435-440` **Vulnerability Type**: Server-Side Request Forgery (SSRF) **Risk Level**: High ### Vulnerable Code #### Sitemap URL obtained from robots.txt ```python sitemap_url = f"{parsed.scheme}://{parsed.netloc}/sitemap.xml" response = self.session.get(sitemap_url, timeout=10) if response.status_code != 200: # Try robots.txt to find sitemap robots = self.fetch_robots_txt() sitemap_match = re.search(r'Sitemap:\s*(.+)', robots, re.IGNORECASE) if sitemap_match: sitemap_url = sitemap_match.group(1).strip() response = self.session.get(sitemap_url, timeout=10) ``` #### Page URLs obtained from the sitemap ```python for url_data in urls: url = url_data.get("loc") if not url or url == self.url: continue try: response = analyzer.session.get(url, timeout=15) ``` ### Technical Analysis The scanned website controls both the `Sitemap:` directive returned in `robots.txt` and the URL values contained in the sitemap. The application passes these values directly to `requests.Session.get()` without validating: - The URL scheme. - Whether the destination belongs to the original scanned site. - Whether the resolved address is loopback, private, link-local, reserved, or otherwise internal. - Whether an allowed hostname resolves to a prohibited address. - Redirect destinations and each redirect hop. - Destination ports. The `requests` library follows HTTP redirects by default. Consequently, even an initially acceptable URL can redirect the scanner to an internal destination. Fetching website resources is part of the intended functionality, but allowing a remote website to select arbitrary cross-origin destinations is unnecessary for ordinary same-site analysis. This creates an SSRF primitive in the environment running the scanner. The initial user-supplied target is also fetched without network-range restrictions ...[truncated 2641 chars]
- Remediation
- <![CDATA[ ## Remediation Suggestions 1. **Restrict URL schemes** - Parse every target with `urllib.parse.urlparse()`. - Permit only `http` and `https`. - Reject URLs containing embedded credentials or malformed hostnames. 2. **Enforce an origin policy** - By default, require sitemap and page URLs to use the original scanned hostname or an explicitly approved registrable domain. - If cross-origin sitemaps are required, make that behavior opt-in and apply an explicit allowlist. 3. **Block prohibited network ranges** - Resolve all destination hostnames before connecting. - Reject every resolved IPv4 and IPv6 address classified as loopback, private, link-local, multicast, reserved, unspecified, or otherwise non-global. - Repeat this check immediately before connection to reduce DNS rebinding risk. - Apply the same controls to the initial user-supplied target. 4. **Validate redirects** - Disable automatic redirects with `allow_redirects=False`, or manually follow a small number of redirects. - Reapply scheme, origin, hostname, port, and resolved-address validation to every redirect hop. 5. **Constrain ports and outbound access** - Permit only required destination ports, normally 80 and 443. - Use operating-system, container, or firewall egress controls to prevent access to internal and metadata networks. - Run the scanner in an isolated environment with no access to sensitive internal services. 6. **Limit fetched content** - Stream responses and enforce maximum response sizes. - Set connection and read timeouts separately. - Validate content types before parsing data as HTML or XML. 7. **Add regression tests** - Verify rejection of loopback, RFC1918, link-local, IPv6 local, integer-encoded IP, mixed-notation IP, and redirect-based destinations. - Test cross-origin sitemap directives, DNS rebinding scenarios, and URLs containing user information or unusual ports. ]]>
