T09 · Insecure Skill Coding Practices
Error
- Location
- parsers/generic.py:64
- Finding
- Unrestricted URL Fetching Enables Server-Side Request Forgery<![CDATA[ ## Vulnerability Details **File Location**: `parsers/generic.py:64-73` and `core/utils.py:21-36` **Vulnerability Type**: Server-Side Request Forgery (SSRF) **Risk Level**: High ### Vulnerable Code URL extraction accepts any HTTP or HTTPS URL without validating whether its destination is public: ```python _URL_PATTERN = re.compile( r"https?://" # scheme r"(?:[a-zA-Z0-9\-._~:/?#\[\]@!$&'()*+,;=%])+", # rest of URI chars re.IGNORECASE, ) def extract_urls(text: str) -> list[str]: """Return a deduplicated, order-preserved list of URLs found in *text*.""" seen: set[str] = set() urls: list[str] = [] for match in _URL_PATTERN.finditer(text): url = match.group(0).rstrip(".,;:!?)") # strip trailing punctuation if url not in seen: seen.add(url) urls.append(url) return urls ``` The generic parser then fetches that destination and follows redirects automatically: ```python def _fetch_html(self, url: str) -> str | None: """Download the raw HTML content of *url*.""" response = requests.get( url, headers=self._get_headers(), timeout=self.timeout, allow_redirects=True, ) response.raise_for_status() return response.text ``` ### Technical Analysis Any URL that is not classified as Twitter, Reddit, or YouTube is passed to `GenericParser`. The implementation does not resolve and inspect the destination address before making the request. Consequently, an attacker can supply URLs targeting: - Loopback interfaces such as `127.0.0.1` or `::1` - RFC1918 private networks - Link-local addresses - Cloud instance metadata services - Internal DNS names - Reserved or otherwise non-public address ranges Setting `allow_redirects=True` also permits a public endpoint to redirect the request to a private destination. No validation is performed after a redirect or after DNS resolution, leaving the implementation exposed to redirect-ba ...[truncated 2019 chars]
- Remediation
- <![CDATA[ ## Remediation Suggestions 1. Parse each URL and permit only `http` and `https` schemes with a valid hostname. 2. Resolve all destination addresses before connecting. 3. Reject IPv4 and IPv6 addresses classified as loopback, private, link-local, multicast, reserved, unspecified, or non-global. 4. Explicitly block cloud metadata destinations, including link-local metadata addresses and provider-specific metadata hostnames. 5. Disable automatic redirects. If redirects are required, follow them manually and repeat scheme, hostname, DNS, and IP validation for every hop. 6. Protect against DNS rebinding by ensuring that the validated address is the address actually used for the connection. 7. Consider enforcing an allowlist or requiring explicit approval for destinations outside expected public domains. 8. Use streamed responses and enforce strict limits on: - Downloaded bytes - Decompressed bytes - Redirect count - URLs processed per invocation - Extracted and persisted content size 9. Apply outbound network controls at the runtime or container layer to prevent access to private networks and metadata services. 10. Return generic network errors rather than exposing detailed internal connection information. ]]>