T09 · Insecure Skill Coding Practices
Error
- Location
- scripts/generate_llms_txt.py:27
- Finding
- Unrestricted URL Fetching Enables Server-Side Request Forgery## Vulnerability Details **File Location**: `scripts/generate_llms_txt.py:27-39`, with additional affected flows at `scripts/generate_llms_txt.py:88-111`, `scripts/generate_llms_txt.py:169-179`, and `scripts/generate_llms_txt.py:196-201` **Vulnerability Type**: Server-Side Request Forgery (SSRF) through user-controlled and sitemap-controlled URLs **Risk Level**: High ### Vulnerable Code ```python def __init__(self, domain, timeout=10): self.domain = domain.replace('https://', '').replace('http://', '').rstrip('/') self.base_url = f"https://{self.domain}" self.timeout = timeout self.visited = set() self.pages = [] def fetch(self, path='', full_url=None): """Fetch a URL with error handling.""" url = full_url or urljoin(self.base_url, path) try: resp = requests.get(url, timeout=self.timeout, allow_redirects=True) if resp.status_code == 200: return resp except: pass return None ``` Sitemap URLs are filtered using an unsafe substring comparison and then fetched: ```python def get_sitemap_urls(self): """Fetch URLs from sitemap.xml.""" urls = [] # Try common sitemap locations for path in ['/sitemap.xml', '/sitemap_index.xml', '/sitemap-index.xml']: resp = self.fetch(path) if resp and resp.status_code == 200: # Parse XML import xml.etree.ElementTree as ET try: root = ET.fromstring(resp.text.encode('utf-8')) # Handle both sitemap and urlset for elem in root.iter(): if elem.tag.endswith('loc'): urls.append(elem.text.strip()) except: pass if urls: break # Filter to same domain return [u for u in urls if self.domain in u][:50] # Limit to 50 ``` Interactive and file-based generat ...[truncated 4442 chars]
- Remediation
- ## Remediation Suggestions Apply centralized URL validation before every network request: 1. Parse URLs with `urllib.parse.urlsplit` and permit only explicit `http` and `https` schemes. 2. Reject URLs containing username or password components. 3. Normalize hostnames and compare the parsed hostname exactly against the approved hostname. If subdomains are needed, require either exact equality or a dot-boundary suffix such as `host.endswith("." + approved_host)`. 4. Resolve all destination hostnames before connecting. Reject every resolved IPv4 and IPv6 address that is loopback, private, link-local, multicast, reserved, unspecified, or otherwise non-global. 5. Protect against DNS rebinding by ensuring the validated address is the address actually used for the connection. 6. Disable automatic redirects or process redirects manually. Validate the scheme, hostname, port, and resolved addresses of every redirect target before following it. 7. In sitemap mode, accept only URLs belonging to the exact original site or a narrowly defined allowlist. Replace the substring test with parsed-host comparison. 8. In interactive and URL-list modes, reject off-domain absolute URLs unless the user explicitly enables a documented and appropriately isolated cross-domain mode. 9. Restrict destination ports to `80` and `443` unless other ports are explicitly required and approved. 10. Enforce maximum response sizes and streaming download limits to reduce denial-of-service exposure. 11. Run the generator in a network-isolated environment that cannot reach cloud metadata services, loopback administration endpoints, or sensitive private networks. 12. Return explicit validation errors rather than suppressing all exceptions with a bare `except`, so rejected or suspicious destinations can be audited.