T09 · Insecure Skill Coding Practices
Error
- Location
- scripts/fetch_site_content.py:27
- Finding
- Unvalidated URL Fetching Enables Server-Side Request Forgery<![CDATA[ ## Vulnerability Details **File Location**: `scripts/fetch_site_content.py`, lines 27–35 and 82–98 **Vulnerability Type**: Server-Side Request Forgery (SSRF) through an unrestricted browser crawler **Risk Level**: High ### Vulnerable Code The crawler accepts an arbitrary URL and navigates to it without validating its scheme, hostname, resolved IP address, or redirect destination: ```python async def _fetch_with_crawl4ai(url): """Use crawl4ai to fetch the page asynchronously.""" from crawl4ai import AsyncWebCrawler, CrawlerRunConfig config = CrawlerRunConfig(magic=True) async with AsyncWebCrawler() as crawler: result = await crawler.arun(url=url, config=config) if result.success: return {'url': url, 'content': result.markdown, 'error': None} else: return {'url': url, 'content': '', 'error': result.error_message} ``` The URL originates directly from a supplied JSON file: ```python url = data.get('url', '') title = data.get('title', 'Unknown title') if not url: print(f"Error: file has no URL field: '{filepath}'") return print(f"Fetching: {title}") print(f"URL: {url}") # Fetch content result = fetch_content(url) if result.get('error'): print(f"Fetch failed: {result['error']}") ``` ### Technical Analysis The input JSON file is treated as trusted even though users or other processes can create or modify it. Its `url` field is passed directly to `crawler.arun()`. There are no controls that: - Restrict the URL to HTTP or HTTPS. - Reject embedded credentials or malformed hostnames. - Block loopback, private, link-local, reserved, multicast, or cloud metadata addresses. - Validate DNS resolution results. - Revalidate destinations after redirects. - Restrict requests to hosts obtained through the intended TopHub workflow. Because Crawl4AI operates through a browser, the resulting request originates from the machine running the Sk ...[truncated 1875 chars]
- Remediation
- <![CDATA[ ## Remediation Suggestions 1. Parse URLs with a standards-compliant parser and permit only explicit `http` and `https` schemes. 2. Reject URLs containing credentials, ambiguous host representations, malformed ports, or missing hostnames. 3. Resolve every hostname before navigation and reject all resolved loopback, private, link-local, multicast, unspecified, and reserved IPv4 and IPv6 addresses. 4. Explicitly block well-known cloud metadata destinations. 5. Revalidate every redirect destination and its resolved addresses before following it. 6. Protect against DNS rebinding by enforcing destination controls at connection time, preferably through an egress proxy or network policy rather than relying only on preflight DNS checks. 7. Consider an allowlist of domains supplied by the trusted TopHub workflow. If arbitrary public websites must remain supported, require explicit user confirmation for hosts outside a trusted set. 8. Run the crawler in a restricted environment with no access to internal networks, metadata services, sensitive local files, or unnecessary credentials. 9. Add tests covering loopback addresses, private IPv4 and IPv6 ranges, encoded IP forms, DNS aliases, redirects to private destinations, and unsupported schemes. ]]>