T09 · Insecure Skill Coding Practices
Error
- Location
- scripts/web_search.py:276
- Finding
- Server-Side Request Forgery Through Arbitrary URL Crawling<![CDATA[ ## Vulnerability Details **File Location**: `scripts/web_search.py:276-278`, with insufficient validation at `scripts/web_search.py:335-352` and the externally reachable call path at `scripts/web_search.py:475-481` **Vulnerability Type**: Server-Side Request Forgery (SSRF) **Risk Level**: High ### Vulnerable Code ```python async def crawl_page_async(url: str) -> Dict[str, Any]: """Asynchronously crawl webpage content.""" if not HAS_CRAWL4AI: return { 'success': False, 'message': 'crawl4ai is not installed; unable to crawl webpages' } try: async with AsyncWebCrawler() as crawler: result = await crawler.arun(url=url) ``` The validation applied before reaching this sink is limited to the URL scheme and length: ```python def validate_url(url: str) -> tuple: """Validate URL.""" if not url: return False, 'URL cannot be empty' if not isinstance(url, str): return False, 'URL must be a string' url = url.strip() if len(url) == 0: return False, 'URL cannot be empty' if not url.startswith(('http://', 'https://')): return False, 'URL must begin with http:// or https://' if len(url) > 2000: return False, 'URL length cannot exceed 2000 characters' return True, '' ``` The crawl action passes the validated but otherwise unrestricted URL to the crawler: ```python elif action == 'crawl': url = kwargs.get('url', '') is_valid, error_msg = validate_url(url) if not is_valid: return {'success': False, 'message': error_msg} return crawl_page(url.strip()) ``` ### Technical Analysis The `crawl` action accepts an attacker-controlled HTTP or HTTPS URL and passes it to `AsyncWebCrawler.arun()`. Validation only confirms that the value is a string, uses an HTTP-based scheme, and is no longer than 2,000 characters. It does not reject: - IPv4 or IPv6 loopback destinations - RFC 1918 private networks - Lin ...[truncated 2008 chars]
- Remediation
- <![CDATA[ ## Remediation Suggestions 1. Parse URLs with a standards-compliant URL parser and allow only explicitly required schemes. 2. Resolve the hostname before connecting and reject every address classified as loopback, private, link-local, reserved, multicast, or unspecified for both IPv4 and IPv6. 3. Revalidate the resolved destination immediately before connection to reduce time-of-check/time-of-use and DNS-rebinding risks. 4. Disable redirects when possible. If redirects are required, validate the scheme, hostname, resolved addresses, and port of every redirect target before following it. 5. Prefer an explicit allowlist of approved external domains and ports instead of attempting to block known-dangerous ranges. 6. Block cloud metadata endpoints at both application and network layers. 7. Run the crawler in a network-isolated sandbox without access to internal networks, host services, or cloud metadata. 8. Apply request timeouts, response-size limits, concurrency limits, and rate limits to reduce scanning and denial-of-service abuse. 9. Add tests covering decimal, hexadecimal, octal, shortened, and IPv4-mapped IPv6 address representations, redirects, user-information URL syntax, and DNS rebinding scenarios. ]]>