T09 · Insecure Skill Coding Practices
Error
- Location
search_fetch.py:91- Finding
Unrestricted URL Fetching Enables Server-Side Request Forgery
- Content
View full analysis
Vulnerability Details
File Location:
search_fetch.py:91-112
Related Locations:search_fetch.py:115-160,search_fetch.py:171-190,search_fetch.py:230-247
Vulnerability Type: Server-Side Request Forgery through unvalidated URLs
Risk Level: HighVulnerable Code
python def _fetch_direct(url: str, timeout: int, max_chars: int) -> dict: import requests from bs4 import BeautifulSoup session = requests.Session() session.headers.update(_BROWSER_HEADERS) # Visit root domain first to pick up cookies (helps with some anti-bot) try: from urllib.parse import urlparse root = f"{urlparse(url).scheme}://{urlparse(url).netloc}/" if root != url: session.get(root, timeout=timeout, allow_redirects=True) time.sleep(0.5) except Exception: pass resp = session.get(url, timeout=timeout, allow_redirects=True) resp.raise_for_status() resp.encoding = resp.apparent_encoding or "utf-8" soup = BeautifulSoup(resp.text, "lxml") title = soup.title.string.strip() if soup.title else "" text = _extract_text(resp.text, url) return _build_result(url, resp.status_code, title, text, "direct", max_chars)The URL originates directly from a positional command-line argument:
python fp.add_argument("url")Third-party strategies also forward the supplied URL externally:
python def _fetch_jina(url: str, timeout: int, max_chars: int) -> dict: """Use r.jina.ai as a reader proxy — free, no API key, returns clean markdown.""" import requests jina_url = f"https://r.jina.ai/{url}" headers = { "User-Agent": _BROWSER_HEADERS["User-Agent"], "Accept": "text/plain, text/markdown, */*", "X-Return-Format": "markdown", } resp = requests.get(jina_url, headers=headers, timeout=timeout)Technical Analysis
Th ...[truncated 2731 chars]
- Remediation
View remediation
Remediation Suggestions
- Restrict accepted schemes to
httpsand, only if required,http. Reject malformed URLs, non-network schemes, embedded credentials, and ambiguous host representations. - Resolve the hostname before connecting and reject every IPv4 and IPv6 address classified as loopback, private, link-local, multicast, reserved, unspecified, or otherwise non-public.
- Explicitly block cloud metadata destinations, including link-local metadata addresses and platform-specific metadata hostnames.
- Disable automatic redirects. Follow redirects manually only after validating the scheme, hostname, port, and resolved addresses of every destination.
- Mitigate DNS rebinding by connecting to a validated resolved address while preserving the intended HTTP
Hostvalue and TLS hostname verification, or by using an outbound proxy that enforces public-address policy. - Restrict destination ports to a minimal allowlist, normally 80 and 443.
- Remove the root-origin preflight request unless it is essential. If retained, apply the same destination checks before issuing it.
- Add an optional hostname allowlist for high-assurance use cases such as government, regulatory, court, and company domains.
- Require explicit user confirmation before sending a URL to Jina Reader or the Internet Archive. Reject private and sensitive URLs even when confirmation is provided.
- Avoid forwarding URLs containing credentials, access tokens, signed query strings, fragments, or other sensitive parameters to third parties.
- Add tests covering decimal and hexadecimal IP representations, IPv4-mapped IPv6, redirects to private addresses, DNS rebinding scenarios, credential-bearing URLs, and metadata endpoints.
- Restrict accepted schemes to