T09 · Insecure Skill Coding Practices
Error
- Location
scripts/fetch_page.py:21- Finding
Unrestricted URL Fetching Enables Server-Side Request Forgery
- Content
View full analysis
Vulnerability Details
File Location:
scripts/fetch_page.py:21-29;scripts/deep_research.py:35-45,99-100
Vulnerability Type: Server-Side Request Forgery (SSRF) through unrestricted outbound HTTP requests
Risk Level: HighVulnerable Code
scripts/fetch_page.py:21-29:python p.add_argument("url") p.add_argument("--searxng-url", default=None) # accepted for consistency p.add_argument("--max-chars", type=int, default=5000) args = p.parse_args() timeout = int(os.environ.get("SEARXNG_TIMEOUT", "20")) r = requests.get(args.url, timeout=timeout, headers={"User-Agent": "wsearch-fetch/1.0"}) r.raise_for_status()scripts/deep_research.py:35-45:python def _fetch(url, max_chars=2500): try: r = requests.get(url, timeout=15, headers={"User-Agent": "wsearch-fetch/1.0"}) r.raise_for_status() ct = (r.headers.get("content-type") or "").lower() if HAS_BS4 and ("text/html" in ct or "<html" in r.text[:200].lower()): text = BeautifulSoup(r.text, "lxml").get_text("\n") text = re.sub(r"\s+", " ", unescape(text)).strip() else: text = r.text.strip() return text[:max_chars]scripts/deep_research.py:99-100:python ranked = _rank(all_results) fetched = [(r, _fetch(r["url"], args.max_chars)) for r in ranked[:args.fetch_top_pages]]Technical Analysis
Both page-fetching paths pass an insufficiently validated URL directly to
requests.get(). The implementation does not:- Restrict the URL scheme to HTTP or HTTPS.
- Resolve hostnames and reject loopback, private, link-local, reserved, multicast, or unspecified addresses.
- Block cloud metadata services and equivalent internal endpoints.
- Disable redirects or revalidate the destination of every redirect.
- Apply a streamed response-size limit before downloading the response body.
In `fetch_pa ...[truncated 2150 chars]
- Remediation
View remediation
Remediation Suggestions
- Parse each destination with
urllib.parse.urlsplit()and permit only explicithttpandhttpsschemes. - Reject URLs containing ambiguous host syntax, embedded credentials, malformed ports, or missing hostnames.
- Resolve all destination addresses before connecting. Reject every address classified by
ipaddressas private, loopback, link-local, multicast, reserved, or unspecified. - Explicitly deny cloud metadata endpoints and provider-specific metadata hostnames.
- Disable automatic redirects with
allow_redirects=False, or manually follow a small number of redirects while applying the same scheme, hostname, DNS, and IP validation to every hop. - Account for DNS rebinding by ensuring the validated address is the address used for the connection, or by enforcing equivalent egress controls outside the application.
- Fetch responses with
stream=True, reject inappropriate content types, enforce a maximumContent-Length, and stop reading after a fixed byte limit. - For
deep_research.py, consider limiting automatic fetching to an approved public-domain allowlist or requiring confirmation before fetching untrusted result URLs. - Apply host-level egress filtering so the Skill process cannot reach loopback, private networks, or metadata ranges unless explicitly required.
- Parse each destination with