T09 · Insecure Skill Coding Practices
Error
- Location
scripts/scrape.py:52- Finding
Unrestricted URL Fetching Enables Server-Side Request Forgery
- Content
View full analysis
Vulnerability Details
File Location:
scripts/scrape.py, lines 52-75
Vulnerability Type: Server-Side Request Forgery (SSRF)
Risk Level: HighVulnerable Code
python def fetch(url, use_js=False, wait=None, timeout=20): if use_js: return fetch_playwright(url, wait, timeout) headers = {"User-Agent": random.choice(USER_AGENTS), "Referer": "https://www.google.com/"} resp = requests.get(url, headers=headers, timeout=timeout) resp.raise_for_status() return resp.text def fetch_playwright(url, wait, timeout): try: from playwright.sync_api import sync_playwright except ImportError: print("❌ 需要 playwright：pip install playwright && playwright install chromium", file=sys.stderr) sys.exit(1) with sync_playwright() as pw: browser = pw.chromium.launch(headless=True) page = browser.new_page() page.goto(url, timeout=timeout * 1000) if wait: page.wait_for_selector(wait, timeout=timeout * 1000) html = page.content() browser.close() return htmlThe untrusted URL reaches these functions through the command-line interface:
python ap.add_argument("url") ... html = fetch(args.url, args.js, args.wait, args.timeout)The snapshot workflow also passes its URL directly to the same fetch function:
python def build_records(url, fields, use_js, wait, timeout): html = fetch(url, use_js, wait, timeout)Technical Analysis
The application accepts a user-controlled URL and passes it directly to either
requests.get()or Playwright'spage.goto(). It does not validate:- The URL scheme.
- The destination hostname.
- The resolved IPv4 or IPv6 address.
- Whether the destination is loopback, link-local, private, reserved, or multicast.
- Whether redirects lead to a prohibited address.
- Whether the destination is a cloud ...[truncated 2305 chars]
- Remediation
View remediation
Remediation Suggestions
- Accept only explicitly supported schemes, preferably
httpandhttps. Rejectfile,ftp,data, browser-internal, and all other schemes. - Parse URLs with a strict parser and reject missing hosts, embedded credentials, malformed ports, and ambiguous host representations.
- Resolve all destination addresses before connecting. Reject IPv4 and IPv6 addresses in loopback, private, link-local, reserved, unspecified, multicast, and carrier-grade NAT ranges according to the deployment policy.
- Explicitly block cloud metadata destinations and hostnames, including link-local metadata addresses.
- Disable automatic redirects or validate every redirect target using the same scheme, hostname, DNS, and IP checks before following it.
- Defend against DNS rebinding by connecting only to the validated resolved address while preserving the intended HTTP host and TLS verification semantics.
- Prefer an allowlist of approved public monitoring domains when the set of targets is known.
- Apply outbound firewall or proxy restrictions so the process cannot reach internal networks or metadata services even if application validation fails.
- In Playwright mode, intercept all browser requests and reject prohibited navigation, iframe, script, image, stylesheet, XHR, fetch, and WebSocket destinations.
- Set response-size, redirect-count, and overall request limits to reduce resource-exhaustion risks.
- Add tests covering decimal, hexadecimal, octal, IPv6, IPv4-mapped IPv6, hostname aliases, redirect chains, and DNS-rebinding cases.
- Accept only explicitly supported schemes, preferably