T09 · Insecure Skill Coding Practices
Error
- Location
scripts/scraper.py:37- Finding
Unrestricted URL Fetching Enables SSRF and Local Resource Access
- Content
View full analysis
str: """Fetch URL content with proper headers and error handling.""" req = Request(url, headers={ "User-Agent": USER_AGENT, "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-US,en;q=0.5", }) try: with urlopen(req, timeout=timeout) as resp: encoding = resp.headers.get_content_charset() or "utf-8" return resp.read().decode(encoding, errors="replace") except HTTPError as e: print(f"HTTP Error {e.code}: {e.reason} — {url}", file=sys.stderr) sys.exit(1) except URLError as e: print(f"Connection Error: {e.reason} — {url}", file=sys.stderr) sys.exit(1) ``` Pagination links obtained from remote HTML are also subsequently passed to this fetching function: ```python next_link = soup.find("a", rel="next") if next_link and next_link.get("href"): return urljoin(current_url, next_link["href"]) for pattern in [".next a", "a.next", ".pagination .next", "[aria-label='Next']", "a[aria-label='Next page']", ".page-next a"]: el = soup.select_one(pattern) if el and el.get("href"): return urljoin(current_url, el["href"]) ``` ### Technical Analysis The scraper passes a user-supplied URL directly to `urllib.request.urlopen()` without validating: - The URI scheme, such as restricting requests to `http` and `https`. - Whether the resolved address belongs to a loopback, private, link-local, multicast, or reserved network. - Whether redirects lead to prohibited addresses. - Whether a pagination URL remains on an approved origin. - Whether DNS resolution changes between validation and connection. - The maximum permitted resp ...[truncated 2033 chars]- Remediation
View remediation