T09 · Insecure Skill Coding Practices
- Location
scripts/parse_sitemap.py:20- Finding
Unrestricted Recursive Sitemap Fetching Enables Server-Side Request Forgery
- Content
View full analysis
Vulnerability Details
File Location:
scripts/parse_sitemap.py, lines 20-39
Vulnerability Type: Server-Side Request Forgery (SSRF)
Risk Level: HighVulnerable Code
python def read_xml(source: str) -> bytes: if source.startswith("http://") or source.startswith("https://"): ctx = ssl.create_default_context() with urllib.request.urlopen(source, context=ctx, timeout=30) as resp: return resp.read() return Path(source).read_bytes() def parse_urls(source: str) -> list[str]: seen_sitemaps = set() urls: list[str] = [] queue = deque([source]) while queue: current = queue.popleft() if current in seen_sitemaps: continue seen_sitemaps.add(current) data = read_xml(current)Nested sitemap locations are subsequently placed into the same unrestricted processing queue:
python if tag == "sitemapindex": for loc in root.findall("sm:sitemap/sm:loc", NS): if loc.text: queue.append(loc.text.strip())Technical Analysis
The parser accepts attacker-influenced HTTP and HTTPS URLs and passes them directly to
urllib.request.urlopen. It does not validate the destination hostname, resolved IP address, port, URL credentials, or redirect target. It also does not require nested sitemap URLs to remain on the origin of the initial sitemap.Because child locations extracted from a remotely retrieved sitemap are recursively fetched, a malicious sitemap can direct the Agent environment to make HTTP requests to loopback addresses, private network ranges, link-local services, or other destinations that are inaccessible to the attacker directly. Python's URL opener also follows HTTP redirects by default, so validating only an initial URL in future changes would remain insufficient unless every redirect target is checked.
Plain HTTP is acce ...[truncated 1715 chars]
- Remediation
View remediation
Remediation Suggestions
- Require HTTPS for all remote sitemap sources and reject plaintext HTTP URLs.
- Parse URLs with
urllib.parse.urlsplitand allow only explicitly supported schemes. - Establish an explicit hostname allowlist or require nested sitemap URLs to use the same approved origin as the initial sitemap.
- Resolve each destination hostname and reject loopback, private, link-local, multicast, unspecified, reserved, and other non-public address ranges for both IPv4 and IPv6.
- Repeat destination validation after DNS resolution and for every redirect target to prevent redirect-based and DNS-rebinding bypasses.
- Disable automatic redirects or implement a bounded redirect handler that validates every hop.
- Reject URLs containing embedded credentials and restrict destination ports to an approved set, normally TCP 443.
- Consider accepting only site owners' preapproved sitemap origins rather than arbitrary user-provided URLs.
- Log rejected destinations without exposing credentials or sensitive URL parameters.
- Add tests for loopback addresses, private IPv4 and IPv6 ranges, mixed-format IP addresses, redirects to internal hosts, cross-origin child sitemaps, and DNS-rebinding scenarios.
