T09 · Insecure Skill Coding Practices
Warning
- Location
- scripts/google_index.py:110
- Finding
- Unrestricted Recursive Sitemap Fetching Enables Server-Side Request Forgery<![CDATA[ ## Vulnerability Details **File Location**: `scripts/google_index.py`, lines 110–143 **Vulnerability Type**: Server-Side Request Forgery through unrestricted URLs and redirects **Risk Level**: Medium ### Vulnerable Code ```python def fetch_sitemap_urls(sitemap_url: str) -> list[str]: """Fetch and parse a sitemap XML, returning all <loc> URLs. Handles both regular sitemaps and sitemap index files. """ import httpx urls: list[str] = [] try: resp = httpx.get(sitemap_url, timeout=30, follow_redirects=True) resp.raise_for_status() except Exception as e: print(f"Error fetching sitemap {sitemap_url}: {e}", file=sys.stderr) return urls try: root = ET.fromstring(resp.content) except ET.ParseError as e: print(f"Error parsing sitemap XML: {e}", file=sys.stderr) return urls # Strip namespace for easier parsing ns = "" if root.tag.startswith("{"): ns = root.tag.split("}")[0] + "}" # Check if it's a sitemap index sitemap_tags = root.findall(f"{ns}sitemap") if sitemap_tags: # It's a sitemap index — recurse into each child sitemap for sm in sitemap_tags: loc = sm.find(f"{ns}loc") if loc is not None and loc.text: child_urls = fetch_sitemap_urls(loc.text.strip()) urls.extend(child_urls) ``` The initial sitemap URL is supplied directly through the CLI without validation: ```python auto_parser.add_argument( "--sitemap", "-s", required=True, help="Sitemap URL to fetch (e.g. https://example.com/sitemap.xml)", ) ``` ### Technical Analysis The application makes HTTP requests to a caller-controlled sitemap URL and follows redirects automatically. It does not validate: - The URL scheme. - The destination hostname. - Resolved IPv4 or IPv6 addresses. - Redirect destinations. - Child sitemap URLs extracted from sitemap-index documents. - Whether child sitemaps rema ...[truncated 2399 chars]
- Remediation
- <![CDATA[ ## Remediation Suggestions 1. Permit only `https` URLs unless insecure HTTP support is explicitly required. 2. Resolve every hostname before connecting and reject all loopback, private, link-local, multicast, unspecified, reserved, and documentation address ranges for both IPv4 and IPv6. 3. Disable automatic redirects or validate every redirect target before following it. 4. Apply the same checks to every child sitemap URL, not only the initial CLI value. 5. Prefer requiring child sitemaps to use the same scheme and registrable domain as the initial sitemap. 6. Protect against DNS rebinding by ensuring the validated address is the one used for the connection and by validating all resolved addresses. 7. Define strict maximums for recursion depth, sitemap count, response size, URL count, and total fetch time. 8. Consider exposing an explicit hostname allowlist for automated deployments. 9. Avoid submitting private, local, credential-bearing, or otherwise non-public URLs to Google. 10. Return a clear validation failure rather than attempting a request when a destination is prohibited. ]]>