T09 · Insecure Skill Coding Practices
- Location
FinancialDataCollection/src/fiscal_crawler.py:90- Finding
Unrestricted outbound requests permit server-side request forgery
- Content
View full analysis
Vulnerability Details
File Location:
FinancialDataCollection/src/fiscal_crawler.py, lines 90-112
Vulnerability Type: Server-Side Request Forgery through unvalidated links and redirects
Risk Level: MediumVulnerable Code
python def _fetch_html(self, url: str) -> str: response = self.session.get(url, timeout=self.timeout) response.raise_for_status() response.encoding = response.apparent_encoding or response.encoding return response.textpython href = anchor.get("href", "").strip() if not href: continue absolute_url = urljoin(page_url, href) title = normalize_text(anchor.get_text(" ", strip=True)) if "财政收支情况" not in title: continue if absolute_url in yielded: continue yielded.add(absolute_url) yield {"title": title, "url": absolute_url}Technical Analysis
The crawler starts from a fixed Ministry of Finance URL, but it does not verify the scheme, hostname, port, or resolved IP address of links extracted from listing pages. Python's
urljoinaccepts absolute URLs, so an absolute attacker-controlledhrefreplaces the trusted base URL completely.The resulting URL is passed to
requests.Session.get()without destination validation. Requests also follows HTTP redirects by default, and the redirect destinations are not validated. Consequently, either a malicious listing link or a redirect from a nominally trusted URL can cause the process to issue a GET request to an arbitrary HTTP or HTTPS endpoint.The title filter does not provide a security boundary. A party able to modify listing content can supply a title matching the expected fiscal-report pattern while setting its link to an internal destination.
Attack Path
- An attacker gains the ability to alter a source listing page, compromises an upstream content-management component, or causes a trusted document endpoint to return an attacker-selected redirect.
- For a listing- ...[truncated 1542 chars]
- Remediation
View remediation
Remediation Suggestions
- Allow only
httpsURLs with the exact expected hostname, such aswww.mof.gov.cn. - Reject URLs containing user information, unexpected ports, fragments where inappropriate, or non-HTTP schemes.
- Resolve the destination and reject loopback, private, link-local, multicast, reserved, and unspecified IP ranges. Repeat this validation for every redirect target to address DNS rebinding and redirect-based bypasses.
- Disable automatic redirects with
allow_redirects=False, then follow redirects manually only after validating eachLocationvalue. - Apply the same validation to listing-page and document-page requests.
- Enforce outbound firewall rules so the runtime cannot access metadata services, loopback services, or internal administrative networks.
- Add tests for absolute external links, protocol-relative URLs, private IP literals, IPv6 loopback, encoded host representations, DNS results resolving to private addresses, and redirect chains.
A minimal hostname check should resemble:
python from urllib.parse import urlparse ALLOWED_HOSTS = {"www.mof.gov.cn"} def validate_url(url: str) -> None: parsed = urlparse(url) if parsed.scheme != "https" or parsed.hostname not in ALLOWED_HOSTS: raise ValueError("Unapproved crawl destination") if parsed.port not in (None, 443): raise ValueError("Unapproved destination port")This check should be supplemented with resolved-IP validation and manual redirect handling.
- Allow only