T09 · Insecure Skill Coding Practices
Error
- Location
- scripts/fetch_detail.py:98
- Finding
- Server-Side Request Forgery Through Unvalidated News URLs## Vulnerability Details **File Location**: `scripts/fetch_detail.py:98-103` **Related Validation Locations**: `scripts/search_news.py:201-202`, `scripts/search_news.py:299-302` **Vulnerability Type**: Server-Side Request Forgery (SSRF) **Risk Level**: High ### Vulnerable Code The search workflow permits a URL when either its host is allowed or its associated text contains a configured country term: ```python if not host_matches(item["url"], allowed_hosts) and not country_terms_match(title, snippet, country_terms): continue ``` The fallback search uses the same alternative condition: ```python fallback_results = [ r for r in fallback_web_search(keyword, country, engine) if host_matches(r.get("url", ""), allowed_hosts) or country_terms_match(r.get("title", ""), "", country_terms) ] ``` The detail fetcher subsequently accepts any URL beginning with `http` and requests it without validating its destination: ```python url = news_item.get("url", "") if not url or not url.startswith("http"): return {**news_item, "content": "", "date": None, "sentiment": "中性", "type": "行业动态"} try: resp = curl_requests.get(url, timeout=8, impersonate="chrome") content = extract_article_content(resp.text) date = extract_article_date(resp.text) ``` ### Technical Analysis The URL trust boundary is enforced incorrectly. In `search_news.py`, matching a country-related term in attacker-influenced article text is treated as an alternative to host authorization. Consequently, an untrusted host can pass filtering merely by including an accepted country term in its title or snippet. The final network sink in `fetch_detail.py` only verifies that the raw string starts with `http`. It does not: - Parse and restrict the scheme to exactly `http` or `https`. - Require the destination to match an approved hostname. - Resolve the hostname and reject private, loopback, link-local, reserved, or unspecified addresses. - Validate redirect destinations. - Block cloud ...[truncated 2044 chars]
- Remediation
- ## Remediation Suggestions 1. **Parse URLs before use** - Use `urllib.parse.urlsplit`. - Permit only exact `http` and `https` schemes. - Reject malformed URLs, embedded credentials, missing hostnames, and unexpected ports where possible. 2. **Enforce host authorization at the network sink** - Require every detail URL to match an explicit hostname allowlist. - Do not treat article title, snippet, country, or keyword matching as authorization to contact a host. - Repeat validation immediately before every request, including URLs loaded from JSON. 3. **Reject non-public destinations** - Resolve all A and AAAA records. - Use Python's `ipaddress` module to reject loopback, private, link-local, multicast, reserved, and unspecified addresses. - Reject hostnames resolving to any prohibited address, including mixed public/private DNS results. - Explicitly block metadata destinations, including `169.254.169.254` and equivalent platform-specific hostnames. 4. **Secure redirect handling** - Disable redirects unless required. - If redirects are enabled, validate every redirect target with the same scheme, hostname, DNS, and IP-address policy before following it. - Set a low redirect limit. 5. **Reduce request and response exposure** - Apply strict response-size limits while streaming the body. - Permit only expected content types. - Retain connection and read timeouts. - Run the fetcher in a network-restricted environment that cannot access internal networks or metadata services. 6. **Separate relevance filtering from security controls** - Country-term matching may remain a relevance signal, but it must never override destination security requirements. - Reject or quarantine results whose destinations are not explicitly trusted.