T09 · Insecure Skill Coding Practices
- Location
scripts/fetch_detail.py:98- Finding
Server-Side Request Forgery Through Unvalidated News URLs
- Content
View full analysis
Vulnerability Details
File Location:
scripts/fetch_detail.py:98-103
Related Validation Locations:scripts/search_news.py:201-202,scripts/search_news.py:299-302
Vulnerability Type: Server-Side Request Forgery (SSRF)
Risk Level: HighVulnerable Code
The search workflow permits a URL when either its host is allowed or its associated text contains a configured country term:
python if not host_matches(item["url"], allowed_hosts) and not country_terms_match(title, snippet, country_terms): continueThe fallback search uses the same alternative condition:
python fallback_results = [ r for r in fallback_web_search(keyword, country, engine) if host_matches(r.get("url", ""), allowed_hosts) or country_terms_match(r.get("title", ""), "", country_terms) ]The detail fetcher subsequently accepts any URL beginning with
httpand requests it without validating its destination:python url = news_item.get("url", "") if not url or not url.startswith("http"): return {**news_item, "content": "", "date": None, "sentiment": "中性", "type": "行业动态"} try: resp = curl_requests.get(url, timeout=8, impersonate="chrome") content = extract_article_content(resp.text) date = extract_article_date(resp.text)Technical Analysis
The URL trust boundary is enforced incorrectly. In
search_news.py, matching a country-related term in attacker-influenced article text is treated as an alternative to host authorization. Consequently, an untrusted host can pass filtering merely by including an accepted country term in its title or snippet.The final network sink in
fetch_detail.pyonly verifies that the raw string starts withhttp. It does not:- Parse and restrict the scheme to exactly
httporhttps. - Require the destination to match an approved hostname.
- Resolve the hostname and reject private, loopback, link-local, reserved, or unspecified addresses.
- Validate redirect destinations.
- Block cloud ...[truncated 2044 chars]
- Parse and restrict the scheme to exactly
- Remediation
View remediation
Remediation Suggestions
-
Parse URLs before use
- Use
urllib.parse.urlsplit. - Permit only exact
httpandhttpsschemes. - Reject malformed URLs, embedded credentials, missing hostnames, and unexpected ports where possible.
- Use
-
Enforce host authorization at the network sink
- Require every detail URL to match an explicit hostname allowlist.
- Do not treat article title, snippet, country, or keyword matching as authorization to contact a host.
- Repeat validation immediately before every request, including URLs loaded from JSON.
-
Reject non-public destinations
- Resolve all A and AAAA records.
- Use Python's
ipaddressmodule to reject loopback, private, link-local, multicast, reserved, and unspecified addresses. - Reject hostnames resolving to any prohibited address, including mixed public/private DNS results.
- Explicitly block metadata destinations, including
169.254.169.254and equivalent platform-specific hostnames.
-
Secure redirect handling
- Disable redirects unless required.
- If redirects are enabled, validate every redirect target with the same scheme, hostname, DNS, and IP-address policy before following it.
- Set a low redirect limit.
-
Reduce request and response exposure
- Apply strict response-size limits while streaming the body.
- Permit only expected content types.
- Retain connection and read timeouts.
- Run the fetcher in a network-restricted environment that cannot access internal networks or metadata services.
-
Separate relevance filtering from security controls
- Country-term matching may remain a relevance signal, but it must never override destination security requirements.
- Reject or quarantine results whose destinations are not explicitly trusted.
-