T09 · Insecure Skill Coding Practices
- Location
scripts/collect_webpages.py:15- Finding
Unrestricted URL Fetching Enables Server-Side Request Forgery
- Content
View full analysis
Vulnerability Details
File Location:
scripts/collect_webpages.py:15-22,scripts/collect_webpages.py:105-114, andscripts/collect_feeds.py:34-45
Vulnerability Type: Server-Side Request Forgery caused by insufficient URL and destination validation
Risk Level: HighVulnerable Code
scripts/collect_webpages.py:15-22:python def fetch_webpage(url: str) -> Optional[str]: """抓取网页HTML内容""" try: import requests response = requests.get(url, timeout=30, headers={ 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36' }) response.raise_for_status() return response.textscripts/collect_webpages.py:105-114:python for url in urls: if not url.startswith(('http://', 'https://')): print(f"警告: 跳过无效URL: {url}") continue print(f"正在抓取: {url}") html = fetch_webpage(url) if html: content = extract_content(html, url)scripts/collect_feeds.py:34-45:python url = source.get('url') name = source.get('name', '未知来源') category = source.get('category', '未分类') if not url: print(f"警告: 来源 {name} 缺少URL配置") return [] try: feed = parse(url) if feed.bozo:Technical Analysis
Both collectors initiate outbound requests to user-controlled URLs. The webpage collector only verifies that a string begins with
http://orhttps://. This does not prevent access to loopback addresses, private networks, link-local services, IPv6-local destinations, or cloud instance metadata endpoints.The RSS collector passes the configured URL directly to
feedparser.parse()without even applying the webpage collector's limited scheme check. Depending on the parser's supported behavior and environment, this allows arbitrary configured network destinations to be queried.Redirects are not explicitly disabled or validated. The `request ...[truncated 1729 chars]
- Remediation
View remediation
Remediation Suggestions
- Permit only explicitly required schemes, normally HTTPS and, where necessary, HTTP.
- Parse URLs with a standards-compliant URL parser rather than using prefix checks.
- Resolve hostnames before connecting and reject all loopback, private, link-local, multicast, reserved, unspecified, and metadata-service address ranges for both IPv4 and IPv6.
- Maintain an allowlist of approved news and feed domains where operationally possible.
- Disable redirects by default. If redirects are required, revalidate the scheme, hostname, port, and resolved destination address at every redirect hop.
- Defend against DNS rebinding by ensuring that the validated address is the address used for the connection.
- Restrict destination ports to those genuinely needed, such as 80 and 443.
- Apply the same validation policy to both
requests.get()andfeedparser.parse(). - Run collectors in a network sandbox that cannot access localhost services, private networks, or cloud metadata endpoints.
- Avoid including complete sensitive response bodies or detailed internal connection errors in downstream reports.
