T09 · Insecure Skill Coding Practices
- Location
scrapling_tool.py:24- Finding
Unrestricted URL Fetching Enables Server-Side Request Forgery
- Content
View full analysis
Vulnerability Details
File Location:
scrapling_tool.py:24,scrapling_tool.py:42-44,scrapling_tool.py:61, andscrapling_tool.py:78
Vulnerability Type: Server-Side Request Forgery through unrestricted user-controlled URLs
Risk Level: MediumVulnerable Code
python def scrape_basic(url, selector=None): """基础抓取""" try: page = Fetcher.get(url)python def scrape_stealth(url, selector=None, solve_cloudflare=False): """隐身模式抓取（绕过Cloudflare）""" try: page = StealthyFetcher.fetch(url, headless=True, solve_cloudflare=solve_cloudflare)python def scrape_dynamic(url, selector=None, wait_for=None): """动态内容抓取（JavaScript渲染）""" try: page = DynamicFetcher.fetch(url, headless=True, network_idle=True)python parser.add_argument('url', help='Target URL to scrape')Technical Analysis
The command-line caller completely controls the URL passed to all three network fetchers. The implementation does not validate the URL scheme, hostname, resolved IP address, destination port, or redirect targets.
Consequently, a caller may instruct the process to connect to destinations that are normally inaccessible from the caller's own network context, including:
- Loopback services such as
127.0.0.1or::1. - RFC1918 private network services.
- Link-local addresses.
- Cloud instance metadata endpoints.
- Internal hostnames resolved by the host's DNS configuration.
DNS-based validation would not be sufficient by itself. A secure implementation must validate every resolved address and repeat the validation after redirects to prevent DNS rebinding and redirect-based bypasses.
In basic mode, a request without a selector stores the response body in
result["text"]. JSON output then serializes the complete result, allowing accessible inter ...[truncated 1638 chars]- Loopback services such as
- Remediation
View remediation
Remediation Suggestions
- Parse URLs with a standards-compliant URL parser and permit only explicitly required schemes, normally
httpsand optionallyhttp. - Reject URLs containing credentials, malformed hostnames, unsupported ports, or ambiguous numeric IP representations.
- Resolve the hostname before connecting and reject every address classified as loopback, private, link-local, multicast, reserved, unspecified, or otherwise non-global.
- Explicitly block cloud metadata addresses and hostnames, including relevant IPv4 and IPv6 forms.
- Disable automatic redirects or validate the scheme, hostname, port, and resolved addresses of every redirect target before following it.
- Protect against DNS rebinding by ensuring the validated address is the address used for the connection, or enforce the restriction through a trusted outbound proxy.
- Prefer an explicit destination allowlist when the expected scraping targets are known.
- Apply egress firewall rules so the process cannot reach loopback, private networks, metadata services, or sensitive management networks.
- Limit response size, request duration, and redirect count to reduce denial-of-service exposure.
- Avoid returning full response bodies by default, especially when requests can be initiated through an Agent or remotely accessible service.
- Add tests covering IPv4, IPv6, encoded IP addresses, redirects to private destinations, and DNS rebinding scenarios.
- Parse URLs with a standards-compliant URL parser and permit only explicitly required schemes, normally