T09 · Insecure Skill Coding Practices
- Location
scripts/fetch_webpage.py:31- Finding
Unrestricted URL Fetching Enables Server-Side Request Forgery
- Content
View full analysis
Vulnerability Details
File Location:
scripts/fetch_webpage.py:31-44
Vulnerability Type: Server-Side Request Forgery (SSRF)
Risk Level: HighVulnerable Code
python def fetch_webpage(url: str, timeout: int = 30) -> dict: """ Fetch webpage content and extract structured information. Args: url: Target webpage URL timeout: Request timeout in seconds Returns: A dictionary containing structured webpage information """ try: # Send HTTP request headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ' 'AppleWebKit/537.36 (KHTML, like Gecko) ' 'Chrome/120.0.0.0 Safari/537.36' } response = requests.get(url, headers=headers, timeout=timeout) response.raise_for_status()The URL is supplied directly through the command-line interface and passed to the vulnerable function without validation:
python parser.add_argument('--url', required=True, help='Target webpage URL') parser.add_argument('--timeout', type=int, default=30, help='Request timeout in seconds') parser.add_argument('--output', choices=['json', 'text'], default='json', help='Output format') args = parser.parse_args() result = fetch_webpage(args.url, args.timeout)Technical Analysis
The fetcher accepts an arbitrary user-controlled URL and passes it directly to
requests.get. It does not validate:- The URL scheme
- The destination hostname
- The resolved IPv4 or IPv6 address
- The destination port
- Whether the address belongs to a loopback, private, link-local, reserved, or unspecified range
- Redirect destinations
Because
requests.getfollows redirects by default, validation of only the initial URL would not be sufficient. A public URL could redirect the request to an internal service.The script subsequently parses and exposes the response body through its structured result:
`` ...[truncated 2539 chars]
- Remediation
View remediation
Remediation Suggestions
-
Restrict URL schemes
- Parse the URL before use.
- Permit only
httpandhttps. - Reject URLs containing embedded credentials.
- Reject malformed or ambiguous host representations.
-
Block non-public destinations
- Resolve all destination hostnames before connecting.
- Reject every resolved IPv4 and IPv6 address classified as loopback, private, link-local, multicast, reserved, or unspecified.
- Explicitly block cloud metadata destinations, including
169.254.169.254and relevant metadata hostnames. - Revalidate immediately before the connection to reduce DNS-rebinding risk.
-
Validate every redirect
- Prefer disabling automatic redirects with
allow_redirects=False. - If redirects are required, impose a small redirect limit and repeat complete scheme, hostname, port, and resolved-address validation for every redirect target.
- Prefer disabling automatic redirects with
-
Restrict ports and destinations
- Permit only the ports necessary for public webpage retrieval, normally 80 and 443.
- Where practical, use a domain allowlist or a controlled outbound proxy that enforces public-network-only access.
-
Limit response resources
- Use streaming requests.
- Enforce a strict maximum response size before buffering or parsing.
- Apply connection and read timeouts separately.
- Reject unsupported content types before parsing.
- Limit the amount of extracted text, links, headings, and image metadata.
-
Harden the execution environment
- Apply outbound firewall rules that prevent access to private networks and metadata services.
- Run the fetcher with minimal privileges and no unnecessary cloud identity.
- Avoid exposing raw internal error details to untrusted callers.
-
Add security tests
- Test loopback, RFC1918, link-local, IPv4-mapped IPv6, encoded IP, DNS-rebinding, and redirect-to-private-address cases.
- Verify that oversized responses and redirect loops are safely rejected.
-
