T09 · Insecure Skill Coding Practices
- Location
scripts/fetch:34- Finding
Unrestricted URL Fetching Enables Server-Side Request Forgery
- Content
View full analysis
Vulnerability Details
File Location:
scripts/fetch:34-45, 58;scripts/crawl:53-70, 82-83, 108-109, 132-134
Vulnerability Type: Server-Side Request Forgery (SSRF, CWE-918)
Risk Level: HighThe scripts accept arbitrary URLs and pass them to the network-capable Scrapling executable without validating the URL scheme, hostname, resolved IP address, or redirect destinations.
Vulnerable Code
In
scripts/fetch, the URL is accepted as a positional argument and passed directly to Scrapling:python def parse_args(): p = argparse.ArgumentParser(add_help=False) p.add_argument("url") p.add_argument("output_file", nargs="?", default=None) p.add_argument("--mode", choices=["get", "fetch", "stealthy"], default=None) p.add_argument("--selector", "-s", default=None) p.add_argument("--wait", type=int, default=0) p.add_argument("--wait-selector", default=None) p.add_argument("--solve-cloudflare", action="store_true") p.add_argument("--text", action="store_true") p.add_argument("--html", action="store_true") p.add_argument("--no-resources", action="store_true") p.add_argument("--timeout", type=int, default=30000) p.add_argument("--network-idle", action="store_true") p.add_argument("--proxy", default=None) return p.parse_args()python def build_cmd(mode, url, outfile, args): cmd = [SCRAPLING, "extract", mode, url, outfile]In
scripts/crawl, both a start URL and a file containing multiple URLs can supply unrestricted destinations:python def parse_args(): p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) p.add_argument("start_url", nargs="?", default=None) p.add_argument("--urls-file", default=None) p.add_argument("--depth", type=int, default=1) p.add_argument("--same-domain", action="store_true", default=True) p.add_argument ...[truncated 4830 chars]- Remediation
View remediation
Remediation Suggestions
Implement one shared URL-validation function and invoke it for every initial URL, URL-list entry, discovered crawl link, and redirect destination.
- Permit only explicitly required schemes, normally
httpsand optionallyhttp. - Reject malformed URLs, missing hostnames, nonstandard URL encodings, and URLs containing embedded usernames or passwords.
- Resolve the hostname before connecting and inspect every resulting IPv4 and IPv6 address.
- Reject loopback, private, link-local, multicast, unspecified, reserved, and other non-global address ranges.
- Normalize IPv4 and IPv6 representations to prevent bypasses using alternate numeric formats or IPv4-mapped IPv6 addresses.
- Disable automatic redirects or validate every redirect target before following it.
- Mitigate DNS rebinding by ensuring the validated address is the address used for the connection, or by validating the actual peer address after connection.
- Prefer an explicit hostname allowlist when the workflow operates on known sites.
- Apply equivalent validation to proxy configuration, or remove user-controlled proxy support when it is unnecessary.
- Add network-layer egress controls blocking loopback, private networks, link-local ranges, and cloud metadata services as defense in depth.
- Add tests covering IPv4, IPv6, decimal or encoded address forms, redirects, DNS rebinding scenarios, URL credentials, and mixed-case or malformed schemes.
- Return a clear validation error without attempting a network connection when a destination is rejected.
- Permit only explicitly required schemes, normally
