T09 · Insecure Skill Coding Practices
Warning
- Location
- scripts/crawl_site.sh:9
- Finding
- Unrestricted Website Crawling Enables Server-Side Request Forgery<![CDATA[ ## Vulnerability Details **File Location**: `scripts/crawl_site.sh`, lines 9-17 and 37-52 **Vulnerability Type**: Server-Side Request Forgery (SSRF) **Risk Level**: Medium ### Vulnerable Code ```bash DOMAIN="${1:?Usage: crawl_site.sh <domain> [output_file]}" OUTPUT="${2:-/dev/stdout}" # Normalize domain — strip protocol and trailing slash DOMAIN="${DOMAIN#https://}" DOMAIN="${DOMAIN#http://}" DOMAIN="${DOMAIN%/}" BASE="https://${DOMAIN}" ``` ```bash for path in "${PAGES[@]}"; do url="${BASE}${path}" status=$(curl -s -o /dev/null -w "%{http_code}" -L --max-time 10 "$url" 2>/dev/null || echo "000") if [ "$status" = "200" ]; then echo "## ${url}" >> "$OUTPUT" echo "" >> "$OUTPUT" # Extract text content, strip HTML tags, collapse whitespace curl -sL --max-time 15 "$url" 2>/dev/null \ | sed 's/<script[^>]*>.*<\/script>//g' \ | sed 's/<style[^>]*>.*<\/style>//g' \ | sed 's/<[^>]*>//g' \ | sed 's/ / /g; s/&/\&/g; s/</</g; s/>/>/g' \ | tr -s '[:space:]' '\n' \ | head -200 \ >> "$OUTPUT" ``` ### Technical Analysis The script accepts a caller-controlled domain and interpolates it directly into URLs passed to `curl`. It does not validate the hostname, port, resolved IP addresses, or URL components. Consequently, a caller can direct requests toward loopback, private, link-local, or other internal network destinations. Both `curl` invocations also use `-L`, which follows redirects. Redirect destinations are not validated, allowing an attacker-controlled public HTTPS endpoint to redirect the crawler to an internal HTTP or HTTPS service. This can bypass the initial construction of an `https://` URL. The script performs the request twice: once to determine the HTTP status and again to retrieve the response body. If the destination returns status 200, up to 200 lines of processed response content are written to the output. DNS rebinding is also possible because no resolved-addr ...[truncated 1850 chars]
- Remediation
- <![CDATA[ ## Remediation Suggestions 1. **Strictly parse the input** - Accept only a hostname or a fully parsed HTTP(S) URL. - Reject user information, embedded credentials, paths, query strings, fragments, malformed ports, and unexpected URL schemes. - Restrict destination ports to an explicit allowlist such as 80 and 443. 2. **Block non-public destinations** - Resolve the hostname before making a request. - Reject every resolved IPv4 and IPv6 address belonging to loopback, private, link-local, multicast, unspecified, documentation, carrier-grade NAT, or other reserved ranges. - Reject hostnames such as `localhost` and internal-only DNS names where appropriate. 3. **Control redirects** - Prefer disabling redirects. - If redirects are required, validate the scheme, hostname, port, and every resolved address at each redirect hop. - Set a low redirect limit and restrict redirect protocols with appropriate `curl` options. 4. **Prevent DNS rebinding** - Pin requests to addresses that were validated immediately before connection. - Ensure all addresses returned for a hostname are public, rather than validating only one address. - Repeat validation for every redirect destination. 5. **Apply network-level isolation** - Run the crawler in a sandbox with egress rules that deny loopback-sensitive endpoints, private networks, cloud metadata addresses, and infrastructure control planes. - Use a controlled outbound proxy designed to enforce public-web-only access. 6. **Limit exposed responses** - Impose strict response-size limits. - Avoid returning internal error details. - Record rejected destinations in security logs without exposing sensitive response content. ]]>