T09 · Insecure Skill Coding Practices
- Location
scripts/crawl_site.sh:9- Finding
Unrestricted Website Crawling Enables Server-Side Request Forgery
- Content
View full analysis
[output_file]}" OUTPUT="${2:-/dev/stdout}" # Normalize domain — strip protocol and trailing slash DOMAIN="${DOMAIN#https://}" DOMAIN="${DOMAIN#http://}" DOMAIN="${DOMAIN%/}" BASE="https://${DOMAIN}" ``` ```bash for path in "${PAGES[@]}"; do url="${BASE}${path}" status=$(curl -s -o /dev/null -w "%{http_code}" -L --max-time 10 "$url" 2>/dev/null || echo "000") if [ "$status" = "200" ]; then echo "## ${url}" >> "$OUTPUT" echo "" >> "$OUTPUT" # Extract text content, strip HTML tags, collapse whitespace curl -sL --max-time 15 "$url" 2>/dev/null \ | sed 's/]*>.*<\/script>//g' \ | sed 's/]*>.*<\/style>//g' \ | sed 's/<[^>]*>//g' \ | sed 's/ / /g; s/&/\&/g; s/<//g' \ | tr -s '[:space:]' '\n' \ | head -200 \ >> "$OUTPUT" ``` ### Technical Analysis The script accepts a caller-controlled domain and interpolates it directly into URLs passed to `curl`. It does not validate the hostname, port, resolved IP addresses, or URL components. Consequently, a caller can direct requests toward loopback, private, link-local, or other internal network destinations. Both `curl` invocations also use `-L`, which follows redirects. Redirect destinations are not validated, allowing an attacker-controlled public HTTPS endpoint to redirect the crawler to an internal HTTP or HTTPS service. This can bypass the initial construction of an `https://` URL. The script performs the request twice: once to determine the HTTP status and again to retrieve the response body. If the destination returns status 200, up to 200 lines of processed response content are written to the output. DNS rebinding is also possible because no resolved-addr ...[truncated 1850 chars]- Remediation
View remediation
