T09 · Insecure Skill Coding Practices
- Location
scripts/news_crawler.js:9- Finding
Unrestricted Fetching of URLs Extracted from Remote HTML
- Content
View full analysis
Vulnerability Details
File Location:
scripts/news_crawler.js:9-10, 27, 54-57, 72, 99-100, 115
Vulnerability Type: Server-Side Request Forgery (SSRF) through unvalidated remote URLs
Risk Level: MediumVulnerable Code
js const rawList = text.match(/title_array_01\((.*)/g) || []; const pageUrls = rawList.slice(1).map(item => item.match(/(http.*)/)?.[0].split('\'')[0] || ''); const data = await Promise.all(pageUrls.map(async pageUrl => { try { const pageResponse = await fetch(pageUrl, { headers }); const pageText = await pageResponse.text(); const soup = parse(pageText); const title = soup.querySelector('h3')?.text.replace('[视频]', '').trim() || ''; const content = soup.querySelector('.cnt_bd')?.text.replace(/\n/g, ' ').trim() || ''; return { date, title, content }; } catch (err) { console.error(`Error fetching page ${pageUrl}:`, err.message); return null; } }));The same vulnerable pattern is repeated in the middle and recent news crawlers:
js const pageUrls = soup.querySelectorAll('#contentELMT1368521805488378 li a') .slice(1) .map(a => a.getAttribute('href') || ''); // ... const pageResponse = await fetch(pageUrl, { headers });js const pageUrls = soup.querySelectorAll('li a').slice(1).map(a => a.getAttribute('href') || ''); // ... const pageResponse = await fetch(pageUrl, { headers });Technical Analysis
The crawler first downloads an index page from CCTV and then extracts article URLs from that remotely supplied HTML. It passes each extracted value directly to
fetch()without validating:- The URL scheme
- The destination hostname
- The destination port
- Whether DNS resolves to a loopback, private, link-local, or reserved address
- Redirect destinations
This crosses a trust boundary: although the initial index URL is fixed to a CCTV do ...[truncated 2608 chars]
- Remediation
View remediation
Remediation Suggestions
-
Apply an explicit destination allowlist
- Parse every extracted link with the standard
URLclass. - Permit only
https:. - Permit only required CCTV hostnames, such as
tv.cctv.comandcctv.cntv.cn. - Reject embedded credentials, nonstandard ports, malformed URLs, and empty links.
- Parse every extracted link with the standard
-
Prevent DNS-based SSRF bypasses
- Resolve the hostname before connecting.
- Reject loopback, private, link-local, multicast, unspecified, and reserved IPv4 and IPv6 ranges.
- Account for IPv4-mapped IPv6 addresses and alternate IP representations.
- Revalidate the destination after every redirect, or disable automatic redirects and process them manually.
-
Remove unnecessary headers
- Remove the hardcoded
Cookie,Host,Proxy-Connection, andUpgrade-Insecure-Requestsheaders. - Do not forward authentication or session headers to URLs obtained from remote pages.
- Allow the HTTP client to derive the
Hostheader from the validated destination.
- Remove the hardcoded
-
Bound network resource usage
- Add an
AbortControllertimeout to every request. - Limit the number of article links processed.
- Enforce maximum response sizes while streaming bodies.
- Restrict redirect counts and concurrent requests.
- Verify
response.okand validate expected content types before parsing.
- Add an
-
Validate the date argument
- Require a strict
YYYYMMDDvalue using/^\d{8}$/. - Confirm that it represents a real calendar date before constructing the index URL.
- Require a strict
-
Use a centralized safe-fetch helper
- Implement URL validation, DNS/IP checks, redirect handling, timeout enforcement, and size limits in one function.
- Use that helper consistently in
fetchOlderNews,fetchMidNews, andfetchRecentNews.
-
