T09 · Insecure Skill Coding Practices
Warning
- Location
- scripts/news_crawler.js:9
- Finding
- Unrestricted Fetching of URLs Extracted from Remote HTML## Vulnerability Details **File Location**: `scripts/news_crawler.js:9-10, 27, 54-57, 72, 99-100, 115` **Vulnerability Type**: Server-Side Request Forgery (SSRF) through unvalidated remote URLs **Risk Level**: Medium ### Vulnerable Code ```js const rawList = text.match(/title_array_01\((.*)/g) || []; const pageUrls = rawList.slice(1).map(item => item.match(/(http.*)/)?.[0].split('\'')[0] || ''); const data = await Promise.all(pageUrls.map(async pageUrl => { try { const pageResponse = await fetch(pageUrl, { headers }); const pageText = await pageResponse.text(); const soup = parse(pageText); const title = soup.querySelector('h3')?.text.replace('[视频]', '').trim() || ''; const content = soup.querySelector('.cnt_bd')?.text.replace(/\n/g, ' ').trim() || ''; return { date, title, content }; } catch (err) { console.error(`Error fetching page ${pageUrl}:`, err.message); return null; } })); ``` The same vulnerable pattern is repeated in the middle and recent news crawlers: ```js const pageUrls = soup.querySelectorAll('#contentELMT1368521805488378 li a') .slice(1) .map(a => a.getAttribute('href') || ''); // ... const pageResponse = await fetch(pageUrl, { headers }); ``` ```js const pageUrls = soup.querySelectorAll('li a').slice(1).map(a => a.getAttribute('href') || ''); // ... const pageResponse = await fetch(pageUrl, { headers }); ``` ### Technical Analysis The crawler first downloads an index page from CCTV and then extracts article URLs from that remotely supplied HTML. It passes each extracted value directly to `fetch()` without validating: - The URL scheme - The destination hostname - The destination port - Whether DNS resolves to a loopback, private, link-local, or reserved address - Redirect destinations This crosses a trust boundary: although the initial index URL is fixed to a CCTV do ...[truncated 2608 chars]
- Remediation
- ## Remediation Suggestions 1. **Apply an explicit destination allowlist** - Parse every extracted link with the standard `URL` class. - Permit only `https:`. - Permit only required CCTV hostnames, such as `tv.cctv.com` and `cctv.cntv.cn`. - Reject embedded credentials, nonstandard ports, malformed URLs, and empty links. 2. **Prevent DNS-based SSRF bypasses** - Resolve the hostname before connecting. - Reject loopback, private, link-local, multicast, unspecified, and reserved IPv4 and IPv6 ranges. - Account for IPv4-mapped IPv6 addresses and alternate IP representations. - Revalidate the destination after every redirect, or disable automatic redirects and process them manually. 3. **Remove unnecessary headers** - Remove the hardcoded `Cookie`, `Host`, `Proxy-Connection`, and `Upgrade-Insecure-Requests` headers. - Do not forward authentication or session headers to URLs obtained from remote pages. - Allow the HTTP client to derive the `Host` header from the validated destination. 4. **Bound network resource usage** - Add an `AbortController` timeout to every request. - Limit the number of article links processed. - Enforce maximum response sizes while streaming bodies. - Restrict redirect counts and concurrent requests. - Verify `response.ok` and validate expected content types before parsing. 5. **Validate the date argument** - Require a strict `YYYYMMDD` value using `/^\d{8}$/`. - Confirm that it represents a real calendar date before constructing the index URL. 6. **Use a centralized safe-fetch helper** - Implement URL validation, DNS/IP checks, redirect handling, timeout enforcement, and size limits in one function. - Use that helper consistently in `fetchOlderNews`, `fetchMidNews`, and `fetchRecentNews`.