T09 · Insecure Skill Coding Practices
Warning
- Location
code.md:17- Finding
robots.txt Verification Fails Open on Retrieval and Parsing Errors
- Content
View full analysis
{ let data = ''; res.on('data', chunk => data += chunk); res.on('end', () => { try { const allowed = evaluateRobotsRules(data, agentLabel, parsed.pathname); resolve(allowed); } catch (e) { resolve(true); // Missing robots.txt = permitted } }); }).on('error', () => resolve(true)); }; fetchRobots(parsed.protocol === 'https:' ? https : http) .then(resolve) .catch(() => resolve(true)); ``` ### Technical Analysis The robots.txt verification routine treats every network, protocol, and parsing failure as authorization to continue. Both the request error handler and the outer rejection handler resolve to `true`. An exception raised while evaluating the response also resolves to `true`. The response status code and content type are not validated. Therefore, an HTTP error page, malformed response, interrupted request, or other non-policy response may be evaluated as though it were a valid robots.txt file. This behavior conflicts with the Skill's stated requirement to retrieve and review robots.txt before extraction and to halt when restrictions exist. The vulnerability is a fail-open policy-control defect: failure to determine whether access is permitted is incorrectly treated as affirmative permission. ### Attack Path 1. The scraper is configured to extract content from a target URL. 2. The target server, an intermediary, or a transient network condition causes the robots.txt request to fail, return malformed content, or return an HTTP error response. 3. The request error, parser exception, or rejected promise reaches a `resolve(true)` branch. 4. `runScraper` receives `permitted = true`. 5. The scraper requests the target resource without having successfully established the site's rob ...[truncated 780 chars]- Remediation
View remediation