T09 · Insecure Skill Coding Practices
Warning
- Location
- code.md:17
- Finding
- robots.txt Verification Fails Open on Retrieval and Parsing Errors<![CDATA[ ## Vulnerability Details **File Location**: `code.md`, lines 17–34 **Vulnerability Type**: Fail-open authorization and policy verification **Risk Level**: Medium ### Vulnerable Code ```javascript protocol.get(robotsUrl, (res) => { let data = ''; res.on('data', chunk => data += chunk); res.on('end', () => { try { const allowed = evaluateRobotsRules(data, agentLabel, parsed.pathname); resolve(allowed); } catch (e) { resolve(true); // Missing robots.txt = permitted } }); }).on('error', () => resolve(true)); }; fetchRobots(parsed.protocol === 'https:' ? https : http) .then(resolve) .catch(() => resolve(true)); ``` ### Technical Analysis The robots.txt verification routine treats every network, protocol, and parsing failure as authorization to continue. Both the request error handler and the outer rejection handler resolve to `true`. An exception raised while evaluating the response also resolves to `true`. The response status code and content type are not validated. Therefore, an HTTP error page, malformed response, interrupted request, or other non-policy response may be evaluated as though it were a valid robots.txt file. This behavior conflicts with the Skill's stated requirement to retrieve and review robots.txt before extraction and to halt when restrictions exist. The vulnerability is a fail-open policy-control defect: failure to determine whether access is permitted is incorrectly treated as affirmative permission. ### Attack Path 1. The scraper is configured to extract content from a target URL. 2. The target server, an intermediary, or a transient network condition causes the robots.txt request to fail, return malformed content, or return an HTTP error response. 3. The request error, parser exception, or rejected promise reaches a `resolve(true)` branch. 4. `runScraper` receives `permitted = true`. 5. The scraper requests the target resource without having successfully established the site's rob ...[truncated 780 chars]
- Remediation
- <![CDATA[ ## Remediation Suggestions - Replace the Boolean result with an explicit state such as `allowed`, `denied`, or `unknown`. - Fail closed when transport, TLS, timeout, parsing, or server errors prevent policy verification. - Validate the HTTP status before parsing: - Parse successful robots.txt responses. - Handle a confirmed `404 Not Found` according to a documented policy. - Treat redirects, authentication responses, rate limits, and server errors explicitly. - Validate that the response is a plausible robots.txt representation rather than an HTML error page. - Add response-size limits and request timeouts to the robots.txt retrieval function. - Require explicit operator approval before continuing from an `unknown` state. - Log the status code, final URL, verification outcome, and reason without representing a failed check as successful. - Add tests covering DNS failures, TLS failures, malformed policy files, HTTP 403/404/429/500 responses, redirects, and parser exceptions. ]]>