T09 · Insecure Skill Coding Practices
Warning
- Location
- scripts/fetch_pdf.py:30
- Finding
- Weak arXiv URL Validation Permits Arbitrary Network Retrieval<![CDATA[ ## Vulnerability Details **File Location**: `scripts/fetch_pdf.py`, lines 30–31, 77–81, and 101–110 **Vulnerability Type**: Insufficient URL validation and server-side request forgery exposure **Risk Level**: Medium ### Vulnerable Code ```python def is_arxiv_url(url: str) -> bool: return bool(re.search(r'arxiv\.org', url, re.IGNORECASE)) ``` ```python def download_pdf(url: str, output: str) -> str: url = normalize_arxiv_pdf_url(url) print(f"Downloading from: {url}", file=sys.stderr) urllib.request.urlretrieve(url, output) return os.path.abspath(output) ``` ```python if source.startswith("http://") or source.startswith("https://"): if not is_arxiv_url(source): print(f"Error: Only arXiv URLs are supported. Got: {source}", file=sys.stderr) sys.exit(1) output = args.output if not output: match = re.search(r'arxiv\.org/(?:abs|pdf)/([\d.]+)', source) if match: arxiv_id = match.group(1).replace(".", "_") output = f"arxiv_{arxiv_id}.pdf" else: output = "paper.pdf" pdf_path = download_pdf(source, output) ``` ### Technical Analysis The URL allowlist is implemented as an unanchored regular-expression search over the entire URL string. It verifies only that the text `arxiv.org` appears somewhere; it does not parse the URL or confirm that the destination hostname is an approved arXiv host. For example, a URL such as the following passes the check even though its destination is not arXiv: ```text http://127.0.0.1:8080/resource?source=arxiv.org ``` The URL remains unchanged by `normalize_arxiv_pdf_url()` unless it contains a matching `arxiv.org/abs/<numeric-id>` substring. It is then passed directly to `urllib.request.urlretrieve()`. Redirect destinations are not independently validated, and the downloaded response is not checked for an expected media type, PDF signature, or maximum size. This creates an arbitrary network-retrieval primitive and p ...[truncated 1612 chars]
- Remediation
- <![CDATA[ ## Remediation Suggestions 1. Parse the URL with `urllib.parse.urlsplit()` rather than searching the raw string. 2. Require the `https` scheme. 3. Compare the normalized hostname against an explicit allowlist such as `arxiv.org` and `export.arxiv.org`; do not use substring or suffix checks that permit deceptive domains. 4. Reject embedded credentials, fragments, unexpected ports, and unsupported arXiv path formats. 5. Restrict accepted paths to recognized `/abs/<id>` and `/pdf/<id>` formats, then construct the final PDF URL internally. 6. Disable automatic redirects or validate the scheme, hostname, port, and path of every redirect destination. 7. Stream the response with explicit connection and read timeouts and enforce a maximum download size. 8. Confirm the expected content type and verify that the downloaded file begins with a valid PDF signature. 9. Write to a newly created file and refuse to overwrite an existing destination unless the user explicitly confirms it. 10. Where appropriate, block loopback, link-local, private, and other internal address ranges after DNS resolution as defense in depth. ]]>