T09 · Insecure Skill Coding Practices
Warning
- Location
scripts/fetch_pdf.py:30- Finding
Weak arXiv URL Validation Permits Arbitrary Network Retrieval
- Content
View full analysis
bool: return bool(re.search(r'arxiv\.org', url, re.IGNORECASE)) ``` ```python def download_pdf(url: str, output: str) -> str: url = normalize_arxiv_pdf_url(url) print(f"Downloading from: {url}", file=sys.stderr) urllib.request.urlretrieve(url, output) return os.path.abspath(output) ``` ```python if source.startswith("http://") or source.startswith("https://"): if not is_arxiv_url(source): print(f"Error: Only arXiv URLs are supported. Got: {source}", file=sys.stderr) sys.exit(1) output = args.output if not output: match = re.search(r'arxiv\.org/(?:abs|pdf)/([\d.]+)', source) if match: arxiv_id = match.group(1).replace(".", "_") output = f"arxiv_{arxiv_id}.pdf" else: output = "paper.pdf" pdf_path = download_pdf(source, output) ``` ### Technical Analysis The URL allowlist is implemented as an unanchored regular-expression search over the entire URL string. It verifies only that the text `arxiv.org` appears somewhere; it does not parse the URL or confirm that the destination hostname is an approved arXiv host. For example, a URL such as the following passes the check even though its destination is not arXiv: ```text http://127.0.0.1:8080/resource?source=arxiv.org ``` The URL remains unchanged by `normalize_arxiv_pdf_url()` unless it contains a matching `arxiv.org/abs/` substring. It is then passed directly to `urllib.request.urlretrieve()`. Redirect destinations are not independently validated, and the downloaded response is not checked for an expected media type, PDF signature, or maximum size. This creates an arbitrary network-retrieval primitive and p ...[truncated 1612 chars]- Remediation
View remediation
` and `/pdf/` formats, then construct the final PDF URL internally. 6. Disable automatic redirects or validate the scheme, hostname, port, and path of every redirect destination. 7. Stream the response with explicit connection and read timeouts and enforce a maximum download size. 8. Confirm the expected content type and verify that the downloaded file begins with a valid PDF signature. 9. Write to a newly created file and refuse to overwrite an existing destination unless the user explicitly confirms it. 10. Where appropriate, block loopback, link-local, private, and other internal address ranges after DNS resolution as defense in depth. ]]>