ClawHub
Back to skill

Security audit

hn-crawler

Security checks for vulnerabilities and agentic risk

Overview

The skill mostly matches its crawler/reporting purpose, but it can be retargeted to arbitrary URLs and local inputs without safeguards, so it should be reviewed before use.

Install only if you are comfortable with a Python crawler that can fetch any URL provided by CLI or environment and write local outputs. Run it in an isolated environment with restricted outbound network access, avoid using it against internal URLs, pin dependencies before installation, and note that organize.py currently contains a syntax error that may prevent the full pipeline from running until fixed.

A.I.G

Vulnerability Patterns
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (2)

T05 · Unauthorized Access and Privilege Escalation

Warning
Location
scripts/crawl.py:69
Finding

Unrestricted URL Fetching Enables Server-Side Request Forgery

Content
View full analysis

Vulnerability Details

File Location: scripts/crawl.py:69-74, with attacker-controlled input accepted at scripts/crawl.py:130-134 and propagated by scripts/run_pipeline.py:108-109,205-209
Vulnerability Type: Server-Side Request Forgery through unrestricted URL fetching
Risk Level: Medium

Vulnerable Code

python
response = session.get(
    url,
    headers=headers,
    timeout=timeout,
    allow_redirects=True
)

The URL is accepted without restriction:

python
parser.add_argument(
    "--url",
    default=os.getenv("TARGET_URL", DEFAULT_URL),
    help=f"Target URL (default: {DEFAULT_URL})"
)

The pipeline propagates the supplied value to the crawler:

python
crawl_args = []
if url:
    crawl_args.extend(["--url", url])

returncode, output_file = run_script(CRAWL_SCRIPT, crawl_args)

Technical Analysis

Although the Skill is declared as a crawler for https://hn.aimaker.dev/, it accepts an arbitrary URL from either the --url argument or the TARGET_URL environment variable. It does not validate:

  • The URL scheme
  • The destination hostname
  • The destination port
  • The IP addresses produced by DNS resolution
  • Whether the address is loopback, private, link-local, reserved, or a cloud metadata endpoint
  • Redirect destinations

Because allow_redirects=True is enabled, an initially acceptable endpoint could redirect the request to an internal service. The returned response body is subsequently written to disk and may be processed by later pipeline stages.

This exceeds the minimum network privilege required for a Skill whose stated purpose is limited to crawling hn.aimaker.dev.

Attack Path

  1. An attacker or untrusted caller influences the --url argument or TARGET_URL environment variable.
  2. The attacker supplies a loopback, private-network, link-local, metadata-service, or redirecting URL.
  3. requests.Session.get() s ...[truncated 898 chars]
Remediation
View remediation

Remediation Suggestions

  1. Restrict destinations to an explicit hostname allowlist, preferably only hn.aimaker.dev.
  2. Require HTTPS and reject unsupported schemes, embedded credentials, unexpected ports, and malformed URLs.
  3. Resolve the hostname before connecting and reject loopback, private, link-local, multicast, reserved, unspecified, and metadata-service IP ranges for both IPv4 and IPv6.
  4. Disable automatic redirects or validate every redirect destination using the same scheme, hostname, port, and resolved-address policy.
  5. Protect against DNS rebinding by verifying the address used for the connection, not only an earlier DNS lookup.
  6. Apply an outbound network policy at the container or host level so the process cannot reach internal networks or metadata services.
  7. Limit response size and redirect count to reduce denial-of-service exposure.
  8. If arbitrary crawling is intentionally supported, require explicit user confirmation and run the network stage in an isolated environment with restricted egress.

T08 · Insecure Dependencies

Note
Location
scripts/requirements.txt:4
Finding

Mutable and Non-Reproducible Third-Party Dependencies

Content
View full analysis

Vulnerability Details

File Location: scripts/requirements.txt:4-9; installation instruction at SKILL.md:45-48
Vulnerability Type: Unpinned dependency resolution without integrity verification
Risk Level: Low

Vulnerable Code

text
requests>=2.31.0
urllib3>=2.0.0

# HTML parsing
beautifulsoup4>=4.12.0
lxml>=4.9.0

The documented installation command is:

bash
cd .trae/skills/hn-crawler/scripts
pip install -r requirements.txt

Technical Analysis

Each dependency specifies only a minimum version. Consequently, installing the Skill at different times can resolve to different package versions, including future versions that were not reviewed with this codebase.

The project provides no lock file, exact version pins, package hashes, or trusted-index enforcement. This makes installation non-reproducible and leaves package integrity dependent on the caller's pip configuration and the current state of the selected package index.

The package names observed during the audit are established packages, and no suspicious third-party index or obvious typosquatted package was identified. The risk arises from mutable dependency resolution rather than evidence that the currently named packages are malicious.

Attack Path

  1. A user follows the documentation and executes pip install -r requirements.txt.
  2. Pip contacts its configured package index and resolves any versions satisfying the lower bounds.
  3. A future compromised, malicious, or incompatible release satisfies those constraints.
  4. Pip downloads and installs the unreviewed package version.
  5. Package installation behavior or subsequent imports execute code with the privileges of the user running the Skill.

Impact Assessment

A compromised dependency could execute arbitrary code during installation or import with the privileges of the installing user. Depending on those privileges, this could expose loca ...[truncated 237 chars]

Remediation
View remediation

Remediation Suggestions

  1. Pin every direct and transitive dependency to an exact, reviewed version.
  2. Generate a lock file containing cryptographic hashes, for example with pip-tools.
  3. Install with hash verification enabled:
bash
python -m pip install --require-hashes -r requirements.lock
  1. Document and enforce a trusted Python package index rather than inheriting arbitrary global pip configuration.
  2. Install dependencies in a dedicated virtual environment or restricted container.
  3. Use automated dependency vulnerability scanning and review updates before regenerating the lock file.
  4. Remove unused dependencies where possible. In particular, verify whether direct declarations of urllib3 and lxml are necessary for the implemented runtime paths.

SkillSpector

Vulnerability Patterns
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Behavioral ASTexec() Call, eval() Call, Dynamic Import
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Findings (23)

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding

The mismatch between advertised end-to-end web processing and actual reliance on pre-organized local JSON creates trust-boundary confusion. In agent systems, confusion about whether data comes from the network or local storage can lead to unsafe handling, improper authorization assumptions, and unreviewed ingestion of attacker-controlled files.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
96% confidence
Finding

The mismatch between advertised end-to-end web processing and actual reliance on pre-organized local JSON creates trust-boundary confusion. In agent systems, confusion about whether data comes from the network or local storage can lead to unsafe handling, improper authorization assumptions, and unreviewed ingestion of attacker-controlled files.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding

The mismatch between advertised end-to-end web processing and actual reliance on pre-organized local JSON creates trust-boundary confusion. In agent systems, confusion about whether data comes from the network or local storage can lead to unsafe handling, improper authorization assumptions, and unreviewed ingestion of attacker-controlled files.

Content

No source excerpt is available for this finding.

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
82% confidence
Finding

The skill declares no explicit tool scope or permission boundaries even though it describes capabilities that imply network access, shell execution, environment use, and file read/write behavior. In an agent setting, missing scope declarations can cause over-broad tool access and make it harder for the host to enforce least privilege, especially for a crawler that accepts configurable target URLs and output paths.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
82% confidence
Finding

The manifest description and main heading/body present the skill as operating in Chinese only, with no indication that users can choose another language. This can violate language/locale policy when a skill imposes a language without explicit user opt-in or documented regional justification.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
91% confidence
Finding

The module docstring states that the script gets raw HTML content from hn.aimaker.dev, which communicates a site-specific intent. In practice, the code exposes a configurable URL input and fetches whatever URL is provided, so the documentation actively understates the implemented scope.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The code hard-codes Accept-Language to prefer zh-CN, which imposes a specific locale on all requests. This is a natural-language/locale policy issue because the skill does not offer user opt-in or explain why a Chinese locale must be forced.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The skill is presented as a crawler for hn.aimaker.dev, but it accepts arbitrary URLs via CLI and environment variables and fetches them without any allowlist or validation. In an agent/tooling context, this creates a server-side request forgery style primitive that can be abused to reach unintended internal or sensitive network resources, making the actual behavior materially more dangerous than the stated scope.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

This code file contains natural-language documentation and CLI output exclusively in Chinese, including the module docstring and later user-facing help/messages. Under the policy, forcing a specific language without user opt-in is a locale/language policy violation because users are not given a choice or documented region-specific justification.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

This file contains natural-language documentation and CLI help strings exclusively in Chinese, starting with the module docstring and continuing through argument descriptions and examples. Under the policy, forcing a specific language without opt-in is a locale/language policy violation when no alternative language option or justification is provided.

Content

No source excerpt is available for this finding.

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/run_pipeline.py (reported line 44)May include surrounding context.

python
print(f"{'='*60}\n")
    
    try:
        result = subprocess.run(
            cmd,
            capture_output=True,
            text=True,

Description-Behavior Mismatch

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

The skill is described as a site-specific crawler for hn.aimaker.dev, but the --url option and TARGET_URL default allow it to fetch arbitrary external URLs. This expands the tool into a general-purpose web fetch pipeline, which can enable out-of-scope access, SSRF-like behavior in agent environments, and processing of attacker-chosen content.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

Allowing --input-file lets the pipeline process arbitrary local files instead of only content produced by the crawler. In an agent context, this broadens the skill from a web-crawling utility into a generic local file processing primitive, which may expose sensitive local data to later extraction or summarization stages.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

This is a natural-language policy issue because the skill forces a specific language for its docstring, CLI help, status messages, and generated report content. Users who do not read Chinese are given no opt-in or alternative locale, and the file contains no justification that this is a region-specific tool.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
85% confidence
Finding

This code file contains natural-language documentation exclusively in Chinese, starting with the module description. Under the policy rule for language/locale constraints, this can be a violation when a specific language is imposed without any opt-in or documented justification.

Content

No source excerpt is available for this finding.

Unpinned Dependencies

Low
Category
Supply Chain
Confidence
94% confidence
Finding

The dependency is specified with a lower bound only, so builds may resolve to different versions over time. This weakens reproducibility and can unintentionally introduce a vulnerable or incompatible release through normal installs or future supply-chain changes.

Content

Scanner excerpt · scripts/requirements.txt (reported line 4)May include surrounding context.

text
# HN Crawler Skill 依赖

# HTTP 请求
requests>=2.31.0
urllib3>=2.0.0

# HTML 解析

Unverifiable Dependency: requests has 16 known advisory(ies) (CVE-2014-1830 (Exposure of Sensitive Information to an Unauthorized Actor in Requests); CVE-2024-47081 (Requests vulnerable to .netrc credentials leak via malicious URLs); CVE-2024-35195 (Requests `Session` object does not verify requests after making first request wi) +13 more), but the manifest does not pin a version, so it is unknown whether the installed release is affected

Low
Category
Supply Chain
Confidence
90% confidence
Finding

The manifest does not pin requests, so there is no assurance that deployed environments avoid known vulnerable releases. In a web-crawling skill that fetches attacker-controlled URLs or content, this uncertainty can expose the agent to request-handling flaws such as credential leakage or other client-side issues if an affected version is installed.

Content

No source excerpt is available for this finding.

Unpinned Dependencies

Low
Category
Supply Chain
Confidence
94% confidence
Finding

Using an unpinned urllib3 version allows the installed package version to vary by environment and time. This increases supply-chain risk and makes it impossible to guarantee that deployments avoid vulnerable releases.

Content

Scanner excerpt · scripts/requirements.txt (reported line 5)May include surrounding context.

text
# HTTP 请求
requests>=2.31.0
urllib3>=2.0.0

# HTML 解析
beautifulsoup4>=4.12.0

Unverifiable Dependency: urllib3 has 16 known advisory(ies) (CVE-2025-66471 (urllib3 streaming API improperly handles highly compressed data); CVE-2024-37891 (urllib3's Proxy-Authorization request header isn't stripped during cross-origin ); CVE-2026-21441 (Decompression-bomb safeguards bypassed when following HTTP redirects (streaming ) +13 more), but the manifest does not pin a version, so it is unknown whether the installed release is affected

Low
Category
Supply Chain
Confidence
91% confidence
Finding

Because urllib3 is unpinned, the installation may resolve to a version affected by known advisories without visibility in the manifest. This is relevant in a crawler because it processes remote responses from untrusted servers, making transport, redirect, and decompression issues more meaningful than in a non-networked tool.

Content

No source excerpt is available for this finding.

Unpinned Dependencies

Low
Category
Supply Chain
Confidence
92% confidence
Finding

A minimum-only specifier for beautifulsoup4 is not deterministic and can pull newer versions without explicit review. While not inherently exploitable on its own, it increases operational and supply-chain risk by reducing control over installed code.

Content

Scanner excerpt · scripts/requirements.txt (reported line 8)May include surrounding context.

text
urllib3>=2.0.0

# HTML 解析
beautifulsoup4>=4.12.0
lxml>=4.9.0

# 数据处理

Unpinned Dependencies

Low
Category
Supply Chain
Confidence
95% confidence
Finding

The unpinned lxml dependency means package resolution may select different versions, including ones later found vulnerable. Because lxml parses untrusted HTML/XML content in a crawler context, version uncertainty is somewhat more concerning than for purely passive libraries.

Content

Scanner excerpt · scripts/requirements.txt (reported line 9)May include surrounding context.

text
# HTML 解析
beautifulsoup4>=4.12.0
lxml>=4.9.0

# 数据处理
# (使用 Python 标准库)

Unverifiable Dependency: lxml has 14 known advisory(ies) (CVE-2021-43818 (lxml's HTML Cleaner allows crafted and SVG embedded scripts to pass through); CVE-2014-3146 (lxml Cross-site Scripting Via Control Characters); CVE-2021-28957 (lxml vulnerable to Cross-Site Scripting ) +11 more), but the manifest does not pin a version, so it is unknown whether the installed release is affected

Low
Category
Supply Chain
Confidence
92% confidence
Finding

The manifest leaves lxml unpinned despite known advisories, so environments may end up with a vulnerable parser version. In this skill, lxml is directly relevant because the crawler parses untrusted web content, which increases the chance that parser bugs could be triggered by malicious pages and lead to content sanitization bypasses, denial of service, or other parser-related issues depending on usage.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Low
Category
Not specified by scanner
Confidence
89% confidence
Finding

Reading TARGET_URL from the environment creates an external control channel for selecting crawl targets outside the declared site-specific scope. While less severe than an explicit CLI override, it still permits unreviewed retargeting in automated or hosted agent deployments.

Content

No source excerpt is available for this finding.