Website Usability Testing using Nova Act

• Outcome: Review
• Latest audit: Sep 10, 2026 · 5:51 PM UTC
• Version: 1.0.3

Overview

This skill is a real usability-testing automation tool, but it needs Review because it can perform live workflow actions, store sensitive browser traces, send page summaries to an undeclared AI provider, and generate unsafe HTML reports.

Install only if you are comfortable running live browser automation against authorized test targets. Prefer a sandbox or VM, test accounts, non-production sites, pinned dependencies, and no real personal/payment/account data. Review or delete generated traces and reports, and avoid opening reports from untrusted sites until HTML escaping/CSP is fixed.

A.I.G

Vulnerability Patterns

• Insecure Dependencies: Introduces malicious components through unsafe dependency sources
• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review

Findings (2)

T09 · Insecure Skill Coding Practices

Error

• Location: scripts/enhanced_report_generator.py:408
• Finding: Stored HTML Injection in Generated Usability Reports<![CDATA[ ## Vulnerability Details **File Location**: `scripts/enhanced_report_generator.py:408-411` **Additional Affected Locations**: `scripts/enhanced_report_generator.py:555-558, 590-603, 660-677, 731-732` **Vulnerability Type**: Stored HTML injection caused by missing output encoding **Risk Level**: High ### Vulnerable Code ```python html += f""" <div class="page-analysis"> <h3>📄 Page Analysis: {page_analysis.get('title', 'Unknown')}</h3> <p><strong>Purpose:</strong> {page_analysis.get('purpose', 'Not analyzed')}</p> <p><strong>Navigation:</strong> {', '.join(page_analysis.get('navigation', ['None found']))}</p> """ ``` The same unsafe rendering pattern is used for Nova Act responses: ```python if observations_list: notes = '; '.join(str(o) for o in observations_list) elif raw_response: notes = f"Response: {raw_response}" elif error_msg: notes = f"Error: {error_msg}" else: notes = "No observations recorded" html += f""" <div class="observation-notes {notes_class}"> <strong>{"⚠️ " if is_issue else ""}Observation:</strong> {notes} </div> """ ``` ### Technical Analysis The report generator inserts website-derived and model-derived values directly into an HTML document without applying contextual HTML escaping. Affected values include: - Page titles, purposes, and navigation entries extracted from the tested website - Raw Nova Act responses - Persona names and test-case descriptions - Actions, expected outcomes, errors, and observations - Trace filenames and paths used in HTML attributes Because the tested website is an untrusted input source, an attacker can place HTML or script-bearing markup in visible page content. If Nova Act preserves that content in a title, navigation result, page description, or raw response, the report generator writes it verbatim into `nova_act_usability_report.html`. The vulnerability is stored rather than reflected: the malicious payload is first collecte ...[truncated 1782 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions 1. Apply HTML escaping to every untrusted text value before interpolation: ```python from html import escape title = escape(str(page_analysis.get("title", "Unknown")), quote=True) purpose = escape(str(page_analysis.get("purpose", "Not analyzed")), quote=True) navigation = escape( ", ".join(map(str, page_analysis.get("navigation", ["None found"]))), quote=True ) notes = escape(str(notes), quote=True) ``` 2. Escape attribute values separately with `quote=True`. Do not place raw trace paths or filenames into `href` attributes. 3. Validate trace links: - Resolve paths to canonical local paths. - Confirm they remain inside the expected log directory. - Convert them to file URIs with `Path.resolve().as_uri()`. - Reject unexpected schemes such as `javascript:`, `data:`, and remote HTTP URLs. 4. Prefer a template engine configured with automatic escaping instead of constructing HTML through f-strings. 5. Add a restrictive Content Security Policy to the generated report, for example: ```html <meta http-equiv="Content-Security-Policy" content="default-src 'none'; style-src 'unsafe-inline'; img-src data: file:"> ``` 6. Avoid enabling scripts in the report. If scripting is later required, use a nonce-based CSP and never permit inline event handlers. 7. Add regression tests using payloads in every dynamic field, including: - `<script>alert(1)</script>` - `<img src=x onerror=alert(1)>` - `" onmouseover="alert(1)` - `javascript:alert(1)` 8. Apply the correction consistently to all dynamic interpolation sites, not only the primary page-analysis section. ]]>

T08 · Insecure Dependencies

Warning

• Location: SKILL.md:24
• Finding: Unpinned Third-Party Dependencies and Browser Installation< | | **Config Location** | `~/.openclaw/config/nova-act.json` | | **Format** | `{"apiKey": "your-nova-act-api-key-here"}` | | **Dependencies** | `pip3 install nova-act pydantic playwright` | | **Browser** | `playwright install chromium` (~300MB download) | ``` The package manifest also specifies dependencies without versions: ```json "requirements": { "python": ">=3.8", "packages": [ "nova-act", "playwright" ] } ``` ### Technical Analysis The Skill instructs users and agents to install the latest available releases of `nova-act`, `pydantic`, and `playwright`. It also downloads a Chromium artifact through Playwright without recording or verifying a reviewed version and integrity hash in the project. Package installation and imports execute code with the privileges of the user running the Skill. Because no exact versions, lock file, or hashes are provided, the effective dependency code can change after this Skill has been audited. This does not prove that the named packages are malicious. The vulnerability is the absence of reproducible and integrity-verified dependency resolution, which expands the supply-chain attack surface and can also introduce incompatible future behavior. ### Attack Path 1. A user or agent follows the documented setup instructions. 2. `pip3 install nova-act pydantic playwright` resolves the newest package releases available from the configured Python package index. 3. An upstream account compromise, package-index compromise, unsafe future release, or local index substitution causes a malicious or altered package version to be selected. 4. Installation hooks or later pa ...[truncated 1127 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions 1. Pin all direct dependencies to reviewed versions: ```text nova-act==<reviewed-version> pydantic==<reviewed-version> playwright==<reviewed-version> ``` 2. Create a hash-locked requirements file using a tool such as `pip-compile --generate-hashes`, then install with: ```bash python3 -m pip install --require-hashes -r requirements.txt ``` 3. Include transitive dependencies in the lock file so resolution is reproducible. 4. Install dependencies inside a dedicated virtual environment or isolated container rather than the user's global Python environment. 5. Use an explicitly configured trusted package index and prevent unreviewed fallback indexes: ```bash python3 -m pip install \ --index-url https://pypi.org/simple \ --require-hashes \ -r requirements.txt ``` 6. Pin the Playwright version and document the corresponding browser revision. Where supported, verify downloaded browser artifacts through trusted checksums or controlled internal artifact storage. 7. Add automated dependency scanning and update review. Version changes should trigger a new security review before lock files are updated. 8. Keep `skill.json`, `SKILL.md`, and the lock file synchronized so automated installation cannot silently use a broader dependency range than the documented setup. ]]>

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• System Prompt Leakage: Direct Leakage, Indirect Extraction, Tool-Based Exfiltration
• Rogue Agent: Self-Modification, Session Persistence

Findings (30)

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 94% confidence
• Finding: The declared description presents a complete usability testing agent with persona generation, workflow testing, outcome interpretation, and report generation. The supplied code does not implement any of that domain logic. It is only infrastructure for session management: loading Nova Act configuration from ~/.openclaw/config/nova-act.json, exporting an API key to the environment, creating a logs directory, and returning a NovaAct instance. While this wrapper could support a usability testing system, the code chunk itself is materially different in primary purpose and omits the core advertised capabilities. Therefore this is a clear description-behavior mismatch.

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 94% confidence
• Finding: The declared description presents a full usability-testing agent that generates personas, runs user-flow tests, interprets outcomes, and produces HTML reports. In contrast, this module is narrowly scoped to packaging raw responses for another agent, creating analysis prompts, and suggesting alternative prompts after failed attempts. The code comments repeatedly state that it does not interpret responses itself. While this chunk could be a supporting component of the larger described system, the specific behavior shown materially differs from the declared capability of response interpretation and broader end-to-end usability testing.

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 98% confidence
• Finding: The declared description presents a full usability-testing agent with AI orchestration, persona generation, workflow execution, response interpretation, and HTML report generation. The supplied code chunk only implements a background status reporter for long-running tests: it tracks phase/persona/test names, counts completed/passed/failed tests, and prints periodic and final text summaries. While such reporting could be a supporting component within a larger usability-testing system, this code chunk by itself does not perform the core declared functionality. Therefore, the code’s actual behavior is materially narrower and different from the declared primary purpose.

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 95% confidence
• Finding: The declared description presents a full usability-testing system that orchestrates tests, analyzes results, and generates reports. The supplied code chunk does none of that; it only performs local filesystem operations to locate HTML trace files in log directories and identify the latest session. While trace discovery could be a supporting component of a larger usability testing skill, this chunk’s actual behavior is narrowly focused on file lookup and lacks the core declared functionality. Therefore, the description does not accurately represent what this code chunk actually does.

Context Leakage

High

• Category: Data Exfiltration

Content

# Parse cookbook for actionable guidance (Bug #14: Use the cookbook!)
    cookbook_hints = parse_cookbook_hints(cookbook)
    
    # Extract context
    archetype = persona.get('archetype', 'user')
    tech_level = persona.get('tech_proficiency', 'medium')
    page_title = page_analysis.get('title', 'this page')

• Confidence: 75% confidence
• Finding: Code or instructions that leak agent conversation context to external services, potentially exposing sensitive user interactions.

Direct Prompt Extraction

High

• Category: System Prompt Leakage

Content

- Clicks, types, scrolls
    - Reports what it sees
    """
    # Return prompt unchanged - no persona-specific modifications
    # Persona context is used by the agent when interpreting results, not by Nova Act
    return base_prompt

• Confidence: 85% confidence
• Finding: Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Session Persistence

Medium

• Category: Rogue Agent

Content

✅ **Real Browser Automation** - Actual Playwright browser control via Nova Act  
✅ **Cookbook Integration** - Loads best practices and workflow patterns automatically  
✅ **Fully Dynamic Testing** - Exploration strategies generated per website/persona (no hardcoded logic!)  
✅ **Smart Persona Generation** - Analyzes page content to create relevant user types  
✅ **Adaptive Testing** - AI tries multiple variations when element text doesn't match exactly  
✅ **Robust Error Handling** - Handles scroll loops, timeouts, and Nova Act failures gracefully  
✅ **Detailed Reporting** - Professional HTML reports with step-by-step observations

• Confidence: 60% confidence
• Finding: Skill establishes unauthorized persistence across sessions via cron jobs, startup scripts, or state files. Session persistence allows an attacker to maintain access beyond the current interaction.

Intent-Code Divergence

Medium

• Confidence: 82% confidence
• Finding: The safety guarantees state the skill will never create real accounts and always stop before final action. However, the supported workflows section says the skill fills registration forms for account signup up to final submission, which is materially closer to account creation than the absolute 'NEVER' wording suggests and creates contradictory operator expectations about what actions the automation may perform.

Sudo/Root Execution

Medium

• Category: Privilege Escalation

Content

```

> **Note:** On Linux, you may also need system dependencies:
> `sudo playwright install-deps chromium`

### Step 3: Configure API key

• Confidence: 88% confidence
• Finding: The README instructs users to run a command with sudo, which normalizes privileged execution of externally installed tooling and dependencies. If copied blindly, this increases the blast radius of installation-time compromise or unintended system changes, especially on shared or sensitive hosts.

Vague Triggers

Medium

• Confidence: 93% confidence
• Finding: The README activates the skill with generic phrases like "Test https://example.com for usability" and "Run a usability test on example.com" without giving clearer scope, exclusions, or narrower trigger constraints. In a markdown skill description, this can create ambiguous invocation boundaries because many routine requests about websites could match.

Lp3

Medium

• Category: MCP Least Privilege
• Confidence: 93% confidence
• Finding: The skill instructs the agent to read a local API key file and to write logs, screenshots, JSON results, and HTML reports, but it declares no explicit tool scope or permission boundaries. In an agent ecosystem, undeclared file and environment access increases the chance of overbroad execution, accidental secret exposure, and misuse of local resources.

Intent-Code Divergence

Medium

• Confidence: 93% confidence
• Finding: The feature section at L057 and guidance around L190-L196 say the script should not make duplicate AI/API calls and that persona generation belongs to the already-running agent. Later, L239-L242 says the system will use Claude AI to infer personas when none are provided, contradicting the earlier documented design and intent.

Intent-Code Divergence

Medium

• Confidence: 96% confidence
• Finding: At L070 the documentation explicitly says the script does not interpret responses or generate the final report. However, later instructions at L557-L560 and example code at L731-L739 state that running the adaptive test script creates `nova_act_usability_report.html` and generates the HTML report, which is a direct contradiction in documented intent.

Intent-Code Divergence

Medium

• Confidence: 90% confidence
• Finding: The mandatory workflow at L388-L395 assigns interpretation and report generation to the agent after reading raw JSON. Yet L717-L739 describes the script's own flow as pooling/analyzing results and generating the report, creating a material contradiction about who performs core evaluation steps.

Session Persistence

Medium

• Category: Rogue Agent

Content

python3 "$SKILL_DIR/scripts/run_adaptive_test.py" "https://example.com"

# This will:
# - Create nova_act_logs/ in current directory
# - Create test_results_adaptive.json in current directory
# - Create nova_act_usability_report.html in current directory
# - Provide 60-second status updates during test

• Confidence: 90% confidence
• Finding: The skill explicitly creates persistent local artifacts including trace logs, screenshots, page HTML/content, JSON results, and reports in the working directory. Because the tested pages may contain PII, credentials, business data, or other sensitive content, indiscriminate retention of these artifacts materially increases exposure, especially on shared systems or when scanning untrusted or production-like sites.

Intent-Code Divergence

Medium

• Confidence: 96% confidence
• Finding: Lines L007-L015 explicitly state that Nova Act only executes browser actions and should not reason about personas, usability, or user importance. Later, lines L267-L270 describe Nova Act trace files as containing 'AI reasoning and decisions,' which directly conflicts with that earlier intent documentation and can mislead developers about where reasoning occurs.

Missing User Warnings

Medium

• Confidence: 97% confidence
• Finding: The documentation encourages enabling detailed HTML trace logging with screenshots, timelines, and session details, but it does not warn that these artifacts can capture sensitive information such as form contents, PII, account data, or internal pages. In a usability-testing skill that navigates real workflows and may fill forms, this materially increases the risk of inadvertent data exposure through stored logs, linked reports, or shared artifacts.

Intent-Code Divergence

Medium

• Confidence: 96% confidence
• Finding: The safety-stop helper suggests material-impact test cases should stop safely, but actual workflow generation advances real booking flows through search, selection, continuation, and personal-detail entry before halting at payment. This mismatch is dangerous because operators may rely on the documented safeguard while the code still performs meaningful real-world actions on live websites.

Intent-Code Divergence

Medium

• Confidence: 97% confidence
• Finding: The code claims workflow testing uses safety stops, but the generated purchasing and booking strategies still direct the browser agent to perform consequential actions such as adding items to a cart, selecting booking options, proceeding through checkout, and filling personal details before stopping. In a usability-testing skill, those intermediate actions can create side effects on third-party sites, trigger reservations, inventory holds, abandoned carts, lead generation, or account/session state changes even if final payment is not submitted.

Missing User Warnings

Medium

• Confidence: 98% confidence
• Finding: The purchasing workflow explicitly tells the agent to search for a product, open it, add it to cart, navigate to the cart, proceed to checkout, and fill shipping information before only stopping at payment. Those are state-changing actions on an external site and can create abandoned carts, reserve stock, trigger anti-fraud or marketing systems, and mishandle test PII in a live environment.

Missing User Warnings

Medium

• Confidence: 90% confidence
• Finding: This code accesses sensitive credentials from ~/.openclaw/config/nova-act.json and sets NOVA_ACT_API_KEY in the process environment, but there is no user-facing warning, prompt, or disclosure about handling credentials. The module docstring and runtime messages also do not mention this sensitive operation.

Missing User Warnings

Medium

• Confidence: 88% confidence
• Finding: The module formats and forwards full prompts and raw Nova Act responses directly into agent-visible output without any redaction, minimization, or sensitivity checks. In a usability-testing workflow, those raw responses can easily contain user-entered form data, account details, addresses, checkout information, or other sensitive page content, so emitting them wholesale increases the risk of inadvertent disclosure to logs, reports, downstream LLMs, or operators.

Context-Inappropriate Capability

Medium

• Confidence: 95% confidence
• Finding: The fallback persona generator makes an outbound call to Anthropic and consumes ANTHROPIC_API_KEY even though the skill’s stated purpose is Nova Act usability testing. This expands the trust boundary and can disclose analyzed website metadata to a third party without necessity, creating privacy, compliance, and secret-handling risk if users do not expect external model providers to receive that data.

Missing User Warnings

Medium

• Confidence: 95% confidence
• Finding: The code sends page-derived content such as title, purpose, and navigation to an external AI API without any explicit disclosure, consent, or policy check. In a usability-testing skill, analyzed pages may be internal, pre-release, or customer-owned, so silently exporting even summarized content can leak sensitive business information outside the expected execution environment.

Missing User Warnings

Medium

• Confidence: 92% confidence
• Finding: This manifest explicitly advertises browser automation and workflow testing for actions like booking, checkout, and posting, but does not warn that the skill may interact with live systems, submit forms, or touch user data. In context, that omission is meaningful because usability testing against real sites can trigger unintended purchases, submissions, account actions, or collection of sensitive information if operators are not clearly warned and constrained.

Static analysis

No suspicious patterns detected.