Website Usability Test Nova Act

• Outcome: Review
• Latest audit: Sep 10, 2026 · 7:33 PM UTC
• Version: 1.0.5

Overview

This appears to be a real usability-testing skill, but it needs review because it can automate live website actions, persist sensitive traces, and lacks strong scoping around external data sharing and generated reports.

Review before installing. Use only on websites you are authorized to test, preferably staging or sandbox environments. Avoid logged-in sessions, commerce flows, account creation, social posting, internal/private URLs, or pages with sensitive data until URL restrictions, safety-stop boundaries, report escaping, dependency pinning, and the broken main script are fixed. Treat generated traces and reports as sensitive files.

A.I.G

Vulnerability Patterns

• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Unauthorized Access and Privilege Escalation: Obtains permissions beyond the task's legitimate needs
• Insecure Dependencies: Introduces malicious components through unsafe dependency sources
• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions

Findings (8)

T09 · Insecure Skill Coding Practices

Error

• Location: scripts/enhanced_report_generator.py:410

Finding

Stored HTML and JavaScript Injection in Enhanced Reports

Content

View full analysis

📄 Page Analysis: {page_analysis.get('title', 'Unknown')}

Purpose: {page_analysis.get('purpose', 'Not analyzed')}

Navigation: {', '.join(page_analysis.get('navigation', ['None found']))}

""" ``` ```python if observations_list: notes = '; '.join(str(o) for o in observations_list) elif raw_response: notes = f"Response: {raw_response}" elif error_msg: notes = f"Error: {error_msg}" action = step.get('action') or step.get('prompt', '') or 'No action' rationale = step.get('rationale') or step.get('expected_outcome', '') html += f"""

Step {step_num + 1}: {action_display}

{f'

Expected: {rationale}

' if rationale else ''}

{"⚠️ " if is_issue else ""}Observation: {notes}

""" ``` ```python html += f""" 📹 Recording {global_recording_index}: {display_name} """ ``` ### Technical Analysis The report generator directly interpolates website-derived page titles, navigation text, Nova Act responses, errors, persona fields, test cases, and trace paths into HTML. It does not apply HTML escaping, attribute escaping, URI-scheme validation, or sanitization. These values cross an untrusted-data boundary because they may originate from an attacker-controlled website or crafted results/persona file. An injected value such as an image element with an event handler will become active markup when the ...[truncated 871 chars]

Remediation

View remediation

T09 · Insecure Skill Coding Practices

Error

• Location: scripts/generate_report.py:92

Finding

Stored HTML and JavaScript Injection in Legacy Report Generator

Content

View full analysis

0 else 0 persona_findings += f"""

{persona_name}

Tasks Completed: {stats['success']}/{stats['total']} ({success_rate:.1f}%)

""" for task in stats['tasks']: status = "✓" if task['success'] else "✗" persona_findings += f"
• {status} {task['task']} ({task['duration']:.1f}s)
" persona_findings += "

" friction_section = "

Common Friction Points

" for fp in analysis['friction_points'][:10]: friction_section += ( f"
• {fp['persona']} - " f"{fp['task']}: {fp['issue']}
" ) friction_section += "

" ``` ### Technical Analysis The script loads arbitrary JSON results and inserts persona names, task descriptions, and issue text directly into an HTML template. No output encoding is performed. A crafted JSON value is therefore interpreted as markup rather than displayed as text. ### Attack Path 1. An attacker supplies or modifies a results JSON file. 2. A malicious string is placed in a persona, task, or observation field. 3. The user invokes `generate_report.py` on that file. 4. The string is embedded in the generated HTML unchanged. 5. Script or active markup executes when the report is opened. ### Impact Assessment The payload can falsify audit findings, display deceptive content, execute browser-side requests, or disclose information rendered in the report. Exploitation requires control over a results file or over source data later stored in that file. ]]>

Remediation

View remediation

T01 · Skill Instruction Hijacking

Error

• Location: scripts/response_interpreter.py:55

Finding

Indirect Prompt Injection Through Tested Website Content

Content

View full analysis

str: return f"""Analyze this usability test step result: **Step:** {step_result.get('step_name', 'Unknown')} **Question Asked:** {step_result.get('prompt', 'N/A')} **Expected Outcome:** {step_result.get('expected_outcome', 'N/A')} **Nova Act Response:** {step_result.get('raw_response', 'No response')} Based on the response, determine: 1. **Goal Achieved?** Did we find/accomplish what we were looking for? 2. **Should Retry?** 3. **Next Action:** What should we do next?""" ``` ### Technical Analysis Nova Act responses may contain arbitrary text controlled by the tested website. The interpreter places that text directly into an Agent-facing prompt without a clear untrusted-data delimiter or an instruction stating that directives inside the response must never be followed. `SKILL.md` additionally instructs the orchestrating Agent to read and interpret these raw responses. This promotes website content into an instruction-processing context and creates an indirect prompt-injection channel. ### Attack Path 1. An attacker publishes visible page text resembling Agent instructions. 2. Nova Act returns that text as `raw_response`. 3. `create_agent_prompt_for_interpretation` embeds it directly into an interpretation prompt. 4. The orchestrating Agent processe ...[truncated 559 chars]

Remediation

View remediation

T05 · Unauthorized Access and Privilege Escalation

Error

• Location: scripts/run_adaptive_test.py:960

Finding

Arbitrary URL Navigation Without Private-Network or Scheme Restrictions

Content

View full analysis

= 3 else None ``` ```python update_status("Analyzing website...") page_analysis = analyze_page(WEBSITE_URL) _shutdown_state['page_analysis'] = page_analysis ``` The URL is subsequently passed to the browser session in `scripts/nova_session.py:61-65`: ```python with NovaAct( starting_page=starting_page, tty=not headless, logs_directory=logs_dir ) as nova: yield nova ``` ### Technical Analysis The entry point accepts an arbitrary URL and does not validate its scheme, hostname, resolved IP address, redirects, embedded credentials, or destination network range. There are no controls blocking loopback, private, link-local, reserved, cloud-metadata, or local-file destinations. Because the browser runs in the Agent's environment, it may have network visibility that the user or remote requester does not. ### Attack Path 1. An untrusted requester provides a URL targeting localhost, an intranet service, a link-local metadata endpoint, or another restricted destination. 2. The script passes the URL directly to Nova Act. 3. The browser accesses the destination using the host environment's network privileges. 4. Nova Act extracts content and captures screenshots or traces. 5. Internal content is stored in results and reports, where it may be disclosed to the requester or an additional API. ### Impact Assessment The browser may read services reachable from the execution environment, including local administration interfaces, internal applications, and metadata endpoints. The exact privileges depend on network placement and browser protocol support. Captured internal content can persist in screenshots, HTML traces, JSON results, ...[truncated 16 chars]

Remediation

View remediation

T09 · Insecure Skill Coding Practices

Error

• Location: scripts/dynamic_exploration.py:242

Finding

Ambiguous Browser Actions Can Violate Material-Impact Safety Guarantees

Content

View full analysis

Remediation

View remediation

T09 · Insecure Skill Coding Practices

Warning

• Location: scripts/safe_nova_wrapper.py:84

Finding

Advertised Nova Act Timeouts Are Not Enforced

Content

View full analysis

ActResult: for attempt in range(max_retries): try: start = time.time() result = nova.act(action) duration = time.time() - start if duration > SLOW_OPERATION_THRESHOLD: return ActResult( success=False, observation=observation, error=f"Action took {duration:.1f}s (possible scroll loop or hang)", duration=duration ) ``` ```python def safe_act_get(nova, query: str, schema: Any, timeout: int = DEFAULT_TIMEOUT, max_retries: int = DEFAULT_MAX_RETRIES) -> QueryResult: for attempt in range(max_retries): try: start = time.time() result = nova.act_get(query, schema=schema) duration = time.time() - start ``` ### Technical Analysis Although both functions accept a `timeout` argument, they never use it. Duration is checked only after `nova.act` or `nova.act_get` returns. Consequently, a hung SDK call cannot be interrupted by this wrapper. A `with_timeout` decorator exists elsewhere in the file but is not applied to these functions, and its Windows fallback would run without any timeout. ### Attack Path 1. A malicious or defective page causes a Nova Act operation to wait indefinitely. 2. The wrapper blocks inside `nova.act` or `nova.act_get`. 3. The post-operation duration check is never reached. 4. The process remains occupied until an external mechanism terminates it. ### Impact Assessment This can cause denial of service, prolonged browser and API resource consumption, incomplete re ...[truncated 139 chars]

Remediation

View remediation

T08 · Insecure Dependencies

Warning

• Location: SKILL.md:18

Finding

Unpinned Third-Party Runtime Dependencies

Content

View full analysis

Remediation

View remediation

T09 · Insecure Skill Coding Practices

Warning

• Location: scripts/run_adaptive_test.py:290

Finding

Undisclosed Transfer of Tested Page Data to Anthropic

Content

View full analysis

Remediation

View remediation

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• System Prompt Leakage: Direct Leakage, Indirect Extraction, Tool-Based Exfiltration
• Rogue Agent: Self-Modification, Session Persistence

Findings (26)

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 95% confidence

Finding

The declared description presents a full usability-testing agent with specialized behaviors: generating personas, running tests, analyzing whether goals were achieved, and producing HTML reports for specific UX workflows. The supplied code does not implement any of that domain logic. Instead, it is infrastructure code only: a session/context manager for Nova Act that loads credentials from a local config file, sets NOVA_ACT_API_KEY, prepares a logs directory, and yields a NovaAct instance. While such a wrapper could support a usability-testing system, this chunk by itself does not perform the described testing, analysis, or reporting. The code also accesses local configuration and writes logs, which are concrete behaviors/resources absent from the declared description. Therefore, the description materially overstates and misrepresents what this code chunk actually does.

Content

No source excerpt is available for this finding.

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 95% confidence

Finding

The declared description presents a full usability-testing system that generates personas, executes user workflow tests, interprets results, and produces reports. In contrast, the supplied code is a narrow helper module for packaging raw responses and prompting an external orchestrating agent to do the actual interpretation. The module explicitly states it 'does NOT interpret responses itself.' It provides dataclasses, formatting functions, prompt templates, and simple alternative prompt generation logic. This is materially narrower and different from the declared behavior, especially around autonomous interpretation, persona generation, and report generation. Therefore, this chunk does not accurately represent the declared purpose.

Content

No source excerpt is available for this finding.

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 98% confidence

Finding

The supplied code is a status-reporting helper, not a usability-testing agent. It starts a background thread, emits periodic progress updates, tracks counters for tests, and prints a final summary. While some field names reference personas and tests, the module itself does not perform the declared core functions of generating personas, executing website workflows, collecting raw data, analyzing responses, or producing HTML reports. This is a materially different primary purpose from the declared description, so it should be flagged as a mismatch.

Content

No source excerpt is available for this finding.

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 94% confidence

Finding

The declared description presents a full usability-testing agent with orchestration, persona generation, workflow testing, analysis, and reporting. The supplied code does none of that directly. It only searches local directories for HTML trace files associated with Nova Act sessions and returns file paths sorted by modification time. While this could be a supporting helper within a larger usability-testing system, the chunk’s actual purpose is materially narrower and different from the declared primary purpose. Therefore, this code chunk does not accurately represent the described skill behavior on its own.

Content

No source excerpt is available for this finding.

Context Leakage

High

• Category: Data Exfiltration
• Confidence: 75% confidence

Finding

Code or instructions that leak agent conversation context to external services, potentially exposing sensitive user interactions.

Content

Scanner excerpt · scripts/dynamic_exploration.py (reported line 115)May include surrounding context.

python

# Parse cookbook for actionable guidance (Bug #14: Use the cookbook!)
    cookbook_hints = parse_cookbook_hints(cookbook)
    
    # Extract context
    archetype = persona.get('archetype', 'user')
    tech_level = persona.get('tech_proficiency', 'medium')
    page_title = page_analysis.get('title', 'this page')

Direct Prompt Extraction

High

• Category: System Prompt Leakage
• Confidence: 85% confidence

Finding

Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Content

Scanner excerpt · scripts/dynamic_exploration.py (reported line 704)May include surrounding context.

python

- Clicks, types, scrolls
    - Reports what it sees
    """
    # Return prompt unchanged - no persona-specific modifications
    # Persona context is used by the agent when interpreting results, not by Nova Act
    return base_prompt

Session Persistence

Medium

• Category: Rogue Agent
• Confidence: 60% confidence

Finding

Skill establishes unauthorized persistence across sessions via cron jobs, startup scripts, or state files. Session persistence allows an attacker to maintain access beyond the current interaction.

Content

Scanner excerpt · README.md (reported line 51)May include surrounding context.

md

✅ **Real Browser Automation** - Actual Playwright browser control via Nova Act  
✅ **Cookbook Integration** - Loads best practices and workflow patterns automatically  
✅ **Fully Dynamic Testing** - Exploration strategies generated per website/persona (no hardcoded logic!)  
✅ **Smart Persona Generation** - Analyzes page content to create relevant user types  
✅ **Adaptive Testing** - AI tries multiple variations when element text doesn't match exactly  
✅ **Robust Error Handling** - Handles scroll loops, timeouts, and Nova Act failures gracefully  
✅ **Detailed Reporting** - Professional HTML reports with step-by-step observations

Sudo/Root Execution

Medium

• Category: Privilege Escalation
• Confidence: 70% confidence

Finding

Commands invoke sudo or root privileges. Verify this elevated access is necessary and justified.

Content

Scanner excerpt · README.md (reported line 103)May include surrounding context.

text

> **Note:** On Linux, you may also need system dependencies:
> `sudo playwright install-deps chromium`

### Step 3: Configure API key

Vague Triggers

Medium

• Category: Not specified by scanner
• Confidence: 88% confidence

Finding

The README suggests activation via generic requests like "Test https://example.com for usability" and "Run a usability test on example.com" without defining stricter trigger conditions, exclusions, or required context. Because this is a markdown file and no negative examples or narrow invocation scope are provided, the activation guidance is somewhat ambiguous and could overlap with ordinary requests to test a site.

Content

No source excerpt is available for this finding.

Undeclared Tool Scope

Medium

• Category: MCP Least Privilege
• Confidence: 90% confidence

Finding

The skill requests access to environment data and performs file reads/writes, including reading a local API key file and writing logs/results, but it declares no explicit tool scope or permissions. This weakens least-privilege controls and can allow broader-than-expected access to sensitive local data or persistence of sensitive artifacts without clear user consent.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium

• Category: Not specified by scanner
• Confidence: 91% confidence

Finding

At L070 the documentation says the script does not interpret responses or generate the final report, requiring the agent to do phases 3-4 manually. But L557-L560 and L573-L576 state that running the script will create nova_act_usability_report.html and even generate a partial report on interruption, and L725-L741 shows the workflow saving results and generating the HTML report. These statements actively contradict each other about whether report generation is automatic.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium

• Category: Not specified by scanner
• Confidence: 88% confidence

Finding

L185-L196 explicitly instructs the agent not to let the Python script make its own AI call and not to rely on generic fallback personas, framing agent-generated personas as the intended design. However, L235-L246 and L629-L634 describe the system/script as automatically generating contextual personas, including using Claude AI when no custom persona is supplied. This is an intent-level contradiction in the skill's own documentation.

Content

No source excerpt is available for this finding.

Session Persistence

Medium

• Category: Rogue Agent
• Confidence: 92% confidence

Finding

The skill explicitly creates persistent local artifacts including trace logs, raw test results, and an HTML report in the current working directory. Because the traces are documented to include screenshots, full page content, and AI/browser actions, sensitive data from tested sites can be retained on disk and later exposed to other local users, processes, backups, or accidental sharing.

Content

Scanner excerpt · SKILL.md (reported line 558)May include surrounding context.

md

python3 "$SKILL_DIR/scripts/run_adaptive_test.py" "https://example.com"

# This will:
# - Create nova_act_logs/ in current directory
# - Create test_results_adaptive.json in current directory
# - Create nova_act_usability_report.html in current directory
# - Provide 60-second status updates during test

Intent-Code Divergence

Medium

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

The document explicitly states Nova Act should not evaluate usability or judge whether tasks are easy or hard, reserving that reasoning for the agent. Later examples contradict this by asking Nova Act questions like whether menu items are 'clearly labeled and easy to understand' and whether search results are 'relevant,' which are evaluative judgments rather than raw browser observations.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

The purchasing workflow executes material actions such as adding an item to cart, navigating to the cart, and clicking 'Proceed to Checkout' before the safety stop is reached. In a usability-testing skill that drives a real browser, these steps can modify user state, reserve inventory, trigger promotions, or advance a live commerce transaction farther than intended even if payment is not submitted.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 92% confidence

Finding

The signup workflow auto-fills registration credentials and navigates into the registration flow before only stopping at the final submit button. Even without final submission, populating identity fields on a live registration form can trigger validation calls, availability checks, bot defenses, analytics events, or accidental account-creation side effects if the site auto-submits or uses multi-step progression.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 92% confidence

Finding

The script writes the modified results back to the same path supplied by the user, with no confirmation, backup, or alternate output file. This can irreversibly destroy original test data or corrupt unrelated JSON files if the operator provides the wrong path, which is a real integrity risk even if it is not a code-execution issue.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium

• Category: Not specified by scanner
• Confidence: 90% confidence

Finding

The manifest describes an AI usability-testing skill, but this helper adds a separate capability to retrieve credentials from ~/.openclaw/config/nova-act.json and later inject them into environment state. While the skill may need Nova Act authentication internally, accessing arbitrary local user config and managing secrets via environment variables is not described as part of the skill's purpose.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium

• Category: Not specified by scanner
• Confidence: 91% confidence

Finding

The fallback persona generation sends page-derived content such as the website title, purpose, and navigation to Anthropic, which is an external AI provider outside the skill's stated Nova Act testing flow. This creates an undisclosed data egress path that may expose sensitive site content, internal URLs, or customer environments to a third party when users reasonably expect testing to stay within the Nova Act workflow.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

Website-derived content is transmitted to an external AI API without a user-facing warning, confirmation, or consent mechanism. In a usability-testing skill, this is more dangerous because users may run tests against private staging sites, authenticated pages, or proprietary product content and may not expect third-party sharing during persona generation.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium

• Category: Not specified by scanner
• Confidence: 97% confidence

Finding

The function documentation says it does not interpret responses and that the orchestrating AI agent should analyze raw responses to determine goal achievement. However, later in the same test flow the code heuristically sets goal_achieved and computes overall_success from raw text, contradicting the documented intent of agent-only interpretation.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Low

• Category: Not specified by scanner
• Confidence: 82% confidence

Finding

Earlier guidance frames Nova Act as a browser automation tool that only executes actions and reports observations, not reasoning. Describing trace files as containing 'AI reasoning and decisions' creates intent-level inconsistency about whether Nova Act itself is performing reasoning.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low

• Category: Not specified by scanner
• Confidence: 84% confidence

Finding

The posting workflow opens the compose UI and types content into it before the safety stop. Although the code avoids clicking 'Post', entering text into live communication interfaces can still create drafts, trigger autosave, invoke moderation or notification systems, or leak test content into shared environments.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low

• Category: Not specified by scanner
• Confidence: 88% confidence

Finding

The report embeds local trace file paths directly into HTML, including absolute Linux or WSL paths such as file://... or file://wsl$/..., which can expose filesystem layout, usernames, distro names, and session directory structure to anyone who views or shares the report. In this usability-testing context, traces may also reveal sensitive artifacts from test sessions, so making local paths clickable increases the chance of unintended disclosure beyond the intended audience.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low

• Category: Not specified by scanner
• Confidence: 88% confidence

Finding

The function unconditionally writes a generated HTML report to the current working directory, and that report includes test cases, observations, personas, and trace-link metadata derived from inputs. While the module docstring describes report generation, there is no user-facing prompt, log, or warning at the point of file creation to disclose that local data will be written.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.