Website Usability Test Nova Act

• Outcome: Review
• Latest audit: Sep 10, 2026 · 7:33 PM UTC
• Version: 1.0.5

Overview

This appears to be a real usability-testing skill, but it needs review because it can automate live website actions, persist sensitive traces, and lacks strong scoping around external data sharing and generated reports.

Review before installing. Use only on websites you are authorized to test, preferably staging or sandbox environments. Avoid logged-in sessions, commerce flows, account creation, social posting, internal/private URLs, or pages with sensitive data until URL restrictions, safety-stop boundaries, report escaping, dependency pinning, and the broken main script are fixed. Treat generated traces and reports as sensitive files.

A.I.G

Vulnerability Patterns

• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Unauthorized Access and Privilege Escalation: Obtains permissions beyond the task's legitimate needs
• Insecure Dependencies: Introduces malicious components through unsafe dependency sources
• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions

Findings (8)

T09 · Insecure Skill Coding Practices

Error

• Location: scripts/enhanced_report_generator.py:410
• Finding: Stored HTML and JavaScript Injection in Enhanced Reports<![CDATA[ ## Vulnerability Details **File Location**: `scripts/enhanced_report_generator.py:410-412, 556-561, 593, 629-670, 697-701, 745, 764-769` **Vulnerability Type**: Stored cross-site scripting and HTML injection **Risk Level**: High ### Vulnerable Code ```python html += f""" <div class="page-analysis"> <h3>📄 Page Analysis: {page_analysis.get('title', 'Unknown')}</h3> <p><strong>Purpose:</strong> {page_analysis.get('purpose', 'Not analyzed')}</p> <p><strong>Navigation:</strong> {', '.join(page_analysis.get('navigation', ['None found']))}</p> """ ``` ```python if observations_list: notes = '; '.join(str(o) for o in observations_list) elif raw_response: notes = f"Response: {raw_response}" elif error_msg: notes = f"Error: {error_msg}" action = step.get('action') or step.get('prompt', '') or 'No action' rationale = step.get('rationale') or step.get('expected_outcome', '') html += f""" <div class="observation"> <div class="observation-header"> <span class="step-name">Step {step_num + 1}: {action_display}</span> </div> {f'<div>Expected: {rationale}</div>' if rationale else ''} <div class="observation-notes {notes_class}"> <strong>{"⚠️ " if is_issue else ""}Observation:</strong> {notes} </div> </div> """ ``` ```python html += f""" <a href="{browser_path}" class="trace-link" target="_blank"> 📹 Recording {global_recording_index}: {display_name} </a> """ ``` ### Technical Analysis The report generator directly interpolates website-derived page titles, navigation text, Nova Act responses, errors, persona fields, test cases, and trace paths into HTML. It does not apply HTML escaping, attribute escaping, URI-scheme validation, or sanitization. These values cross an untrusted-data boundary because they may originate from an attacker-controlled website or crafted results/persona file. An injected value such as an image element with an event handler will become active markup when the ...[truncated 871 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions - Escape every untrusted text value with `html.escape(value, quote=True)` before interpolation. - Use an auto-escaping template engine such as Jinja2 instead of manually concatenating HTML. - Apply context-specific escaping separately for text nodes and attributes. - Restrict links to approved schemes such as `file`, `https`, or relative paths after canonicalization. - Add a restrictive Content Security Policy that blocks inline scripts and remote connections. - Add regression tests using payloads containing tags, quotes, event handlers, and `javascript:` URIs. ]]>

T09 · Insecure Skill Coding Practices

Error

• Location: scripts/generate_report.py:92
• Finding: Stored HTML and JavaScript Injection in Legacy Report Generator<![CDATA[ ## Vulnerability Details **File Location**: `scripts/generate_report.py:92-113` **Vulnerability Type**: Stored cross-site scripting and HTML injection **Risk Level**: High ### Vulnerable Code ```python for persona_name, stats in analysis['persona_stats'].items(): success_rate = (stats['success'] / stats['total'] * 100) if stats['total'] > 0 else 0 persona_findings += f""" <div class="persona-section"> <h3>{persona_name}</h3> <p><strong>Tasks Completed:</strong> {stats['success']}/{stats['total']} ({success_rate:.1f}%)</p> <ul> """ for task in stats['tasks']: status = "✓" if task['success'] else "✗" persona_findings += f"<li>{status} {task['task']} ({task['duration']:.1f}s)</li>" persona_findings += "</ul></div>" friction_section = "<h2>Common Friction Points</h2><ul>" for fp in analysis['friction_points'][:10]: friction_section += ( f"<li><strong>{fp['persona']}</strong> - " f"{fp['task']}: {fp['issue']}</li>" ) friction_section += "</ul>" ``` ### Technical Analysis The script loads arbitrary JSON results and inserts persona names, task descriptions, and issue text directly into an HTML template. No output encoding is performed. A crafted JSON value is therefore interpreted as markup rather than displayed as text. ### Attack Path 1. An attacker supplies or modifies a results JSON file. 2. A malicious string is placed in a persona, task, or observation field. 3. The user invokes `generate_report.py` on that file. 4. The string is embedded in the generated HTML unchanged. 5. Script or active markup executes when the report is opened. ### Impact Assessment The payload can falsify audit findings, display deceptive content, execute browser-side requests, or disclose information rendered in the report. Exploitation requires control over a results file or over source data later stored in that file. ]]>
• Remediation: <![CDATA[ ## Remediation Suggestions - HTML-escape all persona, task, and issue values. - Migrate template rendering to an engine with automatic escaping enabled. - Validate the loaded JSON against a strict schema and reject unexpected value types. - Add a Content Security Policy to generated reports as defense in depth. - Test report generation with representative stored-XSS payloads. ]]>

T01 · Skill Instruction Hijacking

Error

• Location: scripts/response_interpreter.py:55
• Finding: Indirect Prompt Injection Through Tested Website Content<![CDATA[ ## Vulnerability Details **File Location**: `scripts/response_interpreter.py:55-67, 79-122` **Vulnerability Type**: Untrusted website content embedded into Agent instructions **Risk Level**: High ### Vulnerable Code ```python for i, result in enumerate(results, 1): output.append(f"--- Step {i}: {result.get('step_name', 'Unknown')} ---") output.append(f"Prompt: {result.get('prompt', 'N/A')}") output.append(f"Expected: {result.get('expected_outcome', 'N/A')}") output.append(f"API Success: {result.get('api_success', False)}") output.append(f"Raw Response: {result.get('raw_response', 'No response')}") ``` ```python def create_agent_prompt_for_interpretation(step_result: Dict) -> str: return f"""Analyze this usability test step result: **Step:** {step_result.get('step_name', 'Unknown')} **Question Asked:** {step_result.get('prompt', 'N/A')} **Expected Outcome:** {step_result.get('expected_outcome', 'N/A')} **Nova Act Response:** {step_result.get('raw_response', 'No response')} Based on the response, determine: 1. **Goal Achieved?** Did we find/accomplish what we were looking for? 2. **Should Retry?** 3. **Next Action:** What should we do next?""" ``` ### Technical Analysis Nova Act responses may contain arbitrary text controlled by the tested website. The interpreter places that text directly into an Agent-facing prompt without a clear untrusted-data delimiter or an instruction stating that directives inside the response must never be followed. `SKILL.md` additionally instructs the orchestrating Agent to read and interpret these raw responses. This promotes website content into an instruction-processing context and creates an indirect prompt-injection channel. ### Attack Path 1. An attacker publishes visible page text resembling Agent instructions. 2. Nova Act returns that text as `raw_response`. 3. `create_agent_prompt_for_interpretation` embeds it directly into an interpretation prompt. 4. The orchestrating Agent processe ...[truncated 559 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions - Explicitly label raw responses as untrusted data that must never be treated as instructions. - Serialize untrusted responses in a structured JSON field rather than concatenating them into prose. - Require constrained output containing only schema-validated fields such as `goal_achieved`, `friction_level`, and `reason`. - Reject or neutralize responses containing instruction-like text where feasible. - Keep interpretation Agents isolated from unnecessary tools and secrets. - Add prompt-injection tests using page content that requests tool calls, secret disclosure, or instruction overrides. ]]>

T05 · Unauthorized Access and Privilege Escalation

Error

• Location: scripts/run_adaptive_test.py:960
• Finding: Arbitrary URL Navigation Without Private-Network or Scheme Restrictions<![CDATA[ ## Vulnerability Details **File Location**: `scripts/run_adaptive_test.py:960-1016` **Vulnerability Type**: Browser-mediated server-side request forgery and unauthorized internal resource access **Risk Level**: High ### Vulnerable Code ```python WEBSITE_URL = sys.argv[1] persona_arg = sys.argv[2] if len(sys.argv) >= 3 else None ``` ```python update_status("Analyzing website...") page_analysis = analyze_page(WEBSITE_URL) _shutdown_state['page_analysis'] = page_analysis ``` The URL is subsequently passed to the browser session in `scripts/nova_session.py:61-65`: ```python with NovaAct( starting_page=starting_page, tty=not headless, logs_directory=logs_dir ) as nova: yield nova ``` ### Technical Analysis The entry point accepts an arbitrary URL and does not validate its scheme, hostname, resolved IP address, redirects, embedded credentials, or destination network range. There are no controls blocking loopback, private, link-local, reserved, cloud-metadata, or local-file destinations. Because the browser runs in the Agent's environment, it may have network visibility that the user or remote requester does not. ### Attack Path 1. An untrusted requester provides a URL targeting localhost, an intranet service, a link-local metadata endpoint, or another restricted destination. 2. The script passes the URL directly to Nova Act. 3. The browser accesses the destination using the host environment's network privileges. 4. Nova Act extracts content and captures screenshots or traces. 5. Internal content is stored in results and reports, where it may be disclosed to the requester or an additional API. ### Impact Assessment The browser may read services reachable from the execution environment, including local administration interfaces, internal applications, and metadata endpoints. The exact privileges depend on network placement and browser protocol support. Captured internal content can persist in screenshots, HTML traces, JSON results, ...[truncated 16 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions - Permit only explicitly supported `https` and, if necessary, `http` URLs. - Reject URLs containing embedded credentials or nonstandard schemes. - Resolve all hostnames and block loopback, private, link-local, multicast, reserved, and metadata address ranges for both IPv4 and IPv6. - Repeat validation after every redirect and guard against DNS rebinding. - Require explicit authorization before testing internal destinations. - Run the browser in a network-isolated sandbox with egress restrictions. ]]>

T09 · Insecure Skill Coding Practices

Error

• Location: scripts/dynamic_exploration.py:242
• Finding: Ambiguous Browser Actions Can Violate Material-Impact Safety Guarantees<![CDATA[ ## Vulnerability Details **File Location**: `scripts/dynamic_exploration.py:242-286` **Vulnerability Type**: Unsafe state-changing browser automation **Risk Level**: High ### Vulnerable Code ```python { "step_name": "select_product", "action_type": "navigate", "prompt": "Click on the first product in the results", "expected_outcome": "Product page loaded", "fallback_prompts": [] }, { "step_name": "add_to_cart", "action_type": "navigate", "prompt": "Click the 'Add to Cart' or 'Buy' button", "expected_outcome": "Item added to cart", "fallback_prompts": [] }, { "step_name": "navigate_to_cart", "action_type": "navigate", "prompt": "Click on the cart icon or 'View Cart' button", "expected_outcome": "Cart page loaded", "fallback_prompts": [] }, { "step_name": "proceed_to_checkout", "action_type": "navigate", "prompt": "Click 'Proceed to Checkout' or 'Checkout' button", "expected_outcome": "Checkout initiated", "fallback_prompts": [] }, { "step_name": "fill_shipping", "action_type": "navigate", "prompt": "Fill shipping information: Name 'Test User', Address '123 Test St', City 'Test City', ZIP '12345'", "expected_outcome": "Shipping filled", "fallback_prompts": [] }, { "step_name": "verify_payment_page", "action_type": "query", "prompt": "Is there a payment method section, credit card form, or 'Complete Purchase' button visible?", "expected_outcome": "⚠️ SAFETY STOP: Payment page accessible but NO PURCHASE MADE", "fallback_prompts": [], "is_safety_stop": True } ``` ### Technical Analysis The safety stop is applied only to the final verification query. Earlier steps execute state-changing actions, including clicking an ambiguous `Buy` control and initiating checkout. On one-click purchasing interfaces, a `Buy` button may immediately place an order or invoke another material action. The system assumes that every target follows a conve ...[truncated 867 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions - Never permit automation to click controls labeled `Buy`, `Order`, `Pay`, `Submit`, or similarly ambiguous material-action terms. - Make transactional workflows observation-only by default. - Inspect candidate controls and surrounding text before interaction. - Require explicit human confirmation immediately before any state-changing action. - Use isolated test accounts and non-production environments with payment methods disabled. - Add domain-specific allowlists and verify that a target is an authorized test environment. - Stop before checkout initiation rather than only before the final payment page. ]]>

T09 · Insecure Skill Coding Practices

Warning

• Location: scripts/safe_nova_wrapper.py:84
• Finding: Advertised Nova Act Timeouts Are Not Enforced<![CDATA[ ## Vulnerability Details **File Location**: `scripts/safe_nova_wrapper.py:84-171` **Vulnerability Type**: Missing operation cancellation and resource exhaustion protection **Risk Level**: Medium ### Vulnerable Code ```python def safe_act(nova, action: str, timeout: int = DEFAULT_TIMEOUT, max_retries: int = DEFAULT_MAX_RETRIES) -> ActResult: for attempt in range(max_retries): try: start = time.time() result = nova.act(action) duration = time.time() - start if duration > SLOW_OPERATION_THRESHOLD: return ActResult( success=False, observation=observation, error=f"Action took {duration:.1f}s (possible scroll loop or hang)", duration=duration ) ``` ```python def safe_act_get(nova, query: str, schema: Any, timeout: int = DEFAULT_TIMEOUT, max_retries: int = DEFAULT_MAX_RETRIES) -> QueryResult: for attempt in range(max_retries): try: start = time.time() result = nova.act_get(query, schema=schema) duration = time.time() - start ``` ### Technical Analysis Although both functions accept a `timeout` argument, they never use it. Duration is checked only after `nova.act` or `nova.act_get` returns. Consequently, a hung SDK call cannot be interrupted by this wrapper. A `with_timeout` decorator exists elsewhere in the file but is not applied to these functions, and its Windows fallback would run without any timeout. ### Attack Path 1. A malicious or defective page causes a Nova Act operation to wait indefinitely. 2. The wrapper blocks inside `nova.act` or `nova.act_get`. 3. The post-operation duration check is never reached. 4. The process remains occupied until an external mechanism terminates it. ### Impact Assessment This can cause denial of service, prolonged browser and API resource consumption, incomplete re ...[truncated 139 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions - Use SDK-native deadlines and cancellation if supported. - Otherwise, run each browser operation in a separate killable worker process. - Terminate the browser session when a deadline expires. - Ensure timeout behavior works consistently across supported operating systems. - Remove unused timeout code or apply it correctly. - Add tests that simulate indefinitely blocked actions and verify forced termination. ]]>

T08 · Insecure Dependencies

Warning

• Location: SKILL.md:18
• Finding: Unpinned Third-Party Runtime Dependencies<![CDATA[ ## Vulnerability Details **File Location**: `SKILL.md:18-28, 518-536, 839-850` **Vulnerability Type**: Unconstrained software supply chain **Risk Level**: Medium ### Vulnerable Code ```markdown | **Dependencies** | `pip3 install nova-act pydantic playwright` | | **Browser** | `playwright install chromium` (~300MB download) | ``` ```bash pip3 install nova-act pydantic playwright playwright install chromium ``` The package declaration in `skill.json:17-20` is also unconstrained: ```json "packages": [ "nova-act", "playwright" ] ``` ### Technical Analysis The setup instructions install packages and browser artifacts without exact versions, hashes, a lock file, or an approved package index. The effective code installed in future runs can therefore change without a corresponding change to the reviewed Skill. The browser automation and Nova Act packages execute with the same privileges as the Agent and receive access to the Nova API key through the environment. ### Attack Path 1. A dependency account, release, package index, or transitive dependency is compromised. 2. A user follows the documented unpinned installation command. 3. The latest compromised package is installed. 4. Malicious installation or runtime code executes with the Agent's privileges. 5. Local files, API keys, browser traces, or network access may be exposed. ### Impact Assessment A compromised dependency can obtain the full privileges of the Python process, including access to the Nova Act API key environment, local configuration, generated traces, browser sessions, and permitted network resources. ]]>
• Remediation: <![CDATA[ ## Remediation Suggestions - Pin exact versions for all direct and transitive dependencies. - Provide a lock file and use hash verification such as `pip --require-hashes`. - Restrict installation to an approved package index or internal mirror. - Pin and verify Playwright browser artifacts. - Generate and review a software bill of materials. - Run dependency installation and browser automation in a sandbox with limited filesystem and network access. ]]>

T09 · Insecure Skill Coding Practices

Warning

• Location: scripts/run_adaptive_test.py:290
• Finding: Undisclosed Transfer of Tested Page Data to Anthropic<![CDATA[ ## Vulnerability Details **File Location**: `scripts/run_adaptive_test.py:290-329` **Vulnerability Type**: Unintended third-party data disclosure **Risk Level**: Medium ### Vulnerable Code ```python purpose = page_analysis.get('purpose', 'Unknown') title = page_analysis.get('title', 'Unknown') navigation = ', '.join(page_analysis.get('navigation', [])) prompt = f"""Based on this website analysis, identify the 3 most plausible user types who would visit this site in real life: Website Title: {title} Purpose: {purpose} Navigation: {navigation} ... """ ``` ```python try: import anthropic api_key = os.environ.get('ANTHROPIC_API_KEY') if not api_key: print(" ⚠️ ANTHROPIC_API_KEY not set, using fallback personas") return [] client = anthropic.Anthropic(api_key=api_key) response = client.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=2000, messages=[{ "role": "user", "content": prompt }] ) ``` ### Technical Analysis When `ANTHROPIC_API_KEY` is present, the script sends the extracted page title, purpose, and navigation to Anthropic to generate personas. This behavior is not represented in `skill.json` dependencies and conflicts with documentation stating that the Agent should avoid duplicate Claude calls and that there are no extra API calls. The transfer may occur for private or internal pages reached through the unrestricted URL input. ### Attack Path 1. The environment contains `ANTHROPIC_API_KEY`. 2. The user tests a sensitive internal or private website. 3. Nova Act extracts its title, purpose, and navigation. 4. Persona auto-generation invokes `infer_plausible_user_types`. 5. The extracted content is sent to Anthropic without a dedicated consent step. ### Impact Assessment Sensitive internal page metadata, navigation labels, project names, or business-purpose information may be disclosed to an additional third-party service. The ...[truncated 133 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions - Remove the Anthropic fallback unless it is essential to the Skill. - Require explicit user opt-in before transmitting tested-page data to another provider. - Clearly disclose the destination, fields sent, retention implications, and applicable provider policies. - Redact or minimize page content before transmission. - Disable external persona generation for private and internal targets. - Declare and pin the `anthropic` dependency if the feature remains. - Add an offline persona-generation mode as the default. ]]>

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• System Prompt Leakage: Direct Leakage, Indirect Extraction, Tool-Based Exfiltration
• Rogue Agent: Self-Modification, Session Persistence

Findings (26)

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 95% confidence
• Finding: The declared description presents a full usability-testing agent with specialized behaviors: generating personas, running tests, analyzing whether goals were achieved, and producing HTML reports for specific UX workflows. The supplied code does not implement any of that domain logic. Instead, it is infrastructure code only: a session/context manager for Nova Act that loads credentials from a local config file, sets NOVA_ACT_API_KEY, prepares a logs directory, and yields a NovaAct instance. While such a wrapper could support a usability-testing system, this chunk by itself does not perform the described testing, analysis, or reporting. The code also accesses local configuration and writes logs, which are concrete behaviors/resources absent from the declared description. Therefore, the description materially overstates and misrepresents what this code chunk actually does.

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 95% confidence
• Finding: The declared description presents a full usability-testing system that generates personas, executes user workflow tests, interprets results, and produces reports. In contrast, the supplied code is a narrow helper module for packaging raw responses and prompting an external orchestrating agent to do the actual interpretation. The module explicitly states it 'does NOT interpret responses itself.' It provides dataclasses, formatting functions, prompt templates, and simple alternative prompt generation logic. This is materially narrower and different from the declared behavior, especially around autonomous interpretation, persona generation, and report generation. Therefore, this chunk does not accurately represent the declared purpose.

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 98% confidence
• Finding: The supplied code is a status-reporting helper, not a usability-testing agent. It starts a background thread, emits periodic progress updates, tracks counters for tests, and prints a final summary. While some field names reference personas and tests, the module itself does not perform the declared core functions of generating personas, executing website workflows, collecting raw data, analyzing responses, or producing HTML reports. This is a materially different primary purpose from the declared description, so it should be flagged as a mismatch.

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 94% confidence
• Finding: The declared description presents a full usability-testing agent with orchestration, persona generation, workflow testing, analysis, and reporting. The supplied code does none of that directly. It only searches local directories for HTML trace files associated with Nova Act sessions and returns file paths sorted by modification time. While this could be a supporting helper within a larger usability-testing system, the chunk’s actual purpose is materially narrower and different from the declared primary purpose. Therefore, this code chunk does not accurately represent the described skill behavior on its own.

Context Leakage

High

• Category: Data Exfiltration

Content

# Parse cookbook for actionable guidance (Bug #14: Use the cookbook!)
    cookbook_hints = parse_cookbook_hints(cookbook)
    
    # Extract context
    archetype = persona.get('archetype', 'user')
    tech_level = persona.get('tech_proficiency', 'medium')
    page_title = page_analysis.get('title', 'this page')

• Confidence: 75% confidence
• Finding: Code or instructions that leak agent conversation context to external services, potentially exposing sensitive user interactions.

Direct Prompt Extraction

High

• Category: System Prompt Leakage

Content

- Clicks, types, scrolls
    - Reports what it sees
    """
    # Return prompt unchanged - no persona-specific modifications
    # Persona context is used by the agent when interpreting results, not by Nova Act
    return base_prompt

• Confidence: 85% confidence
• Finding: Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Session Persistence

Medium

• Category: Rogue Agent

Content

✅ **Real Browser Automation** - Actual Playwright browser control via Nova Act  
✅ **Cookbook Integration** - Loads best practices and workflow patterns automatically  
✅ **Fully Dynamic Testing** - Exploration strategies generated per website/persona (no hardcoded logic!)  
✅ **Smart Persona Generation** - Analyzes page content to create relevant user types  
✅ **Adaptive Testing** - AI tries multiple variations when element text doesn't match exactly  
✅ **Robust Error Handling** - Handles scroll loops, timeouts, and Nova Act failures gracefully  
✅ **Detailed Reporting** - Professional HTML reports with step-by-step observations

• Confidence: 60% confidence
• Finding: Skill establishes unauthorized persistence across sessions via cron jobs, startup scripts, or state files. Session persistence allows an attacker to maintain access beyond the current interaction.

Sudo/Root Execution

Medium

• Category: Privilege Escalation

Content

```

> **Note:** On Linux, you may also need system dependencies:
> `sudo playwright install-deps chromium`

### Step 3: Configure API key

• Confidence: 70% confidence
• Finding: Commands invoke sudo or root privileges. Verify this elevated access is necessary and justified.

Vague Triggers

Medium

• Confidence: 88% confidence
• Finding: The README suggests activation via generic requests like "Test https://example.com for usability" and "Run a usability test on example.com" without defining stricter trigger conditions, exclusions, or required context. Because this is a markdown file and no negative examples or narrow invocation scope are provided, the activation guidance is somewhat ambiguous and could overlap with ordinary requests to test a site.

Lp3

Medium

• Category: MCP Least Privilege
• Confidence: 90% confidence
• Finding: The skill requests access to environment data and performs file reads/writes, including reading a local API key file and writing logs/results, but it declares no explicit tool scope or permissions. This weakens least-privilege controls and can allow broader-than-expected access to sensitive local data or persistence of sensitive artifacts without clear user consent.

Intent-Code Divergence

Medium

• Confidence: 91% confidence
• Finding: At L070 the documentation says the script does not interpret responses or generate the final report, requiring the agent to do phases 3-4 manually. But L557-L560 and L573-L576 state that running the script will create `nova_act_usability_report.html` and even generate a partial report on interruption, and L725-L741 shows the workflow saving results and generating the HTML report. These statements actively contradict each other about whether report generation is automatic.

Intent-Code Divergence

Medium

• Confidence: 88% confidence
• Finding: L185-L196 explicitly instructs the agent not to let the Python script make its own AI call and not to rely on generic fallback personas, framing agent-generated personas as the intended design. However, L235-L246 and L629-L634 describe the system/script as automatically generating contextual personas, including using Claude AI when no custom persona is supplied. This is an intent-level contradiction in the skill's own documentation.

Session Persistence

Medium

• Category: Rogue Agent

Content

python3 "$SKILL_DIR/scripts/run_adaptive_test.py" "https://example.com"

# This will:
# - Create nova_act_logs/ in current directory
# - Create test_results_adaptive.json in current directory
# - Create nova_act_usability_report.html in current directory
# - Provide 60-second status updates during test

• Confidence: 92% confidence
• Finding: The skill explicitly creates persistent local artifacts including trace logs, raw test results, and an HTML report in the current working directory. Because the traces are documented to include screenshots, full page content, and AI/browser actions, sensitive data from tested sites can be retained on disk and later exposed to other local users, processes, backups, or accidental sharing.

Intent-Code Divergence

Medium

• Confidence: 95% confidence
• Finding: The document explicitly states Nova Act should not evaluate usability or judge whether tasks are easy or hard, reserving that reasoning for the agent. Later examples contradict this by asking Nova Act questions like whether menu items are 'clearly labeled and easy to understand' and whether search results are 'relevant,' which are evaluative judgments rather than raw browser observations.

Missing User Warnings

Medium

• Confidence: 95% confidence
• Finding: The purchasing workflow executes material actions such as adding an item to cart, navigating to the cart, and clicking 'Proceed to Checkout' before the safety stop is reached. In a usability-testing skill that drives a real browser, these steps can modify user state, reserve inventory, trigger promotions, or advance a live commerce transaction farther than intended even if payment is not submitted.

Missing User Warnings

Medium

• Confidence: 92% confidence
• Finding: The signup workflow auto-fills registration credentials and navigates into the registration flow before only stopping at the final submit button. Even without final submission, populating identity fields on a live registration form can trigger validation calls, availability checks, bot defenses, analytics events, or accidental account-creation side effects if the site auto-submits or uses multi-step progression.

Missing User Warnings

Medium

• Confidence: 92% confidence
• Finding: The script writes the modified results back to the same path supplied by the user, with no confirmation, backup, or alternate output file. This can irreversibly destroy original test data or corrupt unrelated JSON files if the operator provides the wrong path, which is a real integrity risk even if it is not a code-execution issue.

Context-Inappropriate Capability

Medium

• Confidence: 90% confidence
• Finding: The manifest describes an AI usability-testing skill, but this helper adds a separate capability to retrieve credentials from ~/.openclaw/config/nova-act.json and later inject them into environment state. While the skill may need Nova Act authentication internally, accessing arbitrary local user config and managing secrets via environment variables is not described as part of the skill's purpose.

Context-Inappropriate Capability

Medium

• Confidence: 91% confidence
• Finding: The fallback persona generation sends page-derived content such as the website title, purpose, and navigation to Anthropic, which is an external AI provider outside the skill's stated Nova Act testing flow. This creates an undisclosed data egress path that may expose sensitive site content, internal URLs, or customer environments to a third party when users reasonably expect testing to stay within the Nova Act workflow.

Missing User Warnings

Medium

• Confidence: 95% confidence
• Finding: Website-derived content is transmitted to an external AI API without a user-facing warning, confirmation, or consent mechanism. In a usability-testing skill, this is more dangerous because users may run tests against private staging sites, authenticated pages, or proprietary product content and may not expect third-party sharing during persona generation.

Intent-Code Divergence

Medium

• Confidence: 97% confidence
• Finding: The function documentation says it does not interpret responses and that the orchestrating AI agent should analyze raw responses to determine goal achievement. However, later in the same test flow the code heuristically sets `goal_achieved` and computes `overall_success` from raw text, contradicting the documented intent of agent-only interpretation.

Intent-Code Divergence

Low

• Confidence: 82% confidence
• Finding: Earlier guidance frames Nova Act as a browser automation tool that only executes actions and reports observations, not reasoning. Describing trace files as containing 'AI reasoning and decisions' creates intent-level inconsistency about whether Nova Act itself is performing reasoning.

Missing User Warnings

Low

• Confidence: 84% confidence
• Finding: The posting workflow opens the compose UI and types content into it before the safety stop. Although the code avoids clicking 'Post', entering text into live communication interfaces can still create drafts, trigger autosave, invoke moderation or notification systems, or leak test content into shared environments.

Missing User Warnings

Low

• Confidence: 88% confidence
• Finding: The report embeds local trace file paths directly into HTML, including absolute Linux or WSL paths such as file://... or file://wsl$/..., which can expose filesystem layout, usernames, distro names, and session directory structure to anyone who views or shares the report. In this usability-testing context, traces may also reveal sensitive artifacts from test sessions, so making local paths clickable increases the chance of unintended disclosure beyond the intended audience.

Missing User Warnings

Low

• Confidence: 88% confidence
• Finding: The function unconditionally writes a generated HTML report to the current working directory, and that report includes test cases, observations, personas, and trace-link metadata derived from inputs. While the module docstring describes report generation, there is no user-facing prompt, log, or warning at the point of file creation to disclose that local data will be written.

Static analysis

No suspicious patterns detected.