Website Usability Test Nova Act

• Outcome: Review
• Latest audit: May 28, 2026 at 11:33 AM UTC
• Version:
  1.0.5

Overview

This appears to be a legitimate usability-testing skill, but it needs review because it can automate state-changing website workflows, persist sensitive browser traces, and may send site analysis to Anthropic despite incomplete disclosure.

Install only if you are comfortable with an agent controlling a real browser against the target site, saving screenshots/page content locally, and using your Nova Act API key. Prefer test or sandbox environments, avoid authenticated or confidential production sites, review/delete nova_act_logs and reports after use, avoid setting ANTHROPIC_API_KEY unless third-party persona generation is acceptable, and supervise workflows that involve carts, checkout, signup, posting, or forms.

SkillSpector

By NVIDIA

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• Rogue Agent: Self-Modification, Session Persistence
• MCP Least Privilege: Underdeclared Capability, Wildcard Permission, Missing Permission Declaration
• MCP Tool Poisoning: Hidden Instructions, Unicode Deception, Parameter Description Injection

Findings (12)

Lp3

Medium

• Category: MCP Least Privilege
• Confidence: 95% confidence
• Finding: The skill explicitly instructs the agent to read a credential file, inspect environment-dependent configuration, and write logs/results/reports, but it does not declare corresponding permissions. Undeclared capabilities undermine least-privilege controls and informed consent, increasing the chance that the skill can access secrets or persist sensitive artifacts without clear platform mediation.

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 91% confidence
• Finding: The documented behavior goes beyond benign UX testing by accessing local API credentials, potentially invoking additional AI services, and generating/shareable trace and report artifacts that may contain screenshots, HTML, and sensitive page content. When a skill's stated purpose does not fully disclose these behaviors, users and enforcement layers may authorize it under false assumptions, which materially raises privacy and data-exposure risk.

Context-Inappropriate Capability

Medium

• Confidence: 91% confidence
• Finding: The script prepends a hard-coded workspace directory to `sys.path` and then imports executable Python code from that location. If that workspace content is modified, replaced, or attacker-controlled, running this script will execute untrusted code with the privileges of the user, creating a code-execution and supply-chain risk.

Description-Behavior Mismatch

Medium

• Confidence: 89% confidence
• Finding: The script overwrites the original JSON input file in place after applying heuristic interpretations. This can destroy raw test evidence, corrupt source data, and make downstream analysis trust modified results as if they were original observations.

Context-Inappropriate Capability

Medium

• Confidence: 93% confidence
• Finding: The script introduces an undeclared outbound dependency on Anthropic to generate personas, which changes the skill's trust boundary and sends page-derived content to a third party. In a usability-testing skill expected to operate with Nova Act, this creates supply-chain, privacy, and compliance risk because behavior now depends on an external LLM service not clearly disclosed or controlled.

Intent-Code Divergence

Medium

• Confidence: 84% confidence
• Finding: The code contradicts its own safety/design contract by saying goal achievement must be determined by the orchestrating agent, then locally inferring success and setting overall pass/fail. This can produce misleading reports and unsafe automation decisions, especially if downstream systems trust the generated success state for business or release decisions.

Missing User Warnings

Medium

• Confidence: 93% confidence
• Finding: The purchasing workflow explicitly instructs navigation through add-to-cart, cart, and checkout steps before stopping only at the payment page. In an automation skill for usability testing, this can still cause material impact such as cart pollution, inventory reservation, abandoned checkout creation, coupon consumption, rate-limit triggering, or accidental progression in merchants with one-click/express flows.

Missing User Warnings

Medium

• Confidence: 94% confidence
• Finding: The script silently overwrites the user-supplied JSON file without warning or confirmation. In a usability-testing workflow, that increases the chance of accidental data loss and enables misuse where a caller can cause modification of arbitrary writable JSON files passed as arguments.

Missing User Warnings

Medium

• Confidence: 94% confidence
• Finding: The code loads a sensitive API key from a local config file and places it into the process environment without any disclosure, minimization, or cleanup. Environment variables are broadly inherited by child processes and may be exposed through debugging, crash reports, process inspection, or downstream tooling, increasing the chance of unintended credential disclosure.

Missing User Warnings

Medium

• Confidence: 91% confidence
• Finding: The wrapper automatically creates a log directory and enables session trace storage on disk, potentially capturing sensitive browsing content, test data, prompts, or authenticated workflow artifacts. Because this happens by default and uses the current working directory, users may unknowingly persist sensitive data in locations with weaker access controls or accidental inclusion in backups, commits, or shared workspaces.

Missing User Warnings

Medium

• Confidence: 94% confidence
• Finding: The persona-generation prompt includes analyzed website title, purpose, and navigation and sends that data to an external API without a user-facing warning or consent checkpoint. Even if the data seems low sensitivity, websites under test may be internal, pre-release, customer-specific, or otherwise confidential, so silent exfiltration to a third party is risky.

Session Persistence

Medium

• Category: Rogue Agent

Content

python3 "$SKILL_DIR/scripts/run_adaptive_test.py" "https://example.com"

# This will:
# - Create nova_act_logs/ in current directory
# - Create test_results_adaptive.json in current directory
# - Create nova_act_usability_report.html in current directory
# - Provide 60-second status updates during test

• Confidence: 88% confidence
• Finding: Create nova_act_logs/ in current directory # - Create test_results_adaptive.json in current directory # - Create nova_act_usability_report.html in current directory # - Provide 60-second status update

VirusTotal

65/65 vendors flagged this skill as clean.