Back to skill

Security audit

Playwright Test Generator

Security checks for vulnerabilities and agentic risk

Overview

This is a coherent Playwright test generator, but it can browse arbitrary URLs and produce executable test files from unescaped user or page data, so it should be reviewed before installation.

Install only if you are comfortable reviewing generated tests before running them, especially in CI. Use mock or trusted test URLs, avoid pointing it at internal or production systems, do not feed untrusted HTML or test data without review, and update/pin dependencies before use.

Vulnerability Patterns
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (3)

T09 · Insecure Skill Coding Practices

Error
Location
playwright_test_generator.py:192
Finding
Unrestricted URL Analysis Enables Server-Side Request Forgery<![CDATA[ ## Vulnerability Details **File Location**: `playwright_test_generator.py:192-225`; `src/generator.js:366-397` **Vulnerability Type**: Server-Side Request Forgery through unrestricted headless-browser navigation **Risk Level**: High ### Vulnerable Code ```python def generate_from_url(url: str, language: str = 'python', pom: bool = False) -> str: """Generate test code from a URL.""" from playwright.sync_api import sync_playwright code_blocks = [] with sync_playwright() as p: browser = p.chromium.launch(headless=True) context = browser.new_context() page = context.new_page() try: page.goto(url, wait_until='domcontentloaded', timeout=30000) page.wait_for_timeout(1000) analyzer = PageAnalyzer(page) if pom: code_blocks.append(_generate_pom_code(analyzer, language)) else: code_blocks.append(_generate_test_code(analyzer, language)) finally: page.close() context.close() browser.close() return '\n\n'.join(code_blocks) ``` The JavaScript implementation has the same weakness: ```javascript async analyzeUrl(url, options = {}) { if (!url) throw new Error('URL is required for analyzeUrl'); const { mock = false } = options; if (mock) { return { url, title: 'Mock Page Title', locators: { 'main-heading': 'h1', 'login-form': '#login-form', 'username-input': '[data-testid="username-input"]', 'password-input': '[data-testid="password-input"]', 'submit-button': '[data-testid="submit-button"]' }, elements: [] }; } const { chromium } = await import('playwright'); const browser = await chromium.launch({ headless: true }); const page = await browser.newPage(); await page.goto(url, { waitUntil: 'domcontentloaded' }); const html = await page.content(); const title = await page.title() ...[truncated 2528 chars]
Remediation
<![CDATA[ ## Remediation Suggestions 1. Parse URLs with a standards-compliant URL parser and permit only `http:` and `https:`. 2. Reject URLs containing embedded credentials or malformed hostnames. 3. Resolve the hostname before navigation and reject every resolved address in: - Loopback ranges. - RFC1918/private ranges. - Link-local ranges. - Multicast and unspecified ranges. - Reserved and documentation ranges. - IPv4-mapped IPv6 representations of blocked addresses. 4. Intercept browser requests and validate each destination, including subresources and redirect targets. 5. Prefer an explicit destination-domain allowlist when URLs are expected to come from a known environment. 6. Run the browser in a network sandbox that cannot reach metadata services, localhost, or internal networks. 7. Add response-size, navigation-time, and redirect-count limits. 8. Add tests for direct private IPs, alternative numeric IP representations, DNS rebinding, IPv6 loopback, and public-to-private redirects. ]]>

T09 · Insecure Skill Coding Practices

Error
Location
src/parser.js:314
Finding
Unescaped Inputs Permit Arbitrary Code Injection into Generated Tests<![CDATA[ ## Vulnerability Details **File Location**: `src/parser.js:314-341`; additional sinks in `src/generator.js:24-257`, `src/locator.js:165-194`, `playwright_test_generator.py:231-355`, and `templates.py:91-246` **Vulnerability Type**: Generated source-code injection **Risk Level**: High ### Vulnerable Code ```javascript export function stepToCode(step, indent = ' ') { const { action, selector, value, key, duration, url } = step; switch (action) { case 'navigate': return `${indent}await page.goto('${url || step.target || selector}');`; case 'click': return `${indent}await page.click('${selector || value}');`; case 'fill': return `${indent}await page.fill('${selector || value}', '${step.value}');`; case 'press': return `${indent}await page.keyboard.press('${key || value}');`; case 'wait': return `${indent}await page.waitForTimeout(${duration || 1000});`; case 'hover': return `${indent}await page.hover('${selector || value}');`; case 'select': return `${indent}await page.selectOption('${selector || value}', '${value}');`; case 'check': return `${indent}await page.check('${selector || value}');`; case 'uncheck': return `${indent}await page.uncheck('${selector || value}');`; case 'screenshot': return `${indent}await page.screenshot({ path: '${selector || 'screenshot.png'}' });`; case 'assert': return `${indent}await expect(page).toHaveTitle(/${value}/);`; default: return `${indent}// TODO: ${selector || value}`; } } ``` Data-driven values are also inserted directly into executable source: ```javascript const TEST_DATA = [ ${testData.map(d => ` { ${Object.entries(d).map(([k, v]) => `${k}: '${v}'`).join(', ')} }`).join(',\n')} ]; ``` Locator names and selectors are directly interpolated: ```javascript const locatorsCode = Object.entries(locators).map(([name, sel]) => { const method = name.replace(/[\s_-]+/g, '_').toLowerCase( ...[truncated 3591 chars]
Remediation
<![CDATA[ ## Remediation Suggestions 1. Do not generate executable source through raw string interpolation. 2. For JavaScript literals, serialize untrusted values with `JSON.stringify`. 3. For Python literals, use `repr` or construct an abstract syntax tree and emit code with a trusted formatter. 4. Validate generated identifiers against strict allowlists: - Permit only letters, digits, and underscores. - Reject leading digits. - Reject reserved words. - Apply a fixed safe fallback when normalization produces an invalid identifier. 5. Escape or remove newline and comment delimiters from generated comments and docstrings. 6. Escape regular-expression metacharacters or avoid generating regular-expression literals from input. 7. Treat remote HTML and locator extraction results as untrusted data. 8. Validate test-data objects against a fixed schema and serialize the complete structure safely rather than interpolating keys and values. 9. Add adversarial tests covering single and double quotes, backslashes, newlines, template delimiters, comment delimiters, regular-expression delimiters, and Unicode line separators. 10. Mark generated tests as untrusted artifacts and require review before execution, especially in CI. ]]>

T09 · Insecure Skill Coding Practices

Warning
Location
src/generator.js:163
Finding
Generated Data-Driven Tests Expose Credential Records in Logs<![CDATA[ ## Vulnerability Details **File Location**: `src/generator.js:163-183` **Vulnerability Type**: Sensitive information exposure through logging **Risk Level**: Medium ### Vulnerable Code ```javascript function generateJestDataDriven(data) { const { testName = 'login', testData = [], steps = [], language = 'typescript' } = data; const importLine = language === 'typescript' ? "import { test, expect } from '@playwright/test';" : "const { test, expect } = require('@playwright/test');"; return `${importLine} // Data-driven test: ${testName} const TEST_DATA = [ ${testData.map(d => ` { ${Object.entries(d).map(([k, v]) => `${k}: '${v}'`).join(', ')} }`).join(',\n')} ]; for (const data of TEST_DATA) { test(data.username + ' login test', async ({ page }) => { await page.goto('https://example.com'); // Use data.username, data.password etc. console.log(data); }); } `; } ``` ### Technical Analysis The generated test explicitly anticipates records containing fields such as `username` and `password`, then logs each complete record with `console.log(data)`. Data-driven test records commonly contain passwords, authentication tokens, API keys, email addresses, session data, or other sensitive values. Logging the complete object exposes every field regardless of whether it is necessary for test diagnostics. The data is written when the generated test runs, so exposure can occur in local terminal history, CI job output, centralized logging platforms, test reports, or retained build artifacts. ### Attack Path 1. A user supplies a test-data JSON file containing credentials or other secrets. 2. The generator embeds those records in a Jest data-driven test. 3. The generated test is executed locally or in CI. 4. `console.log(data)` writes every field, including passwords or tokens, to standard output. 5. CI or logging infrastructure stores and distributes the output according to its retention and access policies. 6. Any person or integra ...[truncated 611 chars]
Remediation
<![CDATA[ ## Remediation Suggestions 1. Remove `console.log(data)` from generated tests. 2. Log only a non-sensitive case identifier, sequence number, or redacted username when diagnostics are necessary. 3. Apply explicit redaction to fields such as `password`, `token`, `secret`, `authorization`, `cookie`, and `apiKey`. 4. Do not embed production credentials directly in generated test files. 5. Load secrets at runtime through protected environment variables or secret-backed test fixtures. 6. Configure CI systems to mask known secrets and restrict access to test logs. 7. Add tests confirming that generated source does not print complete data records or secret fields. ]]>
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
  • Memory PoisoningPersistent Context Injection, Context Window Stuffing, Memory Manipulation
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
Findings (37)

Tp4

High
Category
MCP Tool Poisoning
Confidence
93% confidence
Finding
The declared description presents a broad AI-driven Playwright test generation skill, including generating multiple kinds of tests from natural language, HTML, or URLs across several frameworks. The supplied code chunk is much narrower: it only parses provided HTML, extracts selectors, offers selector lookup, and emits a simple JavaScript Page Object class. While this partially aligns with the claimed ability to build POMs from HTML analysis, it does not implement the larger primary purpose of generating full test scripts or data-driven tests, nor does it show AI/natural-language handling or URL-based analysis. This is therefore a material description-to-behavior mismatch.

Tp4

High
Category
MCP Tool Poisoning
Confidence
89% confidence
Finding
The supplied code is clearly related to Playwright test generation, so it broadly aligns with the domain of the description. However, the declared description significantly overstates the implemented capabilities. This chunk only contains static template generators and utility functions for assembling test code strings, POM scaffolding, and Gherkin text. It does not contain any AI logic, parsing of natural language, HTML inspection, URL fetching/analysis, or data-driven test generation. It also does not show support for a separate 'native Playwright' mode beyond the Python and JavaScript template families explicitly present. Additionally, it contains Gherkin/BDD generation, which is outside the declared purpose. Because the actual implemented behavior is materially narrower and somewhat different from the declared capabilities, this should be flagged as a mismatch.

Known Vulnerable Dependency: brace-expansion==2.0.3 — 3 advisory(ies): CVE-2026-13149 (brace-expansion: DoS via exponential-time expansion of consecutive non-expanding); CVE-2026-14257 (brace-expansion: DoS via unbounded expansion length causing an out-of-memory pro); CVE-2026-69152 (brace-expansion: DoS via unbounded intermediate arrays, bypassing the CVE-2026-1)

High
Category
Supply Chain
Confidence
91% confidence
Finding
brace-expansion 2.0.3 has multiple DoS issues involving exponential or unbounded expansion that can consume CPU or memory. Even though this instance is likely transitive, any code path that expands attacker-influenced glob or brace patterns can be crashed or severely slowed, which is especially relevant in developer tooling and automation contexts.

Known Vulnerable Dependency: browserslist==4.28.2 — 2 advisory(ies): CVE-2026-73088 (Browserslist: Uncaught crash / prototype write via untrusted browserslist-stats.); CVE-2026-73089 (Browserslist: Unbounded memory growth (no cache eviction) via distinct query res)

High
Category
Supply Chain
Confidence
83% confidence
Finding
browserslist 4.28.2 is flagged for crash/prototype-write and memory growth issues when handling untrusted browserslist-stats or distinct queries. In this project it is a dev-tooling dependency, which lowers runtime exposure, but if CI, code generation, or tooling consumes attacker-controlled config or stats files, this can still cause denial of service or unsafe object mutation.

Known Vulnerable Dependency: brace-expansion==1.1.13 — 3 advisory(ies): CVE-2026-13149 (brace-expansion: DoS via exponential-time expansion of consecutive non-expanding); CVE-2026-14257 (brace-expansion: DoS via unbounded expansion length causing an out-of-memory pro); CVE-2026-69152 (brace-expansion: DoS via unbounded intermediate arrays, bypassing the CVE-2026-1)

High
Category
Supply Chain
Confidence
90% confidence
Finding
brace-expansion 1.1.13 carries the same class of CPU and memory exhaustion vulnerabilities as newer affected branches. Because it is nested under other tooling dependencies, exploitation would most likely occur through attacker-controlled glob patterns in test/build automation, resulting in denial of service rather than direct code execution.

Known Vulnerable Dependency: js-yaml==3.14.2 — 4 advisory(ies): CVE-2026-84375 (js-yaml: maxTotalMergeKeys does not limit CPU use for empty merge sources); CVE-2026-59869 (js-yaml: YAML merge-key chains can force quadratic CPU consumption); GHSA-5p4m-2wfm-xmqj (JS-YAML: Quadratic CPU consumption in !!omap resolution (3.x and 4.x) — CVE-2026) +1 more

High
Category
Supply Chain
Confidence
89% confidence
Finding
js-yaml 3.14.2 is affected by multiple quadratic or otherwise excessive CPU-consumption issues when parsing crafted YAML structures. Although present as a dev dependency, YAML parsing in developer tooling or CI commonly touches semi-trusted project files, so a malicious config file could hang or significantly degrade automation workflows.

Missing User Warnings

Medium
Confidence
91% confidence
Finding
The design explicitly supports taking a user-supplied URL, launching a browser, navigating to it, and extracting DOM content, but it does not document any trust boundaries, URL validation, network safety controls, or warnings about interacting with external/live systems. In a test-generation skill, this can cause the agent to visit attacker-controlled or production URLs, triggering outbound requests, script execution in the browser context, and unintended actions against real services during analysis.

Missing User Warnings

Medium
Confidence
89% confidence
Finding
The README encourages URL analysis and writing generated output to files but does not warn users that these actions may trigger outbound network requests or filesystem modifications. In an AI-agent or automation context, that omission can lead to unintended access to internal URLs, SSRF-like behavior through agent tooling, or overwriting local files when users invoke commands based on the documentation.

Lp3

Medium
Category
MCP Least Privilege
Confidence
88% confidence
Finding
The skill advertises capabilities that imply reading files, writing generated code, and accessing URLs, but it does not declare any explicit tool scope or permissions boundaries. In an agent environment, missing scope declarations can allow broader-than-expected tool access or make reviewers unable to verify whether file and network operations are appropriately constrained.

Vague Triggers

Medium
Confidence
93% confidence
Finding
Overly broad activation language can cause the skill to trigger on unrelated requests and start handling inputs outside its intended QA/test-generation scope. In an agent system, this can lead to inappropriate use of file/network capabilities, accidental data exposure, or generation of unreviewed code in contexts the user did not intend.

Vague Triggers

Medium
Confidence
89% confidence
Finding
Ambiguous 'when to use' conditions broaden the skill's activation surface and increase the chance of misrouting prompts to a skill that may read files, analyze HTML, or fetch URLs. Because this skill's context involves potentially sensitive page content and code generation, accidental invocation is more dangerous than for a purely informational skill.

Missing User Warnings

Medium
Confidence
83% confidence
Finding
In the batch path, the CLI automatically saves every generated output to disk via output.save(filepath) and only reports the save after it happens. Unlike a clearly destructive operation, file creation is expected for this tool, but this command lacks any pre-write disclosure or confirmation in the code path itself while performing repeated file writes.

Missing User Warnings

Medium
Confidence
96% confidence
Finding
The generator emits test steps that will navigate to arbitrary user-supplied URLs and then perform live fills and clicks, which can trigger real side effects such as form submissions, account actions, purchases, or workflow changes when the generated code is executed. In the context of a Playwright test-generation skill, this is more dangerous because the tool is explicitly designed to automate browser interactions and may be pointed at production systems without guardrails or user confirmation.

Context Window Stuffing

Medium
Category
Memory Poisoning
Content
'open': 'Navigate to page',
        'navigate': 'Navigate to page',
        'go': 'Navigate to page',
        'fill': 'Fill input field',
        'enter': 'Enter data',
        'type': 'Type data',
        'click': 'Click element',
Confidence
85% confidence
Finding
Skill attempts to fill the context window with filler content, displacing legitimate instructions and safety constraints. This can degrade agent performance or bypass safety boundaries.

Context Window Stuffing

Medium
Category
Memory Poisoning
Content
'open': 'Navigate to page',
        'navigate': 'Navigate to page',
        'go': 'Navigate to page',
        'fill': 'Fill input field',
        'enter': 'Enter data',
        'type': 'Type data',
        'click': 'Click element',
Confidence
85% confidence
Finding
Skill attempts to fill the context window with filler content, displacing legitimate instructions and safety constraints. This can degrade agent performance or bypass safety boundaries.

Vague Triggers

Medium
Confidence
95% confidence
Finding
Allowing invocation whenever the user merely mentions Playwright test generation creates ambiguity in tool selection and increases the chance of unintentional activation. Because this skill supports URL and HTML analysis plus code artifact generation, accidental invocation expands the attack surface and may process external or untrusted content unexpectedly.

Vague Triggers

Medium
Confidence
95% confidence
Finding
Allowing invocation whenever the user merely mentions Playwright test generation creates ambiguity in tool selection and increases the chance of unintentional activation. Because this skill supports URL and HTML analysis plus code artifact generation, accidental invocation expands the attack surface and may process external or untrusted content unexpectedly.

Missing User Warnings

Medium
Confidence
90% confidence
Finding
The skill invites users to supply URLs or HTML for analysis but does not warn that this may involve processing untrusted external content and producing generated files or locators from it. In practice, this can surprise users, increase exposure to prompt-injection-style content embedded in HTML, and create downstream risks if generated artifacts are trusted or executed without review.

Intent-Code Divergence

Medium
Confidence
95% confidence
Finding
The docstring for `generate_page_object` says it generates a complete Page Object Model class, and the surrounding templates imply real POM methods will be rendered. However, the implementation builds `method_code` from `PythonTemplates.generate_step_screenshot.__doc__`, which inserts the text `Generate a screenshot step.` for each method rather than generating the supplied method definitions.

Intent-Code Divergence

Medium
Confidence
91% confidence
Finding
The `generate_scenario` docstring states it generates a Gherkin scenario, and the class also defines `step_when`, `step_then`, and `step_and`, suggesting support for varied step types. In practice, `generate_scenario` formats every input step through `GHERKIN_STEP`, whose template hardcodes `Given`, so `When` and `Then` style steps are contradicted by the implementation.

Intent-Code Divergence

Low
Confidence
97% confidence
Finding
The section is explicitly labeled as pytest-playwright Python output, but the example methods contain `await` inside non-async `def` functions and include JavaScript-style semicolons, which is not valid Python. This is an intent/documentation contradiction in the README example rather than an implementation detail.

Natural-Language Policy Violations

Low
Confidence
95% confidence
Finding
The info command presents subscription pricing only in yen (¥), which imposes a locale-specific presentation in natural-language output without opt-in or explanation. This can violate language/locale policy expectations when the tool is not clearly region-specific.

Known Vulnerable Dependency: @babel/core==7.29.0 — 1 advisory(ies): CVE-2026-49356 (@babel/core: Arbitrary File Read via sourceMappingURL Comment)

Low
Category
Supply Chain
Confidence
77% confidence
Finding
@babel/core 7.29.0 is flagged for an arbitrary file read via sourceMappingURL handling. In this lockfile it appears only as a development/test dependency, so exploitability in the deployed skill is limited, but if untrusted JavaScript is processed during local builds, tests, or CI, it could expose local files accessible to the build runner.

Known Vulnerable Dependency: baseline-browser-mapping==2.10.16 — 1 advisory(ies): CVE-2026-45819 (baseline-browser-mapping process termination on invalid input causes denial of s)

Low
Category
Supply Chain
Confidence
65% confidence
Finding
baseline-browser-mapping 2.10.16 is reported as susceptible to process termination on invalid input, causing denial of service. It is a transitive dev dependency and not obviously part of runtime skill execution, so the real-world risk is mostly limited to build or test workflows that accept attacker-controlled input.

Unpinned Dependencies

Low
Category
Supply Chain
Content
"author": "kay",
  "license": "MIT",
  "dependencies": {
    "cheerio": "^1.0.0",
    "ejs": "^3.1.9",
    "playwright": "^1.42.0"
  },
Confidence
40% confidence
Finding
Dependencies lack version pinning, allowing potential malicious package updates. Consider pinning versions.

Static analysis

No suspicious patterns detected.