Back to skill

Security audit

QA Pilot

Security checks for vulnerabilities and agentic risk

Overview

QA Pilot is a legitimate QA-testing skill, but it gives agents broad automatic permission to test and fix apps without enough built-in scoping.

Install only if you want agents to perform active QA and automatic fix attempts. Use it on local or explicitly approved test environments, with test accounts and disposable data. Avoid enabling automatic runs against production, admin consoles, payments, messaging, or third-party systems, and prefer a pinned or trusted install path over the unpinned npx command.

Vulnerability Patterns
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (2)

T09 · Insecure Skill Coding Practices

Error
Location
demo/taskflow-app.html:394
Finding

Persistent DOM-based cross-site scripting in the TaskFlow demo

Content
View full analysis
{ const origIndex = tasks.indexOf(task); const li = document.createElement('li'); li.className = 'task-item'; li.innerHTML = `
${task.text} ${task.priority} × `; list.appendChild(li); }); ``` ```javascript function addTask() { const input = document.getElementById('taskInput'); const text = input.value.trim(); if (!text) { showToast('Please enter a task!'); return; } tasks.push({ text, priority: document.getElementById('taskPriority').value, done: false, createdAt: Date.now() }); ``` ### Technical Analysis The application accepts arbitrary task text and stores it without validation or output encoding. During rendering, `task.text` is interpolated into an HTML template assigned to `li.innerHTML`. Because `innerHTML` parses its input as markup, an attacker can supply HTML containing executable event handlers. For example: ```html ``` The task is also serialized into `localStorage ...[truncated 2164 chars]
Remediation
View remediation
toggleTask(origIndex)); const textElement = document.createElement('span'); textElement.className = `task-text${task.done ? ' completed' : ''}`; textElement.textContent = task.text; const priorityElement = document.createElement('span'); priorityElement.className = `task-priority priority-${task.priority}`; priorityElement.textContent = task.priority; const deleteButton = document.createElement('button'); deleteButton.className = 'task-delete'; deleteButton.type = 'button'; deleteButton.textContent = '×'; deleteButton.addEventListener('click', () => deleteTask(origIndex)); li.append(checkbox, textElement, priorityElement, deleteButton); list.appendChild(li); ``` Additional hardening should include: 1. Validate records loaded from `localStorage` before using them. 2. Require `task.text` to be a string and apply an appropriate maximum length. 3. Restrict `priority` to the explicit values `low`, `medium`, or `high`. 4. Replace all inline event handlers with `addEventListener`. 5. Add a restrictive Content Security Policy when the demo is served over HTTP. 6. If rich HTML is intentionally supported in the future, sanitize it using a maintained, security-reviewed HTML sanitizer rather than custom escaping. 7. Add regression tests using common XSS payloads and verify that they appear only as literal text. ]]>

T08 · Insecure Dependencies

Warning
Location
INSTALL.md:9
Finding

Unpinned third-party package execution in installation instructions

Content
View full analysis
Remediation
View remediation
add helal-muneer/qa-pilot ``` 2. Document the expected npm publisher, package name, version, and registry. 3. Provide release checksums or signed release artifacts for manual verification. 4. Prefer an installation mechanism that supports lockfiles or integrity verification. 5. Recommend reviewing the package and dependency provenance before execution. 6. Retain the manual Git clone or verified archive option as a non-`npx` installation path. 7. Update the pinned version deliberately after auditing new releases rather than implicitly resolving the latest version. ]]>
Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
Findings (9)

Rp1

Medium
Category
MCP Rug Pull
Confidence
88% confidence
Finding

The installation instruction uses npx skills add helal-muneer/qa-pilot without pinning a specific package version, so users will fetch and execute whatever version of the skills package is current at install time. If the upstream package is compromised, typo-squatted, or a malicious update is published, this creates a supply-chain execution risk during installation.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The README encourages clicking every button, filling every form, and breaking things, but does not warn that these actions can create, modify, or delete data, submit transactions, send messages, or otherwise change application state. Because the skill is explicitly meant to drive real app interactions, missing safety guidance materially increases the chance of harmful testing against live or sensitive environments.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The README says the skill activates automatically when configured, but does not define clear scope boundaries such as only for explicit web-app QA tasks or only against approved local targets. In an agent setting, broad auto-activation can cause the skill to run in unrelated workflows, triggering unsolicited browsing and test actions that may waste resources or interact with applications unexpectedly.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The instruction to always run QA Pilot before marking tasks complete is overly broad and can apply to many routine tasks that are not suitable for automated browser testing. In agent environments, unconditional directives often override situational judgment and may lead to unsafe or disruptive execution against apps, admin panels, or workflows with side effects.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The skill auto-activates on broad conditions such as any completed app modification or any time the agent is about to say work is done. That can cause the agent to launch testing, start services, interact with applications, and potentially mutate project state without explicit user approval or clearly bounded scope. In a powerful agent environment, this increases the chance of unintended actions and overreach.

Content

No source excerpt is available for this finding.

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
84% confidence
Finding

The instruction to infer requirements from prior conversation and common patterns, while explicitly avoiding asking the user for clarification, grants the agent discretion to decide what should exist and then test/fix against those assumptions. That can lead to unauthorized changes, fabricated acceptance criteria, and destructive or wasteful edits based on guessed requirements rather than user intent.

Content

Scanner excerpt · SKILL.md (reported line 64)May include surrounding context.

□ Navigate away with unsaved changes

text

**Important:** If the agent can't find a spec or clear feature list, it should infer the expected features from the original conversation and common patterns for that type of application. Don't ask the user to provide a test plan — that defeats the purpose.

---

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
91% confidence
Finding

The description states the skill will 'Automatically test websites & apps before reporting completion' without defining when it activates, what targets are in scope, or what limits apply. For an agent skill, this can lead to unbounded autonomous interaction with external systems, causing unintended actions against production services, third-party sites, or sensitive environments.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
89% confidence
Finding

The manifest advertises automated testing of websites and apps but provides no warning that the skill may perform active interactions with live systems. Without disclosure, users or orchestrating agents may invoke it in unsafe contexts, increasing the risk of unintended requests, state changes, account actions, or testing against systems that were not authorized.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
93% confidence
Finding

The document declares lang="en", which hard-codes English as the page language. Under the policy, forcing a specific language without user opt-in can be a natural-language locale violation when no alternative or justification is provided elsewhere in the file.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.