Back to skill

Security audit

Ai Autotester

Security checks for vulnerabilities and agentic risk

Overview

This automated tester is purpose-aligned, but it needs Review because it can modify arbitrary accessible directories, install dependencies, and execute project code without clear containment or user approval.

Only use this skill on trusted projects inside a disposable, isolated environment with no sensitive credentials available. Expect it to modify the target project, install packages, and run code from that project; review requirements.txt first and avoid pointing it at arbitrary filesystem locations.

Vulnerability Patterns
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (2)

T05 · Unauthorized Access and Privilege Escalation

Error
Location
run.py:5
Finding
Unrestricted Target Path Allows Unauthorized Filesystem Modification## Vulnerability Details **File Location**: `run.py`, lines 5-9 **Vulnerability Type**: Unrestricted path resolution and unauthorized file modification **Risk Level**: High ### Vulnerable Code ```python ws=Path("/home/jason/.openclaw/workspace");t=(ws/a.task).resolve() if not Path(a.task).is_absolute() else Path(a.task).resolve() t.exists() or (print(json.dumps({"skill":"AI_AutoTester","status":"error","error":f"Target path not found: {t}"},ensure_ascii=False,indent=2)),sys.exit(1)) (td:=t/"tests").mkdir(parents=True,exist_ok=True) (tf:=td/"test_smoke.py").exists() or tf.write_text('from app.main import app\nfrom fastapi.testclient import TestClient\n\ndef test_root():\n c=TestClient(app)\n r=c.get("/")\n assert r.status_code==200\n',encoding="utf-8") (t/"requirements-test.txt").write_text("pytest==8.3.2\nhttpx==0.27.2\n",encoding="utf-8") ``` ### Technical Analysis The `--task` argument is treated as a filesystem path. Absolute paths are accepted directly, while relative paths are resolved against `/home/jason/.openclaw/workspace`. After resolution, the program does not verify that the resulting path remains inside the intended workspace. Consequently, a relative path containing traversal components such as `../../target`, or an explicitly supplied absolute path, can select any existing directory accessible to the process. The program then: - Creates a `tests` directory. - Creates `tests/test_smoke.py` if it does not already exist. - Unconditionally overwrites `requirements-test.txt`. Calling `Path.resolve()` canonicalizes a path but does not enforce a security boundary. The absence of an allowlist or containment check violates least-privilege principles and permits modifications beyond the documented testing workspace. The `--constraints` option does not mitigate this issue because it is parsed but never consulted before filesystem changes occur. ### Attack Path 1. An attacker or untrusted caller invokes the ...[truncated 1119 chars]
Remediation
## Remediation Suggestions - Reject absolute values for `--task` unless an explicitly authorized administrative mode requires them. - Resolve both the workspace and requested target, then enforce containment: ```python workspace = Path("/home/jason/.openclaw/workspace").resolve() target = (workspace / args.task).resolve() if not target.is_relative_to(workspace): raise ValueError("Target must remain within the configured workspace") ``` - On Python versions without `Path.is_relative_to()`, use a safe containment comparison based on `relative_to()`, handling `ValueError`. - Consider restricting targets to an allowlist of registered project directories. - Verify that the target is a directory rather than merely checking that it exists. - Do not overwrite `requirements-test.txt` unconditionally. Fail safely if it exists, create a uniquely named generated file, or require explicit confirmation. - Require confirmation before modifying project files and clearly report every planned change. - Apply defenses against symlink-based boundary bypasses and revalidate paths immediately before writing where concurrent attackers are in scope. - Honor the supplied constraints before performing any filesystem mutation.

T08 · Insecure Dependencies

Error
Location
run.py:10
Finding
Untrusted Project Dependencies Are Installed Without Isolation or Verification## Vulnerability Details **File Location**: `run.py`, lines 10-12 **Vulnerability Type**: Unsafe dependency installation from a user-selected project **Risk Level**: High ### Vulnerable Code ```python (r1:=t/"requirements.txt").exists() and subprocess.run(["python","-m","pip","install","-q","-r",str(r1)],cwd=str(t),check=False) (t/"requirements-test.txt").write_text("pytest==8.3.2\nhttpx==0.27.2\n",encoding="utf-8") subprocess.run(["python","-m","pip","install","-q","-r",str(t/"requirements-test.txt")],cwd=str(t),check=False) ``` ### Technical Analysis The script automatically installs every dependency declared in the selected target's `requirements.txt`. The target is user-controlled and, due to the unrestricted path handling, may be any accessible project directory. No controls are applied to the dependency specification. In particular, the implementation does not: - Require approval before installation. - Restrict package indexes or direct URL sources. - Require cryptographic hashes. - Validate package names or versions against an allowlist. - Disable source distributions or untrusted build backends. - Use a disposable virtual environment or sandbox. - Separate installation privileges from the agent's broader privileges. A requirements file can refer to attacker-controlled package sources, local projects, version-control repositories, or malicious source distributions. Installing a source distribution may execute attacker-controlled build backend code during package preparation or wheel construction. Installation is performed through the active `python` interpreter, so packages can alter or compromise the environment used by the skill and subsequent Python tasks. The use of an argument array avoids shell injection, but it does not make the dependency contents trustworthy. The program also ignores installation failures through `check=False` and continues to testing. This can hide partial or failed environmen ...[truncated 1818 chars]
Remediation
## Remediation Suggestions - Never install target-controlled dependencies directly into the skill's active interpreter environment. - Execute dependency installation and tests inside a disposable container, virtual machine, or tightly sandboxed virtual environment. - Use a fresh environment for each target and delete it after completion. - Require explicit user confirmation after displaying the complete dependency plan and package sources. - Permit packages only from approved indexes and reject unexpected direct URLs, VCS references, local paths, and editable installations. - Use a lock file with exact versions and cryptographic hashes, such as pip's `--require-hashes`. - Prefer prebuilt, verified wheels and consider rejecting source distributions with `--only-binary=:all:` where compatible. - Run installation and testing with minimal filesystem permissions, no unnecessary credentials, and restricted network access. - Check subprocess return codes and terminate safely when installation fails instead of continuing in a partially modified environment. - Maintain dependency scanning and provenance checks for approved packages. - Keep test-runner dependencies separate from target dependencies and avoid overwriting project dependency files.
Vulnerability Patterns
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Behavioral ASTexec() Call, eval() Call, Dynamic Import
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Findings (8)

Context-Inappropriate Capability

High
Confidence
98% confidence
Finding
The skill's core behavior is to install dependencies and execute tests against an arbitrary resolved path, which effectively grants code execution on untrusted project contents. Because the task parameter can point to any accessible directory and the code performs dynamic execution with minimal validation, the skill is especially dangerous in an agent context.

Context-Inappropriate Capability

Medium
Confidence
90% confidence
Finding
The skill creates a tests directory and injects a smoke test into the target project, modifying user-controlled code without warning. This can corrupt repositories, interfere with existing test suites, and create an avenue for unexpected behavior when combined with subsequent execution.

Missing User Warnings

Medium
Confidence
95% confidence
Finding
The skill silently creates test and dependency-related files in the target project without notifying the user. In an automation context, unannounced writes are dangerous because they can alter repository state, break builds, or hide malicious persistence among normal-looking test artifacts.

subprocess module call

Medium
Category
Dangerous Code Execution
Content
t.exists() or (print(json.dumps({"skill":"AI_AutoTester","status":"error","error":f"Target path not found: {t}"},ensure_ascii=False,indent=2)),sys.exit(1))
(td:=t/"tests").mkdir(parents=True,exist_ok=True)
(tf:=td/"test_smoke.py").exists() or tf.write_text('from app.main import app\nfrom fastapi.testclient import TestClient\n\ndef test_root():\n c=TestClient(app)\n r=c.get("/")\n assert r.status_code==200\n',encoding="utf-8")
(r1:=t/"requirements.txt").exists() and subprocess.run(["python","-m","pip","install","-q","-r",str(r1)],cwd=str(t),check=False)
(t/"requirements-test.txt").write_text("pytest==8.3.2\nhttpx==0.27.2\n",encoding="utf-8")
subprocess.run(["python","-m","pip","install","-q","-r",str(t/"requirements-test.txt")],cwd=str(t),check=False)
r=subprocess.run(["python","-m","pytest","-q"],cwd=str(t),capture_output=True,text=True)
Confidence
96% confidence
Finding
This command installs dependencies from the target project's requirements.txt, which is fully controlled by the selected repository. Installing arbitrary packages can trigger execution of malicious setup/install hooks or pull attacker-chosen code, making this an explicit arbitrary code execution and supply-chain vulnerability.

Missing User Warnings

Medium
Confidence
96% confidence
Finding
The skill installs packages via pip without any warning or consent, exposing the environment to untrusted dependency execution and system changes. In this context, package installation is especially risky because it is driven by repository contents and occurs before any trust decision is made.

subprocess module call

Medium
Category
Dangerous Code Execution
Content
(tf:=td/"test_smoke.py").exists() or tf.write_text('from app.main import app\nfrom fastapi.testclient import TestClient\n\ndef test_root():\n c=TestClient(app)\n r=c.get("/")\n assert r.status_code==200\n',encoding="utf-8")
(r1:=t/"requirements.txt").exists() and subprocess.run(["python","-m","pip","install","-q","-r",str(r1)],cwd=str(t),check=False)
(t/"requirements-test.txt").write_text("pytest==8.3.2\nhttpx==0.27.2\n",encoding="utf-8")
subprocess.run(["python","-m","pip","install","-q","-r",str(t/"requirements-test.txt")],cwd=str(t),check=False)
r=subprocess.run(["python","-m","pytest","-q"],cwd=str(t),capture_output=True,text=True)
print(json.dumps({"skill":"AI_AutoTester","status":"ok" if r.returncode==0 else "failed","task":a.task,"target":str(t),"returncode":r.returncode,"stdout":r.stdout[-4000:],"stderr":r.stderr[-2000:],"timestamp":datetime.now(UTC).isoformat().replace("+00:00","Z")},ensure_ascii=False,indent=2))
sys.exit(0 if r.returncode==0 else 2)
Confidence
89% confidence
Finding
This subprocess call installs packages into an arbitrary user-selected project directory. Even though shell injection is not present, running pip against project-controlled dependency files can execute untrusted package installation code and alters the environment without confirmation, creating a real code-execution and supply-chain risk.

subprocess module call

Medium
Category
Dangerous Code Execution
Content
(r1:=t/"requirements.txt").exists() and subprocess.run(["python","-m","pip","install","-q","-r",str(r1)],cwd=str(t),check=False)
(t/"requirements-test.txt").write_text("pytest==8.3.2\nhttpx==0.27.2\n",encoding="utf-8")
subprocess.run(["python","-m","pip","install","-q","-r",str(t/"requirements-test.txt")],cwd=str(t),check=False)
r=subprocess.run(["python","-m","pytest","-q"],cwd=str(t),capture_output=True,text=True)
print(json.dumps({"skill":"AI_AutoTester","status":"ok" if r.returncode==0 else "failed","task":a.task,"target":str(t),"returncode":r.returncode,"stdout":r.stdout[-4000:],"stderr":r.stderr[-2000:],"timestamp":datetime.now(UTC).isoformat().replace("+00:00","Z")},ensure_ascii=False,indent=2))
sys.exit(0 if r.returncode==0 else 2)
Confidence
95% confidence
Finding
Running pytest in an arbitrary target directory executes project code, test hooks, plugins, and import-time side effects under the agent's privileges. In this skill, the path is attacker-controllable via --task, so the subprocess becomes a direct arbitrary code execution mechanism rather than a simple test runner.

Natural-Language Policy Violations

Low
Confidence
91% confidence
Finding
The file mixes English headings with Chinese content for the purpose and notes, which effectively constrains core skill information to a specific language. Under the policy, forcing a language without user opt-in or a justified locale constraint is a natural-language policy violation.

Static analysis

No suspicious patterns detected.