Back to skill

Security audit

UI Element Ops

Security checks for vulnerabilities and agentic risk

Overview

The skill is mostly coherent as a screenshot parser and desktop automation helper, but it needs Review because it can control the user’s desktop and includes an under-scoped shell command hook plus mutable third-party setup.

Install only if you are comfortable with a skill that can capture your screen and operate your active desktop session. Review or remove the wait --refresh-cmd shell hook, prefer dry-run/list/find before clicks or typing, avoid using it around sensitive windows, and pin or verify the OmniParser repository, model files, and Python dependencies before running bootstrap.

Vulnerability Patterns
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (2)

T09 · Insecure Skill Coding Practices

Error
Location
scripts/operate_ui.py:315
Finding
Shell Command Injection Through the Wait Refresh Command## Vulnerability Details **File Location**: `scripts/operate_ui.py:315-322` **Vulnerability Type**: Arbitrary shell command execution through unvalidated CLI input **Risk Level**: High ### Vulnerable Code ```python def run_refresh_command(cmd: str, timeout: float, ignore_errors: bool) -> None: result = subprocess.run( cmd, shell=True, text=True, capture_output=True, timeout=timeout, ) if result.returncode == 0: return ``` The command originates from the command-line option defined at `scripts/operate_ui.py:764-768`: ```python wait_p.add_argument( "--refresh-cmd", default=None, help="Optional shell command run before each poll (e.g. re-parse screenshot).", ) ``` ### Technical Analysis The value supplied through `--refresh-cmd` is passed directly to `subprocess.run()` with `shell=True`. The operating-system shell therefore interprets metacharacters, substitutions, pipelines, redirections, and command separators contained in the value. No command allowlist, argument separation, escaping, or validation is applied. Although this option is exposed as a command-line feature, it becomes a command-injection boundary when an AI Agent constructs it from untrusted task instructions, document content, OCR output, UI text, or another externally influenced source. The command is executed before every polling attempt in the `wait` workflow, so a malicious command may also run repeatedly until the wait condition succeeds or times out. ### Attack Path 1. An attacker places a malicious instruction in content that the Agent processes, such as UI text, a document, or task-supplied parameters. 2. The content persuades or causes the Agent to pass an attacker-controlled string to `operate_ui.py wait --refresh-cmd`. 3. The `wait` command passes that string to `run_refresh_command()`. 4. `subprocess.run()` invokes the string through the system shell because `shell=True` is enabled. 5. Shell meta ...[truncated 1048 chars]
Remediation
## Remediation Suggestions 1. Remove `shell=True` and execute commands as an explicit argument array: ```python subprocess.run( command_arguments, shell=False, text=True, capture_output=True, timeout=timeout, check=False, ) ``` 2. Prefer implementing screenshot refresh and parsing directly in Python rather than accepting a generic command. 3. If configurable refresh behavior is required, expose separate typed options for the executable, image path, output path, and other supported arguments. 4. Restrict executable selection to a small allowlist of reviewed local programs. 5. Reject shell metacharacters and command-substitution syntax as defense in depth, but do not treat filtering as a substitute for removing the shell. 6. Require explicit user confirmation before executing any externally supplied refresh operation. 7. Run refresh operations in a restricted environment with minimal filesystem and network access. 8. Add tests verifying that values containing separators, substitutions, redirects, and pipelines cannot cause additional commands to run.

T08 · Insecure Dependencies

Warning
Location
scripts/bootstrap_omniparser_env.sh:22
Finding
Mutable and Insufficiently Pinned Third-Party Installation Supply Chain## Vulnerability Details **File Location**: `scripts/bootstrap_omniparser_env.sh:22-59` **Vulnerability Type**: Unpinned dependencies and mutable remote code/model retrieval **Risk Level**: Medium ### Vulnerable Code ```bash "$VENV_PATH/bin/python" -m pip install --upgrade pip "$VENV_PATH/bin/pip" install \ pillow \ requests \ openai \ numpy==1.26.4 \ matplotlib \ torch \ torchvision \ easyocr \ supervision==0.18.0 \ ultralytics==8.3.70 \ "transformers==4.49.0" \ accelerate \ timm \ einops==0.8.0 \ opencv-python \ pyautogui \ screeninfo \ huggingface_hub if [ ! -d "$OMNIPARSER_DIR/.git" ]; then git clone --depth 1 https://github.com/microsoft/OmniParser "$OMNIPARSER_DIR" fi mkdir -p "$OMNIPARSER_DIR/weights" /tmp/hf /tmp/xdg HF_HOME=/tmp/hf XDG_CACHE_HOME=/tmp/xdg \ "$VENV_PATH/bin/hf" download microsoft/OmniParser-v2.0 \ icon_detect/train_args.yaml \ icon_detect/model.pt \ icon_detect/model.yaml \ icon_caption/config.json \ icon_caption/generation_config.json \ icon_caption/model.safetensors \ --local-dir "$OMNIPARSER_DIR/weights" if [ -d "$OMNIPARSER_DIR/weights/icon_caption" ] && [ ! -d "$OMNIPARSER_DIR/weights/icon_caption_florence" ]; then mv "$OMNIPARSER_DIR/weights/icon_caption" "$OMNIPARSER_DIR/weights/icon_caption_florence" fi ``` ### Technical Analysis Most Python packages are installed without exact version constraints or cryptographic hashes. The bootstrap process also upgrades `pip` to the latest available release. Consequently, identical bootstrap commands can install different code over time. The OmniParser repository is cloned from the repository's current default branch rather than a reviewed commit SHA or signed release. The Hugging Face download similarly omits a fixed revision and local checksum verification. This makes both remote sources mutable after the Skill package has been reviewed. The retrieved OmniParser repository is later inserted into `sys.path` a ...[truncated 2095 chars]
Remediation
## Remediation Suggestions 1. Pin all direct and transitive Python dependencies using a reviewed lockfile. 2. Install with cryptographic hash enforcement, such as `pip install --require-hashes -r requirements.txt`. 3. Avoid unconditional `pip` upgrades; pin the installer version used by the environment. 4. Replace the default-branch clone with a reviewed immutable commit: ```bash git clone https://github.com/microsoft/OmniParser "$OMNIPARSER_DIR" git -C "$OMNIPARSER_DIR" checkout --detach "<reviewed-commit-sha>" ``` 5. Verify the checked-out commit and, where available, validate a signed release or tag. 6. Pass an immutable reviewed revision to the Hugging Face download command. 7. Maintain expected SHA-256 checksums for every downloaded model and configuration file and verify them before use. 8. Use an internal artifact mirror or approved package index where feasible. 9. Run installation and model parsing inside a sandbox with restricted filesystem and network access. 10. Add automated dependency scanning and periodically update pins through a controlled review process.
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • System Prompt LeakageDirect Leakage, Indirect Extraction, Tool-Based Exfiltration
  • Tool MisuseTool Parameter Abuse, Chaining Abuse, Unsafe Defaults
  • Behavioral ASTexec() Call, eval() Call, Dynamic Import
Findings (13)

Tp4

High
Category
MCP Tool Poisoning
Confidence
95% confidence
Finding
The implemented code clearly matches only the screenshot-parsing portion of the description. It loads an image file, runs OCR and OmniParser detection, normalizes element types and bounding boxes, and outputs JSON plus an optional overlay image. There is no code for controlling the desktop, simulating input, monitoring the screen over time, capturing screenshots, or calibrating coordinates across displays or DPI contexts. Because the declared purpose presents a broader dual capability—both parsing and operating the desktop UI—while this code chunk only implements parsing, the description materially overstates the actual behavior of the supplied code.

Context-Inappropriate Capability

High
Confidence
99% confidence
Finding
The polling helper is supposed to support UI element refresh, but it can run any shell command before each poll. That broad capability exceeds the stated UI-element operation scope and creates a command-execution primitive that could be abused for arbitrary local actions, repeated on a timer until timeout.

Tool Parameter Abuse

High
Category
Tool Misuse
Content
def run_refresh_command(cmd: str, timeout: float, ignore_errors: bool) -> None:
    result = subprocess.run(
        cmd,
        shell=True,
        text=True,
Confidence
99% confidence
Finding
Using subprocess.run with shell=True on a caller-supplied string is a classic tool-parameter-abuse issue. In an agent-integrated skill, an attacker can steer parameters toward arbitrary command execution, and because this sits in a wait loop it can repeatedly execute harmful commands during polling.

Direct Prompt Extraction

High
Category
System Prompt Leakage
Content
if not isinstance(value, list) or not all(isinstance(x, str) for x in value):
            raise ValueError(f"Invalid type rule for '{key}', expected list[str].")
        rules[key] = value
    return rules


def contains_any(text: str, keywords: List[str]) -> bool:
Confidence
85% confidence
Finding
Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Lp3

Medium
Category
MCP Least Privilege
Confidence
92% confidence
Finding
The skill exposes broad capabilities including shell, network, environment access, and file read/write, yet the manifest does not declare any explicit tool scope or permission boundaries. For a skill that can also drive the desktop UI, this increases the blast radius: an invoking agent may use more authority than users expect, enabling unintended file access, command execution, or data exfiltration through supporting scripts.

Missing User Warnings

Medium
Confidence
90% confidence
Finding
The skill instructs use of click, type, key, and hotkey actions without an explicit warning that these actions can modify system state, submit forms, trigger destructive shortcuts, or interact with sensitive applications. In a UI-automation skill, missing operator warnings and confirmation guidance makes accidental harmful execution more likely, especially when coordinates or OCR matching are imperfect.

Missing User Warnings

Medium
Confidence
92% confidence
Finding
The script automatically installs Python packages, clones a GitHub repository, and downloads model weights from Hugging Face without any explicit user consent prompt, integrity verification, or provenance pinning beyond a shallow clone and version pins for some dependencies. In a skill that controls desktop UI and processes screenshots, silently bootstrapping network-fetched code and models increases supply-chain and execution risk because the environment gains powerful local interaction capabilities.

Missing User Warnings

Medium
Confidence
91% confidence
Finding
This path exposes shell execution without any built-in warning, confirmation gate, or safety control, which increases the chance that users or agent workflows invoke dangerous commands unintentionally. In this skill context, that omission matters because the feature is embedded inside a benign-seeming wait operation rather than an explicitly hazardous admin command.

subprocess module call

Medium
Category
Dangerous Code Execution
Content
def run_refresh_command(cmd: str, timeout: float, ignore_errors: bool) -> None:
    result = subprocess.run(
        cmd,
        shell=True,
        text=True,
Confidence
98% confidence
Finding
The wait command accepts an arbitrary --refresh-cmd string and executes it with subprocess.run(..., shell=True). In an agent skill, this enables shell execution unrelated to UI automation and can directly lead to arbitrary command execution, data access, or system modification if a caller or upstream prompt influences that parameter.

Missing User Warnings

Medium
Confidence
88% confidence
Finding
The screenshot command captures the full screen or a selected region and saves it to disk, which can collect sensitive on-screen data and persist it in a file. The code prints the output path after saving, but it does not include any warning or disclosure about potential capture of sensitive information.

Dynamic import via __import__()

Medium
Category
Dangerous Code Execution
Content
This stub allows the import path to succeed without installing paddleocr.
    """
    try:
        __import__("paddleocr")
        return
    except Exception:
        pass
Confidence
75% confidence
Finding
Dynamic __import__() can load arbitrary modules at runtime, bypassing static analysis and potentially importing malicious code.

Missing User Warnings

Low
Confidence
76% confidence
Finding
The calibrate command creates directories and writes a coord_profile.json file to disk, but there is no explicit warning that this command persists data in the current working directory by default. While this write is part of the command's purpose, the default output behavior may still be non-obvious to users invoking calibration casually.

Missing User Warnings

Low
Confidence
83% confidence
Finding
The script automatically derives and writes an output JSON file and overlay PNG path from the input image name. While the usage line documents the arguments, there is no explicit user-facing warning, prompt, or comment calling out that running the script will create or overwrite files at those paths.

Static analysis

No suspicious patterns detected.