OpenClaw Desktop Control

Remote desktop control and automation. Capture screenshots, control mouse and keyboard, automate UI interactions. Supports VNC, RDP, and local desktop enviro...

Security Scan

The skill generally matches a desktop-automation purpose, but the package claims features and environment usage that do not line up with the actual files and registry metadata — and it grants broad local control (screenshots, file read/write, simulated input) which combined with autonomous invocation is risky.

Purpose & Capability

The README and SKILL.md claim VNC/RDP support and list vnc_client.py/rdp_client.py/local_display.py in the architecture, but the repository manifest only contains lib/desktop_controller.py and no vnc/rdp client modules. SKILL.md also documents environment variables (DESKTOP_HOST, DESKTOP_PASSWORD, etc.) but the registry metadata lists no required env vars. These mismatches indicate the advertised capabilities are not fully implemented or the manifest is incomplete.

Instruction Scope

Runtime instructions and examples enable taking screenshots (including base64 output), reading/writing arbitrary desktop files, and simulating keyboard/mouse input. The integration example explicitly captures screenshots via subprocess and embeds them for 'AI analysis' (i.e., prints base64 to stdout). While consistent with desktop automation, these instructions allow capture and potential exfiltration of sensitive screen contents and files, and they reference environment variables not declared in the skill metadata.

Install Mechanism

No installer/remote download is used (instruction-only with included Python files). Dependencies are listed in requirements.txt and system package recommendations are standard (scrot, xdotool, cliclick). No high-risk remote install URLs were found.

Credentials

The SKILL.md suggests using DESKTOP_HOST, DESKTOP_PORT, DESKTOP_PASSWORD and DESKTOP_TYPE, but the skill metadata declares no required env vars. Requesting or recommending secrets without declaring them is a transparency issue. The skill can read and write arbitrary files on the host (file read/write commands), which is proportionate for a desktop-control tool but increases risk if the skill is enabled broadly or given autonomous access.

Persistence & Privilege

always:false (normal) and disable-model-invocation:false (normal), but because the skill can take screenshots, simulate input and read/write files, allowing autonomous invocation expands blast radius. There is no install-time persistence, but agent-side autonomous use should be restricted and reviewed before enabling on sensitive hosts.

What to consider before installing

This skill appears to be a desktop-automation tool, but there are red flags you should consider before installing: - Mismatched claims: SKILL.md advertises VNC/RDP support and files (vnc_client.py, rdp_client.py) that are not present in the package. Treat those advertised network features as unimplemented until verified. - Undeclared secrets: The docs recommend environment variables (DESKTOP_HOST, DESKTOP_PASSWORD) but the registry metadata doesn't declare or require them. Don't set credentials for this skill unless you confirm how/where they will be used and stored. - Powerful local privileges: The skill can capture screenshots, read/write arbitrary files, and simulate keyboard/mouse input. These are expected for desktop control but also allow stealthy data capture or unintended command execution (e.g., typing into a terminal). Only run on machines where you trust the skill and its operator. - Autonomous invocation risk: If you allow the agent to call this skill autonomously, it could take screenshots or manipulate the desktop without further prompts. If you need this functionality, restrict the skill to non-production, isolated environments and limit which agents/contexts can invoke it. Recommendations: - Review the full source (desktop_controller.py and scripts/main.py) to confirm implementations of file read/write and any networked clients before enabling. - If you need remote VNC/RDP support, request the missing modules or a version that actually implements them. - Do not provide real credentials or enable the skill on sensitive systems until you confirm where credentials are used and stored. - Prefer running this in a sandbox/VM and monitor stdout/network traffic the first time it runs to detect unexpected exfiltration.

Desktop Control - 远程桌面控制

基于 Bytebot Computer Use 模式实现的桌面自动化控制工具。

Version: 1.0.0
Features: 截图、鼠标控制、键盘输入、应用管理、文件操作

Purpose

让 OpenClaw 能够:

• 查看远程/本地桌面状态 (截图)
• 控制鼠标和键盘进行 UI 交互
• 自动化执行桌面任务
• 与虚拟机、容器、远程服务器交互

Quick Start

1. 连接桌面环境

# 连接到 VNC 桌面
python3 scripts/main.py connect --host localhost --port 5900 --password secret

# 连接到本地桌面 (Linux)
python3 scripts/main.py connect --local

2. 基本操作

# 截图
python3 scripts/main.py screenshot --output /tmp/screen.png

# 移动鼠标
python3 scripts/main.py mouse move --x 500 --y 300

# 点击
python3 scripts/main.py mouse click --x 500 --y 300 --button left

# 输入文本
python3 scripts/main.py type "Hello World"

# 按键
python3 scripts/main.py key press --keys ctrl,c

3. 自动化脚本

# 执行脚本
python3 scripts/main.py script examples/open_browser.txt

Installation

Requirements

# Ubuntu/Debian
sudo apt-get install python3-pil python3-xlib scrot

# macOS
brew install pillow

# Python dependencies
pip3 install -r requirements.txt

Configuration

# 设置环境变量
export DESKTOP_HOST=localhost
export DESKTOP_PORT=5900
export DESKTOP_PASSWORD=secret
export DESKTOP_TYPE=vnc  # vnc, rdp, local

Commands

screenshot - 截图

# 基本截图
python3 scripts/main.py screenshot

# 保存到文件
python3 scripts/main.py screenshot --output /path/to/save.png

# 指定区域
python3 scripts/main.py screenshot --region "100,100,800,600"

# 返回 base64 (用于 AI 分析)
python3 scripts/main.py screenshot --base64

mouse - 鼠标控制

# 移动
python3 scripts/main.py mouse move --x 500 --y 300

# 点击
python3 scripts/main.py mouse click --x 500 --y 300
python3 scripts/main.py mouse click --button right  # 右键
python3 scripts/main.py mouse click --clicks 2      # 双击

# 拖拽
python3 scripts/main.py mouse drag --from "100,100" --to "500,500"

# 滚动
python3 scripts/main.py mouse scroll --direction down --amount 3

# 获取位置
python3 scripts/main.py mouse position

keyboard - 键盘控制

# 输入文本
python3 scripts/main.py keyboard type "Hello World"

# 按键 (支持组合键)
python3 scripts/main.py keyboard press --keys ctrl,alt,t  # 打开终端
python3 scripts/main.py keyboard press --keys ctrl,c      # 复制
python3 scripts/main.py keyboard press --keys ctrl,v      # 粘贴

# 按住/释放
python3 scripts/main.py keyboard hold --key shift
python3 scripts/main.py keyboard release --key shift

app - 应用管理

# 打开应用
python3 scripts/main.py app open --name firefox
python3 scripts/main.py app open --name terminal
python3 scripts/main.py app open --name vscode

# 关闭应用
python3 scripts/main.py app close --name firefox

# 切换到桌面
python3 scripts/main.py app desktop

file - 文件操作

# 读取文件 (从桌面环境)
python3 scripts/main.py file read --path /home/user/document.txt

# 写入文件
python3 scripts/main.py file write --path /home/user/hello.txt --content "Hello"

# 截图并 OCR 识别文字
python3 scripts/main.py file ocr --region "100,100,400,200"

automation - 自动化

# 执行脚本文件
python3 scripts/main.py automation run --script script.txt

# 录制操作
python3 scripts/main.py automation record --output script.txt

# 等待元素出现 (基于图像匹配)
python3 scripts/main.py automation wait --template button.png --timeout 10

Script Format

创建自动化脚本文件 (script.txt):

# 注释以 # 开头
screenshot
wait 1000
mouse move 500 300
mouse click
wait 500
type "Hello World"
key press return
wait 1000
screenshot

API Usage

作为 Python 库使用:

from desktop_controller import DesktopController

# 初始化
controller = DesktopController(host="localhost", port=5900)

# 截图
screenshot = controller.screenshot()

# 鼠标操作
controller.mouse_move(500, 300)
controller.mouse_click(500, 300)

# 键盘操作
controller.type_text("Hello World")
controller.key_press(["ctrl", "c"])

# 关闭
controller.disconnect()

Integration with OpenClaw

在 Skill 中调用:

import subprocess

def analyze_desktop():
    # 截图
    result = subprocess.run(
        ["python3", "skills/desktop-control/scripts/main.py", 
         "screenshot", "--base64"],
        capture_output=True, text=True
    )
    screenshot_base64 = result.stdout.strip()
    
    # 发送给 AI 分析
    return f"![Desktop](data:image/png;base64,{screenshot_base64})"

def click_element(x, y):
    subprocess.run([
        "python3", "skills/desktop-control/scripts/main.py",
        "mouse", "click", "--x", str(x), "--y", str(y)
    ])

Architecture

desktop-control/
├── scripts/
│   └── main.py              # CLI 入口
├── lib/
│   ├── __init__.py
│   ├── desktop_controller.py # 核心控制器
│   ├── vnc_client.py        # VNC 协议实现
│   ├── rdp_client.py        # RDP 协议实现
│   ├── local_display.py     # 本地显示控制
│   └── image_matcher.py     # 图像匹配
├── templates/               # 图像模板示例
├── examples/                # 脚本示例
│   ├── open_browser.txt
│   └── login_form.txt
└── requirements.txt

Use Cases

1. 远程服务器管理 - 通过 VNC 查看和操作服务器桌面
2. UI 自动化测试 - 自动化测试桌面应用
3. IoT 设备控制 - 控制带屏幕的嵌入式设备
4. 游戏自动化 - 自动化游戏操作 (不推荐用于在线游戏)
5. 数据录入 - 自动化表单填写

Security Notes

⚠️ 重要安全提示:

• 不要在生产环境使用弱密码
• 建议通过 VPN/SSH 隧道连接
• 避免在公共网络暴露 VNC/RDP 端口
• 敏感操作建议在本地执行

License

MIT License - 基于 Bytebot 的 Computer Use 模式实现